About our crawler

What it fetches

RSS/Atom feeds that users have subscribed to, and HTML pages used to detect updates on sites that do not publish a feed. Only publicly accessible pages are fetched; authenticated areas are never accessed.

robots.txt

For pseudo-feed generation from HTML, robots.txt is fully honoured (both our User-Agent and * are evaluated, and the stricter one wins; Crawl-delay is respected). Fetching an ordinary RSS/Atom feed is not stopped by robots.txt. Many sites disallow their subscription feeds there, and honouring that would break the subscription for the reader. To stop fetching, get in touch using the address below.

Frequency

Up to two concurrent connections per host, at least one second apart. Fetch intervals adapt between 5 minutes and 24 hours based on how often a feed actually updates. Conditional GET (If-None-Match / If-Modified-Since) is always sent.

If you return 429 / 503

We back off exponentially and stop fetching that host automatically if it persists.

To stop listing or fetching

Pseudo-feed generation stops automatically once you add a Disallow for the URL in your robots.txt. For removal from public pages, use the form below.

To stop us fetching the RSS/Atom feed itself, write to us: support@freefeed.app