About our crawler
FreeFeedBot/1.0 (+https://freefeed.app/bot)
What it fetches
RSS/Atom feeds that users have subscribed to, and HTML pages used to detect updates on sites that do not publish a feed. Only publicly accessible pages are fetched; authenticated areas are never accessed.
robots.txt
For pseudo-feed generation from HTML, robots.txt is fully honoured (both our User-Agent and * are evaluated, and the stricter one wins; Crawl-delay is respected). For ordinary RSS/Atom feeds, fetching stops when the feed URL itself is explicitly disallowed in robots.txt.
Frequency
One concurrent connection per host, at least one second apart. Fetch intervals adapt between 15 minutes and 24 hours based on how often a feed actually updates. Conditional GET (If-None-Match / If-Modified-Since) is always sent.
If you return 429 / 503
We back off exponentially and stop fetching that host automatically if it persists.
To stop listing or fetching
Adding a Disallow for the URL in your robots.txt stops fetching automatically — nothing needs to be requested from us. For anything else, such as removal from public pages, use the form below.
Request removal