RegensbotBot, the Regensbot crawler
RegensbotBot fetches a page when a Regensbot customer asks for it to be audited, from their AI assistant, the API or the dashboard. It reads that page, its robots.txt, its sitemaps and a few resources the page links to, and checks them against search engines' published guidelines.
A customer can also have RegensbotBot crawl a website they added to Regensbot. The crawl reads the website's robots.txt and sitemaps and follows its links on the website's own hosts. Of a page on another website that a crawled page names as its canonical or language version, it reads only the status code.
Regensbot keeps the results of these checks, not copies of your pages: for each page, facts such as its address, status code, title, description, headings, links and structured-data types, fingerprints of its text to find duplicates, and short excerpts quoted as evidence.
How to recognise it
Its requests carry this user agent:
RegensbotBot/1.0 (+https://app.regensbot.de/bot)To compare what search engines and visitors are served, some checks of a single-page audit repeat a request with the user agent of Googlebot, of Storebot-Google or of a browser. These requests come from Regensbot's address, not from Google. Site crawls always use RegensbotBot's user agent.
Limits
- One audit makes at most 120 requests and stops after 120 seconds.
- One customer audits a site at most 6 times an hour; all customers together at most 30 times an hour.
- A site crawl sends one request after the other, at most one a second to each host; on a website the customer has not verified as their own, at most one every two seconds.
- Each customer crawls a website that is not verified at most once a month, up to 25 pages. Scheduled crawls run only on verified websites, and only one crawl of a website runs at a time, across all customers.
- When your site answers 429 or 503, RegensbotBot stops at once and waits as long as Retry-After asks, up to 60 minutes.
How to opt out
Add this group to your robots.txt. RegensbotBot then fetches nothing from your site but robots.txt:
User-agent: RegensbotBot
Disallow: /The older name SEOAuditorBot works too. Narrower rules in that group, such as Disallow: /private/, keep site crawls away from the addresses they cover.
Crawls of a website whose owner has not verified it with Regensbot also follow the rules for every crawler (User-agent: *) when there is no group for RegensbotBot: Disallow: / there stops the crawl, and narrower rules skip the addresses they cover. Regensbot treats these rules as a reservation of rights under section 44b of the German Copyright Act (UrhG).
If a host's robots.txt cannot be read (a server error, status code 429 or no answer), a crawl fetches nothing else from that host unless the host belongs to a website its owner has verified with Regensbot. If it is the host of the crawl's start address, the crawl stops.
An audit of a single page that a customer asks for, and crawls of a website whose owner has verified it with Regensbot, do not read the rules for every crawler as addressed to RegensbotBot, because they check how search engines see the site. To keep RegensbotBot away from them, use a group for RegensbotBot.
One exception: when another website redirects RegensbotBot to yours, the page it lands on is fetched once before your robots.txt can be read. Regensbot then checks nothing on it.
Contact
To have your domain blocked completely, or to report a problem, write to us at the address in our imprint.