GeorateBot

GeorateBot is the fetcher behind Georate. It visits a page only when a person enters that page's URL on Georate to check it. It does not crawl links, follow a schedule or collect pages for AI training.

User agent

Mozilla/5.0 (compatible; GeorateBot/1.0; +https://georate.me/bot)

Requests come from Cloudflare's network, so the IP addresses are Cloudflare's and are shared with other services. Match on the GeorateBot token, not on IP addresses.

What one check fetches

  • The page that was entered, following up to 5 redirects.
  • /robots.txt, /llms.txt and /llms-full.txt from the same site.
  • The first sitemap listed in robots.txt (or /sitemap.xml) and up to 2 child sitemaps.
  • If the HTML looks like it needs JavaScript, the page is also loaded once in a headless browser with the same user agent. The browser loads scripts and data requests but skips images, fonts, media and stylesheets.

Results are reused for 24 hours, so repeated checks of the same URL usually make no new requests. Each person can start only a few checks per minute.

AI crawler access test

To tell you whether a firewall blocks AI search crawlers, each check also requests the page once with each of these crawlers' public user agents. These requests come from Georate, not from the companies named, and will fail the verification those companies publish (reverse DNS or IP lists). If you block them, the report shows a warning and nothing else changes.

  • OAI-SearchBot
  • PerplexityBot
  • Claude-SearchBot
  • Googlebot
  • bingbot

robots.txt

A check is a request from a person, like a browser visit, so GeorateBot reads robots.txt to report on it but does not use it to decide whether to fetch the page. To stop checks of your site, block the user agent at your server, CDN or firewall instead.

How to block it

Return 403 to requests whose user agent contains GeorateBot. With nginx:

if ($http_user_agent ~* "GeorateBot") {
    return 403;
}

With a Cloudflare WAF custom rule (action: Block):

(http.user_agent contains "GeorateBot")

Checks of a blocked site then report that the page could not be fetched. If GeorateBot causes a problem on your site, email contact@georate.me. To remove an existing report, see the contact page.