AI Crawler Checker
Check whether the crawlers behind ChatGPT, Perplexity, Claude, Google and Bing search can fetch a page on your site. It reads your robots.txt rules for each AI crawler and sends a test request with each search crawler's user agent to see whether your firewall lets it through.
What does the AI Crawler Checker test?
- robots.txt rules. For the exact URL you enter, whether each AI crawler is allowed, and which line or group decides it.
- A live request. One request to that URL for each of the five AI search crawlers, to catch firewalls and bot protection that block the crawler even though robots.txt allows it.
- The robots.txt file itself. Whether it can be fetched, and any syntax problems that make crawlers ignore rules.
Search, user-triggered and training crawlers
AI companies run different crawlers for different jobs, and each has its own robots.txt name. Search crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot build the index that AI search answers come from, so blocking them keeps your pages out of those answers. User-triggered fetchers such as ChatGPT-User and Claude-User load a page when someone asks an assistant about it. Training crawlers such as GPTBot and ClaudeBot collect data to train models, and blocking them is a separate choice that doesn't remove you from AI search.
What next?
To see every rule and syntax problem in your file, use the Robots.txt Checker. To check whether the page itself is ready to be cited (structure, structured data, rendering without JavaScript), run the full Georate check.
Frequently asked questions
How do I check if ChatGPT can crawl my website?
ChatGPT search uses OpenAI's OAI-SearchBot crawler. Enter a page URL above and look at the OAI-SearchBot row. If robots.txt allows it and the live request returns HTTP 200, neither your robots.txt rules nor your server are stopping that crawler. The result shows access, not whether ChatGPT will cite the page.
Does blocking GPTBot stop ChatGPT from finding my site?
No. OpenAI uses GPTBot to collect training data and OAI-SearchBot for ChatGPT search, and each follows its own robots.txt rules. A site can block GPTBot and still allow OAI-SearchBot. The checker lists training crawlers separately for that reason.
Why does robots.txt allow a crawler but the live request is blocked?
Usually a firewall, CDN or bot-protection rule is stopping requests that use the crawler's user agent, either with an error such as 403 or with a challenge page. robots.txt can't override that. Check the bot-management settings of your CDN and allow the verified crawlers from the IP ranges their operators publish.
Is the live request the same as the real crawler visiting?
No. The checker sends one request per search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, bingbot) with its public user agent, from Georate's network. Real crawlers come from the operators' own IP ranges, and sites that verify those ranges may treat the real crawler differently. Treat a blocked result as a lead to check, not as proof.
Does passing this check mean AI search engines will cite my page?
No. Crawler access is the first requirement: a page that can't be fetched can't be cited. Whether it is cited then depends on the page's content, structure and the engine's own ranking. The full Georate check scores those parts too.