Robots.txt Checker
Fetch a site's robots.txt, find syntax problems line by line, and see which AI search, user-triggered and training crawlers it allows or blocks for a page.
What does the Robots.txt Checker test?
- Whether the file can be read. A missing file, a server error, or an HTML page served in place of the file each mean something different to crawlers.
- Syntax. Rules before any User-agent line, misspelled directives, paths that don't start with a slash, relative Sitemap URLs, and directives that Google ignores, such as noindex and Crawl-delay.
- Rules per crawler. For each known AI crawler, whether the URL you entered is allowed, and whether its own group or the
User-agent: *group decides.
This tool reads the file only. It does not send test requests with crawler user agents, so it can't tell you whether a firewall blocks a crawler. The AI Crawler Checker does.
What next?
Once your robots.txt is right, run the full Georate check to see whether the page itself is ready to be cited by AI search engines.
Frequently asked questions
How do I check my robots.txt file?
Enter your site's address above. The checker fetches /robots.txt from that site, reports whether it can be read, lists syntax problems by line number, and shows which AI crawlers it allows or blocks for the page you entered. To test a specific page, enter that page's full URL.
What is the difference between blocking GPTBot and OAI-SearchBot?
They are two crawlers with separate robots.txt names. GPTBot collects training data. OAI-SearchBot builds the index behind ChatGPT search. Blocking GPTBot doesn't remove a site from ChatGPT search results, and blocking OAI-SearchBot does keep its pages out of them.
Where does robots.txt have to be?
At the root of the host, for example https://example.com/robots.txt. It applies only to that protocol, host and port, so a subdomain such as blog.example.com needs its own file.
Does Disallow remove a page from Google?
No. Disallow stops crawlers from fetching a page, but a blocked URL can still be indexed if other pages link to it. To keep a page out of search results, leave it crawlable and add a noindex robots meta tag or an X-Robots-Tag header. Crawlers can't see a noindex on a page they are not allowed to fetch.
What happens if my robots.txt returns an error?
A missing file (404) means every URL is allowed. A server error (5xx) is different: Google treats an unreachable robots.txt as disallowing everything for a while, and other crawlers may behave the same way. The checker reports both.