Googlebot, Bingbot and the AI bots.
Check GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot as well as search engines.
- Ten crawlers to choose from
- Fallback groups explained
How it decides
The same rules Google uses.
robots.txt groups rules by crawler. A crawler follows the group with its own name; Googlebot-Image falls back to Googlebot, and everything else falls back to the * group. Inside the group, the longest matching path wins, and Allow wins a tie.
The tester applies those rules (RFC 9309, the standard Google follows) to the live file and points to the line that matched. It also handles the status codes: no robots.txt means everything is allowed; a server error means Google stops crawling until it recovers.
Clear answers,
with the reason shown.
See why, not just yes or no.
The matched Allow or Disallow rule and its line number, the sitemaps listed, and crawlers blocked from the whole site.
- Matched rule and line
- Sitemaps in robots.txt
More free checkers
Other quick checks.
Questions
About the robots.txt tester.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt stops crawling, not indexing. A blocked page can still appear in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex tag.
Should I block AI crawlers?
It depends on whether you want AI assistants to read and cite your pages. Blocking GPTBot or ClaudeBot keeps your content out of those products; Google-Extended controls Gemini training, not Google Search.
Why does the tester say a crawler follows the * group?
Because the file has no group with that crawler’s name. Crawlers only follow the most specific group that names them, then the * group.