Enter any page address. MrKeyword reads the site’s live robots.txt and shows which AI crawlers may read the page, and the exact line that decides it, from OpenAI, Anthropic, Perplexity, Google, Apple and ByteDance, and CCBot, plus whether the site has a working llms.txt.
Free, no account needed. MrKeyword fetches the page live and shows only what it found.
Built on first-party data from Google and Bing
An AI crawler checker reads a site’s live robots.txt and tells you, crawler by crawler, whether the bots behind ChatGPT, Claude, Perplexity, Gemini and other AI products may read a given page, and which Allow or Disallow line decides it.
AI companies run two kinds of crawler. Search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so their AI search can show and cite them; block one and that search can no longer read your page. Training crawlers such as GPTBot, ClaudeBot and CCBot collect pages to train future models; blocking one keeps your content out of training but does not stop AI search citing you. Many sites block both by accident with one “block AI” rule.
The checker uses the same matching rules as our robots.txt tester (RFC 9309, the standard Google follows): a crawler obeys the group with its own name, otherwise the * group, and the longest matching rule wins. Googlebot and Bingbot are shown as a baseline. It also fetches /llms.txt and checks its format and the first five links it lists.
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider, each labelled training or search.
Status, content type, size, whether it opens with a # heading, how many markdown links it lists, and the first five links checked live.
The fictional northwind-plumbing.example installed a plugin that added one group to its robots.txt to keep its pages out of AI training, and it publishes an llms.txt listing five pages. Checking https://northwind-plumbing.example/services/ gives:
7 of 10 AI crawlers can read this page
https://northwind-plumbing.example/robots.txt · HTTP 200 · tested for /services/
| Crawler | Company | Used for | This page | Rule |
|---|---|---|---|---|
| GPTBotCrawls content that may be used to train OpenAI’s models. ChatGPT search is a separate setting. | OpenAI | Training | Blocked | Disallow: /line 8 |
| OAI-SearchBotFinds pages to show and cite in ChatGPT search. | OpenAI | Search and citations | Allowed | No rule matchesNo group of its own: follows the * group |
| ChatGPT-UserVisits a page when a ChatGPT user’s request needs it. OpenAI says robots.txt rules may not apply to it. | OpenAI | Fetches for a user | Allowed | No rule matchesNo group of its own: follows the * group |
| ClaudeBotCollects web content that could contribute to Anthropic’s model training. | Anthropic | Training | Allowed | No rule matchesNo group of its own: follows the * group |
| Claude-SearchBotCrawls to improve Claude’s search results, where pages are shown and cited. | Anthropic | Search and citations | Allowed | No rule matchesNo group of its own: follows the * group |
| PerplexityBotFinds and links pages in Perplexity’s answers. Perplexity says it is not used for training. | Perplexity | Search and citations | Blocked | Disallow: /line 8 |
| Google-ExtendedNo crawler of its own: controls whether pages Googlebot fetched may train Gemini models and ground Gemini Apps. No effect on Google Search, AI Overviews or AI Mode. | Training | Allowed | No rule matchesNo group of its own: follows the * group | |
| Applebot-ExtendedNo crawler of its own: controls whether pages Applebot fetched may train Apple’s AI models. Apple search is not affected. | Apple | Training | Allowed | No rule matchesNo group of its own: follows the * group |
| CCBotBuilds an open archive of the web that anyone can download, a common source of AI training data. | Non-profit web archive | Training | Blocked | Disallow: /line 8 |
| BytespiderByteDance’s crawler, generally listed as collecting data for its AI models. ByteDance publishes little about it. | ByteDance | Training | Allowed | No rule matchesNo group of its own: follows the * group |
| GooglebotGoogle Search, including AI Overviews and AI Mode. | Search engine (baseline) | Allowed | No rule matchesNo group of its own: follows the * group | |
| BingbotBing search. Microsoft controls Copilot’s use of pages with meta tags, not a separate token. | Microsoft | Search engine (baseline) | Allowed | No rule matchesNo group of its own: follows the * group |
Search or training? Blocking a search crawler (OAI-SearchBot, Claude-SearchBot, PerplexityBot) stops that AI search reading the page, so it can no longer show or cite it. Blocking a training crawler (GPTBot, ClaudeBot, CCBot) only keeps the page out of future model training; ChatGPT, Claude and Perplexity search can still cite it. Google-Extended has no effect on Google Search, AI Overviews or AI Mode, which follow Googlebot’s rules, but it does also cover grounding in Gemini Apps.
# Northwind Plumbing| Status | Link |
|---|---|
| 200 | https://northwind-plumbing.example/services/Services |
| 200 | https://northwind-plumbing.example/boiler-repairBoiler repair |
| 404 | https://northwind-plumbing.example/spring-offer-2024Spring offer |
| 200 | https://northwind-plumbing.example/contactContact |
| 200 | https://northwind-plumbing.example/blog/Blog |
llms.txt is a proposed plain-text guide to a site for language models. It doesn’t control crawling (robots.txt does), and Google says Google Search ignores it.
User-agent: * Disallow: /cart/ # Added by a plugin to block AI training User-agent: GPTBot User-agent: CCBot User-agent: PerplexityBot Disallow: / Sitemap: https://northwind-plumbing.example/sitemap.xml
Example result for a fictional website (northwind-plumbing.example). It is produced by the checker's own code from the setup described above; run the checker for your own pages.
GPTBot and CCBot are training crawlers, so blocking them did what the owner wanted: future model training leaves the site out, and ChatGPT search can still read and cite it through OAI-SearchBot. PerplexityBot is different. It is Perplexity’s search crawler, so the same group quietly removed the site from Perplexity’s answers. The fix is to move User-agent: PerplexityBot out of the blocking group that ends in Disallow: / on line 8 and, if the cart should stay private, give it its own group with Disallow: /cart/, because a named group replaces the * group. In llms.txt one sampled link, the 2024 offer page, now answers 404: point it at a current page or take it out. Our guide to allowing or blocking AI crawlers has copyable robots.txt files for each choice, and what llms.txt is explains the file’s format.
A “block AI” group often lists PerplexityBot or OAI-SearchBot next to GPTBot. Split it: training crawlers in a Disallow group, search crawlers allowed.
AI Overviews and AI Mode are part of Google Search and use Googlebot. Blocking Googlebot removes the page from Google entirely; use nosnippet or max-snippet on specific pages instead.
Google-Extended has no effect on Google Search. It controls Gemini training and grounding only.
A crawler with its own group ignores the * group, so Disallow: /cart/ under * no longer applies to it. Copy those lines into each named group.
Some sites answer every unknown address with their home page and HTTP 200. The checker flags an HTML answer, which means there is no real llms.txt.
Check them whenever robots.txt changes, and whenever you want to know why an AI search never cites your site.
Only if you don’t want OpenAI to train future models on your pages. Blocking GPTBot does not remove you from ChatGPT search: that is OAI-SearchBot, a separate setting. Most businesses that want customers to find them in ChatGPT allow OAI-SearchBot whatever they decide about GPTBot.
No. Google-Extended is not a crawler; it is a robots.txt token that controls whether pages Googlebot fetched may be used to train Gemini models and to ground Gemini Apps. Google says it has no effect on inclusion or ranking in Google Search, and AI Overviews and AI Mode follow your Googlebot rules.
llms.txt is a proposed Markdown file at the root of a site that lists its most useful pages for language models. Google says Google Search ignores it, so it neither helps nor harms rankings there; some AI tools and agents may read it. It is a small optional extra, and it never replaces robots.txt.
Give the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) a group with Allow: /, and the training crawlers and tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) a group with Disallow: /. A named group replaces the * group for that crawler, so copy any private-path rules into each named group you allow.
This checker reads the live file, so it shows the change at once. Crawlers cache robots.txt: Google generally refreshes it within 24 hours, and OpenAI says ChatGPT search can take about 24 hours to adjust. Blocking a training crawler only affects what it collects from then on.