- AI bots do three different jobs: search (finding pages to show and cite in answers), training (collecting data for future models) and user fetch (loading one page because a person asked).
- GPTBot vs OAI-SearchBot: GPTBot is OpenAI's training crawler; OAI-SearchBot decides whether you can appear in ChatGPT search. OpenAI says each setting is independent.
- To stay in AI search but opt out of training, allow
OAI-SearchBot,Claude-SearchBotandPerplexityBot, and disallowGPTBot,ClaudeBot,Google-Extended,Applebot-ExtendedandCCBot. - Google-Extended and Applebot-Extended are control tokens, not crawlers. Neither affects your Google or Apple search rankings.
- robots.txt is a request, not a lock. OpenAI and Perplexity say their user-triggered fetchers may not follow it.
The short answer
To block an AI crawler, add a group to the robots.txt file at the root of your domain with that crawler's user-agent token and Disallow: /. To allow one, either leave it out (bots fall back to your User-agent: * rules) or give it Allow: /. The hard part is not the syntax. It is knowing which token does what, because most AI companies now separate their search crawler from their training crawler, and blocking the search one removes you from their answers.
If you want people to find you through ChatGPT, Claude and Perplexity but do not want your content used to train models, block the training crawlers and leave the search crawlers alone. The copyable file for that is further down.
Search, training and user fetch: three different jobs
Each AI company's documentation splits its bots by purpose. Reading them side by side, three types emerge:
- Search crawlers visit pages so the assistant can show and link them in answers. Blocking one usually means you stop appearing in that product's search answers.
- Training crawlers collect content that may be used to train future models. Blocking one is a signal that your content should not be used for that. It does not, by itself, remove you from search answers.
- User-triggered fetchers load a single page because a person asked the assistant to, for example by pasting your URL. They are not crawling the web on their own, and some companies say robots.txt may not apply to them.
Two more things are not crawlers at all: Google-Extended and Applebot-Extended are tokens you write in robots.txt to tell Google and Apple how they may use pages their normal crawlers already fetched.
Every AI crawler, and what blocking it does
This table covers the tokens we could confirm on each owner's own documentation. The token is what you put after User-agent:; matching is case-insensitive.
AI crawler user-agent tokens
| Token | Owner | Type | What it does | If you block it |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | Search | Surfaces websites in ChatGPT's search features | Your pages are not shown in ChatGPT search answers, though they can still appear as navigational links |
GPTBot | OpenAI | Training | Crawls content that may be used to train OpenAI's foundation models | Signals your content should not be used for training. ChatGPT search is a separate setting |
ChatGPT-User | OpenAI | User fetch | Visits a page when a ChatGPT or Custom GPT user's request needs it | OpenAI says robots.txt rules may not apply, and it is not used to decide what appears in search |
ClaudeBot | Anthropic | Training | Collects web content that could contribute to training | Your future content is excluded from Anthropic's training datasets |
Claude-SearchBot | Anthropic | Search | Crawls to improve the quality of Claude's search results | Anthropic says this may reduce your visibility and accuracy in Claude's search results |
Claude-User | Anthropic | User fetch | Fetches pages when a Claude user asks a question that needs them | Claude cannot retrieve your pages for those questions, which may reduce your visibility in user-directed search |
PerplexityBot | Perplexity | Search | Surfaces and links websites in Perplexity's search results; Perplexity says it is not used for training foundation models | You stop being surfaced in Perplexity search results |
Perplexity-User | Perplexity | User fetch | Loads a page to answer a user's question and link to it | Perplexity says this fetcher generally ignores robots.txt, because a user requested it |
Googlebot | Search | Crawls for Google Search, including all Search features such as AI Overviews and AI Mode | You drop out of Google Search crawling entirely. Not a way to opt out of AI only | |
Google-Extended | Control token | No crawler of its own. Controls whether content Google crawls may be used to train future Gemini models and to ground Gemini Apps and Vertex AI | No effect on inclusion or ranking in Google Search | |
Applebot | Apple | Search | Apple's crawler for search features in Spotlight, Siri and Safari | Your pages may drop out of those Apple search features |
Applebot-Extended | Apple | Control token | Does not crawl. Decides whether pages Applebot fetched may train Apple's generative AI models | Pages can still appear in Apple search, and the rule is not used for ranking |
CCBot | Common Crawl | Open dataset | Builds Common Crawl's open web archive, which anyone can download and analyse | Common Crawl stops collecting your pages; it also offers an opt-out registry |
bingbot | Microsoft | Search | Bing's search crawler. Microsoft controls use of your pages in Copilot (formerly Bing Chat) with meta tags, not a separate token | You leave Bing search. To limit only Copilot, use Bing's meta tags instead (below) |
GPTBot vs OAI-SearchBot in one line. GPTBot is about training; OAI-SearchBot is about appearing in ChatGPT search. OpenAI says each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. If you allow both, OpenAI may use one crawl for both purposes. After a robots.txt change, OpenAI says search can take about 24 hours to adjust.
Copyable robots.txt examples
Pick one of the three files below. Each replaces the AI section of your robots.txt; keep any rules you already have for private paths. A group can list several User-agent lines that share the same rules.
1. Appear in AI search, opt out of AI training
# AI search crawlers: allowed, so you can be shown and cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# AI training crawlers and training controls: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot, bingbot and Applebot
User-agent: *
Allow: /Google-Extended also stops your content being used to ground Gemini Apps and Vertex AI answers. If being cited in Gemini matters more to you than training, remove that line. Google AI Overviews and AI Mode are part of Search and follow your Googlebot rules.2. Block all AI crawlers
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /3. Allow all crawlers, including AI
User-agent: *
Allow: /A named group replaces your * rules. Under the robots.txt standard, a crawler that finds a group with its own name ignores the User-agent: * group (RFC 9309). If your * group disallows /admin/ or /cart/, copy those lines into each named group you allow, or those bots will treat them as open.
Google AI Overviews and Microsoft Copilot
The two largest search engines do not use a separate AI crawler for their search answers, so robots.txt tokens alone do not control them.
- Google AI Overviews and AI Mode use the normal Search index. Google says robots.txt rules for Googlebot are the crawling control, and points to
nosnippet,data-nosnippet,max-snippetandnoindexto limit what is shown (Google Search Central). Those also limit your normal search snippets, so use them on specific pages, not the whole site. - Microsoft Copilot is governed by Bing's page-level meta tags. Bing says
NOCACHElets its chat answers use only the URL, title and snippet, andNOARCHIVEkeeps the page out of those answers and out of training; both still let the page appear in Bing search results. Bing's announcement predates the rename from Bing Chat to Copilot.
What robots.txt cannot do
robots.txt is a public request. The standard itself says its rules “are not a form of access authorization” (RFC 9309). Well-behaved crawlers follow it; nothing forces them to. Keep three limits in mind:
- User-triggered fetchers are a grey area. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says its bots honour robots.txt directives.
- It is not retroactive. Blocking a training crawler today speaks to future collection. Anthropic, for example, describes its opt-out as excluding a site's future content.
- It is per host. Each subdomain needs its own robots.txt. Anthropic's help page calls this out explicitly.
If you need content kept private, put it behind a login. If you need to verify a bot is genuine, check the owner's published IP ranges; Anthropic, for example, publishes its list and advises against relying on IP blocking as an opt-out.
What about llms.txt?
An llms.txt file is a proposed plain-text summary of a site for language models. It does not control crawling, and it is not a replacement for robots.txt. Google says Google Search does not use llms.txt or similar AI text files; creating one will neither help nor harm your visibility or rankings in Google Search (Google Search Central). Some other services may read it, so it is fine to keep one, but it is not where your AI access decisions live.
Check your site in five minutes
-
Open your live robots.txt
Visit
yourdomain.com/robots.txtand search for each token in the table. Look for plugin- or host-added blocks you did not write. -
Decide per job, not per company
For each company, decide separately on search and on training. Most businesses that want customers to find them allow the search crawlers.
-
Check your firewall and CDN
Bot-protection settings can block a crawler that robots.txt allows. If you want AI search traffic, make sure the search bots are not being challenged or blocked.
-
Then see whether you are actually mentioned
Access is only the first step. Our guide to checking whether ChatGPT mentions your brand shows how to test real answers.
If you want both halves in one place, MrKeyword's AI Visibility tracker asks ChatGPT, Gemini, Claude and Perplexity your customers' questions every week and records whether you are named and which sites are cited. It does not edit your robots.txt or verify bot traffic in your server logs.
See if AI answers name youThe Free plan tracks 3 questions every week across ChatGPT, Gemini, Claude and Perplexity. No card needed.
Start freeQuestions and answers
AI crawlers and robots.txt questions
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot finds pages to show in ChatGPT's search features. OpenAI says the settings are independent, so you can block GPTBot and still appear in ChatGPT search by allowing OAI-SearchBot.
How do I allow ChatGPT to crawl my site?
Make sure robots.txt does not disallow OAI-SearchBot, either by leaving it out so your User-agent: * rules apply, or by adding a group for it with Allow: /. Also check that your firewall or CDN is not blocking it.
Does blocking Google-Extended remove me from Google Search or AI Overviews?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews and AI Mode are Search features controlled by your Googlebot rules and snippet controls such as nosnippet.
Do AI crawlers have to obey robots.txt?
No. robots.txt is a voluntary standard, and its own specification says the rules are not access authorization. The major companies say their crawlers follow it, but OpenAI and Perplexity say their user-triggered fetchers may not.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers keeps your content out of future training, as far as the companies honour it. Blocking search crawlers removes you from those assistants' answers, which most businesses that rely on being found do not want.
Does Google use llms.txt?
No. Google says Google Search does not use llms.txt or similar AI text files, and that having one neither helps nor harms visibility or rankings in Google Search.
Sources
- Overview of OpenAI crawlers — OpenAI, on OAI-SearchBot, GPTBot and ChatGPT-User
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Anthropic, on ClaudeBot, Claude-SearchBot and Claude-User
- Perplexity crawlers — Perplexity, on PerplexityBot and Perplexity-User
- Google's common crawlers — Google, on Googlebot and Google-Extended
- About Applebot — Apple, on Applebot and Applebot-Extended
- Common Crawl FAQ — Common Crawl, on CCBot, robots.txt and its opt-out registry
- New options to control usage of content in Bing Chat — Bing Webmaster blog, on NOCACHE and NOARCHIVE
- Optimizing for generative AI features on Google Search — Google Search Central, on llms.txt
