Allow all, block training only, or block every AI bot.
One click sets the AI toggles; you can still edit paths, sitemap and crawl delay before you copy the file.
- Block training, keep AI search
- Block all AI crawlers
What goes in robots.txt
What belongs in a robots.txt file?
robots.txt is a plain-text file at the root of your site (for example https://example.com/robots.txt) that tells well-behaved crawlers which paths they may fetch. It uses groups of User-agent lines followed by Allow and Disallow rules, optional Crawl-delay for some bots, and Sitemap: lines that point search engines at your XML sitemap.
Each crawler follows the most specific group that names it, then falls back to the User-agent: * group. Inside a group, the longest matching path wins, and Allow wins a tie. A separate group for GPTBot or ClaudeBot replaces the wildcard rules for that bot only, so copy any private-path rules into every group you allow.
AI companies now publish several tokens: training crawlers such as GPTBot and CCBot, search crawlers such as OAI-SearchBot and PerplexityBot, and control tokens such as Google-Extended that do not crawl on their own. This generator labels each one and warns when you block a search crawler, because that choice affects whether ChatGPT or Perplexity can cite you. Test the live file with the robots.txt tester, see every AI bot at once with the AI crawler checker, or read the guide to allowing or blocking AI crawlers.
Disallow does not de-index a page. robots.txt tells crawlers what they may fetch; it does not remove URLs from Google or Bing. To keep a page out of search, allow crawling and use a noindex tag instead. Check both with the indexability checker.
Clear answers,
with the reason shown.
Copy, download and compare.
Live robots.txt preview, warnings when a search bot is blocked, a built-in allow/block test for Googlebot and OpenAI bots, and optional load of your current robots.txt next to the new one.
- Copy and download robots.txt
- Load live file from your site
Worked example
Block AI training, keep ChatGPT and Perplexity search.
Choosing the Block AI training, keep AI search preset blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Bytespider, leaves OAI-SearchBot, Claude-SearchBot and PerplexityBot on the wildcard Allow rules, and produces:
robots.txt preview
User-agent: * Allow: / User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://northwind-plumbing.example/sitemap.xml
Built-in test for / on example.com (same matching rules as the robots.txt tester):
| Crawler | May fetch / | Group |
|---|---|---|
| Googlebot | Allowed | * |
| GPTBot | Blocked | gptbot |
| OAI-SearchBot | Allowed | * |
Example robots.txt for a fictional site. It is built by the generator’s own code at build time; change the toggles and paths to match your site.
Training crawlers get their own Disallow: / groups, while search crawlers follow User-agent: * with Allow: /, so ChatGPT search can still read public pages without opting into model training. After you upload the file, confirm one page in the robots.txt tester and every AI token in the AI crawler checker. Our AI crawlers guide lists copyable files for each policy.
Reading the result
How do you read the robots.txt generator result?
- Presets
- Allow everything, block training only, or block every AI bot listed. Presets only change the AI toggles; path rules and sitemap stay as you typed them.
- AI toggles
- Each token is labelled training or search/answers. Blocking a search bot triggers a warning, because that bot fetches pages for AI answers.
- Path rows
- Disallow and Allow paths apply to the
User-agent: *group. When you block an AI bot, its group is alwaysDisallow: /for the whole site unless you edit the file by hand after copying. - Built-in test
- Shows whether Googlebot, GPTBot and OAI-SearchBot may fetch
/on example.com using the same longest-match rules as the robots.txt tester.
Common errors
What are common robots.txt mistakes with AI crawlers?
Blocking PerplexityBot while trying to stay in AI search
PerplexityBot is a search crawler. Use the training-only preset, or turn PerplexityBot back to Allowed.
Expecting Disallow to de-index a page
robots.txt does not remove URLs from Google. Use noindex on the page and check with the indexability checker.
Forgetting private paths in named groups
A crawler with its own group ignores the * rules. Copy /admin/ or /cart/ Disallow lines into each allowed AI group if those areas must stay private.
Uploading to the wrong host
www and non-www, or HTTP and HTTPS, each need their own robots.txt unless one redirects to the other.
Blocking Googlebot to stop AI Overviews
AI Overviews use the normal Search index. Block only the AI tokens you mean to block; keep Googlebot allowed for Google Search.
When to check
When should you generate a new robots.txt?
Regenerate when your AI policy changes, when you add private sections, or before launch if a host or plugin shipped a default block-everything file.
- Before opting out of AI training
- Use the training-only preset so search crawlers stay allowed while GPTBot, ClaudeBot and CCBot are blocked.
- After a plugin adds GPTBot blocks
- Load your live file, compare it to the preview, and merge only the rules you intend to keep.
- When you add /staging/ or /cart/
- Add Disallow paths on the wildcard group and mirror them in any named groups that should stay out of those folders.
- When you publish a new sitemap
- Paste the absolute https Sitemap URL so crawlers find every indexable page.
- After a migration to a new host
- Confirm the root robots.txt on the new host matches your policy; CDN edges sometimes serve a stale default.
Go further
Guides for this check.
More free checkers
Other quick checks.
- Redirect checker
- Robots.txt tester
- SERP snippet preview
- Hreflang checker
- Broken link checker
- On-page SEO checker
- Keyword intent checker
- Keyword density checker
- Schema markup validator
- Schema markup generator
- Open Graph checker
- Indexability checker
- AI crawler checker
- Sitemap validator
- HTTP header checker
- DNS lookup
- Backlink checker
- Whole-site audit
Questions
About the robots.txt generator.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot is OpenAI’s training crawler. OAI-SearchBot is the search crawler that decides whether your pages can appear in ChatGPT search answers. OpenAI treats them separately, so you can block GPTBot and still allow OAI-SearchBot.
Does blocking Google-Extended affect Google Search?
No. Google-Extended is not a crawler. It controls whether pages Googlebot already fetched may be used to train Gemini models. Google Search, AI Overviews and AI Mode follow your Googlebot rules.
Should I use Allow: / or leave paths empty?
An empty wildcard group with no rules means every path is allowed. This generator writes Allow: / for clarity when you allow the whole site, and adds Disallow lines for paths you list. Named AI groups always use Disallow: / when you block that bot.
Does robots.txt remove pages from Google?
No. Disallow stops crawling, not indexing. A disallowed URL can still appear if other sites link to it. Use a noindex meta tag or X-Robots-Tag header to keep a page out of the index while still allowing crawlers to read that directive.
Where do I upload the file?
Upload robots.txt to the root of each host that serves your site (HTTPS and www are separate hosts unless you redirect). After you publish, test one URL in the robots.txt tester and run the AI crawler checker on a public page.