Robots.txt explained: syntax, examples and common mistakes

What is robots.txt and where does it go?

Robots.txt is a plain-text file that tells cooperating automated crawlers which URLs on a site they may fetch. Search engines look for it before crawling pages. It helps keep crawlers away from low-value paths such as internal search results, while leaving useful pages and resources accessible. It is a crawl instruction, not a security system: anyone can read the file, and a bot can ignore it.

Publish it at the root, for example https://example.com/robots.txt, with that exact lowercase filename. A file at /blog/robots.txt does not govern the site. Scope includes protocol, host and port: HTTP and HTTPS, www.example.com and example.com, and subdomains such as shop.example.com need their own applicable file. A redirect can serve a file from another location, but check the final live response for each origin.

Use a UTF-8 plain-text file, one directive per line. Lines beginning with # are comments. Rules are public, so do not list sensitive paths as a substitute for authentication. Google's robots.txt specification and Bing's file guide explain the supported syntax.

The directives and wildcards

User-agent names the crawler to which a group applies. User-agent: * is the fallback for crawlers without a more specific group. A group starts with one or more user-agent lines and contains the rules that follow. Use the published crawler token, such as Googlebot or Bingbot, rather than guessing from a browser name.

Disallow identifies a path that the group should not crawl. Disallow: /private/ covers URLs beginning with that path, including files below it. An empty Disallow: imposes no restriction. Allow creates an exception, for example Allow: /private/public/ inside a blocked folder. Paths are case-sensitive: /Private/ and /private/ are different.

Sitemap points to a sitemap using its full absolute URL, such as https://example.com/sitemap.xml. It is a discovery hint, not an allow rule or a promise of indexing. Put sitemap lines outside the crawl-rule groups for clarity, and use one line per sitemap. Replace example domains in every sample with your own.

In path rules, * matches any sequence of characters, including none; $ matches the end of the URL. For example, Disallow: /*.pdf$ matches a URL ending in .pdf, but not the same path followed by a query string. A path without $ is a prefix: /search also matches /searching. Test actual URLs with query strings before publishing a broad pattern.

Crawl-delay is not interpreted by Google. Bing honours it, and its Crawl Control documentation says the directive takes precedence over its crawl-control settings. An excessive delay can slow discovery and updates; use it only when needed and monitor crawl activity.

How Google and Bing choose rules

First, the crawler selects the most specific matching user-agent group. For example, a Googlebot group takes precedence over User-agent: * for Googlebot; Bingbot likewise uses its own group instead of the generic group. The wildcard group's disallows are not automatically added to a named group. Repeat any restrictions that must apply in each named group.

There is an important distinction: Google combines multiple groups that name the same user agent before evaluating them. It does not combine a specific Googlebot group with the wildcard group. Keep one group per crawler where practical to make the file easier to audit. Bing also documents an old msnbot fallback for Bingbot, so check legacy groups when troubleshooting Bing.

Within the applicable rules, the most specific matching path wins: the longest matching rule overrides a shorter one. For example, Disallow: /shop/ and Allow: /shop/help/ block the shop except its help folder. Google uses the less restrictive rule when equally specific rules conflict. Avoid ambiguous ties and use the robots.txt tester to check both an intended exception and a nearby blocked URL for each relevant bot.

Robots.txt vs noindex

Robots.txt controls crawling, not indexing. A search engine may learn a blocked address from links elsewhere and show the URL without a normal description even though it cannot fetch its content. Blocking a page therefore does not reliably remove it from results, and it does not protect private information.

To remove a public page from search, allow the crawler to fetch it and serve a noindex robots meta tag or HTTP header. If robots.txt blocks the page, the crawler cannot see that instruction. The X-Robots-Tag guide covers header-based noindex for PDFs and other files; the indexability checker checks crawl and indexing signals on a live URL. For confidential staging content, use authentication or network restrictions instead of relying on either signal.

Missing files, server errors and file size

If Google receives a 404 for robots.txt, it treats the file as unavailable and assumes crawling is allowed. That is not the same as a server failure. For a 5xx response, Google initially stops crawling while it retries; later it can use a cached good file, and its subsequent handling depends on the duration of the failure and site availability. Restore a successful response promptly rather than relying on a temporary block.

Google processes only the first 500 KiB of a robots.txt file; rules beyond that limit are ignored. Keep the file short and consolidate patterns carefully instead of listing thousands of individual URLs. These status and size details are Google's documented behavior; verify other crawlers separately rather than assuming identical limits.

Copy-ready robots.txt examples

Each example is a separate starting point, not a set of snippets to paste together blindly. Check existing groups first, adapt paths to your site, and test before uploading. The free robots.txt generator can produce a starting file with crawler controls.

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

This is useful for a public site with no crawl restrictions. It does not override a page's noindex directive or grant access to authenticated content.

User-agent: *
Disallow: /drafts/
Allow: /drafts/public/

The trailing slash matters: this pattern covers the folder and its descendants, rather than every path that merely begins with the word drafts. It does not make drafts private.

User-agent: *
Disallow: /*?ref=
Disallow: /*&ref=

Parameter rules can also block useful pages if those URLs are the only route to content. Confirm parameter order and spelling in real URLs, and avoid blocking URLs that need to be crawled for canonical or noindex signals to be seen.

User-agent: *
Disallow: /search?

Change the path to match your site's actual search URL. Test a search result and a normal content URL. If your search uses a different route, this rule will not match it.

User-agent: *
Disallow: /

A site-wide disallow reduces cooperative crawling, but a linked staging URL can still be indexed and the content remains accessible. Password protection is the stronger control. Before launch, check the production file independently so this rule does not follow the deployment.

User-agent: GPTBot
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

This example addresses only the named training crawlers, not every AI company or use of content. Search and training often use separate tokens. Read the AI crawler rules guide before choosing a policy, and use the AI crawler checker to inspect the live result. The proposed llms.txt file has a different purpose and does not replace these crawl rules.

Common mistakes to fix

• Staging block on production: Disallow: / under the wildcard group stops cooperative crawling across the site. Check the live production origin after every launch.
• Blocked CSS or JavaScript: search engines may need these resources to render a page. Test important assets as well as the page URL, and allow resources required to understand the content.
• Wrong path case: /Admin/ will not match /admin/. Copy the actual URL path and check its capitalization.
• Relative sitemap: Sitemap: /sitemap.xml is not the documented absolute form. Include the scheme and host, then verify that the sitemap URL works.
• Missing rules in named groups: adding a Googlebot or Bingbot group changes the rules that crawler follows. Repeat relevant generic restrictions in that group and test both allowed and blocked paths.

How to test the live file

Open the exact origin's /robots.txt in a browser and confirm the expected text and successful response. Enter representative URLs into the robots.txt tester, select Googlebot and Bingbot, and compare the matched line with your intent. Include a blocked URL, an allowed exception, a query-string URL and a CSS or JavaScript asset. Repeat for named AI bots if you use their groups.

In Search Console, use the robots.txt report to inspect Google's fetched file and any fetch issues. After changing a file, request a recrawl there and allow time for caches to update. Bing Webmaster Tools also offers a robots.txt tester. Finally, check the page itself for noindex and other signals: a successful crawl test alone does not guarantee indexing.

Questions and answers