X-Robots-Tag: what it is, how it differs from meta robots, and how to noindex PDFs

What is X-Robots-Tag?

X-Robots-Tag is an HTTP response header that carries the same indexing and snippet rules as the HTML <meta name="robots"> tag. Google's block indexing documentation lists two ways to keep a URL out of search results: a noindex robots meta tag in the page, or an X-Robots-Tag HTTP header on the response. Crawlers that support Google's rules read whichever signal they receive when they fetch the URL.

Because the header is attached to the HTTP response, not to HTML, it is the practical way to control indexing for resources that are not web pages: PDFs, Word documents, images, video files and plain text. A PDF cannot carry a <meta name="robots"> tag in a way search engines use for indexing; the header on the download URL is what matters.

X-Robots-Tag vs the robots meta tag

Both channels accept the same directive names and values; only the delivery mechanism differs. Use a meta tag when you control the HTML <head>. Use X-Robots-Tag when you do not, or when one server rule should cover many files (every PDF, everything under /private/, or all responses from an API route).

• HTML pages: either a meta tag or the header works. If both are present on the same URL, Google treats them as a combined set of rules; avoid sending conflicting directives.
• Non-HTML files: rely on X-Robots-Tag (or on not linking the file publicly). There is no standard in-page robots meta for a binary PDF body.
• Visibility: the meta tag appears in View Source; the header does not. That is why accidental noindex on PDFs is easy to miss until you inspect response headers.

For a page-level checklist that includes both signals, see our on-page SEO checklist.

Directives Google supports

Google documents these in its robots meta tag reference. You can list several in one header, separated by commas.

Rules for one crawler only

You can prefix a directive with a user-agent token so it applies only to that crawler. Google's documentation gives the form X-Robots-Tag: googlebot: noindex to noindex a URL for Googlebot while leaving other crawlers unchanged. The same pattern works for other supported tokens documented on Google's robots meta tag page.

Use this when you need different behaviour for Googlebot and Bingbot on the same file, not as a substitute for authentication on private content.

Copy-paste examples: noindex PDFs and folders

Pick the layer that serves the file. These examples set noindex, nofollow on purpose; drop nofollow if you still want link equity from URLs inside the file.

  Header set X-Robots-Tag "noindex, nofollow"

  Header set X-Robots-Tag "noindex, nofollow"

location ~* \.pdf$ {
  add_header X-Robots-Tag "noindex, nofollow" always;
}

location ^~ /private/ {
  add_header X-Robots-Tag "noindex, nofollow" always;
}

export async function headers() {
  return [
    {
      source: "/files/:path*.pdf",
      headers: [{ key: "X-Robots-Tag", value: "noindex, nofollow" }],
    },
  ];
}

Set static header
Name: X-Robots-Tag
Value: noindex, nofollow

When: URI Path matches regex \.pdf$

The robots.txt trap

If robots.txt disallows a URL, Googlebot will not fetch it, so it never sees an X-Robots-Tag on that response. Google's noindex documentation is explicit: it does not support noindex in robots.txt, and when robots.txt blocks crawling, the crawler cannot read a noindex directive on the page or in headers either.

That breaks a common mistake: disallow /pdf/ in robots.txt and also set X-Robots-Tag: noindex on those PDFs. The disallow wins for crawling; the header never runs. To remove URLs from the index while keeping them crawlable, allow Googlebot to fetch them and send noindex (meta or header). To keep URLs private, use login walls rather than relying on robots.txt alone.

Test whether a URL is allowed for a given bot with the free robots.txt tester before you depend on header-based noindex.

How to check X-Robots-Tag on a live URL

1. Read every response header

   Paste the URL into MrKeyword's free HTTP header checker. It lists status, redirects and headers, and highlights x-robots-tag when present. That is the fastest way to answer searches for x-robots-tag noindex on a specific file.

2. See indexability in one report

   The free indexability checker combines HTTP status, robots.txt, canonical, meta robots and X-Robots-Tag so you can see whether anything blocks indexing.

3. Confirm in Search Console

   For HTML URLs you control, URL Inspection shows the HTML Google fetched. For PDFs, check the live header first, then use the coverage reports after Google recrawls.

Questions and answers