Status, robots directives and canonicals.
HTTP status and redirect count, X-Robots-Tag noindex and nofollow, and a canonical URL named in a Link header.
- X-Robots-Tag explained
- Link rel=canonical in headers
Enter any URL. MrKeyword requests it the way a crawler would and lists every response header, with plain findings on status, X-Robots-Tag, compression, caching and common security headers.
Free, no account needed. MrKeyword fetches the page live and shows only what it found.
Built on first-party data from Google and Bing
An HTTP header checker sends one request to a URL and shows the response headers the server returns: the HTTP status, redirect count, and each header name and value, sorted alphabetically so you can scan them quickly.
Some headers change SEO directly. X-Robots-Tag can keep a page out of Google and Bing even when the HTML has no noindex tag. A Link header can name a canonical URL. Errors and redirects show up in the status line before you read the HTML.
Others affect speed and trust. Content-Encoding shows whether HTML is compressed. Cache-Control shows how browsers may reuse the response. Security headers such as Strict-Transport-Security, Content-Security-Policy and Referrer-Policy do not replace good content, but they are worth checking on important pages.
HTTP status and redirect count, X-Robots-Tag noindex and nofollow, and a canonical URL named in a Link header.
Content-Encoding (gzip or Brotli), Cache-Control, HSTS, X-Content-Type-Options nosniff, framing protection (CSP or X-Frame-Options) and Referrer-Policy.
The fictional northwind-plumbing.example moved its services page to production but left X-Robots-Tag: noindex on from staging. The HTML has no meta robots tag, and the server sends uncompressed HTML with no HSTS. Checking https://northwind-plumbing.example/services/ gives:
HTTP 200 · 120 ms · 0 redirects before this answer
| Header | Value |
|---|---|
| content-type | text/html |
| x-robots-tag | noindex |
Example result for a fictional website (northwind-plumbing.example). It is produced by the checker's own code from the setup described above; run the checker for your own pages.
The page answers HTTP 200, so it loads for visitors, but the header noindex blocks Google and Bing until it is removed or replaced. That often survives a migration when staging rules stay on the CDN or server. Turn compression on for HTML, add charset=utf-8 to Content-Type, and add HSTS once every URL is on HTTPS. For the rest of an indexing check, use the indexability checker, follow redirects with the redirect checker, or run a full pass with our guide to doing an SEO audit and the free site audit.
Remove the header or change it to index,follow on pages that should rank. Check CDN and cache rules as well as the origin server.
Search engines honour X-Robots-Tag on the response. The indexability checker shows both the header and any meta tag.
Enable gzip or Brotli for text/html on the server or CDN. Smaller responses load faster for visitors and crawlers.
Add Strict-Transport-Security once every URL redirects to HTTPS, so repeat visits stay on a secure connection.
Add charset=utf-8 to text/html responses so browsers and tools read the encoding consistently.
Check headers when a page should index but does not, when speed drops, or after changing host, CDN or security settings.
It is a tool that requests a URL once and lists the HTTP response headers the server sends back, together with short notes on headers that affect indexing, speed or security. It does not read the page body beyond what is needed for the request.
X-Robots-Tag is an HTTP response header that carries indexing directives, similar to a meta robots tag. A value such as noindex tells search engines not to include the URL in results, even if the HTML looks indexable.
Some do. X-Robots-Tag can block indexing. A Link header with rel=canonical can point search engines at another URL. Status codes and redirects decide whether a URL is reachable. Compression and caching mostly affect speed and crawl efficiency rather than rankings directly.
Request a normal HTML URL and look for a Content-Encoding header on the response. gzip or br means the body is compressed. This checker sends Accept-Encoding: gzip, br and reports when HTML comes back with no Content-Encoding.
There is no single required list, but HTTPS sites often add Strict-Transport-Security, X-Content-Type-Options: nosniff, a policy that limits framing (Content-Security-Policy frame-ancestors or X-Frame-Options) and Referrer-Policy. Exact values depend on your site; this checker flags common gaps so you can decide what to add.