Indexed, though blocked by robots.txt and Blocked by robots.txt: what they mean and how to fix them

What does "Indexed, though blocked by robots.txt" mean?

It means Google has the URL in its index but is not allowed to crawl it, so it indexed the address without reading the page. You find it in Search Console under Indexing → Pages, in the list of reasons why pages are not indexed or are indexed with a warning. Google's Page indexing report help lists it among the page indexing statuses, and the behaviour behind it is described in Google's robots.txt introduction:

• Google found the URL somewhere else. A link on another page, a link on your own site, or a sitemap submission tells Google the address exists. Google does not need to fetch a URL to learn that it exists.
• A robots.txt rule stops the fetch. When Googlebot tries to crawl it, the Disallow rule that matches the path says no, so Google never receives the HTML, the title, the canonical tag or any robots meta tag.
• The URL can still be indexed. Google says a blocked page "can still be indexed if linked to from other sites", and that the search result "won't have a description" because Google could not read the content. That is exactly the state this status reports.

So the page is not missing from Google. It is present as a bare URL, often with no snippet or a snippet Google assembled from other sources, and it cannot be updated because Google cannot re-read it. Two consequences follow. First, a noindex tag you add to that page does nothing while the rule stands, because the crawler never sees it. Second, if the page is one you want to rank, the block is costing you the crawl that would let Google judge the content. For where crawling and indexing sit in the whole process, read how search engines work.

How it differs from "Blocked by robots.txt"

"Blocked by robots.txt" means the URL is not indexed: Google knows the rule blocks the crawl and has not added the URL to the index. "Indexed, though blocked by robots.txt" means the same rule applies but the URL is in the index anyway. Both come from a Disallow rule that matches the path; the difference is what Google already knows about the URL.

• Blocked by robots.txt. The URL is out of the index and will not be crawled while the rule matches. If you blocked the path on purpose, this is the report working as intended, not an error.
• Indexed, though blocked by robots.txt. The URL is in the index despite the rule, usually because other pages link to it. Google's robots.txt introduction is explicit that robots.txt "is not a mechanism for keeping a web page out of Google".

That is why the fix is not the same for both. A URL you meant to block and that shows as Blocked needs nothing. A URL you meant to block and that shows as Indexed, though blocked needs a real indexing directive, not a crawl rule. And a URL you never meant to block needs the rule removed in either case. Do not confuse these with Crawled - currently not indexed and Discovered - currently not indexed, where Google can reach the page and the question is whether it wants to keep it.

Decide what you want: should the page rank or stay out?

Every fix starts with one decision, because the two answers need opposite robots.txt changes. Open the URL in Search Console's URL Inspection tool to confirm the status and the referring page or sitemap that told Google about it, then pick a case.

Case A: the page should rank, so remove the Disallow rule

If the URL is a page you want in search, the robots.txt rule is the problem. Find the group and the line that matches it, then delete that line or narrow it so the page is allowed. Remember that the most specific matching user-agent group is the one that applies, and the longest matching path rule wins, so a rule in a Googlebot group is not overridden by anything in the wildcard group. The full matching logic, with copy-ready examples, is in robots.txt explained: syntax, examples and common mistakes.

Also check that the rule was not blocking the resources the page needs. A Disallow on a CSS or JavaScript folder can leave Google able to fetch the HTML but unable to render it. Once the page is allowed, keep it in your XML sitemap, link to it from related pages, and request indexing once in URL Inspection.

Case B: the page should stay out of Google, so allow crawling and add noindex

If the URL should not appear in search at all, a robots.txt block is the wrong tool. Allow the crawler to fetch the page and serve a noindex directive instead, in a robots meta tag in the HTML head or in an X-Robots-Tag response header. Google's block indexing documentation says that for a noindex rule to be effective the page must not be blocked by a robots.txt file and must be otherwise accessible to the crawler, because otherwise "the crawler will never see the noindex rule" and the page can still appear in results through links.

The order matters, and it is the step most people get backwards:

1. Remove the Disallow rule first

   Take out the line that blocks the URL, or add a longer Allow exception for it, so Googlebot can fetch the page. Until you do, no indexing directive on that page can be read.

2. Add noindex while the page is crawlable

   For an HTML page, add <meta name="robots" content="noindex"> to the head. For a PDF, an image, a whole folder or a host where you cannot edit the HTML, send X-Robots-Tag: noindex in the HTTP response header instead. The X-Robots-Tag guide covers the header syntax, the crawler-specific forms and the server configuration for each.

3. Wait for Google to crawl it and drop it

   Google has to fetch the page again to see the directive, so give it a crawl cycle and recheck the URL in URL Inspection. The Page indexing report should move the URL out of the indexed list. Keep the page crawlable and keep the noindex in place for as long as you want the URL out.

4. Only then think about the robots.txt rule again

   Put a Disallow rule back for that path only after Google has dropped the page, and only if you still want to save the crawls. Understand the trade-off: a blocked URL that other pages link to can be indexed again as a bare URL, which is the status you started with. If the content is confidential, use authentication or network restrictions rather than either signal.

One more thing not to try: a Noindex line inside robots.txt. Google's block indexing documentation states that specifying noindex in a robots.txt file is not supported by Google.

Test the rule, check the page, then Validate Fix

Prove the rule changed before you ask Google to recheck, otherwise Validate Fix reports the same problem back to you.

1. Test the exact URL against the live file

   Paste the URL into the free robots.txt tester and select Googlebot. It fetches your published file, shows the group that applies and the exact rule that decides the result, so you can see whether the page is allowed and which line is responsible. Test Bingbot too, and test a URL that should stay blocked so you know your edit did not open the whole folder. If you are rewriting the file, the robots.txt generator builds one with a live preview, copy and download.

2. Check the page itself

   Run the URL through the free indexability checker. It checks the HTTP status and redirects, the robots.txt rules for Googlebot, a noindex in the robots meta tag or the X-Robots-Tag header, and where the canonical points. In Case B you want to see the page allowed and the noindex present; in Case A you want it allowed with no noindex left behind by mistake.

3. Run Validate Fix in Search Console

   Open the status in the Page indexing report and choose Validate Fix. Google rechecks a sample of the affected URLs and reports whether the issue is resolved, still present or no longer detectable. Validation only passes once Google can fetch the URLs again, so allow time for the robots.txt change to be picked up. You can also request a recrawl of the file in Search Console's robots.txt report, which shows the file Google fetched and any fetch issues.

4. Recheck the URL and the report

   Use URL Inspection on the URL you care about, run a live test, and confirm Googlebot can now fetch it and sees the directives you intended. Then watch the counts in the Page indexing report over the following crawls rather than expecting them to change the same day.

Status, meaning and fix at a glance

Match the status to what you want for that URL, then apply the fix in the last column.

The same rule may be blocking more than Googlebot

A Disallow written for one crawler often applies to others, so check which bots the group names before you edit it. A rule under User-agent: * applies to every crawler without a more specific group, which can include the AI crawlers you meant to allow, and a rule you add for GPTBot or CCBot does not restrict Googlebot unless the wildcard group says so too. If the URL is blocked for a bot whose content you want used in AI answers, that is a separate decision from search indexing. Read how to allow or block AI crawlers in robots.txt before changing a group, and test each named bot in the robots.txt tester rather than assuming they share a rule.

What about Bing?

Bing also honours robots.txt, so the same Disallow rule stops Bingbot crawling the URL, and the same crawl-versus-index distinction applies. Bing's robots.txt file guide documents the directives and its crawler-specific rules, including the legacy msnbot fallback for Bingbot, so check old groups when a URL behaves differently in Bing. Test the URL for Bingbot in the robots.txt tester, then check its index status with URL Inspection in Bing Webmaster Tools and use IndexNow or Submit URLs to tell Bing about the change. Bing also feeds Copilot, so a URL stuck as a bare address is a gap in AI answers too. The full setup is in our Bing Webmaster Tools guide.

Questions and answers