Skip to content
IndexMeNow
Sign in

Guide

How to find and fix noindex tags and robots.txt blocks

2 min readPublished

Two settings cause a large share of “why is my page not in Google?” problems: a noindex directive and a robots.txt rule. Both are easy to add by accident and easy to miss. They also do different things, and mixing them up can make a problem worse.

noindex and robots.txt are not the same thing

  • robots.txt controls crawling. A Disallow rule asks Googlebot not to fetch the URL.
  • noindex controls indexing. Googlebot fetches the page, sees the directive, and leaves the page out of the index.

The two interfere with each other. If a page is blocked in robots.txt, Google never fetches it and so never sees a noindex on it. The URL can even stay in the index, without a description, if other sites link to it. To remove a page from search, allow crawling and use noindex.

Finding a noindex directive

There are three places it can hide:

  1. A robots meta tag in the HTML head: <meta name="robots" content="noindex">. A googlebot meta tag works the same way for Google.
  2. An X-Robots-Tag HTTP header. Run curl -I https://example.com/page and look for X-Robots-Tag: noindex. This also covers PDFs and other files that have no HTML head.
  3. A CMS or plugin setting. WordPress has “Discourage search engines from indexing this site” under Settings → Reading, and SEO plugins have a per-page “allow search engines to show this page” switch.

In Search Console the result is Excluded by ‘noindex’ tag. IndexMeNow’s pre-flight scan reports both the meta tag and the header as a blocker.

Finding a robots.txt block

Open https://your-site/robots.txt and read the rules that apply to Googlebot, or to all crawlers (User-agent: *). A line such as Disallow: / blocks the whole site, and Disallow: /blog/ blocks everything under that path.

Search Console shows the file Google last fetched in the robots.txt report under Settings, and reports affected pages as Blocked by robots.txt.

Fixing it

  1. Remove the noindex directive or the Disallow rule, or narrow it so it no longer covers the page.
  2. Deploy the change and confirm it on the live site, not only in the editor.
  3. In URL Inspection, run Test live URL. The result should say crawling and indexing are allowed.
  4. Request indexing once, and give Google time to recrawl.

Mistakes that keep happening

  • Launching with the staging Disallow: / still in robots.txt.
  • Setting noindex with JavaScript. Google may or may not see it, so use the HTML or the header.
  • Listing noindex pages in your XML sitemap. It sends Google two opposite signals.
  • Wildcard rules that match more than you meant, such as Disallow: /*? blocking every URL with a query string.

Keep reading