Skip to content
SERPCompass

Pages blocked from Google by noindex or robots.txt

Remove the noindex rule from every page you want in Google. It sits in a robots meta tag in the page's head or in an X-Robots-Tag HTTP header, and site builders set it with a hide-from-search switch. Take pages you keep noindexed out of your XML sitemap. Check that robots.txt does not disallow pages you want found and answers 200 or 404, never a server error.

Last updated 29 October 2026, when every step was checked against the vendor's documentation.

Why it matters

noindex tells search engines not to show a page. Google says it drops a page with noindex from its results entirely, even when other sites link to it. The rule can sit in the page, <meta name="robots" content="noindex">, or in the HTTP response, X-Robots-Tag: noindex, which you cannot see in the page source. The value none means noindex and nofollow together.

robots.txt works differently: it controls crawling, not indexing. Google says it is not a way to keep a page out of Google, because a blocked page that is linked from elsewhere can still appear, without a description. And Google cannot read a noindex on a page robots.txt keeps it from crawling: it ignores the rule.

A sitemap is the list of pages you want found: Google asks you to include the URLs you want to see in its search results. A noindex page listed there says two opposite things about the same page, so decide which is true. If the page should be found, remove the noindex and the sitemap is right. If it should stay out, keep the noindex and take the page out of the sitemap; leaving it out of the sitemap alone would not keep it out, because Google also finds pages through links.

A page-level nofollow tells crawlers not to follow any link on the page. Google may otherwise use those links to find the pages they point to, so pages linked only from a nofollow page are found less reliably.

A rule can name one crawler. Google reads robots meta tags named robots, googlebot or googlebot-news and ignores other names, so <meta name="bingbot" content="noindex"> or X-Robots-Tag: bingbot: noindex does not apply to Google. It is aimed at another search engine, which can drop a page Google still shows: easy to miss when you only check Google.

robots.txt itself has to answer. Google treats a robots.txt that answers 404 or another 4xx code, except 429, as no restrictions. When it answers a server error or cannot be reached, Google stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days.

SERPCompass rates a robots.txt that answers a server error or cannot be reached as critical, and a page set to noindex as a warning, because the page drops out of search; a noindex page your sitemap lists is a warning too. A page-level nofollow, rules addressed to other crawlers and links to addresses robots.txt blocks are notices. The audit reads the meta tag and the header, and applies the rules Google applies: unnamed ones and those for googlebot.

The audit findings this guide fixes: Pages set to noindex; Noindex pages listed in the sitemap; Pages with a page-level nofollow directive; Pages hidden from Bing or other crawlers; Pages blocked by robots.txt; robots.txt returns a server error or cannot be reached.

Find every page with this problem

Run the free scan on your domain. It reads the robots meta tags and X-Robots-Tag header of every page it crawls, looks up each noindex page in your XML sitemaps, checks each link against your robots.txt and checks that robots.txt itself answers, then shows your most urgent findings with the number of pages each affects. A SERPCompass workspace lists the affected pages and whether each noindex came from the meta tag or the header. To check one page by hand, view its source and search for noindex, and run curl -sI followed by the address to look for an X-Robots-Tag line. In Google Search Console, the Page indexing report lists pages excluded by noindex and pages blocked by robots.txt.

Scan my site free

How to fix it on your platform

Choosing which pages to keep out of Google, and snippet controls such as nosnippet, are not covered here; robots.txt rules for Googlebot, Bingbot and AI crawlers, and the rest of your XML sitemap, have their own guides.

WordPress (no SEO plugin)

  1. Go to Settings > Reading. Under Search engine visibility, clear the box Discourage search engines from indexing this site, and click Save Changes. While it is ticked, WordPress adds noindex, nofollow to every page.
  2. Open your /robots.txt address. WordPress's own file disallows only the admin folder; any other Disallow line comes from a plugin, your theme or a robots.txt file in your site's root folder, served instead of WordPress's. Edit or remove the line where it lives.
  3. If a page still carries noindex after that, a plugin or your theme writes it: check the SEO plugin's settings below, then the theme.
  4. WordPress's own sitemap, /wp-sitemap.xml, is off while Discourage search engines is ticked. Otherwise, if a plugin sets posts to noindex, WordPress's developers advise excluding them from the sitemap too: the wp_sitemaps_posts_query_args filter, adding their IDs to post__not_in in a small plugin or a child theme, does it.

Sources: Settings Reading screen (WordPress documentation); wp_robots_no_robots() (WordPress developer reference); do_robots() (WordPress developer reference); New XML sitemaps functionality in WordPress 5.5 (Make WordPress Core).

WordPress with Yoast SEO

  1. Open the post, page, product, category or tag and open the Yoast SEO sidebar with the Yoast icon at the top right of the editor, or the meta box below it.
  2. In the Advanced tab, set Allow search engines to show this content in search results to Yes. Answer Yes to the question whether search engines should follow links on the page. Update the post.
  3. Only editors and administrators see the Advanced tab by default.
  4. If a whole content type is hidden, go to Yoast SEO > Settings, choose it under Content types, Categories & tags or Advanced (for archives), turn its search results toggle back on and click Save changes.
  5. To fix a robots.txt rule, go to Yoast SEO > Tools > File editor. The menu does not appear when file editing is disabled; edit the file over FTP then.
  6. Yoast leaves a post out of its sitemap when Allow search engines to show this content in search results is set to No, and leaves out content types set to noindex. A noindex page still in Yoast's sitemap most likely gets its noindex from somewhere else, such as the theme, another plugin or an X-Robots-Tag header: remove it there, or set Yoast's own setting to match.

Sources: How to noindex URLs (Yoast); Noindex a post or page in WordPress, the easy way (Yoast); Help, I've accidentally noindexed a post. What to do? (Yoast); How to edit robots.txt through Yoast SEO (Yoast); How to exclude content from the sitemap (Yoast); XML sitemaps in the Yoast SEO plugin (Yoast).

WordPress with Rank Math

  1. Open the post or page and click the Rank Math icon at the top right of the editor.
  2. In the Advanced tab, under Robots Meta, select Index (it turns No Index off) and clear No Follow. Update the post.
  3. Check the defaults: open Titles and Meta from Rank Math's menu, and under Global Meta and each content type, such as Posts and Pages, make sure No Index is not set for content you want found. Click Save Changes.
  4. To fix a robots.txt rule, turn on Advanced Mode, go to Rank Math SEO > General Settings > Edit robots.txt, change the rule and click Save Changes. Rank Math can only edit it when no physical robots.txt file sits in your site's root folder.
  5. Rank Math leaves posts whose Robots Meta is No Index out of its sitemap, so a noindex page it still lists gets its noindex from elsewhere, such as the theme, or a cached sitemap has not caught up. To leave such a page out, go to Rank Math SEO > Sitemap Settings > General, enter its ID in Exclude Posts (several separated by commas) and click Save Changes.

Sources: General tab in the meta box (Rank Math); Advanced tab in the meta box (Rank Math); Titles and Meta options and settings (Rank Math); How to edit robots.txt with Rank Math (Rank Math); Why isn't my URL included in the sitemap? (Rank Math); How to exclude content from sitemap (Rank Math).

Shopify

  1. Look for a noindex added in theme code: in your Shopify admin, go to Online Store, open the theme's menu, click Edit code and open theme.liquid in the layout folder. Remove any <meta name="robots" content="noindex"> block that covers pages you want found, and click Save.
  2. Look for the SEO Hidden metafield (seo.hidden) on the product, page or blog post: delete its value, leave it blank and save.
  3. Shopify hides products with the Unlisted status from search engines and the sitemap. A product you want found needs another status.
  4. If your theme has a robots.txt.liquid template, check its rules in Edit code. To go back to Shopify's default file, right-click robots.txt.liquid, choose Delete permanently and click Delete.
  5. To keep a product, page or blog post out of search results and your sitemap together, use the SEO Hidden metafield: in Settings > Metafields and metaobjects, add a definition for Products, Pages or Blog posts with namespace and key seo.hidden, type Integer, then set SEO Hidden to 1 on the item and save. Shopify says it, like the Unlisted status, hides the item from sitemaps; it does not say that of a noindex added in theme.liquid.

Sources: Hiding a page from search engines (Shopify Help Center); Editing robots.txt.liquid (Shopify Help Center).

Wix

  1. Check the whole site first: go to SEO & GEO in your site's dashboard, scroll to Tools and settings, click SEO Settings and make sure Let search engines index your site is on. While it is off, the page settings cannot override it.
  2. For one page, click Pages & Menu in the editor, click the More Actions icon next to the page, click SEO basics and turn on Let search engines index this page.
  3. If Search Console reports Blocked by robots.txt and you never changed the file, Wix advises against editing it: check the page's indexing setting, password protection and members-only status instead.
  4. To see or reset the file, go to SEO & GEO > Tools and settings > Robots.txt Editor and click View File; Reset to Default restores Wix's own.
  5. Wix updates your sitemap whenever you change your site, and says pages search engines are prevented from indexing may not appear in it. To keep a page out, turn off its Let search engines index this page toggle in SEO basics, publish, and check your sitemap.

Sources: Managing search engine indexing for your site (Wix Help Center); Preventing search engines from indexing your page (Wix Help Center); Editing your site's robots.txt file (Wix Help Center); Understanding your site's sitemap file (Wix Help Center).

Squarespace

  1. For one page, open the Pages panel, hover over the page, click the gear icon, open the SEO tab and turn off Hide page from search results. Hiding a collection page also hides its items, such as blog posts.
  2. The home page has no such toggle; a noindex on it, or on any page, can also come from code injection. Look for <meta name="robots" content="noindex"> in your code injection and remove it.
  3. For the whole site, open the Settings panel, click Crawlers and clear Block Search engine crawlers. While it is ticked, your robots.txt carries Disallow: /.
  4. Squarespace leaves pages hidden from search with the SEO tab's toggle, disabled pages and password-protected pages out of its sitemap, and updates it automatically, usually within an hour and at most within 24 hours. A noindex added through code injection is not on that list: to keep a page out of search and the sitemap together, use Hide page from search results instead.

Sources: Hiding your site from search results (Squarespace Help Center); View your site map (Squarespace Help Center).

Webflow

  1. For a static page, go to Page settings > SEO settings and turn Sitemap indexing on. Off, it adds <meta content="noindex" name="robots"> to the page.
  2. For a CMS item, open the item in its Collection, go to Item details and turn Sitemap Indexing on, then save.
  3. A noindex added by hand in custom code is not controlled by that toggle: remove it from the page's custom code.
  4. Go to Site settings > SEO > Indexing. Remove robots.txt rules that block pages you want found, and keep the Traffic control toggle for search engine crawlers on. Staging indexing affects only your webflow.io address.
  5. Save and publish your site: none of these settings take effect until you do.

Sources: Disable search engine indexing (Webflow Help Center); Set robots.txt rules (Webflow Help Center).

Plain HTML

  1. Remove noindex, nofollow and none from the <meta name="robots"> and <meta name="googlebot"> tags of pages you want found. With no robots tag at all, Google applies no restrictions.
  2. Remove the X-Robots-Tag header from those pages in your server configuration: in Apache a Header set X-Robots-Tag line, in NGINX an add_header X-Robots-Tag line.
  3. Check rules addressed to other crawlers, such as <meta name="bingbot" content="noindex"> or X-Robots-Tag: bingbot: noindex, and remove them unless you mean to keep the page out of that search engine.
  4. In /robots.txt, remove or narrow each Disallow rule that covers pages you want found; leave deliberate blocks, such as cart and internal search pages.
  5. Make /robots.txt answer 200, or 404 if you have none, every time. A 5xx or a timeout makes Google pause crawling the whole site.
  6. Take every page you keep noindexed out of your XML sitemap: delete its <url> entry, or exclude it in whatever builds the file. Google asks you to list the URLs you want to see in its results.

Sources: Block Search indexing with noindex (Google Search Central); Robots meta tag, data-nosnippet, and X-Robots-Tag specifications (Google Search Central); How Google interprets the robots.txt specification (Google Search Central); Robots Exclusion Protocol: joining together to provide better documentation (Bing Webmaster blog, 2008); Build and submit a sitemap (Google Search Central).

Examples

  • Bad 
    <meta name="robots" content="noindex, nofollow">

    On a page you want found: Google drops it from results and does not follow its links.

  • Bad 
    HTTP/1.1 200 OK
Content-Type: text/html
X-Robots-Tag: noindex

    The same rule as a header: nothing in the page source shows it.

  • Bad 
    User-agent: *
Disallow: /

    Blocks every page for every crawler that follows the standard. Squarespace writes this when Block Search engine crawlers is ticked.

  • Good 
    User-agent: *
Disallow: /cart
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

    Blocks only the pages kept out on purpose.

  • Bad 
    <url>
  <loc>https://www.example.com/thank-you/</loc>
</url>

    A sitemap entry for a thank-you page that carries noindex: the page is meant to stay out, so take the entry out of the sitemap.

  • Bad 
    <meta name="bingbot" content="noindex">

    Google ignores it, so the page looks fine in Google while it is meant to drop out of Bing.

  • Good 
    <head>
  <title>Drain cleaning | Corner Plumbing</title>
  <link rel="canonical" href="https://www.example.com/services/drains/">
</head>

    No robots tag at all: Google's default is no restrictions, so nothing needs adding to allow indexing.

Common mistakes

  • Leaving WordPress's Discourage search engines box ticked after launch, which keeps noindex, nofollow on every page.
  • Blocking a page in robots.txt to get it out of Google. Google then cannot see a noindex, and the address can still be listed; Google says to remove the block and use noindex instead.
  • Carrying a test site's settings to the live site: a noindex, a Disallow: / or an X-Robots-Tag header meant for the copy.
  • Undoing a hide-from-search switch while a noindex added by hand in custom code or code injection stays on the page.
  • Hiding a page with a noindex in theme code, custom code or a header, while the platform's sitemap, built from its own hide-from-search setting, keeps listing it.
  • Fixing one page while a site-wide setting still hides everything. Wix says hiding the whole site disables the page settings.
  • Expecting the page back the next day. Google sees the change only when it crawls the page again, which for a less important page may take months.
  • Reading nofollow as a way to keep a page out of Google. It only stops crawlers following the page's links; noindex keeps the page out.

Questions and answers

What is the difference between noindex and blocking in robots.txt?

noindex lets Google crawl the page but tells it not to show it, and Google drops the page from results. A robots.txt block stops Google crawling the page, but the address can still appear without a description when other pages link to it. To keep a page out, use noindex and leave it crawlable; to get it in, remove both.

Why does Search Console say Indexed, though blocked by robots.txt?

Google indexed the address without crawling it, which can happen when other pages link to it. If you want the page in Google, remove the robots.txt rule that blocks it. If you want it out, Google says to remove the block and add a noindex, because it cannot see a noindex on a page it may not crawl.

How long until a page comes back after I remove noindex?

Google has to crawl the page again first, and it says that may take months for a page of little importance on the web. For an important page, the URL Inspection tool in Search Console can ask Google to crawl it again; there is a quota, and asking repeatedly does not make it faster.

Does a nofollow on a page stop Google indexing it?

No. A page-level nofollow tells Google not to follow the links on the page; the page itself can still be indexed. Only noindex, or none, which means noindex and nofollow together, keeps a page out of the results.

What happens if my robots.txt answers an error?

It depends on the error. Google reads a 404 or another 4xx code, except 429, as no robots.txt and no restrictions. A server error or a timeout makes Google stop crawling your site for the first 12 hours, then use the last good copy for up to 30 days. Make robots.txt answer 200, or 404 if you have none.

Sources

Every step above was checked against these documents on 29 October 2026. Vendors change their menus; if a step no longer matches, the vendor's page is the authority.

Start with a free scan of your site.

Two to five minutes, no account. You get your three most urgent findings, and the exact count of everything else.

Free, 2–5 minutes, no account needed.