Skip to content
SERPCompass

robots.txt that blocks search or AI crawlers, is missing or has errors

Open your site's robots.txt and make sure no rule disallows the pages you want found for Googlebot, Bingbot or the AI search crawlers: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, and PerplexityBot. You can still opt out of AI training by disallowing GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Then check your CDN: Cloudflare's AI bot settings can block crawlers that robots.txt allows.

Last updated 28 October 2026, when every step was checked against the vendor's documentation.

Why it matters

robots.txt is the file crawlers read before anything else on your site. Each crawler obeys the group of rules that names it and, when none does, the group for all crawlers, User-agent: *. So one line, Disallow: / under User-agent: *, keeps every crawler that follows the standard out of every page.

For Google this costs more than the crawl. Google says a page it may not crawl can still appear in results, but without a description, and that a page must be indexed and eligible for a snippet to appear as a link in AI Overviews or AI Mode. Google names the robots.txt rules for Googlebot as the control site owners have over that crawl.

AI assistants send separate crawlers for search and for training, and each one obeys its own name. OpenAI says OAI-SearchBot decides whether pages can appear in ChatGPT search answers, while GPTBot collects content that may train its models, and that each setting is independent. Anthropic splits Claude-SearchBot and Claude-User, which find and fetch pages for Claude's answers, from ClaudeBot, which collects training data. Perplexity says PerplexityBot surfaces sites in its search results and is not used to train models.

Google-Extended and Applebot-Extended are not crawlers at all: no bot visits your site under those names. They are robots.txt names that tell Google and Apple whether pages their search crawlers fetch may be used for AI. Google says Google-Extended has no effect on inclusion or ranking in Google Search, and Apple says pages that disallow Applebot-Extended can still appear in its search results. So you can opt out of AI training and stay findable in search and AI answers.

The group rule catches people out with Bing and Apple too. Bing says that when a robots.txt has a section for bingbot, Bingbot ignores every other directive in the file, so rules meant for all crawlers must be copied into that section. Apple says Applebot follows the Googlebot rules when no group names Applebot.

robots.txt is only half of it. A firewall or CDN can refuse a crawler before it ever reads the file. Cloudflare sorts AI crawlers into Search, Agent and Training and lets you block each group; since 15 September 2026, Cloudflare says, blocking Training also stops Googlebot, Bingbot and Applebot, search included. Its Bot Preference Sync setting can also add its own section to the top of the robots.txt your visitors and crawlers receive.

SERPCompass rates a page robots.txt blocks for Googlebot as critical, and pages blocked for Bingbot or a site that blocks AI search crawlers as warnings. A missing robots.txt is a notice: it blocks nothing, because crawlers read a missing file as no restrictions, but it leaves you no place to name your sitemap or state your AI training choices. Lines crawlers cannot parse are a notice too: they are skipped without a warning, so the rule you meant never applies.

The audit findings this guide fixes: Pages robots.txt blocks for Googlebot; Pages robots.txt blocks for Bingbot; AI search crawlers blocked in robots.txt; No robots.txt file; robots.txt syntax problems.

Find every page with this problem

Run the free scan on your domain. It reads your robots.txt, checks each page it crawls against the rules Googlebot and Bingbot follow, and checks whether the AI search crawlers may read your site. It shows your most urgent findings with the number of pages each affects; the Gap Report and a SERPCompass workspace list every finding with its pages, including the robots.txt lines crawlers ignore. For a quick answer about your home page, the free AI crawler checker reads your robots.txt the way each search and AI crawler does. Neither can see a firewall or CDN block: if robots.txt allows a crawler and it still cannot reach your pages, check your CDN's bot settings.

Scan my site free Use the free checker

How to fix it on your platform

Keeping a page out of search results (a noindex tag, not robots.txt), llms.txt, and bot settings on firewalls and CDNs other than Cloudflare are not covered here.

WordPress (no SEO plugin)

  1. Open your site's /robots.txt address in a private browser window to see the file crawlers receive.
  2. If no robots.txt file sits in your site's root folder, WordPress writes one itself: User-agent: *, a Disallow for the admin folder and an Allow for admin-ajax.php, plus, on a public site since WordPress 5.5, a Sitemap: line for its built-in sitemap. That blocks no crawler from your pages, so there is nothing to change.
  3. If the file shows other rules, they come either from a robots.txt file in the root folder, which replaces WordPress's own, or from a plugin that changes WordPress's output through the robots_txt filter. Edit that file, or that plugin's settings.
  4. Remove or narrow every Disallow rule that covers pages you want found, in the User-agent: * group and in any group that names Googlebot, Bingbot or an AI search crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot).
  5. To write your own rules without a plugin, save them as a plain UTF-8 text file named robots.txt and upload it to the root folder with your host's file manager or FTP.
  6. In Settings > Reading, leave Discourage search engines from indexing this site unticked on a live site. Since WordPress 5.3 it adds a noindex robots meta tag instead of a robots.txt rule, so it keeps pages out of search even when robots.txt allows them.

Sources: How to write and submit a robots.txt file (Google Search Central); do_robots() (WordPress developer reference); WP_Sitemaps::add_robots() (WordPress developer reference); Settings Reading screen (WordPress documentation).

WordPress with Yoast SEO

  1. In the WordPress dashboard, go to Yoast SEO > Tools > File editor. If File editor is missing, file editing is turned off on your site: edit robots.txt with FTP or your host's file manager instead.
  2. If you have no robots.txt yet, click Create robots.txt file. Yoast's starting rules allow all search engines to crawl the site and add a link to your sitemap.
  3. Remove or narrow every Disallow rule that covers pages you want found, in the User-agent: * group and in any group that names Googlebot, Bingbot or an AI search crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot). Then save the file.
  4. To opt out of AI training, go to Yoast SEO > Settings, open Crawl optimization under Advanced and use the switches under Block unwanted bots, then click Save changes. They cover Google AdsBot (free) and, in Premium, Google Gemini and Vertex AI bots (Google-Extended), OpenAI GPTBot and Common Crawl CCBot, which Yoast blocks by updating robots.txt. None of them is a search crawler.

Sources: How to edit robots.txt through Yoast SEO (Yoast); How to block unwanted bots from your site with Yoast SEO (Yoast); Bot Blocker (Yoast SEO features).

WordPress with Rank Math

  1. In the WordPress dashboard, go to Rank Math SEO > General Settings > Edit robots.txt. If Edit robots.txt is missing, Rank Math is in Easy Mode: turn on Advanced Mode from the Rank Math SEO dashboard.
  2. If Rank Math shows a notice that a robots.txt file already exists, its editor has no effect. Either edit that file with FTP or your host's file manager, or copy its rules and remove the file so Rank Math's version is served.
  3. Remove or narrow every Disallow rule that covers pages you want found, in the User-agent: * group and in any group that names Googlebot, Bingbot or an AI search crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot).
  4. Click Save Changes at the bottom right of the screen.

Sources: General Settings (Rank Math); I can't edit my website's robots.txt (Rank Math); Common robots.txt issues and how to fix them (Rank Math).

Shopify

  1. Open your store's /robots.txt address. Shopify gives every store a default robots.txt it calls optimal for SEO; unless your theme has a robots.txt.liquid template, that default is what crawlers receive.
  2. In your Shopify admin, go to Online Store, open the menu beside your published theme and choose Edit code.
  3. Open robots.txt.liquid if the theme has one. To create it, click Add a new template, choose robots and click Create template.
  4. Remove or narrow every Disallow rule that covers pages you want found, in the User-agent: * group and in any group that names Googlebot, Bingbot or an AI search crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot).
  5. Write any group of your own, such as User-agent: GPTBot followed by Disallow: /, as plain text outside the Liquid that prints Shopify's default rules. Leave that Liquid in place: Shopify strongly recommends its Liquid objects over typing the default rules out.
  6. Save the file. Shopify calls this an unsupported customisation that its support team cannot help with, and warns that a mistake can cost all your search traffic.
  7. To go back to Shopify's default, right-click robots.txt.liquid in the code editor, choose Delete permanently, then Delete.

Sources: Editing robots.txt.liquid (Shopify Help Center); Customize robots.txt (Shopify developer docs).

Wix

  1. In your site's dashboard, go to SEO & GEO and scroll down to Tools and settings.
  2. Click Robots.txt Editor, then View File.
  3. Edit the rules under This is your current file. Remove or narrow every Disallow rule that covers pages you want found, in the User-agent: * group and in any group that names Googlebot, Bingbot or an AI search crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot).
  4. Click Save Changes, then Save.
  5. To undo your edits, open the same editor and click Reset to Default, then Reset.

Source: Editing your site's robots.txt file (Wix Help Center).

Squarespace

  1. Open the Settings panel and click Crawlers.
  2. Make sure Block Search engine crawlers is unticked. When it is on, Squarespace adds Disallow: / to your robots.txt and search engines stop indexing the site.
  3. Block known artificial intelligence crawlers is your choice. Its list includes GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, and does not include OAI-SearchBot, Claude-SearchBot or PerplexityBot, so ticking it leaves those AI search crawlers allowed. It applies to the whole site, never single pages.
  4. You cannot edit Squarespace's robots.txt itself: every Squarespace site uses the same file, and its other Disallow lines, such as /config/ and /search, are expected and can be left alone.

Sources: Hiding your site from search results (Squarespace Help Center); Request that AI models exclude your site (Squarespace Help Center); Understanding Google SEO emails and console errors (Squarespace Help Center).

Webflow

  1. Open Site settings, then the SEO tab, and find the Indexing section, where your robots.txt rules are.
  2. Remove or narrow every Disallow rule that covers pages you want found, in the User-agent: * group and in any group that names Googlebot, Bingbot or an AI search crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot).
  3. Webflow adds a Sitemap: line for your sitemap by default: leave Remove sitemap.xml from robots.txt switched off.
  4. Click Save, then publish the site: the rules are not live until you do.

Source: Set robots.txt rules (Webflow Help Center).

Plain HTML

  1. Save your rules as a plain text file named robots.txt in UTF-8, using a text editor rather than a word processor.
  2. Upload it to the root of your site so it answers at https://www.example.com/robots.txt. Crawlers do not look in subfolders, and the file covers only the host, protocol and port it is served from: each subdomain needs its own.
  3. Start each group with one or more User-agent lines and put its Allow and Disallow rules under them. A crawler with a group of its own ignores the User-agent: * group, so copy into its group every rule it should follow.
  4. Add a Sitemap: line with your sitemap's full URL, including https://.
  5. Open the file's URL in a private browser window to confirm it is public.
  6. Check it in the robots.txt report in Google Search Console, and after an important fix choose Request a recrawl there. In Bing Webmaster Tools, the robots.txt tester shows whether Bingbot may fetch a given URL.

Sources: How to write and submit a robots.txt file (Google Search Central); How Google interprets the robots.txt specification (Google Search Central); RFC 9309: Robots Exclusion Protocol (IETF); robots.txt report (Search Console Help); Update your robots.txt file (Google Search Central); To crawl or not to crawl, that is BingBot's question (Bing Webmaster Blog); Bing Webmaster Tools makes it easy to edit and verify your robots.txt (Bing Webmaster Blog).

Examples

  • Bad 
    User-agent: *
Disallow: /

    Keeps out every crawler that follows robots.txt, Googlebot and Bingbot included. It is often left behind when a site moves from a test server to its live address.

  • Bad 
    User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /

    Meant to opt out of AI training, but OAI-SearchBot, Claude-SearchBot and PerplexityBot are the crawlers ChatGPT, Claude and Perplexity use to find pages for their search answers. SERPCompass reports this as AI search crawlers blocked.

  • Bad 
    Disallow: /cart/
User-agent: *
Disalow: /checkout/
Sitemap: /sitemap.xml

    Three lines crawlers skip: a rule before any User-agent line, the misspelt Disalow, and a Sitemap that is not a full URL. SERPCompass lists each one under robots.txt syntax problems.

  • Good 
    User-agent: *
Disallow: /cart/
Disallow: /checkout/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

    Googlebot, Bingbot and the AI search crawlers have no group of their own, so they follow the * group and may read everything except the cart and checkout. The second group opts out of training by OpenAI, Anthropic, Google and Apple without touching search.

  • Good 
    User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

    The smallest useful robots.txt: every crawler may read every page, and the Sitemap line names your sitemap by its full URL.

Common mistakes

  • Launching with the test server's robots.txt, which disallows everything, still in place.
  • Blocking the AI search crawlers while meaning to opt out of training: GPTBot and ClaudeBot train models, while OAI-SearchBot and Claude-SearchBot decide whether ChatGPT and Claude can find your pages.
  • Adding a group for one crawler and expecting the User-agent: * rules to apply to it as well: a crawler that finds a group naming it follows that group alone.
  • Misspelling a field, putting rules above the first User-agent line, or writing Sitemap: /sitemap.xml without the full URL. Crawlers skip those lines without telling you.
  • Using robots.txt to hide a page from search. Google can still index a blocked URL without its content; to keep a page out, use a noindex tag and let crawlers read the page.
  • Fixing robots.txt in your site builder while Cloudflare still blocks the crawler at its network edge, or adds its own section above your rules.
  • Choosing Block for Training in Cloudflare to opt out of AI training: since 15 September 2026 that also stops Googlebot, Bingbot and Applebot. Cloudflare's Disallow AI Training option, or a Google-Extended rule, keeps search open.

Questions and answers

Will blocking GPTBot remove my site from ChatGPT?

No. OpenAI says GPTBot collects content that may be used to train its models, while OAI-SearchBot decides whether your pages can appear in ChatGPT search answers, and that each setting is independent. So you can disallow GPTBot and still allow OAI-SearchBot. ChatGPT-User, which opens a page when someone asks ChatGPT about it, acts for that person, and OpenAI says robots.txt rules may not apply to it.

Does blocking Google-Extended remove my site from Google Search or AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal; it covers training of Gemini models and grounding in Gemini Apps and Vertex AI. AI Overviews and AI Mode link to pages indexed for Google Search, so the Googlebot rules and snippet controls such as nosnippet apply there. Apple says the same of Applebot-Extended: pages that disallow it can still appear in its search results.

Do I need a robots.txt file at all?

Without one, nothing is blocked: Google treats a robots.txt that returns 404 as if none existed, so every crawler may read every page. SERPCompass still notes a missing file, because robots.txt is where you name your sitemap with a Sitemap line and state your AI training choices. Three lines are enough: User-agent: *, then Allow: /, then your Sitemap line.

I allow the crawlers in robots.txt. Why are they still blocked?

A firewall or CDN can refuse a crawler before it reads robots.txt. On Cloudflare, open your domain's Security Settings and check Configure AI bot policies: Block on Search or Agent stops AI search and assistant crawlers, and since 15 September 2026 Block on Training also stops Googlebot, Bingbot and Applebot. Disallow AI Training keeps search open. AI Crawl Control lists each crawler with its own Allow or Block action.

How long does a robots.txt change take to work?

Each crawler fetches the file again on its own schedule. Google generally caches robots.txt for up to 24 hours; after an important fix, Request a recrawl in Search Console's robots.txt report asks it to fetch the file sooner. OpenAI says ChatGPT search can take about 24 hours to adjust. Bing's robots.txt tester can fetch your latest file and check a URL against it.

Sources

Every step above was checked against these documents on 28 October 2026. Vendors change their menus; if a step no longer matches, the vendor's page is the authority.

Start with a free scan of your site.

Two to five minutes, no account. You get your three most urgent findings, and the exact count of everything else.

Free, 2–5 minutes, no account needed.