AI crawler checker: which bots may read your site

How robots.txt works for AI crawlers

robots.txt is a plain text file at the root of your site, at yoursite.com/robots.txt. It holds groups of rules. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow lines for paths. A crawler follows the group that names it; when no group does, it follows the group for every crawler, written User-agent: *. When two rules match a page, the longer one wins, and on a tie the Allow wins. That is how the robots.txt standard (RFC 9309) and Google describe it, and how this checker reads your file.

Training crawlers and answer crawlers do different jobs

AI companies run more than one crawler. Some collect pages to train models: GPTBot for OpenAI, ClaudeBot for Anthropic and CCBot for the open Common Crawl corpus. Others build the index an assistant searches before it answers, which decides whether your pages can be cited: OAI-SearchBot for ChatGPT search, Claude-SearchBot and PerplexityBot. A third kind fetches a page only when a person asks the assistant about it: ChatGPT-User, Claude-User and Perplexity-User. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page.

Each of these has its own name, so a rule that blocks a training crawler does not block the search crawlers. OpenAI says each of its settings is independent: a site can disallow GPTBot and still allow OAI-SearchBot, so ChatGPT search can show its pages.

Two names that are not crawlers

Google-Extended and Applebot-Extended never visit your site. They are names you can use in robots.txt to tell Google and Apple whether pages their own crawlers already fetch may be used for their AI models: Gemini training and grounding for Google, Apple's models for Apple. Google says disallowing Google-Extended does not affect your inclusion in Google Search and is not a ranking signal. Google Search, including AI Overviews and AI Mode, follows the rules you give Googlebot.

How to read the result

Allowed means no rule stops that crawler. Partly blocked means your home page is open but some paths are disallowed; we list them. Blocked means the rules stop that crawler at your home page. A missing robots.txt, answering 404, means every crawler may read everything. A robots.txt that answers a server error or does not answer at all is different: Google stops crawling the site for up to 12 hours, so we say so rather than showing every crawler as allowed.

What this check cannot see

robots.txt is a request, not a lock. Crawlers that follow the standard read it, but a firewall, CDN or bot-protection setting can turn a crawler away even when robots.txt allows it, and some CDNs offer a setting that blocks AI crawlers outright. This check reads robots.txt only, so it cannot see those blocks. When our own request to your home page is refused, we tell you, because the same setting may be refusing other crawlers too.

How India's most visited .in websites set these rules

We ran this checker's code over the top .in domains in the Tranco research ranking. Read the study for how many block AI search crawlers, crawler by crawler, with the method and the data.