How many of the top 1,000 .in websites block AI search crawlers?
31 of the 762 top .in websites whose robots.txt we could read (4.1%) block at least one AI search crawler from their home page. 16 of them block Googlebot too, so their rule is not aimed at AI; the other 15 block AI search crawlers while letting Googlebot in. 79 (10.4%) block at least one AI training crawler.
How we did it
The sample. Tranco is a ranking of websites built for research, which anyone can download again by its list ID. We used list GQ3LK, created on 27 October 2026 from Chrome UX Report, Farsight, Majestic, Cloudflare Radar and Cisco Umbrella data for 28 September 2026 to 27 October 2026; it ranks registrable domains, so each entry is one site. We took the first 1,000 domains, in rank order, whose name ends in .in, second-level zones such as .gov.in, .co.in and .ac.in included. They run from rank 676 to rank 91,831 of the list's 1,000,000; the list holds 10,957 .in domains in all.
The reading. On 28 October 2026 we opened each domain's home page over HTTPS (plain HTTP when HTTPS failed at once; www. when the bare domain did not answer), followed its redirects and read the robots.txt of the site it landed on. When a home page redirected to another domain, we read the domain's own robots.txt instead, because those are its rules. Each request stopped after 10 seconds without an answer and each site after 30 seconds; sites that failed were tried once more. We identified ourselves as SERPCompassBot, from one computer. It is the code behind our free AI crawler checker.
Blocked, partly blocked, allowed. We read each crawler's rules the way the robots.txt standard (RFC 9309) and Google describe them: the group that names the crawler, else the group for every crawler (User-agent: *); the longest matching rule wins, and Allow wins a tie. Blocked means a rule stops the crawler at the site's home page. Partly blocked means the home page is open but other paths are disallowed, for example by Disallow: /admin/. Allowed means no rule stops it. When the robots.txt address answers 404, 410 or another 4xx error that is not a refusal, or answers with an HTML page, the site has no robots.txt: crawlers read that as no rules, so every crawler is allowed.
The crawlers. The 14 crawlers and 2 AI-use tokens our checker reads, listed in the tables below. By AI search crawlers we mean the 6 in the group "AI search and assistant crawlers".
The results, crawler by crawler
Of the 1,000 domains, 513 served a robots.txt and 249 had none. We could not read 238; they are left out of every percentage on this page (see the limits below). Each percentage is of the 762 sites we could read.
- Domains in the sample
- 1,000
- The first .in domains in Tranco list GQ3LK.
- Sites we could read
- 762
- 513 with a robots.txt, 249 without one.
- Block an AI search crawler
- 31 4.1%
- At least one, at the home page.
- Block every AI search crawler
- 21 2.8%
- All 6 at once.
- Name an AI crawler
- 108 14.2%
- In a robots.txt group of its own.
Search engines
For comparison: the crawlers behind Google Search and Bing.
| Crawler | Blocked | Partly blocked | Allowed | Named in its own group |
|---|---|---|---|---|
Googlebot |
162.1% |
32642.8% |
42055.1% |
537.0% |
Bingbot |
182.4% |
32943.2% |
41554.5% |
405.2% |
AI search and assistant crawlers
These build the index an AI assistant searches before it answers, or open a page when a person asks the assistant about it. They decide whether a site can be found and cited in AI answers.
| Crawler | Blocked | Partly blocked | Allowed | Named in its own group |
|---|---|---|---|---|
OAI-SearchBot |
212.8% |
32742.9% |
41454.3% |
405.2% |
ChatGPT-User |
273.5% |
32442.5% |
41153.9% |
506.6% |
Claude-SearchBot |
253.3% |
33544.0% |
40252.8% |
172.2% |
Claude-User |
273.5% |
33443.8% |
40152.6% |
212.8% |
PerplexityBot |
253.3% |
31741.6% |
42055.1% |
537.0% |
Perplexity-User |
273.5% |
33343.7% |
40252.8% |
222.9% |
AI training crawlers and AI-use tokens
These collect pages to train AI models. Each has its own name, so blocking one does not block the search crawlers above. Google-Extended and Applebot-Extended are robots.txt names, not crawlers: they say whether pages that Google and Apple already crawl may be used for their AI models.
| Crawler | Blocked | Partly blocked | Allowed | Named in its own group |
|---|---|---|---|---|
GPTBot |
638.3% |
29538.7% |
40453.0% |
8711.4% |
ClaudeBot |
607.9% |
30039.4% |
40252.8% |
7710.1% |
Google-Extendedrobots.txt token, not a crawler |
567.3% |
30339.8% |
40352.9% |
7710.1% |
Applebot-Extendedrobots.txt token, not a crawler |
567.3% |
31841.7% |
38850.9% |
476.2% |
CCBot |
638.3% |
30439.9% |
39551.8% |
668.7% |
Bytespider |
688.9% |
30940.6% |
38550.5% |
587.6% |
meta-externalagent |
557.2% |
31341.1% |
39451.7% |
547.1% |
Amazonbot |
567.3% |
31741.6% |
38951.0% |
496.4% |
By zone
The .in sites we could read, by the zone their domain sits in, and how many of them block at least one AI search crawler.
| Zone | Sites read | Block an AI search crawler |
|---|---|---|
.in |
425 |
174.0% |
.gov.in |
173 |
74.0% |
.co.in |
67 |
46.0% |
.ac.in |
52 |
11.9% |
.nic.in |
20 |
00.0% |
.edu.in |
10 |
00.0% |
.org.in |
10 |
110.0% |
.am.in |
1 |
00.0% |
.com.in |
1 |
00.0% |
.io.in |
1 |
00.0% |
.net.in |
1 |
1100.0% |
.res.in |
1 |
00.0% |
What it means for your site
If you want AI assistants to find your pages and cite them in their answers, the crawlers in the AI search group have to be able to read them. Each crawler has its own name, so a rule against a training crawler such as GPTBot does not block the search crawlers; OpenAI says its settings are independent of each other.
A rule for every crawler (User-agent: *) that disallows the whole site shuts out search engines as well as AI. 16 of the 31 sites that block an AI search crawler block Googlebot too, so their rules are not a choice about AI.
249 of the sites we read have no robots.txt at all, so every crawler may read them. Having no rules is a choice too: to keep pages out of AI training while staying open to AI search, a site needs rules that name each crawler.
The limits of this study
- robots.txt states intent, not enforcement. Crawlers that follow the standard obey it, but some assistant fetchers say they may not apply it to a page a person asked for, and a firewall, CDN or bot-protection setting can turn away a crawler that robots.txt allows. We read robots.txt only, so blocks of that kind are not in these numbers.
- Sites we could not read are left out, and counted. 238 of the 1,000 domains gave no reading: 43 because the domain name did not resolve, with or without www; 45 because no answer within the time limit; 45 because the HTTPS connection failed our checks (a certificate we could not verify, or an outdated security setup), and plain HTTP did not answer either; 8 because the connection was refused or dropped; 79 because our request was refused (HTTP 401, 403, 406 or 451); 12 because robots.txt answered a server error or HTTP 429; 6 because an answer too large to read, or a redirect that did not end. A site that refused our request may refuse other crawlers as well, but we cannot tell that from outside.
- A ranking list is not every business. Tranco ranks domains by how popular its data sources see them, so the top .in domains are the most popular ones by those measures, not typical ones: 173 of the 762 we read are government sites under .gov.in. Most small businesses, and the many Indian businesses whose sites end in .com or another ending, are outside this sample.
- One home page, one day. We read the rules that apply to each home page on 28 October 2026. Rules for other pages, and changes made since, are not counted.
- Redirects. 44 home pages redirect to another domain. For those domains we read the domain's own robots.txt, following its redirects as crawlers do, not the other site's: a domain is counted by its own rules.
- robots.txt that is a web page. 52 sites answer its robots.txt address with an HTML page. Crawlers find no rules in that, so we count them as having no robots.txt.
The data
One row per domain: its rank, the domain, whether we could read its robots.txt and why not, where we read the home page and robots.txt, and each crawler's verdict.
Download the data (CSV, 1,000 rows)
How to cite this study: SERPCompass, "How many of the top 1,000 .in websites block AI search crawlers?", data collected 28 October 2026, https://serpcompass.com/research/indian-websites-ai-crawlers/. The sample comes from Tranco list GQ3LK; Tranco itself is described in Le Pochat, Van Goethem, Tajalizadehkhoob, Korczynski and Joosen, "Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation", NDSS 2019.
Licence: the data is free to reuse under Creative Commons Attribution 4.0 (CC BY 4.0): credit SERPCompass and link to this page.
Sources
Start with a free scan of your site.
Two to five minutes, no account. You get your three most urgent findings, and the exact count of everything else.