One in twelve software companies in Wikidata blocks an AI crawler by name
AI crawler policies of 3,263 software companies in Wikidata (2026). 3,263 software companies in Wikidata with a readable robots.txt (of 3,884 sites tried), 15 AI crawlers, crawled on 3 November 2026 across OpenAI, Anthropic, Google, Perplexity, Apple, Common Crawl, ByteDance, Meta and Amazon.
Data windowCrawled 3 November 2026, 16:13 to 16:33 UTC
8.5%
of 3,263 software companies in Wikidata block at least one of 15 AI crawlers by name
| Value | |
|---|---|
| Block a training crawler by name | 8.4% |
| Block every crawler through * and name no AI crawler | 0.6% |
| Block a search or user crawler by name | 0.2% |
Source: LoudFace crawl of 3,263 software companies' robots.txt (Wikidata frame), 3 November 20263 Nov 2026
Key numbers
block an AI search or user-request crawler by name
6 of 3,263 sitesof training-crawler blockers use Cloudflare's legacy managed section or block only Bytespider (found after review)
147 of 275; 59.4% for entries added to Wikidata since August 2024, 37.0% for earlier onespublish llms.txt: companies added to Wikidata since August 2024, and earlier (found after review)
1,452 of 2,061 against 351 of 1,170block an AI crawler by name: listed and other companies
no significant difference; found after review, the gap could be -3.1 to 5.9 pointsKey takeaways
- 8.5% of 3,263 software companies in Wikidata block at least one of 15 AI crawlers by name in robots.txt (95% interval 7.6 to 9.5).
- Only 0.2% block an AI search or user-request crawler by name, and 270 of the 277 blockers (97.5%) block none of those crawlers by name or through a wildcard.
- Found after review: of the 275 companies blocking a training crawler, 69 carry Cloudflare's legacy managed section and 80 block only Bytespider, 147 in all (53.5%): 59.4% among recent Wikidata entries and 37.0% among earlier ones.
- Found after review: companies added to Wikidata since August 2024 publish llms.txt at 70.5%, earlier entries at 30.0%, so pooled figures mix two populations.
- Listed and other companies show no significant difference in blocking (9.0% against 8.5%); found after review, the gap could lie anywhere from -3.1 to 5.9 points.
Earlier counts show few software companies blocking AI crawlers. Growfusely found 4.8% of 442 B2B SaaS sites blocking at least one by name, and AI Advisors 0.6% of 362 readable B2B SaaS sites blocking an AI search crawler. Directree, across 9,037 AI tools, found 10.5% blocking GPTBot and 88.8% of those still allowing OAI-SearchBot. Frase found 62% of the 93 B2B software vendors it could check publishing llms.txt. Across the Tranco top 10,000 sites, HasData counted 10.3% blocking at least one AI crawler in July 2026 and 8.3% in September, as PPC Land reports.
Directree and HasData both tie blocking to Cloudflare. Directree found GPTBot blocked on 22.4% of Cloudflare-fronted AI tools against 5.2% on Vercel, which it calls a correlation. HasData linked many GPTBot opt-outs to Cloudflare's managed block, which disappeared from 118 robots.txt files after a Cloudflare change on 15 September, and saw GPTBot disallows on Cloudflare-proxied sites fall from 17.1% to 9.9%.
For software companies, this report counts how many of the companies that block a training crawler carry Cloudflare's section (25.1%) or block only one scraper (29.1%), split by when the company entered Wikidata; those cuts were added after review. It also cuts the results by stock-market listing and region, and publishes the per-site data. We tried the 3,884 websites of 3,940 software companies in Wikidata and read 3,263 robots.txt files.
One in twelve blocks an AI crawler, mostly training crawlers
277 of the 3,263 sites (8.5%, 95% interval 7.6 to 9.5) block at least one of the 15 AI crawlers by name; 275 of them block a training crawler such as GPTBot, ClaudeBot, Google-Extended or CCBot. Only 6 sites, 0.2%, block an AI search or user-request crawler by name, and each of those six crawlers is blocked by between 2 and 4 sites. Of the 277 blockers, 270 (97.5%) block none of the search or user-request crawlers, by name or through a wildcard, though 64 of them restrict some folders for one (found after review). Another 18 sites (0.6%) block every crawler through a wildcard rule without naming an AI crawler. Leaving out the 96 sites that refused our crawler outright, the headline is 8.7% (7.8 to 9.8) (found after review).
About half of the blocks come from a CDN file or one scraper
The following cuts were added after the first review and were not planned before the data. Of the 275 companies blocking a training crawler, 69 (25.1%) carry Cloudflare's legacy managed robots.txt section, which Cloudflare writes into the file for sites that switch it on. Cloudflare announced in August 2026 that Bot Preference Sync takes over from it; none of the sites we read carried the new section yet. 80 (29.1%) block only ByteDance's Bytespider, the most blocked crawler here as in Directree's sample. Together that is 147 companies, 53.5% (47.6 to 59.3): 59.4% of the 202 among recent Wikidata entries and 37.0% of the 73 among earlier ones. 157 (57.1%) name an AI search or user-request crawler in their file; 87 (31.6%) name one of the three search crawlers and leave it fully allowed, and 4 block one.
| Value | |
|---|---|
| Cloudflare legacy managed section | 25.1% |
| Block only Bytespider | 29.1% |
| Either of the two | 53.5% |
| Either, recent Wikidata entries | 59.4% |
| Either, earlier entries | 37.0% |
| Name a search crawler and fully allow it | 31.6% |
Source: LoudFace crawl, 3 November 2026; 275 sites blocking a training crawler3 Nov 2026
A site can count in more than one of the first two bars.
Which crawlers are named, and which are blocked
1,104 sites, 33.8%, name at least one AI crawler, about four times as many as block one by name. GPTBot is named by 1,016 sites and blocked by 3.4% of all sites. Bytespider is blocked by 7% and CCBot by 4.7%; each AI search or user-request crawler by 0.1% or less.
| Value | Note | |
|---|---|---|
| Bytespider (ByteDance) | 7.0% | training |
| CCBot (Common Crawl) | 4.7% | training |
| meta-externalagent (Meta) | 3.4% | training |
| GPTBot (OpenAI) | 3.4% | training |
| Amazonbot (Amazon) | 3.3% | training |
| ClaudeBot (Anthropic) | 3.0% | training |
| Applebot-Extended (Apple) | 3.0% | training |
| anthropic-ai (Anthropic) | 2.9% | training |
| Google-Extended (Google) | 2.5% | training |
| OAI-SearchBot (OpenAI) | 0.1% | search |
| PerplexityBot (Perplexity) | 0.1% | search |
| Claude-SearchBot (Anthropic) | 0.1% | search |
| ChatGPT-User (OpenAI) | 0.1% | user request |
| Claude-User (Anthropic) | 0.1% | user request |
| Perplexity-User (Perplexity) | 0.1% | user request |
Source: LoudFace crawl of 3,263 software companies' robots.txt, 3 November 20263 Nov 2026
Blocked means the crawler is named and the whole site is disallowed for it.
Two populations in one list
This section was added after review. 2,068 of the 3,263 readable sites belong to companies whose Wikidata entry was created on or after 28 August 2024; 1,195 to earlier entries. The two differ sharply. Recent entries publish llms.txt at 70.5% (68.4 to 72.4) against 30.0% (27.4 to 32.7), and block an AI crawler by name at 9.8% against 6.3%, a gap of 3.5 points (1.6 to 5.3). Only 0.3% of recent entries are listed companies, against 13.4% of earlier ones. An entry's creation date says when the company entered Wikidata, which is a different thing from its age, and we did not check who created each entry. The pooled llms.txt figure, 55.8% of 3,231 sites checked, mixes the two.
286 sites (8.8%) carry a Content-Signal line, and 147 (4.5%) state ai-train=no. Of the 3,067 home pages we could read, 73.3% carry Organization schema in their server HTML.
Listing and region
Listed companies block at 9.0% (15 of 167, 5.5 to 14.3) and other companies at 8.5% (262 of 3,096): no significant difference (p = 0.81), with a gap anywhere from -3.1 to 5.9 points (found after review). United States companies block at 7.8%, European at 8.5% and Asian at 11.4% (8.4 to 15.2). Asia differs from the rest at p = 0.043 alone, but no region differs significantly after correcting for the four comparisons (adjusted p = 0.172). By industry, software-as-a-service companies block at 5.9%, computer security at 9.0%, cloud computing at 4.7% and the wider software industry at 8.8%. In an analysis added after review, listed companies publish llms.txt less often: 38.0% against 56.7%, a gap of -26.0 to -10.9 points.
| Value | Note | |
|---|---|---|
| Listed | 9.0% | 167 sites |
| Not listed | 8.5% | 3,096 sites |
| United States | 7.8% | 988 sites |
| Europe | 8.5% | 1,184 sites |
| Asia | 11.4% | 343 sites |
| Other regions | 8.1% | 273 sites |
Source: LoudFace crawl, 3 November 2026; listing and country from Wikidata3 Nov 2026
No difference is significant after correcting for four comparisons.
A broader reading of blocking
Counting a site as blocking when a group naming an AI crawler disallows any path while the home page stays open raises the share from 8.5% to 19.8% (647 sites, 18.5 to 21.2). Under the lenient llms.txt definition, any non-HTML text file, adoption is 58.1% against 55.8%.
What to do about it
- Check your robots.txt for the AI search and user-request crawlers: only 0.2% of these companies block one by name. OpenAI states that sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links, and that robots.txt rules may not apply to ChatGPT-User.
- If your robots.txt carries a section Cloudflare writes, read what it says for each crawler: in an analysis added after review, 25.1% of the training-crawler blockers here carry the legacy managed section, which Cloudflare said in August 2026 it is replacing with Bot Preference Sync.
- Weigh llms.txt against the companies you compete with. In an analysis added after review, 70.5% of companies added to Wikidata since August 2024 publish one, against 30.0% of earlier entries and 38.0% of listed companies.
Figure 5
Two populations: companies added to Wikidata since August 2024, and earlier
Added after the first review. llms.txt on 2,061 and 1,170 sites; blocking on 2,068 and 1,195.
| Value | |
|---|---|
| Publish llms.txt: added since August 2024 | 70.5% |
| Publish llms.txt: added earlier | 30.0% |
| Block an AI crawler by name: added since August 2024 | 9.8% |
| Block an AI crawler by name: added earlier | 6.3% |
Source: LoudFace crawl, 3 November 2026; entry dates from Wikidata3 Nov 2026
Method
How we measured it, and where it stops.
Read this before you quote the numbers. The sheet is the paper’s measurement record: what was sampled, over which window, on which engines, and how.
- Sample
- 3,263 software companies in Wikidata with a readable robots.txt (of 3,884 sites tried), 15 AI crawlers, crawled on 3 November 2026
- Window
- Crawled 3 November 2026, 16:13 to 16:33 UTC
- Engines
- OpenAI, Anthropic, Google, Perplexity, Apple, Common Crawl, ByteDance, Meta, Amazon
- Published
- 3 November 2026
Question. What share of software companies block AI crawlers in robots.txt, which crawlers, whether listing and region make a difference, and how many publish llms.txt or a Content-Signal line? We wrote the method and recorded its hash before collecting any data. Amendment 1 (before collection) set how a company with several websites is handled; amendment 2 (after the first and second reviews) fixed a parsing gap and the outlier count, and added the exploratory analyses the report labels as found after review. Both are in the published method file.
Frame. Every company in Wikidata (CC0) whose industry is software industry, software as a service, cloud computing or computer security, with an official website and no dissolution date: 3,940 companies, read once on 3 November 2026. A company with several websites keeps the shortest address. 54 companies shared a host with another and count once (duplicates), leaving 3,886 sites; 2 Google product hosts were excluded as shared-platform hosts, leaving 3,884.
Crawl. Our crawler, LoudFaceResearchBot, read each site's robots.txt and then /llms.txt and the home page only where robots.txt allowed it, at one request per second per host, between 16:13 and 16:33 UTC on 3 November 2026. A 2xx text response or a 4xx is readable (RFC 9309 treats a 4xx as no robots.txt): 3,263 sites, including 96 that refused our crawler with 401, 402, 403 or 429. A 2xx HTML page in place of robots.txt is a bot wall: 190 sites, 135 of which redirected robots.txt to a web page and 55 served one in place; their policy is unknown and they are left out. A 5xx, timeout or failed connection is not collected: 431 sites. No robots.txt was an outlier by size (over 500 KB) or group count (over 1,000 groups).
What counts. A site blocks a crawler by name when a robots.txt group names it and, under RFC 9309 matching, disallows the home page. Of the 15 crawlers, 9 are for training and 6 for AI search or user requests, by their operators' own descriptions. llms.txt: a 200, non-HTML file whose first line is a Markdown H1, checked on 3,231 sites. Cloudflare's managed section is recognised by its BEGIN marker; Organization schema is read from the home page's server HTML without running JavaScript.
Statistics. Shares carry Wilson 95% intervals; a gap between two shares carries Newcombe's interval. Listed against other companies and each named region against the rest use two-proportion tests with a Holm correction across the four comparisons. The sample size was set before collection: ±2 points at 95% from 3,940 companies needs 1,493 readable sites; 3,263 were read.
Robustness. The headline was re-run counting any path a named group disallows, and llms.txt under a lenient definition; both are reported above.
Limitations
The frame. Wikidata is maintained by volunteers and lists companies notable enough, or keen enough, to be added; its industry labels are broad. 2,068 of the 3,263 readable sites belong to entries created since August 2024, which publish llms.txt far more often and are rarely listed, so pooled figures mix two populations. This is an open, re-runnable list of software companies and should not be read as a census of B2B SaaS. Employee counts exist for only 299 readable sites, too few for a size cut beyond listing.
Coverage. 3,263 of 3,884 intended sites (84%) had a readable robots.txt. By segment: Software as a service 86.3%, Computer security 82.6%, Cloud computing 78.5%, Software industry 84.2%, Listed 82.7%, Not listed 84.1%, United States 82.5%, Europe 88.6%, Asia 87.1%, Other regions 88.9%, Region unknown 73.2%. The 190 bot walls, mostly redirects of robots.txt to another page, are left out.
Refusals. 96 sites refused our crawler with 401, 402, 403 or 429; under RFC 9309 they count as having no robots.txt, so as not blocking. Without them the headline is 8.7%.
Stated policy only. A site can block AI crawlers at its CDN or firewall without a line in robots.txt, and a robots.txt rule is a request crawlers may ignore. We kept the first 64 KB of each robots.txt; 5 files were longer, and rules past that point were not read. Where a Crawl-delay line sits between two user-agent lines, we follow Python's standard parser, which ends the group there; Google's parser does not, and two sites would read differently without changing the headline.
After the data. These cuts were added after the first and second reviews: the Cloudflare section, Bytespider-only blocks, which search crawlers blockers name, refusals, bot-wall type, Wikidata entry date, listed companies' llms.txt rate and the intervals on the gaps between groups. They describe this sample and were not tested as hypotheses set in advance.
One day, other studies. The crawl ran on 3 November 2026, and policies change when a CMS or CDN updates its defaults. Growfusely, AI Advisors, Directree and Frase used different site lists and crawler sets, so their figures and ours are close in meaning but not directly comparable. The HasData figure comes from PPC Land's report of it.
No causes. The study does not show why a company chose its policy, or whether a policy changed its visibility in AI answers.
Versions
- Version 1.0 · 3 Nov 2026CurrentFirst edition: 3,263 software companies, crawled 3 November 2026.
- Version 1.1When the paper is re-run, the new version is listed here and this page links to it.
Data and citation
- Per-site data (CSV, one row per software company site)Company names and industries from Wikidata (CC0); policy per crawler as read from robots.txt on 3 November 2026.
Bukva, A. (2026, November 3). AI crawler policies of 3,263 software companies in Wikidata (2026). LoudFace Research. https://www.loudface.co/research/software-companies-ai-crawler-policies
Questions
What readers ask about this paper
- How many software companies block AI crawlers?
- Of 3,263 software companies in Wikidata whose robots.txt we read on 3 November 2026, 8.5% (95% interval 7.6 to 9.5) blocked at least one of 15 AI crawlers by name.
- Do software companies block ChatGPT search or Perplexity?
- Rarely. Only 0.2% blocked an AI search or user-request crawler by name. In an analysis added after review, 53.5% of the companies blocking a training crawler did it through Cloudflare's legacy managed section or by blocking only Bytespider, and 31.6% named a search crawler and left it fully allowed.
- How many software companies have an llms.txt file?
- 55.8% of the 3,231 sites we could check served an llms.txt in the llmstxt.org format. Found after review: 70.5% of companies added to Wikidata since August 2024, against 30.0% of earlier ones.
