Should you block AI crawlers? What the 2026 crawl-to-referral data actually says

The advice circulating since last year is neat enough to be suspicious: block the training crawlers, allow the search crawlers, get cited without feeding the models. It is good advice for a newspaper. For most businesses it is answering a question they do not have, using a metric that does not apply to them.

The three kinds of bot people lump together

Most robots.txt guides list user agents alphabetically, which hides the only distinction that matters. AI crawlers do three different jobs, and blocking them has three completely different consequences.

Two consequences fall out of that table immediately. Blocking Google-Extended stops your content being used for Gemini training without affecting Google Search rankings, because it is a separate token from Googlebot — which is why it is the least controversial block on the list. And blocking Googlebot to stay out of AI Overviews is self-harm, because the same crawl powers your organic listings; AI Overview eligibility is governed by Google's own snippet and generative AI controls, not by refusing the crawl.

The economics that started the argument

The case for blocking rests on one number: how many pages a bot takes for each visitor it sends back. Cloudflare Radar's figures for the 28 days ending 21 July 2026 are stark.

Two pieces of context make those ratios less mysterious. Cloudflare's May 2026 breakdown found that search-purpose crawling — the only kind that can produce a citation with a link — accounted for under 10% of AI crawler requests, with roughly 90% going to training and answer generation where the user never leaves the chat window. And on 3 June 2026 Cloudflare's chief executive put automated requests at 57.5% of all HTML web traffic, against 42.5% human.

By share of verified bot traffic, Google runs about 28.4%, Anthropic 13.2%, Meta 12.2% and OpenAI 7.2%.

Why the ratio does not settle it for most businesses

Crawl-to-referral is a publisher metric. It measures how much content was taken against how much traffic came back, and it is the correct metric when the content is the inventory you sell advertising against. That is why the loudest voices in this argument are news organisations, and they are right about their own case.

Now apply it to a 180-page website for a services business. A 2,237:1 ratio against 180 URLs is a few thousand requests. It is not a bandwidth problem, it is not a hosting cost, and it is not lost revenue, because those pages were never monetised by pageview. The number is alarming as a ratio and negligible as a quantity.

The asymmetry only becomes a real cost when at least one of these is true:

• You have a large URL surface. Tens of thousands of pages, faceted navigation, a big product catalogue, an events or listings archive. Then crawl volume is genuinely a server and cost line, and it is worth rate-limiting regardless of who is doing the crawling.
• Pageviews are the revenue. Advertising, affiliate, subscriptions. An answer given in the chat window is a substituted visit, and substitution is a direct loss.
• The content is the asset itself. Proprietary research, original datasets, course material, anything a buyer would otherwise pay to read.
• You have a licensing position to protect. If you might sell access to a model developer later, giving it away now weakens the negotiation.

If none of those apply — and for most agencies, SaaS companies, professional services firms, clinics, manufacturers and local businesses, none of them do — then the crawl is not a cost you are paying. It is closer to free distribution of your marketing, and the risk of blocking runs the other way.

The decision rule

Robots.txt is a request, not a fence

Whatever you decide, be clear about what the mechanism does. Robots.txt is a politeness convention. The major named crawlers honour it, which is exactly why they are the ones you can see in your logs. Agents that ignore it are, by definition, not in the list you were considering blocking.

So if a block genuinely matters — confidential material, licensed data, a legal obligation — robots.txt is the declaration and not the control. Pair it with an edge or WAF rule that enforces by verified identity, and check your logs afterwards to confirm the rule is doing what you think. If a block does not matter enough to enforce, it probably did not need to exist.

Three practical notes:

1. Scope blocks to paths, not the whole site. A blanket disallow is a blunt instrument that usually catches the pages you most want cited.
2. Do not block what you cannot name. A user-agent string that is guessed or copied from an out-of-date list does nothing except make the file look decisive.
3. Re-read the file every six months. These agent names change, and a robots.txt written in 2024 is now blocking crawlers that no longer exist while allowing ones that did not.

The cost of over-blocking, which nobody puts a number on

The blocking case has a headline number and the allowing case does not, which is why the argument tilts one way. But the risk is real and it is asymmetric in time: an AI assistant that cannot retrieve your pages will answer the buyer's question using a competitor's. You will not see that in any report. There is no impression, no bounce, no line in analytics for the answer you were absent from.

For a business whose buyers research before they enquire, that is a worse outcome than any amount of uncompensated crawling. It is also harder to reverse than it looks: allowing the crawler back does not immediately restore whatever standing your content had built in the systems that were indexing it.

The practical version of this: if you want to be found in AI answers, the work is the same work that makes you findable in search — clear pages, specific claims, primary sources, structure a machine can parse. That is the substance of answer engine optimisation, and it is wasted if the crawler cannot reach the page. It also has a downstream half: traffic arriving from an assistant behaves differently from search traffic, which is why landing pages for ChatGPT and Perplexity traffic need their own treatment rather than inheriting your paid search templates.

A sane default

For the large majority of business websites, the correct robots.txt change this year is none, plus three things that are not robots.txt: log-based monitoring so you can see who is actually crawling you, edge rate limits so that no crawler — AI or otherwise — can degrade the site for real users, and a genuine block, enforced at the edge, on the specific paths that hold anything confidential.

Blocking AI crawlers is a decision about your business model. It is not a security control, it is not a technical best practice, and it is not free. Make it on the strength of what you sell, not on the strength of a ratio calculated for somebody else's business.

Frequently asked questions

Should I block AI crawlers on my website?

It depends on whether content is your product or your marketing. If you monetise pageviews or sell content directly, block the training crawlers and keep the retrieval crawlers. If you sell a service or a product and your content exists to generate enquiries, block nothing except abusive traffic: the crawl costs you almost nothing on a small site, and blocking retrieval crawlers removes you from the AI answers your buyers are reading.

How many pages do AI crawlers take per visitor they send back?

Cloudflare Radar data for the 28 days ending 21 July 2026 shows Mistral crawling about 3,389 pages per referred visitor, Anthropic's ClaudeBot about 2,237, OpenAI's GPTBot about 217, and Google about 4.6. Cloudflare's May 2026 breakdown also found that search-purpose crawling, the only kind that can produce a citation with a link, was under 10% of all AI crawler requests.

What is the difference between training crawlers and retrieval crawlers?

Training crawlers such as GPTBot, ClaudeBot and CCBot collect content used to train models, and never send a link back. Retrieval crawlers such as OAI-SearchBot and PerplexityBot build the index an assistant searches when answering a live question, and produce the citations and referral clicks. Blocking the first category costs you nothing measurable. Blocking the second removes you from AI answers.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended is a separate robots.txt token from Googlebot. Disallowing it stops your content being used for Gemini model training without affecting how your pages are crawled, indexed or ranked in Google Search. Blocking Googlebot itself would remove you from organic results, so it is never the right way to influence AI Overviews.

Can I stop my site appearing in AI Overviews?

Not by blocking a crawler, because AI Overviews draw on the same Googlebot crawl that produces your organic listings. Eligibility is governed by Google's snippet and generative AI controls rather than by refusing the crawl, and Google rolled out a Search generative AI control alongside its generative AI performance reports in 2026. Blocking Googlebot to avoid AI Overviews would cost you organic search entirely.

Does robots.txt actually stop AI crawlers?

Only the ones that choose to obey it. Robots.txt is a politeness convention, not an enforcement mechanism: the major named crawlers honour it, which is why they appear in your logs at all, and agents that ignore it were never going to read your file. If a block matters legally or commercially, pair the robots.txt rule with an edge or WAF rule and verify it in your server logs.

How much traffic on a normal website is bots?

More than most people expect. Cloudflare's chief executive stated on 3 June 2026 that automated requests account for 57.5% of all HTML web traffic, against 42.5% from humans. AI crawlers are only part of that: by share of verified bot traffic, Google runs about 28.4%, Anthropic 13.2%, Meta 12.2% and OpenAI 7.2%. Check your own logs before assuming the web-wide split matches your site.

The takeaway

Pull your logs, count requests by user agent, and decide from your own numbers. If you are a publisher, the standard split is right: block training, keep retrieval, and enforce it at the edge rather than trusting a text file. If you are a business that uses content to win customers, the honest answer is that the crawl is not costing you anything you can measure, and the block would cost you presence in the answers your buyers are already reading. The bandwidth argument is real for a handful of sites and imaginary for almost everyone else.