Blocking AI crawlers without losing search visibility: what each control actually does

Two things happened this year that changed the answer to "should we block AI bots". Cloudflare stopped treating AI crawlers as one category and started sorting them by what they do with the page, and on 15 September it changed its defaults, which means some sites are now blocking crawlers nobody on the team decided to block. The decision is more precise than it used to be. It is also still misunderstood in one specific and expensive way.

What actually changed in 2026

Until this year, a bot block was a blunt instrument: you allowed a crawler or you did not, and if a crawler did more than one job, blocking it for one purpose blocked it for all of them. That is why so much early advice about blocking AI was quietly wrong. Disallowing a multi-purpose crawler to keep your content out of a training set also took you out of the index that sends you traffic.

Since 1 July 2026, Cloudflare has managed AI crawlers by three declared purposes for every customer, free plans included:

1. Search. The crawler builds an index so an engine can answer a question later and link back to you. This is the referral channel, and it is the one that has always paid for itself.
2. Agent. The crawler fetches your page in real time because a person has just asked something. ChatGPT retrieving a page mid-conversation is an agent fetch. It is the closest thing in this list to an actual visit, and it is the category most people block by accident.
3. Training. The crawler collects content to train or fine-tune a model. It sends no referral traffic at the time of the crawl and none later.

Then on 15 September 2026 the defaults moved. For new Cloudflare customers, for new sites added by existing customers, and for all free plans, Training and Agent crawlers are now blocked by default on pages that display ads, while Search stays allowed. Sites already active on Cloudflare were not switched over, so an established site keeps whatever it had. The practical consequence is a split: if your site launched or moved to Cloudflare after mid-September, your AI crawler policy was set by a default, not by you.

The five controls, and what each one actually enforces

Most of the confusion in this area comes from treating these as five versions of the same switch. They are not. Two are enforced, two are requests, and one is a declaration that the largest search engine has documented as unsupported.

Read the "Enforced?" column as the dividing line between a policy and a control. Declaring a preference is worth doing, because it establishes intent and because Cloudflare has said verified bots that ignore stated preferences or reproduce content in full can lose verified status. But a declaration is not a block. If you need a crawler stopped, it has to be stopped at the edge.

Why you cannot opt out of AI Overviews

This is the part that still gets sold as solvable, and it is not. AI Overviews are built from content Googlebot fetched. There is no separate AI Overviews crawler to disallow, and the controls Google honours for AI features are the snippet controls, which apply to your normal listing in the same breath. Set nosnippet and you are not removing yourself from AI Overviews while keeping your blue link intact; you are removing the description under your blue link as well.

Cloudflare's Content Signals Policy was written precisely to close this gap, by letting a site say "index me for search, but do not use me as a real-time input to generated answers". Google's response was unambiguous: its robots.txt documentation lists four supported fields, Content-Signal is not one of them, and Google has said the directive has no effect. So the honest position in October 2026 is that if your traffic is falling because AI Overviews answer the query on the results page, no robots.txt line fixes that. The remedies are editorial and structural, not access control, which is the subject of how AI assistants choose what to cite.

Worth adding the same correction we made about llms.txt: a file or a directive that a search engine has not committed to reading is not a strategy. It costs nothing to add and it changes nothing by itself.

The cost side: crawl-to-refer ratios, and why one number misleads

The economic argument for blocking rests on how many times a crawler takes a page for each visitor it sends back. Trackers using Cloudflare Radar data put Google at roughly 5 crawls per referral in mid-2026, Perplexity near 186:1, OpenAI in the hundreds, and Anthropic in the thousands. Those are real orders of magnitude and they are worth knowing.

They are also extremely unstable, and anyone quoting you a single ratio is quoting a snapshot. The same reporting puts Anthropic near 23,951:1 across January to March 2026, around 10,300:1 at the end of May, and roughly 4,580:1 in June. OpenAI moved from about 1,104:1 in July 2025 to the low hundreds a year later. A ratio that improves by a factor of five in six months is not a stable input to a permanent decision.

Two more figures put the referral side in proportion. Of AI crawler requests Cloudflare observed in May 2026, 51.8% were for training and only 9.3% were for search. And across the same month, every AI chatbot combined accounted for 0.29% of search referrals Cloudflare Radar observed, against 87.63% from Google.

The decision matrix

Blanket advice fails here because the three purposes have genuinely different value depending on what your site is for. A publisher funded by ad impressions and a plumber funded by phone calls should not reach the same answer. Here is how we set it, by purpose and by business model.

The rule behind the table: allow the purposes that can send you a customer, and decide training on whether the content is the product or an advert for the product.

That last clause is where most write-ups stop short. Blocking training crawlers is usually described as free, and in referral terms it is. But training data is how a model comes to know that you exist at all, which is the only route to being named in an answer where nobody searched for your category. For a research publisher or a course business, the content is the asset and that trade is obviously wrong. For a regional web development firm or a retailer, the content exists to win customers, and brand presence in a model's answers is worth more than the licensing value of a service page. Choose blocking when you sell access to the content; choose allowing when the content exists to sell something else.

A six-step exposure audit

Before changing anything, find out what you are actually doing today. In our experience almost no site's real behaviour matches what the team believes it is, usually because a default, a plugin or a security rule made a decision nobody recorded.

1. Read your edge configuration. In Cloudflare, that is AI Crawl Control plus any WAF or rate-limiting rules that match on user-agent or ASN. Note which purposes are blocked and whether a rule is set site-wide or only on ad-carrying pages.
2. Fetch your own robots.txt and read every line. Look for directives you did not write. Bot preference settings can write instructions into robots.txt, and SEO plugins add their own. A site can carry a blanket AI disallow that arrived with a plugin update.
3. Compare the two. The common failure is a robots.txt that politely allows a crawler while an edge rule blocks it, or the reverse. Where they disagree, the edge wins and your stated policy is fiction.
4. Segment bot traffic in your logs. Group requests by user-agent into search, agent, training and unverified, for a full month. This gives you your own crawl ratios instead of someone else's, and it is the only way to know whether crawler load is a real hosting cost on your site or a headline about somebody else's.
5. Test retrieval from the outside. Ask an assistant a question your site should answer and see whether it can fetch and cite the page. A clean robots.txt with an aggressive firewall often fails here, and this is the single fastest way to find an accidental agent block.
6. Check what you are already getting. Pull referrals from AI sources and the trend in branded and direct traffic. If assistant referrals are growing from a small base, blocking agent crawlers is cutting a channel while it is still compounding.

Steps 2 and 3 belong in a recurring technical check rather than a one-off, for the same reason the rest of the items in our technical SEO audit checklist do: these files change without anyone announcing it.

The contradiction in the survey data, and what explains it

A small-business survey reported in late 2025 found 44% of respondents getting more leads from AI-powered search, 42% more traffic and 33% more inquiries, while more than half block AI crawlers. 43% said they block over content ownership and intellectual property, and 16% reported unauthorised scraping in the previous year.

Those findings only look contradictory until you separate the purposes. Intellectual property concern is an argument about training. The leads and traffic come from search and agent crawlers. A business can hold both positions coherently, and the tiered policy above is what holding both positions looks like in a configuration file. The incoherent version is the blanket block, which answers the training concern by also closing the channel that was producing the leads.

What we would actually do, in order

1. Allow Search everywhere. No business model in the table above benefits from blocking it.
2. Allow Agent, and check that it genuinely works by testing retrieval rather than by reading a setting. If you are a publisher paid per impression, meter or price it instead of blocking it.
3. Decide Training once, in writing, using the product-or-advert test, and record the decision where the next person to touch the CDN will find it.
4. Declare it and enforce it. Content signals and robots.txt for intent, edge rules for enforcement. Expect the declaration to be ignored by anyone who wants to ignore it.
5. Stop trying to block AI Overviews. Spend that effort on being the page worth citing instead, which is a content and structure problem.

Frequently asked questions

Does blocking AI crawlers hurt your Google rankings?

Blocking crawlers whose purpose is model training does not affect Google rankings, because Google's search index is built by Googlebot, which is a separate crawler. Blocking Googlebot itself, or applying noindex, removes you from Google. The middle case is the one that costs you something: blocking agent crawlers such as the ones ChatGPT and Perplexity use to fetch a page in real time removes you from those answers without touching your Google position.

Can you block Google's AI Overviews but keep your normal search listing?

No. AI Overviews are fed by Googlebot, the same crawler that builds your normal listing, and the only controls Google honours for AI features are nosnippet, data-nosnippet, max-snippet and noindex. Every one of those also shortens or removes your ordinary search result. Cloudflare's Content Signals Policy was written to close that gap, but Google's robots.txt documentation lists only four supported fields, so the ai-input signal has no effect on Google.

What changed in Cloudflare's AI crawler settings on 15 September 2026?

From 15 September 2026, Cloudflare blocks Training and Agent crawlers by default on pages that display ads, while leaving Search crawlers allowed. The new defaults apply to new customers, to new sites added by existing customers and to all free plans. Sites already active on Cloudflare are not switched over automatically, so an existing site keeps whatever settings it had.

What is the difference between a search, agent and training crawler?

A search crawler indexes your content so an engine can answer questions about it later and link back to you. An agent crawler fetches a page in real time on behalf of a person who is asking a question right now, which is the closest thing in this category to a visit. A training crawler collects content to train or fine-tune a model, and returns no referral traffic at any point. Cloudflare has managed AI crawlers by these three purposes for all customers, including free plans, since 1 July 2026.

Does robots.txt actually stop AI crawlers?

Only the ones that choose to comply. A robots.txt disallow is a request, not an access control, and it is honoured by the major compliant crawlers and ignored by everything else. Enforcement has to happen at the edge, through a CDN or firewall rule that returns a block or a payment-required response. Use robots.txt to declare intent and the edge to enforce it.

Is it worth blocking AI training crawlers?

It depends on whether your content is your product. If you sell access to content, research or courses, blocking training crawlers protects the thing you sell and costs you no measurable referral traffic. If you are a service business or a retailer, the content exists to win customers, and being present in a model's training data is one of the few ways your brand gets named in an answer where nobody searched for you. Blocking training is cheap, but it is not free for everyone.

The takeaway

The useful change this year is that the question stopped being "AI bots: yes or no". It is now three questions with three different answers, and for most businesses the answers are allow, allow, and it depends on whether you sell content. The expensive mistakes are at the edges: a blanket block that closes the agent channel to solve a training complaint, and the belief that some robots.txt line will take you out of AI Overviews while leaving your search listing intact. The first is fixable in an afternoon. The second is not fixable at all right now, and the sooner that is accepted, the sooner the effort goes somewhere it can work.