Cloudflare now blocks AI crawlers by default. Should your site let them in?

On 15 September 2026, Cloudflare changed what happens by default when an AI crawler arrives at a site it sits in front of. Most of the coverage has been written for publishers, and most site owners have concluded it does not affect them. That conclusion is right about the default and wrong about the risk, because the setting that genuinely can hurt you is one a lot of teams switched on themselves, quietly, some time in the last eighteen months.

What actually changed on 15 September 2026

Cloudflare announced in July 2026 that AI companies had until 15 September to separate their crawlers by purpose: one user agent for search indexing, another for AI training, another for fetching a page to answer a live user's question. Crawlers that still blend those jobs into a single agent are classed as "mixed-use", and from that date Cloudflare's defaults block them on pages that host ads.

The scope of the default matters more than the headline. It applies to new Cloudflare customers, to new sites added by existing customers, and to all existing free-tier accounts. Existing paying customers keep the settings they already have and can change them in the dashboard either way. And it is scoped to monetised pages, which is Cloudflare acting on behalf of publishers whose business model is ad impressions against human visits.

So if you run a service business, a B2B site or a store with no display advertising on it, the 15 September flip is not the thing to worry about. It is the wrong story for most of the businesses reading about it.

The setting that can actually cost you rankings

The real exposure is in Cloudflare's AI Crawl Control, where the categories are search, AI training and AI assistants, and where the block on AI training has a side effect that is not obvious from the toggle.

Cloudflare classifies Googlebot and Bingbot as mixed-use crawlers, because Google and Microsoft use them for both search indexing and AI. When a request matches more than one category, the most restrictive rule applies. So a site that sets AI training to block is telling Cloudflare to refuse requests from a set of crawlers that includes the one responsible for its entire organic search presence. Site owners have reported Googlebot receiving 403 responses, including on sitemap fetches, after enabling AI bot blocking.

The damage from that is slow and easy to misread. Nothing disappears the next morning. Pages stop being recrawled, changes stop being picked up, new pages never get indexed at all, and rankings decay over weeks in a pattern that looks like a core update if you are not looking at your server logs. By the time it is obvious in Search Console, you have lost a quarter.

Four controls, four different jobs

Most of the confusion here comes from treating "blocking AI" as one decision. It is four separate controls with different mechanisms and very different costs.

The practical rule that falls out of that table: express training preferences in robots.txt, where the worst case is that a crawler ignores you, and be extremely careful with enforced blocking at the network layer, where the worst case is that you deindex yourself.

One thing blocking Google-Extended does not do

A widespread belief is that adding Google-Extended: disallow to robots.txt keeps you out of AI Overviews. It does not. Google's documentation is explicit that AI Overviews and AI Mode are surfaces within Search, served from the standard Search index that Googlebot builds, while Google-Extended governs whether your content can be used for training and grounding in Gemini Apps and the Vertex AI API.

If you want to limit what Google can display from your page inside those surfaces, the levers are the snippet controls, and they cut both ways: a page that cannot be quoted cannot be cited either. That trade-off is the whole substance of the AI-visibility question, and we went through it in detail in our guide to AEO and GEO.

Which crawlers should you actually let in?

The useful distinction is not human versus machine. It is whether a crawler can send you anything back.

Note where the two positions differ. Publishers, whose product is the content, have a real argument for blocking assistant crawlers and pursuing compensation instead: Cloudflare is building exactly that market, with Pay Per Use replacing the earlier Pay Per Crawl beta and paying publishers when their content is used in an AI answer rather than merely fetched. A dental practice, a law firm or a SaaS company is not in that business. Their content is a customer acquisition channel, and an AI assistant citing it is the channel working.

Choose by what your content is for

1. If content is your acquisition channel, services, B2B, local, e-commerce category and guide pages, allow search and assistant crawlers, and decide on training separately in robots.txt. The cost of being invisible to an assistant is immediate; the benefit of blocking training is speculative.
2. If content is your product, publishing, research, paid courses, data, block training, consider blocking assistants for premium content, and look at the emerging compensation marketplaces. Gate what is genuinely valuable rather than trying to police it after it has been fetched.
3. If you sit in between, a store with a large editorial library, a consultancy with proprietary research, split it by URL path. Block at the level of the premium directory, not the whole domain, and never with a setting that can catch Googlebot along the way.

A fifteen-minute audit

1. Cloudflare dashboard. Check AI Crawl Control, the AI bot blocking toggle and managed robots.txt. Confirm search crawlers are allowed. If AI training is set to block, confirm the mixed-use opt-out that keeps Googlebot and Bingbot flowing.
2. Search Console crawl stats. Look at the last 30 days for 403 responses and any drop in crawl requests after mid-September. This is the evidence, not the setting.
3. Fetch your own robots.txt. If your host or CDN injects a managed file, what is live may not be what is in your repository. Read the deployed version.
4. Index coverage. Check that recently published pages are being indexed. A crawler block shows up here before it shows up in traffic.
5. Decide training separately, in writing. Whatever you choose, record it in robots.txt with Content Signals and note the decision somewhere your next developer will find it.

If the audit turns up a block you did not know about, fix it and expect recovery to take as long as the decay did. Crawl budget is rebuilt gradually, which is one more reason to treat this as a monthly check rather than a one-off. It belongs in the same routine as the rest of your technical SEO checks.

What this means for your site strategy

The AI crawler debate is being framed as a stand for or against AI companies. For most businesses it is not that. It is a plumbing decision with an asymmetric downside: the upside of blocking training crawlers is a principle, and the downside of blocking the wrong crawler is your organic pipeline. Take the principle if you want it, but take it in the file where the worst outcome is being ignored.

The wider shift worth paying attention to is that access to your content is becoming a thing you configure rather than a thing you assume, and that configuration now sits across your CDN, your robots.txt and your page-level tags. Sites built and maintained without anyone owning that surface will keep discovering their settings by accident. If you are planning a build or a rebuild in the next few months, put crawler policy in the handover documentation alongside DNS and hosting. It is now that category of decision.

Frequently asked questions

What changed with Cloudflare's AI crawler settings on 15 September 2026?

From 15 September 2026, Cloudflare's defaults block "mixed-use" AI crawlers, meaning crawlers that combine search indexing with AI training or agent data collection, on pages that carry ads. The default applies to new Cloudflare customers, new sites added by existing customers, and all existing free-tier accounts. Existing paying customers keep their current settings and can override the behaviour in the dashboard.

Does blocking AI crawlers on Cloudflare block Googlebot?

It can. Cloudflare classifies Googlebot and Bingbot as mixed-use crawlers because they serve both search and AI, and when a request matches more than one category the most restrictive rule applies. Site owners have reported Googlebot receiving 403 responses, including on sitemap fetches, after enabling AI training blocks. Verify that search crawlers are explicitly allowed before or after enabling any AI block.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google's documentation states that AI Overviews and AI Mode are surfaces within Google Search, served from the standard index built by Googlebot, while the Google-Extended token controls whether content may be used for training and grounding in Gemini Apps and the Vertex AI API. To limit what Google can display from a page in those surfaces you need snippet controls such as nosnippet, data-nosnippet or max-snippet, which also reduce your eligibility to be cited.

Should a small business block AI crawlers at all?

Usually not all of them. Split the decision by crawler purpose: always allow search indexing crawlers, allow assistant crawlers such as OAI-SearchBot and PerplexityBot because they can cite and link you, and treat training crawlers as a separate choice expressed in robots.txt. For a business whose content is a customer acquisition channel rather than the product itself, being cited in an AI answer is the channel working, not a leak.

What are Content Signals in robots.txt?

Content Signals is a robots.txt extension that lets a site declare how its content may be used, using three yes or no signals: search for building a search index, ai-input for feeding a page into an AI model to answer a live query, and ai-train for training or fine-tuning a model. A newer use parameter expresses how content may be reused. These are stated preferences rather than enforced blocks, and not every automated system honours them.

How do I tell whether a crawler block is already hurting my rankings?

Check Search Console crawl stats for 403 responses and for a drop in crawl requests dated after the change, then check index coverage for recently published pages. A network-level block shows up as reduced crawling and unindexed new pages weeks before it shows up as a traffic decline, which is why it is often misdiagnosed as a core update.

The takeaway

The 15 September default flip is aimed at publishers and scoped to ad-bearing pages, so it probably does not change anything on your site by itself. The setting that can change everything is the AI training block in your own Cloudflare dashboard, because Cloudflare treats Googlebot and Bingbot as mixed-use crawlers and applies the strictest matching rule. Keep search and assistant crawlers allowed, express training opt-outs in robots.txt where being ignored is the worst case, and check Search Console crawl stats for 403s before you assume none of this reached you.