If your website sits behind Cloudflare, you have probably seen a headline in the last few days saying Cloudflare now blocks AI crawlers by default. Most of that coverage skips the two conditions that decide whether it touches your site at all. For the majority of business websites, the change that took effect on 15 September 2026 changed nothing. For a smaller group it changed something that matters. And a third group has been sitting on the wrong setting since long before any of this was announced.
From 15 September 2026, Cloudflare blocks the Training and Agent crawler categories by default on pages that display ads, and leaves the Search category allowed. The default applies to new domains, and to existing sites whose owners never configured the granular AI controls and did not opt out beforehand. If your site does not display ads, the new default does not change which crawlers can reach it. The setting most business sites actually want is Disallow AI Training, which keeps Google, Microsoft and Apple crawling you for search while blocking training-only bots.
What actually changed on 15 September
Cloudflare splits automated AI traffic into three categories, and the distinction is the whole point of the change:
- Search — crawlers that index your content so an engine can answer questions about it later, where you expect referral traffic or other compensation in return.
- Agent — automated activity acting in real time on behalf of a person, such as a chat assistant fetching your page because someone asked about you, or a browser-use agent completing a task.
- Training — crawlers collecting your content to train or fine-tune a model.
The new default blocks Training and Agent on pages that display ads, and leaves Search allowed. Cloudflare's stated reasoning is that an ad is a signal the page owner intended a human to land there and see it, so on those pages human attention is the product being sold.
For each category you can choose to block on all pages, block only on pages that display ads, or not block at all. Those controls are available from the Free plan upward.
Does the new default apply to your site?
Three questions, in order. If you answer no to the first, you can stop.
| # | Question | If no | If yes |
|---|---|---|---|
| 1 | Does your site display ads on any page? | The ad-page default does not apply. Your crawler access is unchanged by the 15 September rollout | Continue to question 2 |
| 2 | Is the domain new to Cloudflare, or on the Free plan? | Your existing preferences were migrated and carried over; check them anyway | Continue to question 3 |
| 3 | Did you configure the granular AI crawler controls, or opt out in Security settings before 15 September? | You are now on the new default: Training and Agent blocked on ad pages | Your own configuration stands |
Question 1 is the one the coverage keeps dropping. A brochure site, a lead-generation site for a professional services firm, a SaaS marketing site and a typical Shopify store do not display ads. For all of them, the 15 September default is a non-event. That is not a reason to ignore the subject — the settings underneath it still matter, and most business sites have never looked at them — but it is a reason not to make emergency changes this week.
Cloudflare also migrated existing customers' preferences onto the new controls. Site owners who never touched the granular settings were migrated based on their old Block AI Bots setting, with a previous Training selection of Block or Block on pages with ads carrying over to the new Disallow AI Training option.
Why "block AI training" used to mean "block Google"
The genuinely useful part of this change is not the default. It is the problem the default was built around.
A mixed-use crawler is one crawler doing more than one job. Googlebot fetches your pages to index them for Search, and that same crawl feeds Google's AI features. Historically that put site owners in a bind: a blunt "block AI training" rule also blocked the crawler keeping you in search results. Refusing one use meant refusing both.
Cloudflare's answer is an Accountable designation for mixed-use operators. To qualify, an operator has to give site owners a way to opt out of AI training through robots.txt or a similar standard, and a way to opt out of AI summaries. Apple, Google and Microsoft all meet or have committed to those requirements. The new Disallow AI Training setting then does something a blanket block cannot: it keeps Accountable mixed-use crawlers allowed for search, publishes your no-training preference in robots.txt through Bot Preference Sync, and blocks every other training crawler — including the training-only bots run by Amazon, Anthropic, Meta and OpenAI.
For a business whose website exists to be found, Disallow AI Training is usually the right setting and a blanket AI block usually is not. It removes your content from model training corpora without costing you search visibility or the AI answers that cite you. A blanket block costs you all three and buys you nothing you can bank.
The crawler map: who is doing what to your site
The category a bot falls into determines what blocking it costs you. This is the part worth writing down before you change any setting.
| Crawler | Operator | What it is for | What you lose if you block it |
|---|---|---|---|
| Googlebot | Search indexing, and the index that AI Overviews and AI Mode draw on | Google search visibility, AI Overviews and AI Mode citations — all of it | |
| Google-Extended | Training control for Gemini and related products | Nothing in Search. Note it does not remove you from AI Overviews, which are generated from the Search index | |
| Bingbot | Microsoft | Search indexing, and the index behind Copilot answers | Bing visibility and Copilot citations |
| OAI-SearchBot | OpenAI | Search indexing for ChatGPT's browsing results | Being findable in ChatGPT search results |
| ChatGPT-User | OpenAI | Live fetch when a person asks ChatGPT about a specific page | ChatGPT cannot read your page when a prospect asks about you by name |
| GPTBot | OpenAI | Model training | Nothing measurable in the short term |
| ClaudeBot | Anthropic | Model training | Nothing measurable in the short term |
| Applebot | Apple | Siri and Spotlight search | Apple search surfaces |
| Applebot-Extended | Apple | Training control | Nothing in Apple search |
| Meta-ExternalAgent | Meta | Training and crawling for Meta AI | Little for most business sites |
Two rows do most of the work. Googlebot is the reason a blanket block is dangerous. ChatGPT-User is the reason blocking the Agent category is a bigger decision than it looks, which is the next section.
Blocking Agent traffic breaks the answer a prospect gets about you
Training crawlers take something and give nothing back in the same session. Agent traffic is different: it is a request made because a person, right now, asked a question your page answers. When someone types your company name into ChatGPT and asks whether you work with businesses like theirs, ChatGPT-User fetches your page to answer. Block that category and the assistant answers from whatever it already has — an old model snapshot, a directory listing, a competitor comparison page — instead of from you.
For a publisher selling ad impressions, blocking agents on ad pages is a defensible commercial decision: the agent consumes the page and the human who would have seen the ad never arrives. For a business whose site exists to generate enquiries, it is close to self-harm. The agent is not intercepting revenue from you; it is carrying a prospect's question to your site and taking the answer back.
Choose a setting by what your site is for
| Site type | Search | Agent | Training | Reasoning |
|---|---|---|---|---|
| Lead-generation site for a service business | Allow | Allow | Disallow AI Training | Every citation is a potential enquiry. Training gives you nothing back |
| E-commerce store | Allow | Allow | Disallow AI Training | Agent-driven shopping is becoming real buyer traffic. Blocking it removes you from the shortlist |
| SaaS marketing site plus public docs | Allow | Allow | Disallow AI Training | Docs answered correctly inside an assistant reduce support load and pre-qualify buyers |
| Ad-supported publisher | Allow | Block on ad pages | Block on ad pages | The page monetises human attention. This is what the new default was designed for |
| Subscription or paywalled content | Allow on free pages | Block on gated paths | Block on all pages | The content is the product. Keep marketing pages open, keep the archive closed |
| Anything with a members' area, checkout or account path | — | Block | Block | Authenticated and transactional paths should never have been crawlable in the first place |
A 20-minute audit
- Establish which default you are on. Check the plan the zone is on and whether the domain was added to Cloudflare recently. Free plans and new domains are the ones the 15 September default reached.
- Open the AI crawler controls in your Cloudflare security settings and read the three categories back. Do not assume a value; the migration mapped old settings onto new ones and the result is not always what the previous owner of the account intended.
- Decide per category using the table above, not per bot. Per-bot allowlists go stale within months as operators rename and split their crawlers.
- Check whether Cloudflare's managed robots.txt is switched on, and reconcile it with the robots.txt your CMS or developer maintains. Two systems writing the same file is the most common way a site ends up blocking something nobody chose to block.
- Verify from outside. Fetch a representative page with each user agent you care about — Googlebot, OAI-SearchBot, ChatGPT-User — and confirm you get a 200 rather than a 403. Do this against a live page, not a staging copy.
- Confirm in Search Console. Use the URL Inspection tool's live test on a key page. If Googlebot is being challenged or blocked at the edge, that test is where it shows up first.
- Re-check after any CDN, WAF or hosting change. Bot rules are the setting most likely to be quietly reset by a migration.
What these controls do not cover
Three limits worth being clear about, because each one is regularly misreported.
robots.txt is a request, not a wall. Bot Preference Sync publishes your preference in a machine-readable form, which is what accountable operators read. Cloudflare's category blocking is enforcement at the edge; the robots.txt line is a statement of intent. Operators who ignore robots.txt were never going to be stopped by it.
Opting out of AI training does not remove you from AI Overviews. Google-Extended governs training for Gemini and related products. AI Overviews and AI Mode are generated from the Search index, so content Google can index can be summarised there. If you want out of those specific surfaces, the control is the Search generative AI setting in Search Console, and we walk through what that costs you in the guide to reading your AI impressions.
Blocking crawlers is not a content strategy. Nothing in Cloudflare's controls makes an AI system more likely to cite you. That depends on whether your pages answer a question completely, in extractable form, with something on them that other sources do not have. We cover that side in AEO and GEO: optimising for AI search, and treat crawler access as part of the build rather than an afterthought on every website we design and develop.
Frequently asked questions
Did Cloudflare block AI crawlers on all sites on 15 September 2026?
No. The new default blocks the Training and Agent categories only on pages that display ads, and only for new domains, new sites added by existing customers, and existing Free-plan sites whose owners never configured the granular controls and did not opt out beforehand. The Search category remains allowed by default. If your site does not display ads, the default did not change which crawlers can reach it.
Will blocking AI training crawlers hurt my Google rankings?
It depends which setting you use. A blanket block on AI crawlers can also block Googlebot, because Googlebot is a mixed-use crawler that serves both search indexing and Google's AI features, and that would remove you from search results. Cloudflare's Disallow AI Training setting exists to avoid exactly that: it keeps Accountable mixed-use crawlers from Google, Microsoft and Apple allowed for search while blocking training-only crawlers such as those run by Amazon, Anthropic, Meta and OpenAI.
What is an Accountable mixed-use crawler?
A mixed-use crawler is one crawler performing more than one job, such as search indexing and AI training in the same fetch. Cloudflare designates an operator Accountable when it gives site owners a way to opt out of AI training through robots.txt or a similar standard, and a way to opt out of AI summaries. Apple, Google and Microsoft meet or have committed to those requirements, so their mixed-use crawlers stay allowed for search under the Disallow AI Training setting.
Should a small business website block AI crawlers?
Generally no, with one exception. Blocking the Search and Agent categories removes you from AI answers that would otherwise cite you and send enquiries, which is the opposite of what most small business sites need. Blocking Training is reasonable because it costs you nothing measurable. The exception is any site whose content is the product it sells, such as a paid archive or subscription publication, where a broader block is a commercial decision rather than a technical one.
Does Google-Extended stop my content appearing in AI Overviews?
No. Google-Extended controls whether your content is used to train Gemini and related products. AI Overviews and AI Mode are generated from Google's Search index, so a page Google can index can still be summarised in them. The control for those specific surfaces is the Search generative AI setting in Search Console, which excludes your property from AI Overviews, AI Mode and generative AI features in Discover, including any impressions and traffic they would have produced.
What happens if I block the Agent category?
Assistants can no longer fetch your pages in real time when a person asks about you. ChatGPT-User, for example, fetches a page when a user asks ChatGPT about that specific site. Blocked, the assistant answers from an older model snapshot or from third-party sources such as directories and competitor comparison pages instead. For an ad-supported publisher that trade may be worth making; for a site that exists to generate enquiries it usually is not.
The takeaway
Check the setting, then ignore the headline. If your site does not display ads, nothing changed for you on 15 September, and the sensible response is a twenty-minute audit rather than an emergency change. If it does display ads, the new default is probably close to what you would have chosen anyway. The decision that matters for almost every business site is the one underneath the news: keep Search and Agent open so that search engines and assistants can find you and answer questions about you accurately, and use Disallow AI Training to keep your content out of model training without paying for that with your visibility. The worst outcome here is not being trained on. It is quietly blocking the crawler that was bringing you customers.
Sources & further reading
- Cloudflare, "Your site, your rules: new AI traffic options for all customers"
- Cloudflare, "Have it both ways: stay discoverable in search while disallowing AI training"
- Cloudflare, "Say it once: introducing Bot Preference Sync"
- Cloudflare Docs, "New options to manage AI traffic" (changelog, 1 July 2026)
- Cloudflare Docs, "Manage AI crawlers" (AI Crawl Control)
- Google Search Central, "AI features and your website"
- OpenAI, "Overview of OpenAI crawlers"