Cloudflare changed its AI crawler defaults. Check whether you blocked Googlebot too

On 15 September 2026 two things happened at Cloudflare on the same day. New defaults began blocking AI training and agent crawlers, and a new setting shipped to stop those blocks from taking Googlebot down with them. If your site sat behind Cloudflare through that window, the honest answer to "am I still being crawled for search?" is that you do not know until you check. Here is what each control actually does, and the audit that settles it in about twenty minutes.

What changed, in order

The sequence matters, because which side of it your settings were written on determines what you need to do now.

Why a training block used to remove you from search

Because Googlebot does two jobs with one user agent. So do Bingbot and Applebot. They crawl to index pages for search, and the same infrastructure feeds AI products. Cloudflare's original purpose-based enforcement bucketed a mixed-use crawler into the most restrictive category that applied to it, so a site that said "no training" said "no Googlebot" without meaning to. Search Engine Journal reported sites losing indexing as a result, which is about as expensive a configuration mistake as a website can make.

The September setting resolves it by separating consent from access. Choosing Disallow AI Training publishes a no-training preference in robots.txt and lets operators Cloudflare classifies as Accountable continue crawling for search, while training-only crawlers stay blocked. Reporting on the launch named the training-only crawlers of Meta, OpenAI, Amazon and Anthropic among those shut out under that setting.

What each setting actually does to each crawler

This is the table that was missing when people were making these decisions in a hurry. Read down the column that matches your current Cloudflare configuration.

Two consequences fall out of that table that are easy to miss. First, blocking Agent is a different decision from blocking Training, and it is the one that affects whether an AI assistant can fetch your page while a prospective customer is asking about you. Second, none of these controls governs AI Overviews or AI Mode. Those are served from Google's search index. If you are indexed for search, you are eligible to appear in them, and the levers there are the ordinary snippet controls — nosnippet, max-snippet, data-nosnippet — not a crawler category at your CDN.

The six-step audit

Run this in order. The first three steps tell you whether you have a live problem; the last three tell you whether your configuration is what you actually intended.

1. Search Console → Settings → Crawl stats. Look at total crawl requests over the last 90 days and the by-response breakdown. A cliff starting mid-September, or a rise in 403 and 503 responses, is the signature of a CDN-level block. This is the single fastest diagnostic and most site owners never open it.
2. URL Inspection on three live URLs — your home page, a money page and a recent article. Use "Test live URL". A live fetch that fails while the indexed version still looks fine means you are currently blocked and simply have not decayed out of the index yet.
3. Cloudflare dashboard → bot analytics. Filter to verified bots and check whether Googlebot, Bingbot and Applebot requests are being allowed. If they are being challenged or blocked, you have your answer and your cause in one screen.
4. Check which AI settings you are actually on. In Cloudflare's security settings, confirm whether you are on Disallow AI Training or a legacy blanket training block, and whether Agent is blocked. The launch described existing configurations as generally migrating; "generally" is not "certainly", and this is a thirty-second check.
5. Fetch your own robots.txt and read it. Cloudflare's setting writes a preference there. If you also maintain robots.txt rules by hand, look for contradictions — a hand-written Disallow: / under a wildcard user agent, or a stale Google-Extended block you no longer want, will quietly beat your intentions.
6. Confirm the decision was deliberate. If nobody on your team can say who chose the current setting and why, you inherited a default. Decide it properly using the table below and write it down somewhere your next developer will find it.

Lag is the thing that makes this dangerous. Blocked crawling does not produce an instant ranking drop; it produces stale indexing, then decay over weeks. By the time traffic visibly falls, the cause is a month behind you and looks like an algorithm update. Crawl stats are where it shows first, which is why step one is step one. If you want the wider version of this check, our guide to diagnosing a site that isn't showing on Google works through the other common causes.

Should you disallow AI training at all?

The honest answer for most businesses is that it matters less than the debate suggests, and the right choice depends on whether your content is the product. Cloudflare's Accountable designation is what makes the choice cheap: to qualify, an operator must give site owners a clear way to opt out of training, let them opt out of AI-generated search summaries, provide URL-level visibility into how content is used for search and training, and publicly confirm that opting out of training will not affect ranking in ordinary search. Apple, Google and Microsoft have been designated Accountable, having met the criteria or committed to timelines, and Cloudflare Radar publishes what operators actually do.

The pattern is that blocking training costs you little and can be defended on principle. Blocking agents costs you the AI-assistant discovery surface that is still cheap to win. Very few businesses that sell services should be doing the second. If your priority is being surfaced and cited by AI answer engines, the work is on the content and structure side — the ground our testing of llms.txt and AI Mode behaviour covers — not on tightening your CDN.

What this tells you about infrastructure risk generally

The specific lesson is narrow: check your Cloudflare settings. The general one is more useful. Your search visibility now depends on defaults set by companies that are not your hosting provider, not your CMS vendor and not you, and those defaults change on their own schedule. In the past year the same pattern has produced a Shopify checkout deadline that broke tracking on stores whose owners changed nothing, a Chrome release cadence that doubled, and now a CDN default that can remove a site from crawling.

The response is not to distrust the platforms. It is to hold a short list of externally-owned settings that can break your site without a deploy, and to check them on a fixed cadence. For most businesses that list is four lines long: CDN bot and crawler settings, DNS and certificate expiry, analytics and conversion tracking, and payment or checkout platform deadlines. None of those live in your codebase, which is exactly why nobody notices when they move. The same reasoning is why a Q4 code freeze protects less than teams assume: freezing your own deploys does nothing about changes that arrive from outside.

Frequently asked questions

Did Cloudflare block Googlebot in September 2026?

Not by default, but a site that used Cloudflare's blanket "block Training" option could block Googlebot without meaning to, because Googlebot is a mixed-use crawler that serves both search indexing and AI products, and mixed-use crawlers were bucketed into the most restrictive category applied to them. On 15 September 2026 Cloudflare released a Disallow AI Training setting that publishes a no-training preference while still allowing accountable mixed-use crawlers such as Googlebot, Bingbot and Applebot to crawl for search.

What changed in Cloudflare's crawler defaults on 15 September 2026?

From 15 September 2026, Training and Agent crawlers are blocked by default on pages that display ads, while Search crawlers remain allowed. The new defaults apply to new domains, new sites added by existing customers and existing free-tier customers. Paying customers can override the defaults in the Cloudflare dashboard.

How do I check whether Googlebot is still crawling my site?

Open Search Console, go to Settings then Crawl stats, and look for a drop in total crawl requests or a rise in 403 and 503 responses from mid-September onward. Then run URL Inspection with "Test live URL" on two or three pages: if a live fetch fails while the indexed copy still looks normal, you are blocked now and have not yet decayed out of the index. Finally, check verified-bot traffic in Cloudflare's bot analytics.

Does blocking AI training remove my site from AI Overviews or AI Mode?

No. AI Overviews and AI Mode are generated from Google's search index, so if your pages are crawled and indexed for search they remain eligible to appear. The controls that limit how your content is used in those features are the ordinary snippet directives, such as nosnippet, max-snippet and data-nosnippet, rather than a crawler category setting at your CDN.

Does the Content Signals directive in robots.txt actually stop AI crawlers?

No. It expresses a preference. As of mid-2026 there was no known adoption of the directive among major crawlers or LLM providers, and Google's John Mueller said he was not aware of any crawler or LLM using it. It is harmless to publish, but it should not be relied on as an enforcement mechanism.

Should a small business block AI crawlers on its website?

For most service businesses, no. Blocking AI training is a defensible choice with little traffic cost, but blocking Agent crawlers removes your pages from the real-time fetches that AI assistants make while someone is comparing suppliers, which is discovery you currently get cheaply. Publishers and sites whose licensed content is the product are the clear exception, and they are better served by path-level rules than a site-wide block.

The takeaway

Two changes landed at Cloudflare on the same day: defaults that block AI training and agent crawlers, and a setting that stops a training block from taking your search crawling with it. Both moved without anybody deploying anything on your side. Spend twenty minutes in Search Console crawl stats, URL Inspection and Cloudflare bot analytics to establish what state you are actually in, then decide the setting deliberately — disallow training if your content is the product, and leave agent access open unless you have a specific reason to close the door on AI-assistant discovery. Then add CDN crawler settings to the short list of externally-owned things you check quarterly, because this will not be the last default that changes on its own.