Cloudflare Blocks AI Crawlers by Default on September 15. What Site Owners Must Do

From September 15, 2026, Cloudflare stops treating AI crawlers as one group and sorts them into three jobs, search, agent, and training, then applies a new default on ad-supported pages that allows search crawlers but blocks training and agent crawlers. Any crawler that serves more than one purpose and will not declare which job it is doing on a given request is blocked entirely on those pages. Crucially, for the millions of sites on Cloudflare’s free tier and for every new sign-up after that date, this blocking is on by default, so allowing AI training access becomes a manual opt-in rather than something you have to actively turn off. Existing paying customers keep the ability to override the defaults from their dashboard. Because this is a policy applied at Cloudflare’s edge, it can change your site’s crawler behavior without any change on your side: it will not appear in your CMS, it raises no error, and your origin server is unchanged. The crawler simply gets a different answer at the edge, which is why it is worth checking your settings before the deadline.

It can, in two ways, which is why it is worth ten minutes to check. First, if your site is on Cloudflare’s free tier or you sign up after September 15, the new blocking defaults apply automatically, so AI agent and training crawlers start getting blocked on your ad-supported pages without you choosing it. Second, and more subtly, blocking the training function can also block multi-purpose crawlers such as Googlebot, Bingbot, and Applebot even when you meant to keep search crawlers allowed, because those crawlers do more than one job. Since Bing’s index is a major input to ChatGPT’s search and Google’s crawl feeds AI Overviews, a training block that cascades to those crawlers can quietly remove you from AI answers. The important thing is that none of this produces a visible symptom: no error, normal analytics, your site up and fine, while the crawlers that put you in front of AI users are turned away at the edge. To avoid an unintended loss, keep search crawlers allowed, and if you block training, verify afterward that you did not also block a search crawler.

They are three different jobs an AI crawler can do, and blocking each costs something different. A search crawler indexes your pages so you appear in search results and in AI answers built on that index, so blocking it can remove you from ChatGPT search, Google AI Overviews, and Perplexity even though your site is up. An agent crawler fetches your page live when a user asks an AI assistant about you, so allowing it lets assistants pull your content in the moment, which is increasingly how people reach sites. A training crawler collects content to train a model, and blocking it, to keep models from learning from your content or to charge for that access, is a legitimate business choice that has no direct effect on whether you appear in search. The reason the distinction matters is that the September 15 default keeps search allowed while blocking agent and training, and that treating them as one decision is the mistake the change is designed to fix. Knowing they are separate lets you block what you intend, usually training, and keep what you need, usually search.

Yes, and free accounts are actually the most affected, which is the opposite of what many owners assume. For the millions of sites on Cloudflare’s free tier, and for every new sign-up after September 15, 2026, the new blocking defaults are on automatically, so AI training and agent crawlers begin getting blocked on ad-supported pages without the owner changing anything, and allowing AI training access becomes a manual opt-in. Existing paying customers are treated differently: they retain the ability to override the defaults through their Cloudflare dashboard and readmit specific crawlers if they choose, but they still have to go and make that change. The practical implication is that if you run a small business site on Cloudflare’s free plan and you have never thought about AI crawler settings, this change will apply to you by default, and the only way to know whether the resulting configuration matches what you actually want is to log in and look. Free-tier does not mean unaffected here; it means affected by default.

That is a genuine business decision that only you can make, and both choices are defensible. Blocking AI training crawlers means models do not learn from your content, and it opens the door to charging for that access, which is why many publishers are choosing to block deliberately; it also has no direct effect on whether you appear in search results or AI answers, because search and training are separate functions. Allowing training means your content can be used to train models, which some owners are fine with and others are not, on principle or on business grounds. There is no universally correct answer, and anyone who tells you there is one is substituting their preference for your judgment. What matters is that the decision is deliberate and informed: understand that training is separate from search, that blocking training does not remove you from AI answers by itself, and that the one real technical risk is a training block cascading to a multi-purpose crawler that also does search. Decide training on the merits for your business, keep search allowed if you want AI visibility, and verify the result rather than trusting the default.

Keep the search crawlers allowed, and confirm it rather than assuming it. The crawlers that put you in AI answers are the search-indexing ones, which feed ChatGPT search, Google’s AI Overviews, and Perplexity, so the single most important setting is to ensure those remain allowed after the September 15 change. Concretely: confirm whether your site is behind Cloudflare and on which plan, open the AI crawler or bot controls in your dashboard, and look at search, agent, and training as three separate switches. Keep search allowed, decide agent based on whether you want AI assistants fetching your pages live for users, and choose training on the merits for your content. If you do block training, verify afterward that you did not also block a multi-purpose search crawler like Googlebot or Bingbot, because that cascade is the most common way sites lose visibility unintentionally. After the deadline, check your server logs or robots handling to confirm the search crawlers you care about are still getting through. If tracking edge changes like this is not something you want to own, a managed edge can keep the decision deliberate and aligned with your goals, but the underlying controls remain Cloudflare’s either way.