If your business publishes content online, something important just changed. Cloudflare, the infrastructure company sitting in front of roughly 20% of all web traffic, announced on July 1 that it is giving every website owner the ability to control how different types of AI bots access their content. And the defaults are about to shift in a meaningful way.

The announcement matters whether you’re running a blog, a content marketing site, a knowledge base, or any kind of web property that AI companies have been quietly harvesting for years.

Three Kinds of AI Crawlers, Three Different Answers

Up until now, most website owners had a blunt choice: allow all AI crawlers, or block them wholesale. Cloudflare has now split them into three distinct categories, each with a different relationship to your content.

Search bots index your site to answer questions about it later. Think of ChatGPT browsing the web to find recent information, or Perplexity pulling in live context. The key word is reciprocal. Search bots should be sending you referral traffic or providing some form of value back. Cloudflare’s default leaves these unblocked.

Agent bots are operating in real time on someone’s behalf. A person asks their AI assistant to book a flight, or check your pricing, or pull a competitor’s specs, and an automated browser does it. These bots visit your site to complete a job, not to build a database. They’re blocked by default on ad-monetized pages for new domains starting September 15.

Training bots are the contentious ones. These crawlers take your content and permanently incorporate it into an AI model’s underlying architecture. Your writing, your data, your research: all of it absorbed into the model with no attribution, no payment, and no referral link. Also blocked by default on ad-monetized pages starting September 15.

The September 15 Deadline

Starting September 15, 2026, any new domain that onboards to Cloudflare will have Training and Agent bots blocked by default on pages that serve advertising. Existing customers can opt into these controls now through their dashboard.

This is significant. Cloudflare is essentially saying: we are changing what “normal” looks like. The default is no longer unrestricted AI access to your content.

The change only applies automatically to ad-monetized pages, because that’s where the economic harm is clearest. An AI bot reading your ad-supported article generates zero ad revenue and may reduce future traffic by summarizing your content elsewhere. But the controls are available across all page types, and the feature is available to all Cloudflare customers including the free tier.

Why This Matters for Content Businesses

The economics of web content have been quietly deteriorating for the past two years. Businesses that invested in building SEO-driven content found that AI search tools were surfacing answers drawn from their content without sending readers back to the original source. Traffic from traditional search engines has dropped for many publishers.

This Cloudflare change doesn’t fully reverse that dynamic. But it gives businesses a legitimate way to draw a line between the two kinds of AI access they’re actually comfortable with.

Allowing search bots means your content gets indexed and surfaced in AI-powered search results, which can still drive discovery and referral visits. Blocking training bots means your proprietary content, your research, your frameworks are no longer free inputs for model training without your consent.

For businesses that produce educational content, industry analysis, or proprietary methodologies, that distinction is material.

What This Means for Business

If you run any kind of web property: Check your Cloudflare settings before September 15. The new controls are available now in the dashboard under the bot management section. Decide which categories you want to allow, and on which pages. It doesn’t have to be all-or-nothing.

If you invest in content marketing: This is a signal to continue that investment. Content that sits behind thoughtful distribution will be harder to scrape, and harder to replace with AI-generated alternatives that were trained on someone else’s work.

If you’re thinking about AI in your business: The web is starting to price itself differently based on how AI accesses it. That’s a structural change in the internet’s economics, and it will affect how AI tools are priced and what data they can access over the next few years.

If you’re building AI applications: Your tools that access web content on behalf of users may now face more restrictions by default. Understanding which category your crawler falls into, and communicating that accurately via user-agent strings, will matter more going forward.

The Bigger Picture

Cloudflare framed this announcement as “Content Independence Day,” a pointed reference to the idea that content creators should have sovereignty over how their work is used. Whether or not that framing sticks, the underlying mechanism is practical and immediate.

The question businesses should be asking isn’t whether AI will continue scraping the web. It will. The question is whether the content you produce will be on the right side of that distinction when the defaults shift in September.

At Enterprise DNA, we produce a significant amount of content across data analytics, AI strategy, and business operations. We think seriously about how that content is accessed and what value it creates for our readers, for our clients, and for our business. Controls like these are a step toward a more rational relationship between content creators and the AI systems that increasingly depend on them.

---

Cloudflare’s AI traffic controls are available to all customers at no additional cost. New default settings for Training and Agent crawlers take effect September 15, 2026, for newly onboarded domains. Existing customers can configure settings now.

If this is the kind of problem agents can help with, the free Working With Claude field guide is the practical next step. Thirty-two pages, no fluff. Get the free guide.