Cloudflare is about to block AI crawlers by default: check this before September 15
On July 1, 2026, Cloudflare announced it’s retiring its old single “block AI bots” switch and replacing it with a three-category system. The date that matters is September 15, 2026: from then on, new domains (and any free-tier account that never set an explicit preference) get two of those three categories blocked by default on ad-supported pages.
The three categories
- Search: bots that index content to answer questions later, usually with referral traffic in return (
OAI-SearchBot,Claude-SearchBot) - Agent: real-time automated activity on a person’s behalf, like ChatGPT fetching your pricing page because a user asked about it
- Training: crawlers pulling your content into a model’s training data (
GPTBot,ClaudeBot,CCBot)
Starting September 15, Training and Agent are blocked by default on ad-supported pages; Search stays allowed.
The detail almost everyone misses
Crawlers that serve more than one purpose at once (and this is the part that matters) get treated under whichever rule is most restrictive. Googlebot, Applebot, and Bingbot all crawl for both search and AI training in a single bot. If your setting blocks Training, those bots can get blocked on those pages too, unless explicitly excluded.
For any brand that wants AI visibility without losing Google Search rankings, this isn’t a minor setting: it’s worth reviewing before the deadline, not after.
Why this matters especially if you’re pursuing GEO/AEO
If you’re investing in getting cited by ChatGPT or Perplexity, and your site sits on Cloudflare with default settings you’ve never reviewed, you risk blocking the exact traffic you’re trying to attract, without ever deciding to.
How to confirm it with your own data, not just the dashboard
The Cloudflare dashboard tells you which policy is configured. It doesn’t tell you whether it’s actually having the effect you think. For that, you need the logs: filter by each bot’s exact user-agent (GPTBot, ClaudeBot, PerplexityBot…) and compare request volume before and after the change. That’s exactly layer 3 — crawlers — of any AI visibility audit worth its name: the setting is the intent, the log is the proof.
What to check today
1. Cloudflare dashboard → Security → Settings
2. Configure AI bot policies
3. Review the current status of Search / Agent / Training
4. Make a deliberate choice: don't let September 15 decide for you
5. Set a reminder for September 16 and confirm your settings
survived the default switchover
This article summarizes Cloudflare’s official July 1, 2026 announcement and subsequent technical coverage from several specialized sources. If you want me to review your domain’s configuration (Cloudflare or not) as part of an AI visibility audit, I can help.