Cloudflare's new AI bot controls are live now, with new defaults landing on 15 September 2026. For insurers, law firms and claims operators, this is a compliance and operational risk sitting quietly in Security settings. Here is what to check, and why the deadline matters.
On 1 July 2026, Cloudflare announced significant changes to how it manages AI bot traffic for the roughly 20% of websites that sit behind its network. New granular controls are live now. New defaults land on 15 September 2026. For most sectors this is a marketing conversation. For insurers, law firms, claims companies and their supply chains, we think it is closer to an operational and compliance conversation, and worth taking seriously this side of August.
This piece walks through what has changed, what it means specifically for regulated professional services operators, and the practical checklist we would run through with any firm whose website sits behind Cloudflare.
What Cloudflare has actually changed
Cloudflare has replaced its single "Block AI Bots" switch with a three-way classification for all automated traffic. Every AI bot or agent that touches a website is now sorted by behaviour into one of three categories.
Search bots crawl and index content so an engine can later answer questions about it. Googlebot's search crawl, Bingbot's search crawl, Perplexity's search crawl. This is the traditional deal, and it still funnels human referral traffic back to the source.
Agent bots visit a website in real time on behalf of a human user. A ChatGPT-User agent fetching a page because someone asked a question. A Claude browser session pulling up a firm's fees page. A Gemini agent scheduling a callback. The defining feature is that a person is on the other end, waiting.
Training bots absorb content to train or fine-tune an AI model. Once absorbed, that content becomes part of the model's underlying knowledge. There is no traffic coming back.
Every Cloudflare customer, on every tier including Free, can now control these three categories independently. That is the operational change.
The default change on 15 September
Setting the controls up is the easier half of the story. The harder half is what Cloudflare is doing to the defaults.
From 15 September 2026, on any new domain onboarding to Cloudflare, the defaults change. On pages that display ads, Training bots and Agent bots will be blocked by default. Search bots remain allowed. The reasoning is that ad-monetised pages are meant for human attention, so the bots that either divert that attention or absorb the content without sending anyone back get blocked at the front door.
The larger practical change applies to existing customers too. Multi-purpose crawlers (bots that do more than one job, for example Googlebot which crawls both for Search and for Google's model training) will be evaluated against all their behaviours from 15 September. Not just one of them. The most restrictive applicable rule wins.
In practice: any firm that ticked the legacy "Block AI Bots" preset in Cloudflare, or the newer "Block Training" toggle, and has not opted out of the new default logic, will start blocking Googlebot, Applebot and BingBot on 15 September. Because those bots crawl for training as well as for search, they will be caught by the training block.
Blocking Googlebot removes a site from Google search results. That is not a small effect for a professional services firm relying on organic discoverability.
Why this is a compliance and operational conversation, not just a marketing one
Three reasons the sector we work in should treat this as more than a marketing setting.
Discoverability under Consumer Duty and the SRA Standards. For consumer-facing legal, claims and insurance firms, being findable when a consumer runs a Google search is not a nice-to-have. It is part of demonstrating the firm is accessible to the public it serves. A silent default change that removes a firm from Google results creates a gap between the firm's stated market accessibility and its actual accessibility. Compliance functions should know whether that gap is opening.
The agentic layer is now a customer service channel. Consumers are increasingly using ChatGPT, Claude and Perplexity as first-stop research tools before choosing a legal or claims provider. When a consumer asks their AI assistant "who handles credit hire in Manchester", or "which law firms take on housing disrepair", the Agent bots that go and check firm websites in real time are effectively front-of-house. Blocking them by default is a decision worth making consciously, not a decision to fall into.
Third-party and supply chain exposure. Most legal, claims and insurance operators rely on portals, referrers and partners whose sites they do not control. If those partners are behind Cloudflare and default-block their agent traffic in September, the operator's own referral flow can degrade in ways that are hard to attribute. This is a vendor management question. Compliance and operations teams should be asking supply chain partners whether they know about the 15 September change.
The five-minute self-check for any firm
Two straightforward questions to answer this week.
Are you behind Cloudflare? The IT function or web agency will know. Failing that, a hosting lookup tool will show whether Cloudflare appears as CDN or nameserver.
What is in the current Cloudflare AI bot configuration? For any firm that is on Cloudflare, someone needs to log into the dashboard, navigate to Security then Settings, and check three things:
- Whether the legacy "Block AI Bots" preset is currently on. If it is, and no one opts out of the new default logic before 15 September, Googlebot will be blocked from that date. This is the single most consequential item.
- Whether the new three-way Search, Agent, Training controls have been configured. If not, the defaults apply.
- Whether the firm has opted out of the new default behaviour. Cloudflare provides an opt-out for customers who want to keep the current behaviour on multi-purpose crawlers. Firms that have made deliberate choices about their current bot posture should record them explicitly by opting out.
The default we would recommend for most professional services firms is: Search allowed, Agent allowed, Training decision made consciously with legal and marketing input. That combination keeps the firm discoverable to Google, addressable to consumer AI agents, and gives the firm control over model training use of its content.
What we would flag as a genuine risk
Two scenarios worth mapping to any firm's risk register.
The first is the accidental disappearance from Google search results on 16 September because a Cloudflare setting was ticked years ago and forgotten. This is a discoverability incident with revenue implications and a Consumer Duty accessibility question attached. It is preventable with a five-minute check now.
The second is the slow attrition of agentic-layer visibility over the second half of 2026. As more consumers use AI assistants for professional services research, firms that have blocked Agent bots (or whose supply chain partners have) will progressively lose surface area in that channel. This is less acute than the Google case but harder to detect once it is happening.
Both are avoidable. Both benefit from a single named individual in the firm being accountable for the Cloudflare AI bot configuration, with the compliance function reviewing that configuration as part of its usual technology risk cadence.
Where CaseFlow Automation fits
We build compliance-first workflow tools for regulated professional services operators. Bot configuration is not our product, but the underlying question this raises (is your firm making deliberate, documented decisions about who accesses your online surface, or defaulting into other people's choices) is squarely on our beat. If a firm's Consumer Duty file, technology risk register or vendor management pack do not currently address AI bot access, this quarter is a good quarter to fix that.
We are happy to talk through the operational implications of the 15 September change with any legal, claims or insurance firm that would find a specialist steer useful, either as part of an existing CaseFlow engagement or as a standalone conversation.
Frequently Asked Questions
- Should our firm block AI training bots?
- That is a documented decision to make, not a default to fall into. The decision should sit with legal, marketing and compliance jointly, informed by whether the firm considers its published content a defensible asset. Training bots absorbing content into a model create a specific data-use question that is worth addressing explicitly rather than by accident.
- Will this affect our Google search rankings?
- It can. If the firm has ever ticked "Block AI Training" or the legacy "Block AI Bots" preset in Cloudflare, and does not opt out of the new default logic before 15 September, Googlebot will be blocked from that date. That removes the firm from Google search results.
- Who in our firm should own this?
- IT typically holds Cloudflare access. But the decision on which bot classes to allow and block is a business decision, not a technical one. In our experience it works best when compliance owns the policy question, marketing owns the visibility question, and IT owns the configuration once the policy is agreed.
- What is a sensible default for a regulated professional services firm?
- Search allowed, Agent allowed, Training decision made consciously with legal and marketing input, and the opt-out from Cloudflare's new default logic ticked so the firm's current posture is recorded rather than inherited.