The Era of Free Scraping Is Over: How Enterprises Must Now Negotiate AI Web Access:

Cloudflare is About to Block AI Agents on 20% of the Web: What You Need to Do:

AI Agent Crawlers Now Need Permission — Here's How Enterprises Get It.

Cloudflare's new September 15 defaults turn agent access to the open web into a negotiated resource, not a given:

Sept 15, 2026: New blocking defaults take effect

3: Crawler categories: Search, Agent, Training

50%+: Of AI crawler traffic re-fetches unchanged pages

1: One Switch Becomes Three Categories:

The open web is getting a permission layer, and it's arriving faster than most enterprise AI roadmaps accounted for.

Cloudflare announced on July 1 that it is replacing its single block-AI-bots toggle with three distinct categories: Search bots that index pages to answer questions later, Agent bots that act in real time on a person's behalf, and Training bots that pull content into a model's weights. All three categories are live now for every Cloudflare customer, including the free tier.

The classification is behavioral, not something an operator opts into. A research agent that browses a page in real time gets treated as Agent-class whether or not its builders think of it as a crawler at all.

2: What Changes on September 15:

From September 15, Training and Agent traffic will be blocked by default on any page that carries advertising, while Search stays allowed.

Cloudflare's reasoning is straightforward: an ad on a page signals it was built for a human visitor. A search crawler that sends a reader back counts as a referral. A bot that reads the page and hands the answer to someone else, without a visit ever happening, does not.

The new defaults apply automatically to newly onboarding domains, new sites from existing customers, and every existing free-tier account. Paid customers who want to keep current access can opt out through their security settings before the deadline, but free-tier publishers are moved over without any action required on their part.

Cloudflare CEO Matthew Prince said the goal is to "encourage mixed-use crawlers to separate search from agent use and training" — a signal that the pressure on bot operators is intentional, not incidental.

3: The Google Complication and the Agent Builder's Homework:

Googlebot crawls for both search and training through a single bot, so blocking Training under the strictest setting also blocks Googlebot — and a site's search visibility with it. For anyone running agents, the immediate task is figuring out which of their own deployments will read as Agent-class under Cloudflare's behavioral rules, since intent doesn't matter, behavior does.

Expect degraded coverage rather than a clean failure: the block lands on ad-supported pages specifically, which happens to be where the news, pricing, and product data most agents rely on tends to live. Negotiated access is the fix, not a rewritten user-agent string pretending to be something else.

A parallel monetization layer is emerging alongside the blocking rules. Pay Per Crawl is becoming Pay Per Use, with services like Ceramic.ai and You.com beginning to pay publishers when their content shows up in AI answers or is reached by an agent. Cloudflare's own data shows more than half of AI crawler traffic re-fetches pages that haven't changed, which is waste both sides have an incentive to price out.

The taxonomy still has a soft spot: Search, Agent, and Training are behaviors that AI companies self-declare for their own bots, and nothing in the announcement explains what stops a company from labeling a training crawler as something less restricted.

4: Why This Is an Infrastructure Problem, Not a Compliance Checkbox:

Every enterprise running a customer-facing agent, a research tool, or a monitoring pipeline just inherited a new dependency it didn't design for.

This is exactly the kind of shifting infrastructure layer that makes in-house agent builds fragile. An agent that quietly loses access to the pages it depends on doesn't fail loudly — it just starts answering from whatever it can still reach, and nobody notices until the output is wrong.

Built for the Access Rules That Keep Changing:

Otherworlds AI's Agent+ Business AI Platform is built and maintained for exactly this kind of moving target — access rules, crawler permissions, and data pipelines that change without warning. At $297/month, powered by Google Opal automated workflows, Agent+ gives enterprises a managed agent layer instead of a homegrown one that breaks the next time a network provider changes its defaults.

For teams that need something more tailored, our custom enterprise AI builds are designed the same way: resilient to a web that increasingly requires permission to read.

Learn more at otherworldsai.com.