The web is walling off AI. That is your opening.
From 15 September 2026, Cloudflare — the network in front of a huge share of the web — blocks AI training and agent bots by default on ad-supported pages. Publishers are repricing access to their content. For brands, the calculus runs the other way.
Cloudflare announced new defaults for AI bot traffic taking effect on 15 September 2026: search crawlers stay allowed, but training bots and agent bots are blocked by default on pages that display ads, and multi-purpose crawlers are blocked outright. New Cloudflare sites inherit the defaults; existing customers confirm their preferences. It is the logical next step in a repricing that began with pay-per-crawl experiments: content that once fed AI models for free is becoming metered, negotiated, or closed.
Why publishers are right to do it
For a publisher, content is the product; an AI that ingests it and answers readers directly consumes the product without paying for it. Blocking, licensing, and metering are rational responses, and the numbers behind them are stark — we covered the collapse of the click in the zero-click data. Expect more walls, not fewer.
Why brands should do the opposite
A brand’s website is not the product; it exists so the product gets found, understood, and chosen. Every publisher wall makes the open, machine-readable sources that remain more valuable to the systems assembling answers — a shrinking supply of trainable, citable material raises the weight of what stays open. We have seen this asymmetry before: Amazon blocks OpenAI’s crawlers, which means a small brand’s own store can appear in ChatGPT’s shopping answers where the largest marketplace is absent. The walls of the big players are the openings of the small ones.
Open by policy, not by accident
Our stance, applied on this site and client builds: explicitly allow the AI crawlers in robots.txt — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and peers — ship clean structured data, and publish content worth lifting. Openness is a strategic choice with real trade-offs; for most brands whose business depends on being known, the trade resolves clearly. The exception is genuine: if your content is your product, think like a publisher, not like a brand.
The read for the next year
The agentic web — software browsing on people’s behalf — keeps growing while the training web shrinks behind walls. The brands that treat machine readers as a first-class audience now are buying share of voice in every answer assembled from what remains open. That is the quiet arbitrage of 2026, and it costs almost nothing to take. To audit where you stand, get in touch.
Sources
Cloudflare blog, 2026 (AI bot traffic defaults from 15 September 2026) · ROI Revolution, August 2026 recap · Imperva/Thales Bad Bot Report 2026 · Ekamoira on Amazon crawler policy, 2026.
While the web walls up, be the open source.
We make brands machine-readable by policy — and present in the answers built from what remains.
Get in touch →