Update, October 2026 — Six weeks after the remarks below, Cloudflare acted on them: from 1 July 2025 new domains block AI crawlers by default, and a pay-per-crawl beta revived an HTTP status code that had gone unused for thirty years. What changed, and what it does not fix, follows the original article.
Matthew Prince runs Cloudflare, the company whose servers sit in front of roughly a fifth of the world’s websites, so when he describes what is happening to web traffic he is reading it off his own dashboards. Speaking at the Council on Foreign Relations this month, he described a system that is failing: the exchange that funded the open web for fifteen years no longer pays.
The old bargain
The web’s economy has run on search. A site let Google copy its pages; Google, in return, sent readers to it; the readers saw advertisements or bought subscriptions; the money paid for the next article. “It was a fair trade,” Prince said. Every part of it depended on the click — the moment the reader left Google and arrived at the source.
Where the clicks went
Google still copies everything. What it no longer does, at the same rate, is send people onward. Prince put a number on it: 75 per cent of queries are now answered on Google’s own results page — a snippet, a knowledge panel, an AI overview — and the searcher never leaves. Ten years ago Google crawled roughly two pages for every visitor it referred; today, by Cloudflare’s measurement, it is six pages per visitor. Sites that lived on ads, memberships or simple readership have lost the traffic that fed them.
The AI ratios
The AI companies are worse by orders of magnitude, because they take content to train and answer with and send almost no one back. Cloudflare’s figures: OpenAI crawls about 250 pages for every visitor it refers; Anthropic about 6,000. The models are built from the web’s writing and then replace the reason to visit the web. Prince’s conclusion is blunt: if publishers cannot earn, they stop publishing, and the system that the AI companies themselves depend on breaks.
What it means here
Bangla-language sites feel this doubly. Their advertising rates are already among the world’s lowest, so a 30 per cent loss of search traffic is not a squeeze but a closure; and because Bangla training data is scarce, AI crawlers value Bangla content highly while sending Bangla readers nowhere. The responses available are the same as elsewhere — block the crawlers with robots.txt and Cloudflare’s bot rules, charge for access, build direct readership through apps, email and social channels rather than depending on search. None of them is comfortable. But the trade Prince describes is not coming back, and a publisher planning around it now is better placed than one still waiting for the clicks.
Source: India Today, via Prothom Alo.
What Cloudflare did on 1 July 2025
Prince described the collapse of a bargain. His company then changed one of the terms, and the change is smaller and more consequential than it sounds.
The default flipped. Blocking AI crawlers was already possible for any Cloudflare customer; the setting had existed for over a year. From 1 July 2025, new domains signing up get it switched on, and crawlers must be explicitly allowed rather than explicitly refused.
That is the whole of the policy change, and it matters because of what defaults do. Almost nobody changes them. A permission that must be granted is granted rarely; a permission that must be revoked is revoked rarely. For a year the tool existed and the web stayed open to crawlers because leaving a setting alone is what people do. Reversing which way the setting points, across a company sitting in front of roughly a fifth of the web, does more than a year of advocacy did.
Pay per crawl. Alongside it Cloudflare opened a private beta letting publishers set a price for crawler access, and built it on HTTP 402 Payment Required — a status code reserved in the original HTTP specification for a payments system that was never built, and left unused for three decades. A crawler that requests a page without paying gets a 402 and a price; one that pays gets the page.
Publishers lined up behind it: the Associated Press, The Atlantic, TIME, Fortune, Sky News, BuzzFeed, Reddit, Pinterest and ADWEEK among the named supporters.
What it does not fix
Three things, and none of them is small.
Reach. Cloudflare can only enforce this for sites behind Cloudflare. That is a large share of the web and not all of it, and the share that matters most to a crawler — the long tail of small sites on cheap shared hosting — is the part least likely to be behind any CDN at all.
Identification. Blocking depends on knowing which requests are crawlers. A crawler that announces itself honestly in its user-agent string can be refused trivially. One that does not has to be caught by behaviour and network fingerprint, which is an arms race with no end state, conducted against companies with more engineers than the people trying to stop them.
Price. A marketplace needs both sides to show up. A publisher can name a figure; nothing obliges an AI company to pay it rather than train on the same facts gathered from somewhere cheaper, and the facts in a news report are not copyrightable even where the expression is. What a crawl is worth remains unsettled, and until it settles, 402 is a mechanism without a market.
Where it leaves a publisher now
The position has genuinely changed since Prince spoke, from one open question to two decidable ones.
The first is whether to block, and the honest accounting is less comfortable than the campaign suggests. Blocking crawlers forfeits whatever discovery comes from being cited in an AI answer, and recovers no revenue, because there was none to recover. It is a bet that content withheld now has a price later. That bet may be right. It is still a bet, and a small publisher placing it pays the cost immediately and collects, if ever, much later.
The second is what to build instead, and that one has no ambiguity in it. Every response that survives this — direct readership, email, apps, membership, a reason to arrive at a site on purpose — is something a publisher would want regardless of what crawlers do. Search referral and crawler compensation are both outside a publisher's control. A reader who comes back without being sent is not.
