Technology

Google now answers 75% of searches without a click, and AI crawlers take 6,000 pages for every visitor they send — Cloudflare’s CEO says the web’s business model is collapsing

For fifteen years the deal was simple: let Google copy your page and it sends you readers. Matthew Prince, whose company sees a fifth of all web traffic, says that deal is dead — Google’s crawl-to-visit ratio has tripled, OpenAI’s is 250 to 1, Anthropic’s 6,000 to 1. What that means for anyone who publishes online, including in Bangla.

Google now answers 75% of searches without a click, and AI crawlers take 6,000 pages for every visitor they send — Cloudflare’s CEO says the web’s business model is collapsing

Matthew Prince runs Cloudflare, the company whose servers sit in front of roughly a fifth of the world’s websites, so when he describes what is happening to web traffic he is reading it off his own dashboards. Speaking at the Council on Foreign Relations this month, he described a system that is failing: the exchange that funded the open web for fifteen years no longer pays.

The old bargain

The web’s economy has run on search. A site let Google copy its pages; Google, in return, sent readers to it; the readers saw advertisements or bought subscriptions; the money paid for the next article. “It was a fair trade,” Prince said. Every part of it depended on the click — the moment the reader left Google and arrived at the source.

Where the clicks went

Google still copies everything. What it no longer does, at the same rate, is send people onward. Prince put a number on it: 75 per cent of queries are now answered on Google’s own results page — a snippet, a knowledge panel, an AI overview — and the searcher never leaves. Ten years ago Google crawled roughly two pages for every visitor it referred; today, by Cloudflare’s measurement, it is six pages per visitor. Sites that lived on ads, memberships or simple readership have lost the traffic that fed them.

The AI ratios

The AI companies are worse by orders of magnitude, because they take content to train and answer with and send almost no one back. Cloudflare’s figures: OpenAI crawls about 250 pages for every visitor it refers; Anthropic about 6,000. The models are built from the web’s writing and then replace the reason to visit the web. Prince’s conclusion is blunt: if publishers cannot earn, they stop publishing, and the system that the AI companies themselves depend on breaks.

What it means here

Bangla-language sites feel this doubly. Their advertising rates are already among the world’s lowest, so a 30 per cent loss of search traffic is not a squeeze but a closure; and because Bangla training data is scarce, AI crawlers value Bangla content highly while sending Bangla readers nowhere. The responses available are the same as elsewhere — block the crawlers with robots.txt and Cloudflare’s bot rules, charge for access, build direct readership through apps, email and social channels rather than depending on search. None of them is comfortable. But the trade Prince describes is not coming back, and a publisher planning around it now is better placed than one still waiting for the clicks.

Source: India Today, via Prothom Alo.

Source: Prothom Alo

Written by

Tech BD

Editorial team of Tech BD.