{"id":236261,"date":"2025-07-04T02:15:15","date_gmt":"2025-07-04T02:15:15","guid":{"rendered":"https:\/\/www.europesays.com\/uk\/236261\/"},"modified":"2025-07-04T02:15:15","modified_gmt":"2025-07-04T02:15:15","slug":"free-lunch-is-over-for-the-ai-that-broke-the-web","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/uk\/236261\/","title":{"rendered":"Free Lunch Is Over for the AI That Broke the Web"},"content":{"rendered":"

The foundational deal of the modern web, a handshake agreement that powered two decades of search and content, is officially dead. Cloudflare just put a price on scraping the internet, and it\u2019s coming for artificial intelligence\u2019s free lunch.<\/p>\n

Almost 30 years ago, two Stanford grad students, Larry Page and Sergey Brin, built Google on a simple bargain: content creators would let them copy the entire web in exchange for traffic. For years, that traffic powered ad revenue, subscriptions, and the growth of online media. Google mostly upheld its end of the deal. But that era is collapsing under the weight of AI.<\/p>\n

On July 1, Cloudflare, one of the internet\u2019s core infrastructure companies, declared \u201cContent Independence Day.\u201d In a landmark policy shift, the company announced<\/a> it will now block AI crawlers from scraping sites hosted on its platform unless those bots pay content creators for the data they consume.<\/p>\n

\u201cCloudflare, along with a majority of the world\u2019s leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content,\u201d CEO Matthew Prince announced in a blog post<\/a>. \u201cThat content is the fuel that powers AI engines, and so it\u2019s only fair that content creators are compensated directly for it.\u201d<\/p>\n

This is a sharp, aggressive turn from the web\u2019s traditionally open access ethos. Cloudflare argues it\u2019s long overdue. AI tools like ChatGPT and Google\u2019s own AI Overviews are now answering user questions directly, effectively strip-mining websites for information while sending almost no traffic back to the original source.<\/p>\n

\u201cInstead of being a fair trade, the web is being stripmined by AI crawlers with content creators seeing almost no traffic and therefore almost no value,\u201d Prince said.<\/p>\n

\n

Publishers, we see you! \ud83d\ude4c Cloudflare just launched pay per crawl to put control over your content back where it belongs. <\/p>\n

Now, crawling is more transparent and controlled, by default, creating a better web ecosystem for creators like you. This is about real content\u2026 pic.twitter.com\/yatB5LSBIm<\/a><\/p>\n

\u2014 Cloudflare (@Cloudflare) July 2, 2025<\/a><\/p>\n<\/blockquote>\n

The Web as Swiss Cheese <\/p>\n

The numbers are stark. Cloudflare claims it\u2019s already 10 times harder to get traffic from Google than it was a decade ago due to features like the answer box. But the new AI models are far worse. According to Cloudflare\u2019s internal metrics, OpenAI drives 750 times less traffic than traditional Google search, while Anthropic drives a staggering 30,000 times less. The reason is simple: people are asking ChatGPT instead of Googling. The content still gets used, but the creators have been completely cut out of the value chain.<\/p>\n

Using its position as a gatekeeper for roughly 20% of all websites globally (around one-fifth of all web traffic passes through Cloudflare\u2019s network), Cloudflare is now forcing the issue by charging a toll.<\/p>\n

But the plan goes further than just blocking bots. Cloudflare aims to build a new content marketplace where AI companies and creators can trade directly. Compensation would be based not on clicks, but on how valuable the content is for training AI models. To explain this, the company uses a quirky metaphor: imagine an AI\u2019s knowledge is a block of Swiss cheese. The holes represent knowledge gaps. The more your content fills one of those holes, the more it\u2019s worth to an AI company.<\/p>\n

It\u2019s an ambitious proposal that challenges the entire web economy, which still judges value by how viral something is. Cloudflare is betting that filling gaps in machine knowledge is a more stable long-term market than chasing fickle human attention. It also hints at something more radical: the end of the free and open web as we knew it.<\/p>\n

This move marks the dawn of the pay-to-train era. OpenAI has already signed high-profile licensing deals with publishers like Reddit and the Financial Times. Other AI giants are quietly inking data partnerships or scraping whatever they can until they get blocked. But Cloudflare\u2019s decision is the first time a major infrastructure provider has flipped the default setting for a huge portion of the internet.<\/p>\n

Our Take <\/p>\n

The real story here is not just technical; it is economical. We are watching the rise of a new class of digital middlemen, companies that will broker access between the creators of web content and the AI models that feed on it. In a post-click internet, training data is the new currency, and Cloudflare just positioned itself as a major bank.<\/p>\n

The company says its goal is to usher in a new golden age for creators. \u201cWe believe that if we can begin to score and value content not on how much traffic it generates, but on how much it furthers knowledge,\u201d Prince said, \u201cwe not only will help AI engines get better faster, but also potentially facilitate a new golden age of high-value content creation.\u201d<\/p>\n

That sounds nice. But it also raises messy questions. Who decides what counts as high value? Who gets paid, and how much? If content is optimized for AI rather than people, what happens to the soul of the web? The darker possibility is a content Cold War, where publishers wall off everything and AI companies hoard exclusive data deals, making the web more fragmented and less open than ever before.<\/p>\n

Whether or not Cloudflare\u2019s \u201cSwiss cheese\u201d model takes off, this much is true: AI broke the old search-based web economy. On July 1, Cloudflare drew a line in the sand. For the first time in the age of generative AI, the pipes of the internet are fighting back.<\/p>\n