{"id":134598,"date":"2026-08-10T05:48:12","date_gmt":"2026-08-10T05:48:12","guid":{"rendered":"https:\/\/www.europesays.com\/ai\/134598\/"},"modified":"2026-08-10T05:48:12","modified_gmt":"2026-08-10T05:48:12","slug":"googles-ai-is-killing-the-web-now-the-web-is-fighting-back","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/ai\/134598\/","title":{"rendered":"Google’s AI is killing the web. Now the web is fighting back"},"content":{"rendered":"

Google became the king of internet search and created a free cornucopia of knowledge by striking a clever bargain with anyone who made a website: Let the company\u2019s crawler index your pages and Google<\/a> would send human readers back. That trade looks very different today thanks to Google\u2019s strategy of using its search infrastructure to strengthen its artificial-intelligence<\/a> products.<\/p>\n

\"Google\u2019sGoogle\u2019s search and AI systems face growing scrutiny over how web content is crawled, used and shared with publishers. (REUTERS)<\/p>\n

The technique, which has largely flown under the radar, gives the company an unfair advantage over other AI firms while also accelerating the rise of automated traffic on the internet at the expense of humans. Google is now meeting some resistance from a plucky British regulator and a crusading industry player, which may finally bring some much-needed change.<\/p>\n

Googlebot, the firm\u2019s automated crawler, has for many years scoured the Internet and downloaded web pages to feed its massive index so that the freshest and most relevant pages could appear on Google\u2019s search results. More recently, though, the line between content collected for search and content used to generate AI answers has become blurred, as the bot is used to help feed both search and Google\u2019s flagship AI model, Gemini<\/a>.<\/p>\n

Also read: Inside Google’s AI reset: How Pichai is redrawing the map & shifting its centre of gravity back to California<\/a><\/p>\n

Google\u2019s web crawler sees 3.2 times more of the web than OpenAI<\/a>\u2019s version and 4.8 times more than Microsoft Corp.\u2019s, according to Matthew Prince, chief executive officer of web security company Cloudflare<\/a>. Prince\u2019s firm is a traffic gatekeeper for about a fifth of the web, handling traffic and blocking attacks from bad actors, and he\u2019s become a vocal critic of Google for bundling its search and scraping systems. He says the strategy makes it difficult for publishers to block AI training without losing the search traffic they rely on.<\/p>\n

That advantage also has broader consequences. If AI increasingly reads, summarizes and rewrites the web without sending readers back to the pages created by humans, the economic incentive to produce original information starts to fade. <\/p>\n

AI agents generated more than 57% of web traffic in 2026 versus about 42% for humans, according to Cloudflare, marking the first time in history that machine activity surpassed humans on the Internet. Prince sees that divergence getting more extreme in the next few years, a trend that echoes the so-called dead internet theory, the idea that machines increasingly create and consume online content. \u201cHumans will be a rounding error on the internet versus what their agents and their bots are doing,\u201d he tells me.<\/p>\n

Also read: Can AI become a public health equaliser for India?<\/a><\/p>\n

Agents are still largely the preserve of software developers and companies; but Mark Zuckerberg <\/a>said last week that Meta<\/a> Platforms Inc. would soon launch a consumer version for Facebook Messenger, WhatsApp and Instagram<\/a> \u201cthat just works out of the box and is easy enough for billions of people\u201d to use.<\/p>\n

There\u2019s nothing wrong with bots doing things that are useful, but the internet will become vapid, less original and ultimately less useful if human contribution is no longer rewarded and peters out. That\u2019s what Google\u2019s new search system threatens, as people create content only for machines to copy and condense into AI answers and generating economic rewards that never reach the original authors.<\/p>\n

Google could have addressed that by letting websites opt out of having their data used for AI while still appearing in search results. The company considered giving them the choice in April 2024, just before it launched AI Overviews<\/a>, then decided not to because it was \u201cevolving into a space for monetization,\u201d according to internal slides that were disclosed as part of a US<\/a> antitrust court case.<\/p>\n

Also read: Should AI labs be treated like the owners of dangerous animals?<\/a><\/p>\n

The company has since faced a reckoning on that decision. Prince says Cloudflare gave Google an ultimatum: From Sept. 15, the infrastructure firm will block mixed-purpose crawlers for its customers who are ad-supported \u2014 and do so by default. Google could lose access to millions of websites using Cloudflare.<\/p>\n

Further pressure came from the UK<\/a>\u2019s antitrust watchdog. In June, the Competition and Markets Authority ordered Google to give website owners a clear choice to block their content from being used for AI products while remaining in search results. As part of the order, Google can\u2019t punish those websites by pushing them lower down in the rankings.<\/p>\n

Prince told me there were ‘rumblings’ that Google wouldn\u2019t just follow that standard for the UK but apply it globally. A Google spokesperson confirmed to me that the company was indeed testing a new setting that lets websites opt out of its AI answers without hurting their search rankings, and that it will roll that out worldwide once UK testing is done. That marks a heartening shift from the status quo, where the world’s biggest tech firms play by their own rules.<\/p>\n

Also read: Will artificial intelligence soon escape human control?<\/a><\/p>\n

But the company’s crawlers remain technically bundled. The same bot still collects a site\u2019s content for both search and AI, and websites have to trust Google to respect their wishes. Fully separating Google’s crawler into two \u2014 indexing for search, and AI scraping \u2014 would be a better remedy. Websites could then block AI scraping themselves and at their own door rather than relying on Google’s word \u2014 which, as the internal slides from 2024 showed, tends to bend when revenue is at stake. It would also beat the cruder alternatives: a blanket ban on grabbing data for AI training or a vague demand that AI companies pay publishers.<\/p>\n

Google still controls 90% of the search market, making it a de facto gateway to the web, but its exploitative ambitions are becoming more problematic in the age of AI. With any luck, the company\u2019s practices will continue to change for the better.<\/p>\n","protected":false},"excerpt":{"rendered":"Google became the king of internet search and created a free cornucopia of knowledge by striking a clever…\n","protected":false},"author":2,"featured_media":134599,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[24,37221,132,1429,66843,45836,66844],"class_list":["post-134598","post","type-post","status-publish","format-standard","has-post-thumbnail","category-google","tag-ai","tag-ai-products","tag-google","tag-google-ai","tag-internet-search","tag-web-crawler","tag-web-pages"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/134598","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=134598"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/134598\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/134599"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=134598"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=134598"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=134598"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}