![\"Robots.txt](\"https:\/\/www.europesays.com\/uk\/wp-content\/uploads\/2025\/09\/shutterstock_2163824051-scaled-e1756904135903-1038x778.webp.webp\")
\n Robots.txt illustration. Picture: Shutterstock\/BestForBest<\/p>\n
\n Robots.txt illustration. Picture: Shutterstock\/BestForBest<\/p>\n
Unwanted AI<\/a> scraping of publisher websites is placing a costly financial burden on publishers, according to one leading industry executive.<\/p>\n Chris Dicker, chief executive of Candr Media Group and board member of the Independent Publishers Alliance, said that the publisher\u2019s Trusted Reviews website was taken down multiple times on 16 August when it was scraped 1.6 million times in a day.<\/p>\n This was up from a previous record of 1.2 million scrapes on the site a day earlier.<\/p>\n He said the average level of AI scraping for Trusted Reviews is running at between approximately 70,000 and 100,000 times a day.<\/p>\n Out of the 1.6 million AI scrapes performed on 16 August, Dicker said he believed these resulted in 603 actual users of generative AI platforms later landing on the Trusted Reviews site, meaning a clickthrough rate from ChatGPT and other similar platforms of 0.037%. He noted in a Linkedin post that this was \u201cdramatically lower than you would expect from traditional search\u201d.<\/p>\n He told Press Gazette that the referral ratio for the previous week from an AI bot scrape resulting in a human visit was 1,888 to one.<\/p>\n He also said engagement from that traffic \u201cperformed poorly\u201d with 58% less time spent on the site than an average user and 10% fewer pages viewed.<\/p>\n Dicker said Candr is currently preparing to change hosting provider and therefore had not taken some steps such as implementing Tollbit\u2019s bot paywall<\/a>, which allows sites to set fees for AI crawlers to access their content and which the publisher plans to do. <\/p>\n But Dicker told Press Gazette the data raised concerns about some AI companies not respecting robots.txt signals that a website does not want to be crawled by specified bots.<\/p>\n He said Tollbit\u2019s dashboard for Trusted Reviews showed that OpenAI had ignored robots.txt to scrape the site 12.2 million times in the past three months, with Meta doing so 2.8 million times, Amazon 2.4 million times, Perplexity 101,000 and Bytedance 95,000.<\/p>\n Trusted Reviews saw an increase in AI bots ignoring robots.txt of 75% compared to the previous three months, he added, while in the past six months it was up 89% compared to the prior six months.<\/p>\n Website hosting company Cloudflare announced in July that it would allow websites to block all scrapers by default.<\/a><\/p>\n In response to Dicker\u2019s Linkedin post<\/a>, OpenAI\u2019s head of media partnerships Varun Shetty said: \u201cTook a look at your robots.txt file and you are allowing OAI-Searchbot access to your site. If that\u2019s something you\u2019d prefer not to do, you can block the crawler and you should see this issue resolved. We\u2019re working closely with publishers on finding ways that our products can drive them value \u2013 will continue to do that.\u201d<\/p>\n However, Dicker told Press Gazette that it was not OAI-Searchbot that was driving the spike in scraping \u2013 it was another OpenAI bot, ChatGPT-User<\/a>, which is blocked by Trusted Reviews.<\/p>\n This bot has been the most prominent of the scrapers on Trusted Reviews, Dicker said, although there are occasional spikes from other tech companies such as Apple, when it said it planned to create an LLM, and Amazon, ahead of sales periods. Bytedance and Meta are also present.<\/p>\n Dicker also noted that with the onus on publishers to block bots via robots.txt, they need to know what the bots are in order to do so. This means that any new ones that spring up are harder to block in advance.<\/p>\n \u2018Actual hard cost\u2019 to publishers of AI scraping is hard to pin down<\/p>\n Dicker also raised concerns around the cost of huge spikes in scraping to publishers, especially smaller ones.<\/p>\n \u201cFrom an Independent Publishers Alliance perspective, some of our members have come to me and said, Chris, we\u2019ve got an issue where we are getting scraped so much that our hosting providers are saying we now need to move up a package and that is costing thousands of pounds a year,\u201d he said.<\/p>\n Dicker added that the impact is wider than just the hosting fees: \u201cWhat\u2019s the impact on the users that were trying to come to our site when the site went down? What brand impact is happening?\u2026 [and] the fact that we won\u2019t be able to serve adverts to them at that particular time. The actual hard cost is almost impossible to actually put a number on. How do you put a number on damage to your brand?\u201d<\/p>\n [Read more: Urgent bid from Independent Publishers Alliance and campaign groups lodged with UK regulator to stop Google AI Overviews \u2018stealing journalism\u2019<\/a>]<\/strong><\/p>\n Dicker said Trusted Reviews has been around for 20 years and spent millions on marketing \u201cto put it in the position it\u2019s in. What\u2019s the impact that all of a sudden users are coming to the site and not able to access it, or come on and have a terrible experience because of the bot activity? A fraction of the millions of pounds we spent? Add it to your bill.\u201d<\/p>\n AI scraping \u2018not matched by meaningful value exchange\u2019<\/p>\n Stuart Forrest, global SEO director for publishing at Bauer Media, told Press Gazette that LLM bots (excluding Google) are making up an average of around 20% of total scraping on Bauer sites and at other publishers according to data he has seen.<\/p>\n He noted that a new tracker from software company Ahrefs<\/a> shows that in July, 42% of referral traffic to 44,000 websites came from Google (pointing out that this does not only include publishers, and Google may index even higher if that was the case). Meanwhile ChatGPT, the biggest LLM by referral traffic sent, only referred 0.2% of all traffic.<\/p>\n \u201cOn one hand, you\u2019ve got this growth in cost of scraping, but it\u2019s just not matched by any kind of meaningful value exchange\u2026 for me, that\u2019s the fundamental \u2013 that there is not a value exchange for publishers.\u201d<\/p>\n