The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them from \u201caccessing content without permission or compensation,\u201d according to an announcement on Tuesday. With the change, Cloudflare will start asking new domain owners whether they want to allow AI scrapers, and will even let some publishers implement a \u201cPay Per Crawl\u201d fee.<\/p>\n
The Pay Per Crawl program will let publishers set a price for AI scrapers to access their content. AI companies can then view pricing and choose whether to register for the \u201cPay Per Crawl\u201d fee or turn away. This is only available for \u201ca group of some of the leading publishers and content creators\u201d for now, but Cloudflare says it will ensure \u201cAI companies can use quality content the right way \u2014 with permission and compensation.\u201d<\/p>\n
Cloudflare has been helping domain owners fight AI crawlers for a while now. The company started letting websites block AI crawlers<\/a> in 2023, but it only applied to ones that abide by a site\u2019s robots.txt file<\/a>, the unenforceable agreement that signals whether bots can scrape its content. Cloudflare began allowing websites to block<\/a> \u201call\u201d AI bots last year \u2014 whether they respect a site\u2019s robots.txt file or not \u2014 and now this setting is enabled by default for new Cloudflare customers. (The company identifies scrapers to block by comparing them to its list of known AI bots.) Cloudflare also rolled out a feature in March that sends web-crawling bots<\/a> into an \u201cAI Labyrinth\u201d to deter them from scraping sites without permission. <\/p>\n Several major publishers and online platforms, including The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora, are on board with Cloudflare\u2019s new AI crawler restrictions, as websites contend with a future where more people are finding information through AI chatbots, rather than search engines. \u201cPeople trust the AI more over the last six months, which means they\u2019re not reading original content,\u201d Cloudflare CEO Matthew Prince said during the Axios Live event last week<\/a>. <\/p>\n Additionally, Cloudflare says it\u2019s working with AI companies to help verify their crawlers and allow them to \u201cclearly state their purpose,\u201d such as whether they\u2019re using the content for training, inference, or search. Website owners can then review this information and determine which crawlers to let in.<\/p>\n \u201cOriginal content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it,\u201d Prince said in the press release. \u201cAI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"The major internet architecture provider Cloudflare will now block known AI web crawlers by default to prevent them…\n","protected":false},"author":2,"featured_media":229073,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3163],"tags":[323,1942,12,326,53,16,15,4715],"class_list":{"0":"post-229072","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-news","11":"tag-tech","12":"tag-technology","13":"tag-uk","14":"tag-united-kingdom","15":"tag-web"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@uk\/114777552944628331","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/229072","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/comments?post=229072"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/229072\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media\/229073"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media?parent=229072"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/categories?post=229072"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/tags?post=229072"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}