{"id":318431,"date":"2025-08-05T00:12:15","date_gmt":"2025-08-05T00:12:15","guid":{"rendered":"https:\/\/www.europesays.com\/uk\/318431\/"},"modified":"2025-08-05T00:12:15","modified_gmt":"2025-08-05T00:12:15","slug":"cloudflare-says-perplexitys-ai-bots-are-stealth-crawling-blocked-sites","status":"publish","type":"post","link":"https:\/\/www.europesays.com\/uk\/318431\/","title":{"rendered":"Cloudflare says Perplexity\u2019s AI bots are \u2018stealth crawling\u2019 blocked sites"},"content":{"rendered":"

The AI search startup Perplexity is allegedly skirting restrictions meant to stop its AI web crawlers from accessing certain websites, according to a report from Cloudflare<\/a>. In the report, Cloudflare claims that when Perplexity encounters a block, the startup will conceal its crawling identity \u201cin an attempt to circumvent the website\u2019s preferences.\u201d<\/p>\n

The report only adds to concerns about Perplexity vacuuming up content without permission, as the company got caught barging<\/a> past paywalls and ignoring sites\u2019 robots.txt files last year. At the time, Perplexity CEO Aravind Srinivas blamed the activity<\/a> on third-party crawlers used by the site.<\/p>\n

Now, Cloudflare, one of the world\u2019s biggest internet architecture providers, says it received complaints from customers who claimed that Perplexity\u2019s bots still had access to their websites even after putting their preference in their websites\u2019 robots.txt file<\/a> and by creating Web Application Firewall (WAF) rules to restrict access to the startup\u2019s AI bots.<\/p>\n

To test this, Cloudflare says it created new domains with similar restrictions against Perplexity\u2019s AI scrapers. It found that the startup will first attempt to access the sites by identifying itself as the names of its crawlers: \u201cPerplexityBot\u201d or \u201cPerplexity-User.\u201d<\/p>\n

But if the website has restrictions against AI scraping, Cloudflare claims Perplexity will change its user agent \u2014 the bit of information that tells a website what kind of browser and device you\u2019re using, or if the visitor is a bot \u2014 to \u201cimpersonate Google Chrome on macOS.\u201d Cloudflare says this \u201cundeclared crawler\u201d uses \u201crotating\u201d IP addresses that the company doesn\u2019t include<\/a> on the list of IP addresses used by its bots.<\/p>\n

Additionally, Cloudflare claims that Perplexity changes its autonomous system networks (ASN), a number used to identify groups of IP networks controlled by a single operator, to get around blocks as well. \u201cThis activity was observed across tens of thousands of domains and millions of requests per day,\u201d Cloudflare writes.<\/p>\n

In a statement to The Verge, Perplexity spokesperson Jesse Dwyer called Cloudflare\u2019s report a \u201cpublicity stunt,\u201d adding that \u201cthere are a lot of misunderstandings in the blog post.\u201d Cloudflare has since delisted Perplexity as a verified bot and has rolled out methods to block Perplexity\u2019s \u201cstealth crawling.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"The AI search startup Perplexity is allegedly skirting restrictions meant to stop its AI web crawlers from accessing…\n","protected":false},"author":2,"featured_media":318432,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[323,51,12,326,16,15,4715],"class_list":{"0":"post-318431","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-business","8":"tag-ai","9":"tag-business","10":"tag-news","11":"tag-tech","12":"tag-uk","13":"tag-united-kingdom","14":"tag-web"},"share_on_mastodon":{"url":"https:\/\/pubeurope.com\/@uk\/114973268205994019","error":""},"_links":{"self":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/318431","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/comments?post=318431"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/posts\/318431\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media\/318432"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/media?parent=318431"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/categories?post=318431"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/uk\/wp-json\/wp\/v2\/tags?post=318431"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}