Most of the world’s largest websites lack policies for managing AI agents that answer customer questions, complete purchases or collect content, according to research from D3 Research.<\/p>\n
The study assessed the 10,000 highest-ranked domains and assigned a median readiness score of 56 out of 100. Among the 5,577 sites with readable policy files, 84% received a D or F grade. Only 13 achieved an A.<\/p>\n
D3 Research found that most sites allowed AI agents to access content without deliberately granting access or recording what the agents collected. Other sites used broad blocking rules that restricted customer-facing agents alongside training crawlers and other automated services.<\/p>\n
Traffic shift<\/p>\n
Automated agents accounted for 57.5% of web traffic in June 2026, according to Cloudflare Radar figures cited in the study. Humans generated the remaining 42.5%.<\/p>\n
Agent traffic grew by 7,851% over the previous year, based on figures from HUMAN Security cited by D3 Research. More than 1,500 AI services were already accessing websites.<\/p>\n
D3 Research separated the traffic into agents that answer questions, agents that act for customers and services that collect content. It found that websites often treated these different activities as a single category.<\/p>\n
The study said websites generally managed automated traffic through robots.txt files. The format was created to tell automated services which parts of a site they could access.<\/p>\n
Those files rely on names supplied by the visiting service. D3 Research found that 95.8% of 2,175 known agents provided no additional information through which a website could verify their identity.<\/p>\n
Poor scores<\/p>\n
The research assigned grades using four measures. Answer eligibility and buyer access each accounted for 35 points. Content exposure contributed 20 points, while identity controls represented the final 10 points.<\/p>\n
A total of 4,198 sites received a D grade. Another 484 received an F. There were 461 sites graded C and 421 graded B.<\/p>\n
D3 Research said 73% of sites had no written rules covering AI traffic. These websites often remained accessible to agents but collected no evidence about which services entered or what content they accessed.<\/p>\n
Of the 100 largest websites examined, 93% received a D or F.<\/p>\n
Nvidia recorded a score of 93 and received an A. Traveloka scored 90. Google, Apple and Microsoft each scored 56, while Amazon scored 27.<\/p>\n
The results were based on the policies visible when the sites were scanned. Individual scores may change when a company updates its robots.txt file.<\/p>\n
Customer access<\/p>\n
The study found that 693 sites blocked at least one agent capable of shopping or acting for a customer. This represented around one in eight of the sites with readable policies.<\/p>\n
Only 151 of those sites had named and deliberately blocked the relevant agents. The remaining 542 blocked customer-facing agents through older blanket rules or inconsistent vendor-specific controls.<\/p>\n
A total of 373 sites blocked every shopping agent assessed. Another 320 allowed an agent from one provider while blocking an equivalent service from another.<\/p>\n
Just 125 sites had policies deliberately welcoming a shopping agent. This represented 2.2% of the sample. Another 4,759 sites neither explicitly permitted nor blocked those agents.<\/p>\n
Online marketplaces recorded the weakest purchasing access among the commercial sectors examined. Forty-three per cent blocked at least one shopping agent. The sector received an average score of 44.<\/p>\n
News organisations also imposed extensive restrictions. Among 177 newspaper sites assessed for purchasing access, 37% turned away at least one customer-facing agent.<\/p>\n
Policy gaps<\/p>\n
D3 Research found that companies had written more policies for services collecting training data than for agents carrying out tasks for customers.<\/p>\n
GPTBot was covered by policies at 1,215 sites. CCBot appeared in 1,076 policies and ClaudeBot in 1,043.<\/p>\n
ChatGPT-User, which acts for an individual, appeared in 642 policies. Perplexity-User appeared in 305, while Claude-User appeared in 280.<\/p>\n
Among the 1,508 sites that had established a deliberate AI policy, 59.1% blocked at least one service that answered customer questions. Almost a third blocked a shopping agent.<\/p>\n
The company-specific approach also produced uneven treatment. D3 Research identified 750 sites that allowed certain answer engines while blocking others. It found 418 sites with rules for OpenAI’s customer-facing services but no equivalent policy for Anthropic.<\/p>\n
Content records<\/p>\n
The study found that 4,114 sites, or 74% of the readable sample, kept no policy record covering companies that collected their content.<\/p>\n
Only 225 sites had rules covering every content-collecting company assessed. D3 Research said such records could help organisations determine which services accessed their material and whether those services generated customer referrals.<\/p>\n
Technology sites allowed customers to reach them in 77% of cases, while 7% recorded any content exposure. Five per cent blocked a buyer.<\/p>\n
E-commerce sites recorded content exposure more frequently, at 25%. Their customer accessibility rate was lower at 64%, and 21% turned away a buyer.<\/p>\n
The study measured declared policies rather than live website traffic. It excluded network-level restrictions, security controls and other enforcement mechanisms that were absent from the published robots.txt files.<\/p>\n","protected":false},"excerpt":{"rendered":"Most of the world’s largest websites lack policies for managing AI agents that answer customer questions, complete purchases…\n","protected":false},"author":2,"featured_media":152895,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[10648,11847,405,7903,8923,321,53,319,111,7537,6860,7129,10380,7135,313,7144,7162,74502,1732,132,7692,320,58,157,724,10583,14868],"class_list":["post-152894","post","type-post","status-publish","format-standard","has-post-thumbnail","category-agentic-ai","tag-access-control","tag-advertising-technologies-adtech","tag-ai-agents","tag-ai-agents-agentic-ai","tag-ai-ethics-governance","tag-amazon","tag-anthropic","tag-apple","tag-artificial-intelligence-ai","tag-artificial-intelligence-agents","tag-cloudflare","tag-conversational-ai","tag-customer-experience-cx","tag-customer-service","tag-cybersecurity","tag-data-collection","tag-data-governance","tag-digital-risk","tag-ecommerce","tag-google","tag-identity-verification","tag-microsoft","tag-nvidia","tag-openai","tag-perplexity","tag-risk-compliance","tag-user-experience-ux"],"_links":{"self":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/152894","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/comments?post=152894"}],"version-history":[{"count":0,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/posts\/152894\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media\/152895"}],"wp:attachment":[{"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/media?parent=152894"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/categories?post=152894"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.europesays.com\/ai\/wp-json\/wp\/v2\/tags?post=152894"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}