Many popular WordPress SEO plugins and content management platforms offer the ability to generate LLMs.txt for the purpose of improving visibility in AI search platforms. With so many popular SEO plugins and CMS platforms offering LLMs.txt functionality, one might come away with the impression that it is the new frontier of SEO. The fact, however, is that LLMs.txt is just a proposal, and no AI platform has signed on to use it.<\/p>\n
So why are so many companies rushing to support a standard that no one actually uses? Some SEO tools offer it because their users are asking for it, while many users feel they need to adopt LLMs.txt simply because their favorite tools provide it. A recent Reddit discussion on this very topic is a good place to look for answers.<\/p>\n
Third Party SEO Tool And LLMs.txt<\/p>\n
Google\u2019s John Mueller addressed the LLMs.txt confusion in a recent Reddit discussion.\u00a0 The person asking the question was concerned because an SEO tool flagged it as 404, missing. The user had the impression that the tool implied it was needed.<\/p>\n
Their question<\/a> was:<\/p>\n \u201cWhy is SEMRush showing that the \/llm.txt is a 404? Yes, I. know I don\u2019t have one for the website, but, I\u2019ve heard it\u2019s useless and not needed. Is that true?<\/p>\n If i need it, how do i build it?<\/p>\n Thanks\u201d<\/p>\n<\/blockquote>\n The Redditor seems to be confused by the Semrush audit that appears to imply that they need an LLMs.txt. I don\u2019t know what they saw in the audit but this is what the official Semrush audit documentation<\/a> shares about the usefulness of LLMs.txt:<\/p>\n \u201cIf your site lacks a clear llms.txt file it risks being misrepresented by AI systems.<\/p>\n \u2026This new check makes it easy to quickly identify any issues that may limit your exposure in AI search results.\u201d<\/p>\n<\/blockquote>\n Their documentation says that it\u2019s a \u201crisk\u201d to not have an LLMs.txt but the fact is that there is absolutely no risk because no AI platform uses it. And that may be why the Redditor was asking the question, \u201cIf i need it, how do I build it?\u201d<\/p>\n LLMs.txt Is Unnecessary<\/p>\n Google\u2019s John Mueller confirmed that LLMs.txt is unnecessary.<\/p>\n He explained:<\/p>\n \u201cGood catch! Especially in SEO, it\u2019s important to catch misleading & bad information early, before you invest time into doing something unnecessary. Question everything.\u201d<\/p>\n<\/blockquote>\n Why AI Platforms May Choose To Not Use LLMs.txt<\/p>\n Aside from John Mueller\u2019s many informal statements about the uselessness of LLMs.txt, I don\u2019t think there are any formal statements from AI platforms as to why they don\u2019t use LLMs.txt and their associated .md markdown texts. There are, however, many good reasons why an AI platform would choose not to use it.<\/p>\n The biggest reason not to use LLMs.txt is that it is inherently untrustworthy. On-page content is relatively trustworthy because it is the same for users as it is for an AI bot.<\/p>\n A sneaky SEO could add things to structured data and markdown texts that don\u2019t exist in the regular HTML content in order to get their content to rank better. It is naive to think that an SEO or publisher would not use .md files to trick AI platforms.<\/p>\n For example, unscrupulous SEOs add hidden text and AI prompts within HTML content. A research paper from 2024 (Adversarial Search Engine Optimization for Large Language Models) showed that manipulation of LLMs was possible using a technique they called Preference Manipulation Attacks.<\/p>\n Here\u2019s a quote from that research paper (PDF<\/a>):<\/p>\n \u201c\u2026an attacker can trick an LLM into promoting their content over competitors. Preference Manipulation Attacks are a new threat that combines elements from prompt injection attacks\u2026 Search Engine Optimization (SEO)\u2026 and LLM \u2018persuasion.\u2019<\/p>\n We demonstrate the effectiveness of Preference Manipulation Attacks on production LLM search engines (Bing and Perplexity) and plugin APIs (for GPT-4 and Claude). Our attacks are black-box, stealthy, and reliably manipulate the LLM to promote the attacker\u2019s content. For example, when asking Bing to search for a camera to recommend, a Preference Manipulation Attack makes the targeted camera 2.5\u00d7 more likely to be recommended by the LLM.\u201d<\/p>\n<\/blockquote>\n The point is that if there\u2019s a loophole to be exploited, someone will think it\u2019s a good idea to take advantage of it, and that\u2019s the problem with creating a separate file for AI chatbots: people will see it as the ideal place to spam LLMs.<\/p>\n It\u2019s safer to rely on on-page content than on a markdown file that can be altered exclusively for AI. This is why I say that LLMs.txt is inherently untrustworthy.<\/p>\n What SEO Plugins Say About LLMs.txt<\/p>\n The makers of Squirrly WordPress SEO plugin acknowledge that they provided the feature only because their users asked for it, and they assert that it has no influence on AI search visibility.<\/p>\n\n
\n
\n
\n