A new standard has been proposed to guide large language models (LLMs) such as ChatGPT, Gemini, and Claude towards relevant web content. This standard, known as llms.txt, is akin to the familiar robots.txt and XML sitemap files. For ecommerce merchants, this new standard could be a game-changer in terms of sharing product information and other content directly with AI systems, especially as consumers increasingly turn to AI platforms for their shopping needs.
Consumers are now relying on generative AI platforms like ChatGPT to recommend products and stores based on their queries. For example, if a user asks, “What are the best trail-running shoes?”, ChatGPT can provide recommendations from various online shops.
The llms.txt standard was proposed by Jeremy Howard, the co-founder of Fast.ai and Answer.AI, in September 2024. The main objective of this standard is to help LLMs process and understand content from large, complex websites. By adopting this standard, websites, particularly ecommerce businesses, can potentially influence AI visibility on their platforms.
The llms.txt file serves as a machine-readable document for AI systems. It signals to AI tools which content is available for their use and which content is relatively easy to summarize and cite. Just like a robots.txt file or a sitemap, llms.txt can allow or disallow various AI tools from using specific content on a website.
For instance, an ecommerce site may disallow content in the shopping cart or checkout flow to prevent AI bots from wasting time on irrelevant pages. Conversely, the file can guide AI towards useful content such as shipping guides, blog posts, and product descriptions.
The structure of an llms.txt file typically follows a pattern in Markdown, a lightweight text markup language. It includes a title, summary, sections, and bullet links to various content pages. This format allows AI to easily navigate and extract information from the website.
One of the key features of llms.txt is its ability to simplify content for AI consumption. For example, an llms.txt file for an ecommerce site could contain links to Markdown versions of product detail pages. These Markdown files are concise, factual, and contain all the necessary information for AI to summarize, share, and cite the products effectively.
During the interface, llms.txt aids LLMs in providing quick and accurate responses to user queries. The AI can quickly access the Markdown files, extract relevant information, and respond to the user’s questions effectively. This process is akin to providing CliffsNotes for the LLMs, enabling them to process information more efficiently.
In essence, llms.txt is like SEO for LLMs, optimizing websites for chatbots and answer engines. While the standard may or may not gain widespread adoption, it signifies a growing trend towards aiding AI in indexing and referencing web pages. It could potentially shape the future of AI discoverability and enhance the shopping experience for consumers on AI-powered platforms.