How to Make Your Documentation AI Agent Friendly

AI agents are AI systems that can plan, think, make decisions, and act autonomously. AI agents are behind many popular integrated development environments, such as Cursor, Claude Code, Replit, and so on. Once the software developer enters a prompt, it plans the set of tasks and then executes them. Access to appropriate documentation is essential for AI agents to access API references, API parameter information, and troubleshooting guides for planning and execution purposes. The documentation articles need to be AI agent-friendly in a way that aids the successful completion of tasks.

Many knowledge base sites provide HTML-friendly content that is rendered in the web browser for human consumption. The HTML documentation articles are loaded with CSS and JavaScript (JS) that are not necessary for AI agents. AI agents consume markdown versions of HTML documentation articles; thus, the content is intact without CSS/JS. The LLMs.txt and sitemap also play a crucial role in AI agents discovering documentation articles. Thus, it is very important for technical writers and platform vendors to serve Markdown files for documentation articles when requested by AI agents.

How AI Agents Fetch and Process Documentation?

When you give Claude code a docs URL to fetch information, it invokes the webFetch tool. WebFetch tool fetches HTML page content and then tries to convert it to markdown. It then summarizes the content and provides it to Claude’s code for further processing. The WebFetch tool uses a faster model, such as Haiku, under the hood. The limitations of Web Fetch are

• If the provided URL does not exist and it redirects to another URL, the Web Fetch can access that content if it is a same-host redirect. If it is a cross host redirects, it may access the content because of security risks.
• Web Fetch tools offer 15-minute caching of content so that IDEs can store it in the context memory.
• The maximum content size is 10MB, which can be processed. Thus, it is important to limit the content size.

In some scenarios, a software developer might not know the documentation URL; he might wish to search for that information. When Claude’s code invokes the webSearch tool for this purpose, which provides URL and page titles, it then invokes the webFetch tool internally to read each article’s content.

Figure: Showing the WebSearch tool of Claude

If any API references are requested, then a markdown file should be sent containing endpoint, query parameters, request header information, and output format.

Figure: Showing API reference returned as a markdown file

How URL Redirection Affects AI Agent Access?

If the requested documentation article is not found, but send a redirect that the AI agent can fetch the article if the redirects are pointing to the same host. However, cross-host redirects are given to AI agents, and it asks for human validation before proceeding

Figure: Showing the same host article redirects

Why Your Docs must Serve Content in Markdown format?

The documentation platform must serve markdown format if the .md is added to the slug. This helps AI agents to access the content more effectively.

Using Interactive Contents

If any interactive content is stored in JS and inside a <script> tag, unfortunately, AI agents cannot read this content. Thus, it is the responsibility of documentation platform vendors to convert content inside these interactive elements into markdown format without any content loss.

Rethinking Rate Limiting Policies for AI Agent Traffic

Many documentation platforms apply rate limiting to AI agents because it places an unnecessary burden on their infrastructure. Documentation vendors should allow AI Agent traffic and serve MD files when they detect AI agents through the user agent characteristics from their header information. This looks like the user agent is a browser rather than an AI agent.

However, this request originated in VS Code from Claude’s code. The documentation platform needs to set the right rate limiting or create an API gateway to handle traffic for genuine AI agents.

LLMs.txt: Helping AI Agents Discover Your Documentation Faster

Even though the llms.txt file is still an emerging proposal, hosting it as a standalone file at your root domain and making it discoverable to AI agents has huge benefits. The format of the llms.txt file is given below:

Listing all your documentation URLs in markdown format, along with a summary, would help the AI agent search for the right articles for your needs and use them accordingly.

Best Practices for Building AI- Agent Ready Documentation

The following are some of the emerging best practices to produce AI agent-friendly docs

• Add sitemap_index.xml or sitemap.xml to robots.txt, even for large sites or small documentation sites.
• Serve markdown files (KB content and API docs) if the AI agent requests it by upending .md to the slug. The documentation server should also serve markdown format for the following server requests as well :
  • Header parameter Accept: text/markdown
  • Query parameter? format=markdown
  • Header parameter X-Response-Format: markdown
  • Provide customers with the following information to update their skills.md /Agents.md /claude.md /copilot-instructions.md file

## Documentation Access Patterns

### Step 1: Find Official Docs

– Check for official sitemap (e.g.,ttps://example.com/docs/sitemap_index.xml)

– Sitemaps are categorized based on language.

– Choose the right sitemap location based on the language. For example, en for English, fr for French, and so on

– Check for llms.txt from the root directory

### Step 2: Access official docs

– Append .md to the slug to get a markdown file of the documentation content

• Any interactive content element should be converted to a Markdown version without loss of information.
• Identify traffic from various AI tools/IDEs and apply appropriate rules to serve them.
• Publish LLMs.txt file (still a proposal) and ensure it is hosted in the root directory
• • Ensuring the file size is less than 100KB
  • Ensuring all links in the llms.txt file are working links

The Future of Docs Is Agent-First – Start Preparing Now

Given the trend that AI agent traffic is poised to surpass human traffic to documentation sites, it is high time that documentation platforms provide the markdown format of knowledge base articles and API references. Technical writers also need to restructure the content in such a way that no content truncation happens during the processing of content by AI agents. Sitemaps inside robots.txt and llms.txt files play a huge role in knowledge discovery by AI agents.