Serving docs to AI agents: Learnings and best practices

As AI assistants like ChatGPT, Claude, Gemini, and Cursor become the primary way users access information, documentation is no longer just read by humans.

In this webinar, Selvaraju Murgeson, Senior Director of Data Science, explored how AI agents discover, fetch, read, and use documentation, and what technical writers and documentation teams can do to make their content AI-ready.

What Was Discussed?

Documentation Is Now Executable

AI agents use documentation to perform tasks, answer questions, and generate solutions. This means technical writers are no longer creating content solely for readers, but also for AI systems that consume and execute instructions.

From SEO to AI Discoverability

Traditionally, users found documentation through search engines, and organizations relied on SEO, sitemaps, and analytics to measure engagement.

Today, users increasingly ask AI assistants instead of searching websites directly. These assistants search for documentation, retrieve content, and summarize answers without the user ever visiting the documentation site.

Why Markdown Matters

AI retrieval tools prefer Markdown because it is lightweight and easy to process.

When documentation is served primarily as HTML with heavy JavaScript and CSS:

• Important content may be buried deep in page code
• Context windows can truncate content
• AI agents may miss critical information

Understanding LLMS.txt

LLMS.txt is a documentation index designed for AI assistants.

Think of it as a navigation guide that helps AI systems understand:

• What content exists
• Where it is located
• Which articles are relevant

A typical LLMS.txt file includes:

• Article title
• Article URL
• Short description

Agent-Friendly Documentation Tips

To make documentation more useful for AI agents:

• Write self-contained articles with complete context
• Use clear headings and structured content
• Ensure tabs, accordions, and interactive elements convert properly to Markdown
• Include accurate, working code samples
• Prefer step-by-step text instructions over screenshots and videos
• Maintain glossaries and define important business terminology

Agent Analytics: The New Documentation Metric

Unlike human readers, AI agents quickly fetch content and leave minimal traces.

Tracking agent traffic helps teams understand:

• Which articles AI accesses most frequently
• Emerging content trends
• Documentation gaps
• Areas requiring higher accuracy

LLMS.txt vs MCP Servers

LLMS.txt

• Public documentation discovery
• No authentication required
• Helps AI assistants find content

MCP Servers

• Secure, authenticated access
• Ideal for private knowledge bases
• Respect user permissions and access controls

Rule of thumb: Use LLMS.txt for public documentation and MCP servers for private content.

Key Takeaways

• AI agents are becoming major consumers of documentation.
• Markdown is the preferred format for AI retrieval.
• LLMS.txt improves documentation discoverability for AI assistants.
• Well-structured, self-contained content reduces hallucinations.
• Agent analytics help teams understand and optimize AI-driven documentation usage.
• Documentation strategy is evolving from SEO-focused content to AI-ready content.

About the Speaker

Selvaraaju Murugesan received the B.Eng. degree in Mechatronics Engineering (Gold medalist) from Anna University in 2004 and the M.Eng. degree from LaTrobe University, Australia, in 2008. He has received his Ph.D. degree in Computational mathematics, LaTrobe University. He is currently working as a Senior Director – Data Science at Kovai.co. His interests are in the areas of Business Strategy, Data Analytics, Artificial Intelligence, and Technical Documentation.