Gartner projects that by 2029, agentic AI will resolve 80% of common customer service issues without human intervention. The shift is already visible at the front door of your documentation. When a customer asks your chatbot a question, it provides an answer from your documentation articles. Your customers are no longer reading the doc; instead, they read the chatbot’s response, which is based on the documentation. If the documentation is incorrect, the chatbot’s response will be incorrect. Technical writers now have additional responsibilities to ensure that the chatbot’s responses are grounded in truth and to validate them over time. If a chatbot produces an incorrect response, it compounds errors in Agentic AI systems. Thus, documentation content is becoming more important than ever, serving as a trusted source of truth for chatbots, AI agents, and humans interacting with it.

What is Docs-as-evals?

Evals are very simple to understand.

• Write the input query
• Get ground truth for the input query from a Subject Matter Expert that is written into the documentation content
• Get the response from the chatbot/Agent from that input query

Now score the gap between ground truth and the generated response from the chatbot. Thus, documentation content can serve as ground truth for both human readers and chatbots. The most important part of a chatbot or AI agent is accurately retrieving an article and its content. Then, content retrieval proceeds to the content generation part, where a response is generated.

Given the nuances in asking questions, a chatbot might produce a slightly different response each time. This means that generated responses may be slightly off from the ground truth. This has to be evaluated over time to ensure that the chatbot does not provide incorrect responses. The nuance in asking questions includes customers

• Using different words in the questions
• Using different business terms in the question
• Using ambiguous terms in the question
• Just using keywords instead of a well-framed question
• Asking multiple questions in the same query
• Trying to hack the chatbot’s response

The Docs-as-eval paradigm enables technical writers to position documentation content as the ground truth, regardless of how the chatbot responds. This helps technical writers and other stakeholders to evaluate the performance of chatbots or any agentic systems. Evals are repeatable, and now documentation content can be considered as regression for chatbots and AI agentic systems. Documentation content written to be read often fails when asked to be reproduced.

Figure: Document360 Eddy AI answering a question based on knowledge grounding

Build AI-ready documentation that powers accurate answers with Document360.

Book a Demo

The Technical Writer’s Role in Docs-as-Evals

Defining a correct answer sounds like it is an editorial decision. However, it is the technical writer’s craft. When the technical writer designs the content, they should know what kinds of questions can be asked and ensure they cover all elements required to answer them. While software developers define what the software does, technical writers define what constitutes a true statement about a software product. They cover all nuances of a particular product feature in detail, providing comprehensive knowledge of the software’s features.

Technical writers’ practice ensures that technical writing is free from ambiguity, uses correct terminology, and chooses words to explain things clearly. If a technical writer is covering a step-by-step procedure, all steps are captured in detail. Screenshots are added at the right step if visual elements would help to bring clarity. Technical writers are thus accountable for ground truth. This is an additional responsibility for modern technical writers: ensuring that docs are used to evaluate chatbots and Agentic AI systems.

How to Write Documentation That Survives as an Eval

Docs-as-eval is an emerging paradigm, and technical writers should keep in mind that docs are verifiable and serve as ground truth for evaluation. Tips for writing a doc that would survive an eval are listed below

• Write testable claims: One fact per sentence so that it can be checked as true or false
• Avoid using any adverbs or hedging language such as “usually” and “in most cases,” as it gives more room for AI Agents to fabricate the generated response
• Remove any ambiguity from the content and ensure that no undefined terms are used, no implied steps in step-by-step procedure, and no pronouns are used without any clear referent
• Ensure that each answer is self-contained such that it does not depend on context the agent never received
• For any step-by-step procedure, do not skip any steps. For any edge cases, write a separate procedure
• State all prerequisites/preconditions explicitly
• Also ensure that all boundaries and nuances are covered in detail
• Use consistent terminology across your knowledge base content because chatbot’s retrieval might be semantic, so synonyms might confuse response generation
• Structure the content into self-sufficient chunks such that they survive on their own being pulled out of the page content
• Add “do not” cases or explicitly state what content does not cover so that Agents/chatbot do not hallucinate

Start Treating Your Documentation as an Answer Key

Creating a live artifact that contains a set of questions and ground truth from the documentation content is a testament to a technical writer’s ability to adapt to a new wave of technology. Before taking a chatbot or AI agent live, the docs-as-eval framework provides assurance that the failure rate is acceptable. Eval scoring mechanisms keep the team honest about the performance of chatbots and Agentic AI systems. Eval metrics help to improve the documentation content. If the underlying architecture of chatbots changes or chatbots are updated to use the latest LLM, then docs-as-eval serves as a regression test. Technical writers need to add more questions and ground truth to the eval tests to strengthen the regression test suite.