How Cloudflare Serves Markdown to AI Agents Explained

Cloudflare envisages that AI agents’ traffic will surpass human traffic to the documentation site. However, many documentation sites still serve HTML content to AI agents, while they prefer Markdown. Cloudflare recently introduced a feature that serves a Markdown version of the documentation article when requested by AI agents. Cloudflare automatically converts HTML format to Markdown using content negotiation headers.

If the AI agent makes an HTTP call to retrieve the documentation article’s content, it needs to add an Accept negotiation header with text/markdown as one of the options. Cloudflare will detect this, fetch the original HTML version from the origin, and convert it to markdown before serving it to the AI agent.

Key Content Translation Problem

The high-level architecture of the solution is shown below

Cloudflare’s feature is elegant. However, it suffers from several content translation problems, such as

1. Unnecessary information is being added to the content

Some interactive elements are included in the Markdown content, including a search button, read-aloud, and feedback. This type of information should be removed before giving it to AI agents.

2. Interactive elements are not being processed correctly

In many documentation sites, there is a chance that they may contain interactive elements such as dropdown lists, tabs, and collapsible elements for displaying content based on user input. The content for those elements is stored in JavaScript and is loaded dynamically. For example, the screenshot shows tabs as interactive elements.

When an AI agent requests this article, Cloudflare serves an inconsistent markdown format as shown below

This adds another layer of complexity for AI agents to understand this markdown content.

3. No good hierarchy of headings with proper tags from H1 to H6

Cloudflare’s conversion loses information on headings and topics. H1 tags are not followed by H2 and so on. This mismatched content hierarchy is shown below:

4. API documentation is served as raw data rather than YAML

API endpoint references are converted to raw data rather than to YAML with proper indentation, so AI agents can use the API reference to build integration pipelines.

An example of a scrambled API definition is shown below

For example, Anthropic API references are served neatly in markdown format as

How AI Agents Request Markdown Using Accept Headers

Modern -day AI agents not only use Accept headers in the request headers; they might also try different methods, such as

1. Adding URL parameter ?format=markdown
2. Adding X-Response-Format: markdown
3. Adding .md extension to the slug to access the markdown version

The above methods do not work for Cloudflare functionality. It is recommended to build your documentation server to support all functionality for serving Markdown rather than relying on Cloudflare’s feature for converting HTML to Markdown. Cloudflare also intelligently adds content-length, x-markdown-tokens, and x-original-tokens to show how many tokens it has saved. This also helps AI agents adapt if they have inherent limits on truncating content due to a limited context window size. The response header shows all information

Using AI Agent Analytics to Improve Documentation Strategy

All the AI agent traffic data offers rich insights. Insights into the number of requests, AI agent client origin, list of popular documentation articles, and specific patterns in accessing documentation help technical writers to focus on the right set of high-value documentation articles. The AI agent traffic data also helps customers to understand the trend in consumption of their documentation. If the AI agent is surpassing human traffic, then it is high time to rethink documentation from the ground up, as your primary customers are AI agents and their characteristics of consumption are different from those of human readers.

Also, AI agent analytics helps technical writers to restructure documentation so that it helps AI agents accomplish activities. Thus, it is vital to keep the high-value documentation up to date and accurate. It is now easier to trace the documentation value chain because technical writers can quantify the value of documentation.

Should You Rely on Cloudflare Markdown or Build Your Own?

Even though Cloudflare offers an option to convert HTML files into markdown format for AI Agent use, it is recommended to implement this functionality in your own server. This is due to content loss, inconsistent content hierarchy, and raw API reference issues. Also, serving markdown for your own documentation server offers a good insight into AI agent behavior and rich analytics that can be used to refine documentation content. Rich analytics helps technical writers to optimize the content length and focus on the right content. This ensures that AI agents are deriving value from documentation to accomplish tasks.