# The manual is the API

## When a coding agent needs to follow your docs

Maya is a backend developer joining a team that maintains **MercuryFlow**, an imaginary platform for managing payment-processing services.

On her first week, she's asked to add automatic retry handling to a MercuryFlow payment worker.

She asks her coding agent:

> How do I configure retries for a MercuryFlow worker?

The agent already has access to the MercuryFlow source repository, so it could search the code directly: the retry implementation, the configuration structs, the defaults, the tests. But that's not what Maya asked. She wants the **supported procedure** for configuring retries, not the implementation.

That distinction matters. Source code tells an agent what the software does. Documentation tells it how developers are expected to use it: which configuration method is supported, what prerequisites apply, what order to follow, and what limitations or warnings matter along the way. Not every consumer even has repo access to begin with: a support chatbot, an external developer assistant, or a RAG pipeline may only ever see the published documentation, never MercuryFlow's private codebase.

So, for this question, the agent needs to do five things:

1. Discover what documentation exists.
2. Identify which of it is procedural rather than conceptual.
3. Find the specific procedure that matches the question.
4. Retrieve the actual text.
5. Follow it.

None of that is exotic. It's what any new contributor does by hand on their first day.

The trouble is doing it against a normal documentation website. HTML is built for a browser and a person scrolling it, not for a program deciding what to fetch next. A crawler has to:

1. Walk the navigation menu.
2. Guess which URLs are likely to hold a procedure rather than a glossary entry.
3. Open pages just to find out what kind of content is on them.

For a small site, that may be acceptable. For a documentation corpus with hundreds of pages, the agent can spend a meaningful share of its context budget scraping before it reaches an answer, the context-window equivalent of reading the whole employee handbook to find the Wi-Fi password. One common failure mode is simpler: rather than pay that cost, the agent skips the docs and answers from general knowledge instead of the project's actual, current procedure.

MercuryFlow's documentation can offer a different interface. Instead of crawling, the agent asks the documentation site directly:

1. What it has.
2. How that content is classified.
3. Where to retrieve it.

It first discovers the documentation contract:

```text
GET /schema.json
```

The schema tells it that the corpus distinguishes `concept`, `task`, and `reference` content, and that those values can be used when querying the index.

The agent needs a procedure, so it asks for tasks:

```text
GET /en/index.json?contentType=task
```

Among the results it finds:

```json
{
  "title": "Configure retry policies",
  "url": "/en/guides/configure-retry-policies/",
  "markdown": "/en/guides/configure-retry-policies.md",
  "contentType": "task",
  "pageType": "topic"
}
```

It follows the `markdown` URL and retrieves the procedure itself, without the navigation, footer, or client-side presentation that surround the human-facing page.

The agent can now give Maya the documented procedure instead of reconstructing one from the site's HTML.

The example is imaginary, but the mechanism is not. It rests on three simple outputs from the documentation corpus.

## What the agent actually finds: schema, index, Markdown

That whole workflow rests on three things, used in order.

| Endpoint         | Role                | What it provides                                                                                                                                                                                                                                                                                                              |
| ---------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/schema.json`   | Discovery step      | Tells a consumer what's queryable before it fetches anything else: which content types and page types exist, and which query parameters are supported. An agent reads it once and knows how to ask for what it wants, instead of hard-coding assumptions about the site's structure.                                          |
| `/en/index.json` | Corpus index        | A machine-readable listing of the documentation corpus, with title, URL, and a `markdown` property pointing at that document's clean text mirror. Accepts the parameters that `schema.json` advertises, so a request such as `?contentType=task` filters it down to procedural entries instead of returning the whole corpus. |
| `/llms.txt`      | Lighter entry point | The same corpus as a short table of contents, following the community [llms.txt](https://llmstxt.org/) convention. A lighter starting point than the full index.                                                                                                                                                              |

The Markdown mirror is the payload: following the `markdown` URL from an index entry returns the same content a human reader would see, without the navigation bar, the footer, or any client-side script.

This is the same mechanism the coding agent used above. It isn't a separate "AI API" bolted onto the site. It's the ordinary build output, queried directly instead of rendered into a page first.

## The same source: chatbots and RAG pipelines

Maya's coding agent is only one possible consumer.

Imagine that MercuryFlow also has a support chatbot. A user asks:

> How do I configure retries for a MercuryFlow worker?

The chatbot doesn't need the whole documentation corpus in its context window, and it doesn't need to guess which pages might be relevant by crawling links. It:

1. Identifies the content type it wants (a task or a reference entry, not a blog-style concept piece).
2. Queries the index for matching entries.
3. Fetches the Markdown for the best match.
4. Uses that text directly as grounding for its answer.

A retrieval-augmented generation pipeline can use the same discovery mechanism when ingesting documentation, before it ever handles a user query. A common way to ingest documentation from a website is to crawl and extract its rendered pages. Publishing a queryable index and Markdown mirrors offers another path:

1. Read `/schema.json` to see what's there.
2. Read `/en/index.json` for the full document list with retrieval URLs already attached.
3. Fetch each `.md` file directly instead of parsing rendered HTML.

That's one useful way to build the ingestion step, not the only valid RAG architecture. Plenty of systems will keep scraping HTML or ingesting a raw Git checkout instead. Nothing about the documentation interface requires a vector database either. What comes after retrieval, embedding, indexing, prompting, is a decision the consuming application makes on its own.

## An agent scoped to procedures only

The more interesting consequence of that taxonomy shows up when a consumer wants to exclude content, not just find it.

Suppose Maya's coding agent is configured specifically to help developers perform MercuryFlow procedures, nothing else. Conceptual background and reference tables just spend its context budget on content it won't act on.

| `contentType` value | Returns                             |
| ------------------- | ----------------------------------- |
| `task`              | Only procedural entries.            |
| `concept`           | Only conceptual background entries. |
| `reference`         | Only reference entries.             |

The agent, or the person configuring it, picks the value that matches what the agent is actually for.

> The documentation taxonomy becomes a selection interface for machines.

That matters because the taxonomy does double duty rather than existing twice:

* Humans use `contentType` to navigate, to tell a how-to apart from background reading before they start clicking.
* Machines use the same field to constrain retrieval before they start fetching.

There's no separate machine-facing taxonomy sitting next to the human-facing one. It's one set of values, read by two kinds of readers.

## A guide from a slice of the corpus

This also means filtering scales up from one document to a whole guide.

Imagine MercuryFlow has a 500-page documentation set and someone asks:

> Create a guide containing only installation procedures.

Against unstructured HTML, that means opening a large fraction of the site to work out which pages qualify. Against the index, it's one filtered query, `pageType=topic` combined with `contentType=task`, narrowed further by title or keyword matching if the installation procedures aren't already grouped together.

The result is a list of Markdown URLs. A consuming agent could then fetch those URLs and assemble them into a new document, without ever needing to understand how the site's navigation or URL structure works.

## The same interface, from inside an editor

Nothing in this workflow is specific to a chatbot or an agent framework either.

Imagine Maya is working in an IDE plugin that understands MercuryFlow documentation. When she asks:

> How do I use this feature?

the plugin can:

1. Query the documentation index.
2. Retrieve the matching Markdown.
3. Display it inline next to the code she is writing.

It's the same HTTP, JSON, and Markdown interface as the coding agent and the chatbot above, just called from a different application.

## One corpus, many consumers

Put Maya's coding agent, the MercuryFlow support chatbot, the RAG pipeline, and the IDE plugin side by side.

None of them depend on which product is asking. They all speak the same discovery contract and read the same Markdown documents.

The point isn't the list of possible consumers. It's that they can all consume the same documentation corpus instead of requiring separate AI-specific documentation.

## A discovery contract instead of a crawler

A conventional crawler's question is: which links should I follow?

`robots.txt` can tell it which paths it's allowed to visit, but not what those paths contain or how to ask for a subset of them. It still has to open pages and infer structure from whatever HTML happens to be there.

A documentation site with a discovery contract answers a different question up front:

1. Here is what this documentation provides.
2. Here is how it's classified.
3. Here is where each document can be retrieved.

The crawler's inference step disappears, because the answer was already published as data.

Still, that doesn't make the site an API in the formal sense. There's no authentication, no write operations, nothing beyond static files served over HTTP. What it does do is make the site closer to a documentation discovery interface than a plain collection of HTML pages.

## The architecture underneath

All of the use cases above run on the same build output.

| Layer            | Role                                                                                       |
| ---------------- | ------------------------------------------------------------------------------------------ |
| Markdown         | Holds the content.                                                                         |
| YAML frontmatter | Holds the metadata, including the classification that makes filtering possible.            |
| `schema.json`    | The discovery contract built from that metadata.                                           |
| `index.json`     | The searchable, filterable listing of the corpus it describes.                             |
| HTML             | The presentation layer, built from the same source for the humans reading it in a browser. |

Retrofitting that frontmatter onto an existing corpus doesn't touch a sentence of prose, because the classification is metadata, not content.

> You don't need to build a second documentation system for AI. You can make the existing documentation directly discoverable and consumable by machines.

The same source corpus serves both audiences at once. Nothing about it required writing the documentation twice.

Maya never had to know any of this to get her retry procedure. That's the point.

### Related reading

* [Strong information typing without the XML overhead](https://redaction-technique.org/strong-information-typing-without-xml-overhead): how a lightweight `contentType` axis buys most of what a heavier content model promises.
* [One YAML file, three outputs: API docs, web, and mobile](https://redaction-technique.org/experimental-astro-api-docs): the earlier YAML-first experiment this manual's `/schema.json` approach grew out of.
* [Transforming a corpus of 7,000 pages into living knowledge](https://redaction-technique.org/transforming-corpus-ai-living-knowledge): structuring a large existing corpus for AI-driven retrieval instead of rebuilding it from scratch.

### External sources

* [llms.txt](https://llmstxt.org/): the plain-text convention `/llms.txt` follows.
* [robots.txt](https://www.robotstxt.org/robotstxt.html): the traditional crawler directive protocol, permissions without content classification.

<small>*Hero image: ["Audio plugs"](https://www.flickr.com/photos/jepoirrier/2040189276) by [Jean-Etienne Minh-Duy Poirrier](https://www.flickr.com/photos/jepoirrier/), licensed under [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/).*</small>

---

Source: https://redaction-technique.org/the-manual-is-the-api