Convert documentation pages to Markdown for RAG

Documentation websites often repeat navigation, version pickers and footer links on every page. Extracting the main content makes each page easier to inspect before it enters a retrieval index. It also makes failures visible before they become missing answers.

Build a list of public documentation URLs for the product version you support. Convert one URL at a time with the URL to Markdown converter, the convert_url MCP tool, or the extraction API. GetMarkdown extracts the requested page; it does not crawl a whole documentation site or follow every link automatically.

GET /api/v1/extract?url=... returns JSON by default. The response contains Markdown, the final URL after redirects, available title and author metadata, word count, cache status and usage. Send Accept: text/markdown when you need only the Markdown body. POST /api/v1/extract accepts a JSON object with a url field.

For a retrieval pipeline, store the final source URL, documentation version and your retrieval timestamp alongside each page. Preserve the YAML frontmatter in the Markdown as another source record. If the page redirects to a newer version, review it before adding it to an older collection.

A documentation page can contain the right prose while losing a code tab, table column or warning. Compare representative output with the source before indexing the collection. Choose pages with command examples, API parameters and nested headings for this check.

Chunk around section headings and keep each example with the explanation and version caveats it needs. Attach the page URL and heading to each chunk. A retrieved code sample is less useful when separated from the condition that makes it valid.

Anonymous extraction allows 20 public pages per IP address per UTC day, then returns HTTP 429 with free_limit_reached. Use Retry-After to determine when to retry. Plan the page list within that allowance and consult developer resources for authenticated usage and current API key availability.

URL extraction happens on our server and results are cached for up to an hour. A repeat request within that window may return an older extraction even if the publisher has just edited the page. Track retrieval dates in your index and replace old page records when you refresh the collection.

Some documentation sites render their content with JavaScript. If the public URL returns an empty shell, open the page in a browser, wait for the content to appear, and save it as HTML. Upload the .html or .htm file through the homepage file converter, then verify that the needed examples were included in the saved file.

HTML file conversion stays local in your browser and has no daily file quota, up to 10 MB per file and eight files per batch. It still cannot recover content missing from the saved HTML. Review permission and access requirements for the documentation you collect.

Test questions that require a definition, a parameter default, an example and a version-specific behavior. Check whether retrieval finds the source section and whether the generated answer cites that page. Include questions that the collection cannot answer so missing coverage is visible.

Keep a mapping from each source URL to its indexed chunks. When a page changes or is removed, that mapping lets you replace its entries instead of mixing outdated examples with current documentation.