Turn any web page into LLM-ready Markdown for RAG
Turn any web page into LLM-ready Markdown for RAG Retrieval-augmented generation (RAG) works better when documents are clean prose instead of raw HTML. Navigation bars, scripts, and layout noise waste tokens and confus
Turn any web page into LLM-ready Markdown for RAG
Retrieval-augmented generation (RAG) works better when documents are clean prose instead of raw HTML. Navigation bars, scripts, and layout noise waste tokens and confuse chunkers.
This tutorial shows a practical pattern: fetch a public URL, convert the main content to Markdown, chunk it, and (optionally) embed the chunks. Weโll call the public URL to Markdown (LLM-ready) API on RapidAPI:
https://rapidapi.com/evan-DewNPsuYp/api/url-to-markdown-llm-ready
Youโll need a RapidAPI key. Start on the free plan while you prototype.
Why Markdown?
- Headings become natural chunk boundaries (
#,##, โฆ). - Lists and tables survive in a form LLMs already understand.
- Token counts are closer to what you send the model than raw HTML is.
This API is static HTML only โ it does not run JavaScript. Sites that render all content client-side may return little text. It also rejects oversized HTML (about 200 KB body cap) with HTTP 413. Those limits are fine for many blogs and docs pages; know them before you design a crawler.
1. Subscribe and set the key
export RAPIDAPI_KEY='your-key-here'
Host header (from the public listing): url-to-markdown-llm-ready.p.rapidapi.com
2. Fetch Markdown
curl --request GET \
--url 'https://url-to-markdown-llm-ready.p.rapidapi.com/v1/markdown?url=https%3A%2F%2Fexample.com&include_links=true&include_images=false' \
--header "X-RapidAPI-Key: $RAPIDAPI_KEY" \
--header 'X-RapidAPI-Host: url-to-markdown-llm-ready.p.rapidapi.com'
Successful responses look like:
{
"ok": true,
"title": "Example Domain",
"markdown": "This domain is for use in documentation examples...",
"chars": 156,
"tokensEstimate": 39,
"truncated": false,
"contentSource": "body",
"finalUrl": "https://example.com/"
}
tokensEstimate is a rough chars / 4 heuristic โ useful for budgeting, not a substitute for your modelโs tokenizer.
Useful query flags:
| Param | Default | Meaning |
|---|---|---|
include_links |
true | Keep [text](url) links |
include_images |
false | Emit  when true |
include_tables |
true | Keep GitHub-style tables |
max_chars |
200000 | Soft-truncate long Markdown |
3. Node loader for a RAG pipeline
// markdownLoader.mjs
const HOST = "url-to-markdown-llm-ready.p.rapidapi.com";
export async function urlToMarkdown(targetUrl, opts = {}) {
const endpoint = new URL(`https://${HOST}/v1/markdown`);
endpoint.searchParams.set("url", targetUrl);
endpoint.searchParams.set("include_links", String(opts.includeLinks ?? true));
endpoint.searchParams.set("include_images", String(opts.includeImages ?? false));
endpoint.searchParams.set("include_tables", String(opts.includeTables ?? true));
if (opts.maxChars) endpoint.searchParams.set("max_chars", String(opts.maxChars));
const res = await fetch(endpoint, {
headers: {
"X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
"X-RapidAPI-Host": HOST,
},
});
const data = await res.json();
if (!res.ok || !data.ok) {
const err = new Error(data.error || `HTTP ${res.status}`);
err.code = data.code;
throw err;
}
return data;
}
/** Naive heading-aware splitter โ replace with your preferred chunker. */
export function chunkMarkdown(md, { maxChars = 1200 } = {}) {
const parts = md.split(/\n(?=#{1,6} )/g);
const chunks = [];
let buf = "";
for (const part of parts) {
if ((buf + "\n" + part).length > maxChars && buf) {
chunks.push(buf.trim());
buf = part;
} else {
buf = buf ? buf + "\n" + part : part;
}
}
if (buf.trim()) chunks.push(buf.trim());
return chunks;
}
const page = await urlToMarkdown("https://example.com", { maxChars: 20000 });
const chunks = chunkMarkdown(page.markdown);
console.log(page.title, chunks.length, page.tokensEstimate);
4. Plug into embeddings (sketch)
Pseudocode only โ swap in your embedding provider:
for (const [i, text] of chunks.entries()) {
// const vector = await embed(text);
// await vectorDb.upsert({ id: `${page.finalUrl}#${i}`, text, vector, meta: { title: page.title } });
console.log(`chunk ${i}:`, text.slice(0, 80));
}
At query time, embed the user question, retrieve top-k chunks, and pass them as context to your LLM.
Failure modes worth handling
| Symptom | Likely cause |
|---|---|
HTTP 413 / too_large
|
HTML over ~200 KB โ try a cleaner URL or a different extractor |
| Sparse Markdown | JS-rendered SPA โ need a browser-based tool instead |
ssrf_blocked |
Private/localhost URL โ expected safety behavior |
upstream_error |
Target site 4xx/5xx |
Related
Need a link-preview card (title + image) instead of full Markdown? Use URL Metadata & Open Graph Extractor.
Keep keys on the server, cache by URL, and stay inside your RapidAPI quota while you validate retrieval quality. Once chunk quality looks good on a handful of real pages, wire the same loader into LangChain / LlamaIndex custom document loaders the same way.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.