Dev.to AI 🤖 Ai Thu, 05 Nov 2026 05:41:45 +0000 👁 0 📖 7 min read

Turn any website into clean Markdown chunks for RAG

Most RAG quality problems start before the embedding model. If you feed it raw HTML, or text extracted with navigation, cookie banners and footers still attached, every chunk carries the same boilerplate. Retrieval then

This post covers the two steps that fix most of it: converting each page to clean Markdown of its main content, and chunking that Markdown along its headings so each chunk is a coherent section with a citation.

Disclosure: I'm Aurelix, and I built the Website to Markdown Actor used for the crawl step. It's a paid tool (prices in the Cost section). The chunking code works on Markdown from any source.

The general approach

Crawl within a scope. Usually one docs section (/docs/...), not the whole domain. Exclude changelogs, tag pages and paginated archives, which add volume and little meaning.
Keep the main content only. Use the page's <article> or <main> element when present, otherwise the largest content block. Strip nav, headers, footers, sidebars, cookie banners and scripts.
Convert to Markdown, not plain text. Headings, lists, tables and code blocks carry meaning. Markdown keeps them; plain text flattens them.
Chunk by structure first, then by size. Split at headings so a chunk is a section, then split oversized sections at paragraph boundaries. Never split inside a code block.
Carry metadata on every chunk: source URL, page title and the heading path (Page > Section > Subsection). It improves retrieval and lets the model cite its source.
Be polite. Respect robots.txt and crawl slowly. It's someone else's server.

Step 1: Crawl to Markdown

The Website to Markdown Actor takes:

startUrls (required): pages to start from; links are followed within the scope.
maxPages (default 50, up to 10,000): stop after this many pages are converted.
scope: "path" (only URLs under the start URL's folder) or "domain" (the whole site of each start URL).
includeGlobs / excludeGlobs: URL patterns, where ** matches anything and * anything except /.

Each page becomes one record:

{
  "url": "https://docs.apify.com/academy",
  "title": "Apify Academy | Academy | Apify Documentation",
  "description": "Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.",
  "language": "en",
  "last_modified": "2026-11-04T07:17:58Z",
  "markdown": "# Apify Academy\n\nLearn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.\n\n## Beginner courses\n\n...",
  "headings": [
    {"level": 1, "text": "Apify Academy"},
    {"level": 2, "text": "Beginner courses"}
  ],
  "links": [
    "https://docs.apify.com/academy/scraping-with-ai",
    "https://docs.apify.com/academy/scraping-basics-javascript"
  ],
  "word_count": 231,
  "crawl_depth": 0,
  "http_status": 200,
  "source_url": "https://docs.apify.com/academy",
  "scraped_at": "2026-11-04T11:25:08Z"
}

Links inside the Markdown are made absolute, and code blocks keep their language tag, which helps the chunker below.

Step 2: Chunk by headings

pip install apify-client

import json
import os
import re

from apify_client import ApifyClient

MAX_CHARS = 2000
MIN_CHARS = 200
HEADING = re.compile(r"^(#{1,6})\s+(.*)$")
FENCE = re.compile(r"^\s*(`{3,}|~{3,})")


def crawl(start_url, max_pages=100):
    client = ApifyClient(os.environ["APIFY_TOKEN"])
    run = client.actor("aurelix/website-to-markdown").call(
        run_input={
            "startUrls": [start_url],
            "maxPages": max_pages,
            "scope": "path",
            "excludeGlobs": ["**/changelog/**"],
        }
    )
    return list(client.dataset(run["defaultDatasetId"]).iterate_items())


def split_sections(markdown):
    """Yield (heading_path, text) per section; ignores '#' inside code fences."""
    path, lines, in_code = [], [], False
    for line in markdown.splitlines():
        if FENCE.match(line):
            in_code = not in_code
        match = None if in_code else HEADING.match(line)
        if match:
            if any(l.strip() for l in lines):
                yield list(path), "\n".join(lines).strip()
            level = len(match.group(1))
            path = path[: level - 1] + [match.group(2).strip()]
            lines = [line]
        else:
            lines.append(line)
    if any(l.strip() for l in lines):
        yield list(path), "\n".join(lines).strip()


def split_blocks(text):
    """Paragraph blocks, keeping each code fence whole."""
    blocks, current, in_code = [], [], False
    for line in text.splitlines():
        if FENCE.match(line):
            in_code = not in_code
        if not line.strip() and not in_code:
            if current:
                blocks.append("\n".join(current))
                current = []
        else:
            current.append(line)
    if current:
        blocks.append("\n".join(current))
    return blocks


def pack(text):
    if len(text) <= MAX_CHARS:
        return [text]
    chunks, current = [], ""
    for block in split_blocks(text):
        if current and len(current) + len(block) + 2 > MAX_CHARS:
            chunks.append(current)
            current = block
        else:
            current = f"{current}\n\n{block}" if current else block
    if current:
        chunks.append(current)
    return chunks


def chunk_page(page):
    chunks = []
    for path, text in split_sections(page.get("markdown") or ""):
        for part in pack(text):
            if chunks and len(part) < MIN_CHARS and chunks[-1]["heading_path"] == path:
                chunks[-1]["text"] += "\n\n" + part
                continue
            chunks.append({"heading_path": path, "text": part})
    for i, c in enumerate(chunks):
        c.update({
            "id": f"{page['url']}#{i}",
            "url": page["url"],
            "title": page.get("title"),
            "language": page.get("language"),
            "last_modified": page.get("last_modified"),
            "embed_text": " > ".join(c["heading_path"]) + "\n\n" + c["text"],
        })
    return chunks


def main():
    pages = crawl("https://docs.apify.com/academy", max_pages=100)
    total = 0
    with open("chunks.jsonl", "w", encoding="utf-8") as f:
        for page in pages:
            for chunk in chunk_page(page):
                f.write(json.dumps(chunk, ensure_ascii=False) + "\n")
                total += 1
    print(f"{len(pages)} pages -> {total} chunks in chunks.jsonl")


if __name__ == "__main__":
    main()

Why these choices

Heading path in the embedded text. A chunk that reads "Set the timeout option to 60" means little alone. Prefixing it with Crawler configuration > Timeouts gives the embedding model the context a human gets from the page layout. Store the bare text separately for display.
Characters, not tokens. 2,000 characters is roughly 400 to 500 tokens of English prose, a common range for retrieval chunks. If your embedding model has a hard token limit, swap len() for a tokenizer count; the logic doesn't change.
Code fences are never split. Half a code example is worse than none. A single code block longer than MAX_CHARS stays one oversized chunk, which is usually the right trade.
Tiny trailing chunks get merged into the previous chunk of the same section, so you don't embed one-line orphans.
Stable IDs (url#index) let you re-crawl and upsert. Pair them with last_modified to re-embed only pages that changed.

From chunks.jsonl, loading into a vector store is one loop: embed embed_text, store text, url, title and heading_path as metadata, and return url with each answer as the citation.

Getting the crawl scope right

Start at the section you need: https://example.com/docs/ with scope: "path" crawls only /docs/....
Versioned docs: exclude old versions with excludeGlobs: ["**/v1/**"], or the same answer appears three times with three different version numbers.
Paginated listings: excludeGlobs: ["**?page=*"] skips archive pages that are lists of links.
Check word_count after a test run of 20 pages. Pages with very few words are often index pages; filter them out before chunking if they add nothing.

Limits

No JavaScript rendering. It reads the HTML the server sends, which covers most docs sites, blogs, knowledge bases and server-rendered React or Next.js. Client-side-only pages (an empty app shell) are skipped, listed in the log and not charged.
HTML only. PDFs, images and other files are skipped.
One site per start URL. www. and the bare domain count as one site; other subdomains aren't followed.
Speed. At most two new requests per second per site, slower when robots.txt sets a Crawl-delay. Conversion is CPU-bound: the README measures about 1,000 large documentation pages an hour at 256 MB, so the default one-hour run fits roughly 1,000 to 3,000 pages. The run stops itself before its timeout and keeps the pages done so far.
Pages over 2 MB of HTML are skipped and not charged.
Main-content detection is automatic, so on unusual layouts a little boilerplate can remain or a side block can be dropped. Spot-check a few pages.
Robots.txt is always obeyed and can't be turned off. Pages behind logins, paywalls or bot challenges aren't accessible.
Crawl only sites you have the right to use this way. Copyright in the content stays with its owner.

Why this tool over the alternatives

Compared with the leading alternatives on the Apify Store (the full table is on each Store page):

A flat $1.50 per 1,000 pages: 34% under Apify's RAG Web Browser, and no compute-unit billing at 8 GB like Apify's Website Content Crawler.
Crawls a whole site at 256 MB, robots.txt always obeyed, failed, duplicate and empty pages never charged.
Not offered: JavaScript rendering, which both Apify Actors have.

Cost

At the time of writing, the README lists $1.50 per 1,000 pages on Apify's free plan and $1.05 per 1,000 on paid plans (Bronze, Silver, Gold), plus $0.001 per run. Skipped pages (robots.txt, errors, non-HTML, duplicate URLs or identical Markdown, JavaScript-only pages, pages with no main content) aren't charged.

The 100-page example costs at most $0.15. Apify's free plan includes $5 of credit a month, which the README puts at about 3,300 pages.