Keep a RAG index in sync with a docs site by re-embedding only the chunks that changed

A support bot or internal assistant built on a documentation site goes stale the week after launch. The docs change, the vector index doesn't, and the bot keeps quoting the old pricing page. The usual fix is a nightly full re-crawl and re-embed, which pays to embed thousands of unchanged chunks to pick up a handful of edits, and still leaves deleted pages in the index unless you diff the URL lists yourself.

What the index actually needs from each sync is three lists: chunks to add, chunks to replace, chunk IDs to delete. This post gets exactly that from a scheduled run and applies it to a vector store in Python.

Disclosure: I'm Aurelix, and I built the Website Change Tracker. It's a paid tool (prices below).

How it decides what changed

You give it start URLs (or sitemap URLs) and a corpusId. The first run with that ID is the baseline: every page comes back as added, split into Markdown chunks. Every later run re-checks the site and returns only pages that were added, changed or deleted.

• Chunk IDs are anchored on content: the page's canonical URL, the heading path and a hash of the normalised text, as valid UUIDs. Chunk boundaries are cut at content-defined points, so editing one paragraph re-cuts only the chunk it sits in, and an unchanged section keeps its ID when the sections around it change.
• Boilerplate is not a change: navigation, footers, cookie banners and lines like "Last updated" or "x min read" are dropped before comparing.
• A broken page is not a deletion: a page is deleted only on a 404/410 or a redirect away. A known page that errors, shows a bot challenge, comes back as an empty JavaScript shell or shrinks by more than 90% is returned as extraction_failed (free), and its previous version is kept.

Each change record carries change, doc_id, url, title, upsert_chunks (each with chunk_id, heading_path, text and a ready citation), delete_chunk_ids, and counts of upserted, deleted and unchanged chunks.

A nightly sync into a vector store

This uses Chroma, which embeds documents itself; the same three steps work with Qdrant, Pinecone, Weaviate or pgvector, since the chunk IDs are UUIDs.

pip install apify-client chromadb

import os

import chromadb
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("aurelix/website-change-monitor").call(
    run_input={
        "startUrls": ["https://docs.apify.com/academy"],
        "corpusId": "apify-academy",
        "maxPages": 500,
        "chunkSize": 1500,
    }
)
changes = list(client.dataset(run["defaultDatasetId"]).iterate_items())

index = chromadb.PersistentClient(path="index").get_or_create_collection("docs")
upserts, deletes = [], []
for c in changes:
    if c["change"] == "extraction_failed":
        print(f"skipped (kept previous version): {c['url']} ({c['reason']})")
        continue
    upserts += c["upsert_chunks"] or []
    deletes += c["delete_chunk_ids"] or []

if deletes:
    index.delete(ids=deletes)
for i in range(0, len(upserts), 100):
    batch = upserts[i:i + 100]
    index.upsert(
        ids=[ch["chunk_id"] for ch in batch],
        documents=[ch["text"] for ch in batch],
        metadatas=[{"url": ch["url"], "citation": ch["citation"]} for ch in batch],
    )
print(f"{len(changes)} pages changed: {len(upserts)} chunks upserted, {len(deletes)} deleted")

Run it once to fill the index, then schedule the same input daily (save it as a task in the Apify Console and click Schedule, then run the script after it, or call it from your own scheduler). A quiet day returns nothing and embeds nothing.

A few input choices that matter:

• Already loaded the site another way? Set "returnBaseline": false and the first run records the corpus silently, so you pay per page checked, not per page returned. The Website to Markdown API is a good way to do that first full load.
• Help centres with orphan pages: "scope": "domain" with "useSitemaps": true tracks pages no menu links to; excludeGlobs such as ["**/search**", "**/tags/**"] keeps out listings that change on every visit.
• No polling: set webhookUrl to an n8n, Make or Zapier webhook and the delta is POSTed in batches of 25 records; the last batch has final: true and the run's counts.
• Chunk size is fixed per corpus so IDs stay stable. To re-chunk, use a new corpusId, which starts a new baseline.

Why this tool

• Built for a machine reading the output: Markdown chunks to upsert and chunk IDs to delete.
• Chunk IDs are anchored to the content, so they stay put when nothing in that section changed.
• Blocked or empty pages are never reported as deleted.
• The delta, chunks and delete IDs, can be pushed to a webhook.
• What it doesn't cover: severity scores or diffs written for a person to read.

Limits

• No JavaScript rendering. Pages that are an empty app shell are skipped, and flagged if they were already in the corpus.
• HTML pages only. PDFs, images and other files are skipped; HTML pages over 2 MB too.
• maxPages caps pages checked per run (up to 20,000). Known pages not reached in a run are kept, not deleted; set it above the size of the site.
• Polite pace: at most two new requests a second per site, about 1,000 large documentation pages an hour. robots.txt is always obeyed.
• rel=canonical is followed within the same site and scope, so a site whose canonical tags all point to one page collapses to that page.
• No login, paywalled or bot-challenged pages.

Cost

Two parts: a small charge per page checked, and a larger one per changed page returned. A run with no changes costs only the checks and the start fee. Extraction failures, robots.txt blocks, fetch errors and duplicates are free. The current prices are on the Store page; Apify's free plan includes $5 of credit a month, enough to check a 250-page docs site every day for a month when only a few pages change.

Links

• Website Change Tracker on the Apify Store