One Engine, Seven Open-Source Projects: PageSift 1.4 and the yolow Family

PageSift started as the ingestion layer of an AI research platform I was building: something had to fetch web pages and documents and turn them into clean Markdown that an LLM pipeline could actually use. It grew into a self-hosted web scraper, crawler and document parser — Firecrawl- and Jina-compatible API, MCP server, Python/Node SDKs, one Rust binary.

Over the 1.x series I kept doing one thing: every time a piece of the engine turned out to be generally useful, I cut it out and gave it its own repository, its own versioning, and an MIT license. PageSift 1.4 finishes that split — and adds the subsystem the extraction was making room for.

PageSift 1.4

The split, completed. Wave one (1.2.7) turned the base64, PDF, HTML and content crates into thin shims over standalone libraries. Wave two (1.4.0) does the same for the remaining three: pagesift-office → yolowdoc, pagesift-syntax → yolowsyntax, pagesift-compress → yolowzstd. The engine is now a composition of seven independently versioned open-source projects — and since 1.3, PageSift itself is Apache-2.0 (it was AGPL before).

DataGraph. The new crate, pagesift-graph (~4,000 lines), is what the cleanup was for. A normal scraper returns text and forgets everything else. DataGraph keeps the receipts:

• Every fetch becomes a node; links between documents become citations.
• Provenance is recorded as edges: redirect chains, canonical alias clusters, attachments and their parent documents.
• Refetch the same resource and you get version lineage instead of an overwrite.
• Extracted fields carry evidence spans — byte offsets into the source document, so any claim can be pointed back at the exact text it came from.
• Claims about the same entity (keyed by subject keys like sku) are clustered into facts across sources, and when sources disagree you get a conflict record: both variants, both excerpts, and a preferred variant with its reason. Two pages pricing the same product at $10 and $12 stop being a silent data-quality bug and become a queryable fact.

It's exposed everywhere PageSift is: CLI (scrape --graph, crawl --graph, graph inspect), the HTTP API (a graph slice on scrape responses, plus /v2/graph/history and /v2/graph/conflicts), the MCP tool pagesift_graph_query, and the Python and Node SDKs.

An LLM pipeline doesn't just need text. It needs to know where a claim came from, what it contradicts, and which version is current. That's the layer DataGraph adds.

The six libraries

yolow64 0.3.0 — base64, ~20 GB/s

A fast Base64 codec with runtime-dispatched SIMD (AVX2/NEON), zero runtime dependencies, streaming and data-URL APIs. The 0.3 release added dedicated short-input kernels and fused hash-table tail processing: around 20.4 GB/s from 8 KiB to 1 MiB, and short payloads (8–64 bytes) got 1.5–2.8× faster than in 0.2. Short inputs are the common case in real pipelines — tokens, hashes, embedded blobs — so that's where the work went.

cargo add yolow64 · GitHub · crates.io

yolowpdf 0.1.2 — PDF to Markdown, no C libraries

A PDF-to-Markdown library and OCR engine that parses the format itself — no Poppler, no MuPDF, nothing to link against. The workspace includes GPU-accelerated OCR (PP-OCRv6). The README benchmark clocks 1,900 pages/s on born-digital documents (i7-10700). This is the component that taught the family its hardest rule: every allocation a hostile file can influence is budgeted before it happens.

cargo add yolowpdf · GitHub · crates.io

yolowhtml 0.1.0 — arena HTML5 parsing and extraction

Two crates: yolowhtml-dom, a zero-copy HTML5 parser over a contiguous arena (Vec<Node>, u32 ids, CSS selectors), and yolowhtml, a single-pass main-content extractor that produces Markdown, metadata and publication dates from the same parse. On the README's test corpus it scores 1.000 structural F1 for Markdown across 15 page categories, and 0.945 F1 for publication-date extraction across 20 languages. Both crates are #![forbid(unsafe_code)].

cargo add yolowhtml · GitHub · crates.io

yolowzstd 0.1.2 — pure-Rust Zstandard

A Zstandard (RFC 8878) encoder/decoder written from the spec: zero dependencies, #![forbid(unsafe_code)], multi-core encode/decode that emits standard concatenated frames — interoperability is tested against the official zstd CLI. It's built for the small-record end of the world: on 64-byte payloads it decodes at 3,257 MB/s (1.6× C libzstd) and encodes at 264 MB/s (4.1×) on an i7-10700, which is exactly the shape of PageSift's cache records. One stated gap: no dictionary support yet.

cargo add yolowzstd · GitHub · crates.io

yolowsyntax 0.1.2 — tree-sitter, without the C

A pure-Rust runtime for tree-sitter grammars: the C runtime reimplemented in Rust, with 41 languages' parse tables converted and embedded — 6.22 MiB total, Zstd-compressed (with yolowzstd, its only runtime dependency) and decompressed lazily per language. It extracts code outlines for 32 symbol-bearing languages. No C FFI, #![forbid(unsafe_code)]. Stated limits: no Query DSL and no incremental re-parsing.

cargo add yolowsyntax · GitHub · crates.io

yolowdoc 0.1.1 — every document format, one call

Documents to Markdown behind a single convert(bytes, format, opts): Office (DOCX/XLSX/PPTX, plus legacy DOC/XLS/PPT and XLSB), OpenDocument, RTF, WordPerfect, HWP/HWPX, e-books (EPUB/MOBI/FB2), email (EML/MBOX/MSG), archives (ZIP/TAR/7Z/GZ/BZ2/XZ — the decompressors are pure Rust too), SQLite, dBASE, CSV and Jupyter notebooks. The per-format parsers go deep — pivot tables, charts, equations — and a Limits budget is checked before any work starts.

cargo add yolowdoc · GitHub · crates.io

How they fit together

The family is a small lattice, not a pile: PageSift depends on all six libraries; yolowsyntax depends on yolowzstd; yolowdoc builds on both; yolowpdf uses yolow64. The rules are shared across all of them — pure Rust, no C dependencies, unsafe code forbidden in the libraries, and resource budgets enforced before allocation, because every one of these parsers eats untrusted input for a living.

PageSift is the engine. These six are its organs, now living their own lives — each usable without the others, each on crates.io under MIT, with PageSift itself under Apache-2.0.

Links: PageSift · yolow64 · yolowpdf · yolowhtml · yolowzstd · yolowsyntax · yolowdoc

Try them, break them, and tell me where they break — issues and benchmarks from other machines are the most useful feedback I can get.