Knowhere hit 4,000 GitHub stars! Here is how we built progressive PDF ingestion

Hey DEV! 👋

Our open-source document memory project, Knowhere, just crossed 4,000 stars on GitHub! 🎉

When we started building Knowhere to wrangle messy PDFs, complex tables, and engineering blueprints for AI agents, we weren't sure how many other devs were fighting the exact same battles. Hitting this milestone means the world to our team—thank you to everyone who has starred the repo, opened issues, or submitted PRs!

To celebrate, I want to share a look under the hood at our latest architectural update—Progressive Ingestion—and show you what happened when we stress-tested the pipeline on 11,945 ML conference papers.

The Problem: Traditional PDF Parsing Works Like Boiling Water 🫖

Until recently, document ingestion worked like boiling a pot of water: until it hits 100°C, you just sit there and wait.

When you pass a PDF into a standard RAG or parsing pipeline, a ton of heavy lifting happens under the hood:

• Text extraction & layout recognition
• Table and figure parsing
• Section tree reconstruction
• Summarization & vector indexing

Historically, a document isn't queryable until every single one of those steps finishes.

For a 5-page PDF? Waiting a few seconds is fine.

For a 500-page technical manual, a financial report, or a dense bundle of academic papers? The UX completely falls apart.

Even if the first 200 pages are already parsed, the entire file stays locked in a "processing" state as long as a few complex pages are still sitting in the queue. If a user—or an AI agent—just wants to look up a single part number on Page 12, they are stuck waiting for Page 480 to finish.

For AI agents, this all-or-nothing blocking is a dealbreaker. Agents work iteratively—they search, read, and gather context on the fly. Forcing an agent to wait until a document is 100% pre-processed kills real-time responsiveness before the workflow even starts.

How We Fixed It: Progressive Ingestion (ingestion_mode=progressive) ⚡

To kill this latency bottleneck, we built Progressive Ingestion into Knowhere.

When you set ingestion_mode=progressive, Knowhere stops holding your documents hostage until the entire pipeline finishes. Instead, it streams context to your agent in three distinct stages:

Stage 1: Instant Baseline (Readable & Searchable in Seconds)

Knowhere immediately publishes a baseline version of the document so your agent can open, read, and run searches within seconds of upload.

Stage 2: Structural Hierarchy (TOC & Section Trees)

Next, Knowhere reconstructs outlines and section hierarchies, turning a flat pile of pages into a structured document tree that agents can navigate via a Table of Contents.

Stage 3: On-Demand Background Enrichment

Deep enrichment and heavy parsing continue asynchronously in the background. Crucially, fine-grained nodes are prioritized on-demand whenever an agent actively queries those specific sections.

Think of it like dining at a restaurant: instead of starving at the table until every dish for the entire party is cooked, appetizers hit the table immediately while the main courses arrive as they’re ready.

We aren't cutting corners or dropping parsing steps—we're just slashing time-to-first-use while finishing the heavy lifting in the background.

💡 When should you use which mode?
Progressive mode is built for latency-sensitive apps (live chat-with-PDF, interactive agents).
Full Ingestion remains the default for batch workloads where you only want documents exposed after 100% of processing is done.

Other Cool Upgrades in This Release 🛠️

• MinerU is now optional: Disabled by default so you have a lighter dependency footprint, though you can still toggle it on for fine-grained extraction.
• More image formats: Native support for GIF, WebP, BMP, TIFF, and HEIC.
• Public Navigation APIs: External agents can now call Knowhere's internal document navigation tools directly via MCP or REST API.

How This Fits Into the Knowhere 2.0 Architecture 🧩

Back in Knowhere 2.0, we introduced two core pillars for complex documents:

VISION-MAP (Visual Parsing): Visually dense files (scanned PDFs, slide decks, engineering blueprints) are routed directly to vision models as full pages to preserve layout fidelity. Clean files (Word, Excel) go through native text parsing. Both converge into one unified document memory.

Agent-Native Retrieval: Instead of rigid, one-shot chunk retrieval, the agent inspects the document map, calls navigation tools, and plans its own multi-step retrieval route.

Progressive Ingestion completes this puzzle:

• Ask while uploading: Drop a 400-page manual into chat and start querying immediately.
• Zero-blocking workflows: Add new reference docs mid-task without stalling the agent.
• No single-page bottlenecks: One diagram-heavy page won't hold the rest of the file hostage.

Stress-Testing on 11,945 ML Conference Papers (2025) 🧪

You can't benchmark a document engine by running demos on three clean PDFs. We wanted a stress test with massive scale, gnarly layouts, and real utility for the AI community.

Academic papers checked every box: double-column text scrambles reading order, equations and tables float across pages, and a casual "see Appendix C" might point to a table 15 pages away.

So we ran all 11,945 accepted papers from ACL, CVPR, ICML, and NeurIPS 2025 through Knowhere.

Instead of dumping flat text into an LLM for a generic summary, our pipeline worked structurally:

1. Parsed each paper into chunks tagged with exact section paths.

2. Reconstructed the full section hierarchy tree (Methodology, Experiments, Appendix, etc.).

3. Extracted 28 structured fields per paper (research problem, core innovations, baselines, experimental results, etc.) while keeping the structural context intact.

4. Packaged the structured fields, grounded source snippets, and section trees into a single verifiable dataset.

Why does this matter? Because claims stay verifiable. If a record says "Method X beats the baseline by 5%," a Research Agent can trace the section path straight back to the exact paragraph to verify the dataset, experimental setup, and ablation tables.

(Disclaimer: Since this dataset was generated via an automated pipeline without manual paper-by-paper proofreading, definitely double-check the original PDF before citing numbers in formal papers!)

Grab the Open-Source Datasets on Hugging Face 🤗

We’ve open-sourced the full 2025 dataset (plus 2023, 2024, and Embodied AI collections) on Hugging Face:

2025 Top ML Papers (11,945 papers): huggingface.co/datasets/JensCS/top-ml-conference-papers-2025

2024 Edition: huggingface.co/datasets/JensCS/top-ml-conference-papers-2024

2023 Edition: huggingface.co/datasets/JensCS/top-ml-conference-papers-2023

Embodied AI Papers (2023–2025): huggingface.co/datasets/JensCS/embodied-ai-conference-papers-2023-2025

Check Out Knowhere & Let Us Know What You Think! 🚀

We're on a mission to get messy, real-world documents into agent workflows without the friction. If you're building RAG pipelines, enterprise knowledge bases, or Research Agents, give Knowhere a spin:

Website: knowhereto.ai

I'd love to hear from you in the comments: What’s the gnarliest document format or latency bottleneck you've run into while building RAG or agent pipelines? Let's geek out below! 👇