Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 3 min read

Taming a 14.6 GB RAM Cliff and 4 GB GPU VRAM Limits in a Local-First AI Desktop App (Tauri v2 + Node.js + PyTorch)

Whenever I tried using cloud LLMs to get through 500-page technical textbooks or multi-thousand-chapter web serials, I ran into the same two failure modes: Lossy summarization flattens nuance: Standard AI summaries st

Whenever I tried using cloud LLMs to get through 500-page technical textbooks or multi-thousand-chapter web serials, I ran into the same two failure modes:

  1. Lossy summarization flattens nuance: Standard AI summaries strip out equations, drop edge cases, and hallucinate connective tissue.
  2. Zero source traceability: When a summary makes a surprising claim, there is no deterministic link back to the exact paragraph in the original PDF or EPUB.

Over the past 9 months, I built Omnitomeβ€”an open-source (AGPL-3.0), local-first Windows desktop application that performs loss-bounded semantic compression (with a documented target of ~7:1) where every synthesized sentence carries a clickable citation back to the immutable source paragraph.

πŸŽ₯ 3-Minute Architecture & UI Walkthrough: https://youtu.be/Nmb-qiZpnrk

Here is how the multi-process architecture works under the hood, and the four hardest systems bugs I had to solve to make the entire pipeline run offline on a 4 GB NVIDIA RTX 3050 laptop GPU.

1. Multi-Process Isolation & The Sole SQLite Writer Law

Packaging a modern React UI, an asynchronous orchestration layer, and heavy CUDA PyTorch models into a single desktop app without freezing the UI required strict process boundaries:

  • Tauri v2 Desktop Shell (Rust): Native window supervision, OS lifecycle management, and child process teardown.
  • Node.js 22 LTS Orchestrator (better-sqlite3): Pipeline coordinator and sole writer to data.db in SQLite WAL mode.
  • Python FastAPI Sidecar (PyTorch FP16 CUDA): Runs BGE-M3 (1024-dim embeddings), BGE Reranker v2, and Kokoro-82M neural TTS.

Why Node.js Is the Sole SQLite Writer

While SQLite in WAL (Write-Ahead Logging) mode supports concurrent readers, allowing Rust, Node.js, and Python processes to write directly to data.db triggers SQLITE_BUSY lock contention during heavy ingestion. In Omnitome, Python sidecars and the React frontend have zero direct database write accessβ€”every mutation flows strictly through Node.js REST boundaries.

2. Crushing the 14.6 GB Node.js RAM Cliff (F65 Sequential Await Barrier)

In early builds, ingesting a 10-book, ~508,000-word multi-format test corpus caused Node.js heap memory to spike to 14.6 GB, crashing consumer laptops.

  • Root Cause: Unbounded Promise.all concurrency queued hundreds of document chunks, embedding payloads, and DOM parsing buffers in memory simultaneously while waiting on the GPU sidecar.
  • The Fix (F65 Sequential Await Barrier): I refactored the 5-stage ingestion pipeline (Parse -> Vectorize -> Classify -> Plan -> Compress & Attribute) behind a strict chunk-by-chunk asynchronous await gate with explicit buffer release.
  • Verified Result: Peak Node.js RSS dropped by 98%β€”from 14.6 GB down to 296 MBβ€”processing all 508,000 words in 5.9 minutes.

3. Running 3 Neural Models on a 4 GB GPU (LifecycleManager VRAM Eviction)

Omnitome's local neural stack runs three models:

  • BGE-M3 (FP16): ~2.1 GB VRAM
  • BGE Reranker v2: ~2.2 GB VRAM
  • Kokoro-82M Neural TTS: ~0.6 GB VRAM

Keeping all three resident requires ~4.9 GB of VRAM, immediately crashing a 4 GB RTX 3050 laptop GPU.

To solve this without tearing down the Python sidecar process between tasks, I implemented an asynchronous LifecycleManager state machine (UNLOADED -> READY -> EVICTED) with a sliding-window TTL governor:

  • Models load in FP16 precision on demand.
  • When a stage completes or sits idle past its TTL window, the governor evicts the model weights and flushes the CUDA cache, reclaiming 3.2 GB of VRAM.

4. Calibrating BGE-M3 Retrieval & Fixing the CJK Token Explosion

  • CLS Pooling Calibration: Switching BGE-M3 vector extraction to calibrated CLS pooling improved our retrieval-separation metric 1.68x (from 0.2520 to 0.4225) across our 19-suite regression harness.
  • CJK Multilingual Token Safety: When testing unspaced Japanese texts, naive whitespace word counters treated entire paragraphs as a single "word," bypassing chunk guards and causing context-window overflows. Adding script-aware regex weighting (~0.7 words per CJK glyph) and sentence-boundary splitting (γ€‚οΌοΌŸ\n) eliminated the overflow completely.

Try Omnitome v1.1.1 (Open Source)

Omnitome v1.1.1 is live today as a single 1.95 GB Windows x64 installer (bundling the Python + CUDA runtime so no local Python setup is needed):

If you're building local-first AI tools with Tauri, Rust, or PyTorch on constrained hardware, I'd love to hear how you handle sidecar supervision and VRAM residency!

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.