Dev.to WebDev 🛠 Dev 👁 0 📖 2 min read

Engineering a 1,000-Page Batch Pipeline for CBZ, EPUB, and PDF Manga Archives

Building a single-image OCR demo takes a weekend. Things start breaking the moment a real user drops a 250-page .cbz volume—or a 900 MB .epub bundle—into your upload zone. While scaling our manga translator to handle fu

Building a single-image OCR demo takes a weekend. Things start breaking the moment a real user drops a 250-page .cbz volume—or a 900 MB .epub bundle—into your upload zone.

While scaling our manga translator to handle full-volume batches up to 1,000 pages (or 1 GB per upload), we ran straight into the limits of traditional server-side file handling. Pushing raw archives to backend routes spiked memory usage, timed out serverless functions, and burned GPU inference on blank divider pages. Moving archive extraction, natural sorting, and page filtering into the browser solved all three bottlenecks.

Here is how we architected the client-to-edge batch pipeline for multi-format comic archives.

1. The Lexicographical Sorting Trap (page_10.jpg Before page_2.jpg)

Comic archives (.cbz, .cbr, .zip, .rar) are just compressed folders of images, and filenames in the wild are rarely zero-padded. When a volume contains page_1.jpg through page_200.jpg, a naive files.sort() orders them lexicographically: page_1.jpg, page_10.jpg, page_100.jpg, page_2.jpg. In a narrative medium, scrambling page order makes the output unreadable.

To preserve exact reading flow without brittle regex hacks, we run every extracted entry list through Intl.Collator(undefined, { numeric: true, sensitivity: 'base' }) before assigning page indices. We also filter out hidden OS metadata artifacts like __MACOSX/ and .DS_Store during the same pass so ghost files never enter the render queue.

2. Unpacking 1 GB Archives Without Blowing Up Server RAM

Accepting a 1 GB archive on a Next.js API route or Edge worker is an anti-pattern—decompressing hundreds of high-resolution scans in server memory quickly triggers OOM crashes.

Instead, we unpack archives locally on the client:

  • ZIP / CBZ & EPUB: Parsed directly in the browser using streaming zip readers, extracting image blobs on demand without holding the entire uncompressed archive in memory at once.
  • PDF & RAR / CBR: Decoded via WebAssembly workers (PDF.js and WASM unarchivers) off the main thread so the UI stays responsive while rendering high-DPI page canvases.

Once unpacked into lightweight client-side page objects, the browser dispatches uploads concurrently with per-page retry logic. If a user's connection blips on page 142, only that single image retries rather than restarting a 900 MB upload.

3. Pre-Flight Page Filtering to Cut Wasted Inference

A printed tankōbon always includes front and back covers, blank spacer pages, tables of contents, copyright colophons, and publisher ads—often 10% to 15% of the total file. Running OCR, LLM translation, and screentone inpainting on blank pages wastes both user credits and backend GPU time.

Before a batch starts, we render a fast thumbnail grid directly from the local blobs. Users can uncheck non-dialogue pages in a couple of clicks, ensuring that only actual story panels hit the inference pipeline.

4. Multi-Volume Queues and Ephemeral Storage

Hardcore readers rarely translate one chapter at a time; they queue up three or four volumes before a weekend flight. Rather than merging multiple archives into one giant fragile job, the client orchestrates each archive as an isolated batch task in a sequential queue—so a malformed file in Volume 3 never blocks Volume 1 or Volume 2 from completing.

Finally, because personal digital libraries shouldn't linger on cloud storage indefinitely, every uploaded page and rendered output travels over encrypted channels and enters a strict TTL lifecycle: an automated cleanup worker purges all session assets from object storage after 10 days.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.