Automating Invoice Data Extraction with PipeWire and Tesseract on My Home Server

Overview

I’m Kaleb, a web developer based in Pompano Beach, Florida. For Dream Futures Tech I run a small home server that powers a handful of automations for my own business – things like nightly backups, form‑submission processing, and a quick OCR pipeline that turns screenshots of vendor invoices into searchable data. In this article I’ll walk you through the exact steps I used to capture a screen region with PipeWire, feed the image to Tesseract, and store the results in a MySQL table that backs the admin panel of my website. The same pattern can be adapted for any local service business in Palm Beach County that needs to digitize paper‑based paperwork without buying expensive software.

Prerequisites

• Ubuntu 22.04 LTS (or any recent Debian‑based distro) running on a home server (I use a 2022 Intel NUC).
• PipeWire installed and running (the default on recent Ubuntu releases).
• Tesseract 5 with English language data.
• Node.js 20 (LTS) – I use it for the glue code because it integrates nicely with my existing Express API.
• MySQL 8 – the same database that powers my website.

1. Capture a Screen Region with PipeWire

PipeWire replaces the old X11 xwd workflow and works under Wayland out of the box. The pw-record utility ships with the pipewire-tools package.

# Install the tools (if not already present)
sudo apt-get install -y pipewire pipewire-tools

# Record a 5‑second clip of a 800×600 region at (100,200)
# The output is a raw video stream; we’ll pipe it directly to ffmpeg.
pw-record --target 100,200,800,600 --duration 5 - | \
  ffmpeg -f rawvideo -pix_fmt rgb24 -s 800x600 -r 30 -i - -vf "fps=1" -qscale:v 2 -f image2pipe -vcodec png -"

The command above produces a series of PNG frames on stdout. For my use‑case I only need the first frame, so I add -vframes 1:

pw-record --target 100,200,800,600 --duration 1 - | \
  ffmpeg -f rawvideo -pix_fmt rgb24 -s 800x600 -i - -vframes 1 -qscale:v 2 -f image2pipe -vcodec png -

2. Install and Configure Tesseract

sudo apt-get install -y tesseract-ocr libtesseract-dev
# Verify the version
tesseract --version

I keep the OCR step in a separate Node.js module so I can call it from my API or a cron job.

3. Node.js Wrapper for the OCR Pipeline

Create a file called ocr.js:

const { spawn } = require('child_process');
const { pipeline } = require('stream');
const { promisify } = require('util');
const mysql = require('mysql2/promise');

const pipe = promisify(pipeline);

async function captureRegion() {
  // Spawn pw-record and pipe into ffmpeg to get a single PNG buffer
  const pw = spawn('pw-record', ['--target', '100,200,800,600', '--duration', '1', '-']);
  const ff = spawn('ffmpeg', [
    '-f', 'rawvideo', '-pix_fmt', 'rgb24', '-s', '800x600', '-i', 'pipe:0',
    '-vframes', '1', '-f', 'image2pipe', '-vcodec', 'png', 'pipe:1'
  ]);

  // Pipe pw -> ffmpeg and collect the PNG data
  const chunks = [];
  await pipe(pw.stdout, ff.stdin);
  ff.stdout.on('data', d => chunks.push(d));
  await new Promise((res, rej) => {
    ff.on('close', code => code === 0 ? res() : rej(new Error('ffmpeg failed')));
  });
  return Buffer.concat(chunks);
}

async function runTesseract(imageBuffer) {
  return new Promise((resolve, reject) => {
    const tesseract = spawn('tesseract', ['stdin', 'stdout', '--dpi', '300', '-l', 'eng']);
    let out = '';
    tesseract.stdout.on('data', d => out += d.toString());
    tesseract.stderr.on('data', d => console.error('tesseract:', d.toString()));
    tesseract.on('close', code => code === 0 ? resolve(out) : reject(new Error('tesseract error')));
    tesseract.stdin.write(imageBuffer);
    tesseract.stdin.end();
  });
}

async function storeResult(text) {
  const connection = await mysql.createConnection({
    host: 'localhost',
    user: 'dream',
    password: process.env.MYSQL_PWD,
    database: 'dreamfuture'
  });
  const sql = 'INSERT INTO invoices (raw_text, captured_at) VALUES (?, NOW())';
  await connection.execute(sql, [text]);
  await connection.end();
}

(async () => {
  try {
    const img = await captureRegion();
    const text = await runTesseract(img);
    console.log('OCR result:', text.trim().split('\n')[0]); // log first line for sanity
    await storeResult(text);
  } catch (e) {
    console.error('Automation failed:', e);
  }
})();

What the script does

1. Capture a 800×600 region where I keep my invoice PDF viewer open.
2. Convert the raw video stream to a PNG using ffmpeg.
3. Pipe the PNG into Tesseract via stdin – no temporary files are written.
4. Insert the raw OCR text into a MySQL table called invoices.

4. Hooking the OCR into My Website

My site runs an Express API that serves the admin dashboard. I added a simple endpoint that triggers the OCR job on demand:

// routes/ocr.js
const express = require('express');
const router = express.Router();
const { execFile } = require('child_process');

router.post('/run', (req, res) => {
  execFile('node', ['ocr.js'], (err, stdout, stderr) => {
    if (err) return res.status(500).json({error: err.message});
    res.json({message: 'OCR completed', output: stdout.trim()});
  });
});

module.exports = router;

The admin UI now has a Run OCR button that calls /api/ocr/run. The result appears in the invoices list, where I can click a row to view the full text and copy the line items into my accounting software. No manual typing required.

5. Deploying on a Home Server (Raspberry Pi 5)

I keep a low‑power Pi on my desk for these jobs. The steps are the same, but I add a systemd service to run the script nightly:

# /etc/systemd/system/ocr-nightly.service
[Unit]
Description=Run PipeWire‑Tesseract OCR nightly
After=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/node /home/kaleb/ocr.js
Environment=MYSQL_PWD=supersecret

[Install]
WantedBy=multi-user.target

sudo systemctl enable --now ocr-nightly.service

Now every night at 02:00 AM (configured in a separate timer unit) the server grabs the latest invoice screenshot, extracts the text, and populates the database. The admin panel shows the new rows the next morning.

6. Why This Matters for Palm Beach County Service Businesses

Local trades – HVAC installers, landscaping crews, pool maintenance firms – still receive a lot of paperwork in PDF or printed form. Rather than hiring a third‑party data‑entry service, a simple OCR pipeline like the one above lets a small office digitize invoices, work orders, or permits with a single click. Because the code runs on inexpensive hardware you already own, the ROI is immediate: less time typing, fewer transcription errors, and searchable records for tax season.

7. Lessons Learned

• PipeWire works everywhere – even on Wayland, so you don’t need to fall back to X11 hacks.
• Avoid temporary files – piping directly between pw-record, ffmpeg, and Tesseract keeps the process fast and reduces disk wear on SSDs.
• Tesseract accuracy – clean, high‑DPI screenshots give the best results. I found that setting --dpi 300 in the Tesseract call improves recognition of small invoice numbers.
• Error handling – the Node wrapper captures exit codes from both ffmpeg and tesseract. In production you’ll want to log failures to a file or monitoring service.
• Scalability – if you need to process dozens of invoices per day, spin up a Docker container that runs the same script and feed it a queue of image URLs.

That’s the entire pipeline I built for my own business, and it can be replicated by any small service provider in Palm Beach County that wants to automate the mundane parts of paperwork.

I build sites and automations for small businesses at dreamfuturestech.com.