The Human Half of the Gate: My 10-Minute Review-Queue SOP for n8n LLM Outputs

My last post ended with a one-command gate — ./prepublish.sh — that verifies the mechanical contract of an n8n LLM workflow: grounded, deduplicated, cost-bounded, loud on failure. One thing deliberately didn't make it into the script: the review-queue spot check. Output quality is subjective; no assertion belongs there. But "just eyeball it" is how you end up reading fifty outputs at 11 PM and calling it reviewed. So here is exactly what the eyeballing looks like — a ten-minute SOP.

Why sampling is enough

The script bounds every mechanical failure mode. If an output is ungrounded, duplicated, over budget, or silently failed, the gate catches it. What remains is prose judgment: is this well-written, does it read like a human wrote it, would I send it to a client? Reviewing 100% of outputs doesn't scale, and — worse — human attention drifts. Item 40 gets a skim that item 4 got a stare. A disciplined sample with a fixed rubric beats a tired full review every time. Ten minutes, then you're done.

The sample: stratified, never random-only

Pure random sampling wastes your ten minutes on the easy middle. I pull 12 items, three ways:

• 5 newest. Fresh eyes on current behavior. If the model or prompt drifted yesterday, it shows up here first.
• 5 flagged by cheap heuristics. These are 30-line checks that don't judge quality — they just pick what deserves your eyes: length outliers (|z| > 2 against a rolling median), banned-phrase hits, repeated-n-gram rate above threshold, anything the escalation path touched.
• 2 pure random. Keeps me honest. The heuristics are only as good as my imagination; random catches what I didn't think to flag.

The 4-question rubric, ~30 seconds per item

One rubric, same four questions every time. No notes, no scores — just four yes/no answers:

1. Would I send this to a paying client as-is? The only bar that matters. Everything else is commentary.
2. Does it answer the question asked, or a nearby question? This is the sneakiest failure: grounded, well-written, completely off-target. The gate can't see it because every assertion passes — the answer is just adjacent to the right one.
3. Is there any sentence no human would write? Stilted phrasing, hedging stacks ("It's worth noting that it may be possible that..."), the occasional "As an AI" leaking through a system prompt.
4. Do any two outputs side by side look like the same template? A sample-diversity check. Twelve outputs that all start with "Here's a summary of" are a prompt bug, not twelve good outputs.

The decision log: every rejection gets a tag

Rejections are data. Every rejected item gets one tag — prompt, fixture, model, or data — and I tally them weekly. The pattern is load-bearing: if one tag is over 50%, that's where the bug lives. In my experience it's the prompt far more often than the model. "The model got dumber" is usually "my prompt drifted and nobody noticed." The tally is what turns a vibe into a work order.

The review queue is a gate factory

This is the part I didn't expect when I started the SOP: the human queue doesn't just catch bad outputs — it tells me what to script next. Three catches that became mechanical checks:

1. Summaries starting with "In summary," every single time. A human spots the monotony in seconds. Fix: an n-gram repetition detector across the batch, now step 0 of the sample heuristics.
2. Action items that quoted whole paragraphs. The output was "grounded" — technically true — but useless. Fix: an output/input length-ratio check; anything above the ratio goes to the flagged five.
3. Polite filler. "I hope this helps!" at the end of a machine-to-machine handoff. Harmless, embarrassing, and invisible to every assertion I had. Fix: a banned-phrase list, checked before the human sample.

Each of these started as a human "that looks off" and ended as a line in the script. The honest answer to "how do you know what to test" is: the review queue tells you. The script handles what you already know; the ten minutes handle what you don't know yet.

What I don't do

Three rules that protect the ten minutes from expanding:

• No marathon sessions. Never batch more than 24 hours of outputs. Batching destroys calibration — by item 30 you've forgotten what "good" looked like. The SOP runs before every version bump, small and frequent.
• No reviewing items the script already rejected. Don't review failures; review passes. The passes are where the lies hide. A rejected output is already handled.
• No hand-fixing without a tag. If I rewrite an output by hand and don't tag the cause, that bug will be back next week wearing a different hat. The tag is the price of the fix.

The split from the last post, completed: the script owns the mechanical contract — grounded, deduplicated, bounded, loud. Ten minutes of human eyes own the prose. The script catches what you know to fear; the review queue catches what you haven't learned to fear yet — and then teaches the script.

I'm building AI Automation Lab — n8n workflow templates for AI automation — and keeping a free lab on GitHub. I also send occasional notes on n8n reliability patterns: get them here.