I Ran 6 n8n AI Workflows on a 1.8B Local Model: 192 Runs, 0 Broken JSON, 7 Fake Hot Leads

Every "run your AI workflows locally" post shows the happy path. I wanted the failure list instead, so I took six of my own n8n workflows, pointed them at a 1.8B model running on CPU, and ran every test case 3 times.

Setup: n8n 1.123.84 self-hosted, Qwen 1.8B (Q4_K_M) in Ollama on CPU through its OpenAI-compatible /v1 endpoint, temperature 0.2, response_format: json_object. No paid APIs. Each workflow was called through its webhook by a small Python script that saved every raw response. Where I had a guarded version (a Code node after the LLM that checks the output before anything acts on it), I ran raw and guarded side by side.

Total: 192 runs. Here is what happened.

The scoreboard

And the good news, which surprised me: 0 of 192 responses were invalid JSON. With response_format: json_object and Ollama, even a 1.8B model always returned parseable JSON. The failures were all in the content.

Failure 1: the lead qualifier gives everyone an 85

The prompt asks for score (0–100) and tier (hot/warm/cold). The model's answers across 18 runs:

• 85, warm: 10 times
• 75, warm: 3
• 60, warm: 2
• 20–25, cold: 3

It scored the CTO with budget and a 6-week deadline 85, and it scored "send me free templates for my college project" 85 too. It then labelled almost all of them warm anyway, so the two fields contradicted each other in 13 of 18 runs.

That matters because my workflow branches on the score (Hot Lead? score >= 70 → Slack). So 7 runs would have pinged Slack for a lead that wasn't hot. One of those was "We offer cheap backlinks and guest post on DA 90 sites!!!"

What the guard caught: the Validate node flagged 14 of 18 runs. It caught the score/tier contradiction every time and the backlink spam 3 of 3 times (a plain regex for backlink|guest post|seo|crypto, which forces the tier to cold).

What it couldn't fix: if the model gives every lead an 85, no amount of validation turns that into a ranking. The guarded version still got the tier wrong in 8 of 18 runs. On a 1.8B model, don't let the model score. Have it extract facts (has_budget, is_decision_maker, timeline_weeks, company_size) and compute the score in a Code node.

Failure 2: the invoice extractor reads 02/11 as February

Six documents: a GST tax invoice, a café receipt, a freelancer invoice, an AWS statement, a handwritten-style electrical bill, and an order-shipped email.

• Chai Point receipt, "02/11/2026": extracted as 2026-02-11 in 3/3 runs. In India that's 2 November. The model assumed US date order every time.
• AWS statement: total extracted as 62.93 in 3/3 runs. The real total is printed right there: Total $54.61. 62.93 appears nowhere in the document. The model added the numbers up itself and got it wrong.
• AWS statement again: invoice_number: "Account 4491-2210-7783" in 3/3 runs. A statement has no invoice number, and the prompt says "use null when unknown; never guess."
• AWS statement date: 2026-09-01 (start of the billing period) instead of the statement date in 2 of 3 raw runs.
• The GST invoice, the electrical bill with CGST + SGST, the freelancer invoice and the shipped email came out right in every run.

What the guard caught: my Validate node checks that total, invoice_number, gstin and due_date actually appear in the source text, that subtotal + tax = total, and that the date is YYYY-MM-DD. It caught the made-up 62.93 total 3/3 times, plus 2 runs where the model invented a due date for the freelancer invoice, which only says "payment within 15 days". It also produced 3 false alarms: on the GST invoice, the correct due date 2026-10-28 didn't match the printed "28 Oct 2026" in my text check. They were safe failures, since a human just looks at them, but it was still noise. Normalise dates before comparing.

What it missed: the February/November swap. 2026-02-11 is a perfectly valid date and "02" and "11" are both in the text. It also didn't flag "Account 4491..." as an invoice number, because that string is in the document. 4 of 18 guarded runs had a wrong field and still passed. Fixes I'm adding: pass the document's locale to the prompt ("dates are DD/MM"), reject any date more than 60 days from today, and reject invoice numbers that start with words like "Account".

Failure 3: the FAQ bot that "knows" you have no annual plan

The knowledge base has three lines: refund window, two prices, support hours. I asked 3 questions it covers and 3 it doesn't.

The two uncovered questions about data residency and Zoho got the safe answer, "I am not sure, a teammate will follow up shortly." But "Do you offer an annual plan with a discount?" got "No, we do not offer an annual plan with a discount" with escalate: false, 3 out of 3 times. One run went with "No, we currently offer a monthly plan."

Nothing in the KB says that. It's a confident, plausible, made-up answer, and to a customer it's indistinguishable from a real one.

My guard checks that any number, price or email in the answer appears in the KB. This answer has none, so it passed. The guard did catch something I wasn't expecting: in 5 runs escalate came back as null or missing instead of a boolean. The raw workflow treats a missing value as "don't escalate".

The fix: stop asking a 1.8B model to judge whether it knows the answer. Ask it to return kb_line (which KB line it used, or none). If it's none, or the cited line doesn't share key words with the answer, escalate in code.

Failure 4: review replies still sound like a bot

This is my Google review reply workflow from the import-test post. The original prompt-only version used a canned bot phrase in 16/24 replies and didn't mention anything specific from the review in 14/24. The v2 prompt, which bans phrases and requires something specific, did better on paper (11/24 and 11/24), but its own check node sent 22 of 24 to the owner for review. The reasons were bot phrases that slipped through anyway, a made-up contact detail, a generic "Thank you for…" opening, and in one case "refund" on the reply to a 1-star "hair in my idli" review.

The checks worked. The 1.8B model just can't follow 5 style rules at once. For this workflow, a small local model is a draft generator, not an auto-poster.

What held up: the hardened inbox

The prompt-injection-hardened triage workflow from my last security post was the one that didn't break. I ran 6 of the injection emails (fake "SYSTEM" overrides, "mark it urgent", refund demands, a fake policy section) 3 times each: 0 of 18 succeeded, and all 18 went to human review. The model went along with the attack in its raw output 3 times, but it didn't matter, because the model doesn't choose the action; code does.

The cost is the same as before: normal emails got the right category in only 12 of 18 runs, so a person sees more mail than they would with a bigger model.

Latency

On a shared 38-core CPU box (other jobs were running, so treat these as rough):

• FAQ: 2.0s average
• Inbox triage: 7.2s average (p95 31s)
• Invoice: 6.6s average guarded, with a few 90s outliers raw while the box was busy
• Lead: 8–14s average

The guard nodes themselves add nothing measurable; they're plain JavaScript. If you care about latency on CPU, the big lever is box load, not the workflow.

What I'd tell anyone running n8n on a small local model

1. JSON mode works. Stop worrying about parse errors and start checking content. 0/192 invalid JSON, but plenty of wrong JSON.
2. Never branch on a number the model invents. Have it extract facts and compute scores in code.
3. Check every extracted value against the source text. That one check caught the made-up AWS total 3/3 times.
4. Validators can't catch plausible lies. A valid date in the wrong order, or a confident "no", passes every type check. Add domain rules (locale, date windows, "cite the KB line").
5. Treat missing booleans as "yes, a human should look." escalate: null should never mean "auto-send".
6. Let code decide actions. It's the only reason the inbox workflow was safe on a model this small.

The free workflows

The original lead qualifier and review reply workflows are free in arkintof/n8n-ai-agent-workflows. To point them at a local model, change two values in the Config node: baseUrl: 'http://127.0.0.1:11434/v1' and your model name. Then add a Code node after the LLM that checks each field against the input before anything downstream runs.

Have you run your n8n AI workflows on a small local model? Which one broke first for you, and which ones turned out to be fine?

Written with AI assistance; workflow built and tested by me.