My chatbot said my résumé listed no jobs. Finding out what else was wrong cost $1.96.

"The passages don't list his job history."

That is what my own demo chatbot said about my own résumé. By the end of the day the bug was fixed with a test that proves it, and I had an eval harness that measured the pipeline on the live models and told me which of my tuning ideas were right. Total spend on the models: $1.96.

Free, step by step: the video series at https://www.youtube.com/@JoelStevick and the site, https://joelstevick.com, with the demo and the book.

The episodes this one comes from: Making it good, with numbers and Caps, the ledger and comparing runs

The question that failed

My site has a free demo: you upload your own PDFs and ask a chatbot questions about them. My two-page résumé went in, with the most obvious question there is: what jobs has he held. It answered that the passages it had didn't list my job history, and cited two passages from page 1. Asked what role I held at one employer, it answered correctly, from page 2, with dates. The text was there. The broad question never reached it.

Was it chunking?

That was my first guess, and it was wrong. My résumé becomes 9 chunks of about 900 characters, and the seven employers sit in 5 of them, across both pages. The answer only ever saw 5 chunks, so seven jobs in five chunks plus a summary could never fit. The words of the question matched the wrong chunks: "jobs" and "held" appear in my summary, not in the headings that name the employers, and keyword search put the summary first. And the model said the information was absent, and my check let it through: the check rejects an answer that cites a passage it wasn't given or states a figure no cited passage has, and "the passages don't list his job history" did neither.

A test first, failing for the right reason

Before changing anything I wanted a test that failed the way the demo had. It uploads a fictional two-page résumé, seven jobs, two on page 1 and five on page 2, through the real demo page and asks exactly "what jobs has he held". The models are scripted, so the scripted answer can only list jobs that appear in the passages it is given. On the old code it named five. The fix: the answer is written from up to 12 passages, and the reranker puts them in order instead of cutting them; a small upload that fits is read whole. With the limit set back to 5, the same test goes red.

Why the fix wasn't the end

A scripted test proves the pipeline does what I designed. It says nothing about whether the real models, given the real context, answer well. For that I built an evaluation that runs the real thing: 25 questions, four on my résumé and 21 on my 267-page book, each with its expected facts quoted from the PDF and checked character for character against the text the demo extracts. Each run uploads both PDFs through the demo's real routes, asks every question with the real models, and records whether the right passage reached the reranker, whether it made the 12 the answer was written from, whether the answer cites it, which required facts it contains, how long it took and what it cost. A run takes ten to fifteen minutes and costs about 27 cents.

The baseline: 21 of 25

My question now passed on the live models, and the pipeline scored 21 of 25. The failures taught me more than the passes. The reranker was quietly broken: it thinks before it answers, and its thinking used up its 256-token limit before it wrote the ranking on 4 of 25 questions, and each time the pipeline fell back to keyword-and-vector order without a word. No test of mine could have seen it, because my tests script the models. The same code was live on my site. Two answers used 587 and 588 of their 600 tokens; any longer and the reply is cut off mid-JSON. And a length rule refused a perfect answer: asked for the chapters, the model listed all thirteen, and a sentence counter read the numbered list as fifteen sentences.

Seven runs

I changed as little as possible each time. Round 1 turned the reranker's thinking off, raised the answer limit from 600 to 2,000 tokens and fixed the sentence counter: 20 of 25. Turning the thinking off made the reranker finish, but a reranker that doesn't think ranks worse. One number would have told me round 1 failed; the per-question view told me which of its three changes helped and which hurt. Round 2 kept the thinking at low effort with room to finish: 22 of 25. Round 3 let the reranker read 48 passages instead of 24: 23 of 25, at a price, since it fixed two questions and broke one that had passed. I ran the final settings three times because the models are not deterministic: 23, 24 and 23 of 25, against 21 before. One question passed once in three runs, and I didn't want to tune toward a lucky run.

Before the tuning shipped, a second agent checked my notes against the run files and caught me overclaiming twice. I had written that the reranker ran out of room on six questions; the files say four.

What I'd tell you

Checks in code catch the failures you have named. An eval on the live models shows you the ones you haven't. You need both. The harness, the 25 cases and every run's results are public, next to the code they measure, and the full story with every run's numbers is at joelstevick.com/blog/tuning-with-evals. Or try the demo on your own PDFs at joelstevick.com/demo. It is the same pipeline, as tuned here.

Watch the series: https://www.youtube.com/@JoelStevick. Everything else, the demo and the book included: https://joelstevick.com