580 quiz questions, 0 structural errors, and 73% of the answers on B

A multiple-choice question bank is a dataset with a schema, and every item can satisfy that schema while the set as a whole is broken. Mine was: 580 items, four options each, a valid answer index on all of them, 500 unique stems - and a bank that rewarded always picking B.

It is the question bank behind a ServSafe Manager practice site. 500 generated items plus 80 hand-written ones, stored as JavaScript rather than JSON: a comment banner, one object per line, a trailing comma on every line. Last night I wrote a linter for it, and then a fixer. Here is what survived contact with the data, and which of my own checks turned out to be worthless.

Loading it

Line-based parsing, not one regex over the whole file:

function loadItems(text) {
  const items = [];
  let inside = false;
  for (const line of text.split(\n)) {
    const t = line.trim();
    if (!inside) { if (t.endsWith("= [")) inside = true; continue; }
    if (t === "];") break;
    if (t[0] !== "{") continue;
    items.push(JSON.parse(t.replace(/,$/, "")));   // strip the trailing comma
  }
  return items;
}

A single regex over the blob, of the shape /=\s*[([\s\S]*)];/, works right up until one string value contains a closing bracket. That is a bug you find later, on one item, in production. Splitting per line costs nothing and cannot fail that way.

Checks that earned their keep

1. Duplicate stems

22 of the 500 generated items were byte-identical to an earlier item: same stem, same four options, same key. If a learner drills 500 questions, hitting the same one twice is not cosmetic - it is the product claiming coverage it does not have.

The check is a Set of normalised stems:

const seen = new Map();
for (const it of items) {
  const key = it.question.toLowerCase().replace(/[^a-z0-9]+/g, "").trim();
  if (seen.has(key)) report("duplicate-stem", seen.get(key), it.id);
  else seen.set(key, it.id);
}

2. Numeric agreement between the keyed option and the explanation

Half of this bank is temperatures, so the explanation is the only proof of the answer. If the keyed option says 165 degrees F and the prose reasons its way to 145, one of them is the wrong answer, and it is usually the key:

const onKey = new Set(numbersIn(options[correct]));
const inProse = new Set(numbersIn(explanation));
if (onKey.size && inProse.size && ![...onKey].some(v => inProse.has(v))) {
  const fromWrongOption = options
    .filter((_, i) => i !== correct)
    .flatMap(numbersIn)
    .filter(v => inProse.has(v));
  if (fromWrongOption.length) report("temp-contradiction", fromWrongOption);
}

Zero hits here. It is still the one check I would keep if I could keep only one, because it targets the failure that copy-paste editing actually produces: the stem gets rewritten, the explanation gets rewritten, and the key stays where it was.

3. Answer position distribution

This one is four lines of arithmetic and it found the real bug:

Blind guessing scores 25%. This bank scored 73% by answering B to everything - and the pass mark on the real exam is 70%. A generated bank keys the answer where the template puts it, and every template put it second.

Two checks I deleted

The which-is-NOT heuristic

The rule I was proudest of: for items of the form which is NOT a member of this group, the keyed option should be the odd one out, so flag items where the keyed option looks most like a member. I built it, ran it, and got five hits.

All five were false. The regex matched the word not anywhere in the stem, so a cooler that is not maintaining temperature and a why is this wrong item both looked like negation items. Precision: 0 for 5.

I deleted the heuristic and kept a one-line inventory of the six genuine which-is-NOT items, for a human to read. A check that produces noise teaches you to ignore the report, which is worse than not having the check.

Reused option sets

Twenty-two groups of items share an identical set of four answer choices. I flagged them, then looked: they are the cooking-temperature items, and the same four temperatures are the correct set of distractors for fish, pork, eggs and casseroles. Reusing distractors is normal. Demoted to info.

The fact-checker, and why a new check should be distrusted

Structural checks cannot tell you an answer is wrong - only that it is internally consistent. So I wrote a second script with a table of the Food Code minimum cooking temperatures and matched food keywords in each stem against it. It found 10 problems in 27 items.

Nine of the ten were my regexes:

• The table was case-sensitive, so ground beef patties missed the rule for ground meat.
• A filter meant to catch cooking-temperature items also caught questions about food that is undercooked, which have no temperature in them at all.
• The rule for meat ran before the rule for meat substitutes, so a plant-based item keyed at 135 degrees read as a 145-degree error.

The tenth is a chicken item whose keyed option carries no temperature, because the answer is a management concept rather than a number. Not a bug either.

The lesson generalises: when a brand-new check reports hits, suspect the check first. A linter is code, and its first run is a test of the linter, not of the data.

Rebalancing the keys

Fixing the position skew without touching content means permuting each item options so the key lands somewhere new. Two things make it safe:

• Skip any item whose options include all of the above or none of the above. Those carry meaning in their position.
• Skip any item whose explanation names an option letter. (There were none, but the guard costs one line and stays.)

Then allocate slots greedily so each one gets a quarter, and shuffle the three non-keyed options with a PRNG seeded from the item id:

let cursor = 0;
for (const it of items) {
  const keyed = it.options[it.correct];
  const rest = it.options.filter((_, i) => i !== it.correct);
  const rand = seededRng(it.id);
  for (let k = rest.length - 1; k > 0; k--) {
    const j = Math.floor(rand() * (k + 1));
    [rest[k], rest[j]] = [rest[j], rest[k]];
  }
  const target = cursor % 4;
  cursor++;
  it.options = [0, 1, 2, 3].map(slot => slot === target ? keyed : rest.shift());
  it.correct = target;
}

Seeding from the id makes the whole thing idempotent: rerun it on the fixed file and you get the fixed file back. That matters because this data file is regenerated by another script, and an unseeded shuffle turns every rerun into a diff.

Result: 125 / 123 / 127 / 125. The 73% became 25%.

Four things worth copying

1. Measure distributions, not just shape. Every structural check passed while the answer key sat on B.
2. Audit each new rule precision on a sample before trusting a single hit.
3. Make fixes idempotent and seeded, so rerunning is a no-op.
4. Leave a marker comment in generated files - mine now says the file was hand-patched and by which script - so the next generator run cannot silently undo the fix.

The bank is 500 items, 500 unique stems, four options each, and the key is spread 125/123/127/125. The linter reports zero errors and zero warnings. It took more time to distrust my own checks than to write them.