Dev.to WebDev 🛠 Dev 👁 0 📖 6 min read

Ten of our tests changed what their score means, and the cut-over is a metric key rather than a date

CogniPrep has 491 practice tests across 77 assessment providers. Every one of them ends with a percentile, and that percentile is not invented per sitting: a nightly job aggregates the recent scores for each test and sto

CogniPrep has 491 practice tests across 77 assessment providers. Every one of
them ends with a percentile, and that percentile is not invented per sitting: a
nightly job aggregates the recent scores for each test and stores a mean, a
median, a standard deviation and nine percentile cut points. The scoring code
then reads that row.

Last weekend we rebuilt ten of those tests to match the real instruments more
closely. Different mechanics, different clocks, in four cases a different
answer format. And with that, raw_score stopped meaning what it used to mean.
Several of them had been storing something on a 0 to 100 scale and now store a
count of questions right.

A percentile computed across both populations is worse than no percentile. It is
a confident number produced from two different measuring sticks.

The obvious fix is a date, and the date is wrong

The first thing you reach for is a cut-over timestamp: ignore sessions before
the rebuild shipped.

The problem is that no single timestamp exists. The commit date is not it: the
code sat in a branch. The merge date is not it either, because the thing that
changes a candidate's experience is a deployment, and these ten games were
rebuilt on five different branches merged over two days. Worse, a date has to be
maintained by hand forever, in a constant nobody will revisit, and it is wrong
in exactly the silent direction: a date too early quietly readmits the old
population.

So the filter does not ask when a sitting happened. It asks what the sitting
wrote.

/**
 * Games rebuilt in a way that changed what raw_score means. Their older sessions
 * are on a different scale and must not shape the population, so only sessions
 * that carry a metric the new scorer always writes are counted. A metric marker
 * rather than a date, because the cut-over is whenever the build is deployed.
 */
const RESCORED_GAMES: Record<string, string> = {
  'sav-verbal': 'seconds_per_answered_%',
  // ... eight more
  'tp-numerical': 'num_bank_version',
};

/** Only sessions on the game's current score scale (see RESCORED_GAMES). */
function currentScaleFilter(gameId: string) {
  const marker = RESCORED_GAMES[gameId];
  if (!marker) return sql``;
  return sql` and exists (
    select 1 from jsonb_object_keys(${gameSessionsTable.metrics}) as k
    where k like ${marker}
  )`;
}

Each session row carries a jsonb bag of metrics beside its score. The new
scorer for a rebuilt test always writes at least one key the old one never
wrote, so the presence of that key is the version stamp. A game that was
not rebuilt gets an empty SQL fragment back and its query is untouched, which
matters because this runs for all 491 games.

Two details in there are worth stealing and one is worth avoiding.

Worth stealing: returning an empty SQL fragment rather than a 1=1 or a
conditional query builder. The filter composes into the existing aggregate as a
string of nothing for 481 of the games.

Worth avoiding: like 'seconds_per_answered_%' is not the prefix match it looks
like. In SQL, _ is a single-character wildcard, so that pattern means
"seconds_per_answered, then any one character, then anything". It happens to
be harmless here because no other key in the bag begins that way, but it is the
kind of pattern that matches something embarrassing a year later. like
'seconds\_per\_answered\_%'
is what was meant.

The floor is where the honesty problem lives

The aggregate has two bounds. It samples at most the 20,000 most recent scores
per game, and it refuses to publish anything from fewer than 10:

const MAX_SAMPLE_SIZE = 20_000;
/** Below this, percentiles are too noisy to publish. */
const MIN_SAMPLE_SIZE = 10;

Below the floor the game is reported as skipped, which is a separate counter
from failed so that a run full of quiet games is not an alert. Nothing is
written, and the row that was already there stays.

Which is the interesting part. Filter out every old sitting from a test that was
rebuilt last night and you are left with a sample of nearly zero, so the row
that remains is whatever was there before: for a new or rebuilt game, a seeded
placeholder distribution. And a seeded row has no idea the scale moved either.

Our own backlog now carries the consequence in one line:

scripts/seed-population-stats.ts seeds for tp-sjt, tp-inductive and
tp-numerical predate the new scoring and banks.

So the filter works and the fallback is now the stale thing. The design has a
version stamp for sessions and no version stamp for the seed.

What the public endpoint says this morning

Population stats are public, because the browser needs them to put a percentile
on a score, so you can read all of this from outside. Three states, same
endpoint:

game sampleSize mean median lastUpdated
sav-numerical 0 4.3 4.3 2026-11-20T14:57Z
tp-inductive 0 58 58 2026-11-16T14:52Z
sav-error 29 86.55 95 2026-11-22T02:00Z

The first row is the design working as intended: no qualifying sittings yet, and
the seeded row is on the new scale, where a mean of 4.3 is four and a bit
questions right.

The second row is the gap. A mean of 58 on a test whose score is now a count of
items is last month's scale, and the timestamp proves it: 16 November, five days
before the rebuild.

The third row is the one I cannot explain yet, and it is the reason this post
exists. It was written by last night's run, after the filter shipped, and its
median of 95 is not a number the rebuilt scorer can produce, because that scorer
returns the number of questions right on a short checking test. Either the row
was computed by a build that did not yet have the filter, or something is
admitting sittings the marker was supposed to exclude. Tonight's 02:00 run is
the experiment, and the timestamp is public, so the answer arrives in a cache
header rather than in a log I have to go and read.

The underlying column, for completeness, is this:

raw_score: decimal({ precision: 5, scale: 2 }).notNull(), // Actual game performance score (0-100)

The comment is from August. Ten games now put item counts in there, and nothing
in the type system noticed, because 7 is a perfectly good decimal.

See it

The stats endpoint is unauthenticated, so this is three curls:

# rebuilt, no qualifying sittings yet: sampleSize 0, new scale
curl -s https://cogniprep.app/api/games/population-stats/sav-numerical

# rebuilt, still serving a seed row from before the rebuild
curl -s https://cogniprep.app/api/games/population-stats/tp-inductive

# the row that disagrees with its own scorer
curl -s https://cogniprep.app/api/games/population-stats/sav-error

Compare lastUpdated against the tests themselves. The rebuilt Saville tests are
described at
/games/saville, the checking one at
/games/saville/error-checking,
and the Test Partnership inductive test at
/games/test-partnership/inductive-reasoning.
The Saville pages were rewritten on 20 November and the inductive one on
21 November. For two of these tests the lastUpdated on the stats row is older
than the page describing the test it scores, and that mismatch is the whole bug
class in one comparison.

If you run anything that compares a user against a population, the lesson is
cheap to apply: version the scale explicitly, store the version on the row, and
make the fallback carry the same version. Inferring it from the shape of the
data works right up to the point where a sample of zero hands the question to a
placeholder nobody versioned.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.