Provenance in the Primary Key: 54 Facts In, 49 Stored, and a Dedupe That Has Never Fired

Originally published on michael-kaminski.io.

Line 82 of my Analysis Engine's fact base says it deduplicates and keeps the most confident copy. On today's run the repo collector emitted 54 observations, the store kept 49, and the branch that keeps the more confident copy executed zero times. Stripe is in the store three times.

The conclusion first: identity and provenance answer different questions, and the answer to the second must never be part of the first. Identity asks whether two observations are the same fact. Provenance asks who reported it and where.

I hashed the collector and the file location into the identity, and from that moment two reports of one thing could never be the same fact. The reconciliation the docstring promises is not buggy. It is unreachable.

This is the third decorative control I have found in my own systems this year. The first was a significance test with a window too short to detect anything. The second was a power calculation on the wrong scale. This one is a primary key that was designed to guarantee uniqueness and guarantees the opposite.

The engine, in one paragraph

The Analysis Engine is a Python 3 tool I wrote to read an unfamiliar system and produce an architecture, a flow map, an ERD, a capacity model and a ranked roadmap. Four collectors feed it: repo, cloud, database and probe. Each one emits Observation objects, each observation carries an Evidence record, and nothing else crosses the boundary into the analyzers.

Everything below was measured on 2026-10-18 against the engine's invented fixture, acme-lending. No client, no employer, no real account.

The identity hash carries the audit trail

An observation's id is sha1(kind, key, collector, evidence.locator), truncated to twelve characters. I put the collector and the locator in on purpose. The engine makes claims about other people's systems, and I wanted every claim traceable to the scan that produced it and the line it was read from.

That decision has a consequence I did not run to the end. Two collectors that see the same component produce two different ids. One collector that sees the same component in two files produces two different ids. The store's add() compares confidence only when ids collide, so the comparison is reserved for the one case that carries no information: the same scanner reporting the same line twice.

The code says the design out loud. The docstring on line 82 reads Append-only fact base. Deduplicates, keeps the most confident copy. The id on line 73 makes the second sentence impossible for any pair of observations worth reconciling.

What the store did today

The five collisions were all config rows for HTTP routes, each the same line of services/api/main.py reported twice by the same scan at the same confidence, 0.85 against 0.85. The comparison is strict greater-than, so the existing copy stayed all five times. That is the entire history of the deduplicator doing its job: it dropped five exact duplicates of itself.

The seven real duplicates never got near it. Stripe appears at 0.75 from web/package.json, at 0.75 from services/api/requirements.txt, and at 0.60 from services/api/main.py:6. Three ids, three rows, zero comparisons. Plaid, PostHog, the Anthropic API, the API service, the decision worker and the primary database each appear twice.

Lookups return the wrong copy in two of seven cases

find(kind, key) walks the store in insertion order and returns the first match. For PostHog it returns the 0.60 observation from web/app.tsx:2, not the 0.75 one from web/package.json. For the Anthropic API it returns 0.60 from worker.py:3 over 0.75 from requirements.txt. In the other five pairs the first copy happens to be the best one, or tied with it.

So the rule the store actually applies is "whichever file the scanner walked first." The rule it was designed to apply is "whichever report is most confident." On this fixture the two rules disagree 2 times in 7, and nothing downstream notices, for a reason that took me longer to see than the bug did.

Nothing broke because the graph fixed it one layer down

The analyzer builds a SystemGraph from the component observations, and add_component() merges by key: the first copy becomes the node, later copies fill in any attribute the node is missing, and their evidence locators are appended to a list. Then _canonicalise() folds provider resources into the logical names they describe, so aws-ecs-service-api becomes part of api and primary-database becomes aws-db-instance-primary. Eighteen component rows become eight real nodes plus a synthetic users actor, and the report reads clean.

That is the reconciliation the store was supposed to do, rebuilt where it was not supposed to live, with a different rule. The graph uses insertion order and fill-the-blanks. The store promised confidence. The two rules happen to pick the same winner today: in the two pairs whose values differ, api and decision-worker, the Kubernetes manifest at 0.90 was also walked before requirements.txt at 0.85.

The loss is still real. The manifest says the API's tech is a Kubernetes Deployment; requirements.txt says it is FastAPI. First-wins kept the Deployment, and the word FastAPI appears in the raw evidence file and in none of the six rendered reports. The confidence scores I attached to every observation had no vote in that, and the first time the walk order and the confidence disagree, the graph will keep the weaker copy without a message.

In a tool whose output is a capacity model, the next value this happens to is a replica count or a service time. The error will not announce itself. It will be a plausible number.

The case the design exists for has never run

Cross-collector conflict is the reason the confidence field exists, and it has never occurred in this store. Cloud, database and probe are all enabled: false in the fixture's target.json. All 49 rows today carry collector=repo.

Three of the four collectors emit components and all four emit metrics, so the collision is designed in and waiting. It has simply never been exercised, and the smoke test, the only test in the repository, contains no assertion about duplicates, confidence or find().

I had an acceptance test for the store in my head. I never wrote it down, and the thing that would have failed it was switched off in the only configuration I ever ran.

The fix costs provenance or reconciliation

There are two honest repairs, and each gives up one of the two things the current hash tries to hold at once.

Option A: identity is (kind, key). Drop the collector and locator from the hash. Today's 49 rows become 41. Six of the seven duplicate pairs resolve on confidence; the primary database, at 0.70 against 0.70, still resolves on order.

The cost is that the losing copy's evidence is gone. The audit trail I built the hash to protect is the thing this option deletes, unless I also move evidence to a list on the surviving row, at which point I have rebuilt the graph's merge inside the store.

Option B: identity stays, lookup changes. Keep every row. Add an index on (kind, key). Make find() return the highest-confidence copy, and add a conflicts() method that lists every pair whose copies disagree on a value.

The cost is that the store stops being a deduplicator and becomes a disagreement log, 49 rows stay 49, and a human has to read the log for it to mean anything.

I prefer B. The point of four collectors is that they disagree, and a tool that quietly picks a winner has thrown away the most useful thing it collected. A conflict between a declared replica count in Terraform and an observed one from the cloud API is a finding, not a dedupe failure.

What it has cost so far

I diagnosed this on 2026-09-03 and wrote that I was building option B. Today is 2026-10-18. The modification time on core.py is 2026-08-28. Forty-five days, zero lines changed.

The reason is the paragraph about the graph. A defect that produces correct output has no priority, and the merge one layer down made the output correct on the only fixture I run. Patching a symptom downstream of its cause is the cheapest way to make a bug permanent.

I know that rule. I still did it, because I did not patch it on purpose: the graph's merge was written for a different reason, and it happened to cover for the store.

The specific rule I am taking from this: hash what makes two facts the same, and nothing else. Who reported a fact, and from where, belongs next to the fact, repeated as many times as it was reported. The moment provenance enters the key, every source becomes its own universe, and reconciliation across sources becomes a word in a docstring.

If your agent pipeline merges observations from more than one collector, run one count before you trust the merge: rows in the store against distinct (kind, key). Mine were 49 and 41. If the first number is larger, the deduplicator is not deduplicating, the key is the first place to look, and some layer below it is already deciding for you.