The Sample Passed. What Did We Actually Prove?

Why a clean control test can still leave a decision maker exposed

By Manfredi Marrocco

The report says every sampled access removal was completed on time. The dashboard turns green. A manager asks whether the company can now rely on its leaver process.

There is a missing question: did the sample come from a complete list of leavers?

If the test began with tickets already created in the identity system, it may have excluded precisely the cases where nobody created a ticket. Ten well-processed tickets can demonstrate that those ten tickets were well processed. They cannot, by themselves, establish that all departures entered the process, or that the same process worked in every subsidiary.

This distinction has practical consequences. A control assessment does not merely collect proof that an activity happened; it establishes how far that proof can support a decision. The boundary depends on the control objective, the population, the test and the period examined. A pass without those boundaries can be more misleading than an explicit limitation.

Return of the Obra Dinn as a decision lens

In Return of the Obra Dinn, the player investigates a ship and reconstructs events through observation and deduction. The official description establishes that investigative premise. The connection to control assessment is the discipline of asking what an observation can support. It is not a claim that a fictional investigation follows an audit standard.

The useful connection is the boundary of a deduction. A ticket constrains the account of a recorded departure; it cannot establish the complete departure population. The game offers a designed mystery. An organisation must also account for evidence that is unavailable and conclusions that remain unresolved.

Start with the claim, not the screenshot

Consider a fictional organisation, Northstar Distribution. Its policy says that relevant access for departing employees must be removed within a defined period after departure. The assessment team receives ten closed tickets, each with a timestamp and an apparent approval. All ten meet the organisation's internal deadline. The example is hypothetical: neither the deadline nor the sample size represents a universal requirement.

What has been shown? For the ten selected records, the ticket system contains information consistent with timely completion. That is a useful observation. It is not yet the same as showing that every departure was captured, that all relevant accounts were disabled, or that an account could not be reactivated later.

The claim can be broken into questions:

1. Coverage: Were all in-scope departures identified, including contractors where the control applies?
2. Execution: Were the required actions completed by the applicable deadline?
3. Completeness of action: Did the actions cover the relevant systems and privileged accounts?
4. Persistence: Did access remain removed, or did a later event restore it?
5. Period and scope: Which entities, services and dates were actually examined?

Each question calls for different evidence. A ticket timestamp may speak to execution. It is weaker evidence of coverage when the ticket list is the only source of the population. A screenshot of a disabled account may demonstrate its state at a particular time, but not necessarily when it changed or whether other accounts existed.

NIST's definition of a security control assessment distinguishes correct implementation, operation as intended and the desired outcome against security requirements. Those are related but separate questions. NIST SP 800-53A provides tailorable assessment procedures and planning guidance; it is a US framework, and its use here is methodological guidance, not a claim that every organisation is legally bound by it.

The population is part of the test

A sample can only be interpreted against the population from which it was selected. If Northstar tests only identity-system tickets, its population is “tickets recorded in that system,” not automatically “all departures.” The distinction matters because the failure of interest may be the absence of a ticket.

A better starting point is to compare independent sources where available: the HR departure list, contractor offboarding records, identity-management tickets and account inventories. Independence is not absolute; two reports may inherit the same upstream error. The assessor should understand how each list is created before treating agreement as corroboration.

Reconciliation can reveal several kinds of mismatch. A person on the HR list may have no ticket. A ticket may have no matching departure. A contractor may appear only in procurement records. An account may be shared or registered under a service identity rather than a person's name. Some discrepancies may be explainable; others require investigation. The objective is not to force all lists to match by deleting inconvenient records, but to explain the differences and establish a defensible population.

If the HR extract covers only permanent employees in one country, it does not support a conclusion about all contractors or other countries. If an acquired subsidiary uses a different process, a clean result at headquarters cannot silently be extended to it. Those exclusions belong next to the result, not in an appendix that the decision maker never sees.

Three types of evidence, three different limits

An assessment can examine documents, interview people and test mechanisms or activities. In Northstar's case, the team might combine:

Corroboration matters, but more evidence is not automatically better evidence. Ten copies of a report generated from the same incomplete database do not create ten independent views. A useful evidence trail records the source, extraction date, scope, owner, relevant timestamp and limitations. Where data can be changed after the event, the team should consider provenance and whether more reliable logs or approvals exist.

The method also depends on the decision. A walkthrough can reveal a design gap early; testing selected historical records can reveal execution issues; a controlled exercise can establish what a mechanism does under particular conditions. None should be advertised as a substitute for all the others.

A clean sample is not a universal pass

Suppose the ten cases were selected because they were easy to retrieve. The result is informative about those cases but cannot support a calculated organisation-wide failure rate. Even a properly designed random sample has uncertainty. Statistical generalisation requires an appropriate sampling design, a known or defensibly defined population, assumptions and a method for interpreting results. A small sample with zero observed errors is not proof of a zero error rate.

Judgemental sampling has a different use. An assessor may deliberately select high-risk cases: privileged users, urgent departures, different sites, weekends and cases involving multiple systems. This can improve the chance of finding meaningful weaknesses, but the selection does not automatically produce a representative failure rate. The report should say why the cases were selected and avoid presenting the percentage of failures in that selection as an estimate for the whole population.

Finding one failure also requires judgement. A late removal could be an isolated mistake, a systematic design flaw, a clock or time-zone discrepancy, or evidence of a broader population problem. The team should verify the facts and explore cause and extent before declaring either “systemic failure” or “human error.” Conversely, a single exception involving a highly privileged account may be important even if the numerical failure rate appears small.

A further challenge is concentration within the sample. Departures handled by one experienced administrator in one subsidiary may tell little about a newly acquired unit using another workflow. Before expanding the conclusion, identify which meaningful variations the sample actually contains. A larger sample drawn repeatedly from the easiest operating condition can add records without adding much assurance about the difficult condition.

The finding should travel with its boundaries

Compare two possible conclusions:

Overstated: “The offboarding control is effective across the organisation.”

Bounded: “For ten selected employee departure tickets at headquarters during the reviewed period, recorded completion met the internal deadline. Completeness of the departure population and access removal in two external applications were not verified. No conclusion is drawn for contractors or other sites.”

The second conclusion is less comfortable but more useful. It tells the service owner what can be relied on and what remains unknown. It also creates a route to action: reconcile the independent departure population; identify which external applications are in scope; test a risk-informed set of cases; and revisit the conclusion if additional evidence changes the picture.

Evidence gaps and control failures must remain distinguishable. A missing log does not prove a removal was late. A valid log showing late removal is evidence of a failure against the stated criterion. Both can justify action, but the actions differ: improve evidence availability in the first case; investigate and correct the process in the second. Depending on the purpose and applicable audit method, an unresolved evidence gap may limit assurance or support a finding. It should not be renamed a successful control simply because no failure has been demonstrated.

Decide what to do with the uncertainty

A decision maker rarely has unlimited time to expand every test. The next step should reflect the potential consequence and the value of better information. If only a low-impact application is excluded, a scheduled follow-up may be proportionate. If the unverified population includes privileged accounts in a critical system, a faster reconciliation or temporary control may be justified. These are choices to make against the organisation's context, not fixed thresholds prescribed by this article.

The resulting record should identify an owner, the unresolved question, a deadline for obtaining evidence and the action to take if that evidence cannot be produced. It should also state whether a proposed safeguard is already operating or remains planned. Calling a future control “compensating” does not make it effective today.

Monitoring can reduce the time between a change and its discovery, but it does not eliminate the population problem. An automated dashboard that counts only completed tickets inherits the ticket system's blind spot. NIST describes continuous monitoring as maintaining awareness at a frequency sufficient to support risk-based decisions; the word “continuous” does not turn a narrow metric into complete assurance. See the NIST glossary explanation.

The question for the next dashboard review is therefore not simply “How many checks passed?” Ask what had to exist before a check could be counted. The most consequential failure may be the record that never entered the dashboard at all.