Reviewing every file doesn't mean the review is complete

A security report can say it reviewed every file and still leave a long list of questions unanswered. I got that result from a small, deliberately vulnerable fixture, and the gap showed up before I even tried to interpret the findings.

The report counted both files as reviewed. It also listed 52 limitations, including 46 application security controls with an unknown status, and those numbers describe different things, so putting them next to each other doesn't give you a percentage.

There was another problem. The report said the Flask app was instantiated on a line that only imported Flask. I checked the fixture, there's no assignment defining app anywhere in the file. When I loaded the source with stubbed dependencies, execution stopped at the route decorator with NameError: name 'app' is not defined.

The scan had read every file, yet its account of the fixture included a claim the source didn't support. That's a useful reminder when you're deciding how much authority to hand a generated report.

What the scan was asked to do

I ran the Hermes Security plugin against a tiny fixture containing Python source and a dependency list. The fixture was built with suspicious patterns on purpose, including a hardcoded token, a request-driven URL fetch and a shell command assembled from request text. The goal was to see what the review would identify and how it would describe its limits.

The finished report recorded three findings, with two critical and one high in its severity totals. It described the shell-command and URL-fetch paths, and grouped dependency advisories into a separate finding. The URL-fetch section itself called that finding high, so its severity doesn't reconcile cleanly with the summary. The report also kept 11 candidates outside the reported findings, including the token literal. Those retained candidates still need human attention, their position outside the main findings doesn't prove they're harmless.

The report treated its code findings as static evidence. It didn't run the application, send requests to the routes or execute the command path, while the dependency result came from matching package versions against published advisories, with no trace to establish whether each affected function was reachable.

That's an important boundary. A version match gives you something concrete to investigate, it doesn't establish that an attacker can reach the affected behavior in your application.

Files and controls count different work

The completed run had all three planned review packets accepted, both files in the fixture were listed as reviewed with none deferred, and the report still marked review coverage incomplete.

I counted the limitations in the report. Forty controls had no explicit coverage unit, six had a finding mapped to them without the control being closed, and six candidate reports had incomplete sections. The first two groups account for the 46 unknown controls, the last group concerns candidate documentation.

Those categories shouldn't be collapsed. A file count tells you which files entered review. A control status tells you whether a particular security question was closed under the review's method. A candidate with an incomplete report is another kind of unfinished work entirely.

That difference changes what I do with the result. I can use a finding to inspect a specific code path and decide what to fix. I can't use the file count to claim that all the relevant security questions were answered, even for a fixture this small.

The source check changed my read

The incorrect Flask claim was buried in a review note about security misconfiguration, which said the application had been instantiated without certain settings, even though the source line it pointed to was an import and the name used by the route decorators was never assigned.

I checked that behavior by loading the source with stand-ins for its imported dependencies, and the missing name stopped execution at the decorator. The scope was narrow. That check isolates the undefined name, it isn't an end-to-end test of Flask, the routes or the reported attack paths.

The distinction affects the finding. The code contains dangerous command construction and an unrestricted fetch, but the report's route-level attack descriptions assume an application that can register and serve those routes. The static review didn't establish that in this fixture. It also listed deployment conditions it couldn't inspect, such as a proxy or network policy that might affect the claimed paths.

I wouldn't throw away the findings because the fixture fails to load. I'd investigate the dangerous code and fix the fixture before trying to validate the route claims. I also wouldn't repeat the report's account of the app initialization without checking it against the source.

Keep the gaps next to the findings

When I read this kind of report, I want the evidence and its limits close enough together that neither gets lost in the handoff. For a dependency advisory, that means the affected version and the unanswered reachability question. For a code finding, I want the relevant path and a clear indication of whether anybody exercised it.

The review took 2 hours and 8 minutes and still finished with incomplete coverage. That time belongs to this one fixture and this one run, it says nothing about how long another repository will take. It does tell me that accepting every planned packet and reviewing every file didn't automatically close every control.

If you use automated reviews in your own work, how do you keep a report's coverage gaps attached to the findings when someone else picks it up?