Vulnerability assessment vs penetration testing: what two scanners found in 27 known flaws

A vulnerability scan is an automated tool that checks a system against known weaknesses, and a penetration test is a person who tries to break in the way an attacker would and shows what they reached. To see the gap, we built 27 web flaws on purpose and ran OWASP ZAP and Nuclei against them in six setups. The best setup found 3 of the 27, found 1 in part and missed 23. All six setups together found 4, found 2 more in part and missed 21 outright. The 21 include SQL injection, five login-flow flaws and six authorization flaws.

We use "vulnerability scan" for the automated tool run and "vulnerability assessment" for the wider job of listing and rating weaknesses, which usually starts with a scan. We tested the scan part: two scanners, no person steering them. The figures below describe scanners, and say nothing about a full assessment that adds a human review of the results.

The test apps are the 27 fixtures in our web-security-fixtures repository (commit 2c4a833). Each one is a small local web app with one known flaw and a fixed twin. We ran OWASP ZAP 2.17.0 and Nuclei v3.11.1 against them on 11 November 2026, inside a sandbox that allows loopback traffic only.

On this page: Results by setup · Results by fixture · What the scanners did well · What scanners miss and why · Noise and false criticals · Scan or penetration test · How we ran it · Caveats · Data

Results by setup: 6 setups, 27 flaws

We ran six setups. "Seeded" means the scanner got every route of the app and a low-privilege token. "Cold" means it got only the root URL. "Found" means the tool named the flaw class the fixture was built to show. "Partial" means it reported a related symptom, such as a missing header, but not the flaw itself.

Flaws found, partly found and missed, by scanner setup (27 flaws)

The best single setup was the ZAP active scan on the seeded app: 3 flaws found, 1 partial, 23 missed. Across all six setups, 4 flaws were found at least once: HSTS, the HttpOnly cookie flags, the open redirect and clickjacking. Two more were found in part: CSRF and GraphQL. The other 21 were missed by every setup.

The row-level security fixture is a PostgreSQL setup with no HTTP surface. A web scanner has nothing to aim at, so we count it as a miss in every row.

Results by fixture

Each fixture name links to the post that reproduces that flaw by hand. The columns show the seeded runs, the best case for each tool.

Result for each of the 27 fixtures (ZAP active scan and Nuclei default templates, seeded runs)

Two cells need a note. The Nuclei security-txt template reports that a security.txt file exists. It fired on the broken server and on the fixed one alike, so we graded it as missed. The Nuclei open redirect hit in the cold run came from a template for CVE-2018-14474 (an old forum product) that happens to match /login?next=. The default seeded Nuclei run did not find the open redirect, and the -dast run did.

What the scanners did well

Both tools found the flaws they exist to find. A missing Strict-Transport-Security header, a cookie without the HttpOnly flag, a page that can be framed and an unvalidated redirect target are all visible in a single response or a short probe, and each tool reported them.

Seeding mattered. ZAP found the cookie flags only after we sent the login request, because its cold run saw a 404 at / and nothing else. ZAP found the open redirect only in the active scan, not in the baseline report. In the cold runs Nuclei found 3 flaws (HSTS, clickjacking and the open redirect) and ZAP found 1 (HSTS).

A scanner also checks things nobody repeats by hand on every release: server banners, missing headers on every page and known CVEs in off-the-shelf software. We built the fixtures ourselves, so they hold no known CVEs, and this test says nothing about how well the tools match them. That matching is what the Nuclei template library is built for.

What scanners miss and why

A scanner sends requests and judges the responses. It can tell that two responses differ. It cannot tell which response the app should have sent, because it has no copy of the rules. Every miss below comes from that gap.

Authorization. Six fixtures break an access rule: api-authorization-testing, idor-vulnerability, broken-access-control, tenant-isolation, owasp-api-top-10 and mass-assignment. No setup found any of them. In api-authorization-testing we gave both tools Bob's token and the URL of Alice's note. The server returns Alice's note to Bob, and nothing in that response marks the note as someone else's. Finding that needs two users and a rule about who owns what.

Login and session logic. Five fixtures break a step in a flow: 2fa-bypass, credential-stuffing, pkce, session-fixation and api-rate-limiting. These flaws sit in the order of steps and the limits between them, such as a password-only session that can already read the account, a rate limit keyed on X-Forwarded-For, or a session id that survives login. No setup found any of them.

Race conditions and business logic. The race-condition fixture has a single-use coupon that two simultaneous requests can both redeem. One request at a time looks correct, and neither tool reported anything. mass-assignment, tenant-isolation and credential-stuffing share the trait: the request is valid, and the flaw is that the app allows it.

Injection and file handling. sql-injection is a class scanners are built for, so we checked the miss by hand. ZAP sent 1,290 requests and reported nothing. A single hand-written request to the search parameter returned every row. We did not run a dedicated injection tool such as sqlmap, so this result applies to ZAP and Nuclei only. path-traversal, zip-slip, host-header-injection and redos were also missed by every setup. timing-attack needs many timed requests compared statistically, and for security-txt the Nuclei template only reports that the file exists.

Blind SSRF. ssrf needs a callback server that the target can reach, so the tool can see the request arrive. Our sandbox blocks all non-loopback traffic and we started Nuclei with -ni, which turns off its callback service. Read the SSRF row as "could not confirm here", and not as proof that the scanners cannot find SSRF.

These are the flaws that a penetration test is built around. A tester signs in as two users, tries one user's token on the other's objects, repeats a request in parallel, reads the business rules and builds a request that the app should refuse.

Noise and false criticals

On the seeded runs of the 26 fixtures with a web surface, ZAP's active scan raised 84 distinct alerts (counted per fixture), and 79 of them were not about the flaw the fixture was built to show. Nuclei printed 691 result lines. They cover 288 distinct template and matcher pairs (counted per fixture), and 281 of those were off-class. Much of it is low or informational: version banners, missing optional headers, technology detection. On the hsts fixture, Nuclei also matched five JARM fingerprint templates named after command-and-control frameworks, for example mythic-c2-jarm, at info severity.

Two results were rated as serious and were wrong. On 2fa-bypass, Nuclei reported chamilo-lms-sqli at critical severity and ZAP reported Spring4Shell at high severity. The fixture has no SQL and no Spring. Both appeared on the vulnerable server and on the fixed one. Someone has to catch results like these before anyone files a ticket.

Scan or penetration test

Run a scanner often. It is cheap to repeat, it catches headers, cookie flags, exposed versions and known CVEs, and it fits in a release pipeline. Treat its output as a list to triage.

Commission a penetration test when the app has logins, roles, money or data that belongs to more than one customer. Those are the places where the scanners missed the most. A tester also reads the scanner output and drops the false criticals, so you get findings you can act on. Our scope pages describe the work: web application penetration testing and penetration testing.

How we ran it

The exact commands, scanner versions and configs are in the original post and its data files.

Caveats

• We adjusted the grading rubric after seeing the output. We wrote a first version before the scans and changed some patterns once we had read what the tools printed. The CSV keeps the raw alert and template names so you can check each grade.

• The active scan had a 6-minute cap and a 1-minute cap per rule. A longer run could find more.

• The fixtures serve one connection at a time, so one idle scanner connection stalled the whole app. We ran them with a threaded launcher that swaps in a threaded HTTP server and counts requests. The flaw logic is unchanged. For the HSTS fixture, the launcher also moves the TLS handshake into the request thread.

• The row-level security fixture has no HTTP surface. It counts as a miss, so a web scanner could find at most 26 of the 27 flaws.

• Blind SSRF needs a callback server. The sandbox blocks one, so the SSRF result is unconfirmed.

• For HSTS we graded only the vulnerable mode, which sends no header.

• These are small test apps with one flaw each, scanned with default settings by two tools. A real application is larger and has real dependencies. We make no claim about other scanners, paid or free.

Data files

• results.csv: one row per fixture, with the grade for every setup, alert and finding counts, and the alert and template names.

• counts.json: the count of found, partial and missed for each setup.

• run.sh: the reproduction script.

• README.txt and SHA256SUMS.

• web-security-fixtures on GitHub.

First published on excessionlabs.ai.