Testing from the outside (2): attack tools and legitimate scanners find different things

This is part 5 of the series "Protect Your Own Service Yourself — Self-Defense Security in the Age of AI." Do the checks you can yourself; when it's hard, EarthLink Network can run the assessment for you, and fix the gaps you can't fix on your own.

Series contents

1. Anyone can attack now, with AI — check your own site first
2. What attackers now hold: when AI runs the attack itself
3. Before you start: the whole picture of checking, and how to test without breaking production
4. Testing from the outside (1): setup and the first run
5. Testing from the outside (2): attack tools and legitimate scanners find different things (this article)
6. Checking the code you wrote: static analysis (coming soon)
7. Checking the parts you depend on: libraries and packages (coming soon)
8. Checking your infrastructure settings for gaps (coming soon)
9. Collect every result in one place and decide what to fix first (coming soon)
10. How to fix what you found: three ways (coming soon)
11. When you cannot fix it yourself: a maintenance contract (coming soon)
12. Keep checking automatically every day — from building to protecting (coming soon)

The short version

We spent 3.3 cents letting a cheap AI model drive an attacker-side tool against a copy of our own login code. In 24 steps it reported six weaknesses, two of them marked critical. Four of the six were false.

In the same week we pointed three more scanners at the same copy: a legitimate code-reading tool driven by the very same cheap model, and two legitimate outside-in scanners that use no AI at all. None of the three raised a single false alarm. One of them is free forever; one ran inside a 14-day free trial; the code-reader cost 45 cents.

So the gap between the attacker-side tool and the legitimate ones was not price, and the AI alone does not explain it. The visible difference was whether the tool re-checks its own findings before reporting them, though the tools also faced different setups. And the kinds of weakness each tool could see depended on how it looked: tools that read the code and tools that probe from outside found different things, and neither replaced the other.

Part 4 showed the setup and the runs. This post sorts the results.

The target throughout is a disposable copy of the login routes of our own service, promptflow. No production system and no third party was touched.

The four tools, and three differences that get mixed up

Three separate things vary across this table: how the tool looks (reading code vs probing a running service), where the tool comes from (legitimate vs attacker side), and whether an AI model drives it. If you compare tools without pulling these apart, you will credit the wrong one for the difference. We pull them apart near the end.

The attacker-side tool: six findings, four false

HexStrike, driven by DeepSeek, chose its own next move at every step until it hit the 24-step cap we set and was cut off before finishing. It reported two critical, two high, one medium, and one low finding, for a total model bill of $0.0331.

We checked each one by hand.

Here is why. The code under test was an exact copy of our login routes. But the identity provider it talks to, and the audit store, were simplified stand-ins, because a sealed test environment needs something to play those parts. Our stand-in identity provider was very lenient: it handed out a token for any authorization code at all. The tool saw that and confidently reported "auth bypass, critical."

That is not a flaw in our product. The lenient part was the stand-in; the real identity provider checks the code. The same thing happened with token refresh: a made-up string like totally-made-up-xyz was rejected, but any string that started with rt_ was accepted, because the stand-in only looked at the first three characters. Three of the four false alarms came from the stand-in. The fourth came from the harness itself sending a broken request; sent properly, logout returned success.

We are spelling this out because recording those four as real would mean inventing a breach that does not exist.

The two real findings matter. Missing security headers can only be seen on a live response, so this was a genuine catch for the outside-in approach. The logout finding shares a root cause with something the code reader found, below.

The way we wired this tool had no step that tries to disprove its own findings. It probed, saw a reaction, and reported. Without a re-check step, false alarms stay in the report alongside the real findings.

The code reader: three findings, zero false, and proof of absence

Strix reads the source instead of probing a running service. It ran on the same cheap model.

First, the step the attacker-side tool did not have. This is Strix the day before the main run (October 19, 2026), on a trial run with a $1 cap against the same login code:

"Two of the specialists' own findings did not survive my review, and I had both withdrawn." Each candidate gets an attempt to disprove it, and only what survives is kept.

On October 20 we ran it on the four login endpoints. This is the findings file it wrote, displayed unchanged in a browser:

Three findings, zero false alarms: two medium and one informational. These severities come from the findings file; the narrative of the same run's report calls the logout issue medium.

• Logout says it succeeded, but the session on the server side is still alive (INFO).
• The audit log records "who did this" from a token payload that was never verified, so an anonymous caller can write any name into the audit trail (MEDIUM).
• When a token refresh fails, the server log gets the first 12 characters of the refresh token, its length, and the raw error from the identity provider (MEDIUM).

The second and third are invisible from outside, and the outside-in tool missed them entirely. It caught part of the first, but blurred by the stand-in's behaviour.

Each finding came with file, line, and either a reproduction or a traced code path. Here is the log leak:

"This could leak" became "this leaked against the stand-in, here is the output." With that much evidence, the decision is easy: fix it. The logout finding had the same treatment: across eight request shapes, seven reported a sign-out that never happened. The audit finding was shown by tracing the code path from the entry point to the audit write, and the tool limited its own claim: the forged name only lands in the audit log and is never used for an access decision, so it called it an audit-integrity problem, not an authentication bypass. Not inflated, not played down.

The same run also ruled whole classes out, with reasons:

"Not found yet" and "cannot happen here" are different statements. The first might just be a miss. The second lets you stop worrying about that area. Outside-in tools can say something close: ZAP ran 120 rules from outside; 117 passed, including SQL and OS command injection, and 3 raised warnings. That is useful, but it only says "we tried and nothing happened." The code reader can say why it cannot happen.

Side by side: different kinds of findings

What one sees, the other cannot. Use only one kind and you will not know what you missed.

Legitimate scanners from outside

That might read as "outside-in tools are noisy." So we also ran two legitimate outside-in scanners on the same copy. Neither uses AI; both walk through a fixed set of checks.

ZAP, free and open source:

The "False Positives 0" line counts alerts a person has marked as false; when we checked the responses ourselves, the header gaps were real.

StackHawk's scanner (HawkScan), in the terminal:

And StackHawk's dashboard:

The 6 and 13 are counts of affected paths; by type there were four, all about response headers, all real, zero false alarms. Note the bottom left of the dashboard: StackHawk has no free-forever plan. It is a 14-day free trial, then from $10 per user per month (official pricing page, October 23, 2026). We ran inside the trial.

Same outside-in approach, no confident false alarms, and they found the same kind of thing the attacker-side tool got right: the header gaps.

Pulling the three differences apart

Three things fall out.

How a tool looks decides what it can find. Code readers found logic, audit, and logging flaws. Outside-in tools found header gaps, whether legitimate or attacker side.

AI alone does not explain the false alarms. Strix and HexStrike ran on the same model. Strix had zero false alarms, HexStrike had four of six. The visible difference was the re-check step, though Strix only read code while HexStrike faced our lenient stand-in, so the setups differed too. ZAP and StackHawk, with no AI, had zero because they run fixed, disciplined checks. The zero-false-alarm results for Strix, ZAP and StackHawk come from runs against the same copy on October 20 and 22, 2026.

The top-right square is empty. Attackers usually do not have your code, so they probe from outside. Reading your own code is an advantage only defenders have. Use it.

Semgrep sits in the top-left as the legitimate, no-AI code reader. It already runs every day in our automated daily checks (CI). Part 6 runs it step by step.

The bill was not the difference

None of the four needed special hardware or a frontier model. That is the uncomfortable part for defenders: an attack loop like this is patience, not intelligence, so a cheap model runs it for pennies. It cuts the other way too. "Legitimate tools are too expensive" no longer holds when a solid one is free and StackHawk has a tier from $10 per user per month.

Three questions to sort real from noise

The tools narrow things down, but a person makes the last call. I ask three questions, in order:

1. Is it actually sensitive, or actually wrong? Something that looks like a key can be a public value.
2. Can it actually be exploited? Theory is not enough; a real weakness lets you do something bad.
3. Can it be reached from outside? If only internal code can touch it, an attacker cannot.

Apply question 2 to the attacker-side tool's "auth bypass" and it fails: it only worked against our stand-in, not against a real identity provider. Most false alarms fall at that step, once you separate "what happened in the test environment" from "what would happen in the real service."

Question 3 mirrors the attacker assumption from part 2: they try every door that can be reached from outside. All three real findings sit on login endpoints anyone on the internet can reach. The log leak is a partial exception: the endpoint is public, but the leaked fragment lands in the server's own log, so it bites when someone can read the logs or the logs themselves leak. It still breaks the rule never to log tokens, so we kept it. That is why medium and informational still made the fix list.

Take both results at face value

Real findings are not a reason to panic. They turn "probably fine" into facts you can act on, and every one of these has a small, local fix (part 10 covers fixes). Proven absence is real reassurance. This run covered four login endpoints, and the absence was proven for those four.

Why you collect and sort

If we had trusted the attacker-side tool, we would believe in two critical auth bypasses that do not exist. If we had trusted only the outside-in scanners, we would have missed the forged audit trail and the token leak. Raw scanner output is not trustworthy on its own. Collect results from different kinds of tools in one place, run each through the three questions, and drop the noise. Part 9 walks through that with real screens.

What to do on your own service:

• Run both kinds: one that probes from outside (ZAP is free) and one that reads your code (Semgrep, in part 6). Do not call one of them "checked."
• Distrust "critical" first. Ask whether it is your code or your test environment. Four of six came from our test setup here (the stand-in server or the harness), and it can happen to you too.
• When choosing a tool, check whether it re-checks its own findings.

Next

Part 6 checks the code you wrote. We run Semgrep, the legitimate code reader from the top-left square, on our own source: install, options, the run, and how to read what comes out, one screen at a time.

In this series, in an age when attack tools are free and one click away for anyone, I show step by step how to check your own service yourself and close the gaps before someone else finds them. The target is always our own service; the series does not cover attacking anyone else's site.
If running the checks yourself is hard, EarthLink Network can do the assessment for you. Found a gap you can't fix? We'll handle the fix. Feel free to reach out.
If this was useful, please like and follow.

Our products are listed at https://www.eln.ne.jp/products

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.