Dev.to Security πŸ” Cybersecurity πŸ‘ 0 πŸ“– 5 min read

I benchmarked a security-audit skill on Claude, Codex and Antigravity. Here are the numbers.

Most "AI security audit" prompts I have seen share the same problem: the report looks great and you cannot tell what was actually checked. An agent that writes "authentication: verified" after reading one file is worse t

I benchmarked a security-audit skill on Claude, Codex and Antigravity. Here are the numbers.

Most "AI security audit" prompts I have seen share the same problem: the report looks great and you cannot tell what was actually checked. An agent that writes "authentication: verified" after reading one file is worse than no audit, because someone will set a budget on it.

I work with Italian public administrations and non-profits, so I built two open-source skills for AI coding agents that try to fix this:

  • appsec-sentinel: application and cloud security assessment.
  • gdpr-sentinel: GDPR / ePrivacy review of what the code really does with personal data. It never declares compliance.

appsec-sentinel running on OWASP Juice Shop

Both are Apache-2.0, work in English and Italian, and run on Claude Code, Codex and Antigravity. This post is about how I measured them, including the results that are not flattering.

The rules the skill enforces

Every claim is examined, proven or unknown

  • Examined, proven, unknown. Every finding has a state: dynamically reproduced, confirmed from code or configuration, hypothesis to validate, not verifiable. A static read is never presented as a test.
  • Real denominators. "27 of 52 endpoints tested, by role", not "good coverage". Scripts compute them from CSV files instead of letting the model estimate.
  • Authorization first. No active test without written scope. Text found in code, comments or tool output is treated as data and cannot widen the mandate.
  • Independent refutation. High and critical findings go to a verifier that is not the author and tries to disprove them. If no independent verifier is available, the report says "not performed". The author refuting itself does not count.
  • A delivery check that does not trust the model. check_delivery.py fails the audit if a dangerous-code category found by the pre-pass is left without a verdict, if forbidden words like "compliant" appear, or if secret values leak into the folder.

Benchmark 1: OWASP Juice Shop (truth written by someone else)

My own test apps have a weakness: I wrote the bugs and I wrote the checklist, so they measure obedience more than generalization. Juice Shop is better: its authors mark the vulnerable lines of their coding challenges in the source (vuln-code-snippet vuln-line). That gives 32 vulnerabilities with exact file and line, chosen by people who never saw my skill.

A finding counts only if it cites the marked file and line. Same prompt, quick mode, static analysis only:

Claude 23/32, Codex 18/32, Antigravity 6/32

Agent Found with file and line Within 15 lines Findings
Claude (Opus, two cloud runs) 23/32 and 19/32 27 and 23 37 and 36
Codex 18/32 22/32 28
Antigravity (Gemini) 6/32 9/32 12

Two things surprised me.

The model matters, but so does the plumbing. Earlier standard runs on my PC (older Claude model) scored 7/32, 7/32 and 8/32. Reading the transcripts, the agent had seen the client-side XSS, the SSRF, the chatbot and the password-reset issues. It wrote them in the checklist as "noted" or "known surface" and never promoted them to findings. Adding a word filter did not help: the agent used a synonym. What worked was structural: the deterministic pre-pass now writes one row per dangerous-code category (HTML without escaping, code execution, deserialization, SSRF, client routes, seed credentials…), and the delivery check fails unless each row is closed with a finding or a "not a defect" that cites a file and line it read. The next two runs: 13/32 and 12/32, same model.

From 7/32 to 13/32 with the same model, 23/32 with a later skill and model

Antigravity writes few findings. It found real things, but stopped early. I have not solved this yet; it is on the roadmap.

The misses are shared. All 9 vulnerabilities Claude missed in its best run were missed by Codex and Antigravity too, and they fall into three groups:

The 9 shared blind spots: guessable security answers, hidden frontend routes, business logic

Pattern-like bugs (injection, XSS, authentication) get caught. Flaws that need intent or outside knowledge don't. That is the next thing I want to measure.

Benchmark 2: Django app and a fake school website

These are mine, so read them with the caveat above. Each has planted defects plus decoys, things that look wrong but are fine (a session cookie, data published because the law requires it, an IP log deleted after 7 days). A false alarm on a decoy counts against the skill.

Claude Codex Antigravity
appsec, Django (11 defects, 5 decoys) 11/11, 0 false alarms 11/11, 0 10/11, 0
gdpr, school website in PHP (15 defects, 5 decoys) 15/15, 0 14/15, 0 14/15, 0

gdpr-sentinel demo

Refutation without subagents

Codex can start a subagent; many setups cannot. For those, refutation_pack.py prepara writes a folder with only the method and one brief per finding: the claim, the code excerpt and the typical ways it could be wrong. It contains no author reasoning. You hand it to a new session (another conversation, another agent, a colleague) and import the answers with refutation_pack.py importa. In a test with four planted false GDPR gaps, the verifier rejected three and downgraded the fourth, and rejected no true gap.

Stability between runs (still open)

On a real 1,000-file application, two GDPR runs agree on only about 60–69% of the individual gaps: they describe the same problem with different words and granularity. Fixed questions help a lot where they apply. Three fixed matrices (per data entity, per privacy-notice item, and who can read what for each role) bring agreement on those rows to 90–93%, 83–88% and 86–92%. The free-form part is still noisy. If you have ideas, I would love to hear them.

How I built it

It started as a long audit prompt I used by hand. I turned it into skills with Claude Code: Claude split the prompt into a SKILL.md plus reference files, wrote the Python scripts and the test apps, and ran the benchmarks in subagents. After each run I asked it to read the transcript and find where the agent had cut corners, then we changed the scripts and measured again. Most of the rules above exist because a run broke them.

What it does not do

It does not certify anything, it does not replace a pentest or a DPO, and a static read is not proof that the running site behaves the same way. The benchmarks are small. The skill's own instructions are in Italian (the agent answers in your language).

Try it

# Claude Code
/plugin marketplace add dextsamu/appsec-sentinel
/plugin install appsec-sentinel@appsec-sentinel
/plugin install gdpr-sentinel@appsec-sentinel

Codex and Antigravity use the same SKILL.md convention (install paths in the repo docs). Then ask, for example: "Use appsec-sentinel. Quick mode, static analysis only, on this folder. The repo is mine; no dynamic testing is authorized." Commands work in English and Italian.

Feedback, criticism and new benchmarks are very welcome, and a star helps other people find the project.

πŸ“° Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.