Coding Agents Pass Up to 91% of Tasks in a New Benchmark. At Most 25% Are Also Secure

• Book: Ship It — the pragmatic startup tech stack
• Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

A coding agent rewrote the missing file at the heart of a Java configuration importer. All 14 hidden functional tests passed. All four security tests failed, because the default SnakeYAML constructor it used "honors every !!class tag" in the uploaded YAML (the paper, Section 5.4 and Appendix F.2). The agent was Kimi-K3. Claude Fable 5 and GPT-5.6-Sol parsed the same input with SnakeYAML's SafeConstructor and passed both suites. Qwen3.8-Max and GLM-5.3 made Kimi-K3's mistake.

That case comes from the Last Secure Code Benchmark (LSCBench), submitted to arXiv on October 10, 2026 by researchers from the National University of Singapore and Alibaba Group, including the Qwen team (author list). The headline: six frontier agents pass 73.6% to 91.3% of tasks functionally, and the best of them writes code that is both working and secure on 25.1% (project page).

What LSCBench actually tests

The benchmark has 450 tasks built from 150 real vulnerabilities, across seven languages and 24 CWE categories (arXiv). Of those 150 scenarios, 115 come from public CVEs and 35 are Java applications written by security experts (project page).

The trick is in the prompt. In the authors' words, "Every task asks only for functionality, through a specification that never mentions security" (arXiv). That is how most people prompt a coding agent. You ask for the import feature, not for a deserialization-safe import feature.

Each task runs at one of three sizes: fill in a masked function, write a removed file, or rebuild an emptied source tree from a spec. A task scores only if two hidden suites both pass, functional tests and security tests that replay the original attack. No partial credit, no judge model (project page).

The leaderboard

All six models ran at their highest reasoning effort. Five ran inside Claude Code v2.1.143 and GPT-5.6-Sol ran inside Codex v0.154.0 (README). Numbers from the project page:

Look at the last row. Kimi-K3 has the highest functional pass rate and the lowest secure one. A green test suite told you nothing about which agent was safer.

Across all 2,700 runs, 66% ended with code that works and is still vulnerable (project page). Of the 150 scenarios, 57 were never secured by any model.

Working code, open window: the shape of most LSCBench runs.

Where the code breaks

The project page breaks down how often runs ended both working and secure, all six models combined. By language: C and Python 30%, JavaScript 27%, C++ 19%, Java 18%, Go 14%, PHP 10% (project page).

The weakness-family chart uses a different base, the share of working code that was also secure, and the spread there is wider. Memory safety comes out best at 44%. Injection sits at 28%. Server-side request forgery is at 3% and information exposure at 1% (project page).

The paper's case studies show what that looks like (Appendix F):

• Command injection, Java: four of the six models passed mount arguments to ProcessBuilder as a list. Qwen3.8-Max and Kimi-K3 built a string for sh -c.
• SSRF, Go: in a LocalAI task, GPT-5.6-Sol's GetURI read any path it was handed, file:// URLs included. The reference fix confines reads to the models directory. No model did that.
• Information exposure, Go: in a syft task, GLM-5.3 followed the tag convention of neighbouring structs, so an attestation password showed up in config dumps and SBOM output. All six models made the same choice.

None of these are exotic. A reviewer who goes looking for them can catch all three.

SSRF and information exposure: 3% and 1% of working code was also secure.

One sentence helped. It didn't fix it

The authors reran four of the six models with one sentence appended to every task, asking for an implementation that meets the functional requirements "while also being secure" (Section 5.5). Works-and-secure scores went up:

• GPT-5.6-Sol: 24.7% to 46.2%
• GLM-5.3: 21.3% to 38.2%
• Qwen3.8-Max: 22.0% to 37.6%
• DeepSeek-V4-Pro: 18.0% to 19.1%

Functional pass rates moved by a few points either way. The paper's verdict: "most tasks stay insecure even with the reminder", and SSRF and information exposure stay near the bottom (Section 5.5). Claude Fable 5 and Kimi-K3 weren't in that rerun.

What to do with this on Monday

Add the reminder anyway. The paper appended it to the task. A line in your AGENTS.md or CLAUDE.md is the nearest everyday equivalent, though the paper didn't test that placement. For three of the four models it added about 16 to 22 points.

Then write the test the benchmark writes: replay the attack. If an agent builds a loader that takes a model name, check that it refuses a URL and a path that climbs out of the directory.

from pathlib import Path

import pytest

MODELS = Path("models").resolve()

def resolve_model(uri: str) -> Path:
    if "://" in uri:
        raise ValueError("plain model names only")
    p = (MODELS / uri).resolve()
    if not p.is_relative_to(MODELS):
        raise ValueError("outside the models dir")
    return p

@pytest.mark.parametrize("uri", [
    "ftp://example.com/model.bin",
    "../secrets.txt",
])
def test_rejects_escape(uri):
    with pytest.raises(ValueError):
        resolve_model(uri)

Review agent work for security on purpose, in small pieces. I build Hermes IDE, so weigh this with that in mind. Its Review Desk (⌘G, Ctrl+Shift+G on Windows and Linux) shows an agent's changes by turn or by file and sends a line comment back to the agent that wrote it (README). It won't spot an SSRF on its own, but it splits a 40-file change into turns you can actually read.

The Review Desk (⌘G) in By turn view for a Claude Code task in orders-api: two turns, turn 1's diff of src/db.ts, the changed-files list, and a commit message drafted from the turns.

You can also run the benchmark against your own agent setup. It ships as a Harbor dataset (LSCBench itself is Apache 2.0), and needs Harbor 0.24+, Docker, Python 3.12+ and about 58 GB of task images. The projects contain exploitable vulnerabilities, so the README says to run it in an isolated environment (README). The sanity check first, where reference solutions should score 1:

pip install harbor

harbor run \
  -d last-secure-code-benchmark/[email protected] \
  -a oracle

I haven't run it myself.

Read the fine print before you quote it

The leaderboard tests the model versions available when the paper was written; newer releases may score differently. Every number is pass@1, one attempt per task. Qwen3.8-Max, one of the six models scored, was also the construction agent that wrote the tests and specifications; the authors argue its mid-field result shows no advantage (arXiv). The paper is an arXiv preprint, so treat it as one until someone replicates it.

If this was useful

Vibe-coding a side project is the easy half. Ship It covers the rest of the road from localhost to production for a small team: hosting, payments, auth, CI/CD and monitoring, the places where "it works" and "it's safe to put online" stop being the same thing. If you run several coding agents on that work, Hermes IDE is free, source-available and on GitHub.