How We Tested Railo on 11 Public Repositories

Railo is a GitHub Action that finds security issues in a pull request and proposes the fix. The fixes are rule-based code transforms, not model output: the same input always gives the same patch, and every patch is a diff you can read.

Until this month we measured fix quality on a benchmark of examples we wrote ourselves. That number looked good, and it was also the easiest test we could have chosen. So we ran the exact pipeline the Action runs on eleven public repositories we did not write, and read every fix it proposed.

The short version:

• 923 findings across the eleven repositories.
• 162 proposed fixes. We read all of them.
• 95 of those 162 should not have been proposed. Most were edits to vendored libraries, test data and seed data.
• None of the 162 broke the syntax of a file. Wrong is not the same as broken, and wrong was the problem.

This post is the list of what went wrong and what we changed. We think it is more useful than another benchmark score.

How we ran it

We picked two kinds of repository on purpose:

• Deliberately vulnerable apps, where real findings are guaranteed: OWASP NodeGoat, OWASP Juice Shop, PyGoat, DVWA, OWASP WebGoat and govwa.
• Well-maintained projects, where almost every finding should be noise: Flask, Express, Gin, Slim and Spring Petclinic.

For each one:

1. Shallow clone, and scan the whole repository rather than a pull request diff.
2. Run the Action's own code path: load the config, scan with the pinned Semgrep version (1.102.0), generate fixes, collect the changed files.
3. Parse every changed file before and after the fix with the language's own tooling: Python's compile, the TypeScript parser for JavaScript and TypeScript, gofmt -e, php -l, and javac stopped after parsing. A fix that adds a parse error counts as breaking the file.
4. Read every proposed fix by hand.

One thing we switched off: the Action normally tries the project's own test command before and after each fix. In this harness we did not execute anything from the repositories.

We now re-run this harness, and read the diffs, before every Action release.

Results

Two things stand out: almost all the wrong fixes landed in code the project did not write or does not ship, and on the well-maintained projects the right number of fixes was zero.

Findings, then fixes proposed before and after our changes:

• WebGoat: 381 findings, 95 fixes before, 15 after
• Juice Shop: 216 findings, 11 before, 6 after
• PyGoat: 132 findings, 16 before, 16 after
• DVWA: 94 findings, 27 before, 24 after
• NodeGoat: 24 findings, 10 before, 6 after
• govwa: 8 findings, 0 before, 0 after
• Flask: 22 findings, 1 before, 0 after
• Gin: 20 findings, 1 before, 0 after
• Express: 14 findings, 0 before, 0 after
• Spring Petclinic: 9 findings, 1 before, 0 after
• Slim: 3 findings, 0 before, 0 after
• Total: 923 findings, 162 fixes before, 67 after

WebGoat alone accounts for 80 of the 95 removed fixes, and about 77 of those were in two vendored editor files.

A follow-up release then cut noise on the maintained projects: Flask went from 22 findings to 11, and NodeGoat from 24 to 21, with one more correct fix (7 instead of 6).

What we got wrong

Every problem below was a fix that parsed, looked plausible in isolation, and was wrong in context. Each case now goes to a person instead: the finding is still reported, but no change is proposed. Each has a regression test.

1. Rewriting vendored libraries

WebGoat ships the Ace editor in static/js/libs/. Our hard-coded-secret rule saw lines like token: "keyword" in Ace's syntax-highlighting rules and turned them into process.env.TOKEN. About 77 fixes, all nonsense. Juice Shop's vendored three.js was edited too.

Now: lib/ and libs/ under static asset folders, minified and bundled files, and any JavaScript file over 5,000 lines are treated as third-party code.

2. Treating test data as production

Gin keeps a certificate and key in testdata/ for its TLS tests. We "fixed" them.

Now: testdata, __tests__, __mocks__, mocks, e2e, cypress, bower_components and similar folders are skipped.

3. Collapsing seed data into one variable

NodeGoat's db-reset.js creates three users with different passwords. We gave all three the same process.env.PASSWORD. Juice Shop's users.yml was worse: 27 users, all set to the literal string ${PASSWORD}.

Now: several different values for the same key in one file mean data, not a secret.

4. Writing ${VAR} where nothing substitutes it

We rewrote a value in Petclinic's Kubernetes manifest, and in Juice Shop's YAML data, to ${VAR}. Neither file is ever processed by anything that expands it, so the app would have read the literal text.

Now: only Docker Compose files and Spring Boot config files get ${VAR}, because those are the formats that substitute it.

5. process.env in browser code

Angular components and in-browser lesson scripts got process.env.SECRET. There is no process.env in a browser.

Now: browser code, detected by path, file extension, or use of document, window or localStorage outside string literals, is not rewritten. (The first version of this check mistook a server config file that mentioned document.write inside a string for browser code. Fixed in the follow-up release.)

6. Names that only look like secrets

A smart-contract address called BeeTokenAddress. DVWA's $token = "userid:2", which is an attack payload, not a credential. Ace's token types again.

Now: names ending in address, id, url, host, name, type and similar are left alone, as are token values that are not shaped like a credential, unless the value matches a known provider format (AWS, GitHub, Slack, Stripe, Google keys), which is caught anywhere.

7. innerHTML to textContent on HTML that is meant to be HTML

Juice Shop renders Markdown to HTML with snarkdown and assigns it to innerHTML; WebGoat builds tables as HTML strings. Switching to textContent would print the tags as text.

Now: values that are HTML by construction are not rewritten.

8. Flask Markup(value) to escape(value)

This double-escapes markup that Flask already treats as safe.

Now: Markup() with no request data in it is not rewritten.

9. "Escaping" a JSONP callback with htmlspecialchars

It makes the code look handled without making the callback safe.

Now: this goes to a person.

What is still not good enough

• Noise on well-maintained code. After the follow-up release Flask still produces 11 findings, all of which we read as noise: 7 from a generic open-redirect rule, 2 from a compile() rule, one Markup and one SHA-1. No fix is proposed for any of them, but in warn mode they still show up on Flask's pull requests. That's the next thing we are working on.
• Partial detection. In DVWA's view_help.php, the if branch is flagged and fixed and the identical else branch is not.
• Context a rule cannot see. Making Juice Shop's token cookie httpOnly is correct in general and breaks Juice Shop, whose frontend reads that cookie. We offer it as a suggestion for review, not an automatic change, which is where it belongs.
• Not automated, by design. NodeGoat's eval(req.body.preTax) is detected and explained, but a person writes the fix. Some fixes need a person, and we would rather say so than guess.

If you maintain one of these projects and think we have something wrong, we would like to hear it.

Try it

The Action is free to use and needs no account or key. It covers Python, JavaScript and TypeScript, Java, Go and PHP. Start in warn mode: it comments the proposed fixes on the pull request and changes nothing.

- uses: actions/checkout@v4
  with:
    ref: ${{ github.head_ref }}
    fetch-depth: 0
- uses: IWEBai/railo-action@v2
  with:
    mode: warn
    base_branch: ${{ github.base_ref }}
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The full workflow, with the permissions it needs, is on the Action's page. If you run it on your own code and it proposes something wrong, please open an issue with the diff: that is exactly the kind of case this post is made of.

If you would rather not fix it yourself

The Action covers the common, pattern-based findings. Plenty of real findings are not that: a pentest report with forty items, a scanner export before an audit, a customer's security questionnaire. For those we run a Fix Sprint: you send us the findings, and in one week you get a pull request per weakness. Some of the fixes Railo writes; the rest an engineer writes by hand, and the evidence pack says which. Every fix is rescanned, and your team reviews and merges everything. It is a fixed price of $1,500 for one repository.

The result is an evidence pack: one row per finding, with the file, the fix, who made it, the rescan result and the merge date. We ran a rehearsal on OWASP NodeGoat, and the finished pack is public: railo.dev/sample-evidence.

If you have findings to close, email [email protected] with the subject "Railo Fix Sprint", or book 15 minutes at cal.com/railo.

Zarif Latif, founder, Railo