Dev.to Security 🔐 Cybersecurity 👁 0 📖 2 min read

I put a security scanner inside Jira as an AI agent — the trick was keeping the judgment out of the prompt

Last week I shipped Bastion: a Rovo agent inside Jira that scans any issue for CVE mentions and vulnerable-package references, enriches them against the live NVD API, and produces a 0–100 risk score with recommendations.

Last week I shipped Bastion: a Rovo agent inside Jira that scans any
issue for CVE mentions and vulnerable-package references, enriches them
against the live NVD API, and produces a 0–100 risk score with
recommendations. It files the remediation ticket without leaving the
workflow where the work already lives.

The interesting engineering decision wasn't the LLM part. It was what
I deliberately didn't let the model decide.

The split: model narrates, code decides

Bastion is an agent, but the security judgment is a pure function:

score += cveData.filter(c => c.severity === 'CRITICAL').length * 40;
score += cveData.filter(c => c.severity === 'HIGH').length * 20;
score += Math.min(cves.length * 5, 20);
return Math.min(score, 100);

CRITICAL findings are worth 40 points, HIGH 20, each additional unique
CVE adds 5 (capped at 20), hard ceiling at 100. Severity bands: 70+
CRITICAL, 40+ HIGH, 15+ MEDIUM. The model reads the issue text, the
scanner extracts CVE IDs with a regex, NVD returns the CVSS data, and
this arithmetic produces the number. The LLM writes the explanation —
"this looks like Log4Shell exposure, patch this sprint" — but it cannot
inflate or deflate the score. Judgment probabilistic, boundary
deterministic: same lesson as prompt-injection defense, applied to risk
scoring.

The unglamorous parts that actually mattered

Rate limits are the real API. NVD's public API returns 429/403 under
load. The client retries once after a 6-second backoff, caches each CVE
for 6 hours, and spaces lookups 1.2 seconds apart. Without that, the
"live lookup" demo dies on the third question.

Extraction is regex, and that's honest. CVE IDs are
CVE-\d{4}-\d{4,7} — trivially extractable. Package names are a fixed
list (log4j, openssl, struts, the usual suspects). The honest
limitation: a vulnerable dependency not on the list gets missed. I
documented it in the README rather than pretending coverage we don't
have. An agent that silently under-reports is worse than one that
admits its blind spots.

Forge constraints shape the design. No direct DB, no long-running
processes — so state lives in Jira itself (labels, issue fields) and
the daily sweep is a scheduled trigger, not a daemon. If your platform
has constraints, let them pick the architecture instead of fighting
them.

What I'd do differently

Package detection should come from the project's dependency manifest
(package.json, requirements.txt, pom.xml) when the issue links a repo —
regex on prose will always have false negatives. And the scoring weights
(40/20/5) are defensible but arbitrary; they should be tuned against
real triage outcomes, which is exactly what a vuln-triage scoring
worksheet is for.

Where this connects

  • Bastion is open source: github.com/wesleygitari2-cloud/bastion — manifest, resolver, security-core, and the React panel are all there.
  • Free: the Security Triage Checklist covers the manual version of this workflow (KEV/EPSS checks, scoring, SLAs) — useful even if you never ship an agent.
  • Paid: the Vuln Triage System ($14, now also in Spanish at $12) is the scoring model + remediation policy + reporting templates Bastion's arithmetic is a simplified version of.

Store link auto-updates; pay in BTC, no account, instant download.

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.