L3: I built continuous runtime monitoring because certification is point-in-time, attacks are runtime
Four independent reviewers said the same thing: "Certification is point-in-time. Attacks are runtime." — @correctover (CrewAI), @wrencalloway (dev.to), @mads_hansen (dev.to), @mayank609 (CrewAI) When 4 people indepen
Four independent reviewers said the same thing:
"Certification is point-in-time. Attacks are runtime."
— @correctover (CrewAI), @wrencalloway (dev.to), @mads_hansen (dev.to), @mayank609 (CrewAI)
When 4 people independently identify the same gap, it's not a gap — it's THE problem. So I built L3.
The gap
My 8-layer Sentinel pipeline audits skills at import time:
- L1.5-L1.8: static analysis (metadata, semgrep, secrets, malware patterns, malware families)
- L2: gVisor sandbox (runs the skill once, captures a behavior baseline)
But after certification, the skill can change:
- A config drift changes
allowed_pathsfrom/datato/ - A supply chain update injects a new payload
- A compromised credential lets it exfiltrate data
- New tools appear in the tool catalog
Static analysis can't see these changes. L2 captured a snapshot. Neither catches drift.
L3 — Continuous Runtime Monitoring
L3 re-runs skills in the sandbox on a schedule (weekly via GitHub Actions) and compares runtime behavior against the L2 baseline. If behavior drifts, the skill is flagged.
6 drift detection types
| Type | Severity | What it catches |
|---|---|---|
| TOOL_CATALOG_NEW_TOOLS | critical | New tools appeared after certification |
| TOOL_CATALOG_CHANGED_SCHEMA | critical | Existing tool changed its inputSchema |
| SUPPLY_CHAIN_GIT_SHA_CHANGED | critical | Git commit changed — repo was updated |
| SUPPLY_CHAIN_NPM_VERSION_CHANGED | high | npm package version bumped |
| NETWORK_NEW_DOMAINS | high | Contacting domains not in baseline |
| CONFIG_PERMISSIONS_EXPANDED | critical | allowed_paths or scopes expanded |
| CREDENTIAL_NEW_ENV_ACCESS | high | Accessing env vars not in baseline |
| PROCESS_NEW_SPAWNS | high | Spawning processes not in baseline |
How it addresses each attack vector
"A config drift changes allowed_paths from /data to /"
→ L3 compares current permissions against baseline. If paths expanded → CRITICAL alert → skill re-quarantined.
"A supply chain update injects a payload"
→ L3 checks git commit SHA and npm version. If changed since certification → CRITICAL alert → skill must be re-audited.
"A compromised credential lets it exfiltrate"
→ L3 re-runs sandbox with --network none. If baseline showed 0 credential accesses and L3 shows N → HIGH alert.
What L3 doesn't solve (honest)
- Client-specific behavior: a skill that returns clean tool descriptions to the scanner but malicious ones to a real agent. L3 mitigates by running with multiple User-Agent strings, but it's not a full solution.
- Zero-day protocol exploits: L3 catches behavioral drift, not protocol-level vulnerabilities.
Architecture
The full pipeline is now 9 layers:
L3 results stored in _data/l3_results/{skill_id}.json — publicly auditable. If a skill drifts, it's flagged at /api/security?view=quarantine with reason: 'l3_drift_detected'.
Try it
- Security overview (now shows 9 layers): https://marketnow.site/api/security
- Trust page: https://marketnow.site/trust
- GitHub: https://github.com/edgarfloresguerra2011-a11y/marketnow
— Edison Flores, AliceLabs LLC
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.