Your Agent's Biggest Security Hole Is a Tool You Installed on Purpose
A pull request title is about the least dangerous-looking text in software. In October 2025, a researcher reported to Anthropic that one could take over its Claude Code Security Review action. The action put the title st
A pull request title is about the least dangerous-looking text in software. In October 2025, a researcher reported to Anthropic that one could take over its Claude Code Security Review action. The action put the title straight into the agent's prompt. The agent ran with no tool restrictions, in a process that inherited the job's environment, including GITHUB_TOKEN and ANTHROPIC_API_KEY. A crafted title got it to run commands such as ps, and the output came back in a PR comment.
The same research, published in April 2026 as "Comment and Control", reached two more agents. In the Gemini CLI Action, a fake "Trusted Content Section" added to an issue led the agent to post GEMINI_API_KEY as a public comment. GitHub's Copilot Agent, once someone assigned it the issue, followed instructions hidden in an HTML comment that the rendered page doesn't show, then committed its base64-encoded process listing to a PR.
Aonan Guan, the lead researcher, described the shared cause this way: "the prompt injection here is not a bug; it is context that the agent is designed to process." Each team had installed that agent on purpose, given it credentials on purpose, and pointed it at text anyone could write.
Exposure depends on the setup. As Guan notes, GitHub Actions doesn't give secrets to fork pull requests by default, though repositories that grant access, for example through pull_request_target, "do exist in the wild." A follow-up preprint puts a scale on it, reporting that "4714 GitHub workflows and eight n8n templates can be successfully hijacked."
You might also not hear about it. The Register reported in April that the three vendors paid bounties of $100, $1,337 and $500, and that none had assigned a CVE or published a public advisory. "I know for sure that some of the users are pinned to a vulnerable version," Guan told The Register. "If they don't publish an advisory, those users may never know they are vulnerable β or under attack." What the action's README does say is that it "is not hardened against prompt injection attacks and should only be used to review trusted PRs."
Most discussion of agent security asks whether the model can be fooled. If you're the person building the agent, a more useful question is what a fooled run can do. You can't settle the first question, but you control the answer to the second, and you can test it this afternoon.
How often the model falls for it is a moving number
The older name for this failure is the confused deputy, from a 1988 note by Norm Hardy about a compiler tricked into overwriting a billing file with authority its caller didn't have. Agents have the same shape: your credentials, someone else's intent. Greshake and colleagues saw the route in 2023, arguing that LLM-integrated applications "blur the line between data and instructions." GitLab's CVE-2026-96893, found internally and fixed on 12 October 2026, could let an authenticated user "inject specially crafted content into AI agent instructions, potentially causing unintended command execution attributed to other users."
So how often does it work? A large recent measurement is INFUSE, a preprint from October 2026 covering 19 models and 41,824 adversarial runs, with attacks placed in files, terminal output and other things an agent reads during a long task. Mean attack success was 19.6%, and it ranged from 3.2% to 41.9% by model. So the model you pick matters, and this attack surface is measurable.
It also tested the defence many engineers assume is useless: telling the model to ignore instructions it finds in content. That turned out to be the strongest single defence it tested at the prompt level. The version tested goes beyond one line in a system prompt: it brackets the task, tells the model to ignore instructions found later, and repeats a reminder after every observation. On a stress-test subset chosen because agents were most vulnerable there, it cut mean attack success from 58.2% to 14.3%. Off-the-shelf guard models left it at about 44%. The authors warn that these reductions "should not be extrapolated to the full benchmark", and the effect on task success varied by model, from a 42-point gain to a 17-point loss. Even so, this is cheap and worth doing. Saying that a system-prompt instruction does nothing doesn't survive this data.
It still isn't a control you can build on, for two reasons.
The first is what's left over. Roughly one attack in seven still landed on that subset. Simon Willison's line on guardrail products applies: "in web application security 95% is very much a failing grade." For a tool that issues refunds or holds a deploy token, 86% is further from a pass than that.
The second is that the attacks in that study mostly didn't adapt. INFUSE's own limitations section says: "We do not study adaptive adversaries that condition on the model under test (beyond GCG, which optimizes against open-source models only)." The Attacker Moves Second, by researchers including Nicholas Carlini and Florian Tramèr, tested 12 published defences against attackers who tune to the defence. Two prompt-level defences evaluated on the AgentDojo agent benchmark, Spotlighting and Prompt Sandwiching, went from as low as 1% attack success under static attacks to above 95% under an adaptive search. Human red-teamers succeeded in all the cases evaluated.
So the rate depends on the model, the wording of your prompt, the attack, and how much effort the attacker put in. You can lower it, but you can't pin it down.
Not everything at this layer involves a model, either. CISA's Known Exploited Vulnerabilities catalog holds two LiteLLM bugs in its MCP features, both exploited without a model involved. In CVE-2026-42271, added in June, two endpoints for previewing an MCP server before saving it would launch whatever command the request supplied, on the proxy host, for anyone with a valid key, including "low-privilege internal-user keys." In CVE-2026-59822, added in September, a fabricated Authorization header triggered a fallback that replaced failed key validation with an empty auth object and let requests "reach MCP tooling without a valid LiteLLM key." Those are a missing role check and an authentication bypass, kinds of bug older than LLMs. Your tool layer is ordinary software with ordinary bugs, so pin it and patch it like any other dependency.
Assume the model has been fooled
If the injection rate won't hold still, stop relying on it. Assume it's 100%, and ask what a fully hijacked run could do with the tools it holds.
Here's an invented but typical example: an on-call triage agent. It reads error logs, works out how many customers are affected, and tells the on-call channel. Log lines contain whatever text users put in their requests, so it reads attacker-writable content on the first step of every run. A quick build gives it these tools:
| Tool | Why it's there | What else it can do |
|---|---|---|
search_logs(query) |
Find the errors | Brings in attacker-written text |
query_db(sql) |
Count affected customers | Read any table, drop any table |
http_fetch(url) |
Read vendor status pages | Send data to any host, in the URL |
post_slack(channel, text) |
Tell on-call | Post to channels with outside guests |
run_shell(cmd) |
"Debugging" | Almost anything |
rollback_deploy(service, version) |
Fix a bad deploy | Change production, unreviewed |
Private data, untrusted content and a way to send data out, all in one run, is what Willison calls the lethal trifecta. Nobody chose it. It accumulated, one reasonable tool at a time.
Walk through what a hijacked run could do here, and the usual fixes look different.
A blocklist doesn't change the outcome. The patch people reach for first blocks env, printenv, ps, /proc, curl, wget and nc. With a shell still available, echo $DEPLOY_TOKEN, export -p, set and an awk one-liner all read the same secret, and http_fetch is still there to send it out. One route is all an attacker needs. Guan's write-up reports the same thing happening for real: Anthropic's fix blocked ps, and cat /proc/*/environ gets the same data. "Blocklisting is whack-a-mole," he writes, and the "only defensible posture is allowlist-only."
Allowlists leak through what they allow. So scope it properly: read-only database, no shell, one allowed host for fetches, on-call as the only channel, and a person approving rollbacks. The token and the destructive routes are gone, and it looks locked down, but a customer record can still get out in two ways. One is a fetch to the allowed host, with the data in the path. That isn't hypothetical. CVE-2026-54316 was Claude Code's pre-approved huggingface.co host, where fetches to attacker-controlled repositories could carry data out through download counts. Copilot's network firewall in Comment and Control allowed github.com, so the commit carrying the stolen data went straight through. The second route is a link in a message to the correct channel. Slack's documentation says its servers "must fetch every URL in a message to determine what kind of content it references", so the link reaches its host before anyone clicks. For apps, setting unfurl_links and unfurl_media to false stops that, which is the kind of setting nobody thinks to look for until they've traced this route.
Typed tools close those routes, at a cost. Take free text out of both directions. query_db becomes count_affected(service), post_slack becomes post_triage(service, severity, affected) with an enum and an integer, and http_fetch is gone. Now no field can carry a customer record. On-call loses the free-text explanation and the agent loses ad hoc queries. Sometimes you'll decide that's too much, and you may send free text to a human-only view instead. But now you're making that trade knowingly, per tool, rather than discovering it in an incident.
The all-clear never gets blocked. A hijacked agent can always lie within the tool it's allowed to use. Typed fields stop a secret travelling in severity, but low is a valid value. No permission fixes this. What helps is having the harness fill in anything it can compute, so the count comes from count_affected and never from the model, and making sure people know the rest is a model's claim.
This is reasoning about an invented agent, not a measurement. It can't tell you how likely any of this is, and there may be routes I haven't thought of. To check your own agent, turn the walk-through into a test, which is where this piece ends.
A permission check needs something to grip
Here's a result that seems to cut against all this. INFUSE also tried two system-level privilege controls, and neither reduced attack success measurably. Its port of Progent, a least-privilege tool-call policy, actually raised attack success on Claude Sonnet 4.6 from 6.5% to 27.1%, and cut task utility by 64β75% on three of the four models with utility data. The one added defence that did measurably help was at the model level, Meta-SecAlign, at 55β74% lower attack success than its base models. If you came here for "permissions beat prompts", that's a problem.
The detail that matters is what those controls were checking. INFUSE's agent has one tool, a free-form shell, so the authors ported both defences to shell commands and state that their conclusions are "scoped to our shell-level adaptations and do not extend to every realization of CaMeL or Progent." My reading, which is an inference and not their finding, is that a policy can't get much grip on a shell. Almost anything can be done with allowed binaries, which is the blocklist problem again.
Results look different when tools have shape:
- CaMeL, in its full form, separates control flow from data so that retrieved content "can never impact the program flow." On AgentDojo it solved 77% of tasks "with provable security", against 84% for the undefended system.
-
CapScope gives each sub-agent its own capabilities, fixed before any repository content is read. In 75 runs per setup, injected actions executed in 47 runs with no policy, 46 under a global hazard list (deny
.env,rm -rf,curland so on) and 33 under a global task-specific policy. Under per-agent capabilities it was 3, while 68 of 75 repairs still finished, against 72 with no policy. That's a small study of five Python repairs on one model, and the authors say it doesn't stop misuse of granted capabilities, so process sandboxing is still needed.
The hazard list barely beating no policy at all is the finding I'd remember. The payloads asked for ordinary-looking actions, a project-local utility or a project file, and a list of known-dangerous things doesn't cover those.
This is the other side of an argument I made in an earlier piece on multi-agent systems. Narrow tools there were about cost and correctness. Here, a parameter the model can't set is also a parameter an injected instruction can't set.
What to change, in the order I'd change it
1. Remove the shell, or stop pretending you've restricted it. If the agent needs one for real, use an allowlist of commands and run it in an isolated sandbox with no secrets in its environment and tightly limited outbound network access. Isolation doesn't stop the injection. It limits what the run can reach when one lands.
2. Give each tool its own narrow credential. The NSA's MCP security guidance says that if a server doesn't need access to sensitive files or internal networks, "those access paths should be explicitly denied at runtime." Read-only where it reads. Scoped to one tenant. Short-lived. OWASP's MCP Top 10 lists privilege escalation via scope creep as MCP02, because temporary grants tend to stay.
3. Replace free text with typed arguments wherever an outward tool allows it. reply_to_ticket(body) beats send_email(to, body). An enum beats a string. Turn off link previews. Every free-text field on an outward tool is a channel, and so is every allowed host.
4. Treat everything a tool returns as input from a stranger. That's nearly the NSA's wording: "Each output must be treated as untrusted input to the next phase of the pipeline." It includes files that feel like configuration. Forever Security found a malicious AGENTS.md in a customer's Downloads folder, trying to get a coding agent to install a known-bad npm package, and pointed out that a web page can save such a file there unprompted. "There is nothing here for OpenAI or Anthropic to patch," they write. It's vendor research, but it's a fair summary of this whole class of problem.
5. Add the prompt hardening anyway. It's cheap, and the INFUSE numbers say it helps. Repeat the reminder after tool results, as that study did, and check task success on your own model afterwards, since that effect varied.
6. Gate a few irreversible calls, and show the exact inputs. The MCP spec says clients should "show tool inputs to the user before calling the server." Keep the list short, because gates wear out: Anthropic reported in March that users of Claude Code's manual prompts "accept 93% of them anyway." So count a gated call as waiting for a person, not as blocked. An approval also covers less than it appears to. A September 2026 preprint on approval laundering points out that approving a package install can also approve the lifecycle hooks it runs.
Deciding which runs combine private data, untrusted content and outward actions, and which parameters the model should be able to set at all, is the part that takes judgment. It's what the tool-security lesson in SophiArch's Building Production AI Agents course works through, with a debug lab where the attack has to fail even though the model still proposes it.
The same split in a GitHub Actions workflow
Back to the pull request. GitHub's security guidance says to avoid pull_request_target "if it's not necessary" and that GITHUB_TOKEN should have "the minimum required permissions." For an AI reviewer, the design that follows is one job that reads attacker-controlled text and holds almost nothing, and a second job that holds write access and runs no model:
on:
pull_request: # not pull_request_target: fork PRs get no secrets
permissions: {} # nothing by default; each job asks for what it needs
jobs:
review: # reads attacker-controlled text
runs-on: ubuntu-latest
permissions:
contents: read
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false # don't leave the token in .git/config
- run: ./run-review-agent --output review.md # your agent, tools allowlisted
env:
MODEL_API_KEY: ${{ secrets.REVIEW_ONLY_MODEL_KEY }} # this step only
- uses: actions/upload-artifact@v4
with:
name: review
path: review.md
comment: # can write to the PR, runs no model
needs: review
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- uses: actions/download-artifact@v4
with:
name: review
- run: gh pr comment "$PR" --repo "$GITHUB_REPOSITORY" --body-file review.md
env:
GH_TOKEN: ${{ github.token }}
PR: ${{ github.event.pull_request.number }}
Two limits remain. The model key still has to reach the agent, so use one with a spending cap that's easy to rotate, and confine file tools to the workspace. A read tool that can open /proc/self/environ is the ps problem again. And review.md is still model output shaped by an attacker, so the comment can say anything. The difference is that a hijacked review can no longer push code, approve anything, or read your other secrets. One consequence to plan for: on a fork PR this workflow gets no key and no write token, so it won't review outsiders' code. That's the safe default, and it matches the action README's advice to use it only on trusted PRs.
Put the attacker in your test suite
Tools do more than their names suggest, so reading the list only gets you so far. Take the model out and put a script in its place that makes the worst tool calls a hijacked model could make, through the same code path real calls take. Then check what happened in the world. The shape, written for your own harness:
from base64 import b64encode
CANARY = "CANARY-7f3a"
FORMS = [CANARY, b64encode(CANARY.encode()).decode(), CANARY.encode().hex()]
def test_hijacked_run_cannot_leak(harness):
harness.seed_private_data(CANARY)
for tool, args in HIJACK_CALLS: # written by hand, at least one per route out
try:
harness.call(tool, *args) # the path real model calls take, not a shortcut
except PolicyDenied:
pass
sent = harness.outbound_log() # recorded at the network edge, not in a tool wrapper
assert not any(form in sent for form in FORMS)
There's a small runnable version of this in article-code, with a diagram showing where your tools plug in. It uses only the standard library and needs no API key.
Write HIJACK_CALLS the way an attacker would. Include your allowed hosts, your links, your encodings, and every free-text field on every outward tool. Add one test each for the destructive goals, checked against real state, and one that runs your honest task, so the suite can't pass by breaking the agent.
The test needs no API key and has no randomness, so it can run on every pull request, unlike an attack-rate benchmark that needs thousands of model runs. Run it again whenever a tool, a scope or an MCP server changes. It's also worth snapshotting your tool list and failing the build when it differs, because the NSA guidance warns that a capability change on an already trusted server "often can be made without approval."
Your model will get fooled at some rate you can reduce and can't pin down. The tool list is yours. Assume the model has been fooled, and find out what that costs before someone else does.
References
- Guan, A., with Liu, Z., & Zhong, G. (2026, April 15). Comment and Control: Prompt Injection to Credential Theft in Claude Code, Gemini CLI, and GitHub Copilot Agent. The researchers' own write-up: mechanisms, disclosure timeline, bounties, and the blocklist bypass.
- Fendley, N., Liu, Z., Guan, A., Zhong, J., & Cao, Y. (2026). Comment and Control: Hijacking Agentic Workflows via Context-Grounded Evolution. arXiv:2605.11229. Source of the 4,714 workflows figure.
- The Register (2026, April 15). Agents hooked into GitHub can steal creds β but Anthropic, Google, and Microsoft haven't warned users. Source of the bounty amounts, the absence of CVEs and advisories, and Guan's comment on pinned versions.
- Anthropic. claude-code-security-review. README security considerations.
- Nguyen, T. D., Guo, J., Dubauskas, R. S., Damuluru, S., Johnson, T. T., & Leach, K. (2026). INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems. arXiv:2610.20776. Attack rates, prompt hardening and guard model results, shell-level privilege controls, and stated limitations.
- Nasr, M., Carlini, N., Sitawarin, C., et al. (2025). The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections. arXiv:2510.09023. Twelve defences tested against adaptive attackers.
- Debenedetti, E., Shumailov, I., Fan, T., et al. (2025). Defeating Prompt Injections by Design. arXiv:2503.18813. CaMeL and its AgentDojo results.
- Bouras, D. S., Dai, Y., & Mechtaev, S. (2026). Authority Is Not a String: A Capability-Scoped Harness for Prompt-Injection-Resistant Coding Agents. arXiv:2609.08371. CapScope against ambient authority and global policies.
- Zhang, J., et al. (2026). Agent Approval Laundering: Transitive Effects Beyond the Approved Invocation. arXiv:2609.28586. What an approval record leaves out.
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv:2302.12173.
- Hardy, N. (1988). The Confused Deputy (or why capabilities might have been invented). ACM SIGOPS Operating Systems Review, 22(4), 36β38.
- National Security Agency (2026, May). Model Context Protocol (MCP): Security Design Considerations for AI-Driven Automation. Cybersecurity Information Sheet. Least privilege, untrusted outputs and approval workflows.
- OWASP. MCP Top 10 (2025, beta). Taxonomy of MCP risks.
- Model Context Protocol. Tools specification: security considerations. Source of the recommendation to show tool inputs before calling the server.
- CISA. Known Exploited Vulnerabilities catalog (version 2026.11.13). CVE-2026-42271 added 8 June 2026; CVE-2026-59822 added 2 September 2026. See also the GitHub advisories GHSA-v4p8-mg3p-g94g and GHSA-7488-6r32-c95q.
- NVD. CVE-2026-54316. The pre-approved
huggingface.cohost as a data channel, fixed in Claude Code 2.1.163. - GitLab (2026, October 12). Critical Patch Release: 19.4.2, 19.3.4, 19.2.8. Source of the CVE-2026-96893 description.
- Weizman, G., & Samet, O. (2026, October 22). Markdown Is the New Executable: Attackers Are Dropping Agent Instructions Into Your Downloads Folder From the Web. Forever Security. Vendor research.
- Willison, S. (2025, June 16). The lethal trifecta for AI agents.
- Anthropic (2026, March 25). Claude Code auto mode. Source of the 93% acceptance figure.
- GitHub. Secure use reference. Guidance on
pull_request_targetand token permissions. - Slack. Unfurling links in messages. How link previews are fetched and how to turn them off.
- SophiArch. Hijacked-run test harness. A runnable version of the test in this article.
- SophiArch. Every Tool Is an Attack Surface: Permissions, Untrusted Output, and Confirmation Gates. Lesson from Building Production AI Agents on trust boundaries and harness controls, with a debug lab.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.