LLM Security Needs Capability Scoping, Not Better Prompts

TL;DR — Filtering and classifier-based guardrails treat prompt injection as a text-matching problem, but attackers only need one novel phrasing to win. The durable fix borrows from network security's 'assume breach' posture: tag content by provenance, propagate that taint through the pipeline, and scope what tools an agent can invoke based on where its instructions actually came from. Defense in depth here means the model can be compromised and the system still doesn't act.

Every team building LLM agents eventually ships the same first line of defense: a classifier or a system prompt rule that tries to spot malicious instructions before they reach the model. Every team building LLM agents also eventually watches that defense fail. Not because the classifier was badly built, but because it was solving the wrong problem. Prompt injection isn't a text-matching problem. It's a trust-boundary problem, and you cannot patch a trust-boundary problem with better pattern recognition.

The filtering arms race is unwinnable by construction

Input filtering assumes malicious intent has a detectable signature. It doesn't, because the "attack" is just natural language doing what natural language does: persuading. A filter trained on "ignore previous instructions" gets bypassed by a document that frames the same request as a customer support ticket, a translation task, or a nested quotation. The attacker's search space is the entire space of human rhetoric. Your defense's search space is whatever patterns you've seen in red-team logs. That's not a fair fight, and it never converges.

This is the same lesson network security learned decades ago with signature-based antivirus. Signatures catch yesterday's malware. They never catch a competent adversary who can test against your detector before shipping. The industry's answer wasn't better signatures — it was to stop assuming the perimeter holds and start designing systems that stay safe even when it doesn't. LLM security needs the same pivot. Stop trying to keep bad instructions out of the context window. Start designing so that instructions arriving from the wrong place simply cannot do anything dangerous.

Provenance is the signal you're throwing away

Here's the structural mistake most agent architectures make: by the time text reaches the model's context window, its origin has been flattened. The system prompt, the user's message, a retrieved document, a tool's JSON response, and a scraped web page all become the same undifferentiated token stream. The model has no architectural reason to treat a sentence buried in a PDF differently from a sentence its operator wrote on purpose. Of course it can be talked into treating the PDF's sentence as an instruction — nothing in the representation tells it not to.

The fix is to stop discarding provenance at ingestion and instead carry it as metadata through the entire pipeline, the way a compiler carries taint annotations through a data-flow graph. Every unit of content entering the context gets tagged with where it came from: operator-authored, authenticated user, retrieved document, third-party API response, untrusted web content. That tag isn't decoration — it has to survive concatenation, summarization, and chunking, and it has to be inspectable at the point where the system decides to act. This is more engineering than prompting. It means your retrieval layer, your tool outputs, and your context assembler all need to emit and preserve provenance, not just text.

Capability scoping: the boundary that actually matters

Provenance only pays off if something downstream uses it to make a decision. That something is the tool-calling layer, and this is where defense in depth has to live, because it's the only place where language turns into consequence. A model hallucinating a wrong answer is embarrassing. A model invoking send_email or execute_transfer because a calendar invite told it to is a breach.

The pattern to copy here is capability-based security, the same principle behind OAuth scopes and least-privilege IAM roles. An agent shouldn't have blanket access to every tool it knows about; it should hold capability tokens scoped to the current task, and — critically — those tokens should be gated by provenance. A request to transfer funds or modify a production record should require that the instruction originated from an authenticated, high-trust channel, not from text that arrived via a retrieved document or a tool response. If the triggering instruction's provenance tag says "untrusted web content," the tool-calling layer refuses, full stop, regardless of how convincingly the model argues for it.

This reframes the model's role. The model proposes actions. It does not get to be the sole authority deciding whether an action executes. That authority sits in a layer that can be audited, tested, and reasoned about independently of the model's weights — which matters, because you cannot formally verify what a transformer will output, but you absolutely can formally verify what a policy engine will authorize.

Where jailbreaks fit in this picture

It's worth being precise about scope, because conflating jailbreaks with injection leads teams to build the wrong defense. A jailbreak targets the model's own alignment — getting it to say something it was trained not to say. Prompt injection targets the application built around the model — getting it to act on attacker-supplied instructions it was never meant to obey. Capability scoping and provenance tagging don't make the model immune to jailbreaks; they make jailbreaks less consequential, because even a model that's been talked into agreeing with an attacker still has to pass its proposed action through an authorization layer that doesn't care how persuasive the conversation was. A compromised model proposing an unauthorized wire transfer is a non-event if the tool layer requires a trust tag the injected text can't produce. That's the entire point of defense in depth: no single layer has to be perfect, because each layer's failure is contained by the next one.

Defense in depth is a design discipline, not a feature list

Teams often interpret "defense in depth" as stacking more filters: an input classifier, then a guardrail model, then an output scanner. That's depth in name only — it's the same category of defense repeated three times, and a single clever prompt can slip through all three if they share the same blind spot. Real depth means layers that fail differently. A provenance tag doesn't care about phrasing, so it's not fooled by the rhetorical tricks that fool a classifier. A capability scope doesn't care what the model said, so it's not fooled by a jailbreak that bypasses the guardrail model. Each layer should be defeated by a fundamentally different kind of attack, which is exactly what makes it expensive for an adversary to chain a bypass across all of them.

This costs real engineering effort: instrumenting your retrieval and tool pipelines to carry metadata, building an authorization layer that's independent of the model, and accepting that some legitimate-looking requests will get blocked because their provenance doesn't clear the bar. That friction is the price of containment. The alternative — hoping your filter has seen every phrasing an attacker might try — isn't a security posture. It's a bet you will eventually lose, and in an agentic system, losing means a tool call executed with real-world consequences, not just an awkward chat response.