Threat-model an agent that can spend or sign

If an agent can move money or sign a release, write the attacker's yield before you pick a model. Charles Guillemet, CTO of Ledger, says securing that agent with software alone is not possible. Alignment fails when ambiguous language meets a non-deterministic agent, so the action can differ slightly from the request. Secrecy fails when the agent holds credentials, because prompt injection can pull them back out. Threat-model the prompt, the model, and the host, then match each loss to a tier of control.

Securing an agent is not something possible. People are telling differently. I think they are either a little bit too optimistic or maybe they don't understand what we are talking about.

Charles Guillemet, CTO of Ledger, on Chain of Thought ep 65

What does a compromised prompt, model, or host hand over?

The prompt includes data the agent reads. Prompt injection hides instructions in that data. Guillemet describes an account on X that was prompt-injected by a tweet and lost money. If the agent holds API keys, or the 24 words behind a crypto wallet, the attacker who owns the prompt can ask for the secret and for any spend that secret authorizes.

The model fails on a different path. Jailbreaking persuades the model to break its safety training with roleplay, a hypothetical, a translation task, or encoded text. Prompt injection smuggles instructions in through data. Jailbreaking persists because safety is trained behavior, not a hard constraint, so another framing can get past it. Filters on input and output, running outside the model, catch a jailbreak the model accepted. They do not prove that an intent maps to the action you meant. A high-level prompt can still become an action that is slightly off, and a signature that is slightly off can be a loss.

The host runs the check. Guillemet describes a typical breach as another program changing how the main program runs. A compromised host can make the policy engine approve an intent the written rules reject. A key on that host can be stolen. A key in hardware that auto-signs can stay in the box while the attacker still collects signatures.

One of these flaws can look too small to schedule. Vulnerability chaining strings low-risk flaws into a high-risk breach. Teams patch the criticals, work down the highs, and let the low-risk findings pile up, because each one alone does not get an attacker much. Guillemet says finding vulnerabilities is now very easy: you just prompt for them. If findings are that cheap, I would assume chaining them gets cheaper too. On an agent that signs, a low finding on the path to the signer belongs in the threat model.

Where does a software policy stop the attacker?

Guillemet's move is to delegate rights. You define policies, and the agent operates inside them. A policy engine turns an intent into a checked action. Formalization is the hard part: the intent has to map to one specific action. He says that mapping is doable and not simple.

Use a software policy when the loss is capped and low-value. On the episode I floated a five-bucks-a-day case. Guillemet's tiering covers it: low-value assets can be secured by software and access control, and friction rises with value. He would not hand his own crypto private keys to an agent, because a prompt injection could make it reveal them. I would keep anything like a treasury, payroll, or a firmware release far above that tier.

A software policy engine checks proposed actions. It does not prove the engine ran as written, and if the key sits inside the agent, a prompt injection that extracts it can bypass the policy entirely. His bottom tier is software and access control, for assets with low value in confidentiality and integrity.

Which tier matches the loss you wrote down?

There is no single design. Classify the asset by value and match the friction to it. The four tiers are his, and he says another CTO might draw them differently. The function below is my sketch for a design review, not Ledger's code.

Low value gets software and access control. More value gets hardware that auto-signs, such as keys in a HashiCorp Vault. He splits that tier: the keys are secure, and the execution is not, so a bad host can still ask for a signature. Higher value gets hardware with a screen, where you verify what you sign. The top tier is the same hardware in a multi-signature setup that needs a quorum. Ledger signs its own firmware releases that way, so one person cannot approve the release alone. For his own crypto, he says he would use something like that top tier.

# Sketch only, for a design review.
def tier(loss: str) -> str:
    if loss == "low_value_account":
        return "software_and_access_control"
    if loss == "keys_held_execution_on_host":
        return "hardware_autosign"
    if loss == "human_must_read_the_intent":
        return "hardware_with_screen"
    if loss in {"firmware_release", "treasury", "crypto_you_cannot_lose"}:
        return "hardware_screen_and_quorum"
    raise ValueError("name the loss before you name the tier")

When you approve a policy, sign it on hardware, such as a Ledger device, so the engine can tell the policy came from you. Then prove the engine ran as written. A secure enclave, a TEE, can run the engine. A zero-knowledge proof can guarantee the computation ran correctly without revealing it. Guillemet says Ledger does not use zero-knowledge proofs at the device level yet. A design in progress there would run the policy engine inside such a proof, and the signer would check the proof.

TEE is not that secure, but it's better than full software.

Charles Guillemet, CTO of Ledger, on Chain of Thought ep 65

The agent proposes an intent. The policy engine checks it. The enclave receives the approval and only then signs. The keys stay in the enclave, and the enclave signs nothing that is not approved. Hardware is the final enforcement layer. You still have to write a policy that maps each intent to the action you want. The enclave signs only after that policy approves.

What should you attack before this agent can sign?

Before the agent can spend or sign, use AI red teaming. Red teaming means you try to break your own system on purpose. Evals measure expected inputs. Red teaming asks what a motivated attacker can force, including jailbreaks, prompt injection, attempts to extract training data, and inputs that trigger unsafe output. Keep each failure as a permanent test. Include the chained path in that set. The attacker needs one path that ends in a signature. You need that signature to require the tier you assigned.

More on attacking agents, defending them, and the part software cannot cover is in Best Chain of Thought Episodes on AI Security.

Checklist

• Write what a compromised prompt, model, and host can steal or sign.
• If the yield includes an API key or a 24-word seed, move that asset above the software tier.
• Delegate rights through a policy a human signs on hardware, with each intent mapped to one action.
• Match the loss to his tiers: software, auto-sign hardware, a screen, or a screen plus a quorum.
• Red-team hostile inputs, including chained low-severity bugs, and store each success as a test.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.