Human Approval Should Follow Consequence, Not Model Confidence
Most agent approval systems ask the wrong question: “Are we confident this action is safe?” The better question is: “What happens if this action is wrong, and can we undo it?” That distinction matters because an agen
Most agent approval systems ask the wrong question:
“Are we confident this action is safe?”
The better question is:
“What happens if this action is wrong, and can we undo it?”
That distinction matters because an agent can be highly confident and still be operating on bad instructions, incomplete context, or a manipulated document.
Consider a hypothetical procurement agent. It reads an email, checks an invoice, and prepares a supplier payment. The system sends every proposed payment to a person for approval.
That sounds safe. In practice, it creates two failure modes.
First, the approver becomes a rubber stamp. If every low-risk action produces an alert, the human learns to approve quickly without inspecting the reasoning.
Second, the approval may arrive after the important decision has already been made. The agent might have selected the supplier, changed payment details, or prepared a transaction with no meaningful boundary between “recommend” and “commit.”
The problem isn’t the presence of a human. It’s where the human is placed in the execution path.
Put approval at the commitment boundary
A useful threshold should be based on consequence and reversibility.
An agent can usually handle actions that are:
- Read-only
- Reversible
- Limited to a known scope
- Covered by an explicit policy
Human review becomes more useful when an action:
- Sends data outside the approved boundary
- Changes permissions or financial details
- Creates an external commitment
- Is difficult to reverse
- Depends on ambiguous or untrusted instructions
The approval request should show the proposed action, the authority being used, the data involved, the policy decision, and what will happen next. “Approve tool call?” is not an approval interface. It’s a prompt for fast clicking.
The system should also make the approval specific. A person approves one defined action with a defined scope, not a vague instruction such as “let the agent continue.”
The control to inspect
Ask three questions about every agent workflow:
- Which actions can the agent complete without a person?
- Which action creates the first material consequence?
- Does approval happen before that action, with enough context to make a real decision?
If the answer to the third question is unclear, the workflow probably has a visibility control rather than an authorization control.
Confidence scores may help prioritize review, but they shouldn’t decide whether review is required. The agent’s certainty is about its interpretation. The business risk belongs to the action.
That is the design choice enterprise teams need to make: approve based on what the agent believes, or based on what the agent is about to change.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.