Separation of Duties for AI Agents: Why Autonomy Needs Checks, Not More Questions
Planner, implementer, independent auditor, policy gate. An architecture for agents that act without a human clicking "approve" 40 times a day. TL;DR Human-in-the-loop on every step tends to degrade into rubb
Planner, implementer, independent auditor, policy gate. An architecture for agents that act without a human clicking "approve" 40 times a day.
TL;DR
- Human-in-the-loop on every step tends to degrade into rubber-stamping. You end up with an approval trail that proves authorization and says nothing about scrutiny [7].
- Unsupervised autonomy removes the accountability that clients, boards and regulators expect.
- A third model borrows from internal-control practice: planner β implementer β independent auditor β deterministic policy gate, with humans owning objectives, exceptions and irreversible decisions.
- This is a pilot hypothesis, not a proven result. The architecture is conceptual, and no controlled baseline-to-outcome case of a full multi-agent consulting engagement was found.
- Suggested next step: a 90-day, single-workflow pilot, using ISO/IEC 42001 and ISO/IEC 27001 as management scaffolding.
The problem: approval fatigue is a security bug
Say you deploy agents to gather client documents, draft analyses, and assemble a benchmark deck. Within a week, the governance question shows up: who checks what the agents did?
The instinctive answer is "a person, at every step." It feels responsible. In practice:
- Throughput is capped at the speed of someone's inbox.
- A reviewer approving dozens of plausible, routine actions a day stops reviewing and starts clicking [7].
- The audit trail says "approved," not "examined."
Why this matters now: agents are moving from drafting text to taking actions: sending messages, changing configs, touching client data. Error cost scales with every tool an agent can call. Meanwhile regulators are formalizing expectations. The EU AI Act requires high-risk systems to be designed so natural persons can effectively oversee them [31], and ISO/IEC 42001 provides a structured set of policies, processes and controls for governing AI [32].
The engineering question: how do you grant agents more autonomy without losing the control that makes the work defensible? The answer owes less to new AI techniques than to an old idea from finance and security: no single actor should be able both to create and conceal an error.
The control logic is older than the technology
Practitioner summaries of the major control frameworks reduce to a triangle: authorization, execution/custody, and recording/reconciliation belong with different parties [27]. COSO names monitoring as one of the five components of effective internal control [25], and the Sarbanes-Oxley Act asks management of listed companies to maintain an adequate internal control structure for financial reporting [29].
ISO/IEC 27001:2022 states it operationally. Control 5.3 requires conflicting duties and areas of responsibility to be separated [22][23]. One guide gives two blunt examples: whoever initiates a transaction shouldn't approve it, and whoever writes code shouldn't push it to production without review [22].
Mapped onto agents, the design rules nearly write themselves:
planner -> cannot execute
implementer -> cannot approve its own output
auditor -> cannot edit the deliverable
any agent -> cannot bypass the gate
The caveat that matters most
Four agents do not create separation of duties. If they share a credential, the same transcript, and the ability to alter evidence, you've staged a play with four actors and one set of keys.
Real separation needs:
- Separate identities
- Separate (least-privilege) permissions
- Append-only logs
- Enforcement outside the language model
Where full separation is impractical, substitute safeguards are accepted. One published self-assessment shows a sole-operator structure offset by an append-only audit log [18].
Reference architecture
+-----------+
Objectives | Human | owns objectives, exceptions,
---------> | owner | irreversible decisions
+-----+-----+
|
+-----v-----+ +-------------------+
| Planner |----->| Implementer |
| (no exec) | | (sandbox, no |
+-----------+ | self-approval) |
+---------+---------+
| artifact + logs
+---------v---------+
| Independent |
| Auditor |
| (read-only on |
| deliverable; |
| evidence access) |
+---------+---------+
|
+---------v---------+
| Policy gate | deterministic,
| (hard-stop list) | outside the LLM
+---------+---------+
|
reversible, bounded -> auto
hard-stop actions -> human
Every component has its own identity, its own permission set, and writes to an append-only log that no operating agent can alter.
Autonomy is assigned by function, not by system
Early research proposed ten levels of human-computer decision-making, from fully manual to fully autonomous [7][10]. A later model argued for setting automation per function type instead of as one dial for the whole system [13][14]. For knowledge work that yields four independent decisions:
| Function | Default automation | Control | Consulting example |
|---|---|---|---|
| Gathering information | High | Approved sources, access scope, data classification | Agent searches authorized client repositories |
| Analyzing information | Medium to high | Traceability to sources, validation tests | Agent drafts a benchmark, cites each figure |
| Choosing an option | Medium | Human-set objectives and thresholds | Agent ranks alternatives; a partner chooses |
| Acting | Low for irreversible, high for reversible | Hard gate, rollback requirement | Agent may update a draft model; may not email the client |
The SAE J3016 driving levels (six steps from no automation to full automation [1][11]) are a useful analogy: a system is autonomous only inside defined conditions. For a consulting agent those conditions are the client, the data classification, the tools, and the ability to roll back.
Worked scenario: the client benchmark deck
An agent team prepares a competitor benchmark.
- Gather and analyze: runs autonomously inside authorized repositories. Every number carries a source link.
- Implement: the implementer assembles the deck in a sandbox (reversible by design).
- Audit: the auditor checks each figure against the source documents, not the implementer's summary, and re-runs the calculations.
-
Gate: three things no agent may do alone:
- send the deck to the client
- change access rights on the client folder
- edit the checklist that governs all of this
- Human: a partner reviews only what the auditor flagged, plus the release itself.
Most routine steps move without a human click, while consequential decisions still reach one.
A short hard-stop list beats case-by-case approval
Case-by-case approval produces inconsistent decisions, blurred accountability, and a queue of low-value prompts. A hard list is predictable, testable, and enforceable by a deterministic gate.
A defensible list covers:
- Production changes
- Destructive deletion
- Credential and privilege changes
- External communication
- Financial or contractual commitments
- Any change to the controls themselves
The CISO, together with risk, should own the list and review it at least quarterly.
Describe outcomes, not commands
A rule against DROP TABLE or "database deletion" is too narrow if an agent can reach the same result by shortening a retention policy. Write the policy against the effect:
# Illustrative policy sketch (outcome-based, not command-based)
hard_stop:
- effect: irreversible_data_loss # deletion, retention shortening, key destruction
- effect: privilege_change # IAM, folder ACLs, credential rotation
- effect: external_communication # email, publish, client-facing send
- effect: commitment # financial or contractual
- effect: control_modification # gate rules, checklists, log config
default:
allow_if: [reversible, bounded]
require: [logging, sampled_review]
Everything outside the list that is reversible and bounded proceeds autonomously, with logging and sampled review.
Independent review must rest on evidence
The auditor is the component most often weakened in practice. Hand it the implementer's reasoning and confident summary and it will tend to agree, because the summary frames the question.
A credible design gives the auditor:
- The task specification
- Acceptance criteria
- The final artifact
- Execution logs
- Independent access to source evidence
...and withholds the implementer's narrative until after a first independent pass.
Evidence ranks in a clear order:
- External system state and deterministic tests
- Signed source documents
- Reproducible calculations
- (far below) the explanation written by the party being checked
Independence has a price. Published analysis catalogues recurring multi-agent failure modes [556], so add only the minimum independent control needed to catch the failure that matters. More agents means more risk surface, not automatically more safety.
The opposite failure: stopping too often
Governance can fail in the other direction. Agents that treat every uncertainty as a reason to halt push routine decisions back to humans, producing alert fatigue and rubber-stamping aimed at the exception queue.
Measure both directions:
| Metric | Why |
|---|---|
| Blocked hard-stop attempts | Gate is working |
| Seeded-error detection rate | Auditor is working |
| Unnecessary escalations | Agents aren't over-stopping |
| False blocks on known-safe work | Policy isn't over-broad |
| Review minutes per completed task | The actual business case |
A seeded error is a mistake planted deliberately to test whether the auditor catches it. If review burden approaches the manual baseline, the architecture has failed even if no incident occurs.
What this costs, and when to stop
Build cost sits in four places:
- Separate identities and permissions
- Immutable logging
- The policy gate
- The auditor's access to source evidence
Run cost adds model usage per extra agent plus control-owner time. No study in the library quantifies these costs, so break-even must come from your own baseline:
reviewer_hours_saved_per_deliverable * loaded_hourly_rate
> added_model_cost + control_ownership_cost
(with rework held flat)
Set a stop-loss before the pilot starts. If review effort stays near the manual baseline, or seeded errors slip past the auditor, pause promotion and fix the design.
What regulators actually require
EU AI Act, Article 14 requires high-risk systems to be designed for effective oversight by natural persons, with measures commensurate with risk, autonomy and context [31][45]. Nothing in that text asks a person to approve each intermediate step. What counts is practical capability:
- Understanding the system's limits
- Staying alert to over-trusting machine output
- Interpreting results
- Overriding or reversing them
- Stopping the system
Application dates for high-risk obligations have been shifting in legislation, so have counsel confirm the current date for your use case. Build the controls now regardless; they're needed on any timeline.
ISO/IEC 42001 supplies the management layer: policies, processes and controls governing how AI is designed, developed, deployed and used [32], including risk and impact assessment, transparency and human oversight across the lifecycle [33]. Certification evidences a management system. It says nothing about agent accuracy or ROI.
ISO alignment (management perspective)
ISO/IEC 42001: governing the agents
Intent: turn oversight from individual heroics into an auditable routine with named owners and continual improvement.
Minimum practices:
- Inventory of agents, tools, and autonomy levels
- A named oversight owner per workflow, with authority to pause and reverse
- Risk assessment before promoting an agent to a higher autonomy level
Artifacts: agent register, authorization matrix, signed hard-stop list, promotion records.
KPI: seeded-error detection rate; share of attempted prohibited actions blocked.
Risk: autonomy expands informally and nobody can say what an agent was allowed to do. Mitigation: a promotion gate that demands evidence before any increase.
ISO/IEC 27001: enforcing the separation
Intent: Control 5.3 anchors the design in language auditors and clients already use [22][24].
Minimum practices:
- Separate identities and least-privilege permissions per role
- Immutable logs no operating agent can alter
- Defined substitute safeguards where separation is impractical
Artifacts: role-to-permission mapping, access reviews, log-integrity records.
KPI: agents holding self-approval permissions (target: zero).
Risk: shared credentials silently collapse the separation. Mitigation: periodic automated conflict checks.
ISO/IEC 20700 and ISO/IEC 21500 bear on engagement quality and delivery governance but are not applied here, to keep compliance load readable.
A 90-day pilot outline
Days 1β30: foundations
- Security and risk write and sign the hard-stop list
- Delivery picks one workflow with a clear reversible/irreversible boundary
- Record baseline: manual cycle time, rework, quality, review effort
Days 31β60: shadow mode
- Agents plan and audit; people execute
- Compare machine and human decisions
- IT provisions separate identities and append-only logging
Days 61β90: gated execution
- Allow reversible execution under the gate
- Inject seeded errors to test the auditor
Promotion rule:
- Promote when seeded-error detection holds and review minutes fall against baseline
- Hold when results are mixed
- Roll back on any unblocked hard-stop action
Assumptions to state up front: a crisp reversible/irreversible boundary, working identity and logging controls, and client consent for agent involvement. If one fails, narrow scope or fix the foundation first.
Human shift: reviewers move from approving steps to owning policies, exceptions and sampling. Track their review time and feed them seeded errors so the exception queue doesn't recreate rubber-stamping. Let skeptical partners inspect the logs themselves.
Fund independence, not agent count. If the answer to "how is the auditor different?" is "a different prompt," separation doesn't exist yet.
Conclusion
Control doesn't come from asking a human more questions. It comes from designing a system in which no single actor can both make and hide a mistake. Finance and information security have relied on this idea for a long time [22][27].
For agents, the payoff is practical: routine, reversible work moves at machine speed; people spend attention on decisions with irreversible consequences; auditors and regulators see role-level evidence instead of reflexive approvals. The EU AI Act asks for effective oversight, and constant interruption isn't part of that [31][45]. ISO/IEC 42001 makes the oversight repeatable [32].
The limits deserve equal weight. This is a conceptual framework, the failure literature is a sober reminder that added agents add added risk [556], and no controlled production case of a full consulting engagement was found. The sensible path is a narrow pilot with a measured baseline, a short hard-stop list, and a genuinely independent auditor. If results hold, widen the domain. If not, the evidence will show where the design must change.
Discussion: If you've shipped agent systems with a separate auditor role, what did you give it that the implementer didn't have? I'm especially curious how people enforce the "gate outside the LLM" boundary in practice.
The cited sources are listed in the full article: https://auranom.ai/separation-of-duties-for-ai-agents-why-autonomy-needs-checks-not-more-questions/
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.
