Everyone Designs the Happy Path. Production Happens at 3 AM.

At 3 AM, architecture diagrams become unusually honest.

The clean arrows are still there. The trust boundaries have not moved. The signature algorithm is still mathematically sound. But the person responding to the incident is no longer asking whether the design is elegant. They are asking a much less comfortable question:

Can I recover this system without quietly disabling the control that was supposed to protect it?

That question changed how I think about security architecture.

This is the first part of Auditability Under Pressure, a seven-part account of a real production system. It is not a series about how we built an architecture. It is about how we learned which parts of it were actually true.

The architecture looked defensible

The system managed remote agents running with elevated privileges. Its data path had three roles:

1. a control plane decided what was allowed;
2. an internet-facing gateway transported requests;
3. an agent inside the managed environment executed the authorized action.

The gateway was deliberately outside the authority boundary. It could carry a work order, but it could not create one. Each privileged work order was signed with Ed25519 by the control plane. The agent verified the signature, its own identity, the order's expiry, clock skew, and a nonce before executing anything.

The canonical signing contract was implemented in two languages. A fixed cross-language test vector made both implementations produce exactly the same bytes. That mattered more than the choice of serialization style: the protocol had to fail in CI before it failed across a mixed fleet.

The audit side was separate. Actions produced structured records linked with a SHA-256 hash chain. A database head and an optional external HMAC anchor made later modification or truncation detectable.

That wording is intentional. The records were tamper-evident, not immutable. An administrator with enough access could alter storage. The control was designed to make that alteration detectable, not physically impossible.

The distinction between the two artefacts also matters:

• the work order proved that the control plane authorized an action;
• the audit record preserved evidence about the decision and result.

Calling both of them “signed receipts” would sound impressive and be technically wrong.

Then we signed tunnel targets

The first uncomfortable discovery was that command execution was not the only privileged operation.

The agent could also open a TCP connection to a target inside the customer network. Terminal, file transfer, and remote desktop traffic depended on that path. The original tunnel header carried a plain host:port. The gateway could not forge a command, but it could choose where the agent connected.

That made the gateway a potential network pivot.

We changed the protocol so that opening a tunnel required a signed work order too. The authorized target became part of the signed content, and the agent required an exact match between that target and the transport header. A gateway could still drop a connection, but it could no longer redirect it to a domain controller, hypervisor, database, or management interface without breaking the signature.

The cryptography was the easy part.

The operational consequence was harder: a new agent would reject tunnels from an old control plane. Deployment order was no longer a preference. It became a safety property:

control plane → gateway → agents

If an automated deployment rolled the control plane back while agents had already moved forward, remote access could disappear across the fleet.

The emergency door

We needed a local break-glass mechanism. Not because unsigned tunnels were acceptable, but because a security control that destroys its own recovery path can turn a routine rollback into a fleet-wide lockout.

The first implementation used the obvious operational tool: an environment flag that allowed unsigned tunnel requests.

It worked. That was the problem.

A boolean emergency flag has no memory of why it was enabled, when it should end, or whether the incident is over. “Set it now, remove it later” is not a security design. It is an optimistic note left for a tired human.

The replacement forced every bypass into a time window:

Every accepted unsigned tunnel emitted a warning containing the expiry time.

This was better, but it was not magic. The bare true form used a process-relative window. Restarting the service recalculated that hour. Repeated restarts could therefore create repeated windows.

The preferred incident command became an absolute expiry, because an absolute deadline survives process lifecycle changes. The one-hour form remained a last-resort convenience, documented with its residual risk.

Microsoft's guidance for emergency access accounts makes the same broader point: emergency access must be deliberately managed, monitored, and validated rather than treated as a forgotten permanent exception. Our mechanism was different, but the operational lesson was the same.

What we believed

We believed that signing privileged commands kept the untrusted gateway outside the authority boundary.

That statement was true, but incomplete. A tunnel destination was also an authorization decision. Protecting exec while leaving dial unsigned secured one path and left another capable of producing serious impact.

We also believed a bypass could be operationally harmless if it was clearly named and disabled by default.

That was false. A control whose safe state depends on someone remembering to restore it is not fail-closed in practice.

What changed our mind

Three pieces of evidence changed the design:

1. tracing the real terminal, file-transfer, and remote-desktop path showed that the gateway selected an unsigned internal target;
2. deployment rollback analysis showed that strict enforcement could remove the very access needed for recovery;
3. testing the emergency flag across time and restart boundaries exposed the difference between “expires after an hour” and “has an absolute expiry.”

None of those findings came from breaking Ed25519. They came from following authority through the whole system.

That distinction is useful well beyond remote agents. Cryptography can correctly protect the wrong boundary. A policy can be perfectly enforced on one operation while a neighboring operation carries equivalent power. A break-glass path can be necessary and still invalidate the assurance story around it.

The first production proof was deliberately incomplete

We proved the signed command path end to end with a real production action. We proved that missing, malformed, and invalid tunnel orders reached the agent and were rejected for distinct reasons. We proved the Windows agent rejected an invalid signature and persisted the evidence.

We did not enable the unsigned bypass in production merely to say we had tested it there.

Doing so would have temporarily disabled the control we had just added. The honest verification status was:

• expiry behaviour: covered by tests;
• bypass closed in production: observed through rejection;
• bypass open in production: intentionally not triggered.

“Not production-triggered” is not the same as “does not work.” It is also not the same as “production-proven.” That middle state belongs in the engineering record.

This decision becomes invalid when…

The local bypass should be reconsidered if all of the following become true:

• the control plane can no longer roll back to a version that omits signed tunnel orders;
• the fleet is fully upgraded and version compatibility is enforced;
• a separate recovery channel exists that does not depend on the affected tunnel path;
• emergency state is centrally visible and alarmed.

Until then, removing the escape hatch would improve the diagram and weaken recovery.

Keeping it without expiry would improve recovery and weaken the security boundary.

The design lives in that tension.

What the diagram did not show

At the end of this stage, we had signed authority, replay resistance, target binding, time-bounded emergency access, and tamper-evident audit records.

What we did not yet have was proof that every platform preserved the evidence needed to investigate those decisions.

On Linux, rejected requests appeared in the journal. On Windows, the same control could run under the Service Control Manager and leave no persistent record at all.

The next part starts with that larger architectural question: should evidence be a feature of the application framework, or a layer that survives it?

References

• RFC 8032 — Edwards-Curve Digital Signature Algorithm (EdDSA)
• NIST SP 800-53 Rev. 5 — AU-9, Protection of Audit Information
• Microsoft — Manage emergency access accounts

Editorial note: This article is based on a real production engineering record. AI tools assisted with structure and language review; the author verified the technical claims, source links, and final wording.