Nine Tools to Avoid Writing One Specification

✓ Human-authored analysis; AI used for formatting and proofreading.

At The Pragmatic Summit in March, Uber's engineering leadership walked through the internal systems they built to support AI-assisted development. The list is impressive: MCP Gateway, Agent Builder, AIFX CLI, Minion, Code Inbox, uReview, Autocover, AutoMigrate, Shepherd. Nine systems, built by dozens of platform engineers, to manage the consequences of generating more code faster.

Every one of them solves a real problem but not the right problem.

The causal chain

AI agents generate more code. More code produces more pull requests. More pull requests overwhelm reviewers, so Uber builds Code Inbox to route and triage. But the reviews themselves are noisy, so they build uReview to filter comments by confidence. But the generated code lacks tests, so they build Autocover to generate unit tests. But the generated tests might be bad, so they build a critique engine to evaluate the tests. But migrations generate even more PRs across the monorepo, so they build Shepherd to manage them end to end.

Each system exists because the previous one didn't solve the problem. It displaced it one step downstream. The displacement is always in the same direction: further from the question nobody asked.

The question is: what does correct look like for this system, and how would we know?

The missing specification

Every system Uber built is a compensating control for the absence of a specification. Without a formal definition of correctness, every change requires a human to judge whether it's right. AI agents increase the rate of changes. The rate of required human judgment scales with it. You can redistribute that judgment (Code Inbox), partially automate it (uReview), or add proxies for it (Autocover), but you cannot eliminate it. Because you never defined what the judgment is against.

This is why Uber's CFO can't get a business impact number. Their engineering director admitted it openly: "These are activity metrics, not business outcomes." The reason they can only measure activity is that they never specified the outcomes. Without invariants, you cannot measure invariant preservation. Without invariant preservation, you are left counting pull requests which is like measuring a hospital's quality by counting the number of procedures it performs.

The metrics that would answer the CFO's question fall out of having specifications:

• Invariant coverage: what fraction of system properties have a formal definition of correct?
• Violation rate: how often does a change break a known invariant?
• Escape rate: how many violations reach production?
• Mean time to restore: how quickly are violations corrected after detection?

These correlate with incidents. Incidents correlate with revenue impact. Now you have something to show finance. None of Uber's nine systems produce any of these measurements.

Prompts are the CGI scripts of agentic development

In the early web, CGI programs built HTML by concatenating strings inside a Perl script. No separation of concerns, type safety or composability. Every request started from scratch. The output was fragile and non-reproducible. The same script could produce different HTML depending on server state, environment variables, and the phase of the moon.

We know how that story ended. CGI gave way to templates, templates to frameworks, frameworks to type-safe component models. Each step replaced stringly-typed construction with structured specification. React doesn't concatenate HTML strings. It declares what the UI should look like, and a reconciliation engine figures out what to change. The specification is the program.

Agentic development is currently in its CGI era. A prompt is a natural-language string that specifies intent without formal semantics. It is not composable. You cannot combine two prompts and know what the combination means. It is not verifiable. You cannot check whether the output satisfies the prompt without a human reading both. It is not reproducible. The same prompt can produce different code on consecutive runs.

Uber's Minion platform accepts a prompt, runs an agent, and produces a diff. That diff then enters a review pipeline staffed by humans and filtered by yet another AI system. The entire pipeline exists because the specification mechanism a prompt cannot express what correct means, so correctness must be adjudicated after the fact, by a reviewer who also doesn't have a written specification to check against.

This is CGI with better typography.

The human reviewer cannot scale this

The central assumption in Uber's architecture and in every agentic workflow covered in Gergely Orosz's reporting is that a human reviewer is the terminal verification step. The human reads the diff, applies judgment, and either approves or rejects.

This assumption has three problems that no tooling can fix.

The rate problem. AI agents produce changes faster than humans can review them. Uber's response was to build smart assignment, SLO-based escalation, and risk profiling to route reviews more efficiently. But efficiency doesn't change the fundamental rate mismatch. If agents produce changes at 10x the rate humans can review them, a 2x improvement in routing still leaves you 5x behind. The only way out of a rate problem is to remove the human from the verification path which requires verification that doesn't need a human.

The specification problem. A reviewer checking a diff is performing two tasks simultaneously: inferring what the change should do (because no specification exists), and checking whether the change does do it. The first task is the hard one, and it is invisible. When uReview filters out "low-confidence" comments, it's filtering on the second task while the first remains entirely in the reviewer's head. A reviewer who doesn't hold the full system model will approve changes that are locally correct and globally wrong. This is not a tooling failure. It is an information failure. The information that would make the review tractable a specification was never written.

The knowledge problem. The most revealing sentence in Uber's presentation was about their static analysis approach to finding misconfigurations. The AI code reviewer flagged issues that were "design trade-offs made deliberately" and things "already mitigated elsewhere in the codebase in ways the model had not cross-checked." Both failure modes with false positives from intentional trade-offs and false positives from mitigations the tool didn't see are symptoms of the same absence. If the trade-off were recorded as a specification ("this endpoint is intentionally unauthenticated because..."), the tool would not flag it. If the mitigation were expressed as an invariant ("cross-account access is permitted only when SourceArn is scoped to..."), the tool would check it. The knowledge exists in the heads of senior engineers. It is not written down. AI tools cannot access what is not written, and building more AI tools does not cause it to be written.

The ideal pipeline

The alternative is to invert the pipeline. Specifications first, generation second, verification mechanical.

Step 1: Define the invariants. For each system, for each domain, the engineering team writes the properties that must hold. Not in natural language. In a language with formal semantics that a machine can evaluate. CEL, Rego, Datalog, Z3. The choice of language matters less than the choice to write anything at all. "No public S3 bucket." "No IAM role trusts a service principal without a source condition." "No dependency older than N versions behind the latest security patch." "No API endpoint without an authentication middleware." These are the things the senior engineers already know. They are currently stored in those engineers' heads, retrieved during code review, and applied inconsistently.

Step 2: Generate against the specification. AI agents receive the invariants as constraints. The agent is not asked to "fix this bug." The agent is given a failing invariant and told to produce a change that makes it pass while preserving all other invariants. The specification is the prompt. This is the difference between "write me a CGI script that outputs a user profile page" and "here is the component interface; implement it so the type checker accepts it." One requires a human to verify the output. The other has a mechanical verification step built into the process.

Step 3: Verify mechanically. The generated change is evaluated against the full invariant set. By a deterministic engine that checks whether the invariants hold. If they do, the change is correct by construction. If they don't, the change is rejected with a specific, machine-readable reason, and the agent can retry. The human reviews the invariant definitions, not every change that satisfies them. That's a one-time cost per invariant, amortized across every change and every agent, versus Uber's current per-change, per-review human cost.

Step 4: Measure outcomes. With invariants defined and mechanically verified, the metrics write themselves. Invariant coverage tells you how much of your system has a definition of correct. Violation rate tells you how often agents produce incorrect changes. Escape rate tells you how often your verification misses something. These are the outcome metrics Uber's CFO is asking for and engineering cannot provide, because the infrastructure to produce them was never built.

Where this is hard

Writing specifications is difficult. It requires senior engineers to externalize knowledge they have internalized over years. It is slower than writing a prompt. It is less exciting than demoing nine new internal tools. Organizations naturally resist it because the output. A formal specification is invisible to leadership in the way a new platform with a web UI is not.

Not everything is specifiable. Feature-level product behavior, where "correct" is partially a product decision that changes with strategy, resists formal specification. The argument is strongest at the infrastructure and platform layers — IAM policy, network configuration, dependency management, API contracts where invariants are naturally expressible and stable over time.

Specifications can be wrong. A system that mechanically verifies against incorrect invariants will produce mechanically verified incorrect results. Invariant definitions need review, testing, and iteration. But this is a one-time cost per invariant, not a per-change cost per review. And a wrong invariant that is written down can be found, debugged, and fixed. A wrong assumption in a reviewer's head cannot.

The obsolescence test

Here is the test that tells you whether you are solving the right problem: when the underlying technology changes, does your solution survive?

Every system Uber built is coupled to the current generation of AI agents, the current set of models, the current prompting paradigm. When the next generation of models ships when agents get faster, or multi-modal, or able to operate across multiple codebases simultaneously every compensating system needs to be re-engineered. Code Inbox's routing heuristics break. uReview's confidence filtering breaks. Minion's optimized defaults need re-tuning. The treadmill continues.

A specification survives all of it. "No public S3 bucket" is true regardless of which model generated the code, which agent framework orchestrated the work, or which IDE the developer used. Invariants are decoupled from the tools that evaluate them. That is the property that makes them durable, and that is the property that none of Uber's nine systems possess.

Dijkstra wrote in 1972 that the purpose of abstraction is "not to be vague, but to create a new semantic level in which one can be absolutely precise." The AI industry is currently being vague at great expense. The alternative is being precise, once, costs less and lasts longer.

The organizations that write specifications first will not need nine tools to manage the output of their agents. They will need one: a verifier.