Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 12 min read

Harvard Says Most Organizations are Using AI Write More Code but Don't Deliver More. Here's How to Fix It.

We gave developers coding agents but kept the old way of working. Then we were surprised when nothing got faster. AI helps us write more code. So why aren't we shipping more software? A new paper from Harvard tries to

We gave developers coding agents but kept the old way of working. Then we were surprised when nothing got faster.

AI helps us write more code. So why aren't we shipping more software?

A new paper from Harvard tries to answer that question. Fiona Chen and James Stratton looked at data from 655 companies and more than 720,000 workers. They found that AI tools help people write a lot more code. But the amount of finished software barely changes.

The reason? Code review. After companies adopt AI agents, code waits longer before someone reviews it. Reviews take longer. And reviewers ask for more changes.

In other words, we sped up one step and created a traffic jam at the next one.

Same data, different conclusion

The study has been spreading fast, and a popular reading of it goes something like this: AI is overhyped. It writes code faster, but the work just moves somewhere else. Companies are spending millions on tokens for nothing. The "country of geniuses in a data center" is really a bunch of engineers waiting for Claude to respond.

I agree with the facts. I don't agree with the conclusion.

The paper tells us where things get stuck. It doesn't tell us why. I have a theory. I can't prove it, and I don't have broad data to back it up. What I do have is one real project where we tried changing how we work, which I'll describe later. And I have a strong feeling about it.

I think many companies are using AI inside a way of working that was never designed for it. The bottleneck isn't a limit of AI. It's a limit of the organizations using it.

And the paper itself leaves room for this. The authors built a model to predict what happens next. It predicts that output grows as companies keep adjusting, and that the gains stay smaller if the review bottleneck doesn't go away. They also point out that there may be other bottlenecks β€” for example, in management or testing β€” that moving more people into review can't fix.

That's not a story about AI failing. It's a story about companies that haven't changed yet.

Writing code was never the only problem

Think about how most teams work today. Work arrives as tickets. Teams plan in two-week sprints. Every change goes through the same handoffs, approvals, and review queues. Managers track progress with the same numbers they always have.

Now add a coding agent. Developers can write code much faster. But everything around them stays the same.

So what happens?

  • More code lands in the review queue, so the queue gets longer.
  • More changes collide with each other, so merging gets harder.
  • If the requirements were unclear, the agent builds the wrong thing β€” just faster.
  • If there aren't good tests, someone has to check every change by hand.

None of this is surprising. If you speed up one part of a system and leave the rest alone, the work just piles up somewhere else.

The engineers interviewed in the paper say the same thing. One described the real bottleneck as "communication between business and product teams, as well as review and QA." Another spent six months waiting for legal approval on something that should have taken a month. The paper also found that AI didn't change how many meetings people have.

None of these are coding problems. They're problems with how work flows through a company. And nobody pointed AI at them.

To be fair, slow reviews don't automatically mean a company has a "Scrum problem." The code might be complex. The tests might be weak. Some changes really do need careful review. My point is simpler: a tool can only be as fast as the system around it.

Not all friction is bad

When AI-generated code causes problems, companies often respond by adding friction. One engineer in the Harvard paper described exactly this. On his project, only one senior engineer was allowed to approve changes, and he could only review code once a week. After some outages caused by AI-generated code, new rules required senior approval for all AI-generated changes. That "created even more review bottlenecks."

Amazon reportedly did something similar after an outage: more sign-offs from senior engineers and more documentation before critical changes. Some people point to this as proof that AI doesn't really speed anything up. If you have to add friction, what's the point?

But the real question isn't whether to add friction. It's what kind.

There is friction that only slows you down. Making a senior engineer read every AI-generated change adds more people to the review queue β€” the exact bottleneck the paper found.

And there is friction that pays for itself. For years, we treated some practices as slow or "overkill":

  • Static types
  • Explicit database schemas
  • API contracts between services
  • Architecture tests
  • Clear boundaries between parts of the system
  • Written specs

With coding agents, these practices do three jobs at once:

  1. They give the AI context. They tell the agent what the system means and what the rules are.
  2. They lead to better output. The agent has fewer ways to guess wrong.
  3. They let us check the work automatically. The compiler, the tests, and the contract checks catch mistakes before a person has to.

Interestingly, one of Amazon's changes could be the good kind. Documentation that only a human will read is paperwork. Documentation that an agent can use as context, and that tests can check against, is friction that pays off.

What I mean by a "factory"

I've written before about the idea of an "agentic factory." Many people see it as a goal for the future β€” something you build once you trust the agents.

I think it's the other way around. You build the factory so that you can trust the agents.

But I want to be clear about what I mean, because "factory" can sound scary. I don't mean a factory with the lights off, where agents ship code and no human is involved. I don't believe in that, at least not with today's models.

There are two pictures of AI in software that I don't agree with:

  • The fully automated factory. No people at all. Agents write, test, and ship everything. This isn't realistic today.
  • The "prompt factory." People become the assembly line. They type prompts, wait for Claude, and then read every line it writes. This is what many developers describe today, and it's exhausting.

What I mean is something in between. People stay involved and in charge. They decide what to build, judge what is risky, and improve the system when something goes wrong. The factory is everything around the agent that gives them evidence:

  • Clear descriptions of what the software should do
  • Automated tests and checks
  • Smooth ways to merge and release changes
  • More review for risky changes, less for safe ones
  • Monitoring in production
  • A quick way to undo mistakes and learn from them

None of this guarantees that every change is correct. It just means mistakes are easier to catch, less harmful when they slip through, and less likely to happen twice. And it means people spend their attention where judgment matters, instead of re-checking every change from scratch.

We've seen this movie before

This reminds me a lot of Agile.

Agile and Lean can change how companies build products. I really believe that. But most Agile transformations I've seen didn't work. What we got instead was "bad Scrum": stand-ups, sprints, story points, and burndown charts β€” with the same old company underneath.

Why does this happen? Because the real ideas behind Agile are hard. Things like "deliver value often," "trust your team," and "adjust based on feedback" need experience and good judgment. You can't follow them like a recipe.

Ceremonies, on the other hand, are easy. You can teach a stand-up in an afternoon. You can track story points in a spreadsheet. So companies adopt the rituals, skip the real change, and call it a transformation.

I worry we're doing the same thing with AI.

Look at the token leaderboards. Some big tech companies have reportedly started ranking engineers by how many AI tokens they use. That's the new story points: easy to count, easy to put on a dashboard, and almost meaningless. It measures whether people are using AI, not whether it's helping.

Buying licenses is easy. Telling people to "use AI more" is easy. Setting productivity targets is easy. Changing how teams are organized, how work flows, how code is reviewed and released, and how managers measure success? That's hard.

And I suspect the reasons are the same ones that held Agile back:

  • People resist change. New tools are fine. New roles and responsibilities are scary.
  • We prefer simple, practical advice. "Use this tool" is easier than "rethink how you work."
  • Managers want control. Old processes feel predictable, even when they slow things down.
  • We measure the wrong things. If you count lines of code or tokens, that's what you'll get β€” not better software.

None of this means people are acting in bad faith. Leaders need to show progress. Managers are asked to keep things predictable. But if we only change the tools, we shouldn't expect a different result.

Cutting the wrong people

The same pattern shows up in layoffs. Some companies are cutting jobs in the name of "AI efficiency." But if the bottleneck is review, judgment, and coordination, then the people being cut may be exactly the people the company is waiting on.

That's not a sign that AI has replaced them. It's another sign of adopting AI without redesigning the work around it.

A new tool doesn't change a company

In a previous post, I argued that coding agents could finally make some Agile ideas real. For example, a business expert can now build a working prototype instead of writing a long requirements document. That directly attacks one of the bottlenecks engineers mention in the paper: communication between business and product teams.

I still believe that. But just because something is possible doesn't mean companies will do it. Agents make new ways of working possible. They don't make companies willing to change.

So we end up stuck in the middle. The technology is new, but the way we work is old. We produce more code at the start, and it gets stuck later on.

What happened when we tried it

I've seen this play out myself. I work on a complex platform in the specialty insurance industry. The project is part of an AI transformation program, and our team was given something unusual: permission to question the way we build software.

First, we made the Harvard mistake

Our first idea was to make Scrum faster. We built agents to refine stories, size them, and plan sprints. It worked, in a way. Work that used to fill a sprint could be built in a morning.

And then we hit exactly the wall the Harvard paper describes. Writing code was no longer the slow part. Checking it was. Estimates became even less reliable, because agents finish some "big" tasks in one go and struggle for days on some "small" ones.

Then we noticed something funny. Once we stopped making sprint commitments, nobody used the sizing agent's estimates anymore. We had automated a ceremony that no longer had a purpose.

That was the turning point. Rethinking the process works better than automating it.

Then we changed how we work

Here is what we changed:

  • We dropped most Scrum ceremonies. No more estimation, sprint planning, or backlog grooming. We kept the retrospective. We moved to a continuous flow of small pieces of work, released when they're ready.
  • We made the teams smaller. Fewer people means fewer handoffs and less lost context. Engineers work directly with users and have become experts in the business, taking on some of the work product owners and business analysts used to do.
  • Business users build prototypes. End users build prototypes with agents in a separate, safe repository: no production systems, no secrets, no live connections. Engineers join these sessions too. Each session produces a working prototype and a written spec, and people refine both until the feature is ready.
  • The prototype becomes the spec, not the code. Agents turn an approved prototype into GitHub issues that describe how the feature should behave. Each issue links back to the prototype. Then the prototype code is thrown away. This deals with the problem the engineer in the Harvard paper raised β€” that features "still need to be validated and approved by humans." The business approves a working prototype at the start, instead of a finished feature at the end.
  • We put all the context in one place. Our code was in GitHub, our issues in another tool, and our documentation in a third. Agents could only see part of the picture. We moved everything into GitHub and added agents that flag when documentation and code drift apart.
  • We invested in friction that pays off. Strict types, business rules written in a form machines can check, and a database design that makes invalid states impossible. Our rule is simple: you can only relax human review where automated checks have become stronger.
  • We measure outcomes, not activity. We don't track velocity, pull requests, or lines of code. We link experiments to business goals. An issue only counts as delivered when its code is running in production. And we watch trends like how long work takes and how many bugs we create compared to features.

What we saw

I can't share specific numbers. But compared to the original plan for the project, we're delivering well ahead of schedule, at a lower cost, with a smaller team, and features reach production in days, not weeks.

To be honest about the limits: this is one project, in one regulated industry, with strong support from leadership. And I'm reporting it as someone on the team, not as an outside researcher. It doesn't prove my theory. Other teams may not get the same results.

But it does show what the other side of the Harvard bottleneck can look like. We used the same kind of AI tools as everyone else. What made the difference wasn't the agents. It was everything we changed around them.

How we could test this

This is a theory, not a fact. So how could we find out if it's true?

First, stop measuring the wrong things. Licenses, tokens, prompts, lines of code, and commits tell you that people are using AI. They don't tell you whether it's helping.

Instead, look at the whole journey, from a real customer need to working software in production:

  • How long does code wait for review?
  • How long does it take for finished code to reach users?
  • How often does work need to be redone?
  • How many bugs reach production?
  • How quickly can we undo a mistake?
  • Are customers actually getting value sooner?

Then compare teams. Some teams use AI with a real factory around them: good tests, explicit contracts, clear ownership, manageable review queues, direct access to the business, and safe releases. Other teams use AI inside their old process.

If both groups get the same results, I'm wrong. But if AI only helps the teams that changed how they work, that tells us something important. The tool was never the problem.

Yes, the human is still required

People who are skeptical of AI often end with: "The work is automated, but the human is still required."

Of course the human is still required. I've never argued otherwise.

The real question is how we use people. Are they checking every line an agent writes, waiting in long review queues, and sitting in the same meetings as before? Or are they deciding what to build, judging risk, and improving the system that does the checking?

Agile taught us that copying the visible parts of a transformation is not the same as transforming. AI might be teaching us the same lesson again.

We're adopting the agents but putting off the factory. And we're expecting the old way of working to give us new results.

The factory isn't something you build after you trust the agents. It's what lets you trust them in the first place. And it's what turns faster coding into better software.

Source: Artificial Intelligence in the Firm: Bottlenecks in Software Production, Fiona Chen and James Stratton, Harvard University (job market paper, version dated November 19, 2026). Quotes from engineers come from the paper's interviews.

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.