Dev.to AI ๐Ÿค– Ai ๐Ÿ‘ 0 ๐Ÿ“– 8 min read

Measuring AI Coding Impact With DORA Metrics, Not Vibes

Six months after a client rolled out AI coding assistants to their whole team, the engineering manager told the board that delivery was "about 40% faster". When I asked where the number came from, the answer was a survey

Measuring AI Coding Impact With DORA Metrics, Not Vibes

Six months after a client rolled out AI coding assistants to their whole team, the engineering manager told the board that delivery was "about 40% faster". When I asked where the number came from, the answer was a survey: developers said they felt faster. Meanwhile the deploy log showed the same two releases a week as before, the change failure rate had crept from roughly one in ten deploys to nearly one in five, and reviewers were spending noticeably longer on each pull request. The team was typing faster. It was not shipping faster, and it was breaking more.

I wrote this with help from AI tools and checked every query and script against the GitHub REST API, GitHub Actions and the published DORA metric definitions. I am a Principal Software Engineer in Karachi with 9+ years across PHP/Laravel, Vue, Node and .NET microservices, and through Loop Origin I maintain business applications where a growing share of new code is AI-assisted.

๐Ÿ“š Series Navigation โ€” Maintaining AI-Written Code in Production

Related earlier posts: Synthetic Checks & SLOs for the Paths Nobody Clicks ยท Backups That Never Restored: Automated Restore Drills

How do you measure whether AI coding tools actually help?

Direct answer: Measure delivery outcomes, not typing speed. Take a baseline of the four DORA metrics (deployment frequency, lead time for changes, change failure rate, time to restore) plus PR cycle time and rework rate before rollout, then compare the same numbers per month afterwards. If lead time drops but failures rise, the tools are moving cost, not removing it.

Engineer's notebook with a hand-drawn line graph next to a keyboard, AI-generated

Why surveys and "lines of code" mislead

Perceived speed is real, but it measures the part of the job AI is best at: producing a first draft. The expensive parts of delivery are review, integration, testing, deployment and fixing what broke. An assistant that generates a 600-line pull request in ten minutes can easily add two hours of review and a hotfix the following week.

Lines of code and "accepted suggestions" are worse. They reward volume, and AI-assisted code tends to be more verbose. On one Laravel project, net lines added per month went up about 60% after rollout while the number of user-facing features shipped stayed flat. Most of the extra lines were duplicated validation logic and tests that asserted very little, exactly the kind of thing Parts 2 and 4 of this series were about.

The DORA metrics work because they measure the system end to end, and because the team cannot inflate them by generating more text.

The metrics I track, and where each one comes from

Metric What it tells you Source Typical trap
Deployment frequency How often value reaches production Deploy workflow runs or release tags Counting staging deploys
Lead time for changes First commit (or PR open) to production PR merge time + deploy time Measuring only merge, not deploy
Change failure rate Share of deploys that need a fix or rollback Hotfix/revert PRs, incident log Not labelling hotfixes
Time to restore How fast you recover from a bad deploy Incident open/close times Only counting "big" incidents
PR cycle time Open to merge, split into wait and review GitHub PR API Ignoring draft time
Rework rate Share of changed lines touched again within 21 days git log per file Counting formatting churn

The first four are the standard DORA set. I add PR cycle time and rework rate because they are where AI-assisted code shows its cost first. Rework in particular rises before change failure rate does: code that was merged quickly gets rewritten quietly a week later.

Step 1: Get a baseline before you change anything

If the tools are already rolled out, use history. GitHub keeps everything you need. This script pulls merged PRs for a period and computes cycle time, using only the public REST API and a read-only token:

#!/usr/bin/env bash
# pr_cycle.sh OWNER/REPO 2026-05-01 2026-07-31
REPO=$1; FROM=$2; TO=$3
gh api --paginate "repos/$REPO/pulls?state=closed&per_page=100" \
  --jq ".[] | select(.merged_at != null) \
        | select(.merged_at >= \"${FROM}\" and .merged_at <= \"${TO}T23:59:59Z\") \
        | [.number, .created_at, .merged_at, .additions + .deletions] | @tsv" \
  > prs.tsv

python3 - <<'PY'
import csv, statistics
from datetime import datetime as d
rows=[r for r in csv.reader(open("prs.tsv"), delimiter="\t")]
f=lambda s: d.fromisoformat(s.replace("Z","+00:00"))
hours=[(f(r[2])-f(r[1])).total_seconds()/3600 for r in rows]
size=[int(r[3]) for r in rows]
print(f"PRs: {len(rows)}  median cycle: {statistics.median(hours):.1f}h  "
      f"p85 cycle: {sorted(hours)[int(len(hours)*.85)]:.1f}h  median size: {statistics.median(size)} lines")
PY

Note that additions + deletions needs the single-PR endpoint on some API versions; if the list response leaves them out, loop over PR numbers with gh api repos/$REPO/pulls/$N. Run the script for the three months before rollout and each month after. Always report the median and the 85th percentile; averages hide the slow PRs that actually hurt.

Step 2: Make deploys and failures countable

Lead time and change failure rate are only as good as your labels. Two small conventions make them reliable:

# .github/workflows/deploy.yml (excerpt)
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
      - run: ./deploy.sh
      - name: Record deploy
        if: success()
        run: |
          echo "$(date -u +%FT%TZ),${GITHUB_SHA},${GITHUB_RUN_ID}" >> deploys.csv
          gh api repos/${{ github.repository }}/deployments \
            -f ref=${GITHUB_SHA} -f environment=production -F auto_merge=false
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Then agree as a team: any PR that fixes a production problem gets the hotfix label, and any revert keeps GitHub's default "Revert" title. Change failure rate becomes "deploys followed by a hotfix or revert within 48 hours, divided by all deploys". It is not perfect, but it is consistent, and consistency is what lets you compare months.

For Laravel apps I also log the deployed release in the app itself so incidents can be tied to a deploy:

// config/app.php
'release' => env('APP_RELEASE', 'dev'),

// app/Providers/AppServiceProvider.php
Log::shareContext(['release' => config('app.release')]);

Step 3: Measure rework, the early warning signal

Rework is the share of lines that are changed again soon after they were written. A rough but useful version from git:

# Files changed in month M that were changed again within 21 days
git log --since=2026-09-01 --until=2026-09-30 --name-only --pretty=format:'%H %cI' \
  | awk 'NF==1' | sort | uniq -c | sort -rn | head -20

I look at the top 20 files and compare against the previous quarter. When AI tools are helping, rework stays flat or drops. When they are generating code nobody fully understood, the same files show up month after month, usually service classes and Vue components with duplicated logic.

Step 4: Split by AI-assisted vs not, carefully

Teams always want to know whether AI-assisted PRs are better or worse. You can label PRs (a checkbox in the PR template works), but be careful: developers reach for AI on certain kinds of tasks, so the two groups are not comparable. I report it, but I never use it as the headline number. The headline is always the whole team's delivery metrics over time.

<!-- .github/pull_request_template.md -->
- [ ] AI assistant generated a significant part of this change
- [ ] I have read and understood every line I am submitting
- [ ] Linked ADR (if this changes a recorded decision): ___

The second checkbox matters more than the first.

What the numbers looked like on that client project

After three months of measuring and applying the earlier parts of this series (stricter review checklist, characterization tests, dependency checks, static analysis gates and ADRs), the same team moved from two deploys a week to about five, median lead time fell from around four days to under two, and change failure rate came back down to roughly where it was before the rollout. Survey "felt speed" barely changed. The difference was that the gains finally showed up where the business could see them.

Diagnostic checklist

  • [ ] We have at least three months of baseline numbers from before (or early in) the rollout
  • [ ] Production deploys are recorded automatically, not from memory
  • [ ] Hotfix and revert PRs are labelled consistently
  • [ ] We report medians and p85, never only averages
  • [ ] PR cycle time is split into waiting-for-review and in-review time
  • [ ] Rework rate is checked monthly for the top changed files
  • [ ] AI vs non-AI comparisons are treated as hints, not conclusions
  • [ ] Metrics are reviewed as a team, never used to rank individuals

FAQ

Are DORA metrics enough on their own?

No. They tell you whether delivery improved, not why. Pair them with PR cycle time, rework rate and a short retro conversation every month.

Our team is small. Is this overkill?

For a team of three, a spreadsheet updated once a month from the scripts above is plenty. The point is a consistent baseline, not a dashboard.

Should we buy an engineering metrics platform?

Maybe later. Most of them compute the same numbers from GitHub and your CI. Start with scripts so you understand the definitions before paying for charts.

What if the numbers get worse after AI rollout?

That is useful information, not failure. Usually it points to review and testing gaps. Tighten the review checklist and gates from Parts 1 and 4 before deciding the tools are the problem.

Can these metrics be gamed?

Anything can, especially if tied to performance reviews. Keep them as team-level signals for improving the system, never as targets for individuals.

How long before we see a real trend?

Give it at least two to three months. Week-to-week numbers swing too much on a small team to mean anything.

That wraps up Maintaining AI-Written Code in Production. The thread across all six parts: AI makes writing code cheap, so the work that keeps software healthy moves to review, tests, dependencies, gates, recorded decisions and honest measurement. If you want a second pair of eyes on your deploy pipeline and monitoring, that is the kind of server setup and maintenance work I do at Loop Origin, or you can get in touch with the numbers you have.

Follow & Connect

๐Ÿ“ฐ Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.