"Just re-run it."
Four words that quietly destroyed the trust of an entire engineering team.

If you've ever typed those words into a Slack thread, this story is for you.

A Red Build Nobody Believed

It was 4:47 PM on a Thursday. Priya, a backend engineer at a 12-person B2B SaaS startup, pushed a one-line fix for a typo in an email template. The pipeline went red.

She didn't read the logs. She didn't even open the failed job. She clicked Re-run all jobs, grabbed a coffee, and came back to a green build.

Nothing in the code had changed. The build had simply decided to pass the second time.

That moment, repeated dozens of times a week, is what we'll dissect in this case study. By the end of this post you will have:

• A clear definition of what a flaky test actually costs you (in hours, not vibes)
• A repeatable way to measure flakiness in your own pipeline
• A practical migration path from brittle scripted tests to a more resilient approach
• Code you can paste into your own repo today

The team's product was a subscription management dashboard. Their stack looked like most modern startups:

On paper, this was a healthy setup. In practice, the end-to-end suite had become the thing everybody worked around instead of relying on.

The Problem: Quantifying the Pain

The team's engineering manager finally did what every engineering manager should do before complaining: she measured.

She pulled 30 days of CI history and wrote a small script to classify every failed run by whether a re-run of the same commit later passed.

#!/usr/bin/env bash
# flaky-report.sh
# Counts runs that failed, then passed on retry for the same commit SHA.

gh run list --limit 500 --json databaseId,headSha,conclusion,attempt \
  | jq -r '
      group_by(.headSha)[]
      | select(any(.[]; .conclusion=="failure")
               and any(.[]; .conclusion=="success"))
      | .[0].headSha
    ' | sort -u | wc -l

The result was uncomfortable:

Total pipeline runs (30 days):     412
Runs that failed at least once:    171   (41.5%)
Failed-then-passed on same SHA:    139   (81% of failures)
Real failures (code was broken):    32

Eighty-one percent of the failures were not bugs. They were noise.

The real cost

She then did the math that got leadership's attention:

# cost_of_flakiness.py
flaky_failures_per_month = 139
avg_minutes_lost_per_failure = 22   # context switch + rerun + wait
engineers_blocked_ratio = 1.4       # others waiting on the same pipeline
hourly_cost = 55                    # blended, USD

hours_lost = (flaky_failures_per_month
              * avg_minutes_lost_per_failure
              * engineers_blocked_ratio) / 60

print(f"Hours lost per month: {hours_lost:.1f}")
print(f"Approx. monthly cost: ${hours_lost * hourly_cost:,.0f}")

Output:

Hours lost per month: 71.3
Approx. monthly cost: $3,923

That is nearly half of one engineer's monthly capacity, burned on re-running tests that were not even telling the truth. And the invisible cost was worse: engineers had started ignoring red builds. Eventually, a real bug slipped through because everyone assumed "it's probably flaky again."

What They Tried First (And Why It Didn't Work)

Before looking for a new approach, the team tried the usual suspects.

Attempt 1: Add retries. They set retries: 2 in the Playwright config.

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,
  timeout: 30_000,
});

Result: the pipeline went green more often, but run time increased by 35%, and the flakiness was now hidden rather than fixed.

Attempt 2: Add hard waits. Somebody sprinkled await page.waitForTimeout(3000) everywhere.

Result: slower tests, same flakiness, and a codebase full of magic numbers.

Attempt 3: A "flaky test quarantine." They tagged unstable tests and skipped them in CI.

Result: the quarantine list grew from 9 tests to 31 in six weeks. A quarantine that only grows is just a graveyard.

Diagnosing the Root Cause

The team finally sorted the 139 flaky failures by cause. This is the table that changed their thinking:

Nearly half of the pain came from tests that were coupled to implementation details of the UI. A selector like this one was typical:

// Looks innocent. Breaks the moment a designer touches the DOM.
await page.click('#root > div.layout > main > section:nth-child(2) button.btn-primary');

A developer changes a wrapper div, nothing visible changes for the user, and the test dies.

The Turning Point

The insight was simple but uncomfortable: the tests were asserting on how the page was built, not on what a user could do.

The team started evaluating tools that describe tests in terms of user intent and use AI to locate elements, instead of hard-coding locators. That is where they tried an AI-driven QA platform. If you want to try the same approach on your own flaky suite, you can start with BotGauge here and run your first AI-generated tests before committing to any large migration.

The workflow they adopted looked like this:

1. Describe the scenario in plain English instead of writing selectors.
2. Let the platform generate and execute the test across the UI, API, and database layers.
3. Rely on self-healing so that cosmetic UI changes don't produce false failures.
4. Wire it into CI so results gate the merge just like unit tests do.

A scenario that used to be 40 lines of Playwright became something like this:

Scenario: Customer upgrades from Basic to Pro plan
  Given I am logged in as a Basic plan customer
  When I open the Billing page
  And I choose the Pro plan and confirm payment
  Then I should see a confirmation message
  And my plan should show as "Pro" in Account Settings

No nth-child. No waitForTimeout. The test reads like the requirement it protects.

The Migration Playbook

They did not rewrite everything at once. Here is the order that worked, and it's a good template for you:

Week 1  Measure flakiness (the scripts above). Rank tests by failure count.
Week 2  Migrate the 10 flakiest critical-path tests (login, checkout, upgrade).
Week 3  Run old and new suites in parallel. Compare signal, not just speed.
Week 4  Retire migrated Playwright tests. Keep the rest until their turn.
Week 5+ Continue by risk tier. Delete the quarantine list as you go.

Wire the new suite into the pipeline like any other quality gate:

# .github/workflows/ci.yml
name: CI
on: [pull_request]

jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm test

  e2e:
    needs: unit
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Trigger AI end-to-end suite
        run: ./scripts/run-e2e-suite.sh
        env:
          E2E_API_TOKEN: ${{ secrets.E2E_API_TOKEN }}

The Results (After 8 Weeks)

Illustrative numbers from the scenario, tracked using the same scripts from earlier:

The most important number is "Failures that were real bugs." When a build turns red now, people read the logs. That is what a healthy signal looks like.

Key Lessons You Can Steal

1. Measure before you migrate. Without the 30-day report, this would have stayed an anecdote.
2. Retries hide problems, they don't solve them. Use them as a temporary bandage and track them as debt.
3. A growing quarantine is a smell. Set a hard cap and an expiry date for every quarantined test.
4. Test intent, not implementation. If a designer's CSS tweak can break your test, your test is testing the wrong thing.
5. Trust is the real product of a test suite. Speed and coverage matter, but only if people believe the result.

Your Turn

Run the flaky-report.sh script against your own repository this week. If more than 15% of your failures disappear on re-run, you don't have a testing problem, you have a trust problem.

Have a flaky-test horror story of your own? Drop it in the comments. I read every one.