Detecting Flaky Tests and Finding Their Root Causes with Strands Agents and Gemini

A few years ago (three years at most), one of the biggest challenges QA Engineers faced was: how do we convince the business and the team that we should invest in automated tests? Today, with AI, this issue has been mostly solved. We don't spend much time coding automated tests because we have AI to do it for us, but a new challenge appears: flaky tests. In this article, I will explain what a flaky test is, how I built a tool to help identify them, and how I used AI to run a first analysis.

What Is a Flaky Test and Why It Matters

Before going into more detail about the solution, let's understand what a flaky test is and why we should worry about it.

In a few words, a flaky test is an automated test that produces both passing and failing results without any change to the code or to the test itself.

If they pass, why should we monitor them? In the end, they pass.

That may be your first thought, but once you understand what a flaky test is, you see that in big projects it can hurt the team's trust in their own product and tests, real failures start being ignored, and the CI time and cost per pipeline go up.

Quick overview of the project

Based on these concerns, I developed this project. The code is available on GitHub: flaky-test-detector.

In the first draft, I planned only to parse the test results, score them per test and commit, and do the analysis manually:

After this first draft, I thought: why not integrate an AI to help me with the analysis? With this in mind, I added new features to support the analysis and the integration with AI models. Today, the solution looks like this:

The project is organized like this:

flaky-test-detector/
├── .github/workflows/
│   └── flaky-report.yml   # weekly analysis + result files
├── agents/
│   └── agent.py           # optional AI agent: root-cause analysis of reports
├── docs/
│   └── how-it-works.svg   # diagram used in this README
├── flaky_detector/
│   ├── __init__.py
│   ├── __main__.py        # CLI
│   ├── parsers.py         # JUnit XML parser
│   ├── github.py          # fetches JUnit artifacts from GitHub Actions
│   ├── scoring.py         # flakiness scoring and ranking
│   └── report.py          # result files (JSON / CSV / Markdown)
├── scripts/
│   └── generate_junit.py  # generates sample JUnit XML files
├── results/               # sample JUnit XML files (git-ignored)
├── reports/               # generated reports and AI analyses
├── tests/
├── pyproject.toml         # project metadata and dev dependencies (uv)
├── requirements.txt
└── README.md

Scoring and Ranking the Flaky Tests

Let's understand how the project scores and ranks the flaky tests. It starts by receiving a JUnit report in XML format (a standard output of most test frameworks, with one <testcase> per executed test), and parsers.py is responsible for parsing the information in this file into the proper format:

class Outcome(str, Enum):
    PASSED = "passed"
    FAILED = "failed"
    ERROR = "error"
    SKIPPED = "skipped"


@dataclass(frozen=True)
class TestResult:
    """A single test execution in a single CI run."""
    __test__ = False  # keep pytest from trying to collect this class

    name: str  # "<classname>::<name>"
    outcome: Outcome
    commit_sha: str | None
    run_id: str | None
    time: float = 0.0
    message: str | None = None
    source: str | None = None  # file the result came from

    @property
    def passed(self) -> bool:
        return self.outcome is Outcome.PASSED

If the file isn't a valid JUnit XML, an exception is raised; if it is valid, the parsing proceeds. When scoring.py is called, it uses the FlakinessScorer class, which is responsible for taking the TestResult objects and turning them into TestScore objects. Each result is grouped by test name and then by commit, and for each (test, commit) pair the scorer checks the outcome. It ignores:

• skipped tests, since they say nothing about flakiness;
• results from JUnit files without the commit_sha property (the commit id), since they can't be compared with each other.

Both failed and error count as a failure.

@dataclass(frozen=True)
class TestScore:
    """Flakiness score of a single test."""
    __test__ = False  # keep pytest from trying to collect this class

    name: str
    commits: int  # commits with usable data for this test
    flaky_commits: int  # commits where the test had mixed outcomes
    commits_where_flaky: tuple[str, ...] = field(default=())
    error_messages: tuple[str, ...] = field(default=())  # distinct failure messages, first seen first

    @property
    def score(self) -> float:
        return self.flaky_commits / self.commits if self.commits else 0.0


class FlakinessScorer:
    def score(self, results: Iterable[TestResult]) -> list[TestScore]:
        outcomes: dict[str, dict[str, set[bool]]] = defaultdict(lambda: defaultdict(set))
        messages: dict[str, dict[str, None]] = defaultdict(dict)  # ordered, de-duplicated
        for r in results:
            if not r.commit_sha or r.outcome.value == "skipped":
                continue
            outcomes[r.name][r.commit_sha].add(r.passed)
            if not r.passed and r.message:
                messages[r.name][r.message] = None
        scores = []
        for name, by_commit in outcomes.items():
            flaky = tuple(sorted(sha for sha, seen in by_commit.items() if len(seen) > 1))
            scores.append(
                TestScore(
                    name=name,
                    commits=len(by_commit),
                    flaky_commits=len(flaky),
                    commits_where_flaky=flaky,
                    error_messages=tuple(messages[name]),
                )
            )
        return scores

The script marks a test as flaky when, for the same commit, the outcomes are both {True, False}: the test failed and passed with the same code base. It then generates the data that the Ranker needs, in the following format:

{
  "name": "tests.checkout.test_cart::test_9",
  "commits": 10,
  "flaky_commits": 3,
  "commits_where_flaky": [
    "gen0007d154385",
    "gen05893e77ff9",
    "gen059095256f8"
  ],
  "error_messages": [
    "AssertionError: timeout waiting for gateway",
    "ConnectionError: connection reset by peer"
  ]
}

When the FlakinessRanker class is called, it uses this data to rank the tests. It only considers tests with data from at least 5 commits (min_commits), so the score can be trusted, and it flags a test as flaky when its score is 0.1 or higher (threshold). Both values can be changed:

def __init__(self, threshold: float = 0.1, min_commits: int = 5) -> None:
    if not 0.0 <= threshold <= 1.0:
        raise ValueError("threshold must be between 0 and 1")
    if min_commits < 1:
        raise ValueError("min_commits must be >= 1")
    self.threshold = threshold
    self.min_commits = min_commits

The score is the total of flaky_commits (the number of commits where the same test passed and failed) divided by the total of commits available for that test in the pipeline.

One test ran on 10 commits, and on 3 of them it both passed and failed on the same commit.

For this example, commits is 10 and flaky_commits is 3, so the score is 0.3.

With this first implementation, we can already see the first results. I created some JUnit reports and, after running the command below, I got this output:

python -m flaky_detector analyze ./results/*.xml --threshold 0.1

Each line shows the score, the test name and the number of flaky commits out of the commits with data for that test.

Reports and Agent Creation

As I mentioned before, after the basics I thought: why not improve it? The first step was creating reports of the results. Since I was already thinking about integrating an AI agent, the report needed a format that suits it well. The report is created by report.py, in JSON, CSV and Markdown formats (using the --output-dir option). In this article, we will focus on the JSON format, since it is the one used by the AI agent:

{
  "generated_at": "2026-11-10T19:32:23+00:00",
  "source": "local",
  "threshold": 0.1,
  "min_commits": 5,
  "results_parsed": 19644,
  "tests_analyzed": 109,
  "flaky_count": 12,
  "flaky_rate": 0.1101,
  "tests": [
    {
      "name": "tests.checkout.test_cart::test_9",
      "score": 0.7,
      "commits": 60,
      "flaky_commits": 42,
      "is_flaky": true,
      "commits_where_flaky": [
        "gen0007d154385",
        "gen059095256f8"
      ],
      "error_messages": [
        "AssertionError: timeout waiting for gateway",
        "ConnectionError: connection reset by peer",
        "AssertionError: race condition on shared state",
        "AssertionError: expected 200, got 503",
        "TimeoutError: operation timed out after 30s"
      ]
    }
  ]
}

To build the AI agent, I used the Strands Agents SDK. My first contact with this SDK was at the Train the Trainers workshop (thank you, AWS EMEA, for the invite), and since the way to configure the agent is very simple, I decided to use it in this project.

In this project I am using GeminiModel (the API key is read from the GOOGLE_API_KEY variable in .env), but you can switch to a different model; see Model Providers.

from strands import Agent
from strands.models.gemini import GeminiModel

model = GeminiModel(
    client_args={"api_key": api_key},
    model_id="gemini-2.5-flash",
)

agent = Agent(
    model=model,
    tools=[fetch_json_report, export_analysis],
    name="Flaky Test Analysis Agent",
    description="This agent analyzes flaky test reports and provides insights on flaky tests, their scores, trends, and it analyzes the error messages to provide insights on the root causes of flakiness. And when it is asked to summarize the flaky test report, it will use the fetch_json_report tool to retrieve the report data and then provide a summary of how many tests are flaky and which are the worst and the possible root causes based on the error messages.",
    system_prompt="You are a helpful assistant that analyzes flaky test reports. Use the fetch_json_report tool to retrieve report data when needed. Always read the error_messages of each flaky test and infer the possible root causes (timeouts, race conditions, network/gateway instability, shared state, etc.), saying which tests each cause affects and how confident you are. After you finish analysing, call export_analysis once with summary, root_causes and recommendations to save the Markdown report, then tell the user the file path.",
)

When building an agent, we need to provide it with tools so it can do its job. For this app, I created two tools:

• fetch_json_report: fetches the generated JSON reports. It is responsible for finding the files based on a timestamp or a period and returning them in a single response, ordered from oldest to newest.
• export_analysis: saves the analysis made by the AI in a Markdown file.

With this, it is possible to call the agent to run the analysis:

def main() -> None:
    parser = argparse.ArgumentParser(description="Analyze flaky test reports with an AI agent.")
    parser.add_argument("period", nargs="?", default="last_24_hours", choices=list(PERIODS),
                        help="period of reports to analyze (default: %(default)s)")
    args = parser.parse_args()
    label = args.period.replace("_", " ")
    agent(f"Summarize the flaky test reports from the {label} (period={args.period}): "
          "how many tests are flaky and which are the worst? After finishing, analyze the error messages to infer the possible root causes of flakiness and give recommendations to fix them. And create the report file with export_analysis tool, then tell me the path of the Markdown file that was written.")


if __name__ == "__main__":
    main()

The period is passed as an argument (for example, python agents/agent.py last_7_days), and the agent provides a report like this one: Flaky Tests Analysis - Last 24 Hours.

What Can Be Done With This

Ok, now it is possible to identify flaky tests and even analyze them. What can I do with this? First, you can start implementing it in your team. Monitoring your tests and flagging the flaky ones can help you improve the team's trust in what they are building, reduce CI costs, and avoid real bugs being ignored because the test passes on the second or third execution.

When should you run this? It depends on how your team works:

• If your team has continuous deployment, and deploys every day or every week, it is interesting to run this once per week. It can also help if you have problems merging PRs in lower environments, to understand what is happening.
• If you don't have continuous deployment and deploy once per month, my recommendation is to run this on every deployment, always comparing the latest executions to see whether the number increased or not.
• The AI analysis is optional. You can run just the Score and Rank steps and, at the end of a cycle, a month or even a quarter, use the agent to give you feedback.

Limitations and Integrations

• Today, the JUnit XML file must have the property <property name="commit_sha" value=""/> (the commit id) and the property <property name="run_id" value=""/>. Since the solution was built with GitHub Actions in mind, it may need changes for other frameworks and pipelines. When the results are fetched from GitHub Actions (--repo), both values are taken from the GitHub run itself.
• There is already a workflow to integrate with GitHub Actions (.github/workflows/flaky-report.yml). It runs every Monday, generates the reports and uploads them as an artifact. If you plan to add it to your own workflow, it may need adjustments, and the analyzed repository must publish its JUnit XML files as workflow artifacts.
• For the AI model, you can set up your favorite one. Remember to check Model Providers to adapt it to your context.
• Another suggestion is to integrate with a reporting tool, like Grafana, so you can see the behavior of your flaky tests at different points in time and whether they are decreasing or not.

Conclusion

Flaky tests are not just an annoyance: they erode the team's trust in the tests and hide real failures. Measuring them is the first step to dealing with them, and a simple score per test and commit is enough to get started. The AI analysis gives you a first look at the possible root causes, but it doesn't replace a human reviewing the results, since failures caused by the infrastructure can also look like flakiness. Give it a try and see how your own pipeline behaves!