MCP CI gates need retry receipts for flaky downstreams

MCP CI gates need to distinguish two very different failures:

1. the server is actually broken
2. the downstream dependency is temporarily flaky

If both become hard failures, CI gets noisy.
If both are ignored, the gate stops meaning anything.

So I shipped @k08200/[email protected] with explicit sidecar retry policy for tool-call dry-runs.

The problem

A readiness gate that calls real MCP tools can hit transient downstream failures:

• 503 Service Unavailable
• 502 Bad Gateway
• 504 Gateway Timeout
• rate limits
• short network timeouts

But auth and permission failures are different. A 401 or 403 usually means the agent will fail in production too.

Those should stay visible unless the contract explicitly says otherwise.

Retry is opt-in per tool

mcp-probe now lets a sidecar contract define retry behavior per tool:

{
  "tools": {
    "logs_query": {
      "input": {
        "query": "service:web status:error",
        "timeframe": "1h"
      },
      "retry": {
        "attempts": 3,
        "delayMs": 1000,
        "retryOn": [429, 500, 502, 503, 504, "timeout", "rate limit"]
      },
      "expect": {
        "status": "pass"
      }
    }
  }
}

The important part: retry is not global magic.

It only happens when the sidecar explicitly opts in.

Receipts still show the flake

If a call fails once and passes on retry, the final result can pass, but the receipt still records every attempt.

That means CI can tolerate a transient downstream blip without pretending the run was clean.

Example shape:

{
  "tool": "flaky_read",
  "status": "pass",
  "source": "sidecar",
  "attempts": [
    {
      "attempt": 1,
      "status": "fail",
      "error": "503 Service Unavailable: transient downstream"
    },
    {
      "attempt": 2,
      "status": "pass"
    }
  ]
}

That is the distinction I want MCP CI gates to preserve:

• hard failures should block
• transient failures can be retried
• pass-after-retry should still leave a receipt

Install

npm install -D @k08200/mcp-probe

Or run directly:

npx @k08200/mcp-probe@latest --config mcp-probe.config.json --github-summary --receipt-file mcp-probe.receipt.json

GitHub release: https://github.com/k08200/mcp-probe/releases/tag/v1.12.0

npm: https://www.npmjs.com/package/@k08200/mcp-probe