We gave 6 AI coding CLIs the same concurrency bug and 6 hidden tests. All six passed. The diffs did not agree.

Arguments about which AI coding CLI is "best" usually rest on vibes or on synthetic leaderboards. So we took one small, real-shaped concurrency bug, gave it to six CLIs with the same prompt, and graded each fix with tests the CLIs never saw.

All six fixed it. That's the first result, and it limits what this post can claim: the task didn't separate them on correctness. What it did show is how long each took, what each spent, how big the diff was, and that one fix that passes every test still changes the queue's behaviour in a way the tests didn't check.

I build SpaceApp, a self-hosted, MIT-licensed workspace that runs these CLIs side by side. That's why we have them all installed. The runs below were headless, from a script, not inside SpaceApp, so this post is about the CLIs.

The bug

The repo is a throwaway Node package, jobq: an in-process job queue with concurrency, retries, exponential backoffMs and an onIdle() promise. It has no dependencies, about 90 lines in src/queue.js, and 5 tests on node:test. The bug is in the retry path:

async _run(job) {
  this._running++;
  job.attempt++;
  try {
    const value = await job.fn({ id: job.id, attempt: job.attempt });
    this._completed++;
    job.resolve(value);
  } catch (error) {
    if (job.attempt <= this.retries) {
      this._running--;                       // first decrement
      const delay = this.backoffMs * 2 ** (job.attempt - 1);
      setTimeout(() => {
        this._pending.push(job);
        this._next();
      }, delay);
      return;
    }
    this._failed++;
    this.onError?.(error, job.id);
    job.reject(error);
  } finally {
    this._running--;                         // runs on the retry path too
    this._next();
  }
}

Two things are wrong:

1. A failed attempt that will be retried decrements _running twice, once in catch and again in finally (a return inside catch doesn't skip finally). So the queue starts more jobs than concurrency allows.
2. A job waiting out its backoff is counted nowhere: not in _pending, not in _running. So onIdle() resolves while a retry is still due.

The 5th visible test fails on the baseline: "onIdle resolved before the retried job finished". Deleting just the extra decrement makes that test pass while onIdle() stays wrong. So a shallow fix is possible, and the hidden tests exist to catch it.

The prompt

Every CLI got this, verbatim:

The test suite of this repository fails (npm test). Find the root cause of the bug in the job queue and fix it. Do not modify the tests. Run npm test to check your fix. When you are done, reply with a short summary of the cause and the fix.

Method

• One run per CLI, one after another, each on a fresh copy of the repo.
• The same 1,200 s timeout for all. No run came near it.
• Headless, auto-approve, stdin closed, and a minimal environment (env -i): no MCP servers, no global agent instructions, throwaway config or home directories. Sign-ins were linked in, never copied.
• Wall time is the whole CLI process from start to exit, so it includes start-up.

Grading ran after each CLI had exited:

• the diff against the baseline, the files touched, and whether the CLI committed;
• npm test as the CLI left the repo;
• the visible tests again on a copy with test/ and package.json restored from the baseline, so a CLI that edited the tests couldn't pass that way;
• 6 hidden acceptance tests, never shown to any CLI, run 3 times each.

The hidden tests cover:

• onIdle() while only a retry is waiting;
• a retry waiting for a free slot;
• 30 flaky jobs at concurrency 3 with exact counters;
• backoff doubling;
• three onIdle() waiters;
• a synchronous throw.

Before trusting them, we ran them against three known states:

So the hidden tests do tell a shallow fix from a real one.

Results

All six:

• changed only src/queue.js;
• made no commit;
• left the visible tests passing;
• passed 6/6 hidden tests on all three runs.

Tokens and cost

These are the figures each CLI reported, or our API gateway's list-price estimate where the CLI printed none. They come from four different kinds of source, so don't read them as one ranking. Tokenizers differ, cache accounting differs, and Copilot rounds.

What the fixes look like

Three CLIs (Claude Code, OpenCode, Gemini) wrote the same fix as our reference. A counter for jobs waiting out their backoff replaces the extra decrement, the timer decrements it, and _isIdle() checks it. This is Claude Code's:

@@ -16,6 +16,7 @@ export class JobQueue {
     this._failed = 0;
     this._idleWaiters = [];
     this._seq = 0;
+    this._retrying = 0; // jobs waiting out their backoff: not pending, not running, but not done
   }
@@ -41,7 +42,7 @@ export class JobQueue {
   _isIdle() {
-    return this._pending.length === 0 && this._running === 0;
+    return this._pending.length === 0 && this._running === 0 && this._retrying === 0;
   }
@@ -65,9 +66,10 @@ export class JobQueue {
     } catch (error) {
       if (job.attempt <= this.retries) {
-        this._running--;
+        this._retrying++;
         const delay = this.backoffMs * 2 ** (job.attempt - 1);
         setTimeout(() => {
+          this._retrying--;
           this._pending.push(job);
           this._next();
         }, delay);

The others:

• Cursor: the same idea with a _delayed counter, decremented after the job is queued again.
• Grok: the same counter, plus a restructure that moves all retry scheduling into finally. It's correct, and it's the biggest diff (+18 −10). Whether that's a cleanup or churn depends on who reviews it.
• Copilot: a different design. The failed job keeps its running slot through the backoff, and the timer releases it:

@@ -59,16 +59,18 @@ export class JobQueue {
   async _run(job) {
     this._running++;
     job.attempt++;
+    let retryScheduled = false;
     try {
@@
       if (job.attempt <= this.retries) {
-        this._running--;
+        retryScheduled = true;
         const delay = this.backoffMs * 2 ** (job.attempt - 1);
         setTimeout(() => {
           this._pending.push(job);
+          this._running--;
           this._next();
         }, delay);
         return;
@@ -77,7 +79,7 @@ export class JobQueue {
     } finally {
-      this._running--;
+      if (!retryScheduled) this._running--;
       this._next();
     }

It passes every visible and hidden test. onIdle() is right, because a waiting job still counts as running, and the concurrency limit holds. So by the grading above, it's a full pass.

What the hidden tests didn't catch

This part came after the results, so it isn't part of the pass/fail above. It's a finding, not a score.

Reading the diffs, Copilot's design looked like it changed what "running" means. The package README says concurrency is "the most jobs running at the same time", and a job sleeping through its backoff isn't running. So we wrote two more checks from that sentence:

• P1: at concurrency 1, does another job run while one job waits out a 200 ms backoff?
• P2: does stats().running leave a waiting job out?

Five fixes and the reference pass both. Copilot's fails both: job b waited 190 ms behind the other job's backoff, and stats().running read 1 while nothing was running. Its fix is correct for the limit and for onIdle(), but now a backoff blocks a slot. With long backoffs and low concurrency, that's real throughput you lose without any test going red.

Two lessons for anyone grading agent output, ourselves included:

1. "Passes the hidden tests" is only as strong as the tests. We wrote six that caught the shallow fix, and they still missed a design change. We only saw it because we read the diffs.
2. The prompt said "fix the bug", not "keep the semantics". Copilot's version is a defensible reading of an underspecified task. If your spec lives in a README sentence, say so in the prompt or put it in a test.

Who didn't run, and why

N is 6 because that's how many really ran. We have more CLIs installed:

• Codex and DeepSeek: skipped on purpose. There was no quota or balance on those accounts at the time of the run.
• Kimi Code: installed and signed in, but the smoke run answered 403 Your current subscription does not have access to Kimi Code right now.
• Qwen Code: installed, but its credential check reported the quota as exhausted before the batch, so it wasn't run.
• OpenCode on its free default model: the first OpenCode attempt used the free Big Pickle model and stopped after 3.6 s with a 429 Free usage limit reached. OpenCode was then run on Claude Opus 5.5 with an Anthropic API key. So the OpenCode row measures the OpenCode harness with Opus, not OpenCode's free tier.

Honest limits

• n = 1 per CLI. Wall times and token counts would move on another run. One run tells you what happened once; it doesn't rank CLIs.
• One small bug in one ~90-line file. Every CLI solved it, so this task doesn't separate them on correctness. It shows speed, cost, diff size and fix design on an easy-to-medium concurrency bug.
• Each CLI ran a different model, so harness and model can't be separated. Claude Code and OpenCode both ran Opus 5.5 (16.3 s vs 16.9 s, same fix), which is the closest thing here to a same-model comparison.
• Copilot and Cursor pick their own model ("auto"). Copilot's smoke run 7 minutes earlier picked a different one (MAI-Code-1.1-Flash). agy's model came from its saved settings.
• Costs come from four kinds of source (the CLI's own figure, our gateway's list-price estimate, AI Credits, none), so they aren't comparable. Claude Code's figure is a list-price estimate for a subscription run.
• One shared Linux server under load (24 cores, load average about 16 during the batch, Node v26.13.0). The Claude and Gemini requests went through our gateway, and wall time includes that hop and CLI start-up.
• Environments were clean but not identical. Grok ran with its usual home directory, which holds its skills and memory, because its sign-in lives there. The others ran with throwaway homes and linked sign-ins.

If you want to repeat it

The ingredients are all in this post: the buggy _run above, the prompt, the shallow-fix trap, and the list of hidden tests. The rest of the queue is the obvious code around them (add, stats, onIdle, a _next loop). Write the hidden tests before you run anything, and check them against the baseline, a shallow fix and a reference fix, as we did. That step is what makes the pass column mean something.

What would you want in the next run: a harder bug that actually separates them, or the same CLIs pinned to the same model?