Claude Batch API Returned Results Out of Order: 59% of My Labels Wrong

My label distribution chart was perfect. 38% billing, 22% login, 15% shipping, and so on down the line, exactly what I expected from six months of support tickets. The classifier I trained on those labels hit a validation accuracy I was happy with. I almost shipped it.

Then I hand-labeled 200 tickets for a gold set, compared them against Claude's labels, and got 41.5% agreement. Claude wasn't that bad at reading support tickets. My pipeline was. The Claude Batch API results came back out of order, I matched them to my inputs by position, and 2,940 of my 5,000 labels ended up attached to the wrong ticket.

Every dashboard I had looked fine, and here's why: every dashboard I had was a histogram.

TL;DR

• The Claude Message Batches API does not return results in the order you submitted requests. The docs say so. I didn't read that line.
• Zipping results with inputs by index silently mislabels data. In my run, 3,861 of 5,000 results were out of position, and 2,940 labels ended up wrong (59%).
• Aggregate checks can't catch it. Shuffling preserves label distributions, counts and per-class ratios. Only a per-row check against ground truth exposes it.
• Fix: set custom_id to your own stable primary key and join on it. Assert the ID sets match, and handle errored, expired and canceled results explicitly.
• Dropping failed results before zipping causes the same bug even if the order happened to be right.

What was I building?

A support-ticket router. I had 5,000 historical tickets with no labels, and I wanted a small, cheap classifier that sorts incoming tickets into six queues. The plan:

1. Have Claude label all 5,000 tickets through the Batch API (half price, no rate-limit babysitting).
2. Train a small classifier on those labels.
3. Route live tickets with the small model and keep Claude out of the hot path.

The labeling batch cost me under four dollars and finished in about 40 minutes. That part worked great. Everything after it was built on sand.

Why does the Claude Batch API return results out of order?

Because a batch isn't a queue. It's a pile of independent requests that Anthropic processes in parallel, on its own schedule, whenever capacity is free. Results get written as requests finish, and short requests finish before long ones. The documentation states plainly that results may come back in any order and that you should match them using custom_id.

I knew custom_id existed. I filled it with f"req-{i}", where i was the loop index. Then I threw it away on the way back in. Here's the code I actually ran:

import anthropic

client = anthropic.Anthropic()

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"req-{i}",
            "params": {
                "model": MODEL,
                "max_tokens": 20,
                "system": LABEL_PROMPT,
                "messages": [{"role": "user", "content": t["body"]}],
            },
        }
        for i, t in enumerate(tickets)
    ]
)

# ...later, after the batch ended
labels = [
    r.result.message.content[0].text.strip()
    for r in client.messages.batches.results(batch.id)
    if r.result.type == "succeeded"
]

for ticket, label in zip(tickets, labels):  # the bug
    ticket["label"] = label

Two bugs in four lines. zip assumes the results stream lines up with tickets. It didn't. And the if r.result.type == "succeeded" filter dropped 14 errored requests, so even a perfectly ordered stream would have shifted every label after the first failure by one slot. zip then quietly truncated to the shorter list, so the last 14 tickets just never got a label, and nothing complained.

Why didn't any of my checks catch it?

Because every check I wrote was permutation-invariant. If you shuffle a column of labels, the following stay exactly the same:

• The label histogram
• The count of each class
• The number of rows
• The fraction of "other" labels I was watching as a quality signal
• The average label length, the parse failure rate, the cost

I checked all of those. All of them were green. I was measuring the shape of the pile and never asking whether any single card was in the right envelope.

Even my validation accuracy lied. I split train and validation from the same batch-labeled set, so the model was being graded against the same scrambled answer key it learned from. It learned something real-ish (some patterns survive partial shuffling), and the validation set happily confirmed it.

How bad was the damage, exactly?

Bad enough to make the classifier useless, but not so bad that it looked broken. Once I re-joined results by custom_id, I could measure it:

That "same label by chance" row explains why nothing looked catastrophic. My classes are skewed: billing alone is 38% of tickets. Pick two random tickets and they share a label roughly 24% of the time. So nearly a quarter of my misplaced labels were correct by luck, and a quarter of the rest were "close enough" (login vs. account access) that spot-checking five rows didn't scream at me.

The order also wasn't random in a pretty way. Results came back in clumps that roughly followed completion time. The first few dozen rows were mostly in place, which is exactly where I looked when I eyeballed the output. The universe has a sense of humor.

From batch to discovery took 9 days. Six of those were spent tuning a classifier on garbage.

How do you match Claude Batch API results to requests correctly?

Use your own stable key as custom_id, build a dict from the results, and join on the key. Never rely on position. Here's what replaced the bug:

by_id = {t["id"]: t for t in tickets}  # ticket IDs like "tkt_88213"

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": t["id"],
            "params": {
                "model": MODEL,
                "max_tokens": 20,
                "system": LABEL_PROMPT,
                "messages": [{"role": "user", "content": t["body"]}],
            },
        }
        for t in tickets
    ]
)

seen, retry = set(), []
for r in client.messages.batches.results(batch.id):
    seen.add(r.custom_id)
    if r.result.type == "succeeded":
        by_id[r.custom_id]["label"] = r.result.message.content[0].text.strip()
    else:  # errored, expired, canceled
        retry.append(r.custom_id)

missing = set(by_id) - seen
assert not missing, f"{len(missing)} requests have no result at all"
print(f"labeled={len(seen) - len(retry)} retry={len(retry)}")

A few rules that matter here:

• custom_id must be unique within the batch, 1 to 64 characters, letters, digits, hyphens and underscores. My ticket IDs already fit. If yours don't, map them to a safe ID and keep the map.
• Use your real primary key, not a loop index. req-{i} works only as long as the list you rebuild later is identical to the one you submitted. Mine got re-sorted by a dedupe step between submit and join. A loop index is a position wearing a fake mustache.
• Treat non-success results as first-class. errored, expired and canceled each need a decision. I resubmit them as a small follow-up batch.
• Assert the ID sets match. The missing check costs one line and would have caught the truncation from zip on day one.

What should you check besides the join?

A per-row test against ground truth, even a tiny one. After the fix, I re-ran the 200-ticket gold set: 184 of 200 matched, 92% agreement, up from 41.5%. That one number would have caught the bug in the first hour if I had built the gold set before training instead of after.

My checklist now, for any LLM batch job that produces labels or extractions:

1. Echo the key. Join on custom_id, never on order.
2. Count every result type. Log succeeded, errored, expired and canceled separately.
3. Set equality, not length equality. len(results) == len(inputs) passes when one ID is duplicated and another is missing.
4. Gold set first. Hand-label 100 to 200 rows before you trust any aggregate.
5. Spot-check from the end and the middle, not the top. The top is where order-dependent bugs hide best.

It's the same lesson for any async fan-out: thread pools with as_completed, Celery results, S3 multipart listings, the OpenAI Batch API. If work finishes out of order, results arrive out of order. The Claude Batch API just made it cheap enough to label 5,000 rows without thinking, so I didn't.

Does the Claude Batch API return results in order?

No. The Claude Message Batches API returns results in whatever order requests finish, which is not the order you submitted them. In my 5,000-request labeling job, 3,861 results came back at a different index, and matching by position mislabeled 2,940 tickets (59%) while every aggregate metric still looked normal. Put your own stable primary key in custom_id, join results to inputs on that key, handle errored, expired and canceled results explicitly, and validate against a small hand-labeled gold set before training anything on the output.

Written by the developer behind Preterview, an interview prep platform.