My LLM pipeline spent 13 of 18 minutes waiting for answers that never came

I maintain an open-source pipeline that turns a research paper or topic into a cited commercialization screening report. Six LLM agents split the work: three analysts, a writer, a reviewer and a scorer. On 9 November one production run took 1,072 seconds. When I read the logs, 774 of those seconds had gone to requests that had stopped delivering anything.

This is how I found out why, what I changed, and why the root cause is still there. Every change and number comes from a public pull request in the repository, linked at the end.

1. First, make retries visible

The first clue was a 781-second run the day before. In the same run, on the same model, the reviewer wrote 2,880 tokens in 50 seconds while the writer took 282 seconds for 6,487 tokens. That gap made no sense as generation speed.

The writer had actually sent two requests. The first was cut at its 150-second timeout; the second finished. None of this showed up anywhere: usage counted only successful requests, and the retry notice went to stdout, which the worker discarded.

So the first change wasn't an optimization. It was logging (#362): every provider attempt, whether it returned, timed out or failed, now leaves a row with its stage, attempt number, outcome and duration.

2. Not slow generation: stalled connections

With per-attempt rows, the 1,072-second run was easy to read (#363):

• Writer: stalled for 302 seconds, then 312 seconds; the third attempt returned in 95.
• Scorer: stalled for 160 seconds; the retry returned in 27.

The same request came back in 27 to 95 seconds on a retry, so generation speed wasn't the problem. The connection was: data stopped flowing mid-answer. A plain request can only learn that by waiting out its whole timeout, and the longer the timeout, the more time it wastes.

3. Stream everything

I switched all six agents to streaming (#363). Answers arrive in chunks, and the wait for each chunk is capped at 60 seconds, so a dead connection is noticed within a minute instead of after several.

Before merging, I wrote down a validation plan and a decision rule (a pre-registration), then ran five topics:

• 0 of 35 attempts stalled.
• All 5 reports completed, with technology readiness in the expected range.
• 31.4 minutes serial, against 53.7 minutes for the same topics the day before.

It wasn't free: 607,840 tokens against 423,040, partly because that model version wrote longer evidence answers.

4. Faster, but 5 of 11 attempts failed

The first production run with streaming took 343 seconds, against 781 and 1,072 for the two before it. But 5 of its 11 provider attempts failed, all in the same pattern (#364):

1. The stream went silent for 60 seconds and was cut.
2. A retry two seconds later failed with a connection error within 15 to 18 seconds.
3. The next attempt succeeded in 4 to 44 seconds.

The writer and the reviewer each used all three attempts. One more failure on either and the run would have failed.

My theory was that retries went down the same connection pool that had just died: the OpenAI SDK's with_options copies a client but shares its HTTP pool. So a transport retry now rebuilds the client on a new connection, and streamed calls get five attempts instead of three (#364).

5. Fresh connections didn't help

The next production run was cancelled at 728 seconds (#365). The writer's request failed on eight new connections in a row: six went silent for 60 seconds, and two raised connection errors at 15 and 33 seconds. In the same run, the three analysts' smaller requests went through.

With a new connection every time, the stale-pool theory was dead. The same request of about 22,000 tokens, sent from my own machine to the same endpoint, got its first chunk in 6.9 seconds. That pointed at the path between the server and the model provider.

6. A probe that costs nothing

The server runs on Railway in Singapore; the model is served from Alibaba Cloud Model Studio's mainland China endpoint. To measure the path itself, I added a link probe to the admin console (#365, #366). It posts requests of 1 KB to 100 KB to the mainland endpoint and to the same service's international endpoint. The requests carry no key: both servers read the whole body before answering 401, so nothing is billed, and the timing is the path's.

Two readings on 9 November, 09:06 to 09:08 UTC (#366):

Across 9 and 10 November (#368), a 100 KB upload failed to finish within 30 seconds 3 times in 10 and a 30 KB request went unanswered once in 3, while the international endpoint answered every size in under 0.3 seconds. Large requests stall intermittently and small ones mostly pass, which matches the writer failing while the analysts got through.

I later ran the probe from my own machine (#369): 100 KB got through once in six tries and 30 KB once in three, while the international endpoint answered all fifteen in under 0.8 seconds. The mainland endpoint stalls from two different networks, so it isn't only the server's route.

7. Why the root cause is still there

The reliable fix is the international endpoint. But it needs a separate international account, which means giving up the free quota on the mainland account. For now I've stayed on the mainland endpoint and rely on streaming and retries to limit the damage.

So the stalls still happen. What I can do is make sure users don't pay for them.

8. Users shouldn't pay for my network

Each account gets a fixed number of runs per day. Until now, a run was refunded only if it never started; once it started, any failure counted.

Now one more case is refunded (#368): a run that fails because the model provider couldn't be reached gets the day's unit back, at most three times a day per account. The cap is there because an unlimited refund would switch the daily limit off exactly when the path is at its worst. Cancellations, rejected keys, too little evidence and other failures still count.

I also fixed a misleading message. A timeout used to suggest "try a narrower topic", although the topic had nothing to do with it. The page now says the model provider couldn't be reached and that nothing in the request caused it.

9. Same rules when switching models

When the deployment moved to a new model, I ran the usual pre-registered spot check (#369): 5 of 5 topics completed and 4 of 5 landed in the expected range, so by the rule set in advance the model stays.

That batch ran from my own machine, and it confirmed the path problem again: 28 of 61 provider attempts never returned, and 46% of the time went into them. Two calls succeeded only on their fifth attempt. Under the old three-attempt cap, 3 of the 5 runs would have failed.

What I'd tell myself a week ago

• Log every attempt before optimizing anything. If you can't see it, every fix is a guess.
• A plain request hides a stall until its timeout. Streaming with a per-chunk bound surfaces it within a minute.
• Retry on a fresh connection. It wasn't the cause this time, but it's the right default.
• Measure the path, not just the model. A probe that sends no key costs nothing and answers faster than paid reruns.
• Don't bill users for failures they didn't cause. Refund the unit and say plainly what happened.

About the project

Academic Commercialization Agent is MIT-licensed. Give it a research paper or topic and it writes a cited screening report on technology readiness, patents and market. It's a first-pass draft, not due diligence.

• Code: https://github.com/shuxiachai/academic-commercialization-agent
• Online app, with sample reports you can read without signing up: https://academic-agent.cc
• The changes above: #362, #363, #364, #365, #366, #368 and #369 in the repository's pull requests; the pre-registrations and results are in docs/.

A question for anyone running LLM calls across regions: have you seen large requests stall like this, and what did you do about it?

I'm Jingbo, a student at the University of Sydney. This post was drafted with AI assistance; every number was checked against the pull requests listed above.