Silent Failure #7: The URL Returned 200 From Everywhere and the Job Died at Dispatch

I checked that image URL twice before I submitted it. Once from my own box, once through a resizer sitting in the Netherlands, because I had been burned before by "works here, not there" and I thought two vantage points made me careful. Both answered 200, image/png. I submitted the keyframe job, the API answered accepted, and six minutes forty-six seconds later it came back with:

media URL could not be downloaded (timeout or source server error)

The file was online. I could see it. The platform that could not download it was telling me about an image I had personally uploaded thirty minutes earlier.

So I did the natural thing: I retried. Same 6m46s, same line, same place. That's 13 minutes and 32 seconds of an afternoon spent queueing for a failure I had "proven" impossible.

What I tried before I stopped arguing

My first theory was caching — maybe the worker hit a stale edge. I appended a cache-buster to the URL. Nothing changed, which should have told me the URL was never the issue, but I kept going: re-uploaded under a different filename, checked the image dimensions were inside the documented bounds (1024×1024, well inside), re-read the mode/schema rules to make sure first_frame wasn't silently rejected as a string. Every one of those checks passed, and every one of them passed for the wrong reason.

For a while I was convinced the validator did a HEAD and the worker did a ranged GET, and that some header was missing. That was me inventing a mechanism to keep my original assumption alive. Dumb, but I've done it before and I'll do it again.

The one variable I had not touched was the host. I mirrored the exact same bytes to a domestic bucket, resubmitted with nothing else changed, and it rendered in 5m56s: 720×720, 5.2 seconds of video, 1.7 MiB. Same image. Same parameters. Same account.

I was wrong about what those two green checks proved. They proved my egress worked, twice, and nothing else. I re-ran the probes today, five days later, on a URL of the same kind, to make sure I wasn't remembering a fluke:

local  GET  -> 200 image/png 1,068,796 bytes 2.85s
DNS         -> 199.232.161.55 / 2a04:4e42:8c::311
weserv GET  -> 200 image/png 1,182,443 bytes 3.81s

Three green answers, and all three green answers are about my network path. The worker that fetches the URL minutes later lives somewhere else entirely. That is the whole bug, and it is not really a bug in the vendor's code — it's a bug in what I accepted as proof.

Accepted is not fetched

The submission endpoint did what it was built to do: it validated my JSON, wrote a record, and returned quickly. That response is a statement about the queue, not about my host. Between "accepted" and "dispatch" there is a gap of minutes and a change of network, and everything I had verified lived on the wrong side of that gap.

Two runs, 6m46s each, and the error string never changed. timeout or source server error names two causes and commits to neither. I asked for a root cause and never got one I could use — I don't know whether it's an egress policy on that region, a prefix that doesn't route, or something about the CDN's anycast edge being fine for TLS-and-HTML traffic and miserable for a 1 MB object pulled by a worker with a 10-second budget. I know the observation and the workaround, and I built the fix on exactly that much.

The fix is a hardcoded list of hosts I no longer hand to a third party's fetcher:

# generate_video.py
UNFETCHABLE_HOSTS = ("ucarecd.net", "ucarecdn.com")   # backend can't fetch, mirror first

if any(h in url for h in UNFETCHABLE_HOSTS):
    url = mirror_to_domestic(url)   # same bytes, host the worker can actually reach

Crude. It also encodes a fact I could not get from documentation: which hosts a specific worker pool cannot reach. Note that the tuple is a list of hosts, not of reasons — I have no idea why, so I can't generalize it, and the next host that behaves like this will get past the check and cost me another 13 minutes.

The second guard came from the same afternoon. The response body reports seconds, size and quality — those are suggestions the backend accepted, not what it produced. So the script stopped trusting the response and started parsing the container: ftyp, moov, mvhd duration, tkhd dimensions, mdat length, hand-rolled because there's no ffprobe in this container. A 200 with a plausible JSON body and a truncated mdat would otherwise pass every check I cared to write.

My take, and I'll take the argument: any URL you hand to a third-party server-side fetch is untrusted until something in that vendor's region has fetched it, and "accepted" should start a timeout window, not end your verification. The corollary is worse and I'll say it plainly — my local curl is a test of my own egress, and I keep filing its results under a heading that says the file is fine.

I've only measured this once, two hosts, two runs, 13m32s on one Tuesday.

What I'd test first

1. Time the terminal state, not the request. Submit a 1×1 PNG into your own pipeline and log accepted_at, terminal_at, reason. If your gap is minutes, your detection lag is minutes and no amount of green curl output changes that.
2. Get one probe on the worker's side of the wall. A cheap box in the dispatch region, or ask support which region runs the worker. All three of my green checks were in the wrong place.
3. Assert on the artifact. Parse duration and dimensions out of the media file itself; treat echoed parameters in the response body as a request echo, because that's all they are.

My take on the fix that matters: alerting on the 202 is a footgun, and a pipeline that can't tell you its own dispatch latency has no failure detection at all.

If you run an async job API on Python 3.11 + Celery that fetches user-supplied URLs: what's your measured accept→terminal-failure gap, and are you alerting on the status code or on the artifact? Mine was 6m46s twice, and I only found it because a video came back empty. If you think a retry-with-backoff policy covers this class of failure, I want the case where the retry happily re-picked the host your worker can't reach — comments are open.