Two-second latency isn't an AI problem. It's an architecture problem your stack was never built to hide.

Intro

There's a gap between the demo and the deployment that nobody puts in the tutorial.

In the demo, you call the model, you get a response, you render it. Done. In production, that same call sits in your request path for one, two, sometimes four seconds, and your users are staring at whatever you left on screen while they wait.

We spend weeks shaving milliseconds off database queries and tuning cache layers to get a page from 180ms to 90ms. Then we bolt an LLM call onto the same request/response cycle and it takes two full seconds to resolve. Nobody notices the discrepancy until users start bouncing.

Here's the actual gap: traditional backend work happens in a latency band humans don't perceive. LLM inference happens in a band they very much do. Same request/response pattern, completely different experience on the other end of it. If you don't change how you architect for it, users feel the friction immediately, no matter how good the model's output is.

Failure mode 1: the silent request

Picture a support-ticket triage feature. User submits a ticket, the backend calls an LLM to classify and route it, and the endpoint doesn't return until the model finishes. On a slow model day, that's a three-second white screen with no feedback. The user assumes the button didn't register and clicks again. Now you've got two calls in flight for the same ticket. This is a made-up example, but if you've shipped an LLM feature behind a synchronous POST, you've lived some version of it.

The fix isn't a faster model, it's not treating the call as instant in the first place.

Failure mode 2: streaming as an afterthought

Streaming gets treated as a nice-to-have polish pass instead of the default. Teams ship the blocking version first, get it working end to end, and plan to "add streaming later." Later rarely comes, because by then the blocking version is load-bearing and touching it means retesting the whole flow. Token-by-token streaming isn't a UX nicety here, it's the difference between "the interface feels instant" and "the interface feels broken."

Failure mode 3: heavy work still living in the request path

Some LLM work genuinely doesn't need to block the response at all: summarizing a document after upload, generating suggested tags, running a background analysis pass. These still get built as synchronous calls because that's the default pattern every other endpoint in the codebase already uses. Moving that work to an async job with a webhook or polling endpoint is often a bigger UX win than any amount of prompt optimization, because the user was never meant to wait on it.

Failure mode 4: no optimistic path forward

Even with streaming and async jobs in place, there's often no way for the user to keep moving while the model works. Optimistic UI, showing the next step as available, prefilling a draft state, letting the user act on a provisional result, keeps people in the workflow instead of parked on a spinner watching a progress indicator that means nothing to them.

The reframe

None of this is really an "AI latency" problem. It's a request/response architecture problem that AI just made visible, because AI is the first thing in most stacks slow enough that users notice the assumption baked into synchronous request handling. The fix isn't specific to LLMs: stream what you can, push what doesn't need to block into the background, and give the user something to do while they wait.

When you're deciding which technique to reach for: if the output needs to be seen as it's produced, stream it. If it doesn't need to block the next user action at all, take it out of the request path entirely. If it has to block but you can predict the shape of the result, use optimistic UI to bridge the gap.

We hit this directly while building the analysis pipeline behind Cyclopt: the moment an LLM call entered the loop, the whole interaction model around it had to change, not just the backend call itself.

How is your team handling this? Streaming everything by default, pushing more to background jobs, or still treating the LLM call like any other backend request?