Dev.to AI 🤖 Ai 👁 0 📖 8 min read

Latency Is a Product Decision

Why slow AI features are usually a design problem, not just a model problem A field sales representative is standing in front of a customer. She opens the mobile app and taps a new button: "Prepare me for this visit."

Why slow AI features are usually a design problem, not just a model problem

A field sales representative is standing in front of a customer.

She opens the mobile app and taps a new button: "Prepare me for this visit." The AI will summarize recent orders, open complaints, overdue invoices, and suggest what to talk about.

A spinner appears.

Five seconds. Ten seconds. Fifteen.

The customer is waiting. The representative puts the phone back in her pocket and starts the conversation without it.

The summary arrives twenty seconds later. It is excellent. Nobody reads it.

This is a hypothetical example, but it describes a very common way AI features fail. Not because the answer is wrong, but because it arrives at the wrong moment.

Most discussions about AI latency focus on the model: which one is faster, how many tokens per second, whether a smaller model is good enough.

Those questions matter. But in most products, the bigger decision is somewhere else:

When does the user actually need the answer, and what should they experience while they wait?

Users Don't Experience Latency. They Experience Waiting.

Latency is a technical measurement. Waiting is a human experience.

The same five seconds can feel very different depending on the situation:

  • Five seconds after submitting a long report for analysis feels fast.
  • Five seconds after pressing a search button feels slow.
  • Five seconds with visible progress feels shorter than five seconds with a spinner.
  • Five seconds while a customer is watching feels like forever.

Traditional enterprise software trained users to expect certain things. Clicking a button should respond immediately. Saving a form should take a moment. Generating a large report can take minutes, and the system will notify you.

AI features often break these expectations, because they put a slow, variable operation behind an interaction that looks instant.

A chat box looks instant. A button labeled "Summarize" looks instant. An autocomplete field looks instant.

When the interaction promises speed and the system cannot deliver it, the feature feels broken, even if the result is good.

The first design question is therefore not "How fast is the model?"

It is "What kind of interaction is this?"

Choose the Interaction Pattern First

Most AI features fit into a small number of interaction patterns. Each one has a different relationship with latency:

Pattern When it fits What the user experiences Watch out for
Inline Small, frequent tasks: classification, short suggestions, field extraction An answer that feels like normal software Needs fast, predictable responses; no room for long reasoning
Streaming Text the user will read: answers, drafts, explanations Progress immediately, content arriving gradually Doesn't help when the result must be complete before it is useful
Asynchronous Longer analysis, multi-step agents, document processing Submit now, get notified when it is done Needs status, notifications, and a place to find the result
Precomputed Predictable needs: daily summaries, visit preparation, risk scores The answer is already there when the user arrives Results can be stale; costs compute for things nobody opens

The visit-preparation example fails because it was designed as an inline feature, but the work behind it is asynchronous by nature. It reads several systems, combines information, and generates a long summary.

The better design is not necessarily a faster model.

It is preparing the summary before the visit: when the visit is planned, when the route for the day is created, or when the representative opens the customer list in the morning.

The fastest response is the one that is already ready.

Streaming Is Not a Universal Fix

Streaming is often the first answer to slow AI features. Show the tokens as they are generated, and the user sees progress immediately.

For text that a person reads, streaming works very well. The time to the first visible word drops, and reading starts while generation continues.

But streaming has limits.

It doesn't help when the output is not meant to be read word by word:

  • Structured output. A JSON object is not useful until it is complete and validated.
  • Actions. You cannot create half an order or update half a customer record.
  • Decisions. A classification or routing decision is one value, not a paragraph.
  • Multi-step workflows. If the model must call three tools before answering, there is nothing meaningful to stream for most of the wait.

There is also a subtle risk. Streaming shows the answer before the system has finished checking it.

If the application validates the output, applies business rules, or checks permissions after generation, a streamed answer may display content that the system later rejects. The user has already read it.

That leads to a simple rule:

Stream what the user reads.
Validate before what the system executes.

For agent workflows, a better form of streaming is often progress, not tokens: "Checking order history", "Reading open support tickets", "Comparing with last quarter". It tells the user what is happening, and it is honest about why it takes time.

Build a Latency Budget

When a feature feels slow, teams often look at the model call first. In practice, the model is usually one stage among many:

User action
  → Authentication and permission checks
  → Retrieval (search, database queries, external APIs)
  → Context building
  → Model call(s)
  → Tool calls and additional model calls
  → Validation and business rules
  → Response rendering

Each stage adds time. Some add a lot more than expected.

A practical exercise is to define a latency budget for the feature, based on the interaction pattern, and then measure each stage against it.

For an inline suggestion, the whole budget might be one second. That immediately rules out some designs: three sequential model calls, a slow ERP query, or retrieval over a large index without caching.

For an asynchronous analysis, the budget might be two minutes. Then the question is no longer speed, but reliability and progress reporting.

A budget changes the conversation. Instead of "the AI is slow", the team can say "retrieval takes 1.8 seconds of our 2-second budget", which is a problem that can actually be solved.

Common improvements are often outside the model:

  • Run independent steps in parallel. Fetching orders, invoices, and tickets one after another is a design choice, not a requirement.
  • Cache stable context. Customer master data and product information rarely change between two requests.
  • Reduce context size. Fewer, more relevant documents are faster to process, and usually produce better answers.
  • Avoid unnecessary model calls. A deterministic check doesn't need a model.
  • Use a smaller model where quality allows. Routing simple tasks to faster models is often more effective than optimizing one large call.

The Average Lies. Watch the Tail.

AI latency is not only slow. It is variable.

The same request can take two seconds one time and twelve seconds the next. Output length varies. Providers have busy periods. Agents sometimes decide to make one more tool call. Retries add hidden time.

This is why the average response time is misleading.

If the average is three seconds but one request in twenty takes fifteen, a frequent user will experience the slow case several times a day. That is the experience they will remember and describe to their colleagues.

Teams should measure percentiles, not just averages: the median, the 95th percentile, and the slowest responses. And they should measure them per stage, so they know whether the tail comes from the model, retrieval, or a slow integration.

Variability also requires explicit decisions:

  • Timeouts. How long is the feature allowed to take before the system stops waiting?
  • Fallbacks. What does the user see when the timeout is reached: a simpler answer, cached data, or an honest message?
  • Limits for agents. A maximum number of steps and a maximum total duration, not just a maximum number of tokens.
  • Cancellation. Can the user stop a long operation, and does the system actually stop the work and its cost?

A feature that is usually fast but occasionally hangs for thirty seconds is worse than a feature that is consistently a little slower.

Predictability is part of performance.

The strongest argument against all of this

There is a fair objection.

Models are getting faster every year. Inference costs are falling, hardware is improving, and smaller models are becoming more capable. Maybe today's latency problems will simply disappear.

Some of them will.

But faster models usually lead to more ambitious features: more context, more tool calls, more steps, more reasoning. Teams tend to spend new speed on new capability. And the parts outside the model, such as ERP queries, permission checks, network calls, and validation, don't get faster because the model did.

The interaction pattern decision doesn't go away either. A long analysis will still be better as an asynchronous job, and a visit summary will still be more useful if it is ready before the visit.

Faster models make good designs better. They rarely fix the wrong design.

A Practical Checklist

Before shipping an AI feature, it is worth asking:

  • When does the user actually need this result, and what are they doing while they wait?
  • Is this an inline, streaming, asynchronous, or precomputed interaction?
  • Does the interface promise a speed the system can deliver?
  • What is the latency budget for the whole feature, and for each stage?
  • Which steps could run in parallel, be cached, or be removed?
  • Are we streaming only what the user reads, and validating before anything is executed?
  • Do we show meaningful progress for multi-step work?
  • Do we measure median and 95th percentile latency, per stage?
  • What happens at the timeout: fallback, cached result, or a clear message?
  • Can the user cancel a long operation, and does the work really stop?

If most of these are unanswered, the feature will probably feel slow, regardless of which model it uses.

Conclusion

In traditional software, performance was often something engineers optimized after the design was done.

In AI products, latency is part of the design itself.

The model determines how long a single generation takes. The product determines whether the user waits for it, watches it, gets notified about it, or never waits at all.

A brilliant answer that arrives after the moment has passed has no value. A good answer that is ready when the user needs it can change how people work.

The question is not only "How fast is our AI?"

It is "Does the answer arrive when it matters?"

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.