Dev.to WebDev 🛠 Dev 👁 0 📖 1 min read

Where 3.6 seconds go in a real-time AI answer, and how we got it to 2.6

We build an interview copilot: it hears a question on a call and starts writing an answer. The number that matters is the time from the end of the question to the first visible words. In our browser call it was a median

We build an interview copilot: it hears a question on a call and starts writing an answer. The number that matters is the time from the end of the question to the first visible words. In our browser call it was a median of 3.6 s. Here is where the time went, measured on 12 recorded questions with real providers.

The breakdown

  • End of speech to a final transcript line: about 1.6 s. We ask the speech engine to end a line after 0.45 s of silence, but in practice the final line arrived a median 1.56 s after speech stopped.
  • Detecting the question and sending the request: under 0.05 s.
  • Tokens, the call's lease and building the prompt: 0 to 0.3 s.
  • Opening a connection to the model: 0.73 s, because the connection was closed between questions.
  • Model request to first text: about 1.1 s. The stable part of the prompt was already cached (about 3,000 of 3,400 input tokens).
  • Rendering: under 0.05 s. Streaming the first sentence sooner would have saved nothing (6 ms).

What worked

Keeping the API's connection to the model open while a call is live. The model stage went from 1.74 s to 1.15 s. The whole path: median 3.58 s to 2.65 s, 17 of 17 questions answered whole. Cost: one tiny keep-alive request every few seconds per server, only while a call runs.

What did not (yet)

Ending the question sooner from our own silence detection brought the final line to about 0.8 s, but in some runs the transcription then cut questions in two a second after they began. A faster wrong answer is worse than a slower right one, so it stays off until we understand why.

Takeaways

  1. Measure each stage on real audio before optimising: our guess (the model) was only half the story.
  2. Connection set-up is invisible in benchmarks and very visible in live calls.
  3. Speech end-of-utterance is the floor you fight next.

We're building RoundReply (roundreply.com): live transcripts and real-time interview answers from your resume.

📰 Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.