Dev.to WebDev šŸ›  Dev šŸ‘ 0 šŸ“– 6 min read

How I cut my real-time voice AI cost by 62%

I build TalkSmart, an app where English learners practise speaking with an AI voice partner. A session is a real-time voice conversation of about 7 to 15 minutes. The voice runs on the Gemini Live API: one model hears th

I build TalkSmart, an app where English learners practise speaking with an AI voice partner. A session is a real-time voice conversation of about 7 to 15 minutes. The voice runs on the Gemini Live API: one model hears the learner and answers with audio.

Voice is the only part of the app that costs real money per use, so I measured it carefully. It turned out I was measuring the wrong thing.

My cost log said $0.11. The invoice said $0.40.
My first estimate was simple. Live audio is billed per token, and audio tokens map to seconds. When I measured it, that came to about 32 tokens per second of input audio and 25 tokens per second of output audio. With $3 per million input tokens and $12 per million output tokens, that is roughly half a cent per minute of learner speech and about two cents per minute of AI speech.

So my server logged seconds of learner audio Ɨ input rate + seconds of AI audio Ɨ output rate for each session. A 6-minute test session came out at $0.046. A full session would be around $0.11. Great margins, I thought.

Then I compared it with the usage numbers in Google's own console. A 12-minute session with no special handling cost about $0.40, roughly 3.5 times my estimate.

Why: every turn pays for the whole conversation again
A live session is stateful. The model keeps the conversation so far in its context, and each new turn bills that whole context again as input tokens. The first turn pays for one turn of context, the tenth turn pays for ten, and so on.

My log only counted the new audio in each turn. It never counted the conversation being re-read.

I hoped prompt caching would soften this. In my sessions the usage metadata showed cachedTok=0 on every turn. Implicit caching discounts did not apply to these live audio sessions.

The proof: cost tracked the square of duration
If every turn re-pays for all previous turns, the total input cost is roughly 1 + 2 + 3 + … + n, which is about n²/2. Doubling the conversation should roughly quadruple the input cost.

Two real sessions, same setup:

Session length Real cost
8.3 min 13.18 TRY
9.7 min 18.12 TRY
If cost were linear, 9.7 minutes would cost 13.18 Ɨ (9.7 / 8.3) ā‰ˆ 15.4. If it is quadratic, it would cost 13.18 Ɨ (9.7 / 8.3)² ā‰ˆ 18.0. The real number was 18.12.

(I'm in Turkey, so my console shows Turkish lira. At the rate I used then, 18.12 TRY is about $0.39.)

Two points don't make a law, but together with the billing model they were convincing enough. For a 15-minute product, a quadratic cost curve is a serious problem. A heavy user doing 33 sessions a month at $0.40 each would eat almost all of a $14.99 subscription.

What didn't work: context window compression
The Live API has a context window compression setting with a sliding window. I tried it first.

contextWindowCompression: {
slidingWindow: {},
}
With the defaults, nothing changed. The default trigger sits near the model's large context limit, and a 12-minute voice chat never gets there. Same cost as before.

Setting the trigger explicitly does kick in:

contextWindowCompression: {
triggerTokens: '8000',
slidingWindow: { targetTokens: '4000' },
}
That cut the cost a lot in my test. But I didn't keep it. A sliding window drops the oldest turns blindly, without knowing which parts matter, and with a window this small it interfered with turn-taking in my push-to-talk setup. For a speaking tutor, remembering that the learner mentioned their sister's wedding five minutes ago is the product. I wanted to choose what to keep.

What worked: summarise and restart every N turns
Instead of letting the context grow, I cut it on purpose and keep the meaning:

Every N learner turns (I started with N = 6), the server sends the transcript so far to a cheap text model and asks for a short summary: topic, what the learner said, mistakes worth coming back to, where the conversation was going.
The server opens a new live session with the same voice. The summary goes into the system instruction, with a note to continue the conversation, not greet the learner again.
When the new session is ready, the server switches over and closes the old one.
In pseudo-TypeScript:

async function resetContext(session: Session) {
const summary = await summarise(session.transcript); // cheap text model
session.swapping = true; // buffer incoming audio from now on

try {
const next = await openLiveSession({
voice: session.voice,
systemInstruction: buildPrompt({ resumeSummary: summary }),
});
const old = session.live;
session.live = next;
old.close();
for (const op of session.buffered) next.send(op); // replay in order
} catch {
// keep the old session; a failed reset must never end the conversation
} finally {
session.buffered = [];
session.swapping = false;
}
}
The details that mattered:

Buffer the learner's audio during the swap. Opening a new live session takes a moment. If the learner starts talking during the swap, their audio and their push-to-talk start/end events are queued and replayed in order on the new session. Otherwise you lose a turn, and the learner thinks the app didn't hear them.
Fail safe. If the summary or the new connection fails, keep the old session and try again later. A cost optimisation must never end someone's conversation.
Reset at a turn boundary, after the AI has finished speaking, not in the middle of an answer.
Keep the cost log honest. Each new session starts its own counters, so the session total has to add up all of them.
The result
Real numbers from a device test, a 10.4-minute session with three resets (after turns 6, 12 and 18):

No reset (9.7 min) With reset (10.4 min)
Input context per turn, near the end 13,232 tokens 1,307 tokens
Total cost 18.12 TRY (~$0.39) 6.86 TRY (~$0.15)
Cost per minute 1.87 TRY 0.66 TRY
The longer session cost 62% less in total and about 65% less per minute. Cost now grows roughly linearly with session length. The quality held: after each reset the tutor remembered the topic and carried on naturally, with no second greeting and no lost turns.

These are single sessions, and sessions vary naturally by 20–30% in my data (some learners talk more, some answers are longer). The direction and size of the change were clear, but treat the exact percentage as one measurement, not a benchmark.

How small should N be?
It's tempting to reset every two turns. The maths says it's not worth it:

Output audio doesn't change. The AI's spoken answers cost the same however you manage context. In my sessions that was close to half the total after the reset, so it's a floor you can't get under this way.
Input cost inside each block grows with N. Over a session of T turns with a reset every N turns, the re-billed context is about T Ɨ N / 2. It is linear in N, so halving N halves the input part, not the total.
From that estimate: going from N = 6 to N = 5 would save about 7% in total, N = 4 about 15%, N = 3 about 22%. With 20–30% natural variation between sessions, a 7% saving is impossible to see in a handful of tests, so I skipped N = 5 and tried N = 4. It stayed in the expected range and the conversation still felt smooth.

Each reset also adds a summary call and a short reconnect, and every summary loses a bit of detail. For me, 4–6 turns is the sweet spot.

The next lever was the other half of the bill: shorter AI answers. Cutting the tutor's typical answer from about 60 to 40 words reduced cost per minute a bit more, and it was better teaching anyway. In a speaking app, the learner should do most of the talking.

Takeaways
With stateful real-time APIs, cost per turn depends on everything before it. Estimate from tokens actually billed, not from seconds of audio.
Compare your own cost log with the provider's console in the first week, before you set prices.
Check whether caching applies to your mode. Don't assume it does.
Generic context compression saves money but can break what makes the conversation good. A summary written for your use case keeps what matters.
Make the context reset invisible: buffer input during the swap, keep the old session if anything fails.
If you've solved this differently (with OpenAI's Realtime API, or with a different reset strategy), I'd love to hear how in the comments.

I'm building TalkSmart, an AI speaking partner for English learners. Next post: the bugs that made most first conversations on mobile silent, and how I found them.

šŸ“° Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.