Shipping a real-time speech translation layer into WhatsApp, Zoom and OBS

Every B2B sales conversation across a language barrier has the same shape: one side writes in English, the other reads with a translation tool, and somewhere in the middle about four minutes get lost. Not on the words — on the round trip.

We wanted to remove the round trip. Here is what actually broke, in the order it broke, and what we would do differently.

The shape of the problem

A procurement manager in Mombasa asks a question over WhatsApp voice. The exporter in Shandong does not read English comfortably. So the exporter:

1. plays the clip
2. opens a translation app
3. copies the text in
4. reads the output
5. types an answer in Chinese
6. translates it back
7. pastes it into WhatsApp

Seven steps, roughly four minutes, for one question. Meanwhile the buyer is asking the same question to five other suppliers. Whoever answers first usually wins.

This is not a language proficiency problem. It is a latency problem, and latency problems are engineering problems.

Architecture, briefly

mic (16k PCM) -> WebSocket -> ASR + MT (streaming) -> TTS -> two output sinks

We use a streaming speech-to-speech model over a persistent WebSocket, chunking audio at a small frame size so the model starts emitting before the sentence finishes. Measured end to end, the delay sits a bit above two seconds — which turns out to be the threshold where the conversation stops feeling like a walkie-talkie and starts feeling like an interpreter sitting next to you.

Two seconds is not a number we chose. It is the point where people stop pausing to wait for the translation and start talking normally.

Lesson 1: you need two output paths, not one

This one cost us the most time.

The naive design routes the translated stream to the call. Then the speaker hears their own translated voice come back through the same device, and you get the classic public-address feedback loop — the rising hum when a microphone gets too close to a speaker.

The fix is two separate sinks:

• Sink A -> the remote participant. Translated audio only. They never hear the original.
• Sink B -> the local operator. Original plus translation. They need to hear what they said to know if it landed right.

Mixing these into one stream is the single most common mistake in this class of product, and it is not a tuning problem — it is a topology problem. No amount of gain staging fixes a wrong topology.

Lesson 2: process priority does not buy you latency

When the first prototype felt sluggish, the obvious move was to raise the process priority and pin threads. It changed nothing measurable.

It is worth internalizing early: the delay lives in the network hop and the model's first-token time. Scheduling on the client is a rounding error next to those. If a user reports lag and you reach for SetPriorityClass before you reach for a network trace, you are optimizing the wrong layer.

We keep a small telemetry breakdown — capture, upload, model first token, synthesis — and it is obvious from that table where the time goes. Any one of the four can dominate, and none of them is the CPU scheduler.

Lesson 3: do not write an audio driver

The tempting shortcut is to write your own virtual audio device so the translation layer can sit transparently inside any app. On Windows that means a signed kernel-mode driver, WHQL submission, and an ongoing compatibility surface across every host application. This is months of work with zero product differentiation.

Use an existing virtual cable. It already works, it is already signed, and users who have ever set up a stream already know what it is.

Lesson 4: you probably do not need to build a mixer

Same reasoning. OBS, Streamlabs and the native live tools on each platform already ship mature routing and mixing. Exposing one more device is a day of work. Building your own mixer is a quarter, and it will be worse than the one users already have open.

What we would do differently

• Start with the telemetry breakdown before writing the pipeline. Four numbers in a table would have saved us the priority-tuning detour entirely.
• Treat the two sinks as a first-class architectural constraint, not a settings toggle. Every code path that assumes a single audio output has to be found and fixed.
• Decide the transport story early. Where the model runs relative to the user sets your floor on latency; a client-side or single-hop-region topology will not save a bad routing decision.

Where this landed

We shipped it inside a larger cross-border sales tool — ad creation, lead intake, and the translation layer above. The translation piece is free to try for 15 days, mostly because we want people to test the two-seconds-plus claim on their own hardware and their own network, not on ours.

The engine is not the interesting part. The topology is. Two sinks, one transport, and honest numbers in a telemetry table — that is the whole trick.

If you want to poke at it: cgwd.wang