Build a Real-Time Voice Translation App

Introduction

Ever wanted to have a conversation with someone who speaks a different language, but without the lag of a human translator? Real‑time voice translation is the dream of many developers building cross‑border apps, customer support bots, or language learning tools. In this post we’ll walk through a practical, end‑to‑end architecture that turns a spoken word into spoken words in another language with minimal latency. We’ll use open‑source libraries for the heavy lifting (speech‑to‑text, translation) and plug in a high‑quality text‑to‑speech engine—ElevenLabs—to give the output a natural voice.

TL;DR: Record audio → Whisper (STT) → Translate → ElevenLabs TTS → Play back.

All you need is a microphone, a server with a GPU (optional), and the free APIs below.

High‑Level Architecture

Client (Web/Mobile) ──► 1️⃣ Speech Capture
                       │
                       ▼
Server (Python/Node) ──► 2️⃣ Speech‑to‑Text (Whisper)
                       │
                       ▼
                     3️⃣ Translate (Google Cloud/DeepL)
                       │
                       ▼
                     4️⃣ Text‑to‑Speech (ElevenLabs)
                       │
                       ▼
Client (Web/Mobile) ──► 5️⃣ Play Synthesized Audio

• Step 1 – The client records short chunks (1–2 s) and streams them to the server via WebSockets or HTTP.
• Step 2 – Whisper (OpenAI) gives us a clean transcription and language code.
• Step 3 – We translate the text with a cloud provider (or an open‑source model).
• Step 4 – ElevenLabs turns the translated text into a natural‑sounding voice.
• Step 5 – The audio bytes are sent back and played immediately.

The most delicate part is keeping the round‑trip under 1 second. Whisper’s small‑model runs fast on a CPU, and ElevenLabs’ API is latency‑optimized for real‑time use.

1️⃣ Speech Capture on the Client

For a web demo we’ll use the MediaRecorder API. It gives us raw audio in 16 kHz mono PCM, which Whisper accepts out of the box.

const mediaRecorder = new MediaRecorder(stream, { mimeType: 'audio/webm;codecs=opus' });
const chunks = [];

mediaRecorder.ondataavailable = e => {
  if (e.data.size > 0) {
    chunks.push(e.data);
    // Send chunk to the server
    const blob = new Blob(chunks, { type: 'audio/webm' });
    sendChunk(blob);
    chunks.length = 0; // reset
  }
};

function startRecording() {
  navigator.mediaDevices.getUserMedia({ audio: true })
    .then(stream => {
      mediaRecorder.start(2000); // 2s intervals
    });
}

Tip: If you need lower latency, reduce the timeslice to 1 s, but remember the network overhead will grow.

2️⃣ Speech���to‑Text on the Server

We’ll use OpenAI’s Whisper‑tiny for speed. Install the whisper package:

pip install git+https://github.com/openai/whisper.git

A minimal endpoint in FastAPI:

import whisper
from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse

app = FastAPI()
model = whisper.load_model("tiny")  # 0.7 GB

@app.post("/transcribe")
async def transcribe(file: UploadFile = File(...)):
    audio_bytes = await file.read()
    audio_path = f"/tmp/{file.filename}"
    with open(audio_path, "wb") as f:
        f.write(audio_bytes)

    result = model.transcribe(audio_path)
    return JSONResponse({
        "text": result["text"],
        "language": result["language"]
    })

Whisper will return the detected language code, so you can skip an explicit language selection step.

3️⃣ Translation

For production you might want a robust cloud API. Below is a lightweight wrapper around DeepL’s free tier, but you can swap it with Google Translate or your own model.

import requests

DEEPL_AUTH_KEY = "YOUR_DEEPL_KEY"

def translate(text, target_lang="FR"):
    payload = {
        "auth_key": DEEPL_AUTH_KEY,
        "text": text,
        "target_lang": target_lang
    }
    r = requests.post("https://api-free.deepl.com/v2/translate", data=payload)
    r.raise_for_status()
    return r.json()["translations"][0]["text"]

If you’re on the free tier, you’re limited to 500 k characters per month. For higher volumes, the paid tier or an open‑source model like MarianMT works too.

4️⃣ Text‑to‑Speech with ElevenLabs

ElevenLabs offers a remarkably natural voice synthesis with a very simple HTTP API. You only need your API key (you can get one for free through the affiliate link below).

import requests

ELEVENLABS_API_KEY = "YOUR_ELEVENLABS_KEY"
ELEVENLABS_ENDPOINT = "https://api.elevenlabs.io/v1/text-to-speech/"

def synthesize(text, voice_id="EXAVITQu4vr4xnSDxMaL"):
    headers = {
        "xi-api-key": ELEVENLABS_API_KEY,
        "accept": "audio/mpeg",
        "content-type": "application/json"
    }
    payload = {
        "text": text,
        "voice_settings": {"stability": 0.5, "similarity_boost": 0.75}
    }
    r = requests.post(f"{ELEVENLABS_ENDPOINT}{voice_id}/stream", json=payload, headers=headers)
    r.raise_for_status()
    return r.content  # MP3 bytes

Why ElevenLabs?

1. The voice quality is on par with commercial TTS services like Amazon Polly.
2. The API is designed for streaming, so you can pipe the MP3 directly to the client without saving to disk.
3. It’s cheap: $0.02 per minute of audio for the free tier.

You can find a quickstart guide and pricing on the ElevenLabs site: https://try.elevenlabs.io/kr07zfuqn1bp

5️⃣ Sending the Audio Back to the Client

In our FastAPI endpoint we can now chain everything:

@app.post("/translate-speak")
async def translate_and_speak(file: UploadFile = File(...), target_lang: str = "FR"):
    # 1️⃣ Transcribe
    audio_bytes = await file.read()
    audio_path = f"/tmp/{file.filename}"
    with open(audio_path, "wb") as f:
        f.write(audio_bytes)
    result = model.transcribe(audio_path)

    # 2️⃣ Translate
    translated = translate(result["text"], target_lang)

    # 3️⃣ Synthesize
    mp3_bytes = synthesize(translated)

    return Response(content=mp3_bytes, media_type="audio/mpeg")

On the client side, you can play the returned MP3 with an <audio> element or the Web Audio API.

async function speak(translatedAudio) {
  const audio = new Audio();
  audio.src = URL.createObjectURL(new Blob([translatedAudio], { type: "audio/mpeg" }));
  await audio.play();
}

Latency Optimizations

A typical round‑trip with a 2 s audio chunk, Whisper‑tiny, DeepL, and ElevenLabs runs in 0.8–1.2 seconds on a modest VPS. That’s fast enough for “real‑time” conversations.

Full Example: Python + FastAPI + ElevenLabs

Here’s a minimal, ready‑to‑run repo you can fork:

https://github.com/your-username/real-time-voice-translate

# Clone
git clone https://github.com/your-username/real-time-voice-translate.git
cd real-time-voice-translate

# Create env
python -m venv venv
source venv/bin/activate

# Install deps
pip install -r requirements.txt

# Set env vars
export ELEVENLABS_API_KEY=your_key
export DEEPL_AUTH_KEY=your_key

# Run
uvicorn main:app --reload

Open index.html in a browser, hit “Start”, and start talking. The translated speech will play back in seconds.

Wrap‑Up

You’ve now seen how to stitch together:

1. Speech capture (client)
2. STT (Whisper)
3. Translation (DeepL/Google)
4. TTS (ElevenLabs)
5. Playback (client)

All with open‑source tools and a single commercial API that delivers studio‑quality voice synthesis. The architecture is modular—swap Whisper for Vosk, DeepL for your own model, and ElevenLabs for Google TTS if you prefer.

If you’re serious about building a polished real‑time translation app, the quality of the voice matters. ElevenLabs gives you that edge without breaking the bank. Try it out today and hear the difference for yourself.

👉 Try ElevenLabs and get started with a free API key: https://try.elevenlabs.io/kr07zfuqn1bp

Happy coding!