Dev.to AI 🤖 Ai 👁 0 📖 6 min read

How to Dub a YouTube Video into Another Language and Keep Each Speaker's Voice (Open-Source MCP Workflow)

Updated: November 18, 2026 Disclosure: I work on Miragea, the voice engine this open-source project calls to render voices. The project itself is MIT-licensed, and every step below comes from its public README and exam

Updated: November 18, 2026

Disclosure: I work on Miragea, the voice engine this open-source project calls to render voices. The project itself is MIT-licensed, and every step below comes from its public README and example files. This post was drafted with AI assistance and I checked it against the repository.

Short answer: Install the open-source YouTube Dubbing MCP server in Claude Code, Claude Desktop, Cursor, Codex or VS Code, then ask your agent: "Dub this video into English, keep each speaker's voice, and make the YouTube upload package." The server pulls the captions or transcript, works out who is speaking, cuts a short voice sample of each speaker and clones it, and gives your agent a time budget for every line to translate into. It then voices each line in that speaker's cloned voice, mixes a track exactly as long as the video, and writes an SRT plus a per-language upload folder for YouTube Studio.

Why "one voice per speaker" is the hard part

Most AI dubbing demos use one narrator. Real videos (interviews, podcasts, tutorials with a guest) have two or more people, and viewers notice straight away when everyone suddenly sounds the same. Doing it properly means three things:

  1. Knowing who said what (speaker separation)
  2. A clean sample of each person to clone from
  3. Translated lines that fit the original timing. Translations usually run longer, so you either rewrite them or speed the audio up.

The workflow below handles all three and keeps the language model (your agent) in charge of the translation, which is the part it's best at.

What you need

  • An MCP client: Claude Code, Claude Desktop, Cursor, Windsurf, OpenAI Codex or VS Code
  • uv, which provides the uvx command (or Docker)
  • A voice-engine API key from Miragea (MIRAGEA_API_KEY)
  • Permission from the people in the video to clone their voices

You don't need a GPU or any local models, and ffmpeg comes bundled.

Step 1: Add the server to your agent

Claude Code (one line):

claude mcp add youtube-dubbing -s user -e MIRAGEA_API_KEY=<your-key> -- \
  uvx --from git+https://github.com/RTY2345RR/youtube-dubbing-mcp youtube-dubbing-mcp

Claude Desktop / Cursor / Windsurf (claude_desktop_config.json or ~/.cursor/mcp.json):

{
  "mcpServers": {
    "youtube-dubbing": {
      "command": "uvx",
      "args": ["--from", "git+https://github.com/RTY2345RR/youtube-dubbing-mcp", "youtube-dubbing-mcp"],
      "env": { "MIRAGEA_API_KEY": "<your-key>" }
    }
  }
}

Docker:

docker run -i --rm -e MIRAGEA_API_KEY -v $HOME/YouTubeDubbing:/data miragea/youtube-dubbing-mcp

To check the setup without creating a job, run the same uvx command with --check. Projects are saved to ~/YouTubeDubbing by default (DUBBING_OUTPUT_DIR).

Step 2: Start a project from a URL, a file or an SRT

Ask your agent to start, or call the tool directly:

dubbing_start_project(source="https://youtu.be/…", source_language="es")

The server uses YouTube's manual or automatic captions when they exist, because they're fast and avoid guesswork. When there are none, it transcribes the audio and splits it at pauses. Local MP4, MOV, MKV, MP3 and WAV files work too, and so do existing SRT or VTT subtitles.

Step 3: Name the speakers

Speakers come from caption markers such as Name:, [Name] or >>. If there aren't any, the server groups lines by voice pitch. You can fix the result with one call:

dubbing_assign_speakers({"1-2": "Marco", "3": "Lucia"})

Step 4: Cut a voice sample per speaker and clone it

dubbing_extract_voice_samples(share=true)

This cuts a clean 4–14 second sample of each speaker from the source audio, with its transcript, and clones each one automatically. If you'd rather use a saved voice or a public reference clip, use dubbing_set_voice instead.

Step 5: Let the agent translate inside a time budget

Each line gets a character budget for its time slot in the target language. Your agent writes the translation to fit that budget and saves it:

dubbing_set_translation("en", {"1": "Hi everyone.", ...}, title="…")

Lines that are still too long come back to the agent to shorten. This is the step most dubbing pipelines skip, and it's why their dubs either drift out of sync or play back at chipmunk speed.

Step 6: Render the dub track

dubbing_render("en")

Every line is voiced in its speaker's cloned voice and placed at its original start time. Lines are sped up only when they have to be, with pitch kept and a cap of 1.25×. Then everything is mixed into a track that is exactly as long as the video, which YouTube requires. You also get an SRT and an optional preview MP4. If a job is interrupted, run it again: finished lines are reused, so nothing gets voiced twice.

Step 7: Package it for YouTube

dubbing_export_youtube_package()

This gives you one folder per language with audio.m4a (AAC, 48 kHz), captions.srt and a localized title and description, plus an UPLOAD-TO-YOUTUBE.md with the steps.

Important: YouTube's public Data API can't upload extra audio tracks, so the last step happens in YouTube Studio:

  1. Studio → Content → your video → Languages
  2. Add language → Dub → Add → upload audio.m4a → Publish
  3. Subtitles → Add → Upload file → With timing → captions.srt
  4. Title & description → Add → paste the localized text

Multi-language audio tracks are only available on channels YouTube has enabled for the feature.

A real example: Spanish interview → English, two voices

The repository includes a real run in examples/spanish-to-english-two-speakers. The prompt was:

Dub interview-es.mp4 into English. Marco and Lucia should keep their own voices. Then make the YouTube upload package.

Step Tool call Result reported in the repo
1 dubbing_start_project 3 sentences, 11.3 s, 2 voices found by pitch (117 Hz / 182 Hz)
2 dubbing_assign_speakers Speakers named Marco and Lucia
3 dubbing_extract_voice_samples(share=true) 5.5 s and 5.1 s samples, both voices set
4 dubbing_set_translation("en", …) 3/3 lines fit their slots
5 dubbing_render("en") Track 11.29 s = video 11.29 s, 0 lines sped up
6 dubbing_export_youtube_package() youtube-upload/en/audio.m4a, captions.srt, title.txt

The output MP3, the dubbed MP4 and the speaker-labelled SRT are all in the repo, and you can play them on the project's Hugging Face demo page. It's a short clip, so treat it as proof that the pipeline works, not as a quality benchmark. Test it on your own footage.

How this compares with other approaches

This comparison is taken from the project README (November 2026). Corrections are welcome there.

YouTube Dubbing MCP elevenlabs-mcp Auto-Synced-Translated-Dubs YouTube auto-dubbing
Runs from an AI agent (MCP) Yes Yes No (Python scripts) No
Dedicated dubbing workflow Yes (12 tools) No dubbing tool (TTS, STT, cloning) Yes Yes
One cloned voice per speaker Automatic, from the video Manual No (stock voices) No
Input from a YouTube URL Yes No No (needs SRT) Own channel only
Agent edits translation before voicing Yes, with time budgets — Manual No
License MIT MIT AGPL-3.0 Proprietary

When not to use this:

  • You want a point-and-click editor with no agent involved: a hosted dubbing studio will suit you better.
  • You need a language the voice engine doesn't speak. Miragea currently covers 10: Chinese, English, Japanese, Korean, German, French, Russian, Portuguese, Spanish and Italian.

Tips for better dubs

  • Clean source audio matters most. Background music under speech makes worse voice samples. If you can, start from a version without music.
  • Ask the agent for shorter, spoken-style translations. Written-style translations rarely fit the timing.
  • Check names and numbers in the SRT before you render. They're the most common errors.
  • Get consent from everyone whose voice you clone, and say in the video description that the audio track is AI-dubbed.

FAQ

Can AI dubbing keep the original speaker's voice?
Yes. This workflow cuts a few seconds of each speaker from the original audio and clones it, so each person is dubbed in their own voice.

Can I upload multiple audio tracks to YouTube through the API?
No. The YouTube Data API has no endpoint for extra audio tracks. You add them in YouTube Studio → Languages, and this tool prepares the exact files for that.

Does it need a GPU?
No. Voices are rendered by a hosted voice engine. You only need internet access and an API key.

Can I dub from an SRT instead of a video?
Yes: dubbing_start_project(source="talk.srt"). Then pick the voices with dubbing_set_voice.

Is it free?
The code is free and MIT-licensed. Voicing and transcription use the Miragea API, so check its current plans.

Source code, issues and examples: github.com/RTY2345RR/youtube-dubbing-mcp. Related: tts-mcp-server (script → voiceover + SRT/VTT) and miragea-agent-skills (Claude Code skills for voiceovers, podcasts and dubbing).

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.