How to Build a Voice AI Pipeline from Scratch

Why Build a Voice AI Pipeline?

Voice AI is becoming a staple in customer support, virtual assistants, accessibility tools, and even creative storytelling. By stitching together a few well‑chosen services, you can go from raw microphone input to a fully‑generated, personalized voice output in minutes. In this post, we’ll walk through a practical, developer‑friendly pipeline that uses open‑source ASR (Automatic Speech Recognition) for transcription, and a modern TTS (Text‑to‑Speech) service for high‑quality, voice‑cloned speech. We’ll also show you how to hook everything up with Python, JavaScript, and simple curl calls.

1. Capture the Voice

The first step is always to capture the audio. If you’re building a web app, you can use the Web Audio API; for a desktop or server‑side tool, you can rely on pyaudio or sounddevice. The goal is a clean, mono, 16‑bit PCM file at 16 kHz or 24 kHz.

import sounddevice as sd
import numpy as np
import scipy.io.wavfile as wav

def record(duration=5, fs=16000):
    print(f"Recording {duration}s …")
    audio = sd.rec(int(duration * fs), samplerate=fs, channels=1, dtype='int16')
    sd.wait()
    wav.write('input.wav', fs, audio)
    print("Done. Saved to input.wav")

Run record() and you’ll have a WAV file ready for ASR.

2. Speech‑to‑Text: Turn Sound into Words

Open‑source ASR models such as Whisper or Vosk are great for local inference, but if you want instant results without worrying about GPU, a hosted API can shave hours off your dev cycle. For this tutorial, we’ll use Whisper’s API via openai-whisper for demo purposes.

import openai

openai.api_key = 'YOUR_API_KEY'

def transcribe(file_path):
    with open(file_path, 'rb') as f:
        audio_data = f.read()
    transcript = openai.Audio.transcribe(
        model="whisper-1",
        file=audio_data
    )
    return transcript["text"]

text = transcribe('input.wav')
print("Transcribed text:", text)

You’ll get a clean string that you can feed into your NLP engine or pass straight to a TTS service.

3. Add Intent or NLP (Optional)

If your application needs to understand user intent, you can plug in a lightweight NLP library like spaCy or send the text to a cloud NLP endpoint (Google NLP, Azure Text Analytics). For the sake of brevity, we’ll skip this step and go straight to speech synthesis.

4. Text‑to‑Speech with ElevenLabs

ElevenLabs offers a state‑of‑the‑art TTS engine that supports realistic voice cloning, expressive prosody, and dozens of voices. Their API is simple, well‑documented, and has a generous free tier—perfect for prototyping.

4.1. Get an API Key

Sign up at ElevenLabs and grab your API key from the dashboard.

4.2. Basic TTS Request (Python)

import requests

API_KEY = 'YOUR_ELEVENLABS_KEY'
headers = {
    'Accept': 'application/json',
    'xi-api-key': API_KEY,
    'Content-Type': 'application/json'
}

def tts(text, voice_id='en-US-Standard-B'):
    data = {
        "text": text,
        "voice_settings": {
            "stability": 0.5,
            "similarity_boost": 0.75
        }
    }
    response = requests.post(
        f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}",
        headers=headers,
        json=data
    )
    response.raise_for_status()
    return response.content

audio_bytes = tts(text)
with open('output.mp3', 'wb') as f:
    f.write(audio_bytes)
print("Saved TTS to output.mp3")

That’s it—no machine learning tricks, just a single POST request and an MP3 file.

4.3. Voice Cloning (Python)

ElevenLabs also lets you upload a short sample (≈5 seconds) of a target voice and generate a custom voice model. The endpoint is POST /v1/voices. After you create the voice, you can pass the returned voice_id to the TTS call.

def create_clone(sample_path, name="My Clone"):
    with open(sample_path, 'rb') as f:
        sample = f.read()
    response = requests.post(
        "https://api.elevenlabs.io/v1/voices",
        headers=headers,
        files={'sample_file': sample},
        data={'name': name}
    )
    response.raise_for_status()
    return response.json()['voice_id']

clone_id = create_clone('sample.wav')
print("Clone ID:", clone_id)

Now call tts(text, voice_id=clone_id) to speak in that cloned voice.

5. End‑to‑End Python Demo

Putting it all together:

def main():
    record(duration=5)
    text = transcribe('input.wav')
    print("You said:", text)

    # Optional: create a cloned voice
    # clone_id = create_clone('sample.wav')
    # voice = clone_id

    voice = 'en-US-Standard-B'  # replace with clone_id if you created one
    audio = tts(text, voice)
    with open('final.mp3', 'wb') as f:
        f.write(audio)
    print("Final audio written to final.mp3")

if __name__ == "__main__":
    main()

Run this script, speak into your microphone, and you’ll hear the same words spoken back in a realistic, cloned voice.

6. JavaScript/Node Example

If you’re building a web app, here’s a minimal fetch‑based TTS snippet:

const fetch = require('node-fetch');

const API_KEY = 'YOUR_ELEVENLABS_KEY';
const text = 'Hello, world!';

fetch(`https://api.elevenlabs.io/v1/text-to-speech/en-US-Standard-B`, {
  method: 'POST',
  headers: {
    'Accept': 'application/json',
    'xi-api-key': API_KEY,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    text: text,
    voice_settings: { stability: 0.5, similarity_boost: 0.75 }
  })
})
  .then(res => res.arrayBuffer())
  .then(buffer => {
    const fs = require('fs');
    fs.writeFileSync('output.mp3', Buffer.from(buffer));
    console.log('Saved to output.mp3');
  })
  .catch(console.error);

7. curl One‑liner

curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/en-US-Standard-B" \
  -H "xi-api-key: YOUR_ELEVENLABS_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text":"Hello, this is a test","voice_settings":{"stability":0.5,"similarity_boost":0.75}}' \
  --output output.mp3

8. Tips & Best Practices

9. Common Pitfalls

10. Wrap‑Up

You now have a complete, end‑to‑end voice AI pipeline:

1. Record audio →
2. Transcribe with Whisper →
3. (Optional) Understand intent →
4. Generate speech with ElevenLabs →
5. Play or stream back to the user.

All the heavy lifting is handled by the APIs, so you can focus on the unique logic of your application.

Call to Action

Ready to take your prototype to the next level? Sign up at ElevenLabs, grab your API key, and start building a voice‑powered experience that feels genuinely human. Happy coding!