Dev.to AI 🤖 Ai 👁 0 📖 5 min read

Build a Voice-Enabled Chatbot with ElevenLabs and OpenAI

Why Voice‑Enabled Chatbots Matter Voice is the most natural way we interact with technology. From Siri and Google Assistant to Alexa and Cortana, the trend is clear: people want to talk to their devices. For developers

Why Voice‑Enabled Chatbots Matter

Voice is the most natural way we interact with technology. From Siri and Google Assistant to Alexa and Cortana, the trend is clear: people want to talk to their devices. For developers building conversational AI, adding speech capabilities transforms a simple text chatbot into an engaging, hands‑on experience. In this post we’ll walk through how to build a voice‑enabled chatbot that uses OpenAI’s powerful language model for dialogue and ElevenLabs for high‑quality, customizable text‑to‑speech (TTS) output. We’ll cover the whole pipeline: receiving user speech, converting it to text, generating a response, and finally turning that response back into speech.

1. Architecture Overview

[User] → (Microphone) → Speech‑to‑Text (OpenAI Whisper) →
        |                                |
        v                                v
   [OpenAI GPT‑4] → Text Response → TTS (ElevenLabs) → Audio Output
  • Speech‑to‑Text: We’ll use OpenAI’s Whisper API to transcribe the user’s voice into text.
  • Dialogue Engine: GPT‑4 (or GPT‑3.5‑turbo) processes the text and generates a natural response.
  • Text‑to‑Speech: ElevenLabs’ TTS engine produces a lifelike voice that can be customized with different voices and speaking styles.

2. Prerequisites

Item What You Need
Python 3.10+ For the example scripts
pip install openai elevenlabs Python client libraries
An OpenAI API key For Whisper and GPT
An ElevenLabs API key For TTS (sign up at https://try.elevenlabs.io/kr07zfuqn1bp)

Tip: Store your keys in environment variables (OPENAI_API_KEY, ELEVENLABS_API_KEY) to keep them secure.

3. Transcribing Speech with Whisper

First, capture audio from the user. Below is a minimal example that records 5 seconds of audio and sends it to Whisper.

import os
import sounddevice as sd
import numpy as np
import openai

# Configure OpenAI
openai.api_key = os.getenv("OPENAI_API_KEY")

duration = 5  # seconds
sample_rate = 16000  # Whisper’s preferred sample rate

print("Speak now…")
audio = sd.rec(int(duration * sample_rate), samplerate=sample_rate, channels=1, dtype='int16')
sd.wait()
print("Recording finished.")

# Convert to bytes
audio_bytes = audio.tobytes()

# Whisper transcription
response = openai.Audio.transcribe(
    model="whisper-1",
    file=audio_bytes,
    language="en"
)

transcript = response["text"]
print(f"User said: {transcript}")

Pro tip: If you’re building a web app, you can stream the audio directly to Whisper using the streaming endpoint for lower latency.

4. Generating a Response with GPT‑4

Once we have the transcript, we feed it into GPT‑4 and let it generate a natural reply.

def get_gpt_response(user_text: str) -> str:
    prompt = (
        f"You are a helpful chatbot. The user says: \"{user_text}\". "
        "Respond in a friendly, concise manner."
    )
    completion = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    return completion.choices[0].message.content.strip()

response_text = get_gpt_response(transcript)
print(f"Chatbot: {response_text}")

Feel free to tweak the prompt or add system messages for persona, tone, or context.

5. Turning Text into Speech with ElevenLabs

ElevenLabs offers a rich set of voices, including the ability to clone a custom voice from a short audio clip. For this tutorial we’ll use a ready‑made voice.

import elevenlabs
import os
import requests

elevenlabs.api_key = os.getenv("ELEVENLABS_API_KEY")

def synthesize_speech(text: str, voice_id: str = "21m00Tcm4TlvDq8ikWAM") -> bytes:
    """
    voice_id: Default is the “Rachel” voice. Find others in your ElevenLabs dashboard.
    """
    audio = elevenlabs.generate(
        text=text,
        voice=voice_id,
        model="eleven_multilingual_v2"  # choose a high‑quality model
    )
    return audio

audio_bytes = synthesize_speech(response_text)

# Save to file (or play directly)
with open("response.wav", "wb") as f:
    f.write(audio_bytes)

print("TTS output saved to response.wav")

Note: ElevenLabs’ API returns raw MP3 bytes by default. If you prefer WAV, set output_format="wav" in the generate call.

6. Putting It All Together

Here’s a full script that ties everything together. Run it, speak, and you’ll hear the chatbot’s reply.

import os
import sounddevice as sd
import openai
import elevenlabs

openai.api_key = os.getenv("OPENAI_API_KEY")
elevenlabs.api_key = os.getenv("ELEVENLABS_API_KEY")

def record_and_transcribe(duration=5, sr=16000):
    audio = sd.rec(int(duration * sr), samplerate=sr, channels=1, dtype='int16')
    sd.wait()
    audio_bytes = audio.tobytes()
    resp = openai.Audio.transcribe("whisper-1", file=audio_bytes, language="en")
    return resp["text"]

def ask_gpt(user_text):
    prompt = f"You are a friendly chatbot. User says: \"{user_text}\". Reply briefly."
    completion = openai.ChatCompletion.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}]
    )
    return completion.choices[0].message.content.strip()

def speak(text, voice_id="21m00Tcm4TlvDq8ikWAM"):
    audio = elevenlabs.generate(text, voice=voice_id, model="eleven_multilingual_v2")
    with open("output.mp3", "wb") as f:
        f.write(audio)
    # Play it back (simple cross‑platform)
    os.system("start output.mp3" if os.name == "nt" else "afplay output.mp3")

if __name__ == "__main__":
    print("Speak to the chatbot:")
    user_msg = record_and_transcribe()
    print(f"User: {user_msg}")
    bot_reply = ask_gpt(user_msg)
    print(f"Bot: {bot_reply}")
    speak(bot_reply)

Debugging tip: If you’re on Linux, replace the os.system call with mpg123 output.mp3 or ffplay -nodisp -autoexit output.mp3.

7. Advanced Features

Feature How to Implement
Custom Voice Cloning Upload 30‑second audio to ElevenLabs and use the returned voice_id.
Multilingual Support Whisper can detect language; ElevenLabs offers multilingual models.
Streaming Responses Use OpenAI’s streaming API for GPT and ElevenLabs’ streaming endpoint for TTS to minimize latency.
Voice‑to‑Voice Interaction Capture user audio, transcribe, respond, synthesize, and play back in a continuous loop.

8. Performance Tips

  1. Batch Audio: For real‑time apps, send 1‑second chunks to Whisper to get faster feedback.
  2. Cache Voices: Store generated MP3s for common responses to avoid repeated TTS calls.
  3. Rate Limits: Both OpenAI and ElevenLabs have quotas. Monitor usage via dashboards and add exponential back‑off.

9. Security & Privacy

  • Never expose your API keys in client‑side code.
  • Delete temporary audio files after use.
  • If you’re handling sensitive data, consider adding encryption or a local Whisper deployment.

10. Next Steps

  • Build a web interface using Streamlit or FastAPI that streams microphone input.
  • Add contextual memory: keep conversation history to make replies more coherent.
  • Explore emotion‑aware TTS: ElevenLabs offers style control to make the voice sound excited, calm, or empathetic.

Call to Action

You’re now ready to turn any text chatbot into a lifelike voice‑enabled assistant. If you’re serious about high‑quality, customizable TTS, give ElevenLabs a try. Sign up today through our affiliate link: https://try.elevenlabs.io/kr07zfuqn1bp and unlock the full potential of voice AI. Happy coding!

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.