Dev.to AI 🤖 Ai 👁 0 📖 5 min read

Create an Interactive Audio Story Generator

What’s an Interactive Audio Story Generator? Imagine you can hand‑write a short story, then let your code turn it into a full‑blown audiobook with multiple characters, sound effects, and even user‑controlled pacing. Th

What’s an Interactive Audio Story Generator?

Imagine you can hand‑write a short story, then let your code turn it into a full‑blown audiobook with multiple characters, sound effects, and even user‑controlled pacing. That’s the goal of an Interactive Audio Story Generator. It blends natural language processing, text‑to‑speech (TTS), and voice‑cloning to create immersive audio experiences that feel like a real conversation.

Below I’ll walk you through building a lightweight prototype in Python, show how to hook up a voice‑cloning service, and give you a few tricks for making the story interactive. If you’re new to voice AI, you’ll see why ElevenLabs is a great starting point.

1. Why TTS and Voice Cloning Matter

  • TTS turns plain text into speech. Modern models can generate speech that sounds natural, with expressive intonation, and can be fine‑tuned to specific accents or emotions.
  • Voice cloning lets you emulate a real person’s voice (or create a custom synthetic voice). That means each character in your story can have a distinct timbre, making the audio feel like a true narrative.
  • Interactivity can be added by letting the user choose plot twists, adjust speed, or even answer questions that influence the story flow.

Combining these pieces gives you a “choose‑your‑own‑adventure” experience entirely in audio form.

2. Tooling Overview

Component Why it’s useful Example
Python Rapid prototyping, rich libraries requests, pydub
ElevenLabs API State‑of‑the‑art TTS & voice cloning https://try.elevenlabs.io/kr07zfuqn1bp
Flask Simple web interface to trigger stories /play endpoint
Web Audio API (in the browser) Control playback, speed, volume AudioContext

We’ll focus on the Python side and show how to integrate ElevenLabs. If you’re more comfortable with JavaScript or curl, the same concepts apply.

3. Setting Up ElevenLabs

  1. Sign up at the affiliate link: https://try.elevenlabs.io/kr07zfuqn1bp
  2. Grab your API key from the dashboard.
  3. (Optional) Create a custom voice by uploading a few minutes of clean audio. ElevenLabs will generate a voice model you can reference by ID.

4. Project Skeleton

audio-story/
├─ app.py
├─ requirements.txt
├─ story.json
└─ static/
    └─ index.html
  • story.json holds the narrative structure.
  • app.py handles the API calls.
  • static/index.html is a minimal UI.

5. Sample Story JSON

{
  "title": "The Lost Lighthouse",
  "characters": {
    "narrator": "narrator",
    "old_man": "old_man",
    "young_woman": "young_woman"
  },
  "script": [
    {
      "speaker": "narrator",
      "text": "It was a stormy night when the lighthouse keeper, old Man, decided to leave the shore."
    },
    {
      "speaker": "old_man",
      "text": "I must check the lantern. The sea will not forgive neglect."
    },
    {
      "speaker": "young_woman",
      "text": "Hold on, old man! I’ve heard the sea is restless tonight."
    }
  ]
}

Feel free to expand this with branching logic or user prompts.

6. Python: Calling ElevenLabs

First, install the dependencies:

pip install requests Flask pydub

6.1. Basic TTS Function

import requests

ELEVENLABS_API_KEY = "YOUR_API_KEY"
BASE_URL = "https://api.elevenlabs.io/v1"

def synthesize_text(text, voice_id, output_path="output.mp3"):
    headers = {
        "xi-api-key": ELEVENLABS_API_KEY,
        "Content-Type": "application/json"
    }
    payload = {
        "text": text,
        "voice_id": voice_id,
        "model_id": "eleven_monolingual_v1"
    }
    resp = requests.post(f"{BASE_URL}/text-to-speech/{voice_id}", json=payload, headers=headers)
    resp.raise_for_status()

    with open(output_path, "wb") as f:
        f.write(resp.content)
    print(f"Saved to {output_path}")

Replace voice_id with the ID of the narrator or any cloned voice.

6.2. Batch Script Generation

import json
from pathlib import Path

def generate_story_audio(story_path, output_dir="audio"):
    Path(output_dir).mkdir(exist_ok=True)
    with open(story_path) as f:
        story = json.load(f)

    for idx, line in enumerate(story["script"], 1):
        speaker = line["speaker"]
        voice_id = story["characters"][speaker]  # map to ElevenLabs voice ID
        text = line["text"]
        output_file = Path(output_dir) / f"{idx:02d}_{speaker}.mp3"
        synthesize_text(text, voice_id, str(output_file))

Now you can call:

generate_story_audio("story.json")

and you’ll get a folder full of MP3s, each representing a line.

7. Making It Interactive

7.1. Speed Control

ElevenLabs supports speed_scale (0.5–2.0). Add it to the payload:

payload = {
    "text": text,
    "voice_id": voice_id,
    "model_id": "eleven_monolingual_v1",
    "voice_settings": {
        "stability": 0.5,
        "similarity_boost": 0.75,
        "style": "default",
        "style_degree": 0.5,
        "speed_scale": 1.2  # user‑controlled
    }
}

7.2. Branching Logic

Add a choices field in your JSON:

{
  "speaker": "young_woman",
  "text": "Do you want to stay or leave?",
  "choices": [
    {"label": "Stay", "next": 5},
    {"label": "Leave", "next": 7}
  ]
}

Your Flask route can parse the user’s choice and jump to the specified index.

8. Frontend Hook (Optional)

<!-- static/index.html -->
<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>Interactive Audio Story</title>
</head>
<body>
  <h1>Play the Story</h1>
  <button id="play">Play</button>

  <script>
    const playBtn = document.getElementById('play');
    playBtn.onclick = async () => {
      const response = await fetch('/play');
      const audioUrl = await response.text();
      const audio = new Audio(audioUrl);
      audio.play();
    };
  </script>
</body>
</html>

And in app.py:

from flask import Flask, send_file
app = Flask(__name__)

@app.route('/play')
def play():
    # Assume we have concatenated MP3s into one file
    return send_file('full_story.mp3', mimetype='audio/mpeg')

9. Production Tips

Tip Why
Cache audio Re‑generating TTS for each request is slow and costly. Store MP3s in S3 or local disk.
Use async For high‑traffic apps, use aiohttp or FastAPI to handle many concurrent requests.
Secure API keys Never expose your ElevenLabs key in client code. Store it in environment variables or a secrets manager.
Add fallback If the TTS API fails, have a local TTS fallback (e.g., pyttsx3) so your app stays responsive.

10. Wrap‑Up

You’ve now seen how to:

  • Pull in ElevenLabs for high‑quality TTS and voice cloning.
  • Structure a story with multiple characters.
  • Add simple interactivity (speed changes, branching).
  • Build a lightweight web interface to play the audio.

The real fun begins when you start adding sound effects, background music, or even real‑time user input. The possibilities are endless.

Call to Action

If you’re ready to bring your audio stories to life, start by signing up at ElevenLabs through this link:

https://try.elevenlabs.io/kr07zfuqn1bp

They offer a generous free tier and a powerful API that will make your interactive audio generator feel like a polished product. Happy coding—and may your stories be loud, clear, and unforgettable!

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.