Voice AI Architecture: Building Scalable TTS Systems
Why TTS Is More Than Just “Read Aloud” Text‑to‑speech (TTS) used to be a novelty feature in mobile apps, but today it’s a core building block for conversational agents, accessibility tools, and even brand‑voice marketi
Why TTS Is More Than Just “Read Aloud”
Text‑to‑speech (TTS) used to be a novelty feature in mobile apps, but today it’s a core building block for conversational agents, accessibility tools, and even brand‑voice marketing. When you start thinking about a production‑grade system, the conversation shifts from “Can I get a decent voice?” to “How do I serve millions of requests per day with low latency, consistent quality, and cost control?”
In this article we’ll walk through a practical architecture for a scalable TTS service, discuss the key components you need to manage, and show how you can plug in a state‑of‑the‑art engine—ElevenLabs—without reinventing the wheel.
Core Components of a Scalable TTS Pipeline
| Layer | Responsibility | Typical Tech Choices |
|---|---|---|
| API Gateway | Auth, rate‑limiting, request routing | AWS API Gateway, Kong, NGINX |
| Request Validation | Sanitize input, enforce size limits | FastAPI, Express, Go net/http |
| Job Queue | Decouple request ingestion from synthesis | RabbitMQ, AWS SQS, Kafka |
| Worker / Synthesizer | Call the TTS engine, post‑process audio | Python workers, Node.js micro‑services |
| Storage | Persist generated audio for caching & replay | S3, GCS, Azure Blob |
| CDN | Fast global delivery of audio files | CloudFront, Cloudflare, Akamai |
| Monitoring & Alerts | Latency, error rates, usage metrics | Prometheus + Grafana, Datadog |
The flow looks like this:
- Client → API Gateway – A request with plain text and optional voice parameters lands.
- Gateway → Validation Service – The payload is checked (max length, allowed characters, etc.).
- Validation → Queue – The job is enqueued for asynchronous processing.
- Worker pulls the job, calls the TTS engine, stores the result, and writes a reference URL back to a database.
- Client polls or receives a webhook with the final audio URL, which is served via a CDN.
This design gives you horizontal scalability (add more workers), fault tolerance (queue can buffer spikes), and cost efficiency (only generate audio when needed).
Choosing the Right TTS Engine
There are three major categories:
| Category | Pros | Cons |
|---|---|---|
| Open‑source (e.g., Mozilla TTS, Coqui) | Full control, no per‑character cost | Requires GPU hardware, maintenance overhead |
| Cloud vendor APIs (AWS Polly, Google TTS) | Pay‑as‑you‑go, built‑in scaling | Limited voice variety, less expressive |
| Specialized SaaS (ElevenLabs, Resemble AI) | High‑quality neural voices, voice cloning, easy SDKs | Subscription cost, vendor lock‑in |
If you need studio‑grade naturalness and the ability to clone a brand voice quickly, ElevenLabs is a solid choice. Their API returns 24‑kHz, 16‑bit PCM WAV files ready for streaming, and the pricing model is friendly for both prototyping and production.
Pro tip: Use ElevenLabs for the “core synthesis” step, but keep the rest of the pipeline (queue, storage, CDN) under your control. That way you can swap engines later without a major rewrite.
Putting It Together with ElevenLabs
Below is a minimal Python worker that pulls a job from an AWS SQS queue, calls the ElevenLabs TTS endpoint, and uploads the result to S3.
import os
import json
import boto3
import requests
from pathlib import Path
# ---- Config ----------------------------------------------------
ELEVEN_API_KEY = os.getenv("ELEVEN_API_KEY")
ELEVEN_VOICE_ID = os.getenv("ELEVEN_VOICE_ID", "EXAVITQu4vr4xnSDxMaL") # default voice
S3_BUCKET = os.getenv("S3_BUCKET")
SQS_QUEUE_URL = os.getenv("SQS_QUEUE_URL")
# --------------------------------------------------------------
sqs = boto3.client("sqs")
s3 = boto3.client("s3")
def synthesize(text: str) -> bytes:
url = f"https://api.elevenlabs.io/v1/text-to-speech/{ELEVEN_VOICE_ID}"
payload = {
"text": text,
"model_id": "eleven_monolingual_v1",
"voice_settings": {"stability": 0.75, "similarity_boost": 0.85},
}
headers = {
"xi-api-key": ELEVEN_API_KEY,
"Content-Type": "application/json",
}
resp = requests.post(url, json=payload, headers=headers)
resp.raise_for_status()
return resp.content # WAV bytes
def upload_to_s3(data: bytes, key: str) -> str:
s3.put_object(Bucket=S3_BUCKET, Key=key, Body=data, ContentType="audio/wav")
return f"https://{S3_BUCKET}.s3.amazonaws.com/{key}"
def process_message(msg_body: dict):
text = msg_body["text"]
job_id = msg_body["job_id"]
wav = synthesize(text)
s3_key = f"tts/{job_id}.wav"
url = upload_to_s3(wav, s3_key)
# Here you could update a DB record or send a webhook
print(f"Job {job_id} ready → {url}")
def main():
while True:
msgs = sqs.receive_message(
QueueUrl=SQS_QUEUE_URL,
MaxNumberOfMessages=5,
WaitTimeSeconds=10,
).get("Messages", [])
for m in msgs:
body = json.loads(m["Body"])
try:
process_message(body)
sqs.delete_message(QueueUrl=SQS_QUEUE_URL, ReceiptHandle=m["ReceiptHandle"])
except Exception as e:
print(f"Failed {body['job_id']}: {e}")
if __name__ == "__main__":
main()
What’s happening?
-
synthesize– Sends a POST request to ElevenLabs (the link is hidden inside the endpoint). -
upload_to_s3– Stores the WAV file in an S3 bucket; the object URL can be served via a CDN. -
process_message– Ties the two steps together and logs the final URL.
You can run multiple instances of this script behind a process manager (systemd, Docker Swarm, Kubernetes) to scale horizontally.
Curl Example for Quick Testing
If you just want to verify your ElevenLabs credentials, a single curl call is enough:
curl -X POST "https://api.elevenlabs.io/v1/text-to-speech/EXAVITQu4vr4xnSDxMaL" \
-H "xi-api-key: $ELEVEN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"text": "Hello, Dev community! This is a test of ElevenLabs TTS.",
"model_id": "eleven_monolingual_v1",
"voice_settings": {"stability":0.75,"similarity_boost":0.85}
}' \
--output hello.wav
The hello.wav file will be saved locally—perfect for debugging audio pipelines before you wire up the queue.
Scaling Strategies You’ll Need
| Situation | Recommended Tactics |
|---|---|
| Burst traffic (e.g., a marketing campaign) | Auto‑scale workers based on SQS queue depth; use spot instances to keep costs low. |
| High‑availability | Deploy workers in multiple AZs/regions; replicate the S3 bucket with cross‑region replication. |
| Cost control | Cache frequently requested phrases (hash of text → S3 URL) to avoid re‑synthesizing identical content. |
| Latency‑critical apps | Warm‑up a small pool of workers that keep an HTTP connection alive to ElevenLabs; use HTTP/2 if supported. |
Because the heavy lifting (the neural synthesis) stays inside ElevenLabs’ managed service, you avoid the GPU provisioning headaches that come with open‑source models. Your scaling budget can focus on queue length, storage, and CDN egress.
Monitoring & Observability
- Latency: Record the time from queue receipt to S3 upload. A Prometheus histogram works well.
- Error Rate: Track HTTP 4xx/5xx responses from the ElevenLabs API. Alert if the rate spikes above 1 %.
- Usage: Log the number of characters per request; this feeds directly into cost forecasting.
- Cache Hit Ratio: If you implement phrase caching, monitor hits vs. misses to see the ROI of the cache layer.
A simple Grafana dashboard can combine these metrics and give you a real‑time view of health and spend.
Wrap‑Up
Building a robust, scalable TTS service is less about reinventing neural networks and more about orchestrating the right pieces—gateway, queue, workers, storage, and CDN. By delegating the synthesis itself to a high‑quality provider like ElevenLabs, you get state‑of‑the‑art voice cloning and expressive speech without the operational overhead of GPU clusters.
The code snippets above should give you a solid starting point. From here you can:
- Add voice selection parameters to the request payload.
- Implement webhook callbacks instead of polling.
- Expand the cache layer to use Redis or DynamoDB for fast lookups.
Ready to give your app a natural‑sounding voice? Try ElevenLabs today and see how quickly you can move from “text” to “talk” in production.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.