AssemblyAI Cut Its Speech-to-Text GPU Bill by 70% — by Letting AWS Kill Its Machines

Every AI vendor will tell you inference is expensive. AssemblyAI actually did something about it: it runs its speech-to-text fleet on EC2 Spot Instances — the cheap compute AWS can yank with two minutes' notice — and still hits its SLAs. The trick isn't a better model. It's an architecture that treats being evicted as a normal Tuesday.

The 70% number is real — and narrower than it sounds

AssemblyAI reports 70% lower inference compute costs versus On-Demand, measured on its own production fleet while compute demand grew nearly fourfold year over year. That's a real, audited-scale result. But note the wording: inference compute, not total platform cost. Nobody quotes a whole-platform saving here, and anyone who repeats "70% cheaper" without that qualifier is overselling the case study.

The queue is the whole design

The core move is decoupling submission from execution. Transcription requests land in an SQS queue; GPU-backed ECS tasks pull jobs, fetch audio from S3, and write results back. When a Spot Instance gets its two-minute eviction notice, an EventBridge-triggered Lambda handler puts the in-flight job back on the queue and another GPU picks it up — possibly a different instance type, in a different Availability Zone. An interruption stops being an outage and becomes a retry. Jobs that exhaust their retries go to a dead-letter queue for investigation. That's the entire SLA story: the queue absorbs the chaos so the customer never sees it.

Diversification, not loyalty, keeps the fleet alive

AssemblyAI calls instance diversification the single most effective lever against interruptions — and it's the part most teams skip. The fleet spans three GPU families (g4dn with T4s, g5 with A10Gs, g6 with L4s) across multiple Availability Zones, drawing on independent capacity pools, with a price-capacity optimized allocation strategy picking the pools with the most spare capacity at the lowest price. An On-Demand capacity provider runs as an always-on baseline; Spot serves the spikes. Bet on one instance type and you deserve your interruptions.

Data residency survived the cost-cutting

This is the detail that makes the case study worth reading instead of just quoting. AssemblyAI serves healthcare, finance, and legal customers whose audio must never leave specific regions. The architecture is fully replicated per region — its own queues, clusters, and fleets in us-west-2 and eu-west-2 — so a job interrupted in London is retried in London, not quietly shipped somewhere cheaper. Cheap compute that violates your residency rules isn't a saving, it's a compliance incident waiting for a date.

The limits are explicit — read them before you copy the recipe

AWS's own guidance is blunt: Spot fits stateless, fault-tolerant, flexible workloads. Stateful, fault-intolerant, or tightly coupled jobs need not apply, and a real-time streaming workload that can't tolerate a retry delay doesn't fit at all. And here's the trap AWS warns against: don't fail over to On-Demand when Spot gets interrupted, because that can drive interruptions for your other Spot Instances. The queue is the safety net, not On-Demand. Copy the pattern only if your workload is short-lived, independently retriable jobs on GPU compute — transcription, batch inference, embedding generation, evaluation runs. And don't quote the 70% for your LLM inference fleet until you've measured it yourself; this number belongs to a speech-to-text pipeline.

Go deeper

Read the full analysis on Agentic Affairs: How AssemblyAI Runs Cost-Optimized Speech-to-Text Inference on AWS with EC2 Spot Instances

Originally published on Agentic Affairs (https://agentic-affairs.com)