Seven speech models on real news broadcasts: Leaderboard #1 ties a 0.6B model
The Open ASR Leaderboard has a clear #1: a 30B mixture-of-experts model. Below it sit models a tenth its size. So, a simple production question: point all of them at real news broadcasts (studio anchors, field reporters
The Open ASR Leaderboard has a clear #1: a 30B mixture-of-experts model. Below it sit models a tenth its size.
So, a simple production question: point all of them at real news broadcasts (studio anchors, field reporters, press conferences, background room noise). Does the leaderboard order hold?
It doesn't. On an NVIDIA H100, the seven models land within 0.94 WER points of each other, and a Holm-corrected paired test cannot tell any two apart. What does differ is speed and cost.
The Setup
Wikimedia Commons news (VOA, White House, Al Jazeera)
โ ffmpeg once โ 16 kHz mono Opus (byte-identical audio)
โผ
human subtitles โโโบ 15โ30 s windows cut at cue boundaries
โผ
7 models ร same windows ยท 1ร H100 ยท batch 16 ยท no language hint
โผ
corpus WER ยท 95% CI by bootstrapping videos ยท paired test vs best
Bootstrapping videos, not individual windows, matters: twelve windows from one briefing share a room and a microphone. Clustered confidence intervals are wider and honest.
Results Summary
| Model | Released | WER (%) | 95% CI | RTF (Speed) | Cost / Audio Hr |
|---|---|---|---|---|---|
| Parakeet Unified EN 0.6B | 2026-04 | 7.34% | [5.16, 9.97] | 596x | ~$0.005 |
| Qwen3-ASR 1.7B | 2026-06 | 7.58% | [5.53, 9.85] | 89x | ~$0.034 |
| Whisper large-v3-turbo | 2024-10 | 7.64% | [5.10, 10.57] | 115x | ~$0.026 |
| Parakeet TDT 0.6B v3 | 2025-08 | 7.97% | [5.88, 10.41] | 658x | ~$0.005 |
| Nemotron-3.5-Transcribe 30B | 2026-10 | 8.00% | [5.85, 10.46] | 49x | ~$0.061 |
| Granite Speech 5.0 2B | 2026-10 | 8.02% | [5.92, 10.47] | 13x | ~$0.220 |
| freetranscribe parakeet-ft | 2026-10 | 8.28% | [6.14, 10.79] | 1,117x | ~$0.003 |
Statistical verdict: Smallest Holm-corrected $p = 0.654$. At this sample size (31 broadcasts, 2.6 hours), none of the differences between the #1 model and the 0.6B models are statistically significant.
The Dirty Laundry (What Broke)
-
Whisper memorized White House transcripts: On crosstalk, Whisper repeatedly outputted
The Press-. The Press-. The Press-., matching the exact format of the web transcripts it was trained on. It did this across 7 windows; no other model did. - One long window crashed an entire batch: A cue over 30s crashed Whisper's batched call and failed all 16 windows in it (WER spiked to 15.5% instead of 7.6%). When deploying, always retry failed batches item-by-item.
- Reference subtitles vs verbatim transcript bias: Human subtitles edit out stutter words. Verbatim models (Nemotron inserted 628 words, 9% stutter repeats) get billed for faithfulness, while 'cleaner' models matched the human captioner style.
Key Takeaways for Audio Pipelines
- Don't pay for size on English broadcasts. At ~$3/hr for a spot H100, an hour of news audio costs ~$0.003 with the fastest model and ~$0.06 with the slowest, at virtually identical accuracy.
- Select by throughput and serving stack: 0.6B to 1.7B models run comfortably on consumer cards with massive throughput headroom.
Full interactive dot-whisker charts, dataset details, and reproduction methodology:
https://everylocalai.com/benchmark/asr-news-broadcasts-2026-10
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.