Dev.to AI 🤖 Ai 👁 0 📖 4 min read

Parakeet vs. Whisper on an AMD iGPU: a speech-to-text benchmark on Linux

On an integrated Radeon 890M, Parakeet Ultra makes 32-57% fewer errors than Whisper small and transcribes a 10 second phrase in 0.11 s. Setup The laptop has an AMD Ryzen AI 9 HX 370 (12 cores, 24 threads) wit

On an integrated Radeon 890M, Parakeet Ultra makes 32-57% fewer errors than Whisper small and transcribes a 10 second phrase in 0.11 s.

Setup

The laptop has an AMD Ryzen AI 9 HX 370 (12 cores, 24 threads) with an integrated Radeon 890M and runs Fedora 44. GPU runs use Vulkan through the RADV driver.

The audio is the test split of FLEURS, 300 sentences per language: 49 minutes of English, 57 of Russian, 65 of German. All files are 16 kHz mono, 16-bit PCM.

Two engines ran the models: whisper.cpp v1.9.5 (libwhisper and libparakeet from the Dabri 2.4.1 RPM) and transcribe.cpp v0.4.2 built with Vulkan and OpenBLAS. Decoding is greedy. Whisper was told the language, Parakeet detects it.

WER is the percentage of words the model got wrong, lower is better. The range in parentheses is a 95% confidence interval. Text is lowercased and stripped of punctuation before scoring and nothing else is normalized, so "20" and "twenty" count as different words and the numbers are higher than on public leaderboards.

Speed is seconds of audio per second of compute, without model loading. The value in parentheses is the time for a 10 second phrase.

Integrated GPU

Model Engine File English Russian German Speed
Whisper small q5_1 whisper.cpp 190 MB 7.23 (6.3-8.2) 12.20 (10.9-13.5) 10.71 (9.7-11.8) 36x (0.28 s)
Parakeet TDT 0.6B v3 q8_0 whisper.cpp 669 MB 5.44 (4.7-6.2) 6.05 (5.2-6.9) 5.46 (4.7-6.3) 79x (0.13 s)
Parakeet TDT 0.6B v3 q4_k whisper.cpp 416 MB 5.50 (4.8-6.3) 6.08 (5.3-6.9) 5.43 (4.7-6.3) 68x (0.15 s)
Parakeet TDT 0.6B v3 q4_0 whisper.cpp 356 MB 5.86 (5.0-6.7) 6.69 (5.8-7.6) 5.84 (5.1-6.7) 87x (0.11 s)
Parakeet TDT 0.6B v3 Q8_0 transcribe.cpp 740 MB 5.37 (4.6-6.1) not run not run 90x (0.11 s)
Parakeet Ultra Q8_0 transcribe.cpp 740 MB 5.09 (4.4-5.8) 5.16 (4.4-5.9) 4.73 (4.0-5.5) 82x (0.12 s)
Parakeet Ultra Q4_K_M transcribe.cpp 486 MB 4.92 (4.3-5.7) 5.45 (4.7-6.3) 4.64 (3.9-5.4) 87x (0.11 s)
GigaAM v3 e2e-rnnt Q8_0 transcribe.cpp 274 MB Russian only 5.16 (4.3-6.1) Russian only 117x (0.09 s)

Against Whisper small, Parakeet Ultra Q4_K_M makes 32% fewer errors in English, 55% fewer in Russian and 57% fewer in German, and runs 2.4 times faster.

Sentence by sentence

The same 300 sentences, one model against another. The number is the change in WER with its 95% interval. Negative means fewer errors. An interval that crosses zero means no measurable difference.

Comparison English Russian German
Ultra Q8_0 vs v3 q8_0 -0.36 [-0.69, -0.03] -0.89 [-1.31, -0.50] -0.72 [-1.17, -0.30]
Ultra Q4_K_M vs v3 q8_0 -0.52 [-0.86, -0.18] -0.60 [-1.03, -0.17] -0.81 [-1.28, -0.37]
v3 q4_k vs v3 q8_0 +0.06 [-0.21, +0.31] +0.03 [-0.30, +0.36] -0.03 [-0.34, +0.28]
v3 q4_0 vs v3 q8_0 +0.42 [0.00, +0.91] +0.64 [+0.22, +1.04] +0.38 [+0.02, +0.75]

Ultra beats v3 in every language. The q4_k file of v3 is as accurate as q8_0 and 250 MB smaller. The q4_0 file is worse.

On Russian, GigaAM v3 and Parakeet Ultra Q8_0 both score 5.16, with a paired difference of 0.00 [-0.73, +0.71]. The same v3 weights on the two engines differ by -0.07 [-0.22, +0.06] in English, so the engines agree.

Large Whisper models

These are slow on this GPU, so they ran on every third sentence, 100 per language. The intervals are wider.

Model File English Russian German Speed
Whisper large-v3 q5_0 1081 MB 5.37 (4.3-6.5) 4.19 (3.1-5.4) 5.31 (3.9-6.9) 9x (1.1 s)
Whisper large-v3-turbo q5_0 574 MB 5.73 (4.5-7.0) 5.18 (3.7-6.9) 5.88 (4.6-7.1) 17x (0.59 s)
Parakeet Ultra Q4_K_M 486 MB 5.24 (4.1-6.4) 5.29 (3.9-6.7) 5.08 (3.9-6.3) 87x (0.11 s)
Parakeet TDT 0.6B v3 q8_0 669 MB 6.26 (5.0-7.5) 6.11 (4.7-7.4) 6.35 (4.9-7.8) 79x (0.13 s)
Whisper small q5_1 190 MB 8.12 (6.6-9.7) 10.46 (8.5-12.4) 11.61 (9.7-13.7) 35x (0.29 s)

Parakeet Ultra is within the margin of error of large-v3-turbo and five times faster.

CPU only

Speed and memory on 40 English sentences (GigaAM on 40 Russian ones).

Model Engine Threads Speed Peak memory
Whisper small q5_1 whisper.cpp 24 7.9x (1.27 s) 393 MB
Parakeet v3 q8_0 whisper.cpp 4 12.5x (0.80 s) 816 MB
Parakeet v3 q8_0 whisper.cpp 12 25.3x (0.40 s) 816 MB
Parakeet v3 q8_0 whisper.cpp 24 12.9x (0.78 s) 817 MB
Parakeet v3 q4_k whisper.cpp 12 39.3x (0.25 s) 581 MB
Parakeet v3 q4_k whisper.cpp 24 24.9x (0.40 s) 587 MB
Parakeet Ultra Q4_K_M transcribe.cpp 12 42.1x (0.24 s) 1008 MB
GigaAM v3 e2e-rnnt Q8_0 transcribe.cpp 12 44.7x (0.22 s) 389 MB

libparakeet defaults to 4 threads. With 12, one per physical core, it runs twice as fast. With all 24 logical threads the speed drops back to where it started. For q4_k, 6, 8 and 16 threads gave 27.7x, 33.1x and 36.1x.

transcribe.cpp maps the model file into memory and that mapping is counted in its peak, so memory is not comparable between the engines.

Limits

One laptop, three languages. FLEURS is read speech, cleaner than dictation. Parakeet covers 25 European languages, Whisper covers 99.

Phonon-2.5 and Parakeet Redux are not in the tables. Both are compressed versions of Parakeet v3 that run only through their own Python runtimes.

In Dabri

Dabri gets the q4_k file of Parakeet v3 in its catalog, and Parakeet now runs one thread per physical core. transcribe.cpp joins whisper.cpp as a second engine library and brings Parakeet Ultra Q4_K_M as a new model in the catalog. Whisper small stays the bundled default: it is 190 MB against 486 MB and covers 99 languages. Ultra needs a CPU with AVX2.

Thanks for reading,
Asher

Originally published on ashbuk.dev.

📰 Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.