Building on LFM3-Omni-2B: a silent-video trap, a text-only fast server and what to measure
The usual voice assistant is four models in a trench coat: speech recognition, a vision model, a language model and a speech synthesizer, so four servers to babysit. Liquid AI's LFM3-Omni-2B is one set of weights, about
The usual voice assistant is four models in a trench coat: speech recognition, a vision model, a language model and a speech synthesizer, so four servers to babysit. Liquid AI's LFM3-Omni-2B is one set of weights, about 4.5 GB, that takes text, images, video with sound and recorded speech, and answers in text, speech or both at once.
We deployed it on two small cards on our cloud, an RTX A5000 with 24 GB and an RTX 2000 Ada with 16 GB, and wrote up every measurement in the LFM3-Omni-2B voice and vision test on inferencekey.com. Here is where it bites.
The video trap: forget one flag and the model says the clip is silent
We sent a 9.7 second clip of a toddler reading, plus a typed question. With the soundtrack, the model reported that the child is speaking. With frames only, it claimed there was no audible speech or music in the clip. Confidently wrong.
In the checkpoint's Transformers code (the reference Python implementation the model ships with), video with sound needs two settings. Leave out either one and the audio never reaches the model:
# Video with its soundtrack, Transformers code path: both, or frames only
load_audio_from_video=True
use_audio_in_video=True
On the hand-built SGLang branch, the flag only worked as a top-level use_audio_in_video: true. Inside mm_processor_kwargs it was silently dropped.
Caveat: the run with sound also had a one-line system prompt asking the model to use what it sees and hears, so two things changed. Even so, the model only noticed sound when it got the sound. If your app takes video, write a test that proves the audio arrives.
The serving split: the fast server is about four times quicker and writes only text
SGLang is an open-source inference server: it keeps a model loaded on the GPU and answers requests. The current release, 0.5.23, does not support this model and crashed at start-up on our card, despite the one-line launch in Liquid AI's docs. Support lives in an open pull request, sgl-project/sglang#46858 by tugot17. We built it by hand. With greedy decoding (always picking the most likely next token, so runs are repeatable) it gave the same answers as the Transformers path, at 158 tokens per second against 38 to 40. (A token is a word piece.) That build has no voice.
So anything spoken goes through the model's own code, loaded by the Hugging Face Transformers library with trust_remote_code. For a single caller the slow route was good enough; we did not test concurrent ones. For heavy text traffic, hold off until SGLang ships a release with support.
What to measure: first speech frame, round trip, and two kinds of memory
The GPU finished a 10-second spoken answer in roughly 6 to 7.5 seconds; the caller got it after 8 to 10, because our backend sends one WAV at the end. Inside the server, the first slice of audio appeared within 0.2 to 0.5 seconds. Streaming means sending audio as it is produced, and we neither built nor tested it. Log both numbers anyway: the gap is what streaming would buy you.
Memory splits too. PyTorch, the library holding the weights on the GPU, reported a peak of 4,285 to 4,434 MiB per request. nvidia-smi, NVIDIA's own monitor, put the whole process at 4,772 to 4,832 MiB. PyTorch never counts the extra 400 MiB or so. To find the floor, we limited PyTorch on the A5000 with torch.cuda.set_per_process_memory_fraction:
| Cap | Voice, photo, transcription, speech, 10 s video |
|---|---|
| RTX 2000 Ada 16 GB, real card | all pass |
| 4.45 GiB (simulated) | all pass |
| 4.35 GiB (simulated) | the 10 s video runs out of memory |
Those rungs are a software limit on a large card, not a real small one. A one-minute video wants about 6 GB. Each rung is in the GPU memory ladder for LFM3-Omni-2B.
Numbers read aloud need a check
We asked "how tall is Kilimanjaro" out loud ten times. Nine answers had the meters right, and most got the feet wrong. Speech is generated with sampling, meaning a controlled dose of randomness (audio_temperature 0.4 in our runs), and the written words follow the voice, so two runs rarely match. Give a number-reading assistant a tool or a check for conversions. Tool calling behaved: a spoken "set the volume to 30" became set_volume({"volume": 30}) in about half a second.
Distrust quotes from muddy audio too: the model gave the toddler a sentence that matched neither Whisper's transcript nor its own.
The license is a gate, not a footnote
The weights ship under the LFM Open License v1.0. Your organization may use it commercially only below 10 million US dollars of yearly revenue. Above that, agree terms with Liquid AI before production.
What we did not prove
One request at a time, our own short inputs, no human rating of the voice (Whisper large-v3 via faster-whisper read the audio back), no GGUF builds (the compressed files llama.cpp runs), no llama.cpp realtime server. No four-model pipeline on the same card either, so this cannot say one model beats it.
Credit where it belongs: Liquid AI built the model, its audio codec, the benchmarks and the sample clips; tugot17 wrote the SGLang port; the Transformers maintainers made the code path we ran; the llama.cpp maintainers added GGUF and realtime support; OpenAI's Whisper gave the independent read-back; LibriSpeech supplied the transcription clip; Kyutai's Pocket TTS voiced our test questions; AI Search's video is where we spotted it. Our part: the custom backend, the two-card deployment and the measurements.
The backend, request options, timings and SDK example are in the full LFM3-Omni-2B deployment write-up.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.