Serving Hy-SimulMT live on one 48 GB GPU: the startup spike, the FP8 trap, what to measure

Hy-SimulMT is Tencent Hunyuan's open model for live Chinese and English interpreting, licensed Apache-2.0. One model both hears and translates, so output can begin while the speaker is mid-sentence. SimulInfer, a small service from the authors, cuts incoming audio into 1.6-second blocks and assembles the partial outputs. Behind it, vLLM (the open-source server that loads the weights onto the GPU and answers requests) runs the model through the authors' simulst plugin.

We ran the FP8 Interleave build on one RTX 6000 Ada 48 GB on our cloud. FP8 means the weights are stored as 8-bit numbers, which brings the files down to 32 GB. Interleave means the output also carries a transcript of the original speech. Four read clips went in at speaking speed, and first words came back 2.3 to 5.5 seconds in. The full Hy-SimulMT test on one 48 GB card holds every table and command. Below is what you hit.

Startup needs the whole card, whatever cap you set

Once serving, the translator sat at 38 GB of the 48, hinting that a 32 GB card might do. Startup says otherwise.

Before it accepts requests, vLLM pushes dummy audio through the audio encoder (the part of the model that turns sound into something the language part can read) to see how much room is left. That check filled the card, 48,510 MiB, in every run, before usage dropped back. With the authors' 16,384 batched tokens (the input vLLM may process in one step), it profiled 3,276 dummy audio items and printed thousands of out-of-memory warnings before recovering. Dropping --max-num-batched-tokens to 4,096 did not remove the peak.

Next we capped the server at 32 GB with --gpu-memory-utilization 0.66, a shorter context (how much text the model keeps in view) and a single session. Translation kept its live pace, but startup still hit 47.4 GiB. The cap does not cover that peak, so a real 32 GB card such as an RTX 5090 could fail before serving anything. Size for 48 GB. Adding the voice clone, Qwen3-TTS-0.6B from Alibaba's Qwen team, puts usage at 48 GB instead of 38.

SimulInfer rejects the FP8 build until it finds a manifest

Point SimulInfer at the FP8 repo and the first error is in Chinese: it cannot identify the model's output format.

ValueError: 无法识别所选模型的输出格式...
# raised in model_discovery.py: no export_manifest.json,
# and the repo name is not in PROVIDED_CHECKPOINTS

SimulInfer only knows the full builds by repo name. Our custom backend (the code that installs and starts the services on the GPU) writes the missing export_manifest.json, and then it starts. Upstream, this was still open at commit 7efcb18c.

The voice needs its own pin. vLLM-Omni 0.32.0 and 0.32.0rc1 both stop on an import error for Qwen3TTSSpeakerEncoder. 0.31.0rc1 on vLLM 0.31.0 works, so the voice runs in its own environment beside the translator on vLLM 0.32.0.

Check the language pair first. The model page tags 40 languages, yet SimulInfer throws a ValueError for any direction except zh-en and en-zh.

Log every arrival, and read the complete flag

"Latency" blurs two numbers. Time to first words depends on when the model judges it has heard enough. Block delay is how long the output for each 1.6-second block takes to arrive after that block's audio ends. Median block delay ran between 0.4 and 0.8 seconds across the four clips. At the 90th percentile it passed 1.5 seconds, and the worst block took 2.9. Timestamp each arrival, or the final text hides this.

Quality depends on direction. We compared the output wording with human reference translations using chrF, a 0 to 100 score that counts shared character sequences. Into English, the two Chinese clips got 60.6 and 51.7. Into Chinese, the two English clips got 42.2 and 23.4.

The clip at 23.4 is the warning. Two of its six sentences went missing, and a third reversed its meaning: "couldn't have included" came out as "could include". SimulInfer flagged that run with complete=false, so have your client check that field. Missing output is easy to spot. A reversed negation reads as normal Chinese. Test English to Chinese on your own recordings first. The per-clip timings and chrF scores for Hy-SimulMT also list the block delay percentiles and the whole-file runs.

The authors report about 3 seconds of lag on long talks, using their StreamLAAL measure. Our lag figure is a rough approximation, not comparable. We did not reproduce their benchmarks or their comparison with human interpreters, and our sample was four read FLEURS clips, one session at a time.

Send whole files unless someone is listening live

Sent as one file, a clip of 50 to 56 seconds came back fully translated in about 26 seconds. For live use, stream PCM audio (raw, uncompressed samples) in short packets at real-time pace to SimulInfer's WebSocket.

Our own API has a gap here. It takes a recorded file of up to about a minute, replays it at speaking speed and returns the arrival timeline at the end. A live mic has to talk to SimulInfer on the GPU directly for now. If you only need recorded speech turned into text, see our Nemotron 3.5 Transcribe test.

Tencent Hunyuan built the model, the simulst plugin, SimulInfer, the TTS integration and the llama.cpp support for the 4-bit build. The Qwen team made Qwen3-TTS, served here through vLLM-Omni. Thanks also to the vLLM and llama.cpp projects, the SGLang-Omni team for their cookbook, Google for the FLEURS clips, and AI Search, whose news video pointed us to the model. Our part was the custom backend, the deployment and the measurements. Setup steps, the full memory ladder and all eight problems with their status are in the Hy-SimulMT live speech translation write-up on inferencekey.com.