When "Just Enable the CUDA Feature" Met Two Free GPUs

If a Rust LLM inference engine already runs on CPU and Apple Metal, and its tensor framework supports CUDA, how hard could it be to run it on NVIDIA GPUs?

I expected a simple recompile. Instead, moving mini-vllm-rs, written in Rust and built on top of Candle, from my M2 Mac mini to Linux and CUDA exposed two separate F16 failures on the GPU, a profiler return value that broke my scope guards, a Mac-shaped assumption in my CLI, and a CPU optimization that had been silently leaking onto the GPU.

My motivation for the move was simple: Kaggle Notebooks give you two free NVIDIA Tesla T4 GPUs. On my Mac mini, heterogeneous serving was limited to one CPU worker and one Metal GPU worker (see my previous post). With a dual-T4 notebook, I could test two GPU workers concurrently, or combine them with a CPU worker into a larger heterogeneous system.

Before I could test three workers together, however, a single CUDA worker had to pass my end-to-end smoke test.

F16 failed twice on CUDA: first loudly, then silently

The smoke test requests F16 activations. On macOS, Candle's quantized Metal matmul does not support F16 activations, so GPU inference fell back to F32 while CPU inference used F16. On CUDA, I initially let F16 run through.

It failed during model loading with unsupported dtype for dequantize F32.

Even in a 4-bit GGUF model, small tensors such as normalization weights are typically left unquantized, so their stored type may be F32, F16, or BF16 depending on the source model and conversion.

When converting weights to match F16 activations, Candle 0.11.0 takes different paths by device: non-CUDA backends dequantize to F32 and cast to F16, whereas CUDA dispatches to a specialized kernel table that handles a subset of quantized formats and rejects the rest, including unquantized F32 weights.

I simply added a check to route F32 weights through the generic dequantize-then-cast path. That got the model past initialization, and Qwen2.5 0.5B started producing the right results. Hooray! That was the first victory.

However, when I tried a larger model, Qwen2.5 7B, it failed the smoke test in a worse way: F16 activations produced corrupted text on CUDA. The prompt asked why leaves change color in autumn, and the output was:

  In!!! the!!!!!! autumn! season!! !,!!!!!!!! leaves!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !change!!!!!! !!their!! !!colors!! !from!! !green!! !!to!!!! !!!yellow! !,! !!orange! !,! !and!!! !red! !. !!This!! !is!!!!!!!!! !!!!!!!!!a!!! !wonder!! !!of!!!!! !!!!nature!! !!!that!! !!happens!!!!! !!!!!!!!!!! !!!!!!!!! !!!! !! !! !! [...]

Beneath the exclamation marks, the model was still emitting relevant words, which pointed to numerical errors rather than a broken tokenizer or weight mapping. When I fell back to F32 activations on CUDA, the output returned to normal immediately, and both Llama 3.1 8B and Llama 3.2 1B worked out of the box as well.

Given more time to evolve this engine, I would dig further into that numerical issue and test more data formats. For now, scoping CUDA GPU backends to fall back F32 with a warning got end-to-end inference working reliably, while F16 is still usable on the CPU backend.

Why my profiler ranges popped as soon as they were pushed

On CPU and Metal, specifying --trace-directory exports Chrome traces for Perfetto. On CUDA, Chrome traces show host-side scopes, but not GPU activity such as kernel execution.

To see both without duplicating instrumentation, I hooked NVTX ranges into the same model, layer, attn, mlp, and qmatmul trace scopes, using an RAII guard to end each range automatically.

In the first capture, NVTX labels such as qmatmul operation=attention-query appeared on the timeline, but each range lasted only up to 600 nanoseconds, and even the longest ones never covered the corresponding CUDA kernel launch call.

My first suspicion was that Candle handed the CUDA API work off to another host thread, so the launches were happening outside the thread-local NVTX range. My coding agent chased the QMatMul call path into Candle, but found no evidence of a thread handoff.

The agent next suggested making the thread sleep briefly after the range starts, to see whether the range would stretch to cover the CUDA dispatch.

Instead of trying that, my intuition told me something was wrong with our RAII guard: the range always closed right before the CUDA dispatch, which looked too deterministic to be a race condition. So I went back to check our code:

impl NvtxRangeGuard {
    fn enter(name: &str) -> Option<Self> {
        (cudarc::nvtx::result::range_push(name) >= 0).then_some(Self)
    }
}

impl Drop for NvtxRangeGuard {
    fn drop(&mut self) {
        cudarc::nvtx::result::range_pop();
    }
}

I asked the agent: would enter() ever return None? Checking the CUDA Profiler documentation turned up this warning:

Some of the NVTX functions are defined to have return values. For example, the nvtxRangeStart() function returns a unique range identifier and nvtxRangePush() function outputs the current stack level. It is recommended not to use the returned values as part of conditional code in the instrumented application. The returned values can differ between various implementations of the NVTX library and, consequently, having added dependencies on the return values might work with one tool, but may fail with another.

The agent had written that check, and I had missed it during review. When we logged the return value under Nsight Systems, it was -1 on every call while still recording the range, matching reports from other developers.

Why did that return value produce a tiny range instead of leaving the range open forever?

1. range_push() recorded the start of the range in Nsight Systems and returned -1.
2. Because the .then_some(...) helper evaluates its argument eagerly, it constructed the NvtxRangeGuard before checking whether range_push(name) >= 0 was true.
3. Because -1 >= 0 was false, .then_some(...) discarded that newly created guard and returned None, running Drop immediately inside enter() and calling range_pop() a few hundred nanoseconds after the push.

The range wasn't ending early because of async GPU work. It was being pushed and popped inside enter() before the caller even received the guard.

As the agent put it in its retrospective:

I treated the push return value as a success flag: nonnegative meant “push succeeded, so create a guard”; negative meant “don’t create one.” I hadn’t verified that assumption against the NVTX implementation we were using. It was a mistake to make range behavior depend on that value.

It is a mistake I could easily have made myself, and a good reminder to be careful with external APIs.

Keeping the guard alive regardless of the return value and assigning a distinct color to each trace scope produced the nested hierarchy below: the blue model span breaks down into orange layer blocks, green attn and mlp phases, and individual qmatmul and attention operations right above the CUDA API launches:

From "mixed" mode to one CPU and two GPUs

Alongside the CUDA fixes, I made a few other changes for the multi-GPU environment:

• Make --inference-device flag repeatable: My old mixed inference mode assumed one CPU plus Metal GPU 0, each managing their own KV cache and scheduler. Replacing it with a repeatable --inference-device flag lets each gpu entry claim the next accelerator index (0, 1, and so on), supporting one CPU plus multiple GPU workers without changing the orchestrator.
• Fix a leaked CPU optimization: While chasing down CUDA dispatch, my coding agent noticed that the contiguous attention path enabled a CPU grouped-query matmul layout by default without checking whether the tensor actually lived on the CPU, so this leaked onto the GPU path.
• Fix model cache: In my Kaggle sessions, the model downloader used Hugging Face Hub’s default cache path (~/.cache/huggingface/hub), which wasn’t retained between sessions. I updated it to honor the environment variable HF_HOME, to choose where the cache lives.

Once those were in place, I ran a smoke test sending 3 requests to a heterogeneous setup of one CPU worker and two GPU workers, with each worker running a copy of Qwen2.5 0.5B and round-robin routing sending one request to each backend:

The CPU backend was definitely much slower than the GPUs, as expected, but all three workers processed their requests concurrently end to end.

What I learned

Moving from a Mac mini to dual T4s tested every boundary I had drawn earlier. The process split and per-backend threads scaled cleanly to a larger heterogeneous system, while the bugs showed where single-device and Metal-specific assumptions had been hiding in plain sight.

There are definitely more optimizations to do. For example, when the paged KV cache is used, my current attention implementation concatenates cache pages before self-attention, which was tolerable on my Mac mini's unified memory but more significantly hurt decode performance on the T4s (smoke-tested with Qwen2.5 7B):

Dedicated GPU memory also opens the door to larger models: on my Mac mini the CPU and GPU share 16 GB of RAM, whereas each T4 GPU has its own 16 GB of VRAM. The orchestration and scheduling algorithms will need to be recalibrated for multi-GPU setups as well.

There is always more work to do, but this marks a steady first step for my inference engine in proving its cross-platform support.