Benchmarking 6 Open LLMs on a 2-Machine RPC Cluster (Consumer GPUs, ~96GB VRAM)

In the endless pursuit of "bigger, bigger, and even bigger," I finally reached this point: two machines directly connected via a local area network, using RPC scheduling to split and load the model layer by layer across both hosts.

The reason is simple—I have a limited budget. Buying a professional-grade graphics card with 96GB of VRAM would definitely not pass my wife's approval. So for consumer-level GPU users on a budget like me, linking multiple machines and GPUs via RPC is currently the only realistic option.

The setup itself is straightforward: connect two machines with an Ethernet cable. I used a nominally 5Gbps CAT6A cable, and the actual tested bandwidth is about 4.7Gbps—yielding a throughput of around 500 MB/s. Under this bandwidth, llama.cpp's RPC chain invocation is currently the only viable choice (a Thunderbolt 5 adapter is on the way; once it arrives, it should close some of the performance gap).

The ambient temperature during testing was 35°C; the CPU reached 76°C, and the RTX 5090 (power-limited to 450W) peaked at 68°C.

No special optimizations were made at the architecture level—I just got the official RPC server running end-to-end first, leaving optimizations for later.

Hardware Setup

This totals approximately 96GB of VRAM, with the two hosts directly connected via a CAT6A network cable.

Pitfall Record: tensor-split Order

Once the two machines were connected, it was time to pick the models. With a 96GB budget, there isn't actually much room for choice—you have to consider production-level concurrency requirements (according to current market expectations, a 96GB setup is considered a poor investment if it can't support a team of 8-15 people), while also balancing KV context size and model precision.

When running the first model (Qwen3.6-35B-A3B Q8_0), I noticed the main card (5090) was using less than 7GB of VRAM—a massive waste. At the time, --tensor-split was set to 32,16,16,16,16, intending to allocate the largest share to the main card. But the actual device order was:
[ local - RPC : 5080 , local - RPC : 5060Ti , local - RPC : 5060Ti , 5090 , 5060Ti ]

In other words, the remote RPC cards were placed at the front, while the local main card was at the end—the exact opposite of what I expected. This is why the VRAM allocation didn't land on the 5090 as intended.

Candidate Models

The comparison between Qwen3.6-35B-A3B and Llama-3.3-70B perfectly illustrates the architectural differences. Llama is a Dense model—every token activates all 80 layers of parameters, so the overhead introduced by splitting across 5 GPUs plus one cross-machine network hop is magnified by about 9 times. In practice, its speed is only about 10 tok/s—barely enough for one person doing translation work (close to reading speed), but with noticeable latency. On the other hand, Qwen3.6 is an MoE model, activating only 3B parameters per token; the computation involved in running it on the same 5 GPUs is significantly lower.

Methodology

With thinking mode disabled, the models were evaluated across 5 dimensions:

The judge was primarily Claude 3.5 Sonnet (Opus 4.8 equivalent internally), supplemented by partial manual review.

A few notes:

• I could only run models my hardware could handle; models under 48GB were tested in previous articles.
• This is a personal test, and the test case design inherently carries some subjectivity.
• All tests were single runs, with no voting mechanisms or self-reflection—accuracy relied entirely on the model's own restraint. If popular multi-path reasoning verification were applied, accuracy would likely improve, but this requires more compute and time, so it wasn't done here.
• The sample sizes in dimensions 3, 4, and 5 are very small (6/24/9 questions respectively). Any disputed judgment is enough to change the rankings. Please view the "Overall Comparison" conclusions below as a directional signal rather than a statistically significant difference.

VRAM Usage per Model (Initial Test Run)

Note: In-host GPU ordering and VRAM allocation ratios are heavily influenced by --tensor-split and loading sequence; host/card allocation is not fixed across models—it reflects the actual RPC scheduling results in a specific run.

Overall Comparison Summary

(Note: "Max Usable Context" reflects the measured upper limit under the specific resource allocation, not the model's inherent capability limit.)

Core Conclusion: In a CAT6A (5Gbps) network and llama.cpp RPC chain invocation environment, Qwen3.6-35B-A3B Q8_0 is the best-performing model overall in this test—it ranks at the top in speed, concurrency degradation, translation error rate, and hallucination suppression.

Final Thoughts

After finishing the Qwen2.5 run, I looked back at the VRAM usage and realized that the SAM/CLIP small models left over from earlier tests hadn't actually been completely offloaded. This is a good reminder: in an environment running multiple models mixed together, you should always manually verify the VRAM that "should be freed," rather than blindly trusting that a process exit will definitely clean it up.

Another thought worth recording: Qwen3.6-35B-A3B Q8_0 only took up about 42GB of VRAM, which means it could actually run entirely on a single machine. This left me in deep thought—I went through all this trouble to build a dual-node RPC cluster, only to end up essentially running a model that a single machine could handle? I'll continue pondering this question when the Thunderbolt 5 adapters arrive for the next benchmarking post.