VLLM gives 5x speed of llama but quants not available (unsloth/gguf). What to do?
Hi - I want to run unsloth dynamic quant on vllm. Why? vllm is giving faster prefill speed - Llama - i get 800-1000 tokens/sec - Vllm - i get 5k-10K tokens/sec Tried using Qwen3.6-35B-A3B FP8 official. Machine is RT
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.