unsloth dynamic quants for vllm?
Hi - I want to run unsloth dynamic quant on vllm. Why? vllm is giving 5X faster prefill speed - Llama - i get 800-1000 tokens/sec - Vllm - i get 5k-10K tokens/sec Tried using Qwen3.6-35B-A3B FP8 official. Machine is
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.