Dynamic KV Cache Quantization and Load-on-demand mmproj/MTP: my llama.cpp wishlist
We all know the struggle of optimizing your VRAM usage: quantized model, quantized kvcache, mmproj off. I'm often frustrated by the tradeoffs I have to make in these areas. On my RTX 5090, I can fit: Qwen3.5-27B @ Q6_K
π
This source provides headlines only. Use the button below to read the complete article on the original site.
π° Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.