Why are quants on KV cache increase before weight quants?
I'm cases where ram is limited I've seen a preference for increasing kvcache precision instead of the weight precision. I.e. 8bit kvcache but only 4bit weights. But I can't seem to find a solid explanation as to why?
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.