r/LocalLLaMA 🤖 Ai 👁 0

2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU

Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big,

📄

This source provides headlines only. Use the button below to read the complete article on the original site.

📰 Read the original article on r/LocalLLaMA

Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.