2-bit copy in VRAM + exact residual in system RAM as the verifier: speculative decoding for models that don't fit your GPU
Hi Guys!, Recently I was making a research and development, but I do not have time to go further with this idea (kids, work, etc.). But maybe it will be interesting for someone. I do not think it would be something big,
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.