Moving a running model's layers between GPUs mid-reply (8 ms, no re-prefill)
I've been working on an inference engine called jitllm. The idea came from my background in operating systems and JIT compilers: generate kernels for whatever hardware is there, and treat layers a bit like processes the
๐
This source provides headlines only. Use the button below to read the complete article on the original site.
๐ฐ Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.