Dev.to Security 🔐 Cybersecurity 👁 0

Hot experts, cold RAM: running a big MoE

How expert offload actually works on a box with more system RAM than VRAM: hot experts stay pinned on the card, cold ones sleep in ordinary memory. The mechanics, measured: https://blog.localai.hex7.com/hot-experts-cold

How expert offload actually works on a box with more system RAM than VRAM: hot experts stay pinned on the card, cold ones sleep in ordinary memory. The mechanics, measured:
https://blog.localai.hex7.com/hot-experts-cold-ram-running-a-big-moe-on-a-machine-with-more-memory-than-vram/

https://blog.localai.hex7.com/hot-experts-cold-ram-running-a-big-moe-on-a-machine-with-more-memory-than-vram/

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.