Is there a limit on the number of active parameters in an MoE model?
Hi. We recently had MoE models as big as 1T and 1.6T total parameters. My expectation on the proportion between total and active parameters so far was around 10 to 1, that we save on smaller, "actually local" models. Ho
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.