I bolted an 8-arm reasoning MoE onto a frozen 1.4B Mamba backbone on a single RTX 3060. Here’s the mechanistic autopsy of what broke and what worked.
I built Mamba-Titan-1.4B-Reasoning (a 2.54B parameter MoE) entirely on a 12GB VRAM budget. I froze a 1.4B pure-PyTorch Mamba-1 backbone, grafted on 8 trainable expert arms at the mid-network level, and trained it on Deep
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.