A 1T-parameter MoE reached 1,300 tokens per second per GPU on a public European supercomputer
π November 12, 2026 β’ domyn.com Domyn, the Milan company leading the EUROPA consortium, published a technical post describing training trials for a mixture-of-experts model on the JUPITER supercomputer. The same write-up
π November 12, 2026 β’ domyn.com Domyn, the Milan company leading the EUROPA consortium, published a technical post describing training trials for a mixture-of-experts model on the JUPITER supercomputer. The same write-up appeared on the PyTorch blog the same day. According to the post, these are trials and not a completed pretraining run.
EUROPA was selected by the European Commission on June 19, 2026 as the winner of the EuroHPC Frontier AI Grand Challenge, with the consortium formed by Domyn and two Fraunhofer institutes. The Commission required an open model of at least 400 billion parameters, support for all 24 official EU languages and compliance with the EU AI Act. JUPITER, hosted at Forschungszentrum JΓΌlich, is Europe's first exascale supercomputer, and each Booster node holds four NVIDIA GH200 Grace Hopper Superchips.
Domyn's post attributes the gain to three engineering changes. Sharding experts by whole expert for the Muon optimizer let each rank own expert matrices and run Newton-Schulz iterations with no communication. Running Newton-Schulz on the GPU while parameters remained offloaded cut the optimizer step from 41.7 seconds to 2.2 seconds and lifted throughput from 441 to 597 tokens per second per GPU. Offloading activation checkpoints to the Grace CPU lifted throughput from 370 to 508 tokens per second per GPU. Running FSDP all-gather on the compute stream cut allocator retries from 384 to zero.
The project reports throughput rising from 90 tokens per second per GPU at the start of the trials to about 1,300, with a median measurement of 1,293, on 384 GH200 GPUs across 96 nodes per data-parallel replica, and 1,536 GPUs for four replicas. The configuration had 951.9 billion total parameters with 32.8 billion active per token, 384 routed experts with top-8 routing and a pretrain sequence length of 65,536 tokens. At about 1,300 tokens per second per GPU, one copy of the model processes about 43 billion tokens a day, and four copies on 1,536 GPUs total about 150 billion tokens a day.
IMPACT: Domyn's post reports trials, not a completed pretraining run. Throughput figures are medians over steps 4 to 10, and the loss comparison covers only ten training steps. No weights, no release date and no licence are named. The post discloses neither training data nor language share, and Domyn flags that spreading tokens evenly across four groups of 96 experts can hurt training. Independent reporting says Mistral AI released the open weights of Mistral Large 4, about 1 trillion parameters, on October 29, 2026, and that Meta's Muse Spark Open has 816 billion parameters.
Sources:
domyn.com/blog/training-1t-moe-eu-public-infrastructure-pytorch
pytorch.org/blog/training-a-1t-parameter-moe-on-eu-public-infrastructure-with-pytorch-and-torchtitan
digital-strategy.ec.europa.eu/en/news/commission-selects-europa-consortium-winner-frontier-ai-grand-challenge-project-build-european-open
maccus.fr/en/eu-open-ai-model-europa
www.quotidiano.net/economia/ia-la-sfida-europea-dei-supercomputer-testato-un-modello-da-quasi-mille-miliardi-di-parametri-nezeq4vy
Continue the conversation: https://t.me/AIAutomationArchitecture Β· Bluesky
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.