Source-linked AI summary
Instella-MoE Technical Report
Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra, Yonatan Dukler, Gowtham Ramesh, Jialian Wu, Ximeng Sun, Wen Xie, Chaojun Hou, Vikram Appia, Zhenyu Gu, Zicheng Liu, Emad Barsoum
TL;DR
The paper tackles limited transparency and reproducibility in high-performing language models by presenting a fully open sparse MoE trained from scratch with a complete development pipeline. Instella-MoE combines architectural and system innovations with staged post-training, achieving strong fully open and open-weight benchmark results while releasing the materials needed for reproduction and extension.
Problem
Proprietary and partially open-weight language models limit scientific understanding, reproducibility, auditing, and equitable access because their data, methods, and evaluation protocols are opaque or incomplete.
Method
Instella-MoE is a fully open 16B-parameter MoE with 2.8B active parameters per token, using Gated MLA, FarSkip-Collective, and a multi-stage training and post-training pipeline.
Results
The base checkpoint averages 76.7 across standard pre-training benchmarks, while the final Think checkpoint averages 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming reported comparable baselines.
Takeaways & Limitations
The complete release shows that transparency and strong performance can coexist in a sparse MoE system and supports reproducible research and community-driven innovation.
Abstract
from arXiv · showhide
In this work, we introduce Instella-MoE, a fully open Mixture-of-Experts (MoE) language model with 16 billion total parameters and 2.8 billion active parameters per token, trained entirely from scratch on AMD Instinct MI300X and MI325X GPUs. Instella-MoE combines a sparsely activated MoE design with architectural and system-level innovations, including Gated Multi-head Latent Attention (Gated MLA) and FarSkip-Collective connectivity, enabling efficient large-scale training and inference. The model is developed through a multi-stage pipeline comprising pre-training, mid-training, long-context extension, supervised fine-tuning with feedback-driven data curation, direct preference optimization, and reinforcement learning with Multi-Teacher On-Policy Distillation. Instella-MoE achieves an average score of 76.7 across standard pre-training benchmarks, outperforming prior fully open models including OLMo-3-7B, SmolLM3-3B, and OLMoE-1B-7B, while remaining competitive with open-weight MoE and dense baselines at comparable active-parameter scales, including Moonlight-16B-A3B and Qwen3.5-4B. After post-training, our final Think checkpoint achieves an average score of 73.2 across instruction-following, reasoning, math, coding, and chat benchmarks, outperforming both fully open and open-weight models with comparable or larger active parameter counts in our evaluation. To support transparent and reproducible research, we release the complete Instella-MoE model flow, including model weights, training configurations, data mixtures, and training code. Together, these contributions establish Instella-MoE a strong, fully open foundation for efficient, high-performing MoE models and reproducible research.
Introduction
Instella-MoE addresses the limited reproducibility of proprietary and partially open-weight language models with a fully open, efficient MoE system and complete training pipeline. It reports strong benchmark performance alongside releases intended to support reproduction, auditing, and extension.
- Model and architecture: Instella-MoE is a fully open 16B-parameter MoE model with 2.8B active parameters per token, trained from scratch on AMD Instinct GPUs.It combines Gated MLA for model expressivity with FarSkip-Collective for training and inference efficiency.
- Training pipeline: The model uses a multi-stage pipeline spanning pre-training, mid-training, long-context extension, supervised fine-tuning, preference optimization, and reinforcement learning.The stages progressively develop language understanding, long-context processing, instruction following, alignment, and reasoning.
- Performance: 76.7 average across standard pre-training benchmarks places the base checkpoint ahead of prior fully open models and competitive with comparable open-weight baselines.The comparison uses active-parameter scales including Moonlight-16B-A3B, Gemma-4-E4B, and Qwen3.5-4B.
- Performance: 73.2 average across instruction-following, reasoning, math, coding, and chat benchmarks makes the final Think checkpoint the strongest overall post-trained result in the comparison.It outperforms OLMo-3-7B-Think, Gemma-4-E4B, and Qwen3.5-4B on the reported comparison.
- Efficiency: FarSkip-Collective increases pre-training throughput by 12.7% by overlapping expert-parallel communication with computation.Expert-parallel serving with SGLang raises time-to-first-token throughput by up to 39.2%.
- Transparency and reproducibility: The release includes stage-by-stage checkpoints, training configurations, data mixtures, training code, and evaluation protocols.This complete model flow is intended to enable reproduction, auditing, and extension of MoE development on AMD platforms.
2 Instella-MoE
Instella-MoE-16B-A3B is a 16B-parameter sparse MoE Transformer with 2.8B active parameters per token, combining Gated MLA with FarSkip-Collective connectivity. Its design targets model quality and compute efficiency through gated attention, expert routing, and overlapped communication.
- Architecture Overview: 16B total parameters and 2.8B active parameters per token define Instella-MoE-16B-A3B as a sparse decoder-only MoE Transformer.The model has 27 layers, including 26 sparsely activated MoE layers.
- Mixture-of-Experts Layers: Each MoE layer routes tokens to the top K=6 of N=64 routed experts alongside two shared experts.The routed experts have intermediate size 1,408, while the shared experts use a fused feed-forward network of intermediate size 2,816.
- Gated Multi-head Latent Attention: Gated MLA inserts an input-conditioned, sigmoid-activated element-wise gate between attention outputs and the output projection.The gate modulates individual value channels within each attention head while retaining MLA’s low-rank key–value factorization.
- Gated Multi-head Latent Attention: 0.47 points: Gated MLA raises the controlled 200B-token ablation average from 49.86 to 50.33 over vanilla MLA.The largest gains occur on MMLU and code generation.
- FarSkip-Collective Connectivity: FarSkip-Collective uses available or partial activations so attention and expert-parallel communication overlap with independent layer computation.The design aims to minimize activation omission while overlapping Dispatch, Combine, and attention communication across the layer.
- FarSkip-Collective Connectivity: 12.7%: FarSkip-Collective increases pre-training throughput by overlapping expert-parallel communication.The architecture also overlaps routed-expert Combine with shared-expert computation and the first part of attention computation.
- FarSkip-Collective Connectivity: The architecture ablation observes comparable performance between Instella-MoE with and without FarSkip-Collective under fixed training settings and seeds.The comparison uses 200B pre-training tokens.
2.2 Training Pipeline Overview
Instella-MoE is trained through successive base-model and post-training stages that expand capability from broad language understanding to long-context processing, instruction following, alignment, and reasoning. Checkpoints from every major stage are released to support reproducible research.
- Base-model training: 7.1T tokens: base-model training begins with pre-training at 4K context length.Mid-training then uses approximately 100B tokens from STEM- and reasoning-focused mixtures combined by model souping.
- Base-model training: The context window is extended in two stages from 4K to 64K, producing the Instella-MoE-16B-A3B-Base model.This stage follows pre-training and mid-training in the base-model pipeline.
- Post-training and alignment: Post-training and alignment comprise supervised fine-tuning, direct preference optimization, and reinforcement learning.The pipeline progressively targets instruction following, alignment, and reasoning after base-model training.
- Reproducibility: Every major training-stage checkpoint is released, including pre-training, mid-training, long-context extension, SFT, DPO, and RL.The release also includes per-stage hyperparameters and data mixtures.
2.3 Training Infrastructure
Instella-MoE uses AMD GPU infrastructure and open-source training stacks to support expert-parallel, mixed-precision training across its multi-stage pipeline.
- All stages through DPO run in Primus with Megatron-LM and ROCm on AMD Instinct MI300X and MI325X GPUs.
- The training configuration uses expert parallelism, tensor parallelism, FlashAttention, bfloat16 precision, and communication–computation overlap.
- Reinforcement learning uses Miles with Megatron-LM learners, SGLang rollout engines, and Ray-based multi-node orchestration.
- Pre-training uses 7.1 trillion tokens at 4,096-token context with a global batch size of 4,096 and a WSD schedule peaking at 4 × 10−4.
- Pre-training and mid-training draw from publicly documented open datasets organized by training stage.
2.5 Mid-training
Mid-training strengthens STEM and reasoning, then long-context extension expands the model from 4K to 64K tokens while Stage 2 restores capabilities that regress during long-document adaptation.
- 2.5 Mid-training: Mid-training creates three variants from the Dolma 3 Dolmino 100B mixture, differing in selected STEM and reasoning subsets.
- Long-context extension: The context window is extended from 4K to 64K tokens through two stages using long-context data and subsequent STEM-focused recovery.
- Long-context extension: Long-context training uses YaRN with RoPE base θ = 8 × 10^6, document-boundary masking, expert parallelism, context parallelism, and activation recomputation.
- Stage 1: Long-document adaptation: Stage 1 uses approximately 194B tokens at 64K context and improves long-input performance, but GSM8K falls from 77.1 to 62.8 and HumanEval+ from 59.9 to 55.0.
- Stage 2: STEM recovery: STEM-balanced recovery restores GSM8K to 81.5 and HumanEval+ to 65.7, while HELMET declines from 43.7 to 41.5 and RULER from 83.9 to 79.4.
2.7 Supervised Fine-Tuning
Supervised fine-tuning initializes from the long-context base model and combines a broad general dataset with targeted mathematics, programming, and science data.
- SFT begins from the long-context base model and uses a phase-1 mixture containing approximately 2.27M general records plus targeted math, Python, and science slices.
- The targeted slices include 300K mathematics samples, approximately 161K Python competitive-programming samples, and approximately 197K science samples.
Structured error analysis
Instella-MoE’s post-training pipeline uses feedback-driven SFT curation, MoE-aware DPO, instruction-following RL, and multi-teacher on-policy distillation to improve capabilities while preserving broader behavior.
- Structured error analysis: Feedback-driven SFT curation diagnoses student failures, aggregates domain analyses into retrieval policies, and selects targeted training examples rather than sampling uniformly.
- Structured error analysis: The curated final SFT phase improves the eleven-benchmark average by 1.5 points, with largest gains on IFEval (+4.8), AIME25 (+3.6), and LCB (+3.2).
- Direct Preference Optimization: DPO disables MoE load-balancing mechanisms because enabled balancing substantially changes expert routing relative to SFT and naive DPO lowers downstream accuracy.
- IF-specialized RL: The RL objective uses GRPO with truncated importance sampling, mean-centered group-relative advantages, zero-gradient filtering, active sampling, and token-level loss normalization.
- Multi-Teacher On-Policy Distillation: MOPD routes IF rollouts to the IF-RL teacher and other rollouts to the frozen DPO teacher using the student’s own on-policy samples.
- Multi-Teacher On-Policy Distillation: MOPD recovers most of the IF expert’s instruction-following gains while maintaining the DPO model’s other capabilities.
3 Results
Instella-MoE achieves strong base and post-training benchmark results while maintaining long-context capability and improving training and inference throughput. Ablations attribute gains to Gated MLA and show that MOPD preserves instruction-following improvements without sacrificing other domains.
- 3.1 Base Model: 76.7 average score makes Instella-MoE-16B-A3B-Base the strongest fully open model evaluated, ahead of SmolLM3-3B (70.5), OLMo-3-7B (70.1), and OLMoE-1B-7B (61.9).The model activates only 2.8B parameters per token and leads evaluated models on WinoGrande with 86.5.
- 3.1 Base Model: Instella-MoE surpasses Moonlight-16B-A3B (76.2) on average and trails only Qwen3.5-4B-Base (79.5), which activates more parameters per token.
- 3.1 Base Model: 41.5 HELMET and 79.4 RULER averages show effective long-context performance at lengths up to 64K tokens.On RULER, the model is competitive with OLMo-3-7B (80.2) and exceeds SmolLM3-3B (78.6).
- 3.2 Post-Trained Model: 73.2 average score makes the final Think checkpoint the strongest fully open post-trained result in the comparison at 2.8B active parameters.SFT reaches 71.6 and DPO raises the average to 72.7 before the final checkpoint.
- 3.3 Ablations: Gated MLA raises the ablation average from 49.86 to 50.33 (+0.47), with clear gains on HumanEval+ (+4.9), MBPP+ (+2.9), and MMLU (+4.3).
- 3.4 Training Efficiency: 12.7% higher pre-training throughput and 39.2% higher TTFT throughput result from overlapping communication with computation under expert parallelism.The inference result uses SGLang serving on AMD hardware.
4 Conclusion
Instella-MoE is presented as a fully open, sparse MoE model that combines parameter efficiency with strong performance across fully open and open-weight comparisons. Its complete release is intended to support reproducible, auditable, and extensible research.
- Instella-MoE-16B-A3B has 16B total parameters and 2.8B active parameters per token, keeping per-token compute close to a dense 3B-class model.The model combines Gated Multi-head Latent Attention and FarSkip-Collective connectivity.
- The release covers pre-training through reinforcement learning, including weights from every major stage, training configurations, data mixtures, evaluation protocols, and training code.
- The authors state that the fully open pipeline supports reproducible research, community-driven innovation, and further advances in efficient open language modeling.
A Overlapped Implementation of Instella-MoE
The overlapped implementation targets expert-parallel communication overhead during training and inference. It coordinates communication and computation while supporting Instella-MoE’s attention and serving configurations.
- FarSkip-Collective passes outdated and partial activations into MoE and attention layers to overlap communication with computation and reduce communication bubbles.The implementation modularly enforces hardware overlap without custom kernels for portability.
- GPU communication-computation overlap keeps computation running while communication proceeds and shares compute units between communication and computation kernels.
- The inference implementation supports tensor-parallel attention and data-parallel attention, aggregates communication, and raises TTFT throughput by 39.2%.These choices reduce total communication volume while supporting Gated-MLA-based attention.
B SFT Data Curation Details
The SFT curation pipeline uses teacher diagnoses to construct weighted retrieval policies, then combines retrieved and random same-domain examples within a fixed 512K budget. Held-out diagnosis and scoring sets separate curation from evaluation to avoid leakage.
- Error analyses and policies: Incorrect student answers are analyzed by a judge model for failure mode, missing skills, and training data that would address the gap.The analyses include representative mathematics and coding cases.
- Error analyses and policies: A reflection model aggregates per-problem analyses into domain policies consisting of weighted retrieval queries whose weights sum to one.The policies define the target distribution for example selection.
- From queries to selected examples: For the 512K budget, targeted-domain retrieval receives approximately half of each domain allocation because α = 0.5, with the remainder filled by random same-domain examples.The non-targeted domain uses entirely random sampling.
- Data splits and budget: Diagnosis and query generation use held-out seed problems, while a separate scoring set is reserved for evaluation and excluded from selection.
- Data splits and budget: The final curated set is the 512K mixture used for the last SFT phase and preserves the base mixture’s sample count per domain.
- Curation prompts: Reproducibility is supported by releasing the judge, error-analysis, and reflection prompts used to perform curation.
C Training Data Details
Instella-MoE’s training data is assembled from publicly available open-source corpora across pre-training, mid-training, long-context extension, SFT, preference optimization, and reinforcement learning. The mixtures combine broad-domain data with targeted mathematics, code, science, reasoning, instruction-following, preference, and teacher-distillation data.
- All training corpora are drawn from publicly available open-source datasets.
- Pre-training: The 7.1T-token pre-training corpus combines web, mathematics, code, curated SFT-style, and other-domain data.Sources include Nemotron, MegaMath, FineMath, RefineCode, and TxT360 subsets.
- Mid-training: Mid-training variants share the Dolma 3 Dolmino 100B base mixture and differ only in selected STEM and reasoning subsets.All remaining subsets are identical across the variants.
- Long-context extension: Long-context Stage 2 uses a curated 37.32B-token blend of mathematics, code, and reasoning data from three source pools.The sources are the Dolma 3 Dolmino 100B mix, the full Dolma 3 Dolmino pool, and Instella-GSM8K-synthetic.
- Post-training: SFT combines general instruction-following data with targeted mathematics, code, and science slices, followed by feedback-driven curation of 512K examples.DPO uses contrastive preference pairs, while RL combines instruction-following and general prompts routed to specialized or frozen teachers.