Source-linked AI summary
Motif 3: Technical Report
Junghwan Lim, Joon Son Chung, Sungmin Lee, Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, Dongpin Oh, Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, Bokki Ryu
TL;DR
Motif 3 addresses the challenge of building more expressive and efficient language-model architectures that translate large-scale training into broadly capable systems. It combines fine-grained sparse MoE scaling, GDLA, stabilization techniques, and targeted capability transfer, achieving strong performance across agentic tool use, reasoning, coding, long-context understanding, and instruction following.
Problem
Further progress requires more expressive and efficient architectures that convert large-scale training into strong, broadly generalizable capabilities.
Method
Motif 3 combines a 314-billion-parameter sparse MoE architecture with GDLA, stabilization components, long-context systems, and multi-teacher post-training.
Results
Motif 3 achieves strong performance across agentic tool use, terminal problem solving, reasoning, coding, long-context understanding, and instruction following.
Takeaways & Limitations
The post-training pipeline consolidates specialized capabilities into a single model without requiring separate models at deployment.
Takeaways & Limitations
Training and evaluation do not cover the full diversity of real-world tasks, domains, languages, interaction patterns, and deployment conditions.
Abstract
from arXiv · showhide
We introduce Motif 3, a decoder-only Mixture-of-Experts language model with 314 billion total parameters and 13.2 billion activated per token. Each sparse MoE layer contains 384 routed experts, with eight selected per token. This fine-grained sparsity provides substantial expert capacity while limiting computation. Motif 3 is built around Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention with the compressed key-value representation of Multi-head Latent Attention. The architecture further incorporates modified manifold-constrained hyper-connections, Expert Specific PolyNorm activations, and multi-token prediction to improve optimization stability, expert specialization, and inference efficiency. We pretrain Motif 3 on approximately 12.5 trillion tokens spanning web documents, STEM, code, mathematics, multilingual content, and domain-specialized corpora. Expert-balancing and numerical-stabilization techniques support stable training at scale, while selective MXFP8 computation and communication, memory-efficient fused kernels, and window-aware context parallelism enable training with context lengths up to 256K tokens. Our post-training pipeline combines general supervised fine-tuning, six specialist teachers trained with reinforcement learning, a software-engineering teacher trained with supervised fine-tuning, and Multi-teacher On-Policy Distillation. The resulting unified model consolidates complementary capabilities in reasoning, coding, tool use, professional work, long-context understanding, calibrated abstention, and instruction following. Across a broad evaluation suite, Motif 3 demonstrates competitive performance against leading open weight models, including strong results on long-horizon agentic tasks, mathematical reasoning, scientific knowledge, and hallucination-sensitive evaluation.
1 Introduction
Motif 3 is introduced as a highly sparse Mixture-of-Experts language model that expands expert capacity while limiting per-token computation. Its development combines large-scale, diverse pretraining with expert-balancing methods and staged post-training focused on capability-specific failures.
- Motivation: Recent open-weight models have narrowed the capability gap with leading proprietary systems, while Mixture-of-Experts architectures scale model capacity without proportionally increasing computation.These developments accompany broader progress in reasoning, coding, tool use, and long-horizon agentic tasks.
- Architecture: Motif 3 uses 314 billion total parameters but activates 13.2 billion per token through 384 routed experts per MoE layer, selecting eight experts per token.This fine-grained sparsity provides a large expert-capacity pool while limiting expert computation for each token.
- Pretraining: Motif 3 is pretrained on approximately 12.5 trillion tokens spanning web, STEM, code, mathematics, synthetic question-answer, and domain-specialized data, with emphasis on multilingual, reasoning, legal, and financial content.A layered expert-balancing strategy is used to train the large expert pool reliably.
- Post-training: The post-training pipeline combines general supervised fine-tuning, specialized teacher training, and Multi-teacher On-Policy Distillation (MOPD) [48] [67].A preliminary SFT model identifies capability-specific failure modes and generates targeted supervision, especially for failure-prone decisions in agentic trajectories.
2 Architecture · 2.1 Overview · 2.2 Grouped Differential Latent Attention
Motif 3 is a 314-billion-parameter decoder-only MoE model that activates approximately 13.2 billion parameters per token through fine-grained expert sparsity. Its GDLA attention combines differential noise suppression, asymmetric signal allocation, and MLA’s compact inference state, with gated outputs and shared compressed KV representations.
- 2.1 Overview: Motif 3 contains 314 billion total parameters, activates approximately 13.2 billion per token, and routes each token through eight of 384 experts per sparse layer.This fine-grained sparsity expands expert capacity without a proportional increase in per-token expert computation.
- 2.1 Overview: GDLA combines GDA’s differential attention with MLA’s compressed KV representation and applies a query-dependent output gate, alongside Expert-Specific PolyNorm and multi-token prediction.These components target attention selectivity, inference efficiency, activations, and prediction structure.
- 2.2.2 Grouped Differential Attention: GDA assigns most attention capacity to signal modeling by using more signal than noise heads, while shared KV heads limit the added computation for noise estimation.Noise outputs are repeated across corresponding signal-head groups, and grouped-query attention associates fewer KV heads with multiple query heads.
- 2.2.1 Latent Query and Key-Value Representations: MLA compresses the KV state into a low-rank latent, while GDLA uses one normalization and up-projection, shared content-key and value heads, and a common rotary key across both paths.The two query paths therefore share a compact KV state rather than maintaining separate KV states for inference.
- 2.1 Overview: The architecture pairs each Transformer block’s GDLA token-mixing layer with a sparse MoE channel-mixing layer and mixes parallel residual streams using mHC.Figure 1 identifies these block-level components and the residual-stream mixing arrangement.
- 2.2 Grouped Differential Latent Attention: GDLA reaches loss 3.2 with 9.2% fewer training tokens than MLA and achieves lower attention training loss than both GDA and MLA.The comparison is reported in the attention training-loss evaluation.
- 2.2.2 Grouped Differential Attention: The differential coefficient is input-dependent and sigmoid-constrained to (0, 1), allowing noise suppression without inverting or amplifying the noise path.Each noise-head output is repeated across the corresponding signal-head group before subtraction.
- 2.2.3 Output Gating: Before output projection, a sigmoid gate computed from the normalized query latent modulates each differential output channel, enabling hidden-state-dependent control of individual attention channels.The gated signal-head outputs are concatenated and projected back to the model dimension.
2.3 Modified Manifold-Constrained Hyper-Connections
Motif 3 replaces fixed residual additions with modified manifold-constrained hyper-connections that dynamically reduce, transform, and redistribute information across four parallel residual streams. A time-dependent post-mapping scale preserves early optimization behavior while limiting late-training activation amplification and outliers.
- 2.3 Modified Manifold-Constrained Hyper-Connections: Motif 3 uses modified manifold-constrained hyper-connections to replace residual additions with token-dependent information flow across multiple parallel streams.The mappings preserve convex mixing and identity-like propagation while enabling richer cross-layer information flow than fixed residual additions.
- 2.3 Modified Manifold-Constrained Hyper-Connections: The expansion rate is n = 4, and each block reduces the streams into a sublayer input before redistributing its output while mixing the original streams.The residual mapping is doubly stochastic, with nonnegative entries and row and column sums of one, preserving convex mixing and identity behavior.
- 2.3 Modified Manifold-Constrained Hyper-Connections: All mappings are generated dynamically per token using a merged projection, sigmoid-based pre-mapping, Sinkhorn-Knopp residual normalization, and FP32 computation for numerical stability.Mapped residual streams are converted back to the model activation dtype after projection, normalization, and iteration steps complete.
- 2.3 Modified Manifold-Constrained Hyper-Connections: Annealing the post-mapping scale from two to one preserves early optimization behavior while limiting repeated sublayer amplification and accumulated activation outliers later in training.The post-mapping range transitions from (0, 2) to (0, 1) without hard-clipping forward activations.
2.4 Expert-Specific PolyNorm
Motif 3 replaces the shared SiLU gate with Expert-Specific PolyNorm, allowing each routed expert to learn its own polynomial coefficients and bias. Coefficient and bias parameterizations constrain the activation’s scale while preserving per-token hidden-dimension normalization.
- 2.4 Expert-Specific PolyNorm: Expert-Specific PolyNorm replaces the shared SiLU gate with independently learned polynomial coefficients and bias for each expert.This gives experts different nonlinear responses for their distinct token distributions.
- 2.4 Expert-Specific PolyNorm: Polynomial coefficients are sigmoid-parameterized within (0, 1), while each bias is clipped to [−0.5, 0.5] to limit output growth.Normalization is computed per token over the hidden dimension and is unaffected by expert grouping.
- 2.4 Expert-Specific PolyNorm: The effective-rank measure is bounded by the weight-matrix rank and reaches that rank when all nonzero singular values are equal.For the measured gate-weight matrices, rank(W) = 512.
2.5 Multi-Token Prediction
Motif 3 adopts standard multi-token prediction as an auxiliary pretraining objective, enabling self-speculative decoding without changing the primary autoregressive architecture.
3 Training System Optimization
Motif 3 training is optimized through topology-aware parallelism, selective and full activation recomputation, memory-saving fused kernels, and selective low-precision computation while retaining optimization-critical states in FP32. The system also accounts for grouped-query attention and context-window effects when controlling numerical stability and communication costs.
- Parallel layout: EP = 8 places each expert-parallel group within one eight-GPU node, using NVLink for expert dispatch and combine communication in a topology-aware hierarchical layout.HybridEP implements dispatch and combine within each expert-parallel group.
- Activation memory: At 256K-token SFT sequence length, full activation recomputation is applied to every transformer block to minimize activation memory at the cost of additional computation.At shorter pretraining and SFT lengths up to 128K, operator-level selective recomputation rematerializes most intermediates while retaining a small set of outputs.
- Output projection and loss: A fused linear cross-entropy kernel processes the approximately 220K-entry output projection in chunks, avoiding materialization of the full logits tensor.Materializing BF16 logits would require more than 13 GB per microbatch even at the pretraining sequence length, with greater use at longer contexts.
- Low-precision training: Optimization-critical states remain in FP32, while low-precision formats are applied selectively to memory-, communication-, and computation-intensive training components.The low-precision recipe uses MXFP8, BF16, and FP32 across different parts of training and communication.
- Context parallelism: Window-aware context-parallel communication is an idealized lower bound beyond W = L/P because halo regions extend beyond the immediately preceding shard and the implementation reverts to block-granular rotation.In that regime, the implementation transfers more data than the plotted window-aware curve.
4 Pre-training
Motif 3 is pretrained on 12.5 trillion tokens from broad multilingual, technical, code, mathematical, and reasoning-oriented sources, using dynamic mixture scheduling and specialized tokenizer training. Pretraining stability is supported by routing safeguards, decaying router noise, and FFN magnitude regularization.
- Data: 12.5 trillion tokens combine web, STEM, code, mathematics, multilingual, synthetic QA, reasoning, and domain-specific data to strengthen technical, coding, mathematical, and Korean capabilities.The corpus includes dedicated Korean and multilingual material alongside general web documents and specialized sources.
- Dynamic mixture scheduling: Dynamic mixture scheduling expands training from 19 to 57 datasets and continuously shifts sampling toward STEM, mathematics, code, synthetic QA, and reasoning sources.General-domain data dominates the early phase, with category ratios later interpolated toward target values using linear or cosine schedules.
- Tokenizer: Motif 3 achieves the strongest tokenizer compression on English, Korean, code, and mathematics, while remaining competitive on other language partitions.Table 2 measures compression in bytes per token, where higher values indicate fewer tokens; the 12.5-trillion-token volume corresponds to more than 15 trillion comparison-tokenizer tokens by probe-based estimation.
- Routing stability: Decaying Gaussian router noise exposes more experts early, accelerates balanced expert-load formation, and is removed as specialization develops to prevent disruption.The approach addresses self-reinforcing expert selection, where favored experts gain stronger gradients while underutilized experts become inactive.
- FFN magnitude regularization: FFN magnitude regularization penalizes only excessive activations while preserving forward computation and recovery gradients, eliminating isolated extreme outliers during training.Afterward, activation growth became distributed across more channels rather than concentrated in isolated extremes.
5 Post-training
Motif 3’s post-training pipeline combines general supervised fine-tuning, seven specialized teachers, and Multi-teacher On-Policy Distillation to produce a unified model with broad, competitive capabilities. Its clearest evaluation strengths are agentic execution and terminal-based problem solving, while performance remains competitive across coding, reasoning, knowledge, long-context, and abstention tasks.
- Post-training pipeline: The three-stage pipeline performs general SFT, trains seven specialized teachers, and uses MOPD to transfer their complementary capabilities back into the general SFT student.Six teachers are trained through domain-specific reinforcement learning and one software-engineering teacher through SFT; the final student retains broad capabilities while acquiring targeted expertise.
- Supervised fine-tuning: The SFT corpus combines Nemotron-family datasets [1] [53] with collected and synthetic data spanning instruction following, reasoning, coding, knowledge, and agentic tasks.Tool-use examples preserve interaction context across successive calls, while failure-prone agentic decisions are augmented with corrected actions or continuations.
- Evaluation results: Motif 3 leads the listed models on τ3-Banking (35.3), reaches 74.9 on Terminal-Bench 2.1 and 94.7 on τ2-Bench Telecom, and scores 51.5 on public ITBench-AA.It also scores 38.7 on GDPval-AA v2; the ITBench-AA result is highest among the available results listed in Table 6.
- Evaluation results: Motif 3 remains competitive on coding, mathematics, general knowledge, hallucination-sensitive, and long-context evaluations, including 76.2 on SWE-bench Verified, 83.4 on GPQA Diamond, and 71.6 non-hallucination on AA-Omniscience.Its AA-Omniscience accuracy is 30.1, while it scores 72.3 on AA-LCR and 78.2 on IFBench; SciCode and CritPt remain below the strongest models.
6 Conclusion, Limitations, and Future Directions
Motif 3 combines highly sparse MoE architecture and unified post-training to support strong performance across diverse capabilities. Its limitations include incomplete real-world coverage and text-only operation, motivating more efficient scaling, longer contexts, and multimodal extensions.
- Conclusion: Motif 3 combines 314 billion total parameters with 13.2 billion activated per token, using 384 routed experts and eight selected per token.The architecture also incorporates GDLA, modified manifold-constrained hyper-connections, Expert-Specific PolyNorm, and multi-token prediction.
- Conclusion: The post-training pipeline consolidates specialized capabilities into one deployable model, with strong performance in tool use, terminal problem solving, reasoning, coding, long-context understanding, and instruction following.It combines general supervised fine-tuning, capability-specific teacher training, and Multi-teacher On-Policy Distillation.
- Limitations: Motif 3 may perform variably on underrepresented or unevaluated tasks because training and evaluation do not cover the full diversity of real-world conditions.As primarily a text model, it is also limited on tasks requiring direct visual-input understanding.
- Future Directions: Future work will target lower-cost architectures that scale beyond Motif 3, native context lengths beyond one million tokens, and visual capabilities for image and video inputs.These directions aim to preserve computational efficiency and effective use of distant information while broadening multimodal applicability.
A Tokenizer · A.1 Pre-tokenization Regexes
The tokenizer uses distinct pre-tokenization regexes for subword and superword training stages. Stage 2 extends word branches to span multiple whitespace-delimited words while preserving digit, punctuation, line-break, and whitespace handling.
- A.1 Pre-tokenization Regexes: Both tokenizer training stages use different pre-tokenization patterns expressed with regex syntax and Unicode property classes.The patterns are specified separately for Stage 1 and Stage 2.
- A.1 Pre-tokenization Regexes: Stage 1 uses a conventional subword pre-tokenization pattern that separates word-like units, contractions, short numeric sequences, symbols, line breaks, and whitespace.The regex includes Unicode property classes and numeric runs of one to three digits.
- A.1 Pre-tokenization Regexes: Stage 1 recognizes contractions through optional case-insensitive suffix groups, alongside its word, numeric, symbol, line-break, and whitespace branches.The listed suffixes include 's, 't, 're, 've, 'm, 'll, and 'd.
- A.1 Pre-tokenization Regexes: Stage 2 augments each Stage 1 word branch with a repeated space-plus-letter-run group, allowing one pre-tokenization unit to span multiple words.This modification applies to both word branches of the regex.
- A.1 Pre-tokenization Regexes: Stage 2 retains Stage 1 handling for digit grouping, punctuation, line breaks, and whitespace while changing only the word branches.The superword regex preserves the corresponding non-word branches from Stage 1.
B Communication-Volume Derivation for Figure 4
The appendix derives per-rank communication volumes for Figure 4 and distinguishes volume from latency when comparing Ring Attention with Ulysses. With causal early exit, Ring averages 2/3 of Ulysses’ full-attention volume, although the last rank reverses this ordering.
- Interpretation: Communication volume is not a latency proxy because Ring can overlap rotation with attention computation, whereas Ulysses’ all-to-all cannot, and Ring volume varies by rank.Ulysses’ all-to-all volume is identical across ranks, unlike Ring’s rank-dependent volume under early exit.
- Early-exit accounting: With causal early exit, rank r receives only the r preceding key/value shards, giving Ring’s rank-average volume as (P −1)/2.A uniform rotation would double the average volume, while the busiest rank remains unchanged because the last rank receives every shard under either scheme.
- Per-rank volume ratio: 2/3: averaged over ranks, Ring moves 33% less data than Ulysses, but the last rank’s ratio is 4/3 and reverses the ordering.Figure 4b reports the rank average and extends it to the last rank’s received volume.
C Evaluation details
The evaluation uses task-specific simulators, agent harnesses, generation limits, step or episode caps, and timeouts across τ2-Bench Telecom, Terminal-Bench 2.1, and SWE-bench Verified.
- τ2-Bench Telecom: τ2-Bench Telecom uses Qwen3-235B-A22B-2507 in non-reasoning mode as the user simulator, with each task repeat capped at 100 steps.
- Terminal-Bench 2.1: Terminal-Bench 2.1 uses the Terminus 2 agent harness, allowing up to 250 episodes per run and imposing two-hour or task-specific timeouts.Each episode is one cycle in which the model observes the current state and plans terminal actions.
- SWE-bench Verified: SWE-bench Verified uses mini-SWE-agent with up to 16K generated tokens per step, 250 steps per task, and a four-hour timeout.
Contributions
The contribution statement lists authors alphabetically by surname and distinguishes technical and management leadership, core contributors, and additional contributors.
- Authors are alphabetically sorted by last name.
- Technical and management leadership is credited to Joon Son Chung, Sungmin Lee, and Junghwan Lim.
- Core contributors include Wai Ting Cheung, Gihun Cho, Minsu Ha, Sangho Kang, Beomgyu Kim, Dongseok Kim, Jangwoong Kim, Taehyun Kim, Taewhan Kim, Jeesoo Lee, Jeongdoo Lee, Junhyeok Lee, and Dongpin Oh.
- Additional contributors include Hyeyeon Cho, Dahye Choi, Jaeheui Her, Hanbin Jung, Changjin Kang, Minjae Kim, Youngrok Kim, Hyukjin Kweon, Hongjoo Lee, Yeongjae Park, and Bokki Ryu.