Motif 2 12.7B technical report

• Motif-2-12.7B is an open-weight model designed for instruction following, multi-step reasoning, and general-purpose language understanding.The report frames its work around the sparse research on competitive capabilities under model-size and computational constraints.Source: §1 · pp. 1–2
• Grouped Differential Attention reorganizes attention to capture multi-granular token interactions while preserving scalability.The report says it improves efficiency without additional computational cost.Source: §1 · p. 2
• Pre-training used 5.5 trillion tokens and a linear curriculum that gradually increased STEM, mathematics, and programming data.MuonClip and high-performance parallelism kernels were used to improve throughput and stability during long-sequence training.Source: §1 · p. 2
• Three-stage supervised fine-tuning progressively targets broad conversational competence, compositional reasoning, and linguistic fluency.The stages use diverse instruction data, curated and synthetic reasoning-focused data, then selective data pruning.Source: §1 · p. 2

• Width-preserving hypercloning scales Motif-2.6B through direct weight replication, preserving its parameter topology and initialization statistics.The expanded model inherits the predecessor’s activations exactly, maintaining functional continuity.Source: §2 · pp. 2–3
• Grouped Differential Attention uses a 4:1 ratio of signal-preserving to noise-control groups, selected as the best balance in preliminary experiments.The report describes this as its first large-scale application within the Motif architecture.Source: §2 · p. 3
• LLaMA-Pro depth scaling adds layers while retaining inherited normalization, positional embeddings, residual scaling, PolyNorm, and tokenizer; Table 1 lists detailed hyperparameters.Source: §2 · p. 3, Table 1 · p. 3

3.1 Pre-training Data

• The 5.5-trillion-token corpus spans general English, question answering, chat, multilingual, scientific, mathematical, and programming domains.It aggregates open-source and in-house datasets across general web, multilingual, reasoning, science, mathematics, and code sources.Source: §3.1 · pp. 3–4

3.2 Pre-training Stage and Recipe

• The data mixture shifts from mostly English web corpora toward gradually increased reasoning, mathematical, and code datasets during training.The mixture is dynamically adjusted at each training step to support a transition from language learning to higher-order reasoning.Source: §3.2 · p. 4
• Effective batch size scales from 16 million to 80 million tokens, using an internally optimized Muon-Clip variant for parallel and distributed training.Source: §3.2 · p. 4
• The final annealing phase keeps reasoning data below 10% of total tokens while increasing its share to limit overfitting and narrow reasoning distributions.Mathematics receives more weight than code within that subset; the phase covers roughly 1 trillion tokens and increases maximum sequence length from 4,096 to 16,384.Source: §3.2 · p. 4

3.3 Pre-training Evaluation

• Motif-2-Base surpasses Gemma-3 12B on MMLU and MMLU-Redux, approaches Qwen-3 14B/32B levels, and records the highest MMLU-Pro score.Source: §3.3 · p. 4
• 94.9 on GSM8K and 73.6 on MATH exceed comparable-scale open-weight baselines; Motif-2-Base leads with 65.9 on HumanEval and 81.5 on MBPP.Source: §3.3 · pp. 4–5
• On commonsense benchmarks, scores are slightly below larger counterparts but remain competitive without significant degradation.Source: §3.3 · p. 5
• Across knowledge, reasoning, coding, and commonsense benchmarks, Motif-2-12.7B-Base outperforms Gemma-3 12B and 27B and Qwen models of comparable size (14B), while approaching Qwen-3 32B.The evaluation compares against officially reported Qwen3 and Gemma3 scores and uses greedy decoding for Motif-2-12.7B-Base.Source: §3.3 · pp. 4–5, Table 2 · p. 5

4.1 Infrastructure and Precision

• Pretraining ran on 400 H100 GPUs for approximately 272K hours using FP8, with gradients kept in BF16 during the backward pass.Source: §4.1 · p. 6

4.2 Kernel Fusion

• Hand-written fused CUDA kernels outperform naive PyTorch and torch.compile in both passes; torch.compile reaches up to 12.9× speedup over naive.Source: §4.2 · p. 6, Table 3 · p. 6

4.3 Parallel Muon

• Parallel Muon distributes gradient subsets across ranks, gathers assigned full matrices, computes Newton–Schulz updates concurrently, then scatters sharded updates; pipelining overlaps phases to improve throughput and reduce memory use.Source: §4.3 · pp. 7–8, Figure 1 · p. 7
• 571 TFLOPS/GPU versus 80 TFLOPS/GPU yields a 7.1× throughput improvement for non-pipelined Parallel Muon over Distributed Muon.Source: §4.3 · p. 9, Table 4 · p. 9
• FLOPs-based sorting lifts pipelined throughput by over 21% after pipelining first drops throughput approximately 15.8% versus non-pipelined.Source: §4.3 · p. 9
• The fully optimized pipelined version is only about 2.1% faster than non-pipelined, while non-pipelined peak memory exceeds pipelined versions by over 4.1× and 3×.The memory comparisons are against the pipelined non-sorted and fully optimized configurations, respectively.Source: §4.3 · p. 9

5.1 Supervised Fine-tuning

• The three SFT stages progress from general instruction learning to synthetic-data reasoning and domain enhancement, then data-pruned targeted refinement.The pipeline explicitly targets multi-step reasoning, code synthesis, and mathematical problem-solving.Source: §5.1 · p. 10

5.1.1 Stage 1: Large-Scale Supervised Fine-tuning

• Stage 1 uses approximately 28 million balanced-domain samples, packs sequences up to 16,384 tokens, and unfreezes all model parameters.The training setup also specifies FP8 precision, FSDP, and Parallel Muon kernels.Source: §5.1.1 · p. 10

5.1.2 Stage 2: Synthetic and Targeted Fine-tuning

• Stage 2 combines quality-screened open instruction data with synthetic examples targeting compositional reasoning, algorithmic problem-solving, and multi-step instruction following.This lower-learning-rate, shorter continuation stage is reported to improve reasoning quality while preserving conversational fluency.Source: §5.1.2 · p. 10

5.1.3 Stage 3: Data-Pruned Refinement

• Stage 3 filters redundant or low-utility Stage 2 data to reinforce diversity and stabilize reasoning and alignment without changing conversational fluency.The filtering criteria emphasize diversity, reasoning coherence, and linguistic fidelity.Source: §5.1.3 · p. 11

5.2 Post-training Evaluation

• Evaluation covers general knowledge and alignment, mathematics and text reasoning, and agentic and programming tasks, using official Qwen3 and Gemma3 scores as baselines.Instruct evaluations use temperature 0.6 and a maximum sequence length of 32,768 tokens.Source: §5.2 · p. 11
• Despite training on 5.5T tokens, Motif-2-12.7B-Instruct performs comparably to or better than similarly sized or larger open-weight models across several benchmarks.The report highlights MATH-500, AIME25, and LiveCodeBench, and contrasts its training corpus with Qwen3's 36T tokens and Gemma3's 12–14T.Source: §5.2 · p. 11

6 Conclusion

• Motif-2-12.7B combines width-preserving hypercloning, Llama Pro depth scaling, Grouped Differential Attention, curriculum-aware pre-training, and three-stage SFT.The report characterizes the result as efficient and broadly capable without extreme parameter counts.Source: §6 · p. 12
• Fused PolyNorm kernels and Parallel Muon are reported to provide throughput and memory gains at long context, making large-scale training more accessible.Source: §6 · p. 12
• The report plans to release Motif-2-12.7B-Reasoning, an RL-enhanced variant focused on mathematics and code while maintaining conversational fluency.Source: §6 · p. 12

7 Appendix

• Sungmin Lee and Junghwan Lim are credited with technical and management leadership.Source: §7.1 · p. 17
• Dongseok Kim, Taehyun Kim, Eunhwan Park, Jeesoo Lee, Jeongdoo Lee, and Junhyeok Lee are listed as core contributors.Source: §7.1 · p. 17
• The appendix lists 17 additional contributors and states that authors are alphabetically sorted by last name.Source: §7.1 · p. 17