PaperlayerView exact revision on arXiv

Source-linked AI summary

Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

Ailin Huang, Ang Li, Aobo Kong, Bin Wang, Binxing Jiao, Bo Dong, Bojun Wang, Boyu Chen, Brian Li, Buyun Ma, Chang Su, Changxin Miao, Changyi Wan, Chao Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengting Feng, Chengyuan Yao, Chunrui Han, Dan Ma, Dapeng Shi, Daxin Jiang, Dehua Ma, Deshan Sun, Di Qi, Enle Liu, Fajie Zhang, Fanqi Wan, Guanzhe Huang, Gulin Yan, Guoliang Cao, Guopeng Li, Han Cheng, Hangyu Guo, Hanshan Zhang, Hao Nie, Haonan Jia, Haoran Lv, Hebin Zhou, Hekun Lv, Heng Wang, Heung-Yeung Shum, Hongbo Huang, Hongbo Peng, Hongyu Zhou, Hongyuan Wang, Houyong Chen, Huangxi Zhu, Huimin Wu, Huiyong Guo, Jia Wang, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiashu Lv, Jiashuo Liu, Jiayi Fu, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yang, Jie Zhou, Jieyi Hou, Jing Bai, Jingcheng Hu, Jingjing Xie, Jingwei Wu, Jingyang Zhang, Jishi Zhou, Junfeng Liu, Junzhe Lin, Ka Man Lo, Kai Liang, Kaibo Liu, Kaijun Tan, Kaiwen Yan, Kaixiang Li, Kang An, Kangheng Lin, Lei Yang, Liang Lv, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lina Chen, Luck Ma, Mengqiang Ren, Michael Li, Ming Li, Mingliang Li, Mingming Zhang, Mingrui Chen, Mitt Huang, Na Wang, Peng Liu, Qi Han, Qian Zhao, Qinglin He, Qinxin Du, Qiuping Wu, Quan Sun, Rongqiu Yang, Ruihang Miao, Ruixin Han, Ruosi Wan, Ruyan Guo, Shan Wang, Shaoliang Pang, Shaowen Yang, Shengjie Fan, Shijie Shang, Shiliang Yang, Shiwei Li, Shuangshuang Tian, Siqi Liu, Siye Wu, Siyu Chen, Song Yuan, Tiancheng Cao, Tianchi Yue, Tianhao Cheng, Tianning Li, Tingdan Luo, Wang You, Wei Ji, Wei Yuan, Wei Zhang, Weibo Wu, Weihao Xie, Wen Sun, Wenjin Deng, Wenzhen Zheng, Wuxun Xie, Xiangfeng Wang, Xiangwen Kong, Xiangyu Liu, Xiangyu Zhang, Xiaobo Yang, Xiaojia Liu, Xiaolan Yuan, Xiaoran Jiao, Xiaoxiao Ren, Xiaoyun Zhang, Xin Li, Xin Liu, Xin Wu, Xing Chen, Xingping Yang, Xinran Wang, Xu Zhao, Xuan He, Xuanti Feng, Xuedan Cai, Xuqiang Zhou, Yanbo Yu, Yang Li, Yang Xu, Yanlin Lai, Yanming Xu, Yaoyu Wang, Yeqing Shen, Yibo Zhu, Yichen Lv, Yicheng Cao, Yifeng Gong, Yijing Yang, Yikun Yang, Yin Zhao, Yingxiu Zhao, Yinmin Zhang, Yitong Zhang, Yixuan Zhang, Yiyang Chen, Yongchi Zhao, Yongshen Long, Yongyao Wang, Yousong Guan, Yu Zhou, Yuang Peng, Yuanhao Ding, Yuantao Fan, Yuanwei Lu, Yuanzhen Yang, Yuchu Luo, Yudi Zhao, Yue Peng, Yueqiang Lin, Yufan Lu, Yuling Zhao, Yunzhou Ju, Yurong Zhang, Yusheng Li, Yuxiang Yang, Yuyang Chen, Yuzhu Cai, Zejia Weng, Zetao Hong, Zexi Li, Zhe Xie, Zheng Ge, Zheng Gong, Zheng Zeng, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhiheng Hu, Zidong Yang, Zili Wang, Ziqi Ren, Zixin Zhang, Zixuan Wang

arXiv:2602.10604v2cs.CLcs.AI

TL;DR

Open-source models still trail frontier systems in complex reasoning, while efficiency bottlenecks constrain long-context agentic tasks. Step 3.5 Flash addresses this with a unified post-training approach combining domain specialization, global synthesis, and stable large-scale reinforcement learning. It achieves competitive results across reasoning and agentic benchmarks despite using 11B active parameters, including 85.4% on IMO-AnswerBench.

  • Problem

    Open-source models still trail closed-source frontiers in complex reasoning, and efficiency bottlenecks hinder their use in long-context agentic tasks.

    Source: §1 · p. 4

  • Method

    The paper proposes a unified post-training framework that alternates domain-specific specialization with global synthesis and uses MIS-PO for scalable, stable reinforcement learning.

    Source: §1 · p. 4

  • Results

    Across reasoning and agentic benchmarks, Step 3.5 Flash is competitive with leading frontier models despite 11B active parameters, including 85.4% on IMO-AnswerBench.

    Source: §1 · p. 5

  • Takeaways & Limitations

    The results support Step 3.5 Flash as a compact model option for reasoning and agentic workloads.

    Source: §1 · p. 5

    Step 3.5 Flash is tailored to coding and work-centric tasks and may be less stable under distribution shifts, especially in specialized domains or long multi-turn dialogues.

    Source: §7 · p. 24

Abstract

from arXiv · show

We introduce Step 3.5 Flash, a sparse Mixture-of-Experts (MoE) model that bridges frontier-level agentic intelligence and computational efficiency. We focus on what matters most when building agents: sharp reasoning and fast, reliable execution. Step 3.5 Flash pairs a 196B-parameter foundation with 11B active parameters for efficient inference. It is optimized with interleaved 3:1 sliding-window/full attention and Multi-Token Prediction (MTP-3) to reduce the latency and cost of multi-round agentic interactions. To reach frontier-level intelligence, we design a scalable reinforcement learning framework that combines verifiable signals with preference feedback, while remaining stable under large-scale off-policy training, enabling consistent self-improvement across mathematics, code, and tool use. Step 3.5 Flash demonstrates strong performance across agent, coding, and math tasks, achieving 85.4% on IMO-AnswerBench, 86.4% on LiveCodeBench-v6 (2024.08-2025.05), 88.2% on tau2-Bench, 69.0% on BrowseComp (with context management), and 51.0% on Terminal-Bench 2.0, comparable to frontier models such as GPT-5.2 xHigh and Gemini 3.0 Pro. By redefining the efficiency frontier, Step 3.5 Flash provides a high-density foundation for deploying sophisticated agents in real-world industrial environments.

1. Introduction

Step 3.5 Flash targets the reasoning and efficiency gaps that limit open models in agentic tasks through an efficient MoE architecture and a unified training recipe. The paper reports competitive reasoning and agentic benchmark performance with 11B active parameters.

Motivation

  • Open models still trail closed-source frontiers in complex reasoning, while efficiency bottlenecks constrain long-context agentic use and resource-constrained deployment.The introduction also identifies costly, fragmented expert iteration and instability from accumulated off-policy discrepancies and MoE routing shifts as challenges for scaling reinforcement learning.Source: §1 · p. 4

Architecture

  • The 196B-parameter MoE activates 11B parameters per token and combines 3:1 sliding-window to full attention with MTP-3 to reduce long-context latency.The design also increases SWA query heads from 64 to 96 and uses head-wise gated attention; it sustained ∼170 tokens/s on Hopper GPUs during its first week on OpenRouter.Source: §1 · p. 4

Pretraining

  • Stable pretraining over 17.2T tokens yields a base model competitive with larger counterparts on math, coding, and knowledge benchmarks.An observability and diagnostic stack, together with an improved Muon optimizer, mitigates MoE failure modes; training had only a single transient loss spike, and the base model scores 31.6% on SimpleQA, surpassing DeepSeek-V3.2-Exp Base with one-third of its parameters.Source: §1 · p. 4

Post-training

  • The unified post-training recipe combines domain specialization and global synthesis with long-context mid-training and MIS-PO for stable, scalable RL.Built on a shared SFT foundation, it scales context to 128k; MIS-PO uses token- and trajectory-level distributional filtering to reduce gradient variance while preserving learning signals for long-horizon reasoning and agentic behavior.Source: §1 · p. 4

Results

  • With 11B active parameters, Step 3.5 Flash achieves competitive performance with leading frontier models across reasoning and agentic benchmarks.It scores 85.4% on IMO-AnswerBench, 86.4% on LiveCodeBench-v6 (2024.08–2025.05), 88.2% on 𝜏2-Bench, 69.0% on BrowseComp (with context management), and 51.0% on Terminal-Bench 2.0.Source: §1 · p. 5

2. Architecture

Step 3.5 Flash co-designs attention, sparse MoE, and multi-token prediction for low-latency agentic workloads, combining a 196B-parameter backbone with only 11B active parameters. Ablations support its 𝑆3𝐹1+Head attention configuration and head-wise gating as efficiency-conscious design choices.

2.1. Design Philosophy

  • Agentic workloads combine extensive context prefilling with prolonged interactive decoding, so the design targets low wall-clock latency through attention, sparse MoE, and multi-token prediction.SWA with 𝑊=512 supports parallel verification for speculative decoding, while GQA-8 aligns eight KV heads with 8-way tensor parallelism and leaves computational slack for speculation.Source: §2.1 · p. 5

2.2. Sparse MoE Backbone with Hybrid Attention

  • The 45-layer backbone has 3 dense and 42 MoE layers, with 288 routed experts plus one shared expert per MoE layer and top-8 routing, for 196B total and 11B active parameters.Its hybrid attention uses a 3:1 SWA-to-full motif; increasing SWA query heads from 64 to 96 mitigates the quality drop from naive interleaving, and head-wise gating adds a parameter-efficient, data-dependent alternative to sink tokens.Source: §2.2 · pp. 6–7
  • EP-level balancing addresses rank-level load imbalance that loss-free expert balancing alone does not prevent, reducing the risk of routing stragglers throttling throughput.An EP-level balancing loss explicitly promotes uniform utilization across expert-parallel ranks.Source: §2.1 · p. 6, §2.2 · p. 7
  • Three lightweight MTP heads, each using SWA and a dense FFN, add 0.81B parameters (∼0.41%) to accelerate speculative decoding on long-context agentic workloads.Only MTP-1 is optimized during most training; MTP-2 and MTP-3 are initialized from it and jointly trained in a lightweight post-training phase, with position-dependent loss reweighting across prediction offsets.Source: §2.1 · p. 6, §2.2 · p. 7

2.3. Architecture Ablations and Results

  • 55.7 vs. 54.1 during pretraining, LongCtx 28.2 vs. 27.5, and Sci 44.0 vs. 42.4 show 𝑆3𝐹1+Head nearing 𝐹𝐹𝐹𝐹 quality.The 𝑆1𝐹1 layout achieves the strongest overall SFT quality and LongCtx score (29.6), but costs ∼1.58/1.65 attention-side prefill/decode FLOPs; the authors adopt 𝑆3𝐹1+Head for its lower cost and stable long-context performance.Source: §2.3 · p. 8, Table 1 · p. 8
  • 62.46 to 64.43 (+1.97) average performance shows head-wise gating outperforming fixed sink tokens across benchmarks.The controlled comparison holds the 𝑆3𝐹1 layout and 𝑊=512 window constant; head-wise gating is adopted as the default mechanism.Source: §2.3 · p. 9, Table 2 · p. 9

3. Infrastructure

Step 3.5 Flash’s infrastructure combines a 4,096-GPU H800 cluster with a unified training stack for pre-training, post-training, and reinforcement learning. The framework optimizes distributed communication and monitoring to reduce training overhead at scale.

3.1. Compute Cluster

  • A 4,096-GPU training cluster uses 8-GPU nodes linked by NVLink and NVSwitch, with 8×200 Gbps RoCE links between nodes.Source: §3.1 · p. 9

3.2. Training Framework

  • Communication optimizations jointly reduce training iteration time by up to 5% through fabric-aware scheduling and communication-aware rank placement.The framework also uses decoupled attention and MoE parallel groups, with 8-way pipeline parallelism, 8-way expert parallelism, and ZeRO-1 data parallelism. Muon ZeRO-1 resharding cuts iteration time by approximately 5% with less than 4 GB additional memory versus naive all-reduce; kernel fusion and selective activation recomputation further target training efficiency.Source: §3.2 · pp. 9–10

3.3. High-Throughput Lightweight Monitoring

  • Asynchronous metric offloading reduces telemetry overhead to approximately 100 ms per iteration, avoiding synchronous global reduction that could double iteration time.The training stack is Steptron, built on PyTorch and Megatron-LM [67], and supports pre-training, post-training, and RL workloads.Source: §3.2 · p. 9, §3.3 · p. 10

4. Pre-Training and Mid-Training

Pre-training and mid-training combine stability diagnostics and mitigations with a staged curriculum that expands context and shifts data toward software, agentic, and long-horizon tasks. The process covers approximately 17.6T pre-training tokens and 750B mid-training tokens.

4.1. Training Stability

  • With diagnostic-guided mitigations, pre-training loss shows only one spike across the full training duration, while an asynchronous micro-batch metrics stack tracks optimizer- and expert-level signals.The diagnostics surfaced transient loss spikes, expert-side collapse despite apparently healthy routing, and localized activation blow-ups.Source: Figure 3 · p. 11, §4.1 · p. 11

4.1.1. Numerical Sensitivity of Muon

  • Casting only Polar Express state and intermediates to float16 stopped rare, unrecoverable loss spikes while the rest of training remained mixed-precision.The team adopted Polar Express for orthogonalization and ran a fixed T=6 steps to balance optimization quality and throughput.Source: §4.1.1 · pp. 11–12

4.1.2. Expert Collapse Beyond Routing Collapse

  • Expert collapse can persist despite healthy routing; calibrate shared-versus-routed expert scaling, avoid overly restrictive micro-batch balancing, and monitor per-expert activation and parameter norms.Broader-scope balancing or loss-free bias adjustment are alternatives; decreasing minimum-to-median activation or parameter-norm ratios can warn of expert death.Source: §4.1.2 · p. 12

4.1.3. Localized Activation Blow-up in MoE Layers

  • Activation clipping bounds rogue-expert norms, whereas weight clipping alone fails to prevent outlier recurrence; the max-to-median activation-norm ratio reveals instability hidden by training loss.The problem is concentrated in a few experts in deeper MoE layers, even while median activation norms and loss remain stable.Source: §4.1.3 · pp. 12–13, Figure 4 · p. 13

4.2. Training Curriculum

  • The curriculum progresses from broad 4k-context pre-training to 32k specialization, then 128k mid-training, for approximately 17.6T and 750B tokens, respectively.Pre-training anneals toward higher-quality knowledge and software-development data; mid-training extends context to support long-horizon reasoning and downstream agentic workloads.Source: §4.2 · p. 13

4.2.1. Data Mixture

  • The data mixture combines curated web and book/document knowledge, refined code, GitHub PR/issue/commit corpora, and synthetic or semi-synthetic tool-use and reasoning samples.Code-centric data is upsampled during annealing and mid-training, which also adds long-context documents and tasks for extended planning and reasoning.Source: §4.2.1 · p. 14

4.2.2. Schedule

  • Pre-training uses 14.6T tokens at 4k context, then 3T annealed tokens split between 2T at 4k and 1T at 32k; mid-training uses 386B tokens at 32k and 364B at 128k.Mid-training replays 81B tokens (21%) in Stage 1 and retains 10.5B replay tokens in Stage 2, which adds long-horizon reasoning, long documents, and agent-specific data.Source: §4.2.2 · p. 14

4.2.3. Hyper-Parameters

  • Pre-training uses Muon, a peak learning rate of 2.5 × 10^-4, and a global batch growing to 16,384; mid-training freezes router weights and uses a 2 × 10^-5 warmup peak.Pre-training also uses stage-specific MTP weights, load-balancing settings, and RoPE scaling; mid-training fixes MTP loss weight at 0.1, disables EP-group balance loss, and raises full-attention RoPE θ to 5,000,000 at 128k.Source: §4.2.3 · p. 15

5. Post-Training

The post-training recipe combines domain-specialized RL, self-distillation, and scalable RL with verifiable and preference-based rewards. It pairs this training approach with curated data and agent infrastructure to support stable improvement across reasoning and tool-use tasks.

5.1. Expert Model Construction and Self-Distillation

  • The recipe alternates domain-specific RL expert construction with self-distillation and scalable RL to produce a generalist competitive with specialized baselines.A two-stage SFT foundation spans multiple domains and adds about 30k expert chemistry trajectories and synthetic arithmetic tasks; expert-generated trajectories are rejection-sampled into a unified student.Source: §5 · p. 15, §5.1 · pp. 15–16

5.2.1. MIS-Filtered Policy Optimization (MIS-PO)

  • MIS-PO filters off-distribution samples with binary masks at token and trajectory levels, treating retained trajectories as effectively on-policy to reduce gradient variance.Truncation-aware value bootstrapping stabilizes training at truncation rates up to 20%, while routing confidence serves as a proxy for MoE training stability.Source: §5.2.1 · pp. 16–17
  • MIS-PO achieved higher reward plateaus, faster convergence, lower actor-gradient noise, and slower entropy decay than PPO in the reported comparison.The approximately 5,000-step ablation reports lower actor-gradient noise; Figure 5 summarizes efficiency, stability, and exploration persistence.Source: Figure 5 · p. 16, §5.2.1 · p. 17
  • +3.2% on IMO-AnswerBench, +6.1% on CF-Div2-Stepfun-cpp, +10.6% on ARC-AGI-1, and +3.4% on HLEtext accompanied RL training’s steady reward growth.Figure 6 shows reward growth and accuracy boosts across these benchmarks.Source: §5.2.1 · p. 17, Figure 6 · p. 18

5.2.2. Reward System

  • The reward system uses task-specific verifiers for verifiable tasks and GenRM preference rewards for non-verifiable tasks, with MetaRM improving GenRM results by 0.5%–3% on every benchmark.GenRM converts pairwise preference confidence into a Bradley–Terry win rate, controls length, and assigns zero reward to fabricated citations, overconfident claims, or language inconsistencies; agent reports use asymmetric binary rewards.Source: §5.2.2 · pp. 18–19

5.3. Data Synthesis & Curation

  • The curated prompt pool combines open-source, synthetic, and user-trajectory data, using strict global filtering and domain-specific refinement to balance difficulty and maximize reasoning density; it contains 871k samples (7.23B tokens).Source: §5.3 · p. 19

5.3.3. Code Agents

  • The code-agent pipeline builds and verifies environments as training data, curating 50k verified environments across over 15k GitHub repositories and more than 20 programming languages.A cross-task memory pool, loop detection, and execution feedback support environment construction and coding improvement.Source: §5.3.3 · p. 20

5.4. Agent Infrastructure

  • Search and research data uses graph expansion and cross-website browsing, while agent infrastructure retains recent reasoning traces and uses XML tool calls for robust long-horizon interactions.The infrastructure also supports thousands of concurrent environments and training across academic and enterprise interaction frameworks.Source: §5.3.4 · pp. 20–21, §5.4 · p. 21

6. Evaluations

Across pre- and post-training evaluations, Step 3.5 Flash performs competitively with larger sparse models and achieves performance on par with GPT-5.2 xHigh and Gemini 3.0 Pro while activating 11B parameters.

6.1. Pre-training Evaluations

  • Pre-training evaluation spans general language understanding and reasoning, mathematics, coding, and Chinese understanding.Benchmarks include BBH, MMLU, GSM8K, MATH, HumanEval, MBPP, C-Eval, CMMLU, and C-SimpleQA.Source: §6.1 · pp. 21–22
  • 31.6 on SimpleQA beats DeepSeek-V3.2-Exp Base (27.0) with 196B versus 671B total parameters; Step 3.5 Flash also scores 88.2 on BBH and 85.8 on MMLU.It activates 11B parameters and scores 81.1 on HumanEval, 67.7 on MultiPL-E HumanEval, and 58.0 on MultiPL-E MBPP.Source: §6.1 · p. 22, Table 4 · p. 22

6.2. Post-training Evaluations

  • Post-training evaluation covers reasoning, coding, agent, general-capability, and long-context benchmarks.Examples include IMO-AnswerBench, LiveCodeBench-v6, τ2-Bench, BrowseComp, ArenaHard v2, and LongBench v2.Source: §6.2 · p. 23
  • Step 3.5 Flash performs on par with frontier models and excels on reasoning-intensive benchmarks while delivering strong results on agentic evaluations.The reported agentic evaluations include SWE-Bench Verified, Terminal-Bench 2.0, BrowseComp (with Context Manager), GAIA, and τ2-Bench.Source: §6.2 · p. 23

7. Limitations … A.3. Meta Token

Step 3.5 Flash’s limitations include longer generation trajectories and reduced stability under distribution shifts, while the appendix describes attention enhancements and metadata conditioning. The paper identifies expert-level open-world tasks as a future frontier and proposes more efficient distillation.

7. Limitations

  • Step 3.5 Flash requires longer generation trajectories than Gemini 3.0 Pro for comparable quality; the authors plan to compress reasoning while maintaining competitive performance.Source: §7 · p. 23
  • The paper aims to combine generalist versatility with deep domain expertise through on-policy distillation variants that help the model internalize expert behaviors more sample-efficiently.Source: §7 · p. 23
  • Competitive academic agentic performance does not yet address professional, advanced engineering, or scientific tasks; applying RL to these expert-level settings is identified as necessary for genuine autonomy.Source: §7 · p. 24
  • The model may become less stable in specialized domains or long multi-turn dialogues, producing repetitive reasoning, mixed-language outputs, or time and identity inconsistencies.Source: §7 · p. 24

A. Architecture Details

A.1. Head-wise Gated Attention

  • Input-dependent head-wise gating modulates attention output with an adaptive sink mass, and the paper reports that it consistently outperforms fixed, input-independent sink tokens.Source: §A.1 · p. 28

A.2. Speed Benchmark of Attention Enhancements

  • With MTP-3, more SWA query heads slightly increase FLOPs but have less latency impact at a query-to-KV ratio of 12; head-wise gating adds no noticeable FLOPs or latency.Source: Table 7 · p. 28, §A.2 · p. 28

A.3. Meta Token

  • Training prepends human-readable metadata about content type, language, domain, and source; after approximately 3.8T tokens, metadata remains in context but its loss is masked while payload tokens are predicted.Source: §A.3 · p. 29

A.4. Pre-training Ablations Details

Controlled pre-training ablations test hybrid attention layouts and compare sink tokens with head-wise gated attention. The results show that increasing SWA query heads narrows the vanilla layout’s gap to full attention, while head-wise gating improves the sink-token metric.

Pre-training ablations

  • The controlled ablations isolate the effects of hybrid attention layout and sink tokens versus head-wise gated attention.GPQA uses 5-shot prompting, while HumanEval and MBPP use 3-shot prompting.Source: §A.4 · p. 29

Hybrid attention layout

  • The hybrid-layout ablation uses a 30B-A3B MoE model trained on approximately 1.4T tokens, followed by SFT on a 0.1× downsampled dataset.Training comprises 30B-token warmup, 1T-token main pre-training, 300B-token cooldown, and 100B-token long-context specialization.Source: §A.4 · p. 29, Table 9 · p. 30
  • Increasing SWA query heads substantially closes the vanilla S3F1 layout’s gap to full attention: MMLU-Pro improves +3.7 and SFT Reasoning +0.4, with SFT Code −0.6.The vanilla layout underperforms full attention on general pre-training benchmarks and degrades SFT quality, including BBH −4.3 and SFT Avg −0.7; the increased-head layout matches or exceeds full attention on several metrics.Source: §A.4 · p. 30

Gate vs. sink

  • Head-wise gated attention improves the sink-token metric from 62.5 to 64.4 (+1.9).The comparison uses a 100B-A10B MoE model pre-trained for approximately 250B tokens.Source: §A.4 · pp. 29–30

B. Detail Analysis of Localized Activation Blow-up … C.4. Data Infrastructure

The analysis attributes localized activation blow-up to high-frequency bigram specialization and explains why activation clipping is favored; the data sections describe curation pipelines spanning web, code, STEM, and infrastructure.

B. Detail Analysis of Localized Activation Blow-up

  • High-frequency bigrams can trigger extreme first-token activations when fine-grained MoE experts specialize without load-balancing regulation, creating deterministic shortcuts; ordinary lexical-token activations are not treated as failure.The lexical-item pattern is common in shallower layers and lacks the rapid increment associated with the failure mode.Source: B. Detail Analysis of Localized… · pp. 30–31
  • RMSNorm makes the model’s final representation converge toward an outlier expert’s direction when its output is enlarged without bound.The analysis expresses the final representation as the sum of layer outputs followed by RMSNorm, and shows the limiting behavior as the outlier magnitude tends to infinity.Source: B. Detail Analysis of Localized… · p. 31
  • Abnormal SwiGLU outputs arise from highly aligned gate and up activations concentrated in few dimensions, motivating activation clipping over weight clipping.The gate and up activation norms show no significant difference between outlier and normal experts, but their element-wise product produces abnormal outputs; clipping has negligible negative effects when normal activations stay below threshold.Source: B. Detail Analysis of Localized… · p. 31

C.1.1. StepCrawl

  • StepCrawl supplies high-signal web pages and document-like content, processing about 1B pages daily with model-guided filtering and crawl-budget allocation to preserve corpus diversity.The system targets SEO-driven and low-utility pages while balancing site categories; collected content also undergoes quality scoring, deduplication, and sanitization under robots.txt and site-specific access policies.Source: §C.1.1 · p. 32

C.1.2. Quality Refinement and Stratification

  • The data pipeline stratifies web, book, and paper corpora by quality, rebalances embedding clusters, and mines knowledge-dense material for augmentation.It retains High/Medium-High/Medium web tiers, but only High/Medium-High book and paper tiers during annealing; cluster rebalancing improves a broad set of benchmarks, while knowledge mining and targeted rephrasing or QA synthesis yield consistent ablation gains.Source: §C.1.2 · p. 33

C.2.1. Pure-Code

  • The hit0–6 code-filtering configuration yields the best overall benchmark performance, balancing data quality and diversity better than hit0-only filtering or no filtering.Accepting documents with up to six heuristic violations retains a wider variety of high-signal code than the original strict constraints.Source: §C.2.1 · p. 33

C.2.2. PR/Issue/Commit Data

  • The software-engineering corpus combines GitHub PR, issue, and commit data with 90B tokens of code-editing dialogues, 12B tokens of rewritten reasoning data, and hundreds of thousands of environment-based seed samples.The dialogue data uses file-localization and code-repair templates, with phase-specific masking; ablations report gains over SWE-Bench Verified and SWE-Bench Multilingual. Rewritten reasoning data further improves SWE-Bench Verified, while environment seeds drive gains in downstream agent tasks.Source: §C.2.2 · pp. 33–34

C.3. Mathematics & STEM Data

  • The mathematics and STEM dataset adds hundreds of billions of math-related tokens distinct from Common Crawl and a diverse 100M-sample educational collection.The collection spans K–12 mathematics, physics, and chemistry, humanities, and adult vocational exams; early experiments find problem-solving data crucial for pre-training token efficiency.Source: §C.3 · p. 34

C.5. Data Ablations Setting … D.2.2. Reward System

The sections describe fixed-budget data ablations and a post-training pipeline combining curated supervised fine-tuning with large-scale reinforcement learning. They also specify RL data curation and reward mechanisms for verifiable training.

C.5. Data Ablations Setting

  • A fixed training budget underpins the 30B-A3B Muon ablation suite, evaluated on comprehensive benchmarks and held-out compression tests.The paper says compression metrics offer a complementary, often more direct measure of knowledge capacity.Source: §C.5 · p. 35
  • The 30B-A3B MoE model provides stronger stability and better fidelity to full-scale trends than smaller proxies in internal experiments.The paper notes that smaller models can miss complex reasoning and long-tail patterns, creating an artificial bias toward data repetition.Source: §C.5 · p. 35

D. Post Training Details

  • Post-training combines quality-controlled SFT with large-scale RL to improve reasoning, tool use, and generalization.The post-training description also covers data processing and quality control for SFT.Source: D. Post Training Details · p. 35, §D.1.1 · p. 35

D.1.1. SFT Data Processing Pipeline

  • SFT filtration removes degenerate, harmful, or personally identifying data, linguistically inconsistent samples, and benchmark contamination; the refined dataset has 871k samples and 7.23B tokens.Decontamination uses exact matching with digit masking and N-gram matching.Source: §D.1.1 · p. 35

D.2. RL Details and Ablations

  • RL post-training covers data curation, asynchronous search-agent training, and dense- and MoE-model ablations.Source: §D.2 · p. 36

D.2.1. Data Curation

  • RL data spans coding, STEM, and synthetic general-reasoning tasks; it excludes competitions from 2024–2026 and filters prompts without unique answers or trivial or degenerate problems.Synthetic additions include 11–13 digit arithmetic, generated coding test cases, and puzzle or instruction-following environments; batches sample domains using predefined probabilities.Source: §D.2.1 · p. 36

D.2.2. Reward System

  • STEM answers receive gpt-oss-120b verification, while coding tasks use sandbox execution against test cases with soft reward.The grader checks answer correctness, reasoning consistency, required format, and completeness across sub-questions.Source: §D.2.2 · p. 36
  • MIS-PO consistently outperforms GSPO in efficiency and stability on both dense and MoE models.Source: Figure 7 · p. 37

D.2.3. RL Ablation Details … E.1.4. Chinese understanding benchmarks

The sections report RL and search-agent training findings, test-time reasoning approaches, and evaluation protocols across agentic, language, math, coding, and Chinese-understanding tasks. They also describe an edge-cloud GUI agent that outperforms its edge-only baseline on AndroidDaily Hard.

D.2.3. RL Ablation Details

  • MIS-PO shows greater sample efficiency and scalability than GSPO, constraining training–inference mismatch where GSPO fails to converge consistently on MoE.The comparison applies the same token- and sample-level masking strategies and selects a clip ratio of 10^-4 after a grid search.Source: §D.2.3 · pp. 37–38
  • MIS-PO’s extended MoE training run maintains rising rewards, stable actor gradient norms, and controlled entropy.Figure 8 displays reward, actor gradient norm, and entropy; its gradient-norm panel uses raw, unsmoothed and undownsampled values.Source: §D.2.3 · p. 38, Figure 8 · p. 38

D.2.4. Search Agent

  • Step 3.5 Flash achieves the highest average tool-use gain (52.0) and leads on GAIA and xbench-DeepSearch.The Search Agent uses FullyAsync generation and updates with sticky scheduling; this configuration yields an approximate 10× efficiency gain while maintaining stability.Source: §D.2.4 · pp. 38–39, Table 11 · p. 39

D.3. Tool-integrated Reasoning and Parallel Reasoning

  • Python tool-integrated reasoning improves performance across challenging math, STEM, and puzzle benchmarks, while interactive PaCoRe improves GPQA and HLEtext over standard reasoning.Tool-integrated reasoning uses a Python sandbox with a 100-turn limit; PaCoRe extends parallel reasoning to interactive agent loops.Source: §D.3 · pp. 39–40, Table 12 · p. 40, Table 13 · p. 40

E.1.1. General language understanding and reasoning benchmarks; E.1.2. Mathematics reasoning benchmarks; E.1.3. Coding benchmarks

  • Official-metric protocols include 5-shot MMLU, 8-shot GSM8K, 4-shot MATH, and 3-shot HumanEval evaluations.The benchmark prompts and settings cover general language understanding, math reasoning, and coding.Source: §E.1.1 · p. 41, §E.1.2 · p. 43, §E.1.3 · p. 43

E.1.4. Chinese understanding benchmarks

  • C-Eval and CMMLU use 5-shot settings, while C-SimpleQA uses 5-shot evaluation with gpt-oss-120b as judge.These protocols use the official evaluation metrics for C-Eval and CMMLU and the official metric and LLM-based judgement protocols for Chinese SimpleQA.Source: §E.1.4 · pp. 44–45

E. Detailed Evaluation Protocols and Prompts

E.3.3. Step 3.5 Flash + Step-GUI

  • 57.0% success on AndroidDaily Hard, versus 40.0% for the edge-only baseline, with cloud planning paired with local execution.The framework is described as generalizing to desktop and automotive infotainment platforms.Source: §E.3.3 · pp. 54–55

E.2.1. Reasoning benchmarks

The section specifies evaluation protocols for competition-level coding, mathematics, and knowledge benchmarks. Its custom CodeForces Div. 2 benchmark uses adversarially tested cases and a validated local evaluator; the other benchmarks follow their stated official methods.

CF-Div2-Stepfun

  • The 53-problem CodeForces Div. 2 benchmark covers September 2024–February 2025 contests, and its evaluator correctly passed 100% of accepted submissions and flagged 92.45% of failed submissions.Its simplified ratings disregard submission-time penalties and are intended for cross-model comparison, not direct comparison with human participants.Source: §E.2.1 · pp. 45–46

LiveCodeBench-v6

  • LiveCodeBench-v6 uses its official evaluation method [12] and a Python-programming prompt requiring solutions that pass all tests.The prompt format supplies starter code and requires the solution to be enclosed in delimiters.Source: §E.2.1 · p. 47

AIME 2025

  • AIME 2025 follows its official evaluation method with repeat@64 and requests answers in boxed MathTeX format.Source: §E.2.1 · p. 47

HMMT 2025 Feb./Nov.

  • HMMT 2025 February and November follow the official evaluation method with repeat@64 and boxed MathTeX answers.Source: §E.2.1 · p. 47

IMO-AnswerBench

  • IMO-AnswerBench follows its official evaluation method [91] with repeat@64 and boxed MathTeX answers.Source: §E.2.1 · p. 47

MMLU-Pro

  • MMLU-Pro follows its official evaluation method, with processing consistent with the paper’s pre-training evaluation and answers formatted as boxed options.Source: §E.2.1 · p. 47

GPQA-Diamond

  • GPQA-Diamond follows its official evaluation method and requests the selected answer option in a box.Source: §E.2.1 · p. 48

HLEtext

  • HLEtext uses the benchmark’s official metric and LLM-based judgment protocol, with gpt-oss-120b as judge.Source: §E.2.1 · p. 48

E.2.2. Code Agent benchmarks

Step 3.5 Flash reports strong results on SWE-Bench and Terminal-Bench 2.0. Terminal-Bench ablations identify long-output limits and task time as important performance factors.

SWE-Bench

  • 74.4% on SWE-Bench Verified and 67.4% on SWE-Bench Multilingual; SWE-Agent scored 74.2% and Claude Code scored 72.0%.Claude Code used an extended 4-hour limit per instance and no per-tool execution time limit.Source: §E.2.2 · p. 48

Terminal-Bench 2.0

  • 67/89 pass@8 and an avg@8 of 50.98% were reported on Terminal-Bench 2.0.Each task ran across 8 trials; checker refinement improved overall accuracy by approximately 1.5%.Source: §E.2.2 · p. 49
  • 88.6% of successful Terminal-Bench trajectories finished within 30 interactions, and 23/89 tasks succeeded in all 8 trials.History pruning was triggered in 9.41% of successful trajectories.Source: §E.2.2 · p. 49
  • Limit 16k caused the largest performance drop; disabling context management under that limit lowered results to 45.22%, while Rounds 100 had minimal impact.Timeout 2h reduced performance on tasks involving model training, heavy compilation, or complex environment configuration.Source: §E.2.2 · p. 49, Table 16 · p. 49

E.2.3. General Agent benchmarks … E.3. Internal Evaluation - Benchmarks and Methodology

The supplied passages evaluate agent performance across deep search, research, tool-use, general, and long-context benchmarks. They report agent results and describe the evaluation protocols and context-management conditions used.

E.2.3. General Agent benchmarks

  • 69.0 on BrowseComp (w. Ctx Manage) used discard-all on the full dataset, triggered above context thresholds of 72k tokens for BrowseComp and 41k for BrowseComp-ZH.The evaluation allowed up to 1000 steps; the discard-all approach discards the full context and reinitializes the operational loop.Source: §E.2.3 · p. 50
  • 66.0% accuracy on a 200-instance BrowseComp subset made Discard-all competitive among single-agent context strategies; Multi-agent ranked highest, followed by Discard-all, Keep-first&lastK, and Summary.The reported hierarchy closely aligned with real steps, reflecting a trade-off between inference cost and accuracy.Source: Table 17 · p. 50, §E.2.3 · p. 50
  • 65.3 on RESEARCHRUBRICS in the standardized ReAct framework surpassed proprietary commercial baselines, while Gemini 3.0 Pro scored 50.1.The authors attribute Gemini 3.0 Pro’s lower score to insufficient search depth and reliance on internal knowledge rather than extensive retrieval.Source: §E.2.3 · p. 50
  • 63.69 was the aggregated RESEARCHRUBRICS score of the leading commercial system, Gemini DeepResearch, based on reports collected through official interfaces from Dec 2–15, 2025.The benchmark covers 101 diverse research tasks with 20–43 expert-written scoring criteria per task.Source: §E.2.3 · p. 50, Table 18 · p. 51

E.2.4. General benchmarks

  • For MultiChallenge, we used its official evaluation metric with o3-mini as judge; for IFBench [156], we used its official evaluation method.This followed findings from the GPT-5 release that GPT-4o frequently mis-scores complex responses, leading to underestimated results.Source: §E.2.4 · p. 51

E.2.5. Long Context benchmarks

  • MRCR-8needle used AUC@128k, summarizing retrieval accuracy across context lengths up to 131,072 tokens and normalizing the trapezoidal area over 8k–128k context bins.The metric penalizes declining retrieval accuracy as context length increases.Source: §E.2.5 · p. 52
  • FRAMES-Oracle provided the question and all ground-truth Wikipedia articles as an upper-bound setting simulating perfect retrieval; LongBench v2 and RepoQA used their official evaluation methods.The FRAMES results focus on the Oracle Prompt subset to assess long-context capabilities.Source: §E.2.5 · p. 52

E.3.1. Data Analysis Benchmark · E.3.2. Consulting and Recommendations Benchmark

The sections introduce two benchmarks for evaluating Step 3.5 Flash on practical data analysis and real-world advisory tasks. Results show competitive performance on both, with favorable efficiency claims for data analysis and consulting workflows.

E.3.1. Data Analysis Benchmark

  • The Data Analysis Benchmark uses rubric-grounded tasks derived from senior practitioners’ expertise to capture realistic business constraints and analytical nuance.Ten senior analytics leaders from major Chinese internet companies, each with over 15 years of experience, contributed cases through structured interviews.Source: §E.3.1 · p. 52
  • Its 50 tasks each include a problem statement, CSV dataset, reference analysis, and weighted checklist rubric, averaging 26.9 rubric items per task.Iterative expert review aligned task definitions, data, reference solutions, and evaluation criteria.Source: §E.3.1 · p. 52
  • A unified pipeline supports code-based, research-oriented, and text-based analyses, with model-based rubric scoring averaged across three identical runs.The framework covers task execution, automated scoring, and report synthesis for scalable, reproducible evaluation across heterogeneous environments.Source: §E.3.1 · pp. 52–53
  • 39.58% placed Step 3.5 Flash second overall on the Data Analysis Benchmark, close to GPT-5.2 (39.31%), while Claude Opus 4.5 ranked first.The paper attributes its competitive result partly to adaptation to the Claude Code environment and reports a favorable speed–capability trade-off.Source: Table 19 · p. 53, §E.3.1 · p. 53

E.3.2. Consulting and Recommendations Benchmark

  • The Consulting and Recommendations Benchmark comprises 500 queries from Reddit, Stack Exchange, and community forums across everyday, academic, entertainment, and workplace contexts.Its anchor reference was synthesized from independent responses by GPT-5.2, Claude Opus 4.5, and DeepSeek V3.2, then refined by human experts and assigned 88/100.Source: §E.3.2 · p. 53
  • Consulting responses are scored on Usefulness, Logic, Instruction Following, and Tone, covering actionable quality, factual and structural soundness, constraint adherence, and communicative calibration.The dimensions assess both explicit requirements and contextual expectations, including uncertainty signaling and avoiding overconfidence.Source: §E.3.2 · p. 53
  • A hybrid judge assigns Logic, Instruction Following, and Usefulness to GPT-5.2 and Tone to Claude Opus 4.5, weighting each dimension equally.An alignment study found high Pearson correlation between AI- and human-assigned scores.Source: §E.3.2 · pp. 53–54
  • 70.5% average Score placed Step 3.5 Flash fourth overall on the Consulting and Recommendations Benchmark, matching Gemini 3.0 Pro’s 70.5% vs. 70.6%.The paper also reports substantially lower inference cost and latency than Gemini 3.0 Pro, and says Step 3.5 Flash surpasses larger models on Logic.Source: Table 20 · p. 54, §E.3.2 · p. 54

Related papers

Loading 2602.10604v2…