PaperlayerView exact revision on arXiv

Source-linked AI summary

Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

Ailin Huang, Ang Li, Aobo Kong, Bin Wang, Binxing Jiao, Bo Dong, Bojun Wang, Boyu Chen, Brian Li, Buyun Ma, Chang Su, Changxin Miao, Changyi Wan, Chao Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengting Feng, Chengyuan Yao, Chunrui Han, Dan Ma, Dapeng Shi, Daxin Jiang, Dehua Ma, Deshan Sun, Di Qi, Enle Liu, Fajie Zhang, Fanqi Wan, Guanzhe Huang, Gulin Yan, Guoliang Cao, Guopeng Li, Han Cheng, Hangyu Guo, Hanshan Zhang, Hao Nie, Haonan Jia, Haoran Lv, Hebin Zhou, Hekun Lv, Heng Wang, Heung-Yeung Shum, Hongbo Huang, Hongbo Peng, Hongyu Zhou, Hongyuan Wang, Houyong Chen, Huangxi Zhu, Huimin Wu, Huiyong Guo, Jia Wang, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiashu Lv, Jiashuo Liu, Jiayi Fu, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yang, Jie Zhou, Jieyi Hou, Jing Bai, Jingcheng Hu, Jingjing Xie, Jingwei Wu, Jingyang Zhang, Jishi Zhou, Junfeng Liu, Junzhe Lin, Ka Man Lo, Kai Liang, Kaibo Liu, Kaijun Tan, Kaiwen Yan, Kaixiang Li, Kang An, Kangheng Lin, Lei Yang, Liang Lv, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lina Chen, Luck Ma, Mengqiang Ren, Michael Li, Ming Li, Mingliang Li, Mingming Zhang, Mingrui Chen, Mitt Huang, Na Wang, Peng Liu, Qi Han, Qian Zhao, Qinglin He, Qinxin Du, Qiuping Wu, Quan Sun, Rongqiu Yang, Ruihang Miao, Ruixin Han, Ruosi Wan, Ruyan Guo, Shan Wang, Shaoliang Pang, Shaowen Yang, Shengjie Fan, Shijie Shang, Shiliang Yang, Shiwei Li, Shuangshuang Tian, Siqi Liu, Siye Wu, Siyu Chen, Song Yuan, Tiancheng Cao, Tianchi Yue, Tianhao Cheng, Tianning Li, Tingdan Luo, Wang You, Wei Ji, Wei Yuan, Wei Zhang, Weibo Wu, Weihao Xie, Wen Sun, Wenjin Deng, Wenzhen Zheng, Wuxun Xie, Xiangfeng Wang, Xiangwen Kong, Xiangyu Liu, Xiangyu Zhang, Xiaobo Yang, Xiaojia Liu, Xiaolan Yuan, Xiaoran Jiao, Xiaoxiao Ren, Xiaoyun Zhang, Xin Li, Xin Liu, Xin Wu, Xing Chen, Xingping Yang, Xinran Wang, Xu Zhao, Xuan He, Xuanti Feng, Xuedan Cai, Xuqiang Zhou, Yanbo Yu, Yang Li, Yang Xu, Yanlin Lai, Yanming Xu, Yaoyu Wang, Yeqing Shen, Yibo Zhu, Yichen Lv, Yicheng Cao, Yifeng Gong, Yijing Yang, Yikun Yang, Yin Zhao, Yingxiu Zhao, Yinmin Zhang, Yitong Zhang, Yixuan Zhang, Yiyang Chen, Yongchi Zhao, Yongshen Long, Yongyao Wang, Yousong Guan, Yu Zhou, Yuang Peng, Yuanhao Ding, Yuantao Fan, Yuanwei Lu, Yuanzhen Yang, Yuchu Luo, Yudi Zhao, Yue Peng, Yueqiang Lin, Yufan Lu, Yuling Zhao, Yunzhou Ju, Yurong Zhang, Yusheng Li, Yuxiang Yang, Yuyang Chen, Yuzhu Cai, Zejia Weng, Zetao Hong, Zexi Li, Zhe Xie, Zheng Ge, Zheng Gong, Zheng Zeng, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhiheng Hu, Zidong Yang, Zili Wang, Ziqi Ren, Zixin Zhang, Zixuan Wang

arXiv:2602.10604v2cs.CLcs.AI

TL;DR

Open-source models still trail closed-source frontiers in complex reasoning, while efficiency bottlenecks limit long-context agentic applications. Step 3.5 Flash addresses these challenges with a unified, scalable reinforcement-learning framework and efficient architecture, achieving competitive performance across reasoning and agentic benchmarks despite 11B active parameters.

  • Problem

    Open-source models remain behind closed-source frontier systems on complex reasoning, while efficiency bottlenecks hinder long-context agentic applications and resource-constrained deployment.

  • Method

    Step 3.5 Flash uses a unified post-training recipe that alternates domain specialization with global synthesis and introduces MIS-PO for stable, scalable off-policy reinforcement learning.

  • Results

    Step 3.5 Flash achieves competitive performance with leading frontier models across reasoning and agentic benchmarks, including 85.4% on IMO-AnswerBench.

  • Takeaways & Limitations

    The results indicate that Step 3.5 Flash substantially narrows the gap with frontier models while using 11B active parameters.

  • Takeaways & Limitations

    Step 3.5 Flash may experience reduced stability during distribution shifts, especially in specialized domains or long-horizon, multi-turn dialogues.

Abstract

from arXiv · show

We introduce Step 3.5 Flash, a sparse Mixture-of-Experts (MoE) model that bridges frontier-level agentic intelligence and computational efficiency. We focus on what matters most when building agents: sharp reasoning and fast, reliable execution. Step 3.5 Flash pairs a 196B-parameter foundation with 11B active parameters for efficient inference. It is optimized with interleaved 3:1 sliding-window/full attention and Multi-Token Prediction (MTP-3) to reduce the latency and cost of multi-round agentic interactions. To reach frontier-level intelligence, we design a scalable reinforcement learning framework that combines verifiable signals with preference feedback, while remaining stable under large-scale off-policy training, enabling consistent self-improvement across mathematics, code, and tool use. Step 3.5 Flash demonstrates strong performance across agent, coding, and math tasks, achieving 85.4% on IMO-AnswerBench, 86.4% on LiveCodeBench-v6 (2024.08-2025.05), 88.2% on tau2-Bench, 69.0% on BrowseComp (with context management), and 51.0% on Terminal-Bench 2.0, comparable to frontier models such as GPT-5.2 xHigh and Gemini 3.0 Pro. By redefining the efficiency frontier, Step 3.5 Flash provides a high-density foundation for deploying sophisticated agents in real-world industrial environments.

1. Introduction

Step 3.5 Flash targets the gap between open-source models and frontier systems by combining efficient MoE architecture with scalable post-training for reasoning and agentic tasks.

  • Open-source models still trail closed-source frontiers in complex reasoning and face efficiency bottlenecks in long-context agentic tasks.
  • The model uses 196B total parameters but activates only 11B per token, combining 3:1 sliding-window/full attention with MTP-3 to reduce long-context latency.
  • Stable training infrastructure and an improved Muon optimizer support training over 17.2T high-quality, diverse tokens while addressing large-scale MoE failure modes.
  • The unified post-training recipe alternates domain specialization with global synthesis and uses MIS-PO to stabilize off-policy RL for long-horizon reasoning and agentic behavior.
  • 85.4% on IMO-AnswerBench, 86.4% on LiveCodeBench-v6, 88.2% on τ2-Bench, 69.0% on BrowseComp, and 51.0% on Terminal-Bench 2.0 demonstrate competitive reasoning and agentic performance with 11B active parameters.

2. Architecture

Step 3.5 Flash co-designs attention, sparse MoE, and multi-token prediction to reduce agentic inference latency while preserving model capacity and benchmark quality.

  • Design Philosophy: Agentic workloads combine extensive context prefilling with prolonged interactive decoding, making wall-clock latency a central design constraint.
  • Attention: Hybrid attention mitigates quadratic long-context processing, while SWA preserves standard attention semantics and supports parallel speculative verification.
  • Sparse MoE and MTP: EP-group-balanced routing addresses rank-level load imbalance and stragglers, while MTP uses lightweight SWA and dense FFN heads for speculative decoding.
  • Sparse MoE Backbone: The backbone has 45 layers, 196B total parameters, 11B active parameters, and 288 routed experts plus one shared expert per MoE layer.
  • Architecture Ablations and Results: S3F1 minimizes normalized attention FLOPs but degrades quality; adding SWA query heads improves LongCtx from 27.5 to 28.2 and Sci from 42.4 to 44.0 with negligible additional cost.
  • Architecture Ablations and Results: S1F1 achieves the strongest LongCtx score, 29.6, but increases attention-side prefill/decode FLOPs by about 60% relative to S3F1+Head.
  • Architecture Ablations and Results: Head-wise gating raises average performance from 62.46 to 64.43, motivating its adoption as the default attention mechanism.

3. Infrastructure

The training infrastructure integrates large-scale parallelism, communication and kernel optimizations, selective checkpointing, and asynchronous telemetry to support efficient model development.

  • Step 3.5 Flash is trained on 4,096 NVIDIA H800 GPUs using high-bandwidth NVLink/NVSwitch within nodes and 8×200 Gbps RoCE links across nodes.
  • The Steptron framework unifies pre-training, post-training, and reinforcement learning under one PyTorch- and Megatron-LM-based engineering stack.
  • Training uses 8-way pipeline parallelism, 8-way expert parallelism, and ZeRO-1 data parallelism, with decoupled parallel groups for attention and MoE modules.
  • Fabric-aware scheduling and pipelined communication reduce iteration time by up to 5% despite concurrent data-parallel traffic across NVLink and RoCE fabrics.
  • Selective activation recomputation reduces peak memory by recomputing only the most memory-intensive submodules, while asynchronous metrics processing avoids main-loop synchronization overhead.

4. Pre-Training and Mid-Training

Step 3.5 Flash treats stability as a first-class requirement during large-scale sparse-MoE pre-training and mid-training. Continuous diagnostics expose optimizer, expert, and activation failure modes, while targeted mitigations stabilize training and internal expert behavior.

  • Training stability: A lightweight asynchronous metrics server with micro-batch-level logging provides fine-grained visibility into optimizer- and expert-level signals.The stack supports systematic identification and mitigation of recurring large-scale MoE failure modes.
  • Training stability: Three dominant instabilities are transient Muon-related loss spikes, expert-side collapse despite healthy router statistics, and localized activation blow-ups.Expert collapse can involve vanishing activations and stagnant or decaying parameter norms even when dispatch appears stable.
  • Training stability: Casting only the Polar Express iteration to float16 prevents the rare unrecoverable loss spikes observed with bfloat16 intermediate computations.The spikes were attributed to extreme intermediate outliers from cumulative addition error under certain update statistics.
  • Localized activation blow-up: Activation clipping stabilizes internal expert norms across layers, whereas weight clipping alone only delays recurrent outlier experts.The figure contrasts stable activation-clipping trajectories with catastrophic final-layer norm growth in the no-clipping baseline.
  • Localized activation blow-up: The max-to-median ratio of per-expert activation norms reliably reveals hidden instability that training loss fails to expose.Final layers can develop exploding maximum norms while median norms and training loss remain stable.

5. Post-Training

The post-training recipe alternates domain-specific specialization with synthesis into one generalist, then uses stability-oriented RL methods to improve reasoning and agentic performance across domains.

  • Unified Post-Training Recipe: The unified post-training recipe alternates domain-specific RL expert construction with self-distillation and scalable RL to preserve specialized capabilities in one generalist.Experts cover Math, Code, STEM, Tool-use, Long Context Understanding, Human Preference, and Agentic Reasoning.
  • Unified Post-Training Recipe: A two-stage SFT pipeline combines broad multi-domain coverage with difficulty-aware filtering, balancing, and out-of-distribution reasoning signals before RL.The second stage includes approximately 30k expert-level chemistry trajectories and synthetic arithmetic tasks.
  • MIS-Filtered Policy Optimization: MIS-PO filters off-distribution samples at token and trajectory levels, treating retained trajectories as effectively on-policy to reduce gradient variance during long-horizon RL.The method replaces continuous importance weighting with binary masks constrained by probability-ratio thresholds.
  • MIS-Filtered Policy Optimization: MIS-PO shows lower actor-gradient noise than PPO over approximately 5,000 training steps, indicating improved scalability under a trust-region constraint.Figure 5 also reports higher reward plateaus, accelerated convergence, and slower entropy decay for MIS-PO.
  • RL Stabilization: Truncation-aware value bootstrapping replaces zero rewards for truncated trajectories with final-state value estimates, stabilizing training at truncation rates as high as 20%.The method treats truncation as a horizon interruption rather than terminal task failure and is especially beneficial for competition-level benchmarks.
  • RL Stabilization: Routing Confidence serves as an MoE stability proxy: low confidence indicates routing uncertainty and brittleness, whereas high confidence supports robust off-policy training.The proxy is defined as the average probability mass of activated experts.
  • RL Outcomes: Step 3.5 Flash gains +3.2% on IMO-AnswerBench, +6.1% on CF-Div2-Stepfun-cpp, +10.6% on ARC-AGI-1, and +3.4% on HLEtext during RL training.The reported improvements accompany a steady rise in training rewards across the RLVR process.
  • Reward System: MetaRM-augmented GenRM outperforms vanilla GenRM by 0.5% - 3% on every benchmark in ablations over 200 RL training steps.GenRM supplies preference-based rewards for non-verifiable tasks, while MetaRM penalizes spurious reasoning.

6. Evaluations

Step 3.5 Flash is evaluated across general understanding, mathematics, coding, Chinese understanding, reasoning, and agentic benchmarks. Despite activating only 11B parameters, it remains competitive with substantially larger sparse baselines and frontier models across these tasks.

  • Evaluation Setup: Step 3.5 Flash is evaluated on general reasoning, mathematics, coding, Chinese understanding, reasoning, coding-agent, general-agent, and long-context benchmarks.The benchmark suite includes BBH, MMLU, GSM8K, MATH, HumanEval, MBPP, C-Eval, CMMLU, HLE, GPQA-Diamond, AIME2025, LiveCodeBench-v6, SWE-Bench, tau2-Bench, Terminal-Bench 2.0, GAIA, BrowseComp, and related tasks.
  • Pre-training Evaluation: 88.2 on BBH and 85.8 on MMLU show broad pre-training performance despite activating only 11B parameters out of 196B total.The model remains within 0.5 of the best BBH result while using fewer activated parameters than substantially larger sparse baselines.
  • Test-time Scaling: PaCoRe uses parallel reasoning trajectories and multi-round coordination to synthesize higher-fidelity solutions, with configuration ®𝐾= [4, 4, 4, 4].The method is applied to reasoning, general, and long-context benchmarks and is reported to yield significant gains across benchmarks.
  • Post-training Evaluation: Step 3.5 Flash performs strongly on reasoning-intensive benchmarks including AIME 2025, HMMT, IMO-AnswerBench, and LiveCodeBench-v6.The evaluation compares the model with strong open and closed baselines across reasoning, coding, agent, and long-context capabilities.

7. Limitations

The paper identifies longer generation trajectories and distribution-shift instability as current operational constraints. It also describes architectural mechanisms intended to reduce latency and maintain efficient information flow, while noting broader open-world agentic work remains ahead.

  • Token Efficiency: Step 3.5 Flash currently requires longer generation trajectories than Gemini 3.0 Pro to reach comparable quality.The authors plan to prune and compress thinking while maintaining competitive performance.
  • RL for Open-World Agentic Tasks: The paper frames reinforcement learning for intricate professional, engineering, and scientific tasks as a prerequisite for agents capable of genuine autonomy.Current results are described as competitive on academic agentic benchmarks, while open-world expert-level tasks remain a next frontier.
  • Operational Scope and Constraints: Step 3.5 Flash may experience reduced stability during distribution shifts, especially in specialized domains or long-horizon, multi-turn dialogues.Observed issues can include repetitive reasoning, mixed-language outputs, and inconsistencies in time and identity awareness.
  • Head-wise Gated Attention: Head-wise gated attention assigns each attention head an input-dependent scalar gate to modulate information flow across the hybrid layout with negligible computational overhead.The gate can be interpreted as an input-dependent sink mass in the softmax normalizer.
  • Efficiency Analysis: Increasing SWA query heads slightly raises FLOPs but has less impact on latency, while head-wise gating produces no noticeable FLOPs or latency difference.The low latency impact is attributed to SWA remaining IO-bound under a query-to-KV ratio of 12 and to gating being lightweight.

A.4. Pre-training Ablations Details

The ablations test hybrid attention layouts and sink-token alternatives under controlled MoE training settings. They show that additional SWA query heads recover much of the quality lost by a vanilla 3:1 layout, while head-wise gating improves the sink-token metric.

  • Ablation Design: The ablation suite isolates different hybrid attention layouts and sink tokens versus head-wise gated attention.The studies use controlled pre-training experiments under fixed or larger-scale MoE settings.
  • Hybrid Attention Layout: The hybrid-attention ablation uses a 30B-A3B MoE with approximately 1.4T pre-training tokens followed by supervised fine-tuning on a 0.1× downsampled dataset.Training includes warmup, main pre-training, cooldown, and long-context specialization stages.
  • Gate Versus Sink: The gate-versus-sink comparison pre-trains a 100B-A10B MoE model for approximately 250B tokens.This provides a larger-scale setting for comparing sink tokens with head-wise gating.
  • Hybrid Attention Layout: The vanilla 𝑆3𝐹1 layout underperforms full attention, while increasing SWA query heads improves MMLU-Pro by +3.7 and SFT Reasoning by +0.4.The higher-head configuration incurs a minor SFT Code regression of −0.6 while matching or exceeding full attention on several metrics.
  • Gate Versus Sink: Head-wise gated attention improves the sink token metric from 62.5 to 64.4, a gain of +1.9.The comparison is reported in the architectural ablation results using the evaluation protocols described for the suite.

B. Detail Analysis of Localized Activation Blow-up

The analysis attributes localized activation blow-up to fine-grained MoE experts specializing in frequent bi-grams, creating deterministic shortcut behavior and abnormal outputs. It motivates activation clipping and training safeguards as responses to these mechanisms.

  • Failure mechanism: Frequent bi-grams can trigger extreme first-token activations when fine-grained experts specialize without sufficient load-balancing regulation.The specialization creates a shortcut in which one expert dominates prediction and other network components no longer influence the output.
  • Failure mechanism: A pathological solution enlarges an expert outlier until RMSNorm makes the final representation effectively follow that outlier’s direction.The magnitude c grows without bound while the unit direction remains fixed, yielding RMSNorm(c·ĥ_outlier + h_others) = RMSNorm(ĥ_outlier).
  • Mechanistic explanation: SwiGLU can generate large expert outputs when gated and up projections become highly aligned on sparse inputs.This alignment concentrates computation in a limited number of rows from W_up, producing abnormal element-wise products and motivating activation clipping over weight clipping.
  • Data and training controls: StepCrawl combines model-guided filtering, crawl-budget balancing, and quality stratification to retain diverse, high-utility training data.The system processes approximately 1B pages per day, while later ablations favor accepting code documents with up to six heuristic violations.

C.5. Data Ablations Setting

The data-ablation setting evaluates curation choices under fixed-budget training, using a 30B-A3B MoE model and broad benchmark and compression-based assessments. The setup also specifies filtered SFT and RL data construction with verifiable rewards.

  • Ablation setup: Fixed-budget experiments evaluate curation strategies on broad benchmarks and held-out compression test sets.Compression metrics are treated as complementary signals that can more directly measure knowledge capacity.
  • Ablation setup: 30B-A3B provides stronger performance, stability, and full-scale trend fidelity than smaller proxy models in internal experiments.Smaller proxies are cheaper but underrepresent complex reasoning and long-tail memorization behavior.
  • SFT data: The SFT pipeline applies rule-based filtering, model-based filtering, and decontamination before producing 871k samples totaling 7.23B tokens.The resulting dataset distribution is reported in Table 3.
  • RL data: RL data spans competitive coding, STEM, and synthetic reasoning, with contamination controls and generated arithmetic, coding, and environment tasks.Filtering removes unsuitable prompts and samples batches across domains using predefined probabilities.
  • RL data: Verifier models assess STEM answer correctness, while coding sandboxes validate execution against test cases with soft rewards.The grading procedure checks final-answer equivalence and the correctness of the reasoning process.

D.2.3. RL Ablation Details

The RL ablations compare MIS-PO with GSPO across dense and MoE models and examine asynchronous training, tool-use evaluation, and test-time scaling. MIS-PO shows stronger stability and scalability for large-scale MoE off-policy RL.

  • MIS-PO versus GSPO: MIS-PO demonstrates higher sample efficiency and scalability than GSPO while constraining training–inference mismatch.GSPO fails to maintain consistent convergence in large-scale MoE RL, whereas MIS-PO remains stable across architectures.
  • MIS-PO versus GSPO: MIS-PO’s extended MoE run shows continuously increasing rewards, stable actor gradient norms, and controlled entropy.The raw gradient norm is displayed without smoothing or downsampling to expose optimization stability.
  • Asynchronous training: FullyAsync training tolerates approximately 20 steps of latency while avoiding the severe long-tail cost bottleneck of one-step off-policy training.About 5% of samples account for roughly 80% of generation cost in the early client–server framework.
  • Asynchronous training: FullyAsync maintains rising rewards and a controllable TIS truncation rate, indicating limited policy drift from asynchrony.Mid-training injection of task-relevant knowledge and tool-use priors also elicits gains.
  • Tool use: Step 3.5 Flash achieves the highest average tool-usage gain at 52.0 and leads on GAIA and xbench-DeepSearch.The metric separates inherent knowledge from the ability to exploit external information.
  • Test-time scaling: Python-based tool-integrated reasoning improves performance across challenging mathematics, STEM, and puzzle benchmarks.The evaluation uses a 100-turn limit across AIME 2025, HMMT 2025, IMO-AnswerBench, GPQA, HLEtext, and ARC-AGI-1.
  • Test-time scaling: PaCoRe extended to interactive tool loops yields significant gains over standard reasoning on GPQA and HLEtext.The approach preserves the standard LLM message interface while adding state-aware serialization.
  • Deployment: Cloud–edge collaboration reaches 57.0% on AndroidDaily Hard versus 40.0% for the edge-only baseline.The framework separates cloud reasoning from low-latency edge execution across resource-constrained deployments.

E.2.1. Reasoning benchmarks

The reasoning-benchmark section defines evaluation procedures across coding, mathematics, knowledge, and reasoning tasks, including official prompts and execution settings. It also cautions that the custom competitive-programming rating deviates from human contest conditions.

  • Coding benchmarks: CF-Div2-Stepfun evaluates 53 official Codeforces Division 2 problems from September 2024 through February 2025 with offline local grading.The benchmark targets fresh, competition-level programming problems.
  • Evaluation caveat: The simplified competitive-programming rating ignores submission-time penalties and may not be directly comparable to human participants.The authors use it as a standardized cross-model comparison despite its deviation from empirical contest conditions.
  • Coding benchmarks: LiveCodeBench-v6 uses its official evaluation method with a Python-programming prompt and starter-code answer format.Solutions must provide complete Python programs enclosed in the prescribed delimiters.
  • Reasoning benchmarks: MMLU-Pro and GPQA-Diamond use official evaluation methods with boxed multiple-choice outputs.HLEtext uses the official metric and an LLM-based judgment protocol with gpt-oss-120b.

E.2.2. Code Agent benchmarks

Step 3.5 Flash is evaluated as a software-engineering agent on SWE-Bench and Terminal-Bench under detailed execution and context constraints. The results show strong coding-agent performance, while ablations identify context limits and management as important factors.

  • SWE-Bench: 74.4% on SWE-Bench Verified and 67.4% on SWE-Bench Multilingual were achieved with the described containerized agent setup.The verified setting used 4GB memory, while the multilingual setting used 12GB; scores were averaged over four repeated runs.
  • Terminal-Bench 2.0: Terminal-Bench evaluations used remote containers with a 16GB memory limit and a six-hour total interaction and testing budget.Timed-out commands were automatically interrupted, preventing subsequent commands in the same round from executing.
  • SWE-Bench: 67/89 final pass@8 and 50.98% avg@8 were obtained on the reported software-engineering evaluation.Each task was executed across eight trials, and 88.6% of successful trajectories finished within 30 interactions.
  • SWE-Bench: Context management was necessary under constrained inference, with 9.41% of successful trajectories triggering history pruning.The evaluation preserved multi-round thinking history and applied pruning when the context window was exceeded.
  • Terminal-Bench 2.0: Limit 16k caused the largest Terminal-Bench performance drop because long reasoning could exhaust the token budget before terminal commands were produced.Disabling context management under the 16k limit further reduced performance to 45.22%.

E.2.3. General Agent benchmarks

The general-agent evaluation covers deep search, long-context understanding, tool use, and multiple standardized benchmarks. Step 3.5 Flash shows competitive search performance and uses explicit context-management strategies for BrowseComp evaluation.

  • Deep Search: BrowseComp with context management reported 69.0 using a discard-all strategy that reinitializes the operational loop after context thresholds are exceeded.The full BrowseComp dataset was evaluated with a maximum iteration constraint of 1000 steps.
  • Deep Search: Among single-agent context strategies on a 200-instance BrowseComp subset, Discard-all achieved 66.0% accuracy.The reported strategy comparison included Summary, Keep-first&last, Discard-all, and Multi-agent orchestration.
  • Research Evaluation: RESEARCHRUBRICS evaluates 101 domain-diverse research tasks using 20–43 expert-written criteria covering factual accuracy, reasoning soundness, and clarity.The leading commercial system, Gemini DeepResearch, achieved an aggregated score of 63.69.
  • Deep Search: 65.3 on the standardized ReAct evaluation surpassed the reported complex proprietary commercial baselines.Gemini 3.0 Pro scored 50.1 in the same standardized ReAct framework.
  • Evaluation Protocols: The general-agent evaluations use benchmark-specific protocols, including AUC@128k for MRCR-8needle to summarize retrieval performance across context lengths up to 131,072 tokens.The AUC calculation uses average retrieval accuracy across context bins and penalizes degradation as context length increases.

E.3.1. Data Analysis Benchmark

The paper introduces an expert-grounded Data Analysis Benchmark for evaluating end-to-end analytical problem solving under realistic business constraints. Step 3.5 Flash ranks second and remains close to GPT-5.2 while offering a favorable speed–capability trade-off.

  • Benchmark Construction: The benchmark targets realistic analytical ambiguity by distilling tacit expertise from ten senior data analytics leaders with more than 15 years of experience.Expert review aligned task definitions, data, reference solutions, and evaluation criteria for validity and reproducibility.
  • Benchmark Construction: 50 business-analysis items were constructed from expert interviews, each pairing a problem statement, CSV dataset, reference analysis, and weighted checklist rubric.Ten senior analytics leaders contributed cases, with an average of 26.9 rubric items per task.
  • Evaluation Method: The evaluation framework covers task execution, automated scoring, and report synthesis across code-based, research-oriented, and text-based analyses.Results are averaged over three identical runs to reduce stochastic variance and support cross-model comparison.
  • Results: 39.58% placed Step 3.5 Flash second overall, close to GPT-5.2 at 39.31%, while Claude Opus 4.5 ranked first.The paper also reports faster responses alongside solid analytical quality in the Claude Code environment.
  • Related Evaluation: The consulting benchmark scores models on Usefulness, Logic, Instruction Following, and Tone with equal 25% weighting across dimensions.A hybrid LLM-as-a-Judge system assigns GPT-5.2 to three dimensions and Claude Opus 4.5 to Tone.
Loading 2602.10604v2…