PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-DeepResearch Technical Report

Chen Hu, Haikuo Du, Heng Wang, Lin Lin, Mingrui Chen, Peng Liu, Ruihang Miao, Tianchi Yue, Wang You, Wei Ji, Wei Yuan, Wenjin Deng, Xiaojian Yuan, Xiaoyun Zhang, Xiangyu Liu, Xikai Liu, Yanming Xu, Yicheng Cao, Yifei Zhang, Yongyao Wang, Yubo Shu, Yurong Zhang, Yuxiang Zhang, Zheng Gong, Zhichao Chang, Binyan Li, Dan Ma, Furong Jia, Hongyuan Wang, Jiayu Liu, Jing Bai, Junlan Liu, Manjiao Liu, Na Wang, Qiuping Wu, Qinxin Du, Shiwei Li, Wen Sun, Yifeng Gong, Yonglin Chen, Yuling Zhao, Yuxuan Lin, Ziqi Ren, Zixuan Wang, Aihu Zhang, Brian Li, Buyun Ma, Kang An, Li Xie, Mingliang Li, Pan Li, Shidong Yang, Xi Chen, Xiaojia Liu, Yuchu Luo, Yuan Song, YuanHao Ding, Yuanwei Liang, Zexi Li, Zhaoning Zhang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Jiansheng Chen, Jing Li, Xiangyu Zhang, Yibo Zhu

arXiv:2512.20491v4cs.CL

TL;DR

Deep Research benchmarks and task formulations often emphasize retrieval rather than open-ended research requiring planning, verification, and synthesis. Step-DeepResearch combines atomic-capability data and progressive training with ADR-Bench for Chinese research evaluation. The 32B model achieves strong reported benchmark performance, while the paper identifies remaining robustness, factuality, and report-auditability challenges.

  • Problem

    Existing benchmarks and search-oriented tasks do not fully capture open-ended research, which requires intent decomposition, planning, tool use, cross-source verification, and structured synthesis.

    Source: §1 · p. 2

  • Method

    Step-DeepResearch combines atomic-capability data synthesis and progressive training from agentic mid-training through SFT and RL, alongside ADR-Bench for application-driven Chinese evaluation.

    Source: §1 · p. 2

  • Results

    The 32B model achieves 61.42 rubric compliance on ResearchRubrics and is reported to perform at a level comparable to OpenAI and Gemini DeepResearch, while experiments show state-of-the-art results among medium-sized models and competitiveness with proprietary models in specific domains.

    Source: §1 · p. 2, §8 · p. 23

  • Takeaways & Limitations

    The authors conclude that medium-scale models can achieve expert-level Deep Research capabilities through refined training.

    Source: §1 · p. 2

    Tool-use robustness remains insufficient, factuality is difficult under noisy or fragmented evidence, and report readability and auditability need improvement.

    Source: §8 · p. 23

Abstract

from arXiv · show

As LLMs shift toward autonomous agents, Deep Research has emerged as a pivotal metric. However, existing academic benchmarks like BrowseComp often fail to meet real-world demands for open-ended research, which requires robust skills in intent recognition, long-horizon decision-making, and cross-source verification. To address this, we introduce Step-DeepResearch, a cost-effective, end-to-end agent. We propose a Data Synthesis Strategy Based on Atomic Capabilities to reinforce planning and report writing, combined with a progressive training path from agentic mid-training to SFT and RL. Enhanced by a Checklist-style Judger, this approach significantly improves robustness. Furthermore, to bridge the evaluation gap in the Chinese domain, we establish ADR-Bench for realistic deep research scenarios. Experimental results show that Step-DeepResearch (32B) scores 61.4% on Scale AI Research Rubrics. On ADR-Bench, it significantly outperforms comparable models and rivals SOTA closed-source models like OpenAI and Gemini DeepResearch. These findings prove that refined training enables medium-sized models to achieve expert-level capabilities at industry-leading cost-efficiency.

1 Introduction

Deep Research requires iterative planning, tool use, cross-source verification, and synthesis, unlike closed-form search; benchmark formulations can miss these real-world demands. Step-DeepResearch combines atomic-capability data, progressive training, and an application-driven Chinese benchmark, with reported strong results from a 32B model.

1 Introduction

  • The framework synthesizes data for atomic capabilities, trains progressively from agentic mid-training through SFT to RL, and introduces ADR-Bench for application-driven Chinese research evaluation.ADR-Bench spans commercial research, policy analysis, and software engineering, using Elo-style ratings and multidimensional quality criteria.Source: §1 · p. 2
  • 61.42 rubric compliance on ResearchRubrics places the 32B model at a level comparable to OpenAI and Gemini DeepResearch, while expert Elo ratings exceed several larger models and Deep Research systems.The report also describes the system as the most cost-effective Deep Research system, citing low deployment and inference costs.Source: §1 · p. 2

2.1 Workflow-based Implementation of Deep Research

  • Workflow-based systems use external orchestration, but predefined workflows can be complex and ReAct approaches may lack depth in specialized scenarios.The cited systems include plan-and-execute and hierarchical multi-agent designs.Source: §2.1 · pp. 2–3

2.2 End-to-End Optimization of Research Capabilities

  • Existing end-to-end research systems emphasize search efficacy, while systematic training for long-horizon reasoning, cross-validation, report composition, and cost-efficient performance remains a challenge.The passage identifies optimization of model size and inference cost while maintaining performance as an industry challenge.Source: §2.2 · p. 3

2.3 Deep Research Related Evaluation Benchmarks

  • Existing Deep Research benchmarks often lack comprehensive task coverage or depth, and retrieval-focused benchmarks such as BrowseComp and HLE do not capture the openness and multidimensionality of industrial research scenarios.ADR-Bench is introduced to address the evaluation gap for Chinese Deep Research scenarios driven by real user demands.Source: §2.3 · p. 3

3 Data Strategy: Constructing Atomic Capabilities

The data strategy treats Deep Research as decisions over a compact space of transferable atomic actions rather than raw-token exploration. It synthesizes and filters training examples across planning, information seeking, reflection and verification, and report generation.

3 Data Strategy: Constructing Atomic Capabilities

  • The strategy reframes training from next-token prediction to next-atomic-action decisions, using transferable high-level action abstractions to reduce raw-token exploration demands.The paper identifies a trade-off between retaining key skills and keeping planning logic clear.Source: §3 · pp. 3–4

3.1 Capability I: Planning & Task Decomposition

  • Planning data is reverse-engineered from high-quality reports into realistic project tasks and hindsight plans, then filtered for trajectory consistency with those plans.The source documents include technical reports, academic surveys, and financial research reports.Source: §3.1.1 · p. 4, §3.1 · p. 4

3.2.1 Graph-based Synthesis

  • Graph-based synthesis expands low-degree seed entities into bounded subgraphs, verifies each edge through search, then generates multi-hop questions and answers from the verified structure.Expansion uses BFS to build 10–40-node subgraphs and truncates high-degree super nodes to limit semantic drift.Source: §3.2.1 · p. 4, §3.2 · p. 4

3.2.2 Multi-document-based Synthesis

  • Multi-document synthesis walks Wiki-doc hyperlinks without knowing the target in advance and reverse-generates query-answer pairs from collected paths; tasks solvable by QwQ-32b are excluded as simple.The difficulty filter uses QwQ-32b under the default ReAct framework.Source: §3.2.2 · p. 4, §3.2.3 · p. 5

3.3.1 The Error-Reflection Loop

  • Reflection data follows expert generation, result verification, and multi-turn retry, retaining successful corrected trajectories and using cross-source comparison to filter unreliable information.The retry process iterates up to three times when an initial result is incorrect.Source: §3.3.1 · p. 5, §3.3 · p. 5

3.3.2 Deep Verification Workflow

  • A multi-agent verification workflow decomposes claims, plans source needs, retrieves and cross-validates evidence, replans, and produces supported conclusions; a judge filters for logical consistency.The workflow traces are built from cleaned paragraph–judge-result pairs from desensitized real data.Source: §3.3.2 · p. 5
  • Verification trajectories are filtered by checking whether each conclusion is logically self-consistent with its evidence, targeting factual accuracy and logical rigor.The verification point, conclusion, and evidence are judged together as a triplet.Source: §3.3.2 · p. 5

3.4 Capability IV: Report Generation

  • Report-generation training separates expert style and content depth in mid-training from plan adherence, user-specific instruction following, and formatting in SFT.Mid-training uses screened human reports; SFT filters plan-deviating samples and regenerates poorly formatted reports from the final trajectory state.Source: §3.4 · p. 5, §3.4.1 · p. 6, §3.4.2 · p. 6

4 Training Pipeline

The pipeline builds agentic research behavior progressively: mid-training teaches atomic capabilities, SFT composes them into full trajectories, and RL optimizes interactive task performance using rubric-based feedback. It expands from pure-text learning to long-context tool use, with synthesis and judging strategies designed for reliable, efficient training signals.

4 Training Pipeline

  • The three-stage pipeline assigns capability expansion to mid-training, behavior composition and alignment to SFT, and task-level optimization to RL.Training proceeds sequentially on a 32-billion-parameter base model.Source: §4 · p. 6

4.1 Stage 1: Agentic Mid-training

  • Mid-training progresses from 32K pure-text capability learning to 128K real-world tasks that require retrieval, planning, and explicit tool use.The first stage includes academic active reading without explicit tool invocation; the second adds web interaction and multi-tool collaboration.Source: §4.1 · pp. 6–7
  • +10.88% on FRAMES, +2.30% on TriviaQA, and +1.26% on SimpleQA are the maximum observed gains during 32K mid-training; RL reward also rises over training.The mid-training curves remain unconverged at 150B tokens.Source: Figure 2 · p. 7, §4.1 · p. 7, Figure 3 · p. 11

4.2 Stage 2: Post-training Supervised Fine-tuning

  • SFT composes atomic capabilities through end-to-end Deep Search and Deep Research trajectories, including multi-hop answers and open-ended research reports.Deep Research trajectories cover intent understanding, planning, cross-verification, and report generation with citation and attribution requirements.Source: §4.2 · p. 8
  • SFT cleaning retains concise successful searches and selected tool-error trajectories, while removing repetitive loops and reinforcing citation alignment.Deep Search uses a “correct and shortest” selection rule; error cases include reflection and correction after empty results or tool failures.Source: §4.2 · pp. 8–9

4.3 Stage 3: Reinforcement Learning

  • RL addresses the limits of teacher-only trajectories by training through real tool interactions and rubric-based feedback on research reports.Report assessment covers task decomposition, planning, tool strategy, evidence validation, and report generation.Source: §4.3 · p. 9

4.3.1 RL Data Synthesis

  • Two-step reverse synthesis generates fine-grained rubrics alongside a hidden task summary, then constructs a task query from those rubrics and filters samples for alignment.An independent judge checks consistency among the summary, synthesized task, and rubric requirements.Source: §4.3.1 · p. 9

4.3.2 Reward Design

  • A trained Rubrics Judge reduces the cost of large-scale scoring, while binary reward mapping treats positive criteria as satisfied only when fully met and penalizes any deviation on negative criteria.The approach is intended to remove noise associated with the intermediate “partially satisfied” category.Source: §4.3.2 · pp. 9–10

5 System Architecture

The system implements Deep Research as a single-agent ReAct loop that plans, invokes tools, and validates returned information. Its infrastructure supports persistent task state, efficient long-report editing, multimodal input, and authoritative information retrieval.

System Architecture

  • The ReAct loop cycles through planning and reflection, tool execution, and feedback with cross-validation of new results against prior context.The loop supports conflict resolution and filtering falsehoods while building an evidence chain.Source: §5 · pp. 11–12

Authoritative Enhanced Information Acquisition

  • Retrieval draws on over 600 curated authoritative sites and a library of over 20 million high-quality items, with paragraph-level retrieval and authority-aware ranking.When relevance scores are comparable, ranking prioritizes authoritative sources.Source: §5 · p. 12

Knowledge Management & File Operations

  • Patch-based editing changes only a report fragment and reduces output-token costs by over 70% while improving complex-edit success rates.The tool applies atomic updates using fuzzy matching and minimal anchor context.Source: §5 · p. 12
  • A summary-aware local-storage strategy limits returned context and lets the agent retrieve raw tool results from files as needed.When results exceed a preset threshold, only high-density summaries enter the context while raw data is persisted locally.Source: §5 · p. 12

Interactive Execution & Multimodal Perception

  • The interaction framework combines sandboxed terminal use, resilient browsing, and document, audio, and image perception tools.The browser suppresses image feedback when consecutive screenshots show negligible visual change.Source: §5 · pp. 12–13

6 ADR-Bench: A Custom Deep Research Benchmark

ADR-Bench is designed around real-world Deep Research queries and uses distinct evaluation approaches for broad user tasks and specialized professional reports. Its criteria assess completeness, depth, requirement fitness, and readability, while its general-domain process acknowledges limits on reliably verifying accuracy in long-form open-ended reports.

6.1 Query Composition

  • ADR-Bench organizes real-use queries into nine domains and separates general tasks from professional Law and Finance tasks.It uses 20 queries in each professional domain and 10 in each of the seven general domains.Source: §6.1 · pp. 13–14

6.2 Evaluation Framework

  • General-domain reports receive blind comparative human review, while professional Finance and Law reports are assessed with expert-designed, cross-validated rubrics.General reviewers judge paired reports across completeness, depth, requirement fitness, and readability; professional tasks use relatively constrained real-world prompts.Source: §6.2 · p. 14
  • The framework combines user-perceived quality comparisons for general tasks with rubric-based monitoring of professional capability.The authors identify declining discriminative power and rising evaluation costs as performance converges as challenges for future evaluation.Source: §6.2 · p. 14

6.3 Evaluation Criteria

  • General-domain criteria cover information completeness, content depth, requirement fitness, and readability.The criteria consider coverage of key aspects, evidence and insight, intent and constraints, and clear organization and presentation.Source: Information Completeness · p. 15, Requirement Fitness · p. 15

Requirement Fitness

  • Readability criteria ask whether reports use suitable visual aids, logical structure, and accessible explanations for complex material.Evaluators also consider headings, summaries, and segmentation that make reports scannable.Source: Requirement Fitness · p. 15
  • Because accuracy is difficult and costly to validate reliably in open-ended long-form reports, evaluators verify only task-relevant key conclusions or data points that materially affect quality.Professional-domain rubrics instead use questions with relatively limited solution spaces and shared core elements.Source: Requirement Fitness · p. 15

2. Verifiability

The supplied passages only indicate that table content continues across pages; they do not state substantive claims about verifiability.

2. Verifiability

  • The passages mark continued table content but provide no substantive verifiability claim.No further section content is represented here.Source: §2 · pp. 15–16

5. Alignment

The exploratory evaluation work finds Deep Research difficult to assess because standards are non-unique and evaluation is subjective. It favors comparative approaches for more stable assessments, while identifying practical trade-offs in rubric design, segmentation, and evaluator resources.

6.4 Exploratory Attempts

  • Roughly 6–12 domains was a reasonable segmentation range, balancing query coverage against overlap and helping direct queries to suitable experts.The authors note that perfect segmentation is unnecessary.Source: §6.4 · p. 16
  • Evaluation dimensions should be independent, explicit, and executable, although coupling can persist and manual verification of long reports is labor-intensive and often unreliable.The authors recommend focusing on one dimension at a time to address coupling.Source: §6.4 · p. 16
  • Model-generated rubrics diverged from human perceptions and remained difficult to evaluate, while substantial rewriting and alignment were still needed despite the goal of reducing human workload.The authors also report that gold-to-ideal report gaps shape rubric quality and can bias human revisions toward the initial draft.Source: §6.4 · p. 16
  • 0–10 overall ratings varied substantially, while 0–3 scores failed to distinguish reports; comparative evaluation was more stable, higher-confidence, and more efficient.The authors describe comparative evaluation as preferable when Deep Research lacks unique standards and unified verification criteria.Source: §6.4 · pp. 16–17
  • 150 versus 50 comparisons are required for a full Elo leaderboard versus ordinary evaluation on 10 queries and five models; ordinary evaluation is recommended for measuring performance gaps.Sampling or limiting Elo opponents may reduce confidence while still costing more than ordinary comparisons.Source: §6.4 · p. 17
  • Senior domain experts with LLM understanding who are willing to evaluate are scarce, and collaboration with them incurs significant time and financial costs.Source: §6.4 · p. 17

7 Experiments and Analysis

Experiments compare Step-DeepResearch with open-source and commercial systems on ResearchRubrics and ADR-Bench, including human-preference and finance-and-law evaluations. Detailed analyses report strengths across evaluation dimensions and domains, alongside targeted case studies and refinements for recurring research-task problems.

7.1 Experimental Setup

  • The experiments use ResearchRubrics and ADR-Bench, including 70 general-domain human side-by-side comparisons and 40 finance-and-law items scored by LLM evaluators.The evaluated ReAct models include Step-DeepResearch, initialized from Qwen2.5-32B-Base and developed through mid-training, SFT, and reinforcement learning.Source: §7.1 · p. 17

7.2 Evaluation Results

  • 61.42 on ResearchRubrics makes Step-DeepResearch the leading single-agent system, 5.25 points above Kimi-k2-thinking (56.17), and second overall behind Gemini DeepResearch (63.69).It also surpasses OpenAI DeepResearch and Kimi-Researcher in the overall leaderboard comparison.Source: Figure 6 · p. 18, §7.2 · p. 18
  • 47 combined wins and ties (67.1%) against Gemini and MiniMax, alongside a 30-win, 21-loss mid-training ablation, support strong human-evaluation performance on ADR-Bench.The ablation compares Step-DeepResearch with its non-mid-trained version; the human evaluation uses pairwise preferences.Source: §7.2 · p. 18, Figure 7 · p. 19
  • Gemini led finance-and-law evaluations; Step-DeepResearch, Kimi, and OpenAI formed the next tier, with domain-specific training cited for Step-DeepResearch’s competitive position.Fatal professional misconceptions receive a zero score, so this evaluation reflects error costs as well as hit rates.Source: §7.2 · pp. 18–19, Table 6 · p. 19

7.3 Detailed Analysis

  • On ResearchRubrics categories, Step-DeepResearch scores 54.5 on Implicit Criteria and 72.0 on Explicit Criteria, leads Citation Quality with 57.0, and leads Communication Quality with 58.2.Its Instruction Following score (64.9) trails Kimi-Researcher (66.7).Source: §7.3 · pp. 19–20, Figure 8 · p. 20
  • Across domains, Step-DeepResearch beats Kimi-Researcher in AI & ML (64.8 vs 57.4), Historical Analysis (65.8 vs 61.4), and Technical Documentation (64.6 vs 53.1), while trailing Gemini in STEM and Philosophy.The report notes these results were achieved without domain-specific data augmentation and identifies room for improvement in high-order reasoning.Source: §7.3 · p. 20, Figure 9 · p. 20
  • Detailed analyses describe relevant task-specific presentation, comprehensive and structured information, and more consistent general analysis and planning, while noting slight gaps in specialized professional reasoning.The general-domain assessment uses pairwise human preferences; Figure 10 presents its fine-grained results.Source: §7.3 · pp. 20–21, Figure 10 · p. 21

7.4 Case study

  • A representative benchmark case study reports that the mid-trained model improves overall report quality, instruction following, and breadth and depth of information exploration over its non-mid-trained version.The authors associate these gains with planning and information-seeking atomic capabilities incorporated during mid-training.Source: §7.4 · p. 21, Table 7 · p. 22

8 Conclusion and Future Work

The paper concludes that atomic-capability data and end-to-end training improve a 32B model’s research proficiency, with ADR-Bench introduced to assess practical usability. It also identifies remaining challenges in tool robustness, factuality, and report auditability, and outlines future directions.

8 Conclusion and Future Work

  • The paper presents atomic-capability data and end-to-end training as a staged route to improved research proficiency, and introduces ADR-Bench for practical-usability evaluation.The reported paradigm progresses from agentic mid-training to post-training.Source: §8 · pp. 22–23
  • The authors identify brittle tool use under API variation, anomalous returns, and complex long-chain tool compositions, as well as persistent factuality and report-readability challenges.Noisy or fragmented evidence can lead to “plausible but unprovable” inferences; reports also need clearer evidence mapping.Source: §8 · p. 23
  • Future work proposes multi-agent collaboration, greater adaptability in dynamic partially observable settings, and training objectives that include factual consistency and traceable citations.The proposed directions include consensus mechanisms and preference learning with process-based supervision.Source: §8 · p. 23

9 Contributors and Acknowledgements

The acknowledgements list contributors in four categories: research and data, evaluation, infrastructure, and project sponsors.

Contributors and Acknowledgements

  • Research and Data contributors include Chen Hu, Haikuo Du, Heng Wang, Lin Lin, and the other listed team members.The paper states contributors within each category are listed alphabetically by first name.Source: §9 · p. 23
  • Evaluation contributors include Binyan Li, Dan Ma, Furong Jia, Hongyuan Wang, and the other listed team members.The paper states contributors within each category are listed alphabetically by first name.Source: §9 · p. 23
  • Infrastructure contributors and project sponsors are separately acknowledged, including Aihu Zhang and Binxing Jiao, respectively.Both categories include additional named contributors and sponsors.Source: §9 · p. 23

Appendix

The appendix gives positive- and negative-rubric evaluation prompts and includes a domain-specific example prompt for an industry-cycle review.

ResearchRubrics Judger System Prompt

  • The positive rubric prompt scores criterion satisfaction as 0.0, 0.5, or 1.0 and directs evaluators to assess evidence and completeness.It instructs evaluators to base judgments only on explicit document content.Source: A. ResearchRubrics Judger System Prompt · p. 26
  • The negative rubric prompt scores the presence of a defined flaw: 0.0 means absent, 0.5 partial, and 1.0 fully present.Under its inverted quality logic, a higher score indicates a worse document on that flaw.Source: A. ResearchRubrics Judger System Prompt · p. 26

B. Examples

  • The example prompt asks for a 2019–2025E live-commerce market review, identifying turning points, causes, risks, growth years, opportunities, and their preconditions.It is framed as an industry-cycle review using multiple years of data and cause analysis.Source: Finance & Business · p. 27, Table 8 · p. 27
Loading 2512.20491v4…