PaperlayerView exact revision on arXiv

Source-linked AI summary

Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration

Bowei He, Minda Hu, Zenan Xu, Hongru Wang, Licheng Zong, Yankai Chen, Chen Ma, Xue Liu, Pluto Zhou, Irwin King

arXiv:2602.03647v1cs.AIcs.CL

TL;DR

Search-integrated agents need finer-grained supervision because trajectory-level rewards do not adequately distinguish effective reasoning from retrieval-driven errors. Search-R2 combines an Actor with a selectively corrective Meta-Refiner, hybrid process-and-outcome rewards, and joint optimization; it reports consistent gains across benchmarks and model scales. The framework is supported by a mixture-policy analysis and targeted cut-and-regenerate refinement.

  • Problem

    Trajectory-level rewards provide insufficient supervision for intermediate retrieval and reasoning decisions, creating a multi-scale credit assignment problem.

  • Method

    Search-R2 uses an Actor to generate search-integrated reasoning trajectories and a Meta-Refiner to diagnose localized failures and repair them through cut-and-regenerate, with joint optimization and a hybrid reward.

  • Results

    Search-R2 consistently outperforms strong RAG and RL-based baselines across seven benchmarks and model sizes, improving final-answer accuracy and search-process quality.

  • Takeaways & Limitations

    Selective refinement and evidence-density supervision provide a framework for making search-integrated reasoning more reliable while retaining valid trajectory prefixes.

Abstract

from arXiv · show

Search-integrated reasoning enables language agents to transcend static parametric knowledge by actively querying external sources. However, training these agents via reinforcement learning is hindered by the multi-scale credit assignment problem: existing methods typically rely on sparse, trajectory-level rewards that fail to distinguish between high-quality reasoning and fortuitous guesses, leading to redundant or misleading search behaviors. To address this, we propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention, with both components jointly optimized during training. Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories, and a Meta-Refiner, which selectively diagnoses and repairs flawed steps via a 'cut-and-regenerate' mechanism. To provide fine-grained supervision, we introduce a hybrid reward design that couples outcome correctness with a dense process reward quantifying the information density of retrieved evidence. Theoretically, we formalize the Actor-Refiner interaction as a smoothed mixture policy, proving that selective correction yields strict performance gains over strong baselines. Extensive experiments across various general and multi-hop QA datasets demonstrate that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales, achieving superior reasoning accuracy with minimal overhead.

1 Introduction

Search-R2 addresses multi-scale credit assignment in search-integrated reasoning by combining an Actor with a Meta-Refiner that selectively repairs flawed trajectory steps. Its hybrid supervision, theoretical analysis, and experiments support improved reasoning accuracy and search-process quality.

  • Problem: Trajectory-level rewards fail to supervise query formulation, retrieval timing, information filtering, and deduction, causing credit misattribution and brittle reasoning chains.The problem is especially acute when an early irrelevant query propagates errors through the rest of a trajectory.
  • Framework: Search-R2 decomposes generation into an Actor that produces tool-using reasoning trajectories and a Meta-Refiner that diagnoses localized failures.The framework preserves valid reasoning prefixes while repairing flawed steps through targeted intervention.
  • Framework: The Meta-Refiner uses cut-and-regenerate refinement to truncate a trajectory at its error point and regenerate the subsequent reasoning.The illustrative example redirects reasoning from the incorrect entity Aguinaldo to the correct one, Quezon.
  • Training: The Actor and Meta-Refiner are jointly optimized with a hybrid reward combining answer correctness and evidence-information density.The process reward distinguishes trajectories supported by useful retrieved context from trajectories that succeed by chance.
  • Analysis: The Meta-Refiner is characterized as a mixture policy, with theoretical conditions under which selective correction strictly improves performance over baseline sampling.The framework is jointly optimized during training rather than treating refinement as a separate post-processing stage.
  • Results: Search-R2 achieves state-of-the-art performance across seven benchmarks, improving both final-answer accuracy and the quality of the underlying search process.The reported evaluation spans models of different sizes and includes strong RAG and RL-based baselines.

2 Related Works

Prior work studies search-integrated reasoning and multi-turn reinforcement learning, while emphasizing persistent challenges from sparse rewards and difficult credit assignment. Search-R2 is presented against this background as a framework for improving search-agent reasoning.

  • Search-integrated reasoning: Prior research applies search-augmented reasoning to multi-hop question answering, deep research, and web-based decision making.These tasks require agents to query external information during problem solving.
  • Search-R2: Search-R2 is framed as an Actor–Refiner framework that separates initial reasoning-chain generation from trajectory assessment and causal correction.This decomposition targets global reasoning coherence and local search quality simultaneously.
  • Reinforcement learning: Multi-turn decision making remains difficult because sparse rewards make credit assignment across intermediate decisions challenging.In search-integrated agents, query formulation and retrieval timing are often evaluated only through final task outcomes.
  • Reinforcement learning: Prior approaches include dense reward shaping and learned reward models for improving supervision in multi-turn agent learning.The passage situates these methods within broader efforts to address credit assignment.

3 Methodology

Search-R2 separates search-integrated reasoning into Actor generation and Meta-Refiner intervention, then jointly trains both with hybrid outcome and process rewards. Its refiner evaluates coherence, localizes failures, and regenerates only flawed suffixes within an iterative accept-or-repair procedure.

  • Framework: Search-R2 decomposes reasoning into an Actor that generates initial chains and a Meta-Refiner that assesses and causally corrects flawed trajectories.The Actor invokes the search engine during generation, while the Meta-Refiner targets local search or reasoning failures.
  • Actor: The Actor generates a trajectory, issues tagged search calls when needed, incorporates retrieved results, and continues until answering or reaching a step limit.The operational cycle alternates reasoning, query execution, evidence insertion, and resumed generation.
  • Meta-Refiner: The Discriminator accepts globally coherent trajectories, while rejected trajectories are sent to the Trimmer for localized repair.Acceptance uses a threshold τ; otherwise, refinement begins.
  • Meta-Refiner: The Trimmer preserves the valid prefix, cuts at the earliest deviation, and regenerates the suffix with the base policy.This cut-and-regenerate strategy addresses failed queries or logical errors without discarding the entire trajectory.
  • Hybrid Reward: The hybrid reward combines exact-match outcome correctness with a process reward measuring the density of useful retrieved information.The overall reward is R(y) = r_outcome(y) · (1 + r_process(y)), gating process supervision by successful outcomes.
  • Joint Optimization: GRPO jointly optimizes the shared Actor and Meta-Refiner using augmented reasoning-and-refinement traces sampled from the mixture distribution.Group-normalized hybrid rewards train the policy to balance generation and correction from interaction outcomes.

4 Formalization

The formalization models Actor–Refiner correction as a smoothed mixture policy and decomposes its reward improvement into selection and trimming mechanisms. It shows that effective correction depends on identifying poor trajectories and selecting cut-points with high regeneration value.

  • 4 Formalization: The theory formalizes Actor–Refiner collaboration as a smoothed mixture policy and decomposes expected reward gain into distinct mechanisms.The framework quantifies contributions from discrimination and refinement phases.
  • 4.1 Performance Analysis: The central quantity is ΔJ = J_meta − J_base, comparing Meta-Refiner performance with the base Actor policy.J_base averages rewards under π_l, while J_meta averages rewards under the refined distribution q.
  • 4.1 Performance Analysis: The refined distribution combines accepted base trajectories with regenerated trajectories selected after rejection.The acceptance probability and trimmer transition distribution determine how rejected samples are corrected.
  • 4.1 Performance Analysis: The performance decomposition identifies selection precision and correction volume gain as the two drivers of improvement.Selection precision measures whether the discriminator preserves better trajectories and exposes poor ones for refinement.
  • 4.2 Decomposing the Correction Volume Gain: The trimming gain ΔJ_trim is decomposed into a covariance term representing trimming skill and a mean baseline-gain term.The decomposition separates cut-point attribution from the average benefit of regeneration.
  • 4.2 Decomposing the Correction Volume Gain: Trimming skill is positive when the trimmer concentrates probability on cut-points with the greatest regeneration gains.This corresponds to locating critical turning points rather than trimming randomly.
  • 4.2 Decomposing the Correction Volume Gain: When the baseline gain is approximately zero, maximizing correction gain relies almost entirely on precise cut-point selection.This condition emphasizes attribution ability as the dominant source of trimming improvement.
  • 4.2 Decomposing the Correction Volume Gain: Table 2 reports main results across seven datasets, distinguishing in-domain and out-of-domain evaluations and marking best and second-best scores.The table compares Search-R2 with baselines under the stated backbone settings.

5 Experiments

Search-R2 is evaluated across seven general and multi-hop QA benchmarks against retrieval, fine-tuning, and reasoning baselines. It delivers consistent accuracy gains, with targeted refinement improving complex-task performance and training efficiency at modest overhead.

  • Experiment Setup: Evaluation covers NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle, using Exact Match after GRPO training.Training uses the union of NQ and HotpotQA training splits, with 512 prompts per step and five rollouts per prompt.
  • Performance Comparison: Search-R2 achieves a 16.1% EM gain over Search-R1 despite using Qwen2.5-7B versus Search-R1's Qwen3-8B backbone.Average EM also rises from 40.4 to 50.8 when scaling Search-R2 from 7B to 32B.
  • Performance Comparison: Search-R2 improves 2WikiMultiHopQA by 5.5 points and Bamboogle by 11.4 points (+25.3% relative gain).The authors attribute larger gains on complex tasks to correcting early retrieval and reasoning deviations before errors propagate.
  • Ablation Study: The Meta-Refiner produces the largest ablation gain (+11.1% on Qwen2.5-7B), while process reward and joint optimization further improve performance.The full Search-R2 configuration with joint optimization achieves the highest accuracy.
  • Sensitivity to the Maximum Revision Limit: Maximum revision = 4 reaches average score 50.9, essentially matching Search-R2 with one revision at 50.8.Absolute EM gains diminish from 0.9 points for revisions 1→2 to 0.3 points for 3→4, motivating max revision = 1 as the default.
  • Efficiency Analysis: Training time increases by only 5.06% on average, falling to 2.43% for the 32B model, while inference adds no latency.The accuracy-per-time ratio rises from 1.78 at 7B to 4.69 at 32B.

6 Conclusions

The paper concludes that Search-R2 addresses retrieval-noise fragility through Actor–Refiner collaboration and dynamic cut-and-regenerate correction. The framework is presented as a route toward more reliable search-integrated reasoning behavior.

  • Conclusions: Search-R2 is designed to mitigate language-model fragility when retrieval introduces misleading context.The conclusion frames retrieval noise as a source of error propagation loops in standard approaches.
  • Conclusions: Jointly optimized Actor–Refiner collaboration interrupts error propagation through dynamic cut-and-regenerate correction of reasoning trajectories.The Meta-Refiner enables real-time trajectory correction while preserving the framework's search-integrated setting.
  • Conclusions: The findings highlight active refinement as important for more reliable agent behavior in search-integrated reasoning.The conclusion presents this as a path toward reliability rather than a claim of universal deployment performance.

B Proof for Performance Decomposition of Meta-Refiner

The proof verifies the Meta-Refiner mixture distribution and derives its expected reward by separating accepted and refined trajectory contributions. Covariance identities then yield the performance-gain decomposition relative to the base actor.

  • Normalization Check: The proof first verifies that the Meta-Refiner distribution q(y|x) integrates to 1.This establishes normalization before evaluating expected reward.
  • Expected Reward Derivation: Expected reward Jmeta is derived by integrating R(y) over the mixture's accepted and refined components.The derivation explicitly treats the rejected component through a change in integration order.
  • Expected Reward Derivation: The derivation applies the covariance identity to separate average reward terms from covariance terms involving acceptance and trimming behavior.It defines average trimmed reward before applying the identity a second time.
  • Performance Gain: Combining the components and subtracting Jbase produces the final expected performance-gain expression.The proof labels this final step as the gain after synthesizing terms A and B.

C Proof for Decomposition of Trimming Strategy

This section decomposes Search-R2’s improvement into selection precision, trimming skill, and intervention volume, then describes joint optimization with GRPO and meta-actions.

  • The total performance gain is decomposed into discriminator judgment, Meta-Refiner localization, and intervention frequency.
  • Selection precision measures whether the discriminator preserves samples whose existing reward exceeds the expected value of correction.
  • Trimming skill measures alignment between selected cut-points and regeneration gains, indicating whether the refiner targets high-value correction points.
  • Intervention volume is the probability mass assigned to trimming and determines the opportunity space available for refinement.
  • GRPO jointly optimizes the Actor and Meta-Refiner by treating reasoning tokens and meta-actions as an augmented execution trace.

E Supplementary Implementation Details

The supplementary material documents hardware, training configuration, resources, and additional training-dynamics and rollout-volume comparisons for Search-R2.

  • Experiments use 8-node GPU clusters with dual-socket AMD EPYC processors, NVMe storage, high-speed interconnects, and distributed dataset storage.
  • Training uses a unified search-integrated reasoning dataset, 4096-token prompts, 3000-token responses, SGLang rollouts, and five trajectories per prompt.
  • The supplementary resources list Qwen and DeepSeek models, the E5 retriever, a 2018 Wikipedia dump, and training and parallelism software.
  • Figure 5 tracks EM scores from 0 to 300 steps at 50-step intervals across seven datasets and three models, with convergence near 300 steps.
  • Search-R2 with n = 5 and one revision consistently outperforms Search-R1 with n = 10, testing targeted refinement against regenerating every trajectory.
  • Table 8 incrementally adds the Meta-Refiner, process reward, and joint optimization modules to the original Search-R1 framework.

H Detailed Ablation Study Results

The detailed ablation material evaluates trajectory quality across seven datasets using six complementary rubrics beyond final-answer correctness and compares Search-R2 with Search-R1.

  • Trajectory quality is evaluated with six rubrics covering evidence use, retrieval usefulness, search efficiency, timing, coherence, and uncertainty handling.
  • Evidence groundedness measures whether claims and intermediate conclusions are supported by retrieved information rather than unsupported or hallucinated content.
  • Information density measures the usefulness of retrieved content relative to total search results, distinguishing task-relevant evidence from noisy retrieval.
  • Non-redundancy efficiency and query timing quality assess search-budget use, query repetition, query formulation, and the timing of searches.
  • Across six rubrics and seven datasets, Search-R2 outperforms Search-R1 more frequently than the reverse on most datasets.
  • Trajectory coherence measures consistency among hypotheses, evidence, and conclusions, while uncertainty handling measures appropriate responses to incomplete or ambiguous information.

I.2 Detailed Results

The detailed results provide trajectory-quality comparisons and reproducibility materials, including the standard multi-turn search rollout procedure.

  • Table 9 compares Search-R2 and Search-R1 across six trajectory-quality dimensions on seven datasets using paired win counts.
  • Search-R2 outperforms Search-R1 more frequently than the reverse on most datasets, supporting higher-quality search-integrated reasoning trajectories.
  • Table 10 supplies the prompt used for trajectory-quality comparison to improve reproducibility.
  • The standard rollout takes input x, policy πθ, search engine Λ, and budget B, then returns a final response ˆy.
  • At each step, the policy generates text until a search, answer, or end marker; search queries retrieve information, which is appended before generation continues.

K Local Process Reward Implementation Details

The Local Process Reward measures the information density of retrieved evidence by counting useful, non-redundant collections relative to search actions. Outcome gating applies this process reward only when the final answer is correct, combining process quality with task success.

  • Reward definition: The process reward quantifies retrieved-evidence information density to encourage efficient, relevant, and non-redundant searches.It is computed as a ratio involving useful collections and total search actions.
  • Collection utility: A collection is useful when it provides information or clues that help identify the correct answer, even partially.Completely irrelevant collections are not useful.
  • Collection utility: Duplicate information without new insights does not count as useful, even when the information is otherwise relevant.The evaluation compares collections for incremental evidence rather than relevance alone.
  • Collection utility: The evaluator counts the collections marked useful and reports that total as the basis for the process-reward calculation.The prompt supplies the question, correct answer, and tool-response collections for evaluation.
  • Outcome gating: Outcome gating prevents reward hacking by applying the process reward only when the final answer is correct.The total reward combines this gated process signal with a binary Exact Match outcome score.
Loading 2602.03647v1…