Source-linked AI summary
SmartSearch: Process Reward-Guided Query Refinement for Search Agents
Tongyu Wen, Guanting Dong, Zhicheng Dou
TL;DR
Search agents often overlook the quality of intermediate queries, allowing inaccurate queries to produce unexpected retrieval results. SmartSearch combines Dual-Level process rewards, selective query refinement, and a three-stage curriculum; across six tasks, it consistently surpasses baselines while improving search efficiency and query quality.
Problem
Search-agent research often overlooks intermediate query quality, although inaccurate queries can produce unexpected retrieval results and limit information-seeking performance.
Method
SmartSearch combines Dual-Level process rewards, selective query refinement, and a three-stage curriculum progressing from imitation through alignment to generalization.
Results
Across four knowledge-intensive tasks and two web exploration tasks, SmartSearch consistently surpasses existing baselines and improves search efficiency and query quality.
Takeaways & Limitations
The framework focuses search-agent training on improving intermediate query quality through process-reward guidance and refinement.
Abstract
from arXiv · showhide
Large language model (LLM)-based search agents have proven promising for addressing knowledge-intensive problems by incorporating information retrieval capabilities. Existing works largely focus on optimizing the reasoning paradigms of search agents, yet the quality of intermediate search queries during reasoning remains overlooked. As a result, the generated queries often remain inaccurate, leading to unexpected retrieval results and ultimately limiting search agents' overall effectiveness. To mitigate this issue, we introduce SmartSearch, a framework built upon two key mechanisms: (1) Process rewards, which provide fine-grained supervision for the quality of each intermediate search query through Dual-Level Credit Assessment. (2) Query refinement, which promotes the optimization of query generation by selectively refining low-quality search queries and regenerating subsequent search rounds based on these refinements. To enable the search agent to progressively internalize the ability to improve query quality under the guidance of process rewards, we design a three-stage curriculum learning framework. This framework guides the agent through a progression from imitation, to alignment, and ultimately to generalization. Experimental results show that SmartSearch consistently surpasses existing baselines, and additional quantitative analyses further confirm its significant gains in both search efficiency and query quality. The code is available at https://github.com/MYVAE/SmartSearch.
1 Introduction
SmartSearch targets the overlooked quality of intermediate search queries, using process rewards, query refinement, and a three-stage curriculum to improve search agents. Experiments across six tasks report stronger overall performance, search efficiency, and query quality than existing baselines.
- Intermediate search-query quality is often overlooked, although inaccurate queries can produce unexpected retrieval results and derail search trajectories.
- SmartSearch uses Dual-Level Credit Assessment to score query novelty with rules and query usefulness with model-based evaluation.The mechanism produces numerical scores and textual feedback for subsequent refinement.
- Query refinement selectively revises low-quality search rounds and regenerates subsequent rounds from the refined queries.
- Experiments on four knowledge-intensive tasks and two web exploration tasks show SmartSearch consistently surpasses baselines and improves search efficiency and query quality.
- The three-stage curriculum progresses from imitation to alignment to generalization, progressively internalizing query-quality improvement.
2 Related Works
Related work presents LLM-based search agents as a response to limitations of static model knowledge, while identifying insufficient attention to intermediate retrieval quality and sparse reinforcement-learning feedback.
- LLM-based search agents autonomously and iteratively invoke external tools to address challenging knowledge-intensive problems.
- Existing search-agent research has advanced through prompt engineering, fine-tuning, and reinforcement learning.
- Current training paradigms may prioritize information utilization, causing information-retrieval abilities to stagnate.
- Final-outcome-only rewards provide sparse feedback for multi-round search, producing insufficient intermediate guidance and unstable, inefficient policy optimization.
3 Preliminaries
The preliminaries describe a ReAct search agent that iteratively produces thoughts, actions, and observations, alongside GRPO-based policy optimization and composite reward design.
- ReAct Search Agent: At each iteration, the ReAct agent generates a thought, queries an external search tool, and receives Top-K document fragments as its observation.
- ReAct Search Agent: The policy produces each thought and action from the complete history of prior context.
- Policy Optimization: GRPO optimizes the policy during Query-Aware Policy Optimization, with augmented rollout and reward modules targeting intermediate query quality.
- Reward Design: Standard agentic-RL rewards combine outcome correctness and format correctness, while process rewards add fine-grained intermediate feedback.
- Reward Design: Composite rewards aggregate step-wise process rewards with the final outcome reward through an aggregation function that varies across works.
4 Our Method
SmartSearch improves search-agent query generation through process rewards, selective query refinement, and a three-stage curriculum that progresses from imitation to alignment and generalization.
- Process Reward for Assessing Query Quality: SmartSearch assesses query quality through Dual-Level Credit Assessment, combining rule-based novelty checks with model-based usefulness evaluation.The usefulness evaluation considers query-intent necessity and whether retrieved results provide the expected answer.
- Process Reward for Assessing Query Quality: Process rewards produce numerical scores and textual feedback that guide query refinement and select trajectories with high-quality search processes.The overall score combines novelty and usefulness assessments, while textual feedback concatenates their explanations.
- Process Reward for Assessing Query Quality: The novelty score identifies redundant queries by measuring document overlap between the current and previous search rounds.The overlap count is compared against a threshold, with the indicator function identifying shared documents.
- Process Reward-Guided Query Refinement: Query refinement identifies low-quality queries, regenerates later search steps from refined queries, and supports comparative-data construction and rollout.A lightweight SFT-tuned model uses the original query, trajectory history, and diagnostic feedback to generate each refinement.
- Query-Oriented Training Framework: The curriculum progresses from quality-screened SFT, through DPO-based query generation alignment, to RL-based query-aware policy optimization.Stage 1 retains only trajectories with correct final answers and high-quality intermediate queries; process rewards also enter the final RL reward design.
5 Experiments
SmartSearch is evaluated across knowledge-intensive and web exploration benchmarks using answer correctness, search efficiency, and search query quality metrics. It consistently outperforms baselines, with analyses attributing gains to process rewards, query refinement, and curriculum training.
- Experimental Setup: SmartSearch is evaluated on four knowledge-intensive datasets and two web exploration tasks, using EM, F1, Search Efficiency, and Search Quality measures.The benchmarks include 2WikiMultihopQA, HotpotQA, Bamboogle, Musique, GAIA, and WebWalker.
- Main Results: SmartSearch consistently surpasses existing approaches across four datasets and shows strong generalization to web search, with nearly 5% higher average F1 across GAIA and WebWalker.The model is trained on Wikipedia-based local search but is evaluated successfully in open-web settings.
- Main Results: Process-supervised RL methods outperform outcome-only RL methods by around 5% in average EM and F1, reflecting the value of fine-grained supervision for intermediate steps.Outcome-only rewards provide sparse feedback and can produce unstable optimization.
- Main Results: SmartSearch improves average EM and F1 by more than 7% over other process-supervised RL methods by explicitly optimizing intermediate search-query quality.The result links query-quality optimization with improved overall search-agent performance.
- Ablation Study: Query-quality filtering enables Stage 1 to achieve superior performance with only 60% of the training data, while ablations identify query refinement as especially important.The ablations cover the two core mechanisms across the three curriculum-learning stages.
- Quantitative Analyses: SmartSearch achieves the highest Search Quality and Search Efficiency, reducing ineffective or failed search rounds through more precise intermediate queries.Search Quality is supported by both Perfect Rate and Partial Rate, while Search Efficiency measures the effectiveness of retrieval relative to search calls.
- Quantitative Analyses: Using a stronger teacher model raises average F1 by less than 1% but increases per-sample processing time by nearly five times, revealing an effectiveness-efficiency trade-off.This supports using a lightweight student model for LLMeval and LLMrefine.
6 Conclusion
SmartSearch optimizes intermediate search-query quality through process rewards and query refinement, supported by a three-stage curriculum from imitation through alignment to generalization. Experiments across four challenging benchmarks show consistent gains over existing baselines in search efficiency and query quality.
- SmartSearch combines process rewards with query refinement to optimize intermediate search queries.Process rewards provide fine-grained supervision through Dual-Level Assessment, while refinement selectively improves low-quality queries and regenerates later search steps.
- A three-stage curriculum guides the agent from imitation and alignment to generalization.The curriculum is designed to help the agent progressively internalize query-quality improvement under process-reward guidance.
- Experiments across four challenging benchmarks show SmartSearch consistently surpasses existing baselines.
- Quantitative analyses confirm significant gains in search efficiency and query quality.
A Prompt Templates
The prompt templates define the agent’s search interaction, query evaluation, and query refinement procedures, including structured inputs, outputs, and refinement constraints. They require query evaluation to assess intent-result alignment and query refinement to produce a more actionable query when needed.
- Search-agent prompt: The search-agent prompt structures reasoning, search queries, results, and final answers with explicit XML-style tags.It instructs the agent to use a Wikipedia search tool during step-by-step problem solving and encloses the final exact answer in boxed LaTeX format.
- Query evaluation: The evaluation prompt assigns a binary query score when the intent is necessary and the result completely contains the intended information.The intended entity and the entity found in the result must also match exactly.
- Query refinement: The query-refinement prompt supplies the user question, search context, and query issues before presenting refinement guidance.
- Query refinement: A refined query replaces the current-round query and must not rely on that round’s result.
- Query refinement: When query intent is unreasonable, refinement should produce a more necessary and actionable intent.
B Implementation Details
SmartSearch is implemented as a three-stage training pipeline followed by constrained inference. The stages use supervised fine-tuning, preference optimization, and reinforcement learning with progressively specialized data and rollout settings.
- Training stages: The imitation stage fine-tunes Qwen2.5-3B-Instruct through SFT using trajectories sampled by ARPO-14B.Training uses a 7e-6 learning rate for 3 epochs with a 16384-token maximum input length.
- Training stages: The alignment stage applies DPO to trajectories from the SFT model as positive and negative samples.It uses LoRA fine-tuning, a 7e-6 learning rate, 3 epochs, a 10000-token maximum input length, batch size 32, and BF16 precision.
- Training stages: The policy-optimization stage uses RL on challenging questions unresolved after four sampling trials.The final SmartSearch model is trained with 8 rollouts per sample, batch size 64, PPO mini-batch size 16, and at most 5 tool calls per rollout.
- Publication information: The paper reports the SIGIR ’26 publication venue and schedule for SmartSearch.
- Inference: Inference permits a maximum output length of 16384 tokens and up to 10 tool calls.
C Case Study
The case study illustrates how SmartSearch forms precise queries, uses only necessary searches, scores query quality, and refines failed queries. Refinement changes a trajectory that deviates from the correct path into one that retrieves the desired information and reaches the correct answer.
- SmartSearch output: SmartSearch first plans the user question, forms an accurate search intent, and generates a precise query that retrieves the desired information.The example also shows the model using its internal knowledge to generate only necessary searches.
- SmartSearch output: Using only necessary search queries enhances SmartSearch’s search efficiency.
- Process reward: Process rewards provide numerical scores and textual explanations for each query.The explanations identify why low-quality queries perform poorly and guide subsequent refinement.
- Query refinement: Query refinement replaces a low-quality query and regenerates subsequent steps from the refined query.The initial trajectory deviates after failing to retrieve expected information, whereas the refined trajectory retrieves the desired information and reaches the correct answer.