PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search

Chuzhan Hao, Wenfeng Feng, Guochao Jiang, Guofeng Quan, Guohua Liu, Yuewei Zhang

arXiv:2604.08124v1cs.AI

TL;DR

RL-based search agents can suffer from inefficient stochastic exploration and unstable training when guided mainly by outcome rewards. HiExp distills hierarchical experience from reasoning trajectories and aligns it with training, yielding substantial gains and cross-task, cross-algorithm generalization.

  • Problem

    RL-based search agents often rely on outcome-reward-guided stochastic exploration, which can produce inefficient reasoning trajectories and unstable training.

  • Method

    HiExp uses contrastive analysis, self-reflection, and multi-level clustering to transform rollout trajectories into hierarchical experience aligned with critic-free RL training.

  • Results

    HiExp achieves substantial performance gains over RL-based search agents and generalizes across task domains and RL algorithms.

  • Takeaways & Limitations

    Experience-aligned guidance transforms stochastic exploration into strategic, experience-driven search while improving policy-optimization effectiveness and stability.

  • Takeaways & Limitations

    Hierarchical-experience construction is semi-decoupled from policy optimization, so initially distilled guidance may not synchronize with evolving model capabilities.

Abstract

from arXiv · show

Reinforcement learning (RL) has become an effective approach for advancing the reasoning capabilities of large language models (LLMs) through the strategic integration of external search engines. However, current RL-based search agents often rely on a process of stochastic exploration guided by carefully crafted outcome rewards, leading to inefficient reasoning trajectories and unstable training. To address these issues, we propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training stability of search agents. Specifically, we extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge. By leveraging experience-aligned training, we effectively regularize stochastic exploration, evolving it into a strategic and experience-driven search process. Extensive evaluations on multiple complex agentic search and mathematical reasoning benchmarks demonstrate that our approach not only achieves substantial performance gains but also exhibits strong cross-task and cross-algorithm generalization.

1 Introduction

RL-based search agents use outcome-reward-guided stochastic exploration, but this can produce inefficient trajectories, weak strategic planning, and unstable training. HiExp addresses these issues by constructing hierarchical experience and aligning it with training, improving performance and generalization.

  • Motivation: RL-based search agents often struggle with strategic planning, efficient trajectories, and stable reward signals during multi-turn training.These difficulties are especially pronounced for small language models handling complex tasks.
  • Approach: HiExp extracts empirical knowledge from reasoning trajectories through contrastive analysis and multi-level clustering.The process identifies distinctions between successful and failed paths, then abstracts instance-level insights into broader strategies.
  • Approach: HiExp constructs hierarchical experience endogenously through self-reflection and agglomerative clustering over internal reasoning trajectories.The method synthesizes meta-knowledge without additional external factual information.
  • Approach: Experience-aligned training transforms stochastic exploration into strategic, experience-driven search while improving optimization effectiveness and stability.The alignment dynamically connects distilled experience with rollout-stage training.
  • Results: HiExp yields substantial performance gains and robust generalization across task domains and reinforcement-learning algorithms.The evaluations cover agentic search and reasoning settings.

2 Related Work

Related work develops increasingly flexible retrieval and reinforcement-learning approaches for complex reasoning and agentic search. These methods span manually designed retrieval workflows, agentic search, real-environment RL, and finer-grained reward design.

  • Retrieval-augmented reasoning: Early retrieval-augmented methods use branching, iterative retrieval, adaptive retrieval, and Chain-of-Thought to guide multi-step reasoning.These approaches rely on manually crafted workflows or intermediate-query refinement.
  • Agentic search and RL: Agentic search methods integrate external retrieval with reasoning to improve flexibility and generalization in complex scenarios.Search-o1 designs an agentic search workflow, while DeepSeek-R1 demonstrates gains from outcome-based reinforcement learning.
  • Agentic search and RL: Recent work applies reinforcement learning to real-world web search, searcher-generator decoupling, self-evolution, and fine-grained query-planning rewards.These efforts broaden how search agents are trained and guided.

3 Methodology

HiExp treats rollout trajectories as an endogenous knowledge base, distills them into hierarchical experiences, and retrieves guidance during critic-free RL training. Strategic experiences guide early planning, while case-based experiences support intermediate reasoning.

  • Framework overview: HiExp has two components: hierarchical experience construction and experience-aligned training.The first extracts and organizes trajectory knowledge; the second injects it into critic-free optimization.
  • Hierarchical Experience Construction: Self-Reflection Experience autonomously extracts, abstracts, and refines knowledge from internal reasoning trajectories without relying on static external factual sources.The framework broadens training-data value beyond annotated labels to the full exploration process.
  • Hierarchical Experience Construction: Contrastive distillation uses multiple rollout trajectories and self-reflection to identify key decisions and reasoning traps.The resulting case-based experience and summary description encode procedural knowledge from successful and unsuccessful paths.
  • Hierarchical Clustering: Multi-level clustering converts instance-specific experiences into strategic knowledge, reducing overfitting and retrieval noise from direct injection.Semantic embeddings and agglomerative clustering group lexically different but semantically equivalent experiences.
  • Experience-Aligned Training: Experience-aligned training embeds the current reasoning state, matches it against hierarchical experience knowledge, and retrieves the most similar experience.This retrieval supplies high-fidelity strategic priors during trajectory generation.
  • Experience-Aligned Training: Guidance changes with rollout stage: global strategic experiences support initial planning, while top-k granular heuristics support intermediate reasoning.A semantic threshold filters intermediate experiences and can support sub-query refinement.
  • Experience-Aligned Training: Conditioning trajectories on hierarchical experiences improves advantage quality and supports more stable policy updates in the RL objective.The objective incorporates both the search engine and hierarchical experience knowledge base.

4 Experiments

HiExp is evaluated across multi-hop question answering, mathematical reasoning, out-of-domain settings, and multiple RL algorithms, using local retrieval and online-search environments. The reported results show performance gains, including for small models and ablated experience configurations.

  • Experimental Settings: Experiments cover six multi-hop datasets, including in-domain and out-of-domain benchmarks, with dense retrieval over Wikipedia and Tavily web search.The evaluation compares HiExp with frontier LLMs, advanced RAG methods, and RL-based agentic search models.
  • Main Results: HiExp produces substantial gains across four multi-hop benchmarks and balances response comprehensiveness with accuracy by guiding reasoning paths.The method is presented as a universal, pluggable enhancement for different agentic frameworks and retrieval environments.
  • Generalization: HiExp generalizes across tasks, retrieval environments, and RL algorithms, with online web search providing diverse and dynamic context.The framework is evaluated beyond multi-hop question answering and across different algorithmic integrations.
  • Main Results: The trained 7B model performs on par with GPT-4.1 and surpasses larger models including DeepSeek-R1 and Qwen3-235B-A22B.These comparisons are reported for multi-hop reasoning benchmarks.

4.3 Further Analysis

HiExp improves performance across experience configurations, task domains, training settings, and RL algorithms. Its gains extend to out-of-domain and web-search environments, while self-distillation supports a self-contained training pipeline.

  • 60.6 in-domain F1 and 57.9 CEM are reached with full HEK training, up from GRPO baselines of 54.2 and 49.6.
  • Pattern-level E2 guidance combined with instance-level E1 corrections consistently outperforms higher-level E3 combined with E1 during training.
  • 38.2 out-of-domain F1 and 34.0 CEM with E2+E1 exceed GRPO baselines of 34.4 and 30.3.
  • HiExp generalizes beyond multi-hop question answering to mathematical reasoning, yielding a reported +17.4 gain over the base model during GRPO training.
  • HiExp provides gains across GRPO and GSPO, while web search supplies diverse dynamic context and supports hierarchical planning and grounding.
  • Self-distillation slightly outperforms strong-teacher distillation, supporting a scalable pipeline without external large-model supervision.

4.4 Training Stability Analysis

HiExp stabilizes search-agent training by steering exploration toward higher-value reasoning paths and reducing optimization noise. Its experience guidance also provides a concrete blueprint for multi-hop constraint decomposition.

  • HiExp produces a more rapid and stable ascent in valid reward than stochastic exploration during training.
  • HiExp significantly reduces variance in advantages and gradients compared with the baseline.
  • More consistent advantage estimates suppress gradient noise and stabilize model updates.
  • Strategy-based E2 supplies a logic blueprint by instructing the model to resolve a temporal anchor before searching for the target information.

5 Conclusions

HiExp constructs hierarchical experience from internal reasoning trajectories and aligns that experience with reinforcement-learning rollouts. The framework improves performance and optimization stability across tasks and RL algorithms.

  • HiExp uses self-reflection and agglomerative clustering to synthesize hierarchical meta-knowledge from internal reasoning trajectories.
  • Dynamic alignment with reinforcement-learning rollouts transforms stochastic exploration into strategic, experience-guided search.
  • Evaluations report substantial performance gains and robust generalization across diverse task domains and reinforcement-learning algorithms.

Limitations

HiExp currently constructs hierarchical experience separately from subsequent policy optimization. This static separation may leave guidance out of sync with the policy’s evolving capabilities.

  • The semi-decoupled design isolates experience construction from policy optimization, so initial guidance may not synchronize with later policy capabilities.
  • The paper identifies a dynamic closed-loop system coupling experience construction and model training as a future direction.

A Implementation Details

The evaluation uses six multi-hop datasets, separating in-domain training-related data from out-of-domain generalization tests. Implementation also includes fixed-size sampling for two large dev sets and specified training, rollout, and hardware settings.

  • 1,000 examples are randomly sampled from the 2WikiMultiHopQA and HotpotQA dev sets using fixed random seed 42.The paper states that subset performance is nearly identical to the full dev-set performance.
  • The retrieval corpus combines supporting passages from five multi-hop datasets with one million randomly sampled documents from the December 2018 Wikipedia dump.The dump contains over 21 million passages, and multilingual-e5-base is used as the retriever.
  • The final training set combines 8,148 selected HotpotQA and 2WikiMultiHopQA examples with 8,000 randomly sampled MuSiQue training examples.
  • Training runs for 2 epochs with train_batch_size 16, learning rate 1e-6, rollout batch size 8, and temperature 1.0.Prompt and response maximum lengths are 512 and 8192; KL regularization and clipping ratio are 1e-3 and 0.2.

B Prompt Examples

The implementation prompt for LLM-as-Judge scoring uses stronger language models to assess multi-hop QA responses more accurately, while adding computational overhead to training.

  • Larger judge language models provide more accurate response judgments in multi-hop question answering.The resulting evaluation can be incorporated into the training process but introduces additional computational overhead.

C Quantitative Analysis

The paper reports implementation prompts, hierarchical experience construction procedures, dataset-complexity analysis, and computational-cost measurements. These materials describe contrastive trajectory processing and clustering, while also documenting evaluation and prompt-based scoring components.

  • Computational Cost: The offline experience pipeline consists of contrastive distillation over pre-sampled trajectories followed by hierarchical clustering; on MusiQue, about 7,000 trajectories require roughly 1 hour for contrastive distillation with Qwen-7B/72B/Max under parallel inference.
  • Hierarchical Experience Construction: The hierarchical experience construction algorithm initializes an atomic experience set and builds a hierarchical experience knowledge base from training data, rollouts, rewards, and a maximum depth.
  • Hierarchical Experience Construction: Reward-based contrastive splitting separates positive and negative rollout outcomes before an LLM extracts success-critical insights.
  • Hierarchical Experience Construction: Agglomerative clustering performs semantic grouping at each level, while LLM summarization induces patterns and termination indicates that global principles have been reached.
  • Dataset Complexity: Figure 4 summarizes query-complexity distributions across five multi-hop QA datasets.
  • Evaluation Materials: The Frames analysis is presented as a quantitative study of efficient reasoning, while the system prompt specifies trajectory generation through environment interaction during training and inference.
  • Prompt Examples: The supplied prompt materials include vanilla RAG, contrastive distillation, hierarchical clustering, and LLM-as-Judge scoring prompts.
  • Prompt Examples: The LLM-as-Judge prompt scores response accuracy from 0 to 5 according to coverage of reference-answer key points and responsiveness to the question.
Loading 2604.08124v1…