Source-linked AI summary
Meta-Reinforcement Learning with Self-Reflection for Agentic Search
Teng Xiao, Yige Yuan, Hamish Ivison, Huaisheng Zhu, Faeze Brahman, Nathan Lambert, Pradeep Dasigi, Noah A. Smith, Hannaneh Hajishirzi
TL;DR
Agentic search RL often relies on sparse outcome rewards, creating difficult credit assignment and limiting exploration. MR-Search uses in-context meta-RL with explicit cross-episode self-reflection and turn-level advantages to adapt search across episodes. It reports 9.2% to 19.3% average relative improvements over strong baselines across multiple benchmarks.
Problem
Sparse and delayed outcome rewards in agentic search produce ambiguous credit assignment and ineffective exploration, while external process rewards require costly annotations and add bias risks.
Method
MR-Search conditions each episode on prior trajectories and explicit reflections, using critic-free multi-turn RL with turn-level grouped advantages for fine-grained credit assignment.
Results
9.2% to 19.3%: MR-Search achieves these average relative improvements over strong baselines across multiple multi-hop QA benchmarks.
Takeaways & Limitations
MR-Search transforms independent search attempts into a progressively informed process and improves exploration without relying on external process reward models.
Takeaways & Limitations
The study does not evaluate long-form benchmarks and focuses on a fixed Wikipedia search tool rather than heterogeneous tool environments.
Abstract
from arXiv · showhide
This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.
1 Introduction
MR-Search addresses sparse-reward agentic search by using meta-RL, sequential self-reflection, and cross-episode context to improve exploration. It combines this framework with turn-level credit assignment and reports relative gains over strong baselines.
- Agentic search RL methods primarily optimize final-answer correctness, receiving sparse rewards that make intermediate decisions difficult to credit.
- External process rewards and LM judges add annotations, reward-hacking and bias risks, plus computational overhead.
- MR-Search conditions later search episodes on prior trajectories and explicit self-reflections, turning independent attempts into progressively informed exploration.
- MR-Search applies critic-free multi-turn RL with grouped relative advantages estimated at the turn level for localized credit assignment.
- 9.2% to 19.3%: MR-Search achieves these average relative improvements over strong baselines across multiple multi-hop QA benchmarks.
2 Related Work
Related work spans RL-based agentic search, in-context meta-RL, self-reflection, and test-time scaling. MR-Search connects these directions through cross-episode reflection for progressively targeted search.
- RL for Agentic Search: RL-based search agents train language models to reason, interact with search environments, receive feedback, and refine strategies toward task rewards.
- RL for Agentic Search: MR-Search differs from independent search attempts by using explicit cross-episode reflection to drive progressively targeted exploration.
- Meta-Reinforcement Learning: In-context meta-RL uses histories from initial exploration episodes to guide later exploitation episodes in robotics and game domains.
- LLMs with Self-Reflection: Self-reflection research studies iterative refinement through intrinsic feedback, including prompting-based and finetuning-based approaches.
- Test-time Scaling: Test-time scaling includes parallel independent sampling and sequential refinement conditioned on previous attempts.
3 MetaSearch: Meta-Reinforcement Learning for Agentic Search
MR-Search formulates agentic search as cross-episode meta-reinforcement learning, using self-reflection to condition later episodes on accumulated context. It combines this process with turn-level relative advantages for finer credit assignment and reports improved performance as reflection turns increase.
- Background: Sparse and delayed outcome rewards create ambiguous credit assignment and ineffective exploration for RL-based search agents.The standard interaction consists of iterative thought-action-observation cycles ending in a final answer evaluated by a verifier.
- Meta-RL Framework: MR-Search organizes a meta-episode as N sequential episodes, propagating trajectories and reflections from earlier episodes through the accumulated context.The context length grows linearly with reflection steps; retaining only the immediately preceding episode also works well empirically.
- Empirical Results: MR-Search substantially outperforms sequential-reflection and parallel-sampling baselines, with performance increasing as the number of turns grows.Figure 3 compares MR-Search with Search-R1-S and Search-R1-P using the most frequent answer across generated trajectories.
- Meta-RL Framework: MR-Search models each complete search trajectory as an episode followed by explicit self-reflection, allowing subsequent episodes to condition on prior attempts.This changes exploration from isolated episodes into a sequential process that consolidates cross-episode knowledge.
- Policy Optimization: The training procedure samples groups of meta-episodes, computes per-episode rewards with leave-one-out estimation, and propagates discounted relative rewards to earlier turns.The leave-one-out baseline is independent of the current action, supporting unbiased turn-level advantage estimation.
- Policy Optimization: The policy optimization objective broadcasts each turn’s advantage to its tokens, masks tool-output tokens, and captures both global trajectory quality and local step effectiveness.MR-Search uses a clipped surrogate objective rather than separate value functions as in PPO, while remaining critic-free.
4 Experiments
Experiments evaluate MR-Search on search-based QA benchmarks and analyze its optimization, exploration, training, test-time scaling, and inference behavior. MR-Search generally outperforms outcome-reward and external-process-reward baselines while benefiting from multi-turn reflection and exploration.
- Experimental Setup: Experiments cover seven QA datasets plus the more complex long-horizon ASearcher dataset, using Qwen2.5-3B-Base and Qwen2.5-7B-Base.Evaluation reports Exact Match for the last valid prediction from one sampled trajectory per question.
- Main Results: 9.2% and 19.3% relative improvements over Search-R1 are achieved on average with Qwen2.5-7B-Base and Qwen2.5-3B-Base, respectively.MR-Search also outperforms methods using external models for process rewards, including StepResearch and PPRM.
- Main Results: 10.2% and 9.5% relative improvements over Search-R1 are reported on the ASearcher datasets, which require longer-horizon multi-turn search.Figure 4 presents ASearcher test performance alongside reward and search-frequency training curves.
- Ablation Study: The multi-turn RL algorithm consistently outperforms PPO and MT-GRPO with episode turn-level rewards, while removing discounting substantially degrades performance.The ablation attributes the optimization advantage to leveraging dense reflection feedback and reports convergence to poor local optima when γ = 0.
- Test-time Scaling: Additional reflection turns yield steep performance gains for MR-Search but only marginal gains for single-turn Search-R1 with reflection.MR-Search is evaluated beyond its three-turn training setting by appending the full interaction history at each test-time turn.
- Further Analysis: MR-Search selectively revises answers using newly retrieved evidence, preserves already accurate answers, and calls the search engine more frequently than Search-R1.The training analysis reports stable convergence and dynamically adjusted search calls according to task complexity.
5 Conclusions
MR-Search addresses sparse-reward agentic search by using explicit self-reflection to structure cross-episode exploration. Its turn-level grouped advantage formulation supports fine-grained, critic-free credit assignment, and experiments show consistent gains over outcome-only RL baselines.
- Explicit self-reflection after each interaction episode enables structured cross-episode exploration without external process reward models.
- MR-Search conditions each episode on prior trajectories and reflections, turning independent attempts into a progressively informed search process.
- A turn-level grouped advantage formulation provides unbiased, fine-grained credit assignment while keeping the training procedure critic-free.
- MR-Search consistently outperforms outcome-only RL baselines across diverse benchmarks.
Limitations
The study identifies scope and evaluation boundaries for MR-Search. It does not test long-form benchmarks, heterogeneous tool environments, or large-scale frontier-model RL training.
- MR-Search is not evaluated on long-form benchmarks, where verifying progress and final correctness remains challenging.
- The study focuses on agentic search with a fixed Wikipedia search tool rather than multiple heterogeneous tools.
- Scaling MR-Search to large agentic RL runs and studying Meta-RL scaling properties with frontier models are left for future work.
Ethics Statement
The paper states that its societal implications are those broadly associated with machine learning systems and reports no additional ethical concerns beyond those commonly linked to large language models.
- The research may benefit practical applications requiring efficient and reliable reasoning through more effective training of language-model-based agents.
- The paper reports no additional ethical concerns beyond those commonly associated with large language models.
A.1.1 The Details of Datasets
The evaluation uses general and multi-hop question-answering datasets spanning open-domain, evidence-based, entity-centric, and synthetic multi-turn settings. The datasets vary substantially in size and construction.
- NQ contains real anonymized, aggregated search queries, with 79,168 training samples and 3,610 test samples.
- TriviaQA is a reading-comprehension dataset with question-answer-evidence triples, and its test set contains 11,313 samples.
- PopQA contains 14,267 entity-centric question-answer pairs generated from Wikidata knowledge tuples.
- HotpotQA is a Wikipedia-based multi-hop benchmark with sentence-level supporting evidence and 7,405 test samples.
- 2WikiMultiHopQA tests inference across multiple evidence pieces and contains 7,405 test question-answer pairs.
- MuSique enforces genuine multi-step reasoning and contains 2,417 test question-answer pairs.
- Bamboogle contains 125 multi-hop test questions whose supporting evidence is available in Wikipedia despite being difficult for popular search engines.
- ASearcher is a synthetic Wikipedia-based multi-turn dataset filtered to 14k samples and split into 90% training and 10% evaluation sets.
A.1.2 The Details of Baselines
The baselines span inference-only systems and fine-tuning methods with or without step-level supervision, including search-enhanced reasoning and RL-based search agents.
- Baselines are organized into inference without fine-tuning, fine-tuning without step-level supervision, and fine-tuning with step-level supervision.
- Search-o1 combines agentic retrieval-augmented generation with a reason-in-document module.
- ReSearch trains language models to interleave reasoning with explicit search actions and use retrieved evidence for multi-hop reasoning.
- Search-R1 enables language models to generate search queries autonomously during multi-turn reasoning.
- PPRM provides step-wise signals for GRPO-based reinforcement learning, while StepResearch uses intermediate rewards and token-level supervision with step-wise PPO.
A.1.3 The Details of Implementation
The implementation builds on Search-R1 and evaluates MR-Search through specified training resources, training dynamics, and case studies of search and self-reflection.
- MR-Search is implemented on top of Search-R1 using VeRL, with generation and reflection prompts supplied for the agent.
- Training uses AdamW with a 1e-6 learning rate, 300 steps, three retrieved documents, group size five, and 8K or 16K context lengths by dataset.
- RL training uses 8× NVIDIA Tesla H100 GPUs, with 2× additional H100 GPUs serving the retriever.
- MR-Search shows more stable convergence and consistently higher reward than Search-R1 in training-dynamics figures on Qwen2.5-3B-Base and Qwen2.5-7B-Base.The figures track test accuracy, training accuracy, and tool-call counts.
- Case studies show MR-Search revisiting intermediate answers through additional search and self-reflection before producing correct final answers.One case explicitly describes correcting an early answer after ambiguous retrieval results.
- The trajectories include iterative reasoning, search queries, retrieved information, and revised answers across multiple interactions.