PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Assessment Reinforcement Learning for Search-State Awareness in Retrieval-Augmented Large Language Models

Taewhoo Lee, Minju Song, Jungwoo Park, Chanwoong Yoon, Junha Jung, Hyeon Hwang, Sanghoon Lee, Minbyul Jeong, Jaewoo Kang

arXiv:2610.30435v1cs.CLcs.AI

TL;DR

Search agents often lack awareness of what information they have gathered and what remains missing. SARL trains this capability through self-assessment and a dual reward, and achieves a 24.3% relative improvement over the strongest baseline, with further gains on multi-hop and deep-research benchmarks.

  • Problem

    Current RL-based search agents struggle to recognize what information has been obtained and what remains missing, and existing methods show instability under extended search horizons.

    Source: §1 · pp. 1–2

  • Method

    SARL trains search-state awareness with a self-assessment phase and a dual reward that rewards corrective assessments while penalizing excessive assessment use.

    Source: §1 · pp. 2–3

  • Results

    SARL achieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% on multi-hop reasoning and 27.9% on deep research benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SARL shifts search-state awareness from a prompted token-level process toward an internalized reasoning capacity, reflected in more unique search queries and higher document recall.

    Source: §6 · p. 9

    Whether SARL combines synergistically with broader pipelines such as SFT-warmup or multi-agent orchestration remains unexplored.

    Source: §A.1 · p. 13

Abstract

from arXiv · show

Recent advances in search agents have enabled large language models to tackle complex tasks by retrieving and synthesizing external information. However, existing search agents often lack search-state awareness, i.e., the ability to recognize what information has been obtained and what remains missing, leading to hallucinated reasoning and prematurely terminated search. To address this limitation, we propose Self-Assessment Reinforcement Learning (SARL), a reinforcement learning framework in which models learn to recognize their own search state during training and improve their search behavior accordingly, without requiring additional assessment at test time. SARL achieves this through a dual reward design that captures the corrective effect of each assessment while penalizing excessive reliance on it, jointly encouraging effective and efficient search trajectories. Under this setup, SARL achieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% on multi-hop reasoning and 27.9% on deep research benchmarks against the strongest per-benchmark baselines. SARL further surpasses methods built on auxiliary supervision or multi-agent coordination, despite operating as a standalone RL framework with no inference-time scaffolding. Beyond final performance, SARL maintains training stability under extended search horizons, a regime where existing methods consistently degrade, enabling sustained gains as task complexity grows.

1. Introduction

SARL targets search-state awareness: recognizing what information has been obtained and what remains missing. It adds train-time self-assessment and a dual reward design to improve search without explicit assessment at inference time.

1. Introduction

  • Current RL-based search agents lack search-state awareness, leading to hallucinated reasoning and premature termination; existing methods also degrade under extended search horizons.Search-state awareness means recognizing what has been obtained and what remains missing.Source: §1 · pp. 1–2
  • SARL adds a self-assessment phase in which the model identifies question constraints, checks them against retrieved evidence, and revises the trajectory when information is missing.The assessment occurs in an isolated context after an initial answer; an incomplete judgment returns the assessment as feedback for revision.Source: §1 · p. 2
  • A dual reward captures each assessment’s corrective effect on answer correctness while penalizing frequent assessments, encouraging thorough search rather than repeated assessment retries.The resulting search behavior does not require explicit assessment steps at inference time.Source: §1 · p. 2
  • SARL achieves a 24.3% relative improvement over the strongest baseline, with average gains of 35.9% on multi-hop reasoning and 27.9% on deep research benchmarks.These comparisons are against the strongest per-benchmark baselines for the multi-hop and deep-research averages.Source: §1 · p. 2
  • Across ten benchmarks, SARL maintains stable training and generalizes across model scales and families.It also surpasses SFT+RL pipelines and multi-agent frameworks with larger backbones while operating without inference-time scaffolding.Source: §1 · pp. 2–3

2. Method

SARL adds a separate self-assessment and revision process to search trajectories, then uses rewards for assessment effects and frequency to train the policy. Its setup spans single-hop, multi-hop, and long-horizon search tasks.

2.1. Problem Setup and Framework

  • SARL assesses completed trajectories in an isolated context and, when they are judged incomplete, uses the assessment to prompt further search and a revised answer.The same policy produces the assessment; the process repeats until COMPLETE or the maximum turn limit is reached.Source: §2.1 · p. 3

2.2. Dataset Curation

  • The 45K-example training corpus combines 15K samples each from NQ, HotpotQA, and ASearcher to cover single-hop, multi-hop, and complex multi-step search.The ASearcher portion includes 8,537 original examples and 6,463 synthesized examples.Source: §2.2 · pp. 3–4

2.3. Training Objective

  • SARL uses GRPO, sampling groups of trajectories and updating the policy through group-relative comparisons, with advantages computed from its combined assessment reward.The objective also uses asymmetric clipping and a fixed trajectory-length denominator.Source: §2.3 · p. 4

2.4. Reward Design

  • The total reward combines final outcome reward, exponentially penalized by assessment count, with a weighted assessment reward based on changes in answer correctness.The assessment reward assigns +1.0 to incorrect→correct and correct→correct, −0.1 to incorrect→incorrect, and −2.0 to correct→incorrect transitions.Source: §2.4 · p. 4

3. Experiments

SARL is evaluated on ten benchmarks spanning single-hop QA, multi-hop reasoning, and deep research, against retrieval and search-agent baselines. It improves the overall benchmark average and shows gains on more complex tasks while maintaining stability under extended search horizons.

3.1. Experimental Setup

  • Evaluation covers ten in-domain and out-of-domain benchmarks across three task categories, using GPT-4.1 as an LLM judge and avg@4 scores averaged over four sampled completions per query.The maximum number of interaction turns is 32. Reproduced baselines use Qwen2.5-7B, except Direct Generation and SFT; published RAG-R1 and MA-RAG results were not reproduced under this setup.Source: §3.1 · pp. 4–5

3.2. Main Results

  • 24.3% relative improvement over the strongest baseline (ASearcher: 0.345), with SARL averaging 0.429 across the benchmark suite.Table 1 compares performance across benchmarks of varying complexity; the reported peak gains over the best per-benchmark baselines are 58.1% on Musique and 48.3% on FRAMES.Source: Table 1 · p. 5, §3.2 · p. 5
  • Average gains over the strongest per-benchmark baselines are 3.9% on single-hop QA, 35.9% on multi-hop QA, and 27.9% on deep research tasks.Only NQ and HotpotQA overlap with SARL’s training data, and the paper reports gains across all benchmarks.Source: §3.2 · p. 5
  • With a 32-turn limit, AutoRefine and ReSeek show training instability and performance degradation; AutoRefine falls below one search turn on average and ReSeek’s validation scores approach zero late in training.The paper attributes this to inline reflection text and reports that SARL avoids the failure mode through trajectory-assessment decoupling and fine-grained assessment rewards.Source: §3.2 · pp. 5–6
  • On seven shared benchmarks, SARL’s average is 0.554 versus 0.449 for RAG-R1, a 23.4% relative improvement.On HotpotQA and 2Wiki, SARL also exceeds the reported 70B MA-RAG results: 0.614 vs. 0.507 and 0.645 vs. 0.431, respectively.Source: §3.2 · p. 6

4. Analysis

The analyses track how SARL’s search behavior changes during training, test the reward components, and examine transfer across model families and scales. They report increasing query diversity and recall, stable gains from the full reward design, and transfer to additional backbones.

4.1. Search Diversity and Evidence Recall

  • SARL’s unique-query count rises to over six distinct queries on average toward training’s end, while AutoRefine and ReSeek trend downward.The increase is consistent across NQ, HotpotQA, and ASearcher and is more pronounced on datasets requiring more complex reasoning.Source: §4.1 · p. 6, Figure 4 · p. 7
  • SARL steadily improves document recall, particularly on NQ and HotpotQA, while AutoRefine and ReSeek remain largely static and rarely retrieve relevant evidence on BrowseComp-Plus.BrowseComp-Plus has sparsely distributed target evidence weakly aligned with surface-level queries, yet SARL maintains a consistent recall trend.Source: §4.1 · pp. 6–7

4.2. Effect of Assessment Penalty and Reward

  • Only the full Self-Assessment + Penalty + Reward configuration yields stable improvements across evaluation score, unique search cost, and document recall.Self-assessment alone becomes unstable; adding the penalty mitigates this, and the full model has over twice as many unique searches as the self-assessment-only variant at later training stages.Source: Figure 4 · p. 7, §4.2 · p. 7

4.1. Search Diversity and Evidence Recall

  • The paper links SARL’s gains in query diversity and evidence recall to stronger search-state awareness, also reported in its trajectory analysis.It describes recognizing insufficient evidence as driving more diverse queries and recovery of relevant evidence.Source: §4.1 · p. 7

4.2. Effect of Assessment Penalty and Reward

  • With α = 0.8 fixed, λ = 0.2 falls from a best score of 55.5 to 42.6 by training’s end, while λ = 0.5 and λ = 0.8 remain stable.With λ = 0.5 fixed, α ∈{0.2, 0.5, 0.8} yields final validation scores in a 4-point range (51.7–55.5).Source: §4.2 · pp. 7–8

4.3. Generalizability across Model Scale and Family

  • On Llama-3.1-8B-Instruct, SARL’s average score rises from 51.2 to 54.3, whereas GRPO falls from 52.3 to 44.4; SARL leads by +9.9 at step 300.The comparison covers validation at steps 100, 200, and 300 on NQ, HotpotQA, and ASearcher.Source: §4.3 · p. 8
  • The Qwen2.5-14B variant outperforms Qwen2.5-7B at every reported step and benchmark, retaining a +3.6 average-point advantage at step 300.Its advantage is largest on ASearcher at step 300 (+5.6).Source: §4.3 · p. 8

5. Related Work

Related work covers retrieval-augmented generation and RL-based search agents, including methods using supervised warmup, intermediate rewards, planning, or multi-agent coordination. SARL instead learns self-assessment as an internal signal within a single policy.

Retrieval-Augmented Generation

  • RAG methods retrieve external evidence, while multi-document aggregation and self-controlled retrieval approaches extend beyond a single retrieve-then-generate step.The section notes that static or shallowly iterative retrieval can be brittle when evidence is scattered or requires sustained clarification.Source: §5 · p. 8

Internalizing Search-State Awareness through RL

  • RL-based search agents optimize sequential search policies from outcome feedback, but final-outcome training may permit non-faithful reasoning, premature termination, or unsupported assumptions.Related approaches address sparse rewards with SFT warmup, multi-agent frameworks, intermediate rewards, planning, or memory management.Source: §5 · p. 8
  • SARL trains self-assessment within a single policy using outcome-based credit assignment, rewarding measurable assessment effects and penalizing excessive assessment reliance.The design is presented as requiring no additional long-horizon data, auxiliary judges, or inference-time orchestration.Source: §5 · p. 8

6. Conclusion

SARL internalizes search-state awareness in retrieval-augmented LLMs through rewards balancing corrective assessment with efficiency penalties. It outperforms baselines across ten benchmarks, with stronger gains on complex tasks and increased search activity and document recall.

6. Conclusion

  • SARL internalizes search-state awareness through a reward design balancing corrective assessment rewards with efficiency penalties.The conclusion describes this as shifting awareness from a token-level prompted process to an internalized reasoning capacity.Source: §6 · pp. 8–9
  • 35.9% on multi-hop QA and 27.9% on deep research benchmarks are SARL’s reported gains across ten benchmarks.The conclusion also reports a twofold increase in unique search queries and higher document recall.Source: §6 · p. 9
  • The reported improvements are reflected in a twofold increase in unique search queries and notably higher document recall.Source: §6 · p. 9
  • SARL aims to let agents self-correct without relying on external feedback at inference time.Source: §6 · p. 9

A.1. Limitations

The appendix identifies limitations in SARL’s base-model assumptions, evaluated task scope, and compatibility with broader pipelines. Its reported evaluation is limited to search-augmented QA, and pipeline combinations remain unexplored.

A.1. Limitations

  • SARL assumes the base model can judge whether retrieved evidence supports a candidate answer, but this capability does not hold universally.The authors report that the judgment was reliable enough to provide an effective training signal at the 7B scale.Source: §A.1 · p. 13
  • The evaluation focuses exclusively on search-augmented question answering, leaving applicability to code generation, mathematical reasoning, and general agentic planning open.Source: §A.1 · p. 13
  • Whether SARL combines synergistically with SFT-warmup or multi-agent orchestration remains unexplored.The paper presents SARL as a standalone RL framework to isolate self-assessment as a training signal.Source: §A.1 · p. 13

A.2. Example Trajectories of Search-State Awareness

The example trajectories contrast agents that commit despite missing evidence with SARL’s trajectory, which tracks resolved and unresolved subgoals and searches for remaining information. The examples illustrate hallucinated reasoning and premature termination alongside sequential, targeted retrieval.

A.2. Example Trajectories of Search-State Awareness

  • The Catherine I example gives an unsupported answer after one search, while its refine token asserts rather than verifies the needed relationship.Source: §A.2 · pp. 13–14
  • The Lauderdale example commits to an answer despite acknowledging that no political office held by successive castle occupants was identified.It confuses the Earl of Lauderdale title with a political office; another example stops after one search despite the question requiring multiple steps.Source: §A.2 · pp. 13–14
  • SARL’s example decomposes a multi-hop question into subgoals, tracks satisfied constraints, and issues targeted follow-up searches for missing evidence.In the date-of-birth example, the agent identifies the person, finds her mother, then searches for the mother’s birth date.Source: §A.2 · pp. 14–15
  • The SARL trajectory states what remains missing before searching again, rather than committing to an unsupported answer or stopping after one search.Source: §A.2 · pp. 14–15

A.3. Validating the Central Premise of SARL

The experiments test whether base models can assess incomplete trajectories and whether SARL benefits from meaningful rather than randomized assessment. The base model identifies many flawed trajectories as incomplete, while randomized feedback underperforms SARL and becomes unstable during training.

A.3. Validating the Central Premise of SARL

  • 73.2% of flawed Search-R1 trajectories were correctly identified as INCOMPLETE by Qwen2.5-7B before SARL training.The authors interpret this as evidence that the base model has a starting self-assessment capability.Source: §A.3 · p. 15
  • At step 339, SARL reaches 55.5 average accuracy while randomized assessment reaches 41.2.The randomized variant falls from 46.6 at step 200 to 41.2 at step 339; Table 2 compares SARL and randomized assessment across training stages.Source: §A.3 · p. 15, Table 2 · p. 15
  • The experiments support the premise that search-state awareness can be cultivated when the model has sufficient starting capability and the training signal carries corrective information.The authors report that arbitrary feedback under otherwise identical conditions fails to improve performance and destabilizes training.Source: §A.3 · p. 15

A.4. Performance Evaluation on EM and F1 Metrics

SARL demonstrates consistently superior performance across benchmarks on both Exact Match and F1 metrics.

A.4. Performance Evaluation on EM and F1 Metrics

  • SARL consistently outperforms other methods across all benchmarks on Exact Match (EM) and F1 metrics.Table 3 presents the main results for single-hop QA, multi-hop QA, and deep research benchmarks.Source: §A.4 · p. 15

A.5. Overview of Training Stability and Performance

This section surveys SARL’s search-state awareness and training stability, including the accompanying training-cost and search-trajectory considerations. It also covers comparisons across training paradigms, multi-turn behavior, hyperparameter sensitivity, model-family and scale generalization, and evaluation and deployment safeguards, including the assessment prompt’s focus on retrieved information and remaining gaps.

A.5. Overview of Training Stability and Performance

A.6. Training Cost Analysis

  • 2.3× training cost yields a 25.4% relative improvement in average performance for SARL over Search-R1 under matched training settings.The 2.5× trajectory-length ratio closely aligns with the time ratio, indicating that overhead primarily comes from longer search trajectories rather than self-assessment calls.Source: Table 4 · p. 16, §A.6 · p. 16

A.7. Comparison with Broader-Paradigm Methods

  • 0.554 average performance across seven shared benchmarks for SARL compares with 0.449 for RAG-R1-sq and 0.457 for RAG-R1-mq.SARL uses RL without RAG-R1’s SFT warmup; against MA-RAG (Thang Nguyen, 2025), it leads the 70B variant on HotpotQA (0.614 vs. 0.507) and 2Wiki (0.645 vs. 0.431), but trails Llama3-8B on TriviaQA (0.762 vs. 0.826).Source: Table 5 · p. 17, §A.7 · p. 17

A.8. Analysis of Multi-Turn Efficiency

A.9. Evaluation of Search-State Awareness

  • 42% Correct SSA for SARL exceeds Search-R1’s 24% and AutoRefine’s and ReSeek’s 17%, while SARL’s Incorrect SSA rate is 51% versus 65–76% for baselines.Correct SSA requires recognizing obtained information, identifying missing gaps, and giving the correct final answer; partial failures count as Incorrect SSA.Source: §A.9 · p. 18

A.10. Hyperparameter Sensitivity

  • With λ = 0.5 fixed, performance remains stable across α ∈{0.2, 0.5, 0.8}, while λ = 0.2 with α = 0.8 drops from 55.5 best to 42.6 final validation score.The sharp decline indicates that under-weighting the assessment reward leads to training instability.Source: §A.10 · p. 18, Table 7 · p. 19

A.11. Generalization across Model Scale and Family

  • SARL on Llama-3.1-8B-Instruct improves from 51.2 to 54.3 average validation score, while GRPO declines from 52.3 to 44.4; SARL outperforms GRPO on every benchmark by step 300.With the Qwen2.5 backbone, 14B outperforms 7B across benchmarks and training steps, particularly on ASearcher; the default α = 0.8, λ = 0.5 configuration yields the strongest result.Source: §A.11 · pp. 18–19, Table 8 · p. 19

A.12. Prompt Details

  • The self-assessment prompt judges search-process integrity by whether essential constraints were addressed before answering, then returns COMPLETE or INCOMPLETE and prompts revision after an INCOMPLETE judgment.It evaluates the process rather than the answer itself and directs the model to revise its search using the assessment feedback.Source: §A.12 · pp. 19–20
  • The GPT-5.2 assessment prompt scores whether agents recognize retrieved information and identify missing information, independently of final-answer correctness.Correct gap identification includes recognizing when no gap remains.Source: §A.12 · pp. 21–22

A.13. Broader Impacts

  • Search agents that detect incomplete reasoning may reduce confident unsupported answers and support research, education, healthcare information access, and decision support.These applications are highlighted because fabricated or unverified outputs can be costly.Source: §A.13 · p. 22
  • Biased or inaccurate retrieval can amplify misinformation, and malicious actors could exploit agents for information extraction or scaled disinformation; safeguards include evaluation, source attribution, bounded corpora, uncertainty signaling, and verification.SARL is a training method built on public base models, retrievers, and benchmarks; the paper says it does not meaningfully expand the backbones’ misuse surface, but greater coherence may encourage uncritical trust.Source: §A.13 · p. 22
  • SARL’s training and validation scores increased stably and monotonically, with validation peaking at approximately 0.56 by 325 steps.The paper attributes this pattern to its dual reward-penalty mechanism internalizing search-state awareness without over-reliance on explicit feedback.Source: Figure 5 · p. 23
  • 42% Correct SSA for SARL exceeded 24% for Search-R1 and 17% for AutoRefine and ReSeek.The comparison is reported in Figure 6.Source: Figure 6 · p. 24
Loading 2610.30435v1…