PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Information Control for Search-Augmented LLM Reasoning

Siheng Xiong, Oguzhan Gungordu, James C. Kerce, Faramarz Fekri

arXiv:2602.01672v2cs.CL

TL;DR

Search-augmented reasoning agents need better control over retrieval because uncontrolled acquisition can saturate context and outcome-only RL poorly guides intermediate decisions. DeepControl uses information utility to regulate retrieval extent and resolution through continuation, hierarchical granularity, and annealed control. Across seven datasets, it consistently outperforms strong baselines, including by 9.4 average EM points with Qwen2.5-7B and 8.6 with Qwen2.5-3B versus Search-R1-instruct.

  • Problem

    Uncontrolled retrieval can accumulate redundant or noisy evidence, while outcome-only RL provides limited guidance for intermediate information-acquisition decisions.

  • Method

    DeepControl uses information utility to control retrieval extent and resolution through continuation control, hierarchical selective expansion, and annealed training-time intervention.

  • Results

    DeepControl consistently outperforms strong baselines across seven datasets, improving average EM over Search-R1-instruct by 9.4 points with Qwen2.5-7B and 8.6 points with Qwen2.5-3B.

  • Takeaways & Limitations

    The framework improves reasoning accuracy, training stability, and evidence utilization while enabling effective information acquisition without external intervention at test time.

  • Takeaways & Limitations

    The evaluation focuses on text-based search-augmented question answering and does not study dynamic corpora, multi-tool or multi-agent systems, or multimodal reasoning.

Abstract

from arXiv · show

Search-augmented reasoning agents interleave multi-step reasoning with external retrieval, but uncontrolled retrieval can introduce redundant evidence, saturate the context, and destabilize reinforcement learning (RL). Existing outcome-based RL methods provide only sparse terminal rewards, offering limited guidance for intermediate information-acquisition decisions. We propose DeepControl, an adaptive information-control framework based on information utility, a state-dependent estimate of the marginal value of retrieved evidence. The framework regulates information acquisition along two axes: extent, i.e., whether retrieval should continue, and resolution, i.e., how much retrieved detail should be exposed. It implements these controls through retrieval-continuation guidance, hierarchical granularity control, and an annealed control-forcing scheme. This enables the policy to internalize effective acquisition behavior during training and operate without external control at test time. Across seven benchmarks, DeepControl consistently outperforms strong RL and retrieval baselines without explicit information control; compared with Search-R1, it improves average performance by +9.4 and +8.6 points on Qwen2.5-7B and Qwen2.5-3B, respectively. Additional analyses show improved search effectiveness, training stability, and evidence utilization.

1 Introduction

Search-augmented agents face a central information-acquisition problem: more retrieval can add redundancy, noise, and context interference, while outcome-only RL gives limited guidance for intermediate retrieval decisions. DeepControl addresses this by controlling both how long agents retrieve and how much retrieved detail they expose, using utility-driven signals and annealed intervention.

  • Motivation: More retrieval can cause context saturation, redundant or noisy evidence, and interference that degrades decision quality.These failures mean that additional retrieval does not necessarily improve reasoning.
  • Motivation: Outcome-only RL provides limited guidance for intermediate retrieval decisions, allowing both over-retrieval and premature termination.Agents may accumulate unnecessarily long contexts when evidence is weak or stop while additional evidence remains beneficial.
  • Core idea: DeepControl treats information acquisition as a unified control problem over extent—whether to continue—and resolution—how much detail to expose.This differs from approaches that address retrieval quality, stopping, or evidence expansion in isolation.
  • Core idea: Information utility estimates the state-dependent marginal value of retrieved evidence and combines novelty with effectiveness.It provides a practical signal for controlling information acquisition during training.
  • Framework: The framework combines retrieval-continuation control, hierarchical granularity control, and annealed control to regulate acquisition and internalize effective behavior.The controls target premature stopping, over-retrieval, and selective expansion of high-utility content.
  • Results: Experiments across multiple tasks and datasets show consistent improvements in reasoning accuracy, training stability, and evidence utilization.The experiments cover diverse search-augmented reasoning benchmarks.

2 Preliminaries

Search-augmented reasoning agents combine multi-step reasoning with external retrieval, but online RL must balance task success against policy deviation while handling retrieval-related failure modes. Existing methods commonly use outcome-based rewards and can suffer from suboptimal search behavior, information overload, and unstable training.

  • Problem formulation: Search-augmented agents interleave reasoning with external retrieval while maintaining accumulated evidence and intermediate reasoning in their state.At each step, the policy samples structured actions such as retrieval actions with query parameters, and trajectories terminate with a final answer or step limit.
  • Online RL: Online RL alternates between rollout generation and policy updates to maximize task success while regularizing deviation from a reference policy.
  • Policy optimization: PPO uses a clipped surrogate objective with GAE-based advantages and a value function to regulate update stability.
  • Policy optimization: GRPO estimates advantages within groups of responses for the same prompt instead of learning a value function, while applying KL regularization to a reference policy.
  • Search-augmented RL: Outcome-based search RL evaluates final-answer correctness with Exact Match, while retrieved content is produced externally and policy gradients update only model-generated tokens.
  • Limitations: Outcome-based RL can yield suboptimal search, information overload, and unstable training because intermediate retrieval decisions receive limited guidance and raw evidence can accumulate excessively.Agents may over-retrieve when evidence is unavailable or queries are poorly specified; long contexts can miss key evidence, saturate context, increase costs, and destabilize optimization.

3 Adaptive Information Control

DeepControl uses state-dependent information utility to control both how much information agents acquire and how much retrieved detail they expose. It combines selective expansion, search-continuation intervention, annealed control forcing, and auxiliary reward signals to guide training while reducing external control over time.

  • Information Utility: Information utility provides a shared, state-dependent training signal for controlling retrieval extent and information resolution.It evaluates the marginal value of retrieved evidence relative to the current reasoning state.
  • Granularity Control: Granularity control exposes coarse retrieved evidence first, then selectively expands high-utility content through hierarchical refinement actions.Training selects high-utility leaves and traces their ancestors to construct target expansions while limiting context growth.
  • Search Continuation Control: Search-continuation control intervenes when utility indicates premature stopping or unnecessary continuation, issuing stop or one-step continuation signals.Stopping follows sustained low utility, while continuation is triggered by sustained high utility and insufficient confidence in the answer.
  • Reinforcement Learning with Information Control: Annealed control-forcing training shifts rollouts from frequent external intervention to autonomous policy behavior.The curriculum uses controlled and uncontrolled rollout modes, with control removed entirely in the final stage.
  • Reinforcement Learning with Information Control: A composite reward preserves answer correctness while adding penalties for tool-usage violations and control non-compliance.The outcome component is F1-based and includes a format floor for valid outputs.

4 Experiments

DEEPCONTROL is evaluated across seven benchmarks against retrieval, non-retrieval, and fine-tuning baselines. It consistently improves performance and shows benefits in training dynamics, schedule choice, robustness, and failure-mode reduction.

  • Main Results: DEEPCONTROL improves average EM over Search-R1-instruct by 9.4 points with Qwen2.5-7B and 8.6 points with Qwen2.5-3B.The gains are reported across seven datasets and task types.
  • Main Results: DEEPCONTROL improves on all benchmarks, with gains consistent across general and multi-hop question answering.The evaluation covers NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle.
  • Training Dynamics: DEEPCONTROL achieves higher training rewards than vanilla PPO, while PPO remains more stable than GRPO under annealed control-forcing.As control messages decrease, the policy increasingly performs search and expansion actions independently.
  • Sensitivity and Ablations: Intermediate control settings provide the best trade-off between premature stopping and unnecessary continuation, while performance is most sensitive to the tool-usage penalty.Search continuation and granularity control are complementary and both important in ablations.
  • Schedule Sensitivity: The default annealing schedule achieves the best final no-control performance, whereas overly fast or slow schedules reduce performance.Later checkpoints also show higher early-step utility under no-control evaluation.
  • Error Analysis: Removing continuation mainly increases insufficient-retrieval errors, while removing granularity mainly increases missing supporting evidence.DEEPCONTROL reduces major behavioral failure modes overall, although reasoning errors remain the largest residual category.

5 Related Work

Prior work often treats additional retrieved information as beneficial and addresses retrieval quality, stopping, or context selection separately. DeepControl instead frames acquisition extent and resolution as a unified reinforcement-learning control problem.

  • Retrieval-Augmented Reasoning: Retrieval-augmented approaches commonly append retrieved content using fixed or heuristic strategies, which can accumulate redundancy, saturate context, and introduce noisy reasoning.These limitations motivate adaptive control over retrieval continuation and information granularity.
  • Reinforcement Learning: Outcome-based reinforcement learning provides final-answer supervision but limited guidance for intermediate retrieval decisions in long-horizon settings.Agents may consequently over-retrieve when evidence is weak or queries are poorly specified.
  • Information Control: Existing information-control methods typically address improved retrieval, stopping heuristics, or selective context construction separately rather than as one unified RL problem.DeepControl targets both acquisition extent and resolution together.

6 Conclusion

DeepControl regulates search continuation and hierarchical expansion using information utility, then uses annealed control-forcing to internalize effective acquisition behavior. Across multiple tasks and datasets, it reports gains in reasoning accuracy, training stability, and evidence utilization.

  • Conclusion: DeepControl uses information utility to control search extent and resolution during online reinforcement learning, with annealing enabling behavior internalization without test-time intervention.Search continuation controls whether acquisition continues, while hierarchical expansion controls how much retrieved detail is exposed.

Limitations

The evaluation is limited to text-based search-augmented reasoning and does not cover dynamic corpora, multi-tool or multi-agent systems, or multimodal reasoning.

  • Scope: The study evaluates question answering across text-based search settings but leaves dynamic corpora, multi-tool agents, multi-agent systems, and multimodal reasoning for future work.The stated scope boundary limits direct conclusions about those broader settings.
  • Deployment Caveat: Retrieved-evidence errors and imperfect training signals may still propagate errors or produce suboptimal retrieval behavior, and high-stakes deployment is not studied.Additional safeguards and evaluation would be required for such use.

A Information Utility

Information utility estimates the state-dependent marginal value of retrieved evidence by separating novelty from effectiveness. It provides a training-time control signal whose properties capture useful evidence, redundancy, and diminishing returns.

  • Information utility measures the marginal value of newly acquired information relative to the agent’s current reasoning state.
  • Novelty evaluates redundancy across the full retrieved leaf pool, while effectiveness evaluates the helpfulness of newly injected nodes.This separates coverage from the impact of content actually exposed to the agent.
  • Effectiveness is high only when newly injected evidence increases confidence in an acceptable gold answer and is zero otherwise.The signal is available only during training because it requires gold answers.
  • Later evidence can remain effective after novelty declines, indicating that less novel retrieval may still improve answer confidence.
  • Information utility increases with novel, beneficial evidence and exhibits diminishing marginal returns after sufficient task evidence has been acquired.
  • The utility is used as an external control signal rather than directly incorporated into the RL reward, supporting modularity and simpler, more stable training.

B.1 Granularity Control via Hierarchical Selective Expansion

Granularity control represents retrieved information as hierarchical trees, initially exposes coarse root nodes, and selectively expands high-utility content. This decouples retrieval from refinement while limiting context growth.

  • Granularity control presents retrieved information coarsely first and selectively expands finer details when needed.This addresses voluminous retrieval results whose details are not uniformly useful across reasoning stages.
  • Each retrieved source is represented as a rooted tree whose edges connect evidence units to progressively finer-grained refinements.
  • The agent initially injects only root nodes, then incrementally expands selected children to grow the observed evidence set.
  • During training, the controller scores retrieved leaves, selects the top-kexpand leaves, and traces their ancestors upward to construct target evidence sets.
  • The policy learns expansion actions aligned with utility-derived target edges, prioritizing high-utility information while minimizing context growth.

B.2 Search Continuation Control

Search continuation control treats whether to retrieve again as an explicit decision monitored by information utility. It intervenes to stop persistently unproductive search or continue when useful evidence remains and confidence is insufficient.

  • The controller addresses both premature search termination and continued retrieval when no useful evidence remains.
  • Search stops when utility remains below δstop for mstop consecutive search steps, triggering an explicit stop-search signal.
  • A one-shot continuation signal is triggered when recent utility stays high but the model remains insufficiently confident in the gold answer.
  • Search continuation is primarily governed by the learned policy, while utility monitors behavior and triggers corrective control when needed.

B.3 Reinforcement Learning with Information Control

DeepControl trains search-augmented agents with external information-control signals early in learning and gradually removes them. The training objective combines answer correctness with tool-use regularization so the final policy can operate without intervention.

  • External control signals improve exploration and stabilize early learning, but the policy must internalize these behaviors for test-time operation.
  • Rollouts mix controlled trajectories, where a utility-monitoring controller can trigger interventions, with uncontrolled trajectories generated from the current state.
  • An annealed curriculum shifts training from frequent control to reduced control and finally no external control.
  • The composite reward combines F1-based answer correctness with penalties for improper tool usage and control non-compliance.Penalty totals are capped to avoid over-penalization.
  • Valid outputs receive a non-zero format-floor reward, while malformed tool interactions incur explicit penalties.

C Dataset Overview

DeepControl is evaluated on seven benchmarks spanning general and multi-hop question answering. The datasets cover real search queries, reading comprehension, entity-centric knowledge, and multi-hop reasoning across Wikipedia-based and compositional questions.

  • Task Categories: Seven benchmarks cover general question answering and multi-hop question answering.The general QA group includes NQ, TriviaQA, and PopQA; the multi-hop group includes HotpotQA, 2WikiMultiHopQA, Musique, and Bamboogle.
  • General Question Answering: Natural Questions pairs real Google search queries with human-annotated Wikipedia answers.Its stated splits contain 79,168 training samples and 3,610 test samples.
  • General Question Answering: TriviaQA is used as a large-scale reading-comprehension benchmark with an 11,313-example test set.
  • General Question Answering: PopQA measures parametric knowledge coverage on long-tail entities using 14,267 entity-centric triples.
  • Multi-Hop Question Answering: The multi-hop benchmarks require reasoning across multiple paragraphs, structured and unstructured information, or composed hops.Musique contains 2–4 hop problems, while Bamboogle contains 125 manually curated two-hop questions selected because search engines originally answered them incorrectly.

D Implementation Details

The implementation uses hierarchical retrieval and adaptive control within a PPO-based training setup, with experiments spanning two Qwen2.5 model sizes and seven datasets. Control increases training cost but adds no inference-time runtime overhead because the final policy operates without external control.

  • Retrieval and Corpus: Hierarchical retrieval exposes passage summaries initially and reveals full passage text through an expand action.The main experiments use a two-level extractive structure built from the 2018 Wikipedia passage corpus.
  • Retrieval and Corpus: A three-level Wikipedia-search variant was excluded from the main results because it changes the retrieval corpus and weakens direct comparison with prior baselines.The exploratory hierarchy contains document summaries, section summaries, and paragraph-level content.
  • Control Configuration: The control system uses E5 embeddings, 512-token encoding truncation, and kNN novelty estimation with knn = 5.Continuation intervention requires utility above δcont = 0.3 for two consecutive search steps and a gold-answer score below τscore = −2.0.
  • Models and Retrieval: Experiments use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct with the 2018 Wikipedia dump and E5 retrieval.The approach uses hierarchical selective expansion rather than appending raw retrieved passages, with three retrieved passages for existing retrieval baselines.
  • RL Training: PPO is the default RL algorithm, while GRPO was abandoned after training collapse appeared after a few dozen optimization steps.Training merges NQ and HotpotQA, runs for five epochs, and anneals control probability through 0.9, 0.5, 0.2, and 0.
  • Training Cost: Compared with vanilla PPO, DeepControl increases rollout time from 5.2 to 6.7 minutes and 1000-step training time from 3.61 to 4.66 days.The added cost occurs during training, while inference has no additional runtime overhead because external control is removed.

E Additional Results

Additional analyses show that DeepControl improves training stability, learns task-sensitive stopping behavior, and remains effective across utility formulations, reward settings, evidence interfaces, and retrievers.

  • Training stability: 8.3% average improvement over vanilla PPO shows that control signals improve training stability and final performance in online RL.Control signals provide corrective guidance early, then are gradually internalized as external control is removed.
  • Training stability: PPO is more stable than GRPO under control annealing, although GRPO improves faster early before reward degradation after extended training.The comparison links algorithm choice to stabilization during the annealing process.
  • Utility and reward design: The default additive utility performs best overall, while alternative formulations remain competitive but less effective.The results suggest that continuous effectiveness shaping and additive combination provide the most reliable control signal in this setting.
  • Utility and reward design: Gold-answer-free decision-impact utility still outperforms Search-R1, indicating that the framework can remain effective without gold-answer access.The variant estimates utility from changes in the model’s answer distribution after evidence injection.
  • Robustness: Moderate reward and schedule perturbations preserve performance, but malformed tool usage and control non-compliance receive the strongest reward sensitivity.The model is most sensitive to the violation penalty, while excessive ceiling rewards can favor incorrect trajectories.
  • Search behavior: DeepControl adapts search effort to task difficulty, and counterfactual tests show that both forced continuation and removing the final search step reduce accuracy.It searches less on NQ, more on HotpotQA, and the final retrieval is often useful rather than redundant.
  • Ablations and robustness: Adaptive information control, rather than hierarchical evidence construction alone or reward shaping alone, accounts for the main gains.The control mechanism remains beneficial with an outcome-only objective and is necessary for hierarchical expansion to work effectively.
  • Ablations and robustness: DeepControl still outperforms Search-R1 with BM25, suggesting that its control mechanism is not limited to the E5 retriever.The retriever is changed while the utility estimation procedure remains unchanged.

Prompt: Search-Augmented Reasoning

Search-augmented reasoning follows a controlled protocol in which agents search, optionally expand retrieved documents, and then answer. The examples show both continuation and stopping decisions, including answers produced after retrieved evidence or available context is deemed sufficient.

  • Round structure: Each round follows a structured sequence of thinking, searching, receiving results, optionally expanding documents, receiving information, and reasoning again.The protocol distinguishes answering rounds, which end with an answer, from continuing rounds, which return to further search.
  • Control: Control messages determine whether the agent stops searching, continues for an additional step, or changes the amount of retrieved detail exposed.Examples include stopping search, continuing search once, expanding documents, and declining expansion while continuing retrieval.
  • Protocol: The protocol requires reasoning around new search or information inputs before requesting expansion or producing a final answer.Search results provide snippets, while expansion returns fuller document text for subsequent reasoning.
  • Example: In the India example, the agent expands a promising document after snippets are insufficient and answers 1951–52 from the expanded information.The search results identify a relevant 1951–52 elections document, which is then expanded before the final answer is produced.
  • Example: In the Reading Football Club example, stopping search leads the agent to answer Xiu Li Dai from the available document context.The response states that the available information identifies Xiu Li Dai and Yongge Dai as owners, then outputs Xiu Li Dai as the shortest answer span.
Loading 2602.01672v2…