PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent

Ziyang Huang, Xiaowei Yuan, Yiming Ju, Jun Zhao, Kang Liu

arXiv:2505.07596v1cs.CLcs.AI

TL;DR

RL-based search agents can underuse internal knowledge, while retrieval may add redundancy, conflicts, and latency. IKEA uses knowledge-boundary-aware RL to coordinate parametric knowledge and external search; it outperforms baselines across multiple tasks, reduces retrievals, and generalizes to out-of-distribution data.

  • Problem

    RL-based search agents often underuse parametric knowledge, while external retrieval can be redundant, conflict with correct internal knowledge, and delay generation.

    Source: §1 · p. 1

  • Method

    IKEA uses RL with a knowledge-boundary-aware reward and balanced training data to prioritize internal knowledge and retrieve when it is insufficient.

    Source: §1 · p. 2

  • Results

    Across multiple single-hop and multi-hop knowledge-reasoning datasets, IKEA outperforms baselines, reduces retrievals versus naive RL, and generalizes to out-of-distribution datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    IKEA demonstrates an adaptive search agent that combines internal and external knowledge while using fewer retrievals than naive RL approaches.

    Source: §1 · p. 2

    The paper identifies more dynamic boundary learning, broader task applicability, and reduced training-resource requirements as future-work needs.

    Source: §7 · p. 10

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) is a common strategy to reduce hallucinations in Large Language Models (LLMs). While reinforcement learning (RL) can enable LLMs to act as search agents by activating retrieval capabilities, existing ones often underutilize their internal knowledge. This can lead to redundant retrievals, potential harmful knowledge conflicts, and increased inference latency. To address these limitations, an efficient and adaptive search agent capable of discerning optimal retrieval timing and synergistically integrating parametric (internal) and retrieved (external) knowledge is in urgent need. This paper introduces the Reinforced Internal-External Knowledge Synergistic Reasoning Agent (IKEA), which could indentify its own knowledge boundary and prioritize the utilization of internal knowledge, resorting to external search only when internal knowledge is deemed insufficient. This is achieved using a novel knowledge-boundary aware reward function and a knowledge-boundary aware training dataset. These are designed for internal-external knowledge synergy oriented RL, incentivizing the model to deliver accurate answers, minimize unnecessary retrievals, and encourage appropriate external searches when its own knowledge is lacking. Evaluations across multiple knowledge reasoning tasks demonstrate that IKEA significantly outperforms baseline methods, reduces retrieval frequency significantly, and exhibits robust generalization capabilities.

1 Introduction

Existing RL search agents can overuse external retrieval while underusing model knowledge, creating redundancy, conflicts, and latency. IKEA trains agents to judge their knowledge boundary and coordinate internal and external knowledge.

1 Introduction

  • Existing search agents underuse parametric knowledge, causing redundant searches, while noisy retrieval can conflict with correct internal knowledge and interrupt generation.The introduction frames adaptive retrieval timing as the central research question.Source: §1 · pp. 1–2
  • An adaptive agent should determine whether each decomposed subquery is within its knowledge boundary, recall internal knowledge, and search externally for questions beyond it.The introduction identifies retrieval timing as the core and notes prior approaches based on external indicators or complex data engineering.Source: §1 · p. 2
  • IKEA uses RL to identify its knowledge boundary, prioritize parametric knowledge, and retrieve external information when internal knowledge is uncertain or insufficient.Its reward encourages correct answers while minimizing unnecessary retrievals and encouraging searches beyond the model’s knowledge; its training set equally mixes questions likely to be answerable internally and questions requiring external knowledge.Source: §1 · p. 2
  • Across single-hop and multi-hop knowledge-reasoning datasets, IKEA outperforms baselines and generalizes to out-of-distribution datasets while reducing retrievals versus naive RL approaches.The cited comparison names Search-R1 as the naive RL baseline.Source: §1 · p. 2
  • The paper’s contributions include IKEA and analyses identifying knowledge-boundary-aware reward design and data construction as key to efficient adaptive search agents.The framework prioritizes parametric knowledge before external retrieval.Source: §1 · p. 2

2 Preliminary

The preliminary section describes multi-turn RL as policy learning over agent trajectories and introduces PPO and GRPO. It also defines the Knowledge Boundary as the distinction between a model’s internal knowledge and its external complement.

2.1 Multi-turn Reinforcement Learning with Verifiable Reward for Large Language Model

  • In multi-turn RL, an agent takes actions and receives observations over rounds; its state concatenates the task and preceding action-observation tokens, and a completed trajectory receives a final reward.The policy is trained on collected trajectories to maximize expected cumulative reward.Source: §2.1 · p. 3
  • PPO optimizes a clipped surrogate loss on action tokens, but requires a separate value model to estimate advantages, adding memory overhead.Observation tokens are masked from the loss because they come from the external environment rather than the LLM.Source: §2.1 · p. 3
  • GRPO uses multiple rollouts per task and relative rewards within each group to compute advantages, avoiding a separate value model.The paper adopts GRPO as its default RL algorithm.Source: §2.1 · p. 3

2.2 Knowledge Boundary of Large Language Model

  • Knowledge Boundary distinguishes a specific LLM’s parameter-held knowledge from its complement in world knowledge; a question is inside or outside it according to where its answer-relevant knowledge resides.Source: §2.2 · p. 3

3 Method

The method frames search as multi-turn interaction and trains IKEA to use internal knowledge when sufficient, turning to retrieval when it is not. Its design combines a knowledge-boundary-aware reward with a balanced easy/hard training set and reinforcement learning.

3.1 Basic Setting

  • In the multi-turn setup, the agent emits reasoning followed by a search query or final answer; retrieved context is inserted as an observation, and completed rollouts receive exact-match rewards.The context tokens are masked from loss calculation because they are supplied by the environment rather than generated by the model.Source: §3.1 · pp. 3–4

3.2 IEKA: Reinforced Internal-External Knowledge Synergistic Reasoning Agent

  • The method targets redundant retrieval, latency, and harmful knowledge conflicts by using parametric knowledge within the agent’s boundary and retrieval beyond it.The paper identifies erroneous external knowledge overriding correct internal knowledge as one potential conflict.Source: §3.2 · p. 4
  • IKEA adaptively uses internal knowledge, retrieved knowledge, or both, with reinforcement learning aimed at synergistic reasoning within and beyond its knowledge boundary.The agent prompt encourages parametric knowledge when the model is confident and external retrieval when it is uncertain.Source: §3.2 · pp. 4–5
  • The reward combines answer correctness with a retrieval-sensitive knowledge-boundary reward, while incorrect trajectory formatting receives a total reward of -1.For correct answers, the boundary reward increases as retrievals decrease; for incorrect answers, it is zero without retrieval and a small value with retrieval.Source: §3.2 · p. 5
  • A 1:1 mix of Qeasy and Qhard training questions is used to encourage balanced internal-knowledge use and external retrieval.Questions answered correctly at least once across N sampled answers are labeled Qeasy; the rest are Qhard.Source: §3.2 · pp. 5–6

4 Experiment

Across the evaluated settings, IKEA combines internal and retrieved knowledge, outperforming baselines while reducing retrievals relative to Search-R1. Training logs show convergence across model initializations and sizes, alongside changes in retrieval counts and response length.

4.1 Setting

  • Tests use easy and hard subsets from two in-distribution and two out-of-distribution sets, evaluating exact match and valid-search counts.Training logs show IKEA and IKEA-Zero reach similar reward levels; larger models start and finish higher and converge faster, while retrieval counts initially rise and then fall.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Overall Results

  • Without parameter updates, IR-COT improves Hard-task performance but degrades Easy-task performance, while FLARE’s few retrievals yield performance similar to Direct.The passage attributes IR-COT’s Easy-task degradation to knowledge conflicts and says un-finetuned models struggle to decide when to retrieve.Source: §4.2 · p. 7
  • Reinforcement-learning baselines activate internal-only or retrieval-only capabilities, but neither R1 nor Search-R1 effectively integrates the two knowledge sources.R1 significantly improves Easy-subset performance but has limited gains on Hard subsets; Search-R1 addresses internal-knowledge deficits and outperforms other methods such as Iter-Retgen with fewer retrievals.Source: §4.2 · p. 7
  • IKEA improves performance by over 10% compared with R1, primarily on difficult subsets, and significantly reduces retrievals while improving performance compared with Search-R1.The authors also report good performance on two out-of-distribution datasets and effective generalization of the learned knowledge-seeking behavior.Source: §4.2 · p. 7

5 Ablation Study

The ablations show that both reward design and training-data difficulty affect how IKEA balances internal knowledge and retrieval. Removing either reward component or training on only easy or hard data reduces the effectiveness of that balance.

5.1 The effects of reward design

  • Removing either reward component hurts the balance between answer quality and retrieval: without rkb, EM is similar but retrievals increase, while without rkb−, performance degrades as retrievals decrease.The authors conclude that the reward must balance internal and external knowledge use for synergy.Source: §5.1 · p. 8

5.2 The effects of dataset difficulty

  • Hard-difficulty training produces more effective searches and longer responses than mixed training, which in turn exceeds easy training on both measures.Easy-data training decreases retrieval attempts and response length; on test, both easy and hard variants have substantially lower EM than original IKEA.Source: §5.2 · pp. 8–9
  • The hard-data variant increases retrieval attempts substantially, whereas the easy-data variant decreases them significantly relative to original IKEA.The results indicate that disproportionately favoring either internal or external knowledge hinders full performance.Source: §5.2 · p. 9

6 Related Work

The related work frames reinforcement learning as a way to train LLMs for downstream tasks and autonomous environment interaction. It also reviews knowledge-boundary concepts and approaches for probing them.

RL for LLM-based Agent

  • LLM-based search agents use environments, reward functions, and RL to learn autonomous decisions and interaction, with search engines as a typical application.The section situates this work among PPO, DPO, RLOO, ReMax, and GRPO algorithms.Source: §6 · p. 9

The Knowledge Boundary of LLM

  • Knowledge boundary distinguishes an LLM’s internal parametric knowledge from external knowledge and can be probed with template-based or internal-state methods.Internal-state approaches classify using features such as hidden states or SAEs.Source: §6 · p. 9
  • Knowledge-boundary understanding is described as important for adapting RAG behavior to questions and avoiding hallucinations.Source: §6 · p. 9

7 Conclusion and Limitations

The conclusion presents IKEA as an RL search agent that uses internal knowledge first and retrieves when that knowledge is uncertain or insufficient. It reports improved reasoning efficiency and accuracy, while noting limits tied to dataset construction, probing, reward tuning, and RL cost.

7 Conclusion and Limitations

  • IKEA uses a knowledge-boundary-aware reward and training dataset to prioritize parametric knowledge and search externally when internal knowledge is insufficient or uncertain.The conclusion describes this approach as improving efficiency and accuracy on knowledge-intensive tasks.Source: §7 · p. 9
  • Applicability may be limited by reliance on specific dataset construction and model probing; reward parameters may require grid search, and RL training is computationally expensive.Source: §7 · p. 9
  • Future work could explore more dynamic boundary learning, broader task applicability, and lower training-resource requirements.Source: §7 · p. 10

A IKEA agent template

The appendix introduces a system template for prompting the agent to interact with the environment.

A IKEA agent template

B Dataset Construction

The authors construct in-distribution and out-of-distribution datasets, then divide each into easy and hard subsets using a knowledge-boundary dataset construction method.

C Baselines

The baselines span methods using internal knowledge, retrieved knowledge, or both, alongside approaches trained with supervised, preference-based, or reinforcement learning.

  • Direct answers from internal knowledge, RAG relies solely on retrieved knowledge, and Iter-Retgen [30] iteratively combines parametric and non-parametric knowledge.Source: C Baselines · p. 14
  • IR-COT [35] interleaves retrieval and chain-of-thought steps, while FLARE retrieves when predicted upcoming sentences contain low-confidence tokens.Source: C Baselines · p. 14
  • DeepRAG frames retrieval as a Markov decision process and uses offline trajectories with SFT and DPO; its baselines include prompting, SFT/DPO, and RL methods.Source: C Baselines · p. 14
  • R1 uses RL to activate internal knowledge, whereas Search-R1 uses multi-turn RL to activate external retrieval.Source: C Baselines · p. 14

D Implementation Details

Implementation uses Qwen2.5 models, GRPO training, Wikipedia retrieval, and the FlashRAG and verl frameworks; the system prompt is presented in Table 4.

  • Experiments use Qwen2.5-3B and Qwen2.5-7B Base and Instruct models, with e5-base retrieving from wikipedia2018.Source: D Implementation Details · p. 14
  • GRPO trains IKEA with NQ and HotpotQA data, while FlashRAG reproduces baseline results and verl provides the training framework.Source: D Implementation Details · p. 14
  • Training uses 4,000 easy and 4,000 hard samples per dataset, 16 rollouts per task, 120 steps, and eight A100 GPUs.The learning rate is 5e-7, the warmup ratio is 75%, and the batch size is 256.Source: D Implementation Details · p. 14
  • Table 4 contains the system prompt used to guide IKEA's interaction with the environment.Source: Table 4 · p. 15

NeurIPS Paper Checklist

The checklist records the authors' statements on reproducibility, experimental reporting, ethics, and research scope, with supporting details assigned to paper sections and supplementary materials.

  • The authors state that the paper has no theoretical results and that experimental details are provided in Sections A, B, and D.Source: NeurIPS Paper Checklist · p. 17
  • The authors say data and code are included in supplementary materials and that training and test details are provided in Sections A, B, and D.Source: NeurIPS Paper Checklist · pp. 18–19
  • The authors report no error bars or statistical significance information because repeating experiments is too computationally expensive for their lab.Source: NeurIPS Paper Checklist · p. 19
  • The authors state that the work conforms to the NeurIPS Code of Ethics and that they expect no significant social impact.Source: NeurIPS Paper Checklist · p. 20
  • They report no high-risk release requiring safeguards, and say that used assets are credited and new-asset documentation is in the code repository.Source: NeurIPS Paper Checklist · pp. 20–21
  • The authors state that the work involves no human subjects and that LLMs are not an important, original, or non-standard part of core method development.Source: NeurIPS Paper Checklist · pp. 21–22
Loading 2505.07596v1…