PaperlayerView exact revision on arXiv

Source-linked AI summary

BRIDGE: Bilevel Retrieval-Credit-Aware Agentic Reinforcement Learning

Quan Xiao, Mingda Liu, Gaowen Liu, Katsuki Fujisawa, Tianyi Chen

arXiv:2609.36505v1cs.AIcs.LGmath.OC

TL;DR

Agentic RL methods often exclude retrieved evidence from the learning signal, leaving a gap in how retrieval failures receive credit. BRIDGE models retrieval-augmented agentic RL as a bilevel problem and co-adapts retrieval and policy while preserving their hierarchy; it reports improved benchmark averages across open-domain and medical QA.

  • Problem

    Most ARL methods mask retrieved information from the RL loss, so failures involving missing or misleading evidence are credited to the LLM policy rather than the retriever.

    Source: §1 · p. 1

  • Method

    BRIDGE formulates retrieval-augmented ARL as bilevel optimization and uses a memory-efficient first-order method to adapt the retriever with RL credit and RAG likelihood, then optimize the policy with RL.

    Source: §1 · p. 2

  • Results

    BRIDGE has the highest overall average on open-domain QA, improving EM over VT-Search by 7.1 points with 3B and 2.3 with 7B, and leads medical benchmarks in answer accuracy and reasoning quality.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The reported gains are largest on multi-hop QA and with smaller backbones, while the bilevel formulation preserves the retriever-before-policy hierarchy during online adaptation.

    Source: §5 · p. 10

Abstract

from arXiv · show

Agentic reinforcement learning (ARL) with verifiable rewards improves the ability of large language models (LLMs) to tackle knowledge-intensive tasks by learning to interleave search and reasoning. However, most existing ARL methods optimize only LLM-generated tokens and treat retrieved evidence as environment observations. This creates an information-credit gap: failures caused by missing or misleading evidence are attributed to the LLM policy rather than to the retriever, which motivates training the LLM and the retriever jointly. In this paper, we show that retrieval and LLM policy learning are order-sensitive: adapting the retriever before optimizing the policy yields a larger reward gain than the reverse order. To preserve this hierarchy while allowing both components to co-adapt, we formulate retrieval-augmented agentic RL as a bilevel optimization problem. To solve it efficiently, we introduce BRIDGE, a memory-efficient first-order bilevel method motivated by a loss-landscape analysis of the RL and retrieval objectives. Across seven open-domain QA benchmarks, BRIDGE achieves the highest average accuracy with both 3B and 7B backbones, improving the multi-hop average over the strongest baseline by 9.6 and 3.4 EM points, respectively. It also achieves the best averaged answer accuracy and reasoning quality across medical QA benchmarks.

1 INTRODUCTION

Existing agentic RL often leaves retrieved evidence outside the policy’s learning signal, while prior retriever adaptation and joint-optimization work does not address the ordered dependency between retrieval and policy learning. BRIDGE formulates retrieval-augmented agentic RL as a bilevel problem and introduces a memory-efficient first-order method to co-adapt the retriever and LLM while preserving that hierarchy.

1 INTRODUCTION

  • Most ARL methods optimize LLM-generated tokens but mask retrieved information from the RL loss, leaving retrieval failures credited to the LLM policy rather than the retriever.The gap is especially consequential in multi-hop QA, where reasoning depends on retrieval exposing evidence spread across documents.Source: §1 · p. 1
  • Retriever tuning before policy optimization yields larger gains than the reverse order, motivating a bilevel formulation that preserves this hierarchy during joint adaptation.Prior work either adapts retrieval for a frozen generator or jointly optimizes retriever and policy without considering their ordered dependency.Source: §1 · pp. 1–2
  • BRIDGE updates a query-side retriever LoRA using retrieval-turn RL credit and RAG likelihood, then optimizes the LLM policy with RL credit.Its peak memory per retriever update is 49.7% lower than the two-state first-order estimator, with a gradient estimate closely aligned to that estimator.Source: §1 · p. 2
  • 7.1 and 2.3 points: BRIDGE improves overall average EM over VT-Search on open-domain benchmarks with 3B and 7B backbones, respectively, and leads medical QA averages in answer accuracy and reasoning quality.The evaluation covers seven open-domain and five medical QA benchmarks.Source: §1 · p. 2

1.1 RELATED WORK

  • Related work covers tool-use ARL and RAG optimization, but the cited approaches do not address multi-turn RL with search queries and answers induced by an evolving policy.BRIDGE complements joint policy–retriever optimization by explicitly modeling their hierarchical dependency.Source: §1.1 · p. 3

2 PRELIMINARIES: MULTI-TURN TOOL-USE RL AND RAG

Multi-turn tool-use RL models a rollout as alternating LLM-generated actions and tool-returned observations, with the trajectory depending on both policy and retrieval parameters. The section describes verifiable-reward policy optimization and a dense retrieval setup whose RAG likelihood trains the retriever to favor documents useful for answering.

2.1 AGENTIC RL WITH TOOL USE

  • A rollout alternates LLM-generated action segments with environment observations returned by external tools, ending in an LLM action.For search, reasoning, search calls, and final answers use distinct special tokens, while retrieved documents appear in an information block.Source: §2.1 · p. 3
  • The policy is optimized with verifiable rewards and KL regularization; the binary exact-match outcome reward is 1 for a correct answer and 0 otherwise.The rollout distribution depends on both the LLM policy parameters and retrieval parameters.Source: §2.1 · p. 3
  • GRPO compares multiple rollouts for the same prompt, normalizes their rewards within the group, and assigns the resulting advantage to action tokens.The multi-turn surrogate averages the clipped token objective over action tokens and normalizes the trajectory-level advantage by rollout length.Source: §2.1 · pp. 3–4

2.2 TOOL-USE RAG AND RAG LIKELIHOOD

  • A dense retriever scores each trajectory-prefix query against fixed document representations, then returns the highest-scoring documents as the tool observation.The trainable query encoder and fixed document encoder define the relevance score; retrieval changes subsequent policy context and thus the trajectory distribution.Source: §2.2 · p. 4
  • The retriever’s RAG negative log-likelihood marginalizes the correct answer over top-k documents, encouraging higher ranks for evidence useful to answer generation.The top-k candidate set also supplies the RL outcome-credit surrogate, with k chosen larger than the number of documents returned to the LLM.Source: §2.2 · pp. 4–5

3 BILEVEL RETRIEVAL-AUGMENTED AGENTIC RL

Retrieval and policy optimization are order-sensitive, motivating a bilevel formulation that prioritizes retriever adaptation while allowing both components to co-adapt. BRIDGE uses a memory-efficient first-order method to implement this hierarchy.

3.1 BILEVEL FORMULATION

  • Retriever-first tuning outperforms the reverse order, motivating a bilevel formulation that keeps retrieval optimization lower-level as the policy changes.The lower-level RAG objective finds the retriever best suited to the current policy; the upper-level RL objective optimizes that policy with the resulting retriever.Source: §3.1 · p. 5

3.2 MEMORY EFFICIENT BILEVEL ALGORITHM DESIGN VIA LANDSCAPE ANALYSIS

  • The two-state first-order estimator is memory-intensive because it tracks separate retriever states, optimizer states, and retrieval logs.The implicit gradient also involves an inverse Hessian and mixed Jacobian, making direct computation infeasible for large neural networks.Source: §3.2 · pp. 5–6
  • The RAG objective changes substantially along policy directions, while the RL objective is relatively flat along retriever directions across a wide range of policies.This landscape analysis motivates omitting two correction terms when the penalized and unpenalized retriever solutions are sufficiently close.Source: §3.2 · p. 6, Figure 3 · p. 6
  • BRIDGE updates a query-side LoRA retriever with retrieval-turn RL credit and RAG likelihood, then updates the LLM policy using RL on the same rollouts.It tracks one penalized retriever state and updates the retriever adapter periodically.Source: §3.2 · pp. 6–7
  • An idealized version of BRIDGE's updates with exact gradients converges to an approximate stationary point of the bilevel problem.The method combines rollout-level RL feedback with a RAG signal that scores how strongly documents support the correct answer.Source: §3.2 · pp. 7–8

4 EXPERIMENTS

Across open-domain QA tasks, BRIDGE has the highest overall average at both model scales, with particularly large gains on multi-hop QA. It also improves medical QA reasoning quality and answer accuracy over VT-Search.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers seven open-domain and five medical benchmarks, with open-domain training on NQ and HotpotQA and five other open-domain benchmarks testing out-of-domain generalization.Open-domain evaluation uses exact match; medical QA also evaluates reasoning quality.Source: §4.1 · p. 8

4.2 OPEN-DOMAIN QA TASKS

  • BRIDGE achieves the highest overall average at both scales, with its largest margins on multi-hop QA and larger gains with the 3B backbone.It trails the best baseline only on PopQA at both scales and TriviaQA at 7B.Source: §4.2 · p. 8
  • Online retriever–policy variants do not surpass sequential RAG-then-RL, while BRIDGE preserves that hierarchy and improves on the sequential baseline.All ablation variants outperform the RL-only baselines Search-R1 (Base) and VT-Search.Source: §4.2 · p. 8
  • BRIDGE's estimator has higher cosine similarity and lower relative gradient error than Drop across tested penalty coefficients, while reducing mean peak memory by 49.7% and update time by 55.8% versus the two-state estimator.Training rewards improve faster than Search-R1; at comparable training time, BRIDGE is more accurate than VT-Search on all four multi-hop datasets despite using twice as many GRPO rollouts.Source: §4.2 · pp. 8–9

4.3 EVALUATION ON MEDICAL QA BENCHMARKS

  • BRIDGE has the highest reasoning quality on all five medical benchmarks and improves macroaveraged reasoning quality and answer accuracy over VT-Search.The evaluation uses Qwen2.5-7B-base; table cells report GPT-5.6 Luna-judged reasoning quality with answer accuracy in parentheses.Source: Table 2 · p. 9, §4.3 · p. 10

5 CONCLUSION

The conclusion emphasizes the order-sensitive hierarchy between retriever and LLM optimization and presents BRIDGE as a way to co-adapt them. An appendix section derives the retrieval RL gradient and states a convergence guarantee.

5 CONCLUSION

  • Adapting the retriever before optimizing the policy yields larger gains than reversing the order; BRIDGE preserves this hierarchy while enabling online co-adaptation.Across seven open-domain and five medical QA benchmarks, it improves answer accuracy and medical reasoning quality, with largest gains on multi-hop QA and smaller backbones.Source: §5 · p. 10

A RETRIEVAL GRADIENTS AND THEORETICAL GUARANTEE

  • The retrieval-gradients appendix derives the retrieval RL gradient and states a convergence guarantee for BRIDGE.The supplied section statement does not specify the guarantee's conditions or result.Source: A RETRIEVAL GRADIENTS AND THEORETICAL… · p. 15

A.1 DERIVATION OF RETRIEVAL GRADIENT

The appendix derives retrieval-credit and RAG gradients for BRIDGE, then analyzes convergence of its periodic retriever and policy updates under stated assumptions. The analysis links approximate solutions to stationarity of the original bilevel objective, while noting that the implemented LoRA and empirical-gradient setting is outside the theorem's scope.

A.1 DERIVATION OF RETRIEVAL GRADIENT

  • Under stochastic retrieval, η affects trajectory probability only through retrieval-turn probabilities, so its log-gradient sums their log-probability gradients.The derivation fixes θ and holds the recorded trajectory fixed.Source: §A.1 · p. 16
  • BRIDGE’s retrieval-credit surrogate distributes trajectory-level GRPO advantage across retrieved documents using answer-likelihood weights and document-level importance ratios, with probability-ratio clipping.The document-level ratio replaces token probabilities with document probabilities; the advantage sets credit sign and the weights allocate it by answer support.Source: §A.1 · pp. 16–17
  • The retriever update combines the clipped RL gradient with γ times the RAG gradient, while LoRA updates adapt the query encoder and keep the base query and document encoders frozen.The adapted query-encoder weights are parameterized as η = η0 + AB.Source: §A.1 · p. 17

A.2 CONVERGENCE GUARANTEE OF BRIDGE

  • Under Assumptions 1–3, periodic retriever updates with per-round policy updates are analyzed as BRIDGE’s central alternating scheme, supporting approximate stationarity without explicit hypergradients.The analysis uses exact population gradients; extending it to the implemented LoRA parameterization and empirical surrogate gradients is beyond scope.Source: §A.2 · pp. 17–18

If we choose

  • With R ≳ T/ϵ iterations and the stated penalty choice, idealized BRIDGE updates yield an O(ϵ^(p−1)) stationary point of the original bilevel problem.The guarantee assumes the stated flatness and regularity conditions.Source: §A.2 · p. 17, If we choose · p. 18

B.1 EXPERIMENTAL DETAILS

Experiments train BRIDGE and baselines from matched LLM initializations on a shared NQ–HotpotQA training split, using GRPO rollouts at both 3B and 7B scales. BRIDGE adapts a query-side E5 LoRA retriever, with vLLM rollouts and FAISS retrieval.

B.1 EXPERIMENTAL DETAILS

  • All methods start from the same specified LLM and train for 250 steps at both 3B and 7B, using a shared 169,615-example NQ and HotpotQA split.GRPO uses 512-question batches and eight trajectories per question.Source: §B.1 · p. 20
  • BRIDGE adapts rank-16 LoRA on the E5-base-v2 query encoder, with a 10−5 retriever learning rate and γ = 0.25 weighting the RL and RAG terms 0.8 and 0.2.Retrieval and posterior temperatures are 0.4 and 0.5, respectively.Source: §B.1 · p. 20
  • The implementation uses verl-tool, vLLM for language-model rollouts, and FAISS for dense retrieval.Source: §B.1 · p. 20

B.2 COMPONENT-WISE ABLATIONS

The component ablation compares BRIDGE with RAG-then-RL, while the full loss landscapes examine how RAG and RL objectives vary across policy and retriever directions at three BRIDGE checkpoints.

B.2 COMPONENT-WISE ABLATIONS

  • 31.3 multi-hop and 40.3 overall EM for BRIDGE exceed RAG-then-RL’s 23.8 and 35.9 with the same backbone and rollout group size.Table 3 reports component-wise ablations; its M-Avg. and Avg. denote multi-hop and overall QA averages.Source: Table 3 · p. 21, §B.2 · p. 21

B.3 FULL LOSS LANDSCAPES

  • Across initial, middle, and final checkpoints, RAG and RL losses vary more along θ than η; the RL objective also becomes flatter along θ during training.Flatness of the RL loss along η is the property required by the flatness assumption in Appendix A.2.Source: §B.3 · p. 21

B.4 RUNTIME–ACCURACY COMPARISON

The runtime comparisons examine whether retriever adaptation or additional policy rollouts better use comparable compute, and compare BRIDGE with joint-training and Search-R1 baselines across backbones. BRIDGE is more accurate in the reported comparisons, with runtime and backbone-specific qualifications.

B.4 RUNTIME–ACCURACY COMPARISON

  • 31.3 M-Avg. in 79.4 hours for BRIDGE exceeds doubled-rollout VT-Search’s 23.3 M-Avg. in 82.4 hours.Table 4 compares runtime and accuracy on Qwen2.5-3B-base using 4×A100 GPUs.Source: Table 4 · p. 21, §B.4 · p. 21
  • At 3B, BRIDGE is more accurate than Agentic-R on every dataset while taking 22.5 versus 43.8 hours; it also leads Agentic-R in multi-hop and overall averages at both scales.At 7B, TriviaQA is the exception to BRIDGE’s dataset-level advantage.Source: §B.4 · p. 21, Table 5 · p. 22

B.5 PERFORMANCE ACROSS BACKBONES

  • BRIDGE exceeds Search-R1 in overall EM for every matched Base/Instruct backbone; instruction tuning raises its 3B overall EM from 40.3 to 41.5, while 7B Base performs better.Table 6 reports answer EM and the multi-hop and all-benchmark averages.Source: Table 6 · p. 23, §B.5 · p. 23
  • Qwen3-14B has the highest score on every benchmark, while DeepSeek-R1-Distill-Llama-8B reaches 46.9 overall and 41.1 multi-hop EM without reported baselines.These rows demonstrate applicability to additional backbones rather than comparative performance.Source: §B.5 · p. 23, Table 6 · p. 23

C.1 TRAINING DETAILS

The appendix specifies BRIDGE’s training and retriever settings, its medical reasoning-quality evaluation, and the presentation of qualitative trajectory case studies.

C.1 TRAINING DETAILS

  • Policy training uses Qwen2.5-7B-Base, GRPO, and 169,615 NQ and HotpotQA examples, with 16 trajectories sampled per question for 100 steps.The policy learning rate is 10^-6; batches contain 512 questions.Source: §C.1 · p. 23
  • BRIDGE trains rank-16 LoRA adapters on the E5-base-v2 query encoder and exposes the top 3 retrieved documents to the LLM.Retriever training uses a 40-document candidate set; the corpus combines Wikipedia, PubMed, and textbooks.Source: §C.1 · p. 23

C.2 EVALUATION PROTOCOL

  • Reasoning quality is scored ordinally from 0 to 4, with 0 indicating absent or unusable reasoning and 4 correct, coherent reasoning supporting the reference answer; scores are multiplied by 25 for a 0–100 scale.Source: §C.2 · p. 24

D QUALITATIVE TRAJECTORY CASE STUDIES

  • The qualitative trajectory case studies compare BRIDGE with VT-Search using abridged or author-condensed retrieved passages, with answer-relevant spans marked.Final-choice correctness is evaluated separately from reasoning quality.Source: D QUALITATIVE TRAJECTORY CASE STUDIES · p. 24, §C.2 · p. 24

D.1 RAG PROVIDES USEFUL LEARNING SIGNALS UNDER ZERO REWARD

The case studies show how answer-likelihood-based retrieval credit can distinguish useful documents when correctness-based GRPO feedback is absent, alongside examples of evidence coverage, evidence use, and query refinement.

D.1 RAG PROVIDES USEFUL LEARNING SIGNALS UNDER ZERO REWARD

  • BRIDGE can retrieve answer-relevant facts yet still answer incorrectly and receive zero reward, as in the Matthieu Chedid case.A second search retrieves that Chedid’s father is Louis Chedid, but the answer remains wrong.Source: §D.1 · p. 24
  • At training step 100, 85.26% of audited retrieval turns have zero GRPO advantage, while answer-likelihood credit assigns the most-favored document an average 88% of the weight.Retrieval probabilities alone have a mean maximum of 0.1093 over ten candidates; the RAG likelihood term can still distinguish documents.Source: §D.1 · p. 25

D.2 OPEN-DOMAIN QA CASE STUDIES

  • In open-domain cases, BRIDGE retrieves facts needed for multi-hop answers, while VT-Search may miss comparison evidence or retrieve a distractor.Examples include comparing birth years and species counts.Source: §D.2 · p. 25, Table 8 · p. 26, Table 9 · p. 27
  • BRIDGE refines ambiguous searches with disambiguating terms, whereas VT-Search follows irrelevant retrieval drift.The examples contrast searches for the Hang-On video game and Greek mythology’s Chimera.Source: §D.2 · p. 25, Table 12 · p. 30, Table 13 · p. 31

D.3 RETRIEVAL BEHAVIOR ACROSS TRAINING CHECKPOINTS

The case studies contrast BRIDGE’s evolving search behavior across training checkpoints with medical examples involving targeted retrieval and reasoning over retrieved evidence.

D.3 RETRIEVAL BEHAVIOR ACROSS TRAINING CHECKPOINTS

  • At a later checkpoint, BRIDGE uses the first retrieval to form an actor-specific second query and retrieves missing advertisement evidence, while VT-Search remains with one broad query.At the early checkpoint, both methods retrieve the episode clue but not the advertisement clue.Source: §D.3 · pp. 32–33, Table 14 · p. 32, Table 15 · p. 33

D.4 MEDICAL QA CASE STUDIES

  • Medical case studies show BRIDGE retrieving brown-tumor evidence for hyperparathyroidism and searching specifically for haloperidol-induced dystonia before selecting benztropine.The corresponding VT-Search cases retain multiple myeloma and select baclofen.Source: Table 16 · p. 34, Table 17 · p. 35
  • In a dental question, both methods retrieve the same passages, but BRIDGE selects reduced leverage while VT-Search chooses the cingulum distractor.Source: Table 18 · p. 36

Related papers

Loading 2609.36505v1…