Source-linked AI summary
Bridging the Preference Gap between Retrievers and LLMs
Zixuan Ke, Weize Kong, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky
TL;DR
RAG research often treats retrievers and LLMs independently, despite a preference gap between retrieved information and the context LLMs use effectively. The paper trains a bridge between frozen components with supervised and reinforcement learning, reporting gains across QA and personalized generation. Its generalization across datasets and domains remains limited.
Problem
Most RAG work studies retrievers and LLMs separately, although their preferences for ranking and selecting information may differ.
Source: §1 · p. 1Method
Results
BGM enhances QA and personalized generation across datasets, and its modified passages surpass strong retrievers and baseline reranking models.
Source: §1 · p. 2Takeaways & Limitations
The experiments support BGM as a bridge approach for improving downstream RAG performance across QA and personalized generation tasks.
Source: §1 · p. 2BGM has limited generalization across datasets and domains, and its supervised silver passage sequences are synthesized using greedy search.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) have demonstrated superior results across a wide range of tasks, and Retrieval-augmented Generation (RAG) is an effective way to enhance the performance by locating relevant information and placing it into the context window of the LLM. However, the relationship between retrievers and LLMs in a RAG is still under-investigated. Most existing work treats the retriever and the LLM as independent components and leaves a gap between retrieving human-"friendly" information and assembling a LLM-"friendly" context. In this work, we examine a novel bridge mechanism. We validate the ranking and selection assumptions of retrievers in the context of RAG and propose a framework that chains together supervised and reinforcement learning to train a bridge model that optimizes the connection between the retriever and the LLM. Empirical results demonstrate the effectiveness of our method in both question-answering and personalized generation tasks.
1 Introduction
RAG commonly treats retrievers and LLMs separately, despite their differing preferences for information ordering and selection. The paper introduces BGM, a bridge trained with supervised and reinforcement learning, and reports improvements across QA and personalized generation tasks.
1 Introduction
- Most RAG work studies retrievers and LLMs separately, although human-oriented retrieval preferences may not align with the preferences LLMs need for accurate predictions.The paper calls this discrepancy the preference gap.Source: §1 · p. 1
- Top-5 passage-order randomization changes RAG performance by around 1%, while changing the selected top-1 passage produces variation exceeding 5%.The authors interpret this as evidence that selection can matter more than ranking in these experiments.Source: §1 · p. 2
- BGM keeps the retriever and LLM fixed and trains an intermediate bridge model to adapt retrieved information to LLM preferences.The paper motivates this design as an alternative to expensive LLM fine-tuning or difficult production-retriever adjustment.Source: §1 · p. 2
- BGM’s flexible passage-sequence formulation can potentially include repetition, which may help LLMs weight context items despite repetition being considered detrimental for retrieval systems.The authors identify repetition as a possible preference difference between retrievers and LLMs.Source: §1 · p. 2
- BGM enhances QA and personalized generation across public QA, Amazon reviews, and private email tasks, surpassing strong retrievers and baseline reranking models.The evaluation spans diverse datasets and downstream tasks.Source: §1 · p. 2
- The seq2seq bridge jointly reranks and selects retrieved information, and the proposed training scheme combines supervised learning with reinforcement learning.The framework is intended to adapt retrieved information into an LLM-friendly sequence.Source: §1 · p. 2
2 Related Work
Prior RAG approaches bridge retriever–LLM preference differences by updating the retriever, the LLM, or both, but these choices can face practical constraints. BGM instead connects frozen components with a lightweight bridge model.
2 Related Work
- BGM connects a frozen retriever and frozen LLM through a lightweight bridge model, unlike approaches that update either or both components.The figure caption states this design makes BGM applicable to large LMs and off-the-shelf retrievers.Source: Figure 2 · p. 3
- Joint fine-tuning of retrievers and LLMs is widely used, but prior work in this family is mostly based on relatively small LMs below 1B parameters.Examples include Atlas and RAG, which train retriever-related components and LLMs together.Source: §2 · p. 3
- Updating only LLMs can avoid retriever updates, while updating only retrievers suits settings where LLMs are black-box APIs; both strategies have practical limits.Retriever updates can require costly index refreshes, and retriever-only methods described here cannot perform sample-level selection.Source: §2 · p. 3
- Retriever-only methods such as REPLUG and RECOMP adapt retrieved content, but the paper says they select top passages using a fixed threshold rather than sample-level selection.The cited passage characterizes this as a limitation of that family of approaches.Source: §2 · p. 3
3 Problem Formulation
The paper formulates retrieval as supplying ranked passages to a bridge model that adapts them for a frozen LLM. The seq2seq bridge takes the query and retrieved passages and outputs an adapted passage sequence.
3 Problem Formulation
- A typical frozen dense retriever ranks corpus passages by similarity to the input and returns the top-k passages.The passage representations and input are encoded, and their cosine similarity is used for comparison.Source: §3 · p. 4
- The bridge model takes the query and retrieved passages as input and produces adapted passages for the LLM.The adapted sequence is concatenated with the query and passed to the LLM for downstream prediction.Source: §3 · p. 4
- The seq2seq formulation represents ranking through generation order, selection through where generation ends, and repetition through generating a passage ID again.The output sequence can be shorter or longer than the retrieved set because of selection and repetition.Source: §3 · p. 4
- Each retrieved passage receives a unique sentinel ID, allowing the bridge to generate passage IDs rather than rewrite passage content.The generated IDs are mapped back to their corresponding passages for downstream processing.Source: §3 · p. 4
4 Training the Bridge Model
The bridge model is trained by chaining supervised learning (SL) and reinforcement learning (RL): SL initializes ranking and selection, while RL optimizes the model for downstream performance. SL uses greedy search to create silver passage sequences, but sparse supervision can leave SL-only performance mixed.
4 Training the Bridge Model
- BGM chains SL and RL: SL provides an initial model for ranking and selection, then RL optimizes it for the downstream task.The framework uses a seq2seq bridge model that takes a query and passages with prepended passage IDs as input and outputs passage IDs.Source: §4 · p. 4, Figure 3 · p. 5
4.1 Supervised Learning
- Greedy search builds a silver passage sequence (SPS) by incrementally adding the passage that maximizes downstream task performance, stopping when no improvement remains.The empty sequence is evaluated as performance without retrieval augmentation; the resulting SPS is used for supervised training with cross-entropy loss.Source: §4.1 · pp. 4–5
4.2 Reinforcement Learning
- SL alone yields mixed performance, attributed to sparse supervision and the lack of end-to-end training on downstream results.Source: §4.2 · p. 5
- RL trains the bridge model end to end using downstream task performance as reward, with passage IDs as actions and sampled sequences providing supervision beyond silver sequences.Unlike SL's permutations or deletions, RL can also explore operations such as repeating passages; the reward can use measures such as Exact Match or BLEU, and PPO is one possible algorithm.Source: §4.2 · p. 5
5 Experiments
The experiments evaluate BGM on open-domain QA and personalized generation against retrieval and non-retrieval baselines. BGM outperforms the four baselines across four datasets, while ablations highlight the importance of dynamic selection, supervised initialization, and training/evaluation alignment.
5.1 Datasets and Baselines
- The evaluation spans NQ and multi-hop HotpotQA, plus personalized completion on user-authored Email and Book documents.The Email dataset contains private conversations described as unlikely to appear in LLM pretraining.Source: §5.1 · pp. 5–6
- The baselines include GTR, randomized GTR ordering, PSR reranking, and Naive generation without retrieval augmentation.PSR lacks dynamic selection, distinguishing it from BGM's evaluated capabilities.Source: §5.1 · p. 6
5.2 LLM and Hyperparamters
- The experiments use T5-XXL as the bridge model and frozen PaLM2-S as the LLM, with EM or BLEU also serving as the RL reward.The retrieved-passage limit K is set to 5.Source: §5.2 · p. 6
5.3 Evaluation Results and Analysis
- BGM outperforms all four baselines across all four datasets, indicating that it adapts retrieved passages effectively.The comparison covers the overall performance reported in Table 2.Source: §5.3 · p. 6, Table 2 · p. 6
- Thresholding PSR passages, including top-k variants, consistently performs below BGM, supporting joint reranking and dynamic selection.The authors conclude that a manual PSR threshold is insufficient for the intended objective.Source: §5.3 · p. 7, Table 3 · p. 7
- Greedy-search passage sequences for supervised learning outperform GTR- and PSR-derived sequences; finding better sequences remains future work.The authors report that sequence quality significantly affects downstream performance.Source: §5.3 · p. 7, Table 4 · p. 7
- BGM with supervised learning alone underperforms the full model on NQ, HotpotQA, and Email, and sometimes falls below GTR.The authors describe supervised learning alone as inadequate and note that removing it would likely leave RL poorly initialized with an excessively large search space.Source: §5.3 · pp. 7–8, Table 5 · p. 7
- All tested bridge-model sizes outperform GTR, with larger models performing better; on NQ and HotpotQA, BGM also beats baselines with PaLM2-XXS.For PaLM2-XXS, Email and Book BLEU scores were below 1% and were not reported.Source: Table 6 · p. 7, Table 7 · p. 7, §5.3 · p. 8
6 Conclusion
The paper argues that retrievers and LLMs have a preference gap affecting ranking and selection in RAG. It proposes BGM, a seq2seq bridge trained with supervised and reinforcement learning, and reports that extensive experiments validate its effectiveness.
- BGM adapts a frozen retriever’s output for frozen LLMs by framing the bridge task as sequence-to-sequence generation.Source: §6 · p. 8
- The bridge model jointly addresses ranking and selection, which the paper identifies as affected by the preference gap in RAG.Source: §6 · p. 8
- BGM chains supervised and reinforcement learning for dense supervision and end-to-end training.Source: §6 · p. 8
- The paper reports that extensive experiments validate BGM’s effectiveness.Source: §6 · p. 8
7 Limitations
BGM has limitations in generalizing across datasets and domains, and its supervised-learning passage sequences are synthesized using greedy search.
- BGM is limited in its generalization across datasets and domains.The authors identify this limitation by reference to their evaluation in Section 5.3.Source: §7 · p. 9
- The supervised-learning silver passage sequences are synthesized with greedy search, and alternative synthesis approaches remain for future exploration.Source: §7 · p. 9
8 Ethics Statement
The authors do not anticipate negative consequences for individuals from this research and foresee no notable ethical implications. They state that system failure would primarily result in suboptimal LLM performance.
A Case Studies
The case-study material compares GTR, PSR, and BGM on NQ examples, including questions about seafloor spreading and FaZe Clan. The seafloor example shows BGM predicting an answer matching the stated answer, while GTR and PSR predict a different answer.
- For the seafloor-spreading question, GTR and PSR predict “the upper mantle,” while BGM predicts “volcanic activity,” matching one of the stated answers.Source: A Case Studies · pp. 11–12
- For the FaZe Clan subscribers question, GTR predicts Tfue, whereas the stated answer is FaZe Rug.Source: A Case Studies · p. 12
- The cited table is described as comparing GTR, PSR, and BGM cases from the NQ dataset.Source: Table 9 · p. 13