Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

Bowen Jin; Jinsung Yoon; Jiawei Han; Sercan O. Arik

Source-linked AI summary

Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

Bowen Jin, Jinsung Yoon, Jiawei Han, Sercan O. Arik

arXiv:2410.05983v1Oct 8, 2024cs.CLcs.AIcs.LG

TL;DR

Long-context RAG lacks sufficient evidence on how larger retrieval sets and hard negatives affect LLM performance. The paper analyzes this issue and proposes retrieval reordering and RAG-specific fine-tuning, finding that performance can decline as more passages are added and that stronger retrievers may provide more harmful hard negatives. It concludes that evaluation should incorporate retriever-specific hard negatives.

Problem
Long-context RAG lacks sufficient evidence on how increasing retrieved context and retriever-specific hard negatives affect LLM performance.
Method
The paper analyzes hard negatives and develops retrieval reordering, implicit robustness fine-tuning, and explicit relevance fine-tuning with intermediate reasoning.
Results
Performance initially improves but then declines with more retrieved passages, while stronger retrievers can produce more harmful hard negatives; the proposed approaches improve long-context RAG accuracy and robustness.
Takeaways & Limitations
Robust long-context RAG requires attention to hard-negative characteristics and evaluation settings, not only retrieval quantity or precision.
Takeaways & Limitations
Existing long-context benchmarks may not adequately capture hard-negative challenges because they predominantly use random negatives.

Abstract

from arXiv · show

Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases. This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key contributor. To mitigate this and enhance the robustness of long-context LLM-based RAG, we propose both training-free and training-based approaches. We first showcase the effectiveness of retrieval reordering as a simple yet powerful training-free optimization. Furthermore, we explore training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, demonstrating their capacity for substantial performance gains. Finally, we conduct a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.

1. Introduction

Long-context LLMs create opportunities to provide more retrieved information, but this paper finds that additional passages can eventually reduce RAG performance. It analyzes hard negatives and proposes retrieval reordering plus two RAG-specific fine-tuning approaches.

Motivation: Longer-context LLMs enable larger retrieved inputs, but optimizing RAG for them remains under-explored.Directly supplying entire large corpora remains infeasible and costly.
Findings: RAG performance initially improves but then declines as more passages are retrieved, so additional context does not guarantee better outcomes.The paper attributes this pattern partly to irrelevant passages that can mislead generation.
Findings: Hard negatives from stronger retrievers can confuse LLM generation more than hard negatives from weaker retrievers.This finding challenges the assumption that stronger retrieval alone mitigates long-context RAG problems.
Methods: The paper proposes retrieval reordering, implicit robustness fine-tuning, and explicit relevance fine-tuning with intermediate reasoning.These methods span training-free and training-based approaches.
Contributions: The study systematically examines hard negatives and evaluates how data distribution, retriever choice, and training context length affect RAG-specific tuning.The contribution summary frames this as a comprehensive study of robust long-context RAG.

2. Related Work

Prior work improves retrievers, generators, or RAG adaptation, while long-context evaluation often uses random negatives that do not reflect hard-negative challenges. This paper therefore takes a holistic view of long-context LLMs as RAG generators.

RAG foundations: Traditional RAG combines a retriever that selects corpus information with a generator that synthesizes an answer.RAG is used to provide external knowledge for knowledge-intensive tasks.
Research positioning: Prior research commonly improves retrievers or generators in isolation, whereas this paper analyzes the entire long-context RAG system.The paper focuses on challenges and opportunities created by long-context LLM generators.
Evaluation gap: Long-context benchmarks often use random negatives, which differ from the hard negatives encountered in real-world RAG.Multi-document studies may also assume one golden document and random negatives rather than multiple relevant passages and hard negatives.
RAG adaptation: Existing RAG adaptation methods include instruction tuning, robustness tuning with retrieved passages, dual instruction tuning, and frameworks such as Self-RAG.These approaches motivate further study of RAG-specific adaptation for long-context LLMs.

3. Challenges of Long context LLMs in RAG

Across long-context RAG experiments, adding retrieved passages produces gains followed by degradation, especially with stronger retrievers. The analyses identify hard negatives as a central challenge and show that common random-negative evaluations can miss it.

3.1. Retrieved context size: On NQ, four long-context LLMs are evaluated with BM25 and e5 while the number of retrieved passages varies.e5 has Recall@40 0.90 versus 0.73 for BM25.
3.1. Retrieved context size: With e5, performance initially improves and then sharply declines or plateaus as more passages are added; BM25 usually improves or declines only slightly.The resulting pattern is an inverted U for strong retrieval and more consistent, potentially limited improvement for weak retrieval.
3.2. Retrieval quality and LLM capabilities: Increasing retrieved passages raises recall but lowers precision, while RAG accuracy remains below recall across retrieval sizes.Relevant information can therefore be present without producing a correct answer.
3.2. Retrieval quality and LLM capabilities: Despite higher precision, e5 causes more performance degradation than BM25, showing that precision alone does not capture hard-negative effects.The characteristics of irrelevant passages matter, not only their quantity.
3.3. Hard negatives: A controlled experiment combines one golden passage with varying hard negatives from e5, Contriever, BM25, or random sampling.Holding the golden passage constant isolates the effect of the negative passages.
3.3. Hard negatives: Across LLMs, more hard negatives generally reduce answer accuracy, and negatives from stronger retrievers are more difficult than random or weaker-retriever negatives.Gemini-1.5-Pro is robust to random negatives but remains susceptible to hard negatives.
3.3. Hard negatives: Random-negative benchmarks may not adequately evaluate long-context RAG because hard negatives are prevalent in real-world applications.The paper calls for evaluation methodologies incorporating retriever-specific hard negatives.

4. Simple and effective training-free RAG improvement

Retrieval reordering places higher-scoring passages at the beginning and end of long-context inputs, using the models’ positional attention pattern to reduce the influence of hard negatives. Across tested configurations, it provides the clearest gains when retrieval sets are large.

Retrieval reordering places higher-scoring passages at the beginning and end of the input sequence.The strategy exploits long-context models’ tendency to prioritize information at sequence boundaries.
Reordering aims to guide attention toward relevant passages and reduce the influence of hard negatives positioned in the middle.
Across e5 and BM25 retrieval with Gemma-2-9B-Chat and Mistral-Nemo-12B-Instruct on NQ and PopQA, reordering improves RAG performance.
Reordering yields negligible improvements for small retrieval sets but significantly and consistently outperforms original ordering for large sets.The larger-set benefit is attributed to stronger lost-in-the-middle effects and more prevalent hard negatives.

5. Improving Robustness for RAG via Data-Augmented Fine-Tuning

The paper improves long-context RAG robustness through RAG-specific fine-tuning, first exposing models to retrieved contexts and then adding explicit intermediate reasoning for relevance identification. These approaches improve generalization and help models handle irrelevant passages more effectively.

5.1. Implicitly improving LLM robustness through fine-tuning: RAG-specific fine-tuning trains LLMs to answer from instructions, queries, and retrieved passages, implicitly improving robustness to hard negatives.
5.1. Implicitly improving LLM robustness through fine-tuning: RAG FT is evaluated against chat models with retrieval augmentation and Direct FT across unseen datasets and multiple base models.
5.1. Implicitly improving LLM robustness through fine-tuning: RAG FT consistently outperforms the chat model with RAG and Direct FT, while its flatter performance curve indicates greater robustness to increasing hard negatives.
5.2. Enhancing relevance identification through reasoning augmentation: Intermediate-reasoning fine-tuning teaches the model to identify relevant passages before producing the final answer.Training uses labeled reasoning paragraphs, and inference generates the reasoning paragraph before the answer.
5.2. Enhancing relevance identification through reasoning augmentation: RAG FT with intermediate reasoning further improves performance over implicit RAG FT and Direct FT.The paper attributes this to explicit relevance training and a structured analysis of retrieved context.

6. Data-Centric Perspectives on Fine-tuning LLMs for RAG

Fine-tuning data choices strongly affect RAG generalization. Diverse training sources, mixed retrievers, and full-context training each improve robustness across tasks, retrievers, or retrieval sizes.

Impact of training data distribution on generalization: A mixed training distribution from NQ, WoW, Fever, and MMLU yields superior generalization to unseen RAG tasks versus any single source.Each distribution contains 50k samples, with the mixed setting using 12.5k samples from each source.
Influence of retrievers on generalization: Fine-tuning on passages from both BM25 and e5 produces the best performance across seen and unseen retrievers.
Influence of retrievers on generalization: Retriever similarity influences how well a model fine-tuned with one retriever generalizes to another.
Optimizing training for variable retrieval sizes: For Gemma-2-9B-Base with an 8192-token capacity, training with the maximum 40 retrieved passages performs best across varying inference retrieval sizes.

7. Conclusions

The paper finds that adding retrieved passages to long-context RAG can improve performance initially but later degrade it, largely because hard negatives and irrelevant information mislead generation. It proposes reordering and RAG-specific fine-tuning, with data diversity, retriever choice, and context length shaping training robustness.

Long-context RAG performance initially improves but then degrades as more retrieved passages are included.
The proposed remedies combine retrieval reordering, implicit RAG fine-tuning, and intermediate-reasoning fine-tuning.
On NQ, retriever performance ranks e5 > bge > contriever > BM25, with contriever resembling BM25 and bge resembling e5.
The degradation is attributed to hard negatives and added irrelevant or misleading passages that reduce precision and distract generation.

B.2. The importance of hard negatives for long-context LLM evaluation

Hard negatives make long-context LLM evaluation sensitive to both their quantity and retrieval difficulty. Controlled studies distinguish related-but-irrelevant passages from unrelated negatives and show that stronger retrievers can produce more misleading negatives.

Increasing hard-negative passages generally decreases RAG answer accuracy across tested long-context LLMs.
The hard-negative study constructs contexts with a golden passage and varying retrieved negatives, then asks LLMs to answer the query from that context.
Hard negatives from stronger retrievers, such as e5, are more difficult for LLMs than negatives from BM25 or random sampling.
Related-but-irrelevant passages mention entities from the question without containing the ground-truth answer, making them more misleading than unrelated passages.
The broader evaluation uses multiple RAG datasets, four answer types, and a 21M-chunk Wikipedia retrieval corpus.

H.1. Training settings

The training setup specifies templates, generation settings, and dataset-specific instructions for RAG tuning with intermediate reasoning. Inference uses fixed sampling and output-length settings across model families.

Intermediate-reasoning training and evaluation use dedicated instruction and answer templates for different datasets.
Gemini-1.5-Pro is prompted to generate intermediate reasoning for the RAG training process.
All compared LLMs use top-p sampling with p = 1 and a maximum of 256 generated tokens.

I. Data-Augmented RAG Case Studies

The case studies illustrate how RAG-oriented reasoning selects useful evidence and qualifies answers by the available documents. One example identifies a single best document, while another combines two documents.

For the film question, Documents 1 and 2 both connect the named bank to Mary Poppins.
The case studies include retrieved documents containing both directly useful evidence and unrelated or less useful material.
For the English-chemist question, the reasoning identifies Doc 2 as the best document because it states that Humphry Davy discovered nine elements.
The chemist example limits its conclusion to the supplied documents and notes that a document reporting a higher count would change the answer.

J. Data-Augmented RAG Finetuning on Gemma-2-9B

Gemma-2-9B evaluations examine implicit RAG fine-tuning and intermediate reasoning across broad and model-specific settings. The reported figures consistently describe intermediate reasoning as improving over implicit RAG fine-tuning and direct fine-tuning.

Figure 5 evaluates implicit RAG fine-tuning on eight datasets with three different base models.
RAG fine-tuning consistently outperforms chat LLMs with RAG and direct fine-tuning on unseen tasks.
Figure 6 evaluates RAG fine-tuning with intermediate reasoning on five datasets and samples 1,000 queries per dataset because reasoning inference is computationally expensive.
Intermediate-reasoning RAG fine-tuning further improves performance over implicit RAG fine-tuning and direct fine-tuning.
The same ordering is reported for Mistral-Nemo-12B and Gemini-1.0-Pro model evaluations.

M. Training data scaling and RAG performance.

The experiments examine how RAG-specific training-data scale and mixtures with general SFT data affect RAG performance and general language capabilities. Larger training-data amounts improve RAG performance, while combining RAG-specific and general-purpose data improves RAG performance without reducing general-task performance.

Training data scaling and RAG performance: Training-data scale is evaluated by fine-tuning Gemma-2-9B-Base with 5k to 200k mixed samples from NQ, WoW, Fever, and MMLU.The evaluation reported here uses the NQ dataset.
Training data scaling and RAG performance: Increasing the amount of training data consistently improves the resulting LLM’s performance in RAG on NQ.The passage describes a clear positive correlation between training-data scale and RAG effectiveness.
Combining RAG-specific and general SFT data: The combined-data setting trains the LLM on Ultrachat 200k plus 50k RAG-specific samples.The RAG-specific samples are the same data used in Figure 5.
Combining RAG-specific and general SFT data: Adding RAG-specific data to SFT significantly improves RAG-task performance while maintaining general language performance.MT-Bench measures general language capabilities, while NQ and TriviaQA measure RAG performance.