PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, Michael Bendersky

arXiv:2407.16833v2cs.CLcs.AIcs.LG

TL;DR

The paper asks whether retrieval augmentation or direct long-context processing is preferable as LLMs become better at understanding lengthy inputs. It benchmarks both approaches and proposes SELF-ROUTE, which uses model self-reflection to select between them. LC generally performs better when sufficiently resourced, while SELF-ROUTE preserves comparable performance at lower cost.

  • Problem

    The paper addresses the need to compare RAG with long-context LLMs because recent models can understand long contexts directly while RAG remains efficient.

  • Method

    The study benchmarks RAG and LC across public datasets with three LLMs and proposes SELF-ROUTE, which routes queries using model self-reflection.

  • Results

    LC consistently outperforms RAG on average when sufficiently resourced, while SELF-ROUTE achieves comparable performance to LC with lower cost.

  • Takeaways & Limitations

    RAG remains useful for its lower cost and for inputs exceeding a model's context window, while routing can combine the strengths of RAG and LC.

  • Takeaways & Limitations

    Evaluation-dataset leakage may occur because black-box LLMs are pretrained on unknown data, and addressing this remains an open question.

Abstract

from arXiv · show

Retrieval Augmented Generation (RAG) has been a powerful tool for Large Language Models (LLMs) to efficiently process overly lengthy contexts. However, recent LLMs like Gemini-1.5 and GPT-4 show exceptional capabilities to understand long contexts directly. We conduct a comprehensive comparison between RAG and long-context (LC) LLMs, aiming to leverage the strengths of both. We benchmark RAG and LC across various public datasets using three latest LLMs. Results reveal that when resourced sufficiently, LC consistently outperforms RAG in terms of average performance. However, RAG's significantly lower cost remains a distinct advantage. Based on this observation, we propose Self-Route, a simple yet effective method that routes queries to RAG or LC based on model self-reflection. Self-Route significantly reduces the computation cost while maintaining a comparable performance to LC. Our findings provide a guideline for long-context applications of LLMs using RAG and LC.

1 Introduction

The paper compares retrieval-augmented generation with long-context LLMs, finding that LC generally performs better while RAG is substantially cheaper. SELF-ROUTE combines both through model self-reflection to approach LC performance at reduced cost.

  • RAG retrieves query-relevant information and prompts an LLM using the retrieved context, expanding access to external knowledge at low cost.
  • Recent models such as Gemini and GPT-4 can understand long contexts directly, motivating systematic comparison of LC and RAG on performance and efficiency.Gemini 1.5 can process up to 1 million tokens.
  • LC consistently outperforms RAG in almost all sufficiently resourced settings across public datasets.The study benchmarks both approaches to characterize their respective strengths and weaknesses.
  • RAG remains relevant because reducing LLM input length lowers computational cost, and its predictions match LC for over 60% of queries.For those queries, RAG can reduce cost without sacrificing performance.
  • SELF-ROUTE routes queries to RAG or LC using model self-reflection, reducing cost while achieving overall performance comparable to LC.Reported cost reductions are 65% for Gemini-1.5-Pro and 39% for GPT-4O.
  • The paper compares LC, RAG, and SELF-ROUTE with three recent LLMs and analyzes failure patterns, cost-performance trade-offs, and synthetic datasets.The analysis is presented as guidance for building long-context applications.

2 Related Work

Prior work develops long-context models and methods to reduce their cost, while RAG research improves retrieval quality and long-context evaluation uses both synthetic and real benchmarks.

  • Long-context LLMs can process larger windows, but transformer computation grows quadratically with input length, making long-context prompting expensive.Related methods include prompt compression, model distillation, and LLM cascading.
  • RAG research has introduced correction, critique, verification, and adaptive-search methods to improve retrieval quality on knowledge-intensive tasks.
  • Long-context evaluation combines synthetic tests such as needle-in-a-haystack with real datasets including LongBench and ∞Bench.

3 Benchmarking RAG versus LC

The benchmark evaluates RAG and LC across selected real and synthetic long-context tasks using three LLMs and dense retrievers. Overall, LC leads RAG, except when contexts exceed GPT-3.5-Turbo's window, where RAG can perform better.

  • Datasets and metrics: The study selects seven LongBench and two ∞Bench datasets focused mainly on English, real, query-based tasks.The ∞Bench datasets have substantially longer contexts than LongBench.
  • Datasets and metrics: The evaluation reports F1 for open-ended QA, accuracy for multiple-choice QA, and ROUGE for summarization.
  • Models and Retrievers: Three evaluated LLMs are Gemini-1.5-Pro, GPT-4O, and GPT-3.5-Turbo, supporting context windows of up to 1 million, 128k, and 16k tokens respectively.
  • Models and Retrievers: The benchmark uses Contriever and Dragon, splitting contexts into 300-word chunks and selecting the top k chunks by query-chunk cosine similarity.The default is k = 5, with chunk indices prepended before prompting.
  • Models and Retrievers: Evaluation leakage may occur because black-box LLMs were pretrained on unknown datasets, so the study prompts models to answer based only on the provided passage.The paper identifies leakage mitigation as an open question.
  • Benchmarking results: LC surpasses RAG on average by 7.6% for Gemini-1.5-Pro, 13.1% for GPT-4O, and 3.6% for GPT-3.5-Turbo.The gap is larger for the more recent Gemini and GPT-4O models.
  • Benchmarking results: RAG outperforms LC for GPT-3.5-Turbo on En.QA and EN.MC, where contexts average 147k words and exceed its 16k context window.This is a specific setting in which RAG is effective when the input greatly exceeds the model window.

4 Self-Route

SELF-ROUTE routes queries between RAG and long-context prediction using model self-reflection. It reduces token use and cost while achieving performance comparable to long-context models, although results vary by model and dataset length.

  • Motivation: 63% of queries have exactly identical RAG and long-context predictions, motivating selective use of long-context inference.For 70% of queries, the absolute prediction-score difference is less than 10; identical predictions can be either correct or incorrect.
  • Self-Route: SELF-ROUTE combines RAG-and-Route with long-context prediction, accepting RAG answers for answerable queries and sending unanswerable queries to the full-context model.The first step uses retrieved chunks; the second provides the full context to the long-context model.
  • Results: SELF-ROUTE surpasses RAG by over 5% across all three models, with performance changes versus long-context of -0.2% for GPT-4O, -2.2% for Gemini-1.5-Pro, and +1.7% for GPT-3.5-Turbo.These results make SELF-ROUTE comparable to long-context performance while improving substantially over RAG.
  • Results: Gemini-1.5-Pro routes 81.74% of queries as answerable, while all three models route more than half of queries toward RAG.The high answerable rate means most queries can be handled without long-context inference.
  • Results: GPT-4O uses 61% of long-context tokens and achieves comparable performance of 46.83 versus 47.04, while Gemini-1.5-Pro uses 38.6% of the tokens.Lower token counts reduce computation and API costs because transformer computation and many API prices scale with input-token count.
  • Results: On longer datasets, SELF-ROUTE outperforms long-context for GPT-4O by 2.3% on EN.QA and 7.4% on EN.MC, but performs lower than long-context for Gemini-1.5-Pro.The paper attributes differing performance-cost trade-offs possibly to differences in LLM alignment and willingness to answer from RAG context.

5 Analysis

The analysis examines how retrieval depth affects SELF-ROUTE’s performance and cost, why RAG fails, and how evaluation design and internal knowledge affect comparisons. It also finds that retriever choice preserves the reported findings while synthetic datasets can produce artifact-sensitive results.

  • Ablations of k: Larger k improves RAG and SELF-ROUTE performance, while their results approach LC when k exceeds 50.At k = 1, RAG reaches 20.24% and SELF-ROUTE 37.9%; when k is larger than 50, all three methods have similar performance.
  • Ablations of k: SELF-ROUTE’s cost is lowest at k = 5 on average, but k = 1 is cheapest on some extractive datasets.Increasing k raises retrieval and routing cost but can route more queries to RAG; the optimal k depends on task nature and performance requirements.
  • Why does RAG fail?: RAG failures commonly involve multi-step, general, long and complex, or implicit queries requiring broader context understanding.The analysis classifies unanswerable examples using manually annotated few-shot examples and reports distributions across seven LongBench datasets.
  • Retriever robustness: Using the Dragon retriever produces findings consistent with Contriever for LC, RAG, and SELF-ROUTE, supporting generalizability across retrievers.
  • Results on synthetic data: Synthetic evaluation can reverse the apparent RAG-versus-LC comparison when query wording changes, revealing sensitivity to dataset construction artifacts.On PassKey, RAG scores 80.34% versus LC’s 65.25% for one query, but falls to 4.58% after a wording change while LC remains at 69.32%.
  • Exclusion of LLM’s internal knowledge: The comparison limits parametric knowledge with a passage-only instruction, which lowers Gemini-1.5-Pro’s average performance from 50.57 to 45.53.Results on non-commonsense MuSiQue questions preserve the same trend, while broader study of internal-knowledge control remains future work.

6 conclusion

The paper frames RAG and long-context models as a performance–cost trade-off and proposes dynamically routing queries between them. It also notes that model-internal knowledge can produce different results on non-commonsense questions.

  • LC models demonstrate superior long-context understanding, while RAG remains viable because of lower cost and advantages beyond the model context window.
  • Self-Route dynamically routes queries using model self-reflection, achieving performance comparable to LC at significantly reduced cost.
  • GPT-3.5-Turbo and Gemini-1.5-Pro obtain different MuSiQue performance using only internal knowledge.The reported scores are 14.53 for GPT-3.5-Turbo and 23.58 for Gemini-1.5-Pro.

A Dataset details

The evaluation covers diverse long-context question-answering and summarization datasets from LongBench and ∞Bench. These datasets span stories, academic papers, mixed-source documents, multi-hop questions, meetings, and long novels.

  • The study evaluates seven LongBench datasets spanning question answering, multi-hop reasoning, and query-based summarization.The datasets are NarrativeQA, Qasper, MultiFieldQA, HotpotQA, 2WikiMultihopQA, MuSiQue, and QMSum.
  • The LongBench tasks include contexts from novels, movie scripts, academic NLP papers, legal documents, government reports, encyclopedias, Wikipedia, and meeting scripts.
  • ∞Bench contributes En.QA and EN.MC, which use long novels with manually replaced entity names and, for EN.MC, four challenging answer choices.
  • Dataset statistics report the number of evaluation queries and average context length in words.

B Ablations of k

The ablation varies k and compares performance with token ratios relative to long-context inference. The long-context result serves as the upper-bound reference for these comparisons.

  • Performance and token ratio are reported for different k values in the ablation corresponding to Fig. 3.
  • 45.53 is the performance of LC, which serves as the upper bound.
  • The token ratio divides tokens used by RAG or SELF-ROUTE by the tokens required by LC.

C Prompts

The paper uses dataset-specific prompts adapted from released LongBench and ∞Bench prompts. It also provides separate prompts for failure-case analysis.

  • Dataset prompts are modified from the released LongBench and ∞Bench prompts.
  • The study documents prompts used for each dataset.
  • Failure-case analysis uses prompts corresponding to Section 5.2.
Loading 2407.16833v2…