Source-linked AI summary
RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, Bryan Catanzaro
TL;DR
RAG must select useful contexts without overwhelming generation, while separate retrieval and ranking components can limit relevance modeling. RankRAG instruction-tunes one LLM for context ranking and answer generation, and it significantly outperforms evaluated baselines across nine general-domain and five biomedical RAG benchmarks. The method adds reranking latency, although the paper reports that this overhead is lower than the corresponding reranking range in its efficiency study.
Problem
Current RAG pipelines face limited retriever capacity and a top-k trade-off: fewer contexts reduce recall, while more contexts can add noise and impair generation.
Method
RankRAG instruction-tunes a single LLM with ranking and generation data, then retrieves, reranks, and generates using the refined contexts.
Results
RankRAG significantly outperforms existing expert ranking models and evaluated RAG baselines on nine general-domain and five biomedical benchmarks.
Takeaways & Limitations
A small fraction of ranking data can enable one instruction-tuned LLM to support both context ranking and answer generation across knowledge-intensive RAG tasks.
Takeaways & Limitations
Reranking introduces additional processing time because relevance scoring is performed for retrieved contexts before generation.
Abstract
from arXiv · showhide
Large language models (LLMs) typically utilize the top-k contexts from a retriever in retrieval-augmented generation (RAG). In this work, we propose a novel instruction fine-tuning framework RankRAG, which instruction-tunes a single LLM for the dual purpose of context ranking and answer generation in RAG. In particular, the instruction-tuned LLMs work surprisingly well by adding a small fraction of ranking data into the training blend, and outperform existing expert ranking models, including the same LLM exclusively fine-tuned on a large amount of ranking data. For generation, we compare our model with many strong baselines, including GPT-4-0613, GPT-4-turbo-2024-0409, and ChatQA-1.5, an open-sourced model with the state-of-the-art performance on RAG benchmarks. Specifically, our Llama3-RankRAG significantly outperforms Llama3-ChatQA-1.5 and GPT-4 models on nine knowledge-intensive benchmarks. In addition, it also performs comparably to GPT-4 on five RAG benchmarks in the biomedical domain without instruction fine-tuning on biomedical data, demonstrating its superb capability for generalization to new domains.
1 Introduction
RAG retrieves contexts for LLM answer generation, but selecting and processing top-k contexts creates recall, noise, and efficiency challenges. RankRAG addresses these challenges by instruction-tuning one LLM for both context ranking and answer generation.
- RAG retrieves top-k contexts from documents or databases before an LLM generates an answer.
- Smaller context sets can reduce recall, while larger sets can introduce irrelevant or noisy context that misleads generation.
- RankRAG instruction-tunes a single LLM for both context ranking and answer generation.
- The framework adds context-rich QA, retrieval-augmented QA, and ranking data to help filter irrelevant contexts during retrieval and generation.
2 Related Work
Related work improves RAG through retriever alignment, multi-step retrieval, and separate ranking models, but these approaches add complexity or may lack sufficient relevance modeling and generalization.
- RAG research has explored aligning retrievers with LLM generation and designing multi-step retrieval processes.
- End-to-end retriever–LLM optimization can complicate training and require frequent embedding-database re-indexing.
- Separate ranking models improve retrieval quality but may insufficiently capture query–context relevance and lack zero-shot generalization.
- Figure 1 examines ChatQA-1.5 performance as context size k changes, highlighting the trade-off between recall and irrelevant context.
3 Preliminaries
The paper formalizes RAG as retrieval followed by answer generation and identifies limitations in retriever capacity and top-k context selection.
- 3.1 Problem Setup: Given a question q, retriever R selects top-k contexts C, after which an autoregressive language model generates the final answer.
- 3.2 Limitation of Current RAG Pipelines: The paper frames current RAG as a retrieve-then-generate pipeline whose limitations motivate RankRAG.
- 3.2 Limitation of Current RAG Pipelines: Sparse or moderate-size embedding retrievers independently encode questions and documents, limiting their ability to model fine-grained relevance across large collections.
- 3.2 Limitation of Current RAG Pipelines: Accuracy can saturate near k = 10; smaller k harms recall, whereas larger k increases irrelevant content and hampers generation.
4 RankRAG
RankRAG uses two-stage instruction tuning to unify ranking and generation, then applies a retrieve-rerank-generate inference pipeline. Its training blends instruction-following, context-rich QA, retrieval-augmented QA, ranking, and multi-context ranking data.
- 4 RankRAG: RankRAG instruction-tunes an LLM to assess question–context relevance and use retrieved context for answer generation.
- 4.1 Stage-I: Stage-I applies supervised instruction tuning to establish basic instruction-following capabilities before RAG-specific training.
- 4.2 Stage-II: Unified Instruction-Tuning for Ranking and Generation: Stage-II combines Stage-I data with context-rich QA, retrieval-augmented QA, context ranking, and retrieval-augmented ranking data.
- 4.2 Stage-II: Unified Instruction-Tuning for Ranking and Generation: Retrieval-augmented ranking combines gold and BM25-retrieved contexts, training the model to identify relevant contexts among five candidates.
- 4.2 Stage-II: Unified Instruction-Tuning for Ranking and Generation: All Stage-II tasks are cast into a unified QA format (x, c, y), enabling ranking and generation tasks to support knowledge transfer.
- 4.3 RankRAG Inference: Retrieve-Rerank-Generate Pipeline: At inference, the retriever selects top-N contexts, RankRAG reranks them using the probability of generating “True,” and generation uses the refined top-k contexts.
- 4.3 RankRAG Inference: Retrieve-Rerank-Generate Pipeline: Reranking adds processing time because the model computes relevance scores before generation.
5 Experiments
Experiments evaluate RankRAG across knowledge-intensive tasks, baselines, retrievers, model backbones, ablations, and biomedical transfer. Results show broad gains, especially on challenging datasets, with robustness to retriever choice and modest ranking-data requirements, alongside added reranking latency.
- Tasks and evaluation: Experiments cover open-domain QA, fact verification, and conversational QA under zero-shot evaluation.The study uses nine datasets, with task-specific metrics including Exact Match, Accuracy, and F1.
- Main results: RankRAG consistently outperforms ChatQA-1.5 and remains competitive with substantially larger retrieval-augmented baselines.RankRAG 8B significantly outperforms InstructRetro and RA-DIT 65B, while RankRAG 70B outperforms ChatQA-1.5 70B and earlier InstructGPT-based RAG systems.
- Main results: More than 10% improvement over ChatQA-1.5 appears on PopQA and 2WikimQA, indicating larger gains on long-tailed and multi-hop QA.The authors associate these gains with ranking contexts when retrieved top documents are less relevant to the answer.
- Ablation studies: Removing context ranking hurts performance on all tasks, while retrieval-augmented QA and ranking data improve outcomes on most tasks.The ablation uses Llama3-8B and distinguishes RQA from RAR training data and inference without reranking.
- Ablation studies: Average performance gains remain consistent across Llama2 backbones: 7.8%, 6.4%, and 6.3% for the 7B, 13B, and 70B variants.This comparison supports gains across different LLM types and scales.
- Retriever robustness: RankRAG surpasses ChatQA-1.5 by more than 10% on average across two dense retrievers, showing robustness to retriever choice.The comparison uses DPR and Contriever-MS MARCO on three representative tasks.
- Biomedical transfer: RankRAG 8B surpasses Meditron 70B by 6.3% and RankRAG 70B reaches more than 98% of GPT-4 performance on biomedical QA without biomedical fine-tuning.The benchmark is Mirage, a zero-shot biomedical RAG evaluation.
- Ranking module: Five thousand ranking examples, approximately 1% of MS MARCO, already produce compelling results, while 50k examples yield further non-marginal gains.The finding highlights ranking-data efficiency, though increasing reranking depth adds latency overhead.
6 Conclusion
RankRAG instruction-tunes one LLM for context ranking and answer generation, outperforming strong RAG systems across general-domain and biomedical benchmarks.
- RankRAG instruction-tunes a single LLM for both context ranking and answer generation.
- Adding only a small fraction of ranking data enables instruction-tuned LLMs to outperform existing expert ranking models.
- RankRAG significantly outperforms state-of-the-art RAG models on nine general-domain and five biomedical benchmarks.
A Dataset Description
The appendix reports dataset information for the 14 datasets used in RankRAG.
- The appendix lists information for 14 datasets used in RankRAG.
A.1 Main Experiments
The main experiments cover open-domain question answering, fact verification, and conversational question answering across diverse datasets and retrieval settings.
- Open-domain QA: PopQA focuses on long-tail entities, using 1,399 rare entities with fewer than 100 monthly Wikipedia page views.
- Open-domain QA: HotpotQA requires answering complex questions by linking information across multiple documents.
- Open-domain QA: 2WikimQA evaluates cross-lingual and cross-cultural retrieval and question answering across two Wikipedia entities.
- Fact verification: FEVER provides the fact-verification benchmark, using manually verified claims and Wikipedia evidence.
- Conversational QA: Doc2Dial, TopiOCQA, and INSCIT provide conversational QA settings grounded in documents or the whole of Wikipedia.Doc2Dial covers four domains, while TopiOCQA tests topic switching and INSCIT tests underspecified questions requiring clarification.
A.2 Biomedical Benchmarks
The biomedical benchmarks include six biomedical task groups and use instruction formats tailored to retrieval, ranking, answer length, and arithmetic requirements.
- Biomedical benchmarks: MMLU-med contains six biomedical task subsets and 1,089 questions.
- Biomedical benchmarks: MedQA contains 1,273 four-option questions from the United States Medical Licensing Examination.
- Biomedical benchmarks: MedMCQA uses a 4,183-question development set covering 2,400 healthcare topics across 21 medical subjects.The development set is used because the test set lacks provided ground truths.
- Biomedical benchmarks: PubMedQA contains 1,000 manually annotated questions based on PubMed abstracts, with yes/no/maybe answers.
- Training and instructions: Stage-II training blends retrieval-augmented QA and ranking data with conversational QA, MS MARCO ranking, and supervised fine-tuning data.The listed dataset ratios are normalized so their total equals 1.
- Training and instructions: The instruction templates ask models to assess passage relevance, return relevant passage IDs, answer with short or complete responses, or perform arithmetic when required.
E Additional Experiment Results
RankRAG improves retrieved-content quality across DPR and Contriever and remains effective with a small reranked context set. Reranking makes k = 5 work well for most datasets by prioritizing relevant contexts.
- RankRAG-8B shows consistent performance gains across three datasets when using both DPR and Contriever.These results indicate applicability across multiple popular retrieval models.
- k = 5 works well for most datasets with RankRAG, unlike vanilla RAG approaches without ranking.Reranking prioritizes the most relevant contexts at the top, reducing the need to include additional contexts.
F Performance of NQ and Trivia QA on DPR Splits
Because NQ and TriviaQA have both DPR and KILT versions, RankRAG reports results on DPR splits for a fair comparison. It consistently outperforms its direct competitors across the dataset subsets.
- NQ and TriviaQA have distinct DPR and KILT versions, with different numbers of evaluation questions.The DPR versions contain 3610 NQ questions and 11316 TriviaQA questions, while KILT uses smaller subsets.
- RankRAG consistently outperforms ChatQA, Llama-3-instruct, and other InstructGPT-backed methods regardless of the dataset subset used.The comparison uses DPR splits to support fair evaluation across versions.
G Additional Case Studies
Case studies examine RankRAG on long-tailed and multi-hop questions, where retrieval is difficult. Its reranking step helps recover pertinent evidence, filter distractors, and pinpoint correct answers.
- PopQA and HotpotQA case studies compare RankRAG-8B with ChatQA-1.5-8B using distractors and evidences in retrieved contexts.PopQA focuses on long-tailed QA, while HotpotQA focuses on multi-hop QA.
- The Universe Around Us case includes retrieved passages about Sir James Jeans and his science book.
- The Monster case includes passages about Rihanna, Eminem, and their song collaborations.
- RankRAG’s reranking helps counteract poor initial retrieval by finding more pertinent evidence.Combined with RAG-oriented fine-tuning, it filters distracting entities and pinpoints correct answers.