Source-linked AI summary
The Power of Noise: Redefining Retrieval for RAG Systems
Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, Fabrizio Silvestri
TL;DR
RAG extends LLM knowledge by adding passages retrieved from an IR system, but the retrieval strategy itself remains insufficiently understood. This paper systematically examines passage type, position, and number in RAG prompts, finding that distracting top-scoring documents can hurt accuracy while correctly positioned random documents can improve it. The findings motivate further investigation of retrieval strategies for integrating IR with LLMs.
Problem
The study addresses limited evidence about which retriever characteristics and passage types best optimize prompt construction for RAG systems.
Method
The paper conducts a comprehensive study of retrieved-document type, position, and number across RAG experiments, including relevant, distracting, and random passages.
Results
Random documents can improve LLM accuracy by up to 35%, while top-scoring documents that do not contain the answer negatively affect effectiveness.
Takeaways & Limitations
RAG retrieval should balance relevant and random information, place relevant information near the query, and avoid excessive distracting or non-relevant documents.
Takeaways & Limitations
The binary accuracy evaluation can mark semantically equivalent answer variants as incorrect, and more advanced analysis is left for future research.
Abstract
from arXiv · showhide
Retrieval-Augmented Generation (RAG) has recently emerged as a method to extend beyond the pre-trained knowledge of Large Language Models by augmenting the original prompt with relevant passages or documents retrieved by an Information Retrieval (IR) system. RAG has become increasingly important for Generative AI solutions, especially in enterprise settings or in any domain in which knowledge is constantly refreshed and cannot be memorized in the LLM. We argue here that the retrieval component of RAG systems, be it dense or sparse, deserves increased attention from the research community, and accordingly, we conduct the first comprehensive and systematic examination of the retrieval strategy of RAG systems. We focus, in particular, on the type of passages IR systems within a RAG solution should retrieve. Our analysis considers multiple factors, such as the relevance of the passages included in the prompt context, their position, and their number. One counter-intuitive finding of this work is that the retriever's highest-scoring documents that are not directly relevant to the query (e.g., do not contain the answer) negatively impact the effectiveness of the LLM. Even more surprising, we discovered that adding random documents in the prompt improves the LLM accuracy by up to 35%. These results highlight the need to investigate the appropriate strategies when integrating retrieval with LLMs, thereby laying the groundwork for future research in this area.
1 INTRODUCTION
This study examines how retrieved-document types affect RAG effectiveness, challenging assumptions that relevance alone determines useful prompt context. It introduces a systematic analysis and retrieval heuristics motivated by findings about distracting and random documents.
- RAG augments LLM prompts with passages retrieved by an IR system to address limitations in long-context handling and reliance on pre-trained knowledge.The introduction links these limitations to increased hallucination risk.
- The study asks which retriever characteristics optimize prompt construction and whether current retrievers are ideal.
- Relevant, distracting, and random passages represent three retrieval types whose relative value for LLM effectiveness is examined.Relevant passages answer or inform the query; distracting passages are related but do not answer it; random passages are unrelated noise.
- The work conducts the first comprehensive study of how retrieved-document type affects RAG effectiveness.
- The authors propose RAG retrieval heuristics based on the study’s unexpected results and release associated code and data for further research.
2 RELATED WORKS
Related work traces RAG to advances in transformers, language representation, and retrieval, while distinguishing sparse lexical methods from dense semantic retrievers. The paper positions itself as the first comprehensive analysis of how retrieved-document type, number, and position affect RAG prompts.
- Transformer-based language models built on attention mechanisms and bidirectional representation methods underpin the modern LLM era.
- Passage and document denote the same indexing or retrieval unit in the paper.
- Traditional sparse retrieval methods use lexical matching, whereas dense retrievers use low-dimensional vectors to capture semantic relationships.DPR and related dense methods have demonstrated competitiveness with sparse methods.
- RAG combines retrieval-based and generative models, with early work demonstrating improvements on knowledge-intensive tasks.
- This paper analyzes how the type, number, and position of retrieved documents affect the prompt supplied to an LLM.
3 RAG
RAG systems divide open-domain question answering into retrieval and generation: a retriever selects a small document set, and an LLM synthesizes an answer from the query and retrieved documents. The paper formalizes this pipeline and studies how changing the retrieved set affects end-to-end effectiveness.
- 3.1 Open-Domain Question Answering: Open-domain question answering seeks accurate, contextually relevant answers to natural-language questions without restricting the domain or dataset.
- 3.1 Open-Domain Question Answering: A typical OpenQA architecture first retrieves documents that may answer the query and then synthesizes an answer with a generator.
- 3.2 Retriever: Dense retrieval maps queries and documents into vector representations and ranks documents by embedding similarity.Neural encoders such as BERT are used to create these representations.
- 3.3 Reasoner: The generator, typically an LLM, predicts tokens sequentially to synthesize an answer from the query and retrieved documents.
- 3.3 Reasoner: RAG retrieval distributions select top-scoring documents, while the generator models each token conditioned on the query, retrieved document, and prior output tokens.
- 3.3 Reasoner: The paper studies how changing the retrieved document set affects the generator and whole end-to-end system, seeking the best set for effectiveness.
4 EXPERIMENTAL METHODOLOGY
The experiments use NQ-open and categorize retrieved passages by their relationship to the query, then evaluate their effects in a two-step RAG setup across several LLMs. Accuracy is assessed by binary answer matching, with acknowledged difficulties for equivalent answer variants.
- Dataset: NQ-open queries are answered using 100-word passages segmented from an English Wikipedia dump dated 20 December 2018.The design also addresses possible temporal mismatches between the dump and question-answer pairs by integrating gold-document information.
- Document Types: Retrieved passages are categorized as gold, relevant, distracting, or random according to answer content and query relationship.Relevant passages contain the answer and are contextually useful; distracting passages are semantically similar but answer-free; random passages are unrelated.
- Retrieval and Prompting: Contriever retrieves the top-k documents from a corpus of about 21 million documents, using averaged final-layer embeddings and FAISS IndexFlatIP for similarity search.Contriever is a BERT-based dense retriever trained without supervision using contrastive loss.
- Retrieval and Prompting: The LLM receives the task instruction, query, and retrieved documents, and generates a response with greedy decoding capped at 15 tokens.Models are quantized to 4 bits, and no few-shot exemplars are provided.
- Evaluation: Response correctness is measured as a binary accurate-or-inaccurate outcome based on whether the answer appears, which can penalize valid alternative phrasings.The authors specifically note problems with date formats and variants such as “Roosevelt” versus “President Roosevelt.”
5 RESULTS
The results show that adding highly ranked but answer-free distracting passages progressively harms LLM accuracy, and changing dense retrievers does not remove this effect. The experiments vary prompt composition and document count, including an oracle setting with the gold passage known.
- Experimental Setup: The experiments vary contextual document type and number, using sequences containing gold, relevant, distracting, or random passages.Document position is also treated as a configuration variable in the reported results.
- Impact of Distracting Documents: More than 0.38 (−67%) accuracy degradation occurs in some cases as the number of distracting documents increases, consistently across all tested LLMs.The first experiment uses 10K NQ-open training queries in an oracle setup where the gold document is known.
- Impact of Distracting Documents: 0.24 (−25%) peak accuracy reduction occurs when one distracting document is added to the gold-only scenario.This compares prompts with zero versus one distracting document.
- Retriever Robustness: ADORE yields Llama2 accuracies of 0.4068, 0.3815, and 0.3626 with 1, 2, and 4 distractors, versus 0.5642 without distractors.The authors conclude that distinguishing relevant from distracting information is not mitigated simply by changing the dense retrieval method.
5.2 Impact of Gold Positioning
The experiments vary the gold document’s position relative to the query and compare accuracy across near, mid, and far placements. Near placement performs best, while middle placement performs worst in the distracting-document setting.
- Position Definitions: Gold-document positions are defined as near, mid, and far according to increasing distance from the query within the prompt.Near places gold adjacent to the query; mid places it centrally; far places it as far as possible.
- Position Effects: With 18 distracting documents, Llama2 accuracies are 0.37 near, 0.23 far, and 0.17 mid.The reported ordering is consistent across all tested models in this setting.
- Position Effects: Accuracy is higher when the gold document is near the query, lower when it is farthest, and lowest when it is placed in the middle.These results partially corroborate prior evidence cited by the authors.
5.3 Impact of Noise
Adding random documents to a gold document did not degrade performance and improved MPT accuracy by 0.08 (+36%) in the best-performing near setting. Model responses varied with noise position: Llama2 and Phi-2 handled distant noise better, whereas MPT improved across settings and Falcon showed no comparable oracle-setting gain.
- Random-document noise did not deteriorate performance overall in the gold-document experiment.The experiment compared a gold document alone with the gold document supplemented by randomly sampled corpus documents.
- 0.08 (+36%) improvement in MPT accuracy occurred when random documents were added in the best-performing near setting.The near configuration places noise closer to the query in the reported prompt structure.
- Llama2 and Phi-2 improved when noise was positioned farthest from the query but declined when noise moved closer.Their decline near the query was less pronounced than with distracting documents.
- MPT improved under all tested noise-position settings, unlike Falcon in the oracle setting.Falcon did not show the performance increase observed for other models when random documents were added to the gold document.
5.4 RAG in Practice
In a more realistic retrieval setting, adding random documents to retrieved relevant or distracting documents was almost always beneficial. The effect persisted with BM25 and became stronger when random documents came from Reddit or consisted of nonsensical random-word sentences.
- Up to 0.07 (+35%) accuracy gains occurred when random documents were added to four retrieved documents.Adding random documents until the context length was filled was almost always beneficial regardless of the number of retrieved documents.
- BM25 replicated the improvement from adding random documents, producing an average accuracy increase of 3-4 percentage points.The authors attribute this improvement to the quality of documents retrieved by BM25.
- 5.4.2 Increasing The Randomness.: Reddit documents enhanced the previous best score by 0.023 (+9% accuracy).These documents were drawn from a corpus with substantially different tone and style from Wikipedia.
- 5.4.2 Increasing The Randomness.: Nonsensical sentences made from random words also improved performance relative to the base case of Wikipedia random documents.
- Adding random documents improved Falcon when they were placed on top of retrieved documents.This differed from Falcon’s behavior when random documents were added to a gold document in the oracle setting.
5.5 Retriever Trade-Off
The experiments indicate a trade-off between relevant and totally irrelevant documents: a minimal relevant retrieval set supplemented with random documents performed best, while excessive retrieval increased distracting information. The paper offers an attention-entropy explanation but leaves the mechanism unresolved.
- Retrieving 3 to 5 documents and supplementing them with random documents until the context limit was reached was most effective for the examined queries.Adding more retrieved documents increased the risk of including distracting and non-relevant information and sharply declining performance.
- Accurate answers still required some relevant information, despite the positive effect of random documents.The reported trade-off is between a minimal relevant set and additional totally irrelevant documents.
- Adding random documents produced a 3X increase in attention entropy compared with supplying only the gold document.The authors suggest that random documents may better condition the generation function, but they do not establish this explanation definitively.
On The Unreasonable Effectiveness Of Random Documents.
Random documents improved accuracy even when drawn from a stylistically different corpus or formed from nonsensical random words. The accompanying analysis found a 3X increase in attention entropy after random documents were introduced, while identifying the causal explanation as an open question.
- 3X increase in attention entropy occurred when random documents were added to the gold document.The comparison was between the gold-document-only prompt and the prompt containing additional random documents.
- The reason noisy contexts are more advantageous remains unresolved and is identified as a target for future research.
6 CONCLUSIONS
The study finds that RAG effectiveness depends on how retrieved documents are selected and positioned, with relevant information near the query and carefully positioned random documents improving outcomes while answer-free top-scoring documents hurt performance.
- Relevant information should be placed near the query because the model otherwise seriously struggles to attend to it.
- Top-scoring retrieved documents that do not contain the answer negatively impact LLM effectiveness when added to the prompt.
- Random noisy documents can increase system accuracy when correctly positioned within the prompt.
- The authors propose heuristics to exploit these findings, while calling for research into the underlying mechanisms and retrieval techniques designed to interact with generative components.