PaperlayerView exact revision on arXiv

Source-linked AI summary

Sufficient Context: A New Lens on Retrieval Augmented Generation Systems

Hailey Joren, Jianyi Zhang, Chun-Sung Ferng, Da-Cheng Juan, Ankur Taly, Cyrus Rashtchian

arXiv:2411.06037v3cs.CL

TL;DR

Prior RAG research leaves open whether errors reflect poor use of retrieved context or context that lacks enough information to answer. This paper defines and classifies context sufficiency, analyzes model behavior, and develops selective generation. Models err even with sufficient context, while selective generation improves the correct-answer fraction for Gemini, GPT, and Gemma.

  • Problem

    Prior work does not directly distinguish whether RAG errors arise from failure to use retrieved context or from context that lacks enough information to answer.

    Source: §1 · pp. 1–2

  • Method

    The paper defines sufficiency for question-context pairs, uses an autorater to classify instances, and combines sufficiency and confidence signals for selective generation.

    Source: §1 · p. 2, §5.1 · p. 9

  • Results

    Models err on some sufficient-context instances and hallucinate more than they abstain with insufficient context; selective generation improves the fraction of correct answers among total model responses by up to 2–10% for Gemini, GPT, and Gemma.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Open-book question answering cannot be solved by improving retrieval alone, because models also err when context is sufficient.

    Source: §1 · p. 2

    The analysis focuses on QA datasets, while context sufficiency may also matter for summarization tasks.

    Source: §6 · p. 10

Abstract

from arXiv · show

Augmenting LLMs with context leads to improved performance across many applications. Despite much research on Retrieval Augmented Generation (RAG) systems, an open question is whether errors arise because LLMs fail to utilize the context from retrieval or the context itself is insufficient to answer the query. To shed light on this, we develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query. We then use sufficient context to analyze several models and datasets. By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance (Gemini 1.5 Pro, GPT 4o, Claude 3.5) excel at answering queries when the context is sufficient, but often output incorrect answers instead of abstaining when the context is not. On the other hand, smaller models with lower baseline performance (Mistral 3, Gemma 2) hallucinate or abstain often, even with sufficient context. We further categorize cases when the context is useful, and improves accuracy, even though it does not fully answer the query and the model errs without the context. Building on our findings, we explore ways to reduce hallucinations in RAG systems, including a new selective generation method that leverages sufficient context information for guided abstention. Our method improves the fraction of correct answers among times where the model responds by 2--10\% for Gemini, GPT, and Gemma. Key findings and the prompts used in our autorater analysis are available on our github.

1 INTRODUCTION

The introduction distinguishes failures to use retrieved context from cases where context lacks enough information, then uses context sufficiency to analyze RAG behavior and guide selective generation.

1 INTRODUCTION

  • Prior RAG work studied incorrect predictions, distraction, and extraction failures, but relevance ranges from directly answering a question to being merely topical, leaving sufficiency unaddressed.The introduction frames the open question as whether errors reflect context use or insufficient information.Source: §1 · pp. 1–2
  • Sufficiency is defined per question-context pair as enough information to construct an answer, without requiring a ground-truth answer; an autorater classifies instances with 93% accuracy.The autorater enables scalable analysis of model responses with and without sufficient context.Source: §1 · p. 2
  • Models err on some sufficient-context instances, hallucinate more than they abstain with insufficient context—especially on multi-hop questions—and sometimes answer correctly despite insufficiency.Correct answers with insufficient context persist after filtering questions answered correctly without RAG.Source: §1 · p. 2
  • The selective-generation method improves the fraction of correct answers among total model responses by up to 2–10% for Gemini, GPT, and Gemma.It uses confidence and sufficient-context signals to guide generation or abstention.Source: §1 · p. 2

2 RELATED WORK

Related work studies imperfect retrieval and hallucination reduction, but evaluates context relevance in varied ways and often assumes the context is sufficient.

2 RELATED WORK

  • Prior studies examine robustness to imperfect retrieval and methods to reduce hallucinations, often by improving generation and retrieval quality through fine-tuning.Long RAG contexts also raise the “lost in the middle” problem.Source: §2 · pp. 2–3
  • Prior evaluations use differing relevance criteria, including irrelevant added text, counter-memory context, and relevance proxies that do not require context to contain the answer.Cuconasu et al. (2024) distinguishes golden and relevant documents but uses “does not contain the answer” as a proxy for irrelevance.Source: §2 · pp. 2–3
  • On Musique, adding RAG improves correct answers, but models hallucinate more than they abstain; 55.4% of instances have insufficient context.The figure caption notes similar trends for FreshQA and HotPotQA.Source: Figure 1 · p. 3
  • Many hallucination-reduction studies assume the query and context are precisely answerable, so they analyze findings only in the sufficient-context scenario.Separate work studies confidence calibration and performance across data subsets.Source: §2 · p. 3

3 SUFFICIENT CONTEXT

The paper defines sufficient context as information that can support a plausible answer, then evaluates methods for identifying it without requiring a ground-truth answer. Gemini 1.5 Pro (1-shot) is the strongest tested autorater, and the authors use it to label instances for later analyses.

3.1 DEFINITION OF SUFFICIENT CONTEXT

  • An instance has sufficient context when some plausible answer to its query can be constructed from the context, without presupposing the ground-truth answer.Unlike entailment, this criterion asks whether an answer exists rather than whether a specified answer is supported; it can therefore allow a context containing an answer that conflicts with the ground truth.Source: §3.1 · p. 4
  • Sufficiency requires supported reasoning across multi-hop facts and, for ambiguous queries or answers, enough context to disambiguate them.The examples distinguish supported links between facts from unsupported inference, and require context to resolve ambiguity before answering.Source: §3.1 · p. 4

3.2 SUFFICIENT CONTEXT AUTORATER

  • 93% accuracy: Gemini 1.5 Pro (1-shot) performs best overall on the gold-labeled sufficiency task, leading the other methods in F1 score and accuracy.The evaluation uses 115 human-labeled query-context instances; the authors select this method because it uses only the query and context and has balanced precision and recall.Source: §3.2 · pp. 4–5, Table 1 · p. 5
  • The autorater classifies query-context pairs; FLAMe checks 1,600-token chunks and labels the instance sufficient if any chunk qualifies.TRUE-NLI and Contains GT additionally use the ground-truth answer, while Gemini can operate without it.Source: §3.2 · p. 5
  • Gemini 1.5 Pro (1-shot) is chosen for dataset analyses because real-world evaluation cannot be assumed to provide candidate answers.The authors describe classifying sufficient context as different from, and more complex than, determining whether a specific answer is entailed.Source: §3.2 · p. 5

4 A NEW LENS ON RAG PERFORMANCE

The analysis uses sufficient context to examine benchmark datasets and model responses, then qualitatively studies correct answers despite insufficient context. It finds that context sufficiency and model behavior vary, and that models can still err when context is sufficient or answer correctly when it is not.

4.1 DO BENCHMARK DATASETS HAVE HIGH SUFFICIENT CONTEXT?

  • FreshQA has the highest sufficient-context share; truncation changes it modestly from 2,000 to 6,000 tokens and effectively not at all from 6,000 to 10,000.The study uses 6,000-token contexts for the three datasets in the remainder.Source: §4.1 · p. 6, Figure 2 · p. 6

4.2 INITIAL FINDINGS BASED ON SUFFICIENT CONTEXT

  • RAG reduces abstention: Claude 3.5 Sonnet falls from 84.1% to 52%, GPT 4o from 34.4% to 31.2%, and Gemini 1.5 Pro from 100% to 18.6%.The authors suggest increased confidence from contextual information may contribute to more hallucination rather than abstention.Source: §4.2 · pp. 6–7
  • With sufficient context, models are generally more accurate, yet hallucinations often outnumber abstentions; Gemma hallucinates most, while Claude abstains more when context is insufficient.Claude answers fewer questions correctly with insufficient context than Gemini and GPT; response patterns vary across models and datasets.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 QUALITATIVELY ANALYZING RESPONSES WITH INSUFFICIENT CONTEXT

  • On HotPotQA, all three models correctly answer upwards of 35% of instances despite insufficient context.The paper also reports that correct answers with insufficient context occur in the datasets it analyzes, and discusses common case types.Source: §4.3 · p. 7, Table 2 · p. 8
  • Correct answers despite insufficient context can reflect knowledge gaps bridged by retrieval, ambiguity clarified without an answer, or autorater and evaluator errors.The authors categorize these cases into eight types and relate them to prior findings by Yoran et al. (2024).Source: §4.3 · p. 8, Table 2 · p. 8
  • When context is sufficient but the model is wrong, errors can stem from ground-truth conflicts or failures to compose information, including in multi-hop or arithmetic questions.In many cases, the source of the error remains difficult to determine.Source: §4.3 · p. 8

5 TECHNIQUES TO REDUCE HALLUCINATIONS WITH RAG

The section compares selective abstention with fine-tuning as ways to reduce RAG hallucinations. Selective generation combines sufficient-context labels with model confidence to choose when to answer, while fine-tuning teaches abstention directly.

5.1 SELECTIVE RAG USING SUFFICIENT CONTEXT SIGNAL

  • Abstaining on every insufficient-context instance can lower overall performance because models sometimes answer those questions correctly.The method instead aims to prioritize answers likely to be correct, using a threshold to select coverage.Source: §5.1 · pp. 8–9
  • Unlike generation-time fine-tuning, the independent abstention mechanism offers a controllable accuracy-coverage trade-off for different application settings.The section contrasts strict accuracy compliance in medical domains with maximal coverage in creative generation tasks.Source: §5.1 · p. 9
  • A logistic regression combines sufficient-context labels with self-rated answer probability to predict hallucinations, then abstains below a chosen score threshold.P(True) is used for open-source models and P(Correct) for proprietary models; the signals allow coverage-accuracy thresholds to be tuned.Source: §5.1 · p. 9
  • Over 10% selective-accuracy gains for Gemma 27B on HotpotQA in the highest-accuracy regions; over 5% for Gemini 1.5 Pro near 70% coverage.On Gemma 27B for Musique, overall performance was 18.4% and the sufficient-context signal added no benefit.Source: §5.1 · p. 9, Figure 4 · p. 10

5.2 FINE-TUNING

  • Fine-tuned models have a higher rate of correct answers in many cases, but still hallucinate quite often and more than they abstain; more work is needed to determine a reliable balance.Source: §5.2 · p. 10

6 CONCLUSION

The conclusion presents sufficient context as a lens for analyzing RAG behavior and reports a selective-generation method that reduces hallucinations among answered queries. It also identifies evaluation scope and retrieval analysis as limitations, with extensions proposed for finer-grained and multimodal settings.

6 CONCLUSION

  • The selective-generation method applies to Gemini, GPT, and Gemma and reduces hallucinations by 2–10% on queries the model answers.The analysis also finds frequent hallucinations despite sufficient context and cases where models answer correctly with insufficient context.Source: §6 · p. 10
  • The analysis is limited to QA datasets, leaving context sufficiency in summarization tasks outside its evaluated scope.The conclusion notes that summarization prompts can depend on multiple kinds of information in the context.Source: §6 · p. 10
  • The paper does not explore how often different retrieval methods produce sufficient context, and suggests autorater-guided retrieval as a possible performance improvement.The proposed future directions also include a score-based autorater for ranking contexts and extending sufficient context to multimodal RAG.Source: §6 · p. 10

A SUPPORTING EXPERIMENT INFORMATION

The appendix lists the models and settings used for the experiments, including alternative context-sufficiency classification and fine-tuning configurations.

A SUPPORTING EXPERIMENT INFORMATION

A.1 MODELS

  • The experiments use GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Gemma 2 27B, and Mistral 3 7B, alongside FLAMe for sufficient-context classification.The appendix identifies the API-accessible or publicly available model variants used.Source: §A.1 · p. 15
  • The TRUE-NLI classifier labels context sufficient when the maximum chunk probability exceeds 0.05, a threshold selected for highest F1 on the human-labeled dataset.The experiments use the t5_xxl_true_nli_mixture version.Source: §A.1 · p. 15

A.2 FINE-TUNING SETTINGS

  • Fine-tuning uses LoRA on 2,000-example subsets from Musique-Ans training or HotPotQA development data.The appendix reports rank 4, alpha 8, two epochs, batch size 16, and a learning rate of 1 × 10^-5; it selects the best checkpoint by Correct %.Source: §A.2 · p. 15

A.3 DATASETS

The appendix describes the evaluation samples and retrieval pipelines for HotpotQA and FreshQA, and includes the scope of the fine-tuning comparison.

A.3 DATASETS

  • The HotpotQA and Musique-Ans development sets contribute 500 examples each, while FreshQA uses all “True Premise” questions.Source: §A.3 · p. 15
  • HotpotQA retrieval uses FlashRAG with wiki-18 and e5-base-v2, retrieving the top five documents for each query.The appendix also evaluates REPLUG, which prepends documents individually and ensembles output probabilities across passes.Source: §A.3 · p. 15
  • FreshQA contexts come from dataset-provided URLs, with webpage titles and tables retained and table headers duplicated across chunks when needed.Source: §A.3 · p. 16
  • Table 3 compares closed-book, Vanilla RAG, and three fine-tuning settings on % Correct, % Abstain, and % Hallucinate; “idk” replaces training answers with “I don’t know.”The caption says bold marks the best %C for each model and dataset.Source: Table 3 · p. 17

B.1 FINE-TUNING FULL RESULTS

The fine-tuning experiments test whether abstention examples improve Mistral 3 7B under RAG, alongside a breakdown of performance by context sufficiency. Fine-tuning gains vary by dataset, and errors persist when context is insufficient.

B.1 FINE-TUNING FULL RESULTS

  • For Musique, fine-tuned variants exceed closed-book and Vanilla RAG in %C, but not for HotPotQA; Data Mixes 2 and 3 abstain more than Mix 1, not more than Vanilla RAG.Fine-tuned RAG models often hallucinate more than they abstain.Source: §B.1 · p. 17

B.2 PERFORMANCE BREAKDOWN BY SUFFICIENT CONTEXT

  • The breakdown compares closed-book and open-book performance, then partitions open-book results by sufficient versus insufficient context.Source: §B.2 · p. 17
  • FreshQA has 77.4% sufficient context, attributed to hand-curated source URLs; HotpotQA has more questions answerable without context, with 48.0% closed-book correct for GPT 4o.HotpotQA includes yes/no and multiple-choice questions; FreshQA includes timestamp-dependent answers.Source: Figure 5 · p. 18, Figure 6 · p. 18
  • With sufficient context in 54.8% of cases, top performers still have a 14–16% error rate; with insufficient context in 45.2%, abstention is 50–73% and hallucination is 15–40%.These context-dependent patterns are consistent across HotpotQA, FreshQA, and Musique despite differences in task complexity.Source: Table 4 · p. 19

B.3 COMPARISON OF QA EVALUATION METRICS

The comparison finds that exact-string matching is generally stricter than semantic evaluation, though both metrics reveal similar patterns in model behavior. The examples show how formatting, equivalent wording, and partial matches can change their judgments.

B.3 COMPARISON OF QA EVALUATION METRICS

  • Contains Answer checks exact ground-truth strings, whereas LLMEval uses semantic correctness judgments and distinguishes correct, abstain, and incorrect responses.LLMEval is described as handling variations in verbosity and formatting.Source: §B.3 · pp. 20–21
  • Across FreshQA, Musique, and HotpotQA, Contains Answer is generally stricter than LLMEval, while both metrics reveal similar model-behavior patterns.Source: Table 5 · p. 20, §B.3 · p. 20
  • Examples show Contains Answer missing a date-format variant and a semantic equivalent, while a partial string match is counted as correct despite LLMEval marking it a hallucination.Source: §B.3 · p. 20
  • Across both metrics, LLMs hallucinate even with sufficient context and struggle to abstain when context is insufficient.Source: §B.3 · p. 21

C PROMPTS

The appendix specifies prompts for labeling context sufficiency and evaluating answers, then gives question-answering prompts for several settings. These instructions define the criteria, expected response formats, and abstention behavior used in the paper.

C.1 SUFFICIENT CONTEXT AUTORATER PROMPT

  • The autorater prompt asks an evaluator to determine whether context suffices to infer an answer, reason through question assumptions and calculations, and return an explanation plus a JSON label.Source: §C.1 · p. 22

C.2 FLAME PROMPT

  • The FLAMe prompt supplies documents and a question and requests a judgment about whether the documents suffice to infer the answer.Source: §C.2 · p. 23

C.3 LLMEVAL PROMPT

  • LLMEval compares predicted answers with ground truth by meaning rather than exact wording, assigning perfect, acceptable, incorrect, or missing judgments.The evaluation description also notes answer normalization and treatment of empty strings and variations of “I don’t know” as missing.Source: §C.3 · p. 23

C.4 DATASET QUESTION ANSWER PROMPTS

  • The CoT question-answering prompt requests a concise, journalistic response based only on search results and requires the answer to end with “The answer is:”.The final answer is extracted by pattern matching text after that phrase.Source: §C.4 · p. 24
  • A separate question-answering prompt asks for only the answer based on provided search results, with example question, references, and answer fields.Source: §C.4 · p. 24

D FINE-TUNING AND RAG PROMPTS FOR MISTRAL

Loading 2411.06037v3…