PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval Helps or Hurts? A Deeper Dive into the Efficacy of Retrieval Augmentation to Language Models

Seiji Maekawa, Hayate Iso, Sairam Gurajada, Nikita Bhutani

arXiv:2402.13492v3cs.CL

TL;DR

Because retrieved information can override knowledge an LM already has, the paper examines the underexplored interplay of entities and relations. It introduces WiTQA and finds that recall and retrieval performance vary with fact popularity, with retrievers more robust on long-tail information for long-tail entities.

  • Problem

    Prior work largely examined entity effects and considered retrievers or LM recall separately, leaving limited evidence on their fact-level interplay.

    Source: §1 · p. 1

  • Method

    WiTQA pairs QA examples and supporting passages with popularity measures for question entities and entity-relation pairs, enabling fact-centric analysis.

    Source: §1 · p. 2

  • Results

    Across 10 LMs and four retrieval methods, recall varied with model size, small RALMs were robust with supporting passages, and retrievers were more robust on long-tail information for long-tail entities.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Retrieval's benefit depends on fact popularity: it suits long-tailed entity-relation pairs, while well-known pairs can face override issues.

    Source: §1 · p. 2

    Prompt tuning was limited, and encouraging answers regardless of passage relevance could increase incorrect answers, a trade-off left for future work.

    Source: Limitations · pp. 9–10

Abstract

from arXiv · show

While large language models (LMs) demonstrate remarkable performance, they encounter challenges in providing accurate responses when queried for information beyond their pre-trained memorization. Although augmenting them with relevant external information can mitigate these issues, failure to consider the necessity of retrieval may adversely affect overall performance. Previous research has primarily focused on examining how entities influence retrieval models and knowledge recall in LMs, leaving other aspects relatively unexplored. In this work, our goal is to offer a more detailed, fact-centric analysis by exploring the effects of combinations of entities and relations. To facilitate this, we construct a new question answering (QA) dataset called WiTQA (Wikipedia Triple Question Answers). This dataset includes questions about entities and relations of various popularity levels, each accompanied by a supporting passage. Our extensive experiments with diverse LMs and retrievers reveal when retrieval does not consistently enhance LMs from the viewpoints of fact-centric popularity. Confirming earlier findings, we observe that larger LMs excel in recalling popular facts. However, they notably encounter difficulty with infrequent entity-relation pairs compared to retrievers. Interestingly, they can effectively retain popular relations of less common entities. We demonstrate the efficacy of our finer-grained metric and insights through an adaptive retrieval system that selectively employs retrieval and recall based on the frequencies of entities and relations in the question.

1 Introduction

Retrieval augmentation can mitigate factual errors, but irrelevant passages may override knowledge an LM already has. The paper addresses missing fact-centric evidence by introducing WiTQA and examining recall and retrieval across entity-relation popularity.

1 Introduction

  • Prior work focused on entity effects or examined retrievers and LM recall separately, while irrelevant retrieved passages can override correct LM knowledge.Source: §1 · pp. 1–2
  • WiTQA supports fact-centric analysis with QA pairs, supporting passages, and popularity scores for question entity-relation pairs.Source: §1 · p. 2
  • Larger LMs recall frequently encountered entity-relation pairs, yet performance drops on minor facts; they can learn long-tailed relations about renowned entities.Source: §1 · p. 2
  • Retrievers are more robust than LM recall for long-tailed entity-relation pairs, but that advantage does not extend to well-known pairs and may create override issues.Source: §1 · p. 2
  • For long-tailed entities, LMs achieve higher accuracy than retrievers on well-known entity-relation pairs.Source: §1 · p. 2
  • Selective memory integration chooses retrieval or LM memory using entity and relation frequencies, and improves QA performance by up to 10.1%.Source: §1 · p. 2

2 Background

The background describes open-domain QA as generating answers to questions with retrieved passages and an LM-based reader. It then frames the challenge of combining parametric knowledge with retrieved information, motivating WiTQA for error-pattern analysis.

2.1 Open domain Question Answering

  • Open-domain QA generates an answer from a question, typically retrieving a passage for a reader model; Figure 1 outlines WiTQA's four-stage dataset construction.Source: §2.1 · pp. 2–3, Figure 1 · p. 3

2.2 Parametric vs Non-Parametric Knowledge

  • LM-based readers can answer without external passages, but irrelevant retrieved passages may override parametric knowledge and misguide their answers.Source: §2.2 · p. 3
  • Prior approaches combined models based on entity popularity or trained smaller models to decide when to retrieve and assess passage relevance.Source: §2.2 · p. 3
  • Earlier work had not clarified interconnected LM and retrieval-model error patterns, motivating WITQA as a dataset for analyzing them independently and together.Source: §2.2 · p. 3

3 The WITQA dataset

WiTQA is designed to study how Wikipedia popularity of subjects and subject-relation pairs relates to LM and retrieval performance. Its construction combines triple extraction, popularity-based sampling, answer expansion, and round-trip question generation.

3.1 Dataset creation

  • Four stages create WiTQA: Wikipedia triple extraction, triple sampling, answer-candidate expansion, and question generation with round-trip refinement.Source: §3.1 · p. 3

3.1.1 Triple extraction

  • Wikipedia abstract entities are mapped to Wikidata IDs to extract triples; subject and subject-relation counts measure popularity, and the highest-entailment passage supports each triple.Source: §3.1.1 · pp. 3–4

3.1.2 Triple sampling

  • To cover the long-tailed S-R-count distribution, the authors sampled 32 relations across seven count intervals, taking up to 200 triples per relation per interval.Source: §3.1.2 · p. 4

3.1.3 Answer candidate expansion

  • Because a triple’s object may not be the only valid answer, acceptable answers include Wikidata entities and aliases linked to the same subject-relation pair.Source: §3.1.3 · p. 4

3.1.4 Question generation with roundtrip refinement.

  • Model-assisted round-trip generation addresses the poor quality and diversity of fixed relation templates by generating a question from context and then generating its answer.Source: §3.1.4 · pp. 4–5
  • 95% (14,094/14,837) of questions passed the three validity criteria within three round-trip iterations; annotators rewrote the remaining 743.Source: §3.1.4 · p. 5

3.2 Dataset Statistics

  • 14,837 questions make up WiTQA; compared with EntityQuestions and PopQA, it includes popularity indicators and supporting passages and has more diverse S-R-count distributions.Source: Figure 2 · p. 4, Table 2 · p. 4, §3.2 · p. 5

4 Experiments: Recall or Retrieve

Across the experiments, recall and retrieval effectiveness vary with model size and question popularity, so retrieval does not help every question uniformly. The authors use these patterns to test a selective strategy based on entity and entity-relation frequencies.

4.1 Setup

  • The experiments evaluated 10 language models with four retrieval methods, using top-one retrieved passages and exact-match substring scoring.The setup also included GenRead and Oracle among the retrieval settings.Source: §4 · p. 5, §4.3 · p. 6, §4.1 · p. 6

4.2 Analysis of Model’s Recall Ability

  • Large models recalled popular facts well, but their accuracy fell on less popular subject-relation pairs; Flan-T5-large reached 80% on questions with over 1,000 S-R counts.Accuracy rose consistently with S-R and subject counts, but not with subject page-view counts.Source: §4.2 · p. 6, Figure 3 · p. 6, Figure 4 · p. 6

4.3 When Do Retrievers Help

  • BM25, Contriever, GTR, and BGE improved small and medium models by up to 49.2%, whereas GenRead improved larger models by up to 4.1%.The authors suggest larger models may maintain a coherent chain of thought more effectively.Source: §4.3 · p. 7
  • Retrieval augmentation generally helped smaller models more than larger ones, while Oracle passages improved every model; incorrect passages often hindered correct answers.With Oracle passages, Flan-T5-small reached 78.1% accuracy, and the gap from ordinary retrievers was attributed to retrieval errors.Source: §4.3 · pp. 6–7, Figure 6 · p. 8

4.4 Deep Dive into Errors

  • For GPT-3.5, retrieval methods were more robust than Vanilla on less popular questions, while Vanilla performed better on popular questions; retrieval accuracy tracked RALM accuracy closely.Agreement between RALM performance and passage accuracy ranged from 82.4% for Contriever to 93.1% for Oracle.Source: §4.4 · pp. 7–8
  • In head-tail and tail-head groups, Llama and GPT series achieved accuracy comparable to or exceeding passage accuracy, while medium-scale models underperformed passage accuracy and could benefit from augmentation.Tail-head passage accuracy was less than 60%, significantly lower than other groups. The authors attribute this retrieval error to frequent entity mentions but infrequent subject-relation mentions, making relevant passages difficult to identify from a large pool.Source: §4.4 · p. 9
  • In the tail-tail group, even GPT models struggled to recall rare-entity facts, while retrievers were more robust, making retrieval augmentation generally helpful.Supervised GTR and BGE outperformed vanilla LMs across all four popularity groups; the authors relate this to fine-tuning on Wikipedia QA datasets.Source: §4.4 · pp. 8–9
  • Using S-R and S counts as thresholds, selective memory integration improved GPT-3.5 accuracy by 10.1% over Vanilla and GPT-4 accuracy by 8.1% over BM25.The method uses the vanilla LM by default and selectively applies retrieval augmentation.Source: §4.4 · p. 9

5 Further Related Work

Prior work improves RALM robustness by assessing passage relevance, rewriting queries, or combining retrieved and generated passages. This study instead uses question popularity metrics to complement those model-centric approaches.

5 Further Related Work

  • Earlier approaches assess passage relevance, rewrite queries, or integrate retrieved and generated passages, while this study uses Wikipedia-derived question popularity as a complementary, data-centric signal for deciding when to retrieve.Examples include Chain-of-Note for passage relevance, query rewriting, and combining retrieved with LM-generated passages.Source: §5 · p. 9

6 Conclusion

The paper introduces WITQA, a QA dataset with supporting passages that enables evaluation by question popularity and retrieval augmentation. Experiments report size-dependent recall, benefits from supporting passages, and greater retriever robustness for long-tail information about rare entities.

6 Conclusion

  • WITQA pairs questions and answers with supporting passages and enables evaluation of language models by question popularity and retrieval augmentation.Experiments investigate zero-shot performance across 10 language models and four retrieval methods.Source: §6 · p. 9
  • The experiments find that recall depends on model size, even GPT models struggle with minor facts, and retrievers are more robust on long-tail information about long-tail entities.Small RALMs performed robustly when given supporting passages, which the authors interpret as evidence that retrieval errors are a primary source of RALM errors.Source: §6 · p. 9

Limitations

The paper identifies limits in its corpus assumptions, prompt tuning, and question scope. These constraints bound what its findings cover and leave a prompt-related trade-off for future work.

Limitations

  • The analysis assumes Wikipedia text reflects pre-training distributions, but cannot verify that assumption for proprietary models such as GPT-4.The pre-training texts of recent proprietary models are inaccessible.Source: Limitations · p. 9
  • Only limited prompt tuning was conducted, and encouraging answers despite passage irrelevance could raise incorrect-answer rates, a trade-off left for future work.The authors note that this outcome is undesirable in practical applications.Source: Limitations · pp. 9–10
  • The study focuses on triple-based questions, not multi-hop questions, where identifying the subject and relation can be challenging.The authors choose triple-based questions to analyze LMs using clearly characterized questions.Source: Limitations · p. 10

Ethical Consideration

The paper reports that internal annotators handled question rewriting and describes how it selected supporting passages for triples using entailment scores.

Ethical Consideration

  • Internal annotators handled question rewriting and were told directly how the authors would use the data; they earned more than the minimum wage.Source: Ethical Consideration · p. 10

A Detailed Setup for NLI Scores of Supporting Passages

B Prompts for Question Generation/Refinement and Experiments

The appendix provides prompt templates for question generation and QA, and describes an iterative refinement procedure that checks generated question-answer pairs against three criteria.

B Prompts for Question Generation/Refinement and Experiments

  • The appendix shows QA prompt templates for vanilla, GenRead, BM25, Contriever, and Oracle, plus a separate GenRead passage-generation prompt.Source: B Prompts for Question… · p. 13
  • Roundtrip refinement checks whether a generated question is answerable, contains the subject, and omits the object, then adds qualifying questions to WiTQA.If criteria are not met, the procedure appends feedback and regenerates a question; after k iterations without a qualifying pair, the authors write questions based on the triples.Source: B Prompts for Question… · p. 13
  • Figure 10’s question-generation prompt takes the supporting passage, subject, relation, object, and Wikidata relation description as inputs.Source: Figure 10 · p. 13, B Prompts for Question… · p. 13

C Additional Experimental Details

The appendix details experimental implementation, WiTQA statistics, and a popularity-based selective retrieval procedure, including how its thresholds are estimated and evaluated.

C Additional Experimental Details

C.1 Additional WiTQA Statistics

  • The appendix describes roundtrip question refinement: GPT-3.5 iteratively revises questions using answerability, subject-presence, and object-absence checks, with a special case for substring overlap.Source: §C.1 · p. 14
  • WiTQA includes long-tail relations such as cuisine and medical condition, which the authors say improves question diversity; its statistics also report counts by relation.Source: §C.1 · pp. 13–14

C.2 Selective Memory Integration

  • The selective system uses an RALM for questions with low subject-relation and subject counts, and Vanilla LM for other groups.The authors motivate this choice by complementary performance patterns across question popularity groups.Source: §C.2 · p. 14

C.2.1 Settings

  • The authors estimate subject-relation and subject-count thresholds on 50% of WiTQA questions to maximize overall accuracy, then evaluate the selection on the remaining questions.They report averages and standard deviations across five random dataset splits.Source: §C.2.1 · pp. 14–15

C.2.2 Results

Selective memory integration outperforms Vanilla and BM25 across models, while larger models use retrieval less often.

C.2.2 Results

  • 7.7%, 10.1%, and 8.1% improvements over baselines for Llama-2-70B, GPT-3.5, and GPT-4, respectively; selective integration beats Vanilla and BM25 across models.For GPT-series models, contexts often do not support the queried facts, so models may avoid answering; selective integration performs lower than or comparably to Llama-2-70B.Source: Figure 14 · p. 15, §C.2.2 · p. 15
  • Retrieval use shifts downward as model size grows: GPT-4 retrieves for 27.7% of questions, versus 43.0% for Llama-2-70B.The authors attribute this trend mainly to larger models recalling facts more correctly.Source: Figure 15 · p. 15, §C.2.2 · p. 15

C.3 Retriever setup

  • Wikipedia documents are chunked with chunk_size 256 and chunk_overlap 0 for retrieval.Source: §C.3 · p. 15
  • Tables 4 and 5 report counts of WiTQA questions containing relations and relations in extracted Wikipedia-abstract triples, respectively.Source: Table 4 · p. 16, Table 5 · p. 16
Loading 2402.13492v3…