PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models Struggle to Learn Long-Tail Knowledge

Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, Colin Raffel

arXiv:2211.08411v2cs.CLcs.LG

TL;DR

Language models may learn common web facts more readily than rarely occurring long-tail knowledge, but the relationship between memorization and pre-training support remains to be characterized. The paper counts relevant entity-linked documents, finds accuracy strongly related to their number across settings, and evaluates scaling and retrieval augmentation as responses.

  • Problem

    It remains unclear whether language models capture broad knowledge or mainly the facts that frequently appear in their pre-training data.

  • Method

    The paper entity-links pre-training corpora and QA pairs, then counts documents where salient question and answer entities co-occur.

  • Results

    Accuracy strongly correlates with relevant document count across QA datasets, corpora, and model sizes; BLOOM-176B rises from 25% to above 55% as counts increase from 10^1 to 10^4.

  • Takeaways & Limitations

    Scaling improves long-tail learning but may require dramatic model growth, while retrieval augmentation reduces dependence on extensive relevant pre-training information.

  • Takeaways & Limitations

    The pipeline sometimes misidentifies entities and treats all documents containing salient entities as relevant, and the study leaves other task types open.

Abstract

from arXiv · show

The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models. However, while certain pieces of information are ubiquitous on the web, others appear extremely rarely. In this paper, we study the relationship between the knowledge memorized by large language models and the information in pre-training datasets scraped from the web. In particular, we show that a language model's ability to answer a fact-based question relates to how many documents associated with that question were seen during pre-training. We identify these relevant documents by entity linking pre-training datasets and counting documents that contain the same entities as a given question-answer pair. Our results demonstrate strong correlational and causal relationships between accuracy and relevant document count for numerous question answering datasets (e.g., TriviaQA), pre-training corpora (e.g., ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models are better at learning long-tail knowledge, we estimate that today's models must be scaled by many orders of magnitude to reach competitive QA performance on questions with little support in the pre-training data. Finally, we show that retrieval-augmentation can reduce the dependence on relevant pre-training information, presenting a promising approach for capturing the long-tail.

1. Introduction

The paper links language-model question-answering ability to the amount of relevant information in pre-training data, finding that long-tail knowledge remains difficult even for large models. It evaluates document-count relationships, counterfactual retraining, scaling, and retrieval augmentation as possible explanations and remedies.

  • Rarely occurring web knowledge forms a long tail that language models may fail to learn reliably.
  • The study measures relevant pre-training information by counting documents where salient question and answer entities co-occur.The pipeline applies entity linking to pre-training corpora and downstream factoid QA pairs.
  • 25% to above 55%: BLOOM-176B accuracy rises when relevant documents increase from 10^1 to 10^4.
  • Removing relevant documents significantly reduces accuracy, supporting a likely causal relationship between relevant pre-training information and question answering.
  • Scaling improves long-tail learning but may require one quadrillion parameters for competitive performance, whereas retrieval augmentation reduces dependence on extensive relevant pre-training text.Retrieval helps when it finds a relevant document, although retrieval systems retain a mild dependence on relevant document count.

2. Identifying Relevant Pre-training Data

The paper operationalizes relevant pre-training knowledge through entity linking and entity co-occurrence across massive corpora. Human evaluation finds moderate precision, while the authors acknowledge errors and limitations in the counting pipeline.

  • Background and Research Question: The research question is whether language models mainly capture facts that frequently appear in their pre-training data.
  • Background and Research Question: The method uses closed-book few-shot QA to relate answerability to the number of relevant pre-training documents.
  • Our Approach: Relevant documents are identified by extracting salient question and answer entities and counting documents containing both.For the Dante example, the pipeline counts documents containing Dante Alighieri and Florence.
  • Our Approach: Entity linking is parallelized across large pre-training datasets, with indexes recording document-level entity occurrences and co-occurrences.The pipeline processed 2.1TB of data in approximately 3 weeks on 128 CPU cores.
  • Human Evaluation: Approximately 60% precision: human evaluation found 33% of sampled documents answerable and another 27% relevant to TriviaQA questions.
  • Human Evaluation: The pipeline can misidentify entities and overcount relevance, but remains sufficiently precise and recall-complete for large-scale correlational and causal analyses.

3. LM Accuracy Depends on Relevant Document Count

Across tested models and datasets, QA accuracy rises with the number of relevant pre-training documents, while sparse support produces poor long-tail performance. Counterfactual retraining and scaling analyses further connect document availability and model size to factual accuracy.

  • QA accuracy is strongly correlated with relevant document count across tested models and datasets.
  • 25% to above 55%: BLOOM-176B accuracy increases as relevant documents rise from 10^1 to 10^4.
  • Simpler counts using only question or answer entities fail to correlate with accuracy when entity co-occurrences are rare.This supports defining relevant documents as those containing both entities.
  • Human accuracy is highest on questions with few relevant documents, opposite to the model trend.The human evaluation provides the necessary background text.
  • R^2 = 0.98: extrapolated scaling trends imply immensely large models would be needed for high accuracy on rare facts.
  • Removing relevant training documents significantly worsens counterfactual performance when the original document count is high.For questions with few original relevant documents, baseline and counterfactual performance differ little.

4. Methods to Improve Rare Fact Learning

The paper evaluates dataset scale, model scale, and retrieval augmentation as ways to mitigate long-tail knowledge gaps. Retrieval and background context reduce the dependence on sparse pre-training support, while model scaling remains extremely costly.

  • 4.1. Can We Scale Up Datasets?: Moderate dataset scaling, such as 5x, usually yields only small accuracy gains on less-prevalent knowledge.
  • 4.1. Can We Scale Up Datasets?: Relevant-document counts are highly rank-correlated across ROOTS, The Pile, C4, OpenWebText, and Wikipedia.The reported correlations range from 0.87 to 0.97.
  • 4.2. Can We Scale Up Models?: Over 10^18 parameters: extrapolation suggests matching supervised or human performance on rare Natural Questions facts would require a BLOOM model of this scale.
  • 4.2. Can We Scale Up Models?: Providing gold background paragraphs makes GPT-Neo accuracy trends roughly match human trends, including on questions with low document counts.
  • 4.3. Can We Use Retrieval Augmentation?: BM25-augmented models outperform closed-book models across all document-count ranges, especially for rare examples.BM25 itself retains a mild dependence on relevant document count.

5. Related Work

Related work connects language-model behavior to training-data frequency, source, memorization, and retrieval. This paper distinguishes itself by entity-linking relevant documents and combining correlational analysis with causal retraining and scaling studies.

  • Prior work links mathematical reasoning and few-shot accuracy to training-data frequency and corpus source.
  • Unlike influence-identification approaches, this work directly estimates relevant-document counts through entity linking.
  • Memorization research often studies verbatim disclosure, whereas this paper examines non-verbatim fact learning in question answering.
  • Concurrent work uses proxies such as Wikipedia entity popularity, while this paper counts occurrences with entity-linking methods.
  • This paper additionally conducts causal retraining experiments and studies model scaling for long-tail QA.

6. Conclusion and Future Work

The paper links LLM QA success to how widely knowledge appears in pre-training data, with long-tail retention remaining difficult. It argues that tracing performance to pre-training data clarifies current model behavior and motivates retrieval-focused future work.

  • LLMs are mainly successful on open-domain QA questions whose knowledge appears widely in their pre-training datasets.
  • Retaining long-tail knowledge remains an open problem because simply scaling model and dataset size may be insufficient.
  • The analysis may inform tasks involving memorized knowledge, including private-text analysis, commonsense reasoning, and source-code prediction.
  • Tracing accuracy back to pre-training-data properties can help elucidate model successes and failures and identify paths toward improving existing systems.

A. Additional Results: Relevant Document Scaling

Additional experiments extend the relationship between QA accuracy and relevant pre-training documents across BLOOM and GPT-3 settings. The GPT-3 document-count estimates are uncertain because its training data are not public.

  • BLOOM on Natural Questions and GPT-3 on TriviaQA and Natural Questions show that QA performance relates to relevant pre-training-document counts.
  • GPT-3 relevant-document counts may be inaccurate because its training data are unavailable and counts are estimated using OpenWebText.
  • Natural Questions accuracy is lower overall than TriviaQA accuracy, although the trends match those observed for BLOOM.
  • GPT-3 results across Natural Questions and TriviaQA match trends observed for BLOOM and GPT-Neo.

B. Additional Results: Model Scaling

Additional BLOOM and GPT-Neo experiments show that long-tail QA accuracy follows the same log-linear relationship with model size across the tested datasets and model families.

  • Long-tail QA accuracy scales with model size for BLOOM on TriviaQA and GPT-Neo on Natural Questions and TriviaQA.
  • The additional scaling results reproduce the log-linear trend reported in the main text.
  • Across BLOOM TriviaQA and GPT-Neo Natural Questions and TriviaQA, the trends remain similar, but the required scaling degree varies by setting.
  • The scaling results indicate that models would need to be scaled up dramatically to reach high QA accuracy.

C. Relevant Document Counting Heuristics

The paper compares entity-co-occurrence document counts with question-only and answer-only counts as heuristics for predicting QA accuracy. All correlate overall, but the baselines fail for low-co-occurrence questions.

  • All three document-counting heuristics correlate with QA accuracy when counting question-answer co-occurrences, question entities, or answer entities.
  • For QA pairs with fewer than 5 question-and-answer entity co-occurrences, neither question-only nor answer-only counts correlates with QA accuracy.
  • QA accuracy increases as the counts of question entities and answer entities increase when each is considered alone.
Loading 2211.08411v2…