PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable, Adaptable, and Attributable Language Models with Retrieval

Akari Asai, Zexuan Zhong, Danqi Chen, Pang Wei Koh, Luke Zettlemoyer, Hannaneh Hajishirzi, Wen-tau Yih

arXiv:2403.03187v1cs.CLcs.AIcs.LG

TL;DR

Parametric LMs face factual errors, difficulty adapting to new data, and weak verifiability. This position paper advocates retrieval-augmented LMs that use datastore text at inference and proposes a roadmap for their development. It concludes that adoption remains limited despite their advantages, and calls for advances in retrievers and datastores, retriever–LM interaction, and infrastructure.

  • Problem

    Parametric LMs face factual errors and difficult verification and adaptation, while retrieval-augmented LMs remain limited beyond knowledge-intensive tasks and have weak component interaction and scaling support.

    Source: §1 · p. 1

  • Method

    The paper advocates retrieval-augmented LMs that use external datastore text at inference and proposes advances in retrieval, architecture and training, and infrastructure.

    Source: §1 · pp. 1–2

  • Results

    The paper argues retrieval-augmented LMs can improve reliability, attribution, and adaptation, but their adoption remains limited and current approaches focus narrowly on post-hoc retrieval added to parametric LMs.

    Source: §1 · pp. 1–2, §6 · p. 9

  • Takeaways & Limitations

    The roadmap calls for reconsidering relevance and datastore contents, deepening retriever–LM interaction, and developing open, interdisciplinary infrastructure for efficient training and inference.

    Source: §1 · p. 2

    Current retrieval approaches can miss helpful text outside conventional relevance, while input augmentation has shallow retriever–LM interaction and increases inference costs as context grows.

    Source: §4.2 · pp. 6–7

Abstract

from arXiv · show

Parametric language models (LMs), which are trained on vast amounts of web data, exhibit remarkable flexibility and capability. However, they still face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. In this position paper, we advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs. By incorporating large-scale datastores during inference, retrieval-augmented LMs can be more reliable, adaptable, and attributable. Despite their potential, retrieval-augmented LMs have yet to be widely adopted due to several obstacles: specifically, current retrieval-augmented LMs struggle to leverage helpful text beyond knowledge-intensive tasks such as question answering, have limited interaction between retrieval and LM components, and lack the infrastructure for scaling. To address these, we propose a roadmap for developing general-purpose retrieval-augmented LMs. This involves a reconsideration of datastores and retrievers, the exploration of pipelines with improved retriever-LM interaction, and significant investment in infrastructure for efficient training and inference.

1. Introduction

Parametric LMs can produce factual errors, are difficult to verify and adapt, and require large models; retrieval-augmented LMs instead use external text at inference. This position paper argues for their broader adoption and outlines a roadmap addressing retrieval, retriever–LM interaction, and scaling.

1. Introduction

  • Parametric LMs face factual errors, difficult verification and adaptation, privacy-related data-removal challenges, and prohibitively large model sizes.The paper argues that scaling alone has not overcome these limitations and may exacerbate them.Source: §1 · p. 1
  • Retrieval-augmented LMs use datastore text at inference rather than relying only on knowledge internalized during pretraining, and can be more reliable, adaptable, attributable, and parameter-efficient.The contrast is depicted in Figure 1; the paper lists reduced factual errors, better attributions, flexible sequence opt-in and out, easier adaptation, and fewer required model parameters.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Adoption beyond knowledge-intensive tasks remains limited because retrievers may miss helpful text, input augmentation yields shallow interaction, and scalable training and inference infrastructure is lacking.The paper also associates shallow interaction with unsupported generations, susceptibility to irrelevant text, and difficulty using multiple documents.Source: §1 · p. 2
  • The proposed roadmap calls for reconsidering relevance and datastores, improving retriever–LM interaction through architecture and training, and investing in open, interdisciplinary infrastructure.The infrastructure agenda includes scaling datastores to trillion tokens.Source: §1 · p. 2

2. How Far Can We Go with Parametric LMs?

The paper defines parametric LMs as predicting outputs from learned parameters without external data at test time. It describes five practical weaknesses associated with storing knowledge in those parameters: factuality, verification, data removal, adaptation cost, and model size.

2. How Far Can We Go with Parametric LMs?

  • A parametric LM predicts outputs from trained parameters without accessing external data beyond the task input at test time.The paper represents this as y = fθ(x), with θ trained to predict future or masked tokens from a large-scale text dataset.Source: §2 · p. 2

2.1. Weaknesses of Parametric LMs

  • Parametric LMs can hallucinate, particularly on long-tail facts that are less represented in pretraining; scaling yields only minor improvements on this knowledge.The passage also notes that increasing synthetic instruction-tuning data may not improve factuality.Source: §2.1 · p. 2
  • Their lengthy, plausible outputs lack clear provenance, making post-hoc attribution and factual verification challenging and largely unsolved.Source: §2.1 · p. 2
  • Identifying and removing privacy- or copyright-protected training sequences is difficult, and missing attributions make tracing specific instances non-trivial.Source: §2.1 · p. 2
  • Adapting models trained on static text to new distributions requires continued training or costly post-adaptation, with potential catastrophic forgetting.Parametric knowledge can become obsolete, and models that perform well on general-purpose domains may struggle on expert domains.Source: §2.1 · pp. 2–3
  • Models often exceeding a hundred billion parameters remain impractical for many groups to host, despite performance and recall benefits from scaling.Source: §2.1 · p. 3

3. How Can Retrieval-Augmented LMs Address These Issues?

Retrieval-augmented LMs pair a retriever and a parametric LM, using retrieved datastore text alongside the prompt at inference. Prior work reports benefits across factuality, attribution, sequence control, adaptation, and parameter efficiency, while the paper also traces a shift toward adding retrieval to existing LMs.

3. How Can Retrieval-Augmented LMs Address These Issues?

  • At inference, a retriever uses an index built from a datastore to select text, which the LM combines with the original prompt to predict its output.Term-based indexes may use weighted bag-of-words vectors, while neural retrieval indexes may contain document embeddings.Source: §3 · p. 3
  • Earlier NLP retrieval-augmented systems included task-specific QA and machine-translation work, as well as models that jointly or sequentially trained retrieval and language-model components.More recent RAG methods often add retrieval to existing powerful parametric LMs without additional training by concatenating retrieved text with the input.Source: §3 · p. 3

3.1. Effectiveness of Retrieval-Augmented LMs

  • Retrieval-augmented LMs can capture long-tail knowledge and reduce hallucinations, improving the factuality of generated outputs.Source: §3.1 · p. 3
  • Inference-time retrieved evidence can help practitioners inspect outputs, and retrieval using that evidence provides more accurate attributions than post-hoc attribution.Source: §3.1 · pp. 3–4
  • Improved attributions and datastore updates support sequence exclusion, while inference-time datastores can maintain performance on domains absent from training data.Source: §3.1 · p. 4
  • Interchangeable datastores allow domain, application, and timestamp customization without additional training; retrieval augmentation has also outperformed downstream-domain fine-tuning on QA.Datastore updates with current knowledge can bypass the obsolescence of parametric knowledge, and adaptation has also been reported for machine translation.Source: §3.1 · p. 4
  • Retrieval-augmented LMs can use far fewer LM parameters than larger parametric LMs and still outperform them, including on knowledge-intensive QA.The paper reports that QA comparisons include parametric LMs with orders of magnitude more parameters.Source: §3.1 · p. 4

4. Why Haven’t Retrieval-Augmented LMs Been Widely Adopted?

The paper reviews why retrieval-augmented LMs remain less widely adopted despite their promise, and organizes their architectures, training approaches, applications, and datastore choices. It identifies limitations in broad-task retrieval, retriever–LM interaction, and scaling infrastructure.

4.1.1. ARCHITECTURE

  • Architectures vary by retrieved-text granularity, incorporation method, and retrieval frequency, with input augmentation, intermediate fusion, and output interpolation as the main incorporation groups.Retrieval may occur once, at token intervals, or adaptively.Source: §4.1.1 · pp. 4–5
  • Input augmentation flexibly combines retrieval and LM models, but encoding many documents in the prompt raises context-window and inference-cost concerns.FiD parallelizes encoding but still encodes repeatedly for each input.Source: §4.1.1 · p. 5
  • Intermediate fusion incorporates pre-encoded chunks in intermediate LM layers, while output interpolation combines token distributions but requires a much larger index.Intermediate fusion can require extensive architecture modification and pre-training; output interpolation needs datastore-token embeddings.Source: §4.1.1 · p. 5

4.1.2. TRAINING

  • Training ranges from independent or sequential component training to joint training, which improves component interaction but incurs costly index updates.Joint-training work uses asynchronous index updates or in-batch approximations; update frequency trades performance against computational overhead.Source: §4.1.2 · pp. 5–6

4.1.3. APPLICATIONS AND DATASTORES

  • Retrieval augmentation is most pronounced on knowledge-intensive tasks, while datastore choice is task-dependent and ranges from general corpora to domain-focused collections.Prior work commonly uses datastores on the order of 10^9 tokens, while some scale beyond one trillion tokens.Source: §4.1.3 · p. 6

4.2. Limitations of Current Retrieval-Augmented LMs

  • Retrieval-augmented LMs have restricted success beyond knowledge-intensive tasks; lexical or semantic similarity may not identify contexts that help reasoning or general understanding.The paper hypothesizes that retrieval and LM objectives and datastore choices are misaligned, and notes reliance on Wikipedia may limit coverage of queries without direct answers there.Source: §4.2 · p. 6
  • Common input-augmentation approaches lack close retriever–LM interaction, and can amplify unsupported generations, sensitivity to irrelevant context, and inference costs.Longer contexts can also lead LMs to overlook significant input portions.Source: §4.2 · pp. 6–7
  • Specialized infrastructure remains limited: synchronously updating large indexes is computationally expensive, while searches across billions of embeddings can require hundreds of GPUs or prohibitive RAM.Inference can also be substantially more expensive than with standard parametric LMs, especially with datastores above one trillion tokens.Source: §4.2 · p. 7

5. How Can We Further Advance Retrieval-Augmented LMs?

The roadmap calls for retrieval-augmented LMs that can use helpful text across diverse tasks, interact more deeply with LMs, and scale through stronger infrastructure.

5. How Can We Further Advance Retrieval-Augmented LMs?

  • The roadmap prioritizes finding helpful text for diverse tasks, improving retriever–LM interaction, and building infrastructure for wider adoption.It frames these as technical constraints to address through community effort.Source: §5 · p. 7

5.1. Rethinking Retrieval and the Datastore (C1)

  • Retrievers should identify useful documents beyond lexical or semantic matches, while datastore composition should suit the task and domain.Helpful text may include demonstrations or unlabeled text; how to balance datastore domains and filter for quality remains open.Source: §5.1 · pp. 7–8

5.2. Enhancing Retriever-LM Interactions (C2)

  • Integrated architectures such as output interpolation or intermediate fusion may address input augmentation’s limitations, but require scalable designs and pre-training.Smaller-granularity retrieval poses scaling challenges, and these architectures are often smaller than massively pre-trained parametric LMs because of pre-training costs.Source: §5.2 · p. 8
  • Retrieval-aware pre-training can help LMs use retrieved context, while corpus reordering with similar text chunks offers improvements without modifying pipelines or architectures.The paper identifies methods that induce use of retrieved context with minimal or no additional costs as a promising direction.Source: §5.2 · p. 8
  • Post-training adaptation and inference-time methods, including filtering irrelevant context, could improve retrieval-augmented LMs under resource constraints.Existing instruction-tuned LMs may perform suboptimally when augmented because they were not explicitly trained to use retrieved context.Source: §5.2 · p. 8
  • Jointly optimizing retrievers with LMs, including through end-to-end reranking or training without retrieval labels, is a route to broader applications.Some tasks update only input encodings without modifying the index after pre-training.Source: §5.2 · p. 8

5.3. Building Better Systems and Infrastructures for Scaling and Adaptation (C3)

  • Scaling requires faster search, compact indexes, and efficient datastore loading, alongside infrastructure suited to retrieval’s CPU-heavy workloads.The paper also calls for standardized, open-source retrieval-augmented LM pipelines and evaluation benchmarks beyond RAG.Source: §5.3 · pp. 8–9

6. Conclusion

The paper advocates retrieval-augmented LMs as a next generation of language models and outlines a roadmap across architectures, training, and infrastructure.

6. Conclusion

  • The paper advocates retrieval-augmented LMs for more reliable, adaptable, and attributable systems, while linking limited adoption to narrow, post-hoc retrieval augmentation.Its roadmap emphasizes architectures, training methodologies, infrastructure, and collaborative interdisciplinary efforts.Source: §6 · p. 9

Impact Statements

The paper sees retrieval-augmented LMs as a possible way to address parametric-model limitations, while noting unresolved questions about hallucinations and performance beyond knowledge-intensive tasks.

Impact Statements

  • Effectiveness beyond knowledge-intensive domains remains an open question requiring thorough assessment, and retrieval-augmented LMs may not completely address hallucinations.The paper presents addressing parametric-model limitations as a possibility, not a settled outcome.Source: Impact Statements · p. 9

A. Progress of Parametric LMs

  • Parametric LMs include encoder-only masked models, encoder-decoder models, and decoder-only autoregressive models, with recent systems showing strong performance across tasks.The passage describes BERT, BART and T5, GPT and GPT-2, and later large models.Source: A. Progress of Parametric LMs · p. 19
  • Large parametric LMs can adapt through few-shot in-context learning, while supervised training and RLHF have produced instruction-following models.In-context learning adapts to new tasks without parameter updates.Source: A. Progress of Parametric LMs · p. 19
  • Parametric-LM infrastructure work includes sharding and optimizer techniques for training, FlashAttention for training and long-context processing, and memory-efficient inference algorithms.The passage cites PyTorch FSDP, DeepSpeed, and vLLM using PagedAttention.Source: A. Progress of Parametric LMs · p. 19

B.1. Architectures

The paper organizes retrieval-based LM architectures by what they retrieve, how they incorporate retrieval, and when retrieval occurs.

B.1. Architectures

  • Retrieval granularity describes what information an architecture retrieves.It is one of the three axes in the paper’s taxonomy.Source: §B.1 · p. 19
  • The incorporation method describes how an architecture uses retrieved information.It is one of the three axes in the paper’s taxonomy.Source: §B.1 · p. 19
  • Retrieval frequency describes when an architecture retrieves information.It is one of the three axes in the paper’s taxonomy.Source: §B.1 · p. 19

B.1.1. GRANULARITY OF RETRIEVAL

Retrieval-augmented LMs can retrieve text chunks or smaller units such as tokens and phrases. The choice of granularity involves a trade-off between the information retrieved and index size.

B.1.1. GRANULARITY OF RETRIEVAL

  • Text chunks are a prevalent retrieval unit; retrieved chunks are integrated into the input or intermediate layers, and their length can significantly affect performance.Chunks may contain more information than necessary, resulting in redundancy.Source: §B.1.1 · p. 19
  • Token- or phrase-based retrieval matches the input prompt to similar prefixes in a datastore to search possible next tokens directly.For a datastore of the same size, this approach can require a much larger index because embeddings by default correspond to datastore tokens.Source: §B.1.1 · pp. 19–20

B.1.2. INCORPORATION METHOD

Retrieval-augmented LMs vary in how retrieved information is incorporated and how often retrieval occurs. These choices bring different trade-offs in flexibility, efficiency, and computational overhead.

B.1.2. INCORPORATION METHOD

  • Input augmentation combines retrieved results with the original input, enabling flexible plug-ins but increasing inference costs and facing context-length limits.FiD uses parallel encoding to address inefficiency, but the same documents still need repeated encoding for each input.Source: §B.1.2 · p. 20
  • Intermediate fusion adds chunked cross attention to integrate pre-encoded text chunks in intermediate Transformer layers, but requires architecture modification and pre-training.The chunks are independent of the query, unlike input augmentation.Source: §B.1.2 · p. 20
  • Output interpolation manipulates output token distributions; kNN LM interpolates the original LM’s softmax distribution with retrieved token distributions without additional training.Other approaches add training objectives or replace the distribution with a nonparametric distribution over corpus phrases.Source: §B.1.2 · p. 20

B.1.3. FREQUENCY OF RETRIEVAL

  • More frequent retrieval tends to enhance performance but increases computational overhead.Methods retrieve once before generation, at every token or every k tokens, or decide adaptively when to retrieve.Source: §B.1.3 · p. 20

B.2. Applications and Datastore

Retrieval-augmented LMs have been applied across discriminative and generative tasks, using datastores ranging from task examples and web sources to specialized corpora. Prior work reports significant improvements over parametric LMs.

B.2. Applications and Datastore

  • Most prior evaluations focus on knowledge-intensive tasks such as open-domain QA, fact verification, and knowledge-grounding dialogue, often using Wikipedia or commercial search APIs.For other tasks, datastores include labeled training instances, pre-training corpora, commonsense documents, similar code, and API documentation.Source: §B.2 · p. 20
  • Across the reviewed papers, retrieval-augmented LMs have shown significant improvements over parametric LMs.Source: §B.2 · p. 21
  • Retrieval-augmented LMs have been applied to multilingual and multimodal data, and targeted datastores have helped with expert-domain tasks such as theorem proving and molecule generation.The passage notes robot controls among the underexplored modalities.Source: §B.2 · p. 21
Loading 2403.03187v1…