PaperlayerView exact revision on arXiv

Source-linked AI summary

Lost in the Middle: How Language Models Use Long Contexts

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang

arXiv:2307.03172v3cs.CL

TL;DR

Long-context language models can accept more input, but it is unclear how reliably they use information across that context. The paper evaluates multi-document question answering and key-value retrieval, finding that performance is often best at context beginnings and ends and degrades in the middle.

  • Problem

    It is unclear how reliably language models use relevant information across long input contexts, despite their ability to accept them.

  • Method

    The paper evaluates positional information use through multi-document question answering and synthetic key-value retrieval tasks.

  • Results

    Performance is often highest when relevant information appears at the beginning or end of context and substantially degrades when it appears in the middle.

  • Takeaways & Limitations

    Longer contexts do not ensure effective use of all included information, and extended-context models are not necessarily better at using their input context.

  • Takeaways & Limitations

    GPT-4 was evaluated only on a subset of multi-document question-answering experiments because full evaluation would cost upwards of $6000.

Abstract

from arXiv · show

While recent language models have the ability to take long contexts as input, relatively little is known about how well they use longer context. We analyze the performance of language models on two tasks that require identifying relevant information in their input contexts: multi-document question answering and key-value retrieval. We find that performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts. In particular, we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context language models.

1 Introduction

The paper examines how language models use long input contexts through controlled multi-document question-answering and key-value retrieval experiments. It finds that performance depends strongly on where relevant information appears, motivating evaluation protocols that test positional robustness.

  • Transformer memory and compute scale quadratically with sequence length, while newer models support context windows from 4096 to 100K tokens.These expanded context windows make it unclear how models use their inputs during downstream tasks.
  • The experiments vary context length and relevant-information position across open and closed language models, using multi-document question answering and synthetic key-value retrieval.Multi-document QA tests reasoning over retrieved documents, while key-value retrieval isolates matching-token retrieval.
  • Changing relevant-information position substantially affects performance, producing a U-shaped curve with strongest results at the beginning or end and significant degradation in the middle.In one example, GPT-3.5-Turbo performs below its closed-book baseline of 56.1% when relevant information is placed in the middle.
  • Encoder-decoder models resist positional changes within training-time sequence lengths but show U-shaped degradation beyond them, while query-aware contextualization helps key-value retrieval more than multi-document QA.Query-aware contextualization enables near-perfect synthetic key-value performance but minimally changes multi-document QA trends.
  • The results frame longer contexts as a trade-off between providing more information and increasing the content models must reason over, and propose testing minimal positional performance differences.The paper argues that robust long-context use requires performance to be minimally affected by relevant-information position.

2 Multi-Document Question Answering

The study evaluates multi-document question answering by varying context length and answer-document position, finding that models use relevant information best at context boundaries and struggle in the middle. Extended-context models are not necessarily better when contexts fit both model variants.

  • Task and setup: The experiments independently vary answer-document position and context length by reordering documents and changing the number of distractors.Accuracy is the primary metric, based on whether any correct NaturalQuestions answer appears in the output.
  • Task and setup: The task pairs a question with k documents, exactly one containing the answer and k −1 distractors, requiring the model to locate and use the relevant document.Experiments use 2655 NaturalQuestions-Open queries and Wikipedia passages of at most 100 tokens.
  • Position effects: Performance is highest when relevant information appears at the beginning or end of the context and substantially decreases when it occurs in the middle.This produces a distinctive U-shaped curve associated with primacy and recency biases.
  • Position effects: More than 20%: GPT-3.5-Turbo’s multi-document QA performance can drop by more than 20% in the worst case across 20- and 30-document settings.In those settings, performance is lower than without any input documents, i.e., closed-book performance.
  • Extended-context models: When contexts fit both model variants, GPT-3.5-Turbo and GPT-3.5-Turbo (16K) have nearly identical performance as answer position changes.The 10- and 20-document settings fit within both models’ context windows, indicating extended-context models are not necessarily better at using context.

3 How Well Can Language Models Retrieve From Input Contexts?

The paper tests basic context retrieval with a synthetic key-value task that minimizes natural-language confounds. Performance is generally strongest at context boundaries and degrades in the middle, especially for several models and longer contexts.

  • The task asks models to return the value associated with a specified key in a JSON object containing unique random UUID key-value pairs and distractors.Each input contains one relevant pair and k −1 irrelevant pairs.
  • Random UUIDs minimize natural-language semantics and potential linguistic-feature confounds in this retrieval testbed.
  • 75, 140, and 300 key-value pairs define the evaluated context lengths, with 500 examples per length.
  • Claude-1.3 and Claude-1.3 (100K) perform nearly perfectly, while other models struggle especially at 140 or 300 key-value pairs.
  • GPT-3.5-Turbo, GPT-3.5-Turbo (16K), and MPT-30B-Instruct perform worst when retrieving key-value pairs from the middle of context.LongChat-13B (16K) shows a different trend at 140 pairs and sometimes generates retrieval code instead of directly outputting the value.

4 Why Are Language Models Not Robust to Changes in the Position of Relevant Information?

The paper investigates whether architecture, query-aware contextualization, and instruction fine-tuning explain positional brittleness in long contexts. Encoder-decoder models are more robust within training-length contexts, query-aware contextualization greatly improves key-value retrieval but not QA trends, and U-shaped behavior persists across instruction-tuning comparisons and sufficiently large models.

  • Model architecture: Within its 2048-token training-time context, Flan-UL2 shows only a 1.9% absolute best-to-worst performance difference as relevant information moves.Beyond 2048 tokens, encoder-decoder performance develops a U-shaped positional pattern.
  • Query-aware contextualization: Query-aware contextualization yields near-perfect key-value retrieval across 75, 140, and 300 pairs, versus 45.6% worst-case performance without it.GPT-3.5-Turbo (16K) achieves perfect performance with 300 key-value pairs under query-aware contextualization.
  • Query-aware contextualization: In multi-document QA, query-aware contextualization slightly improves performance at the beginning but slightly decreases it elsewhere, without substantially improving positional robustness.This pattern is summarized in Figure 9.
  • Instruction fine-tuning: MPT-30B and MPT-30B-Instruct both show U-shaped performance, while instruction fine-tuning raises absolute performance and slightly reduces worst-case disparity.The supplied passage states that the disparity reduction is from nearly 10%, but the remainder of that value is truncated.
  • Model scale: The U-shaped curve appears only in sufficiently large Llama-2 models: 7B models are solely recency biased, whereas 13B and 70B models exhibit the U-shaped pattern.This holds with or without additional supervised fine-tuning and reinforcement learning from human feedback.

5 Is More Context Is Always Better? A Case Study With Open-Domain QA

In open-domain QA, adding context creates a task-dependent trade-off: more retrieved information may help, but the extra content can reduce effective reasoning and increase latency and cost. Reader performance saturates well before retriever recall, showing that models struggle to use additional retrieved documents.

  • Implication: Longer input contexts are therefore a trade-off, because more information may help downstream performance while increasing the content the model must reason over.Whether a 16K-token context is beneficial ultimately depends on the downstream task.
  • Method: The study evaluates a standard retriever-reader pipeline on NaturalQuestions-Open, measuring retriever recall and reader accuracy as retrieved-document count increases.Contriever, fine-tuned on MS-MARCO, retrieves Wikipedia documents that are inserted into the language-model prompt.
  • Results: Reader performance saturates long before retriever performance, indicating that models do not effectively use the extra retrieved context.Figure 11 summarizes retriever recall and model performance as a function of retrieved-document count.
  • Results: ∼1.5% for GPT-3.5-Turbo and ∼1% for Claude-1.3: retrieving more than 20 documents only marginally improves reader performance.The additional documents significantly increase input-context length, latency, and cost.

6 Related Work

Prior work has pursued cheaper long-context processing through modified or alternative architectures, while this work highlights precise knowledge access as a distinct challenge beyond perplexity evaluation. Earlier studies also found that language models often use longer context coarsely or favor recent information, connecting this work’s U-shaped pattern to the human serial-position effect.

  • Efficient long-context architectures: Transformer variants reduce long-context cost through recurrence, factorized or low-rank attention, while faster exact attention uses an IO-aware CUDA kernel.These approaches target cheaper scaling with context length.
  • Efficient long-context architectures: Other models remove attention’s quadratic sequence-length complexity using convolution and/or linear RNNs, including RWKV, S4, and Hyena.These architectures avoid attention entirely rather than modifying its computation.
  • Long-context evaluation: Perplexity on diverse web corpora is often used as a proxy for long-context ability, but this work identifies precise knowledge access as an additional challenge.The distinction concerns retrieving specific information from long contexts rather than only modeling corpus statistics.
  • Historical context use: Earlier studies found that LSTMs and dialogue models make increasingly coarse use of longer-term context, while attentive LSTMs mainly use recent history.Khandelwal et al. (2018), Sankar et al. (2019), and Daniluk et al. (2017) report these context-use patterns.
  • Serial-position effect: The observed U-shaped curve resembles psychology’s serial-position effect, in which humans best remember the first and last elements of a list.The analogy is notable because Transformer self-attention is technically equally capable of retrieving any context token.

7 Conclusion

The section concludes that language models struggle to robustly access relevant information in long contexts, especially when it appears in the middle. It also examines factors affecting context use and finds that reader performance saturates before retriever recall in open-domain question answering.

  • 7 Conclusion: Controlled experiments show that performance degrades significantly when the position of relevant information changes, often reaching its lowest level when information is in the middle of long contexts.These experiments empirically study how language models use long input contexts.
  • 7 Conclusion: A preliminary investigation examines how model architecture, query-aware contextualization, and instruction fine-tuning affect language models’ use of context.These factors are investigated to better understand context use.
  • 7 Conclusion: In an open-domain question-answering case study, language-model reader performance saturates far before retriever recall.The case study provides a practical analysis of the relationship between readers and retrievers.

A Ambiguity in Multi-Document QA Distractor Documents · B Random Distractors in Multi-Document QA

The QA experiments address temporal ambiguity in the retrieval corpus and test whether conclusions persist with unambiguous questions and random distractors. Results remain consistent across these checks: models still struggle to reason over long contexts despite higher accuracy with random distractors.

  • A Ambiguity in Multi-Document QA Distractor Documents: The retrieval corpus uses a late-2018 Wikipedia dump, which has a small temporal mismatch with NaturalQuestions annotations.
  • A Ambiguity in Multi-Document QA Distractor Documents: For example, NaturalQuestions lists Robert Griffin III as “currently a free agent,” while the corpus says he plays for the Baltimore Ravens.
  • A Ambiguity in Multi-Document QA Distractor Documents: The authors use Min et al.’s ambiguity annotations to construct a subset of unambiguous questions.
  • A Ambiguity in Multi-Document QA Distractor Documents: Experiments on the unambiguous subset produce results and conclusions similar to those on the full question collection.
  • B Random Distractors in Multi-Document QA: The study also evaluates multi-document question answering with random Wikipedia documents serving as distractors.
  • B Random Distractors in Multi-Document QA: Random distractors enable ablation of retrieved distractors, or hard negatives, from the experiment.
  • B Random Distractors in Multi-Document QA: In this setting, simple heuristics such as lexical overlap with the query can often identify the document containing the answer.
  • B Random Distractors in Multi-Document QA: Although all models achieve higher absolute accuracy with random distractors, they still struggle to reason over their entire input context.

C Randomizing Distractor Order in Multi-Document QA

Randomizing distractor order and explicitly stating that search results are randomly ordered does not eliminate the U-shaped performance pattern: performance still degrades when relevant information appears in the middle of long contexts.

  • Method: The experiment instructs models to use search results whose order is random and randomly shuffles the k −1 distractor documents.This tests whether the observed positional effects arise from a learned bias that earlier search results are more relevant.
  • Results: Randomizing distractor order and mentioning this in the prompt slightly decreases performance when relevant information is at the beginning and slightly increases it when relevant information is later.The supplied passage reports these directional changes relative to the results in §2.3, but does not provide numerical magnitudes.
  • Results: The results continue to show a U-shaped performance curve, with degradation when models must use information in the middle of their input contexts.Thus, the positional weakness persists even after distractor ordering is randomized.

D GPT-4 Performance

On 500 multi-document question-answering examples with 20 documents per context, GPT-4 achieves the highest absolute performance among evaluated language models but still struggles with information in the middle of long contexts. Its performance follows a U-shaped pattern, peaking when relevant information appears at the beginning or end of the context.

  • D GPT-4 Performance: GPT-4 achieves higher absolute performance than any other evaluated language model, yet its performance forms a U-shaped curve across information positions.The evaluation uses 500 random multi-document QA examples, each containing 20 documents.
  • D GPT-4 Performance: Performance is highest when relevant information appears at the very start or end of the context and degrades when GPT-4 must use information in the middle.This positional degradation occurs despite GPT-4’s stronger absolute performance than the other evaluated models.

E Llama-2 Performance · F Token Counts

The paper evaluates Llama-2 on 20-document multi-document QA across model sizes and training variants, finding size-dependent positional biases and substantial benefits from additional fine-tuning. It also reports token-count statistics across the evaluated QA and key-value retrieval settings, accounting for tokenizer differences and context-length limits.

  • E Llama-2 Performance: Llama-2 is evaluated on multi-document QA inputs containing 20 total documents.Because its tokenizer produces longer sequences, 20 of 2655 examples exceeding the 4096-token context limit are discarded.
  • E Llama-2 Performance: The experiments compare 7B, 13B, and 70B Llama-2 models with and without supervised fine-tuning and reinforcement learning from human feedback.The fine-tuned variants are designated “-chat-” models.
  • E Llama-2 Performance: Only the 13B and 70B models exhibit U-shaped performance curves reflecting both primacy and recency bias, whereas 7B models are solely recency-biased.The authors hypothesize that earlier studies missed primacy bias because they used models smaller than 1B parameters.
  • E Llama-2 Performance: Additional supervised fine-tuning and reinforcement learning from human feedback dramatically improve Llama-2 performance on multi-document QA.The 13B base model shows a 20-point accuracy disparity between its best- and worst-case performance.
  • F Token Counts: The token-count analysis reports average and maximum input-context lengths for all experimental settings.These statistics are presented in Tables 2–4 for closed-book, oracle, document QA, and key-value retrieval settings.
  • F Token Counts: MPT-30B and MPT-30B-Instruct share a tokenizer, as do GPT-3.5-Turbo and GPT-3.5-Turbo (16K), while Claude-1.3 and Claude-1.3 (100K) also share one.The Claude-1.3 tokenizer matches GPT-3.5-Turbo’s apart from additional special tokens absent from the data.

G Full Multi-Document Question Answering Results … G.3 30 Total Retrieved Documents

The section reports multi-document question-answering performance across contexts containing 10, 20, and 30 retrieved documents. It defines answer-document position using an index whose value is one less than the document’s position in the input context.

  • G Full Multi-Document Question Answering Results: “Index n” denotes performance when the answer-containing document is at position n + 1.Lower indices place the answer document closer to the beginning of the input context.
  • G Full Multi-Document Question Answering Results: Index 0 represents the condition where the answer-containing document is first among all documents.This places the relevant document at the very start of the context.
  • G Full Multi-Document Question Answering Results: The evaluation varies the number of retrieved documents in the multi-document QA task.The reported settings include 10, 20, and 30 total retrieved documents.
  • G.1 10 Total Retrieved Documents: Table 5 reports model performance for 10 total retrieved documents.The table is part of the multi-document QA evaluation across different context lengths.
  • G.3 30 Total Retrieved Documents: Table 7 reports model performance for 30 total retrieved documents.This is the largest retrieved-document setting listed in the section.
Loading 2307.03172v3…