PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Long-Range Language Models Actually Use Long-Range Context?

Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, Mohit Iyyer

arXiv:2109.09115v1cs.CL

TL;DR

Most language models are trained on small local windows, while whether long-range Transformers use distant context to encode discourse information remains unclear. The paper analyzes two such models and finds that long-range context has little overall effect, with gains limited to a small token subset and no benefit for sequence-level prediction.

  • Problem

    Most language models are trained on small local windows, and whether long-range Transformers use distant context to encode discourse information remains unclear.

    Source: §1 · p. 1

  • Method

    The paper conducts fine-grained analyses of the Routing Transformer and a simpler baseline under context perturbations, examining effects across token types.

    Source: §1 · p. 1

  • Results

    Beyond 2K tokens, long-range context has negligible aggregate token-level perplexity impact, helps only a small token subset, and does not help sequence-level prediction.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The models rarely exploit long-term context, and when they do, use is mostly superficial, such as copying rare tokens from far away.

    Source: §7 · p. 9

    An 8K-token Routing Transformer example takes around 1.3–1.4 seconds for one forward pass on RTX8000 GPUs.

    Source: Ethical concerns · p. 9

Abstract

from arXiv · show

Language models are generally trained on short, truncated input sequences, which limits their ability to use discourse-level information present in long-range context to improve their predictions. Recent efforts to improve the efficiency of self-attention have led to a proliferation of long-range Transformer language models, which can process much longer sequences than models of the past. However, the ways in which such models take advantage of the long-range context remain unclear. In this paper, we perform a fine-grained analysis of two long-range Transformer language models (including the \emph{Routing Transformer}, which achieves state-of-the-art perplexity on the PG-19 long-sequence LM benchmark dataset) that accept input sequences of up to 8K tokens. Our results reveal that providing long-range context (i.e., beyond the previous 2K tokens) to these models only improves their predictions on a small set of tokens (e.g., those that can be copied from the distant context) and does not help at all for sentence-level prediction tasks. Finally, we discover that PG-19 contains a variety of different document types and domains, and that long-range context helps most for literary novels (as opposed to textbooks or magazines).

1 Introduction

The introduction asks whether long-range Transformers use distant context to capture discourse information. It reports that benefits are narrow despite models’ ability to process longer sequences.

1 Introduction

  • Language models’ small local context windows limit their ability to use discourse-level information, motivating tests of whether longer-range models exploit it.Source: §1 · p. 1
  • The authors compare the Routing Transformer with a simpler baseline using perturbations and token-type analyses, and recommend future analyses of when models use distant context.Source: §1 · pp. 1–2
  • Beyond 2K tokens, long-range context has negligible aggregate perplexity impact, though it helps a small set of tokens and particular types of books.The benefiting tokens include those in subword clusters and those copyable only from distant context; gains are reported for fictional and continuous books.Source: §1 · pp. 1–2
  • Word shuffling and random document replacement barely change overall perplexity, suggesting that the evaluated models encode long-term context superficially at best.Source: §1 · p. 2
  • Long-range context is not used for sequence-level prediction tasks outside the teacher-forced setting.Source: §1 · p. 2

2 Background & Setup

This section introduces Local and Routing Transformers and describes the experimental setup used to assess long-range context. The analyses use pretrained checkpoints and sampled PG-19 validation tokens.

2 Background & Setup

  • Self-attention compares tokens across a sequence, so its compute and memory scale quadratically with sequence length, limiting very long inputs.The section introduces Local and Routing Transformers as long-range language models and describes the experimental setup used to measure long-context effects.Source: §2 · p. 2, §2.1 · p. 2

2.1 Long-range Language Modeling

  • A Local Transformer restricts each layer to the previous k tokens, giving the lth layer access to k × l preceding tokens.This local-window design allows the model to scale to large input lengths.Source: §2.1 · p. 2
  • The Routing Transformer assigns queries and keys to learned clusters and computes attention only within the query’s nearest-centroid cluster.This content-based sparse attention reduces complexity from O(N^2) to O(N^1.5).Source: §2.1 · p. 2

2.2 Experimental setup

  • The analyses use PG-19 validation data, sampling 220K tokens from 50 books with 10 target tokens per context because evaluating every token with long prefixes is infeasible.Tokens sampled from each book are proportional to its length; the annotated edition of The Canterbury Tales and Other Poems is excluded.Source: §2.2 · p. 3
  • The Routing Transformer checkpoint has 490M parameters, processes up to 8,192 tokens, and achieves 33.2 perplexity on the PG-19 test set.The experiments use publicly available pretrained checkpoints because training long-range Transformer LMs requires immense computational resources.Source: §2.2 · p. 3
  • The Local Transformer is derived from the Routing Transformer checkpoint, replacing clustering heads with local heads; its heads attend to 256 previous tokens, yielding an effective receptive field of about 5K tokens.The study also excludes the final 40 tokens from all evaluations because an RT artifact raises their losses.Source: §2.2 · p. 3
  • Compressive Transformer and Longformer are not analyzed because public PG-19 checkpoints are unavailable and their model sizes differ from RT and LT, complicating controlled experiments.Source: §2.2 · p. 3

3 The effect of longer context

Beyond 2K tokens, longer prefixes do not improve aggregate perplexity for the evaluated models, but benefits appear for selected tokens and book types. Routing Transformer gains are strongest for infrequent or copyable tokens, later subwords, and fictional, continuous books.

3 The effect of longer context

  • RT perplexity plateaus beyond 2K tokens, while LT flattens around 1K; their relatively consistent gap suggests much of RT’s advantage comes from local-context modeling.The LT’s effective context size is 5K tokens.Source: §3 · p. 4
  • RT infrequent-token perplexity falls from ∼1200 at 2K tokens to 1180 at 5K, then returns to 1200 with longer prefixes.Long-range context does not improve either model’s frequent-token predictions; LT infrequent-token perplexity worsens as the prefix grows.Source: §3 · p. 4, Figure 2 · p. 4
  • For tokens inside subword clusters, RT gains through 5K tokens occur in later subwords, while first-subword perplexity follows the infrequent-token pattern.The perplexities cover 4.1K first subwords and 5.1K remaining subwords.Source: §3 · pp. 4–5, Figure 3 · p. 4
  • For tokens last seen more than 2K tokens earlier, RT perplexity keeps falling through 8K, while LT plateaus after 2K; tokens absent from the prefix plateau around 5K.The analysis covers 22K tokens last seen over 2K tokens earlier and 36K tokens never seen in the prefix.Source: Figure 4 · p. 4, §3 · p. 5
  • RT also learns to increment a chapter number from a title 2300 tokens earlier; replacing “Chapter IV” with “Chapter V” raises the predicted token’s loss by over 10.Source: §3 · p. 5
  • Fictional and continuous books benefit more from long-range context, and account for most improvements on later subwords.Their perplexity plateaus at around 5K tokens.Source: Figure 6 · p. 5, Figure 5 · p. 5, §3 · p. 6

4 The perturbation of long-range context

The perturbation analysis tests whether long-range context benefits depend on its content and order. Overall perplexity changes little when distant context is perturbed, but selected token groups respond to those changes, and sequence copying works only at shorter distances.

4 The perturbation of long-range context

  • The analysis defines three perturbation operations and averages results over five runs: sequence shuffling, random sequence replacement, and specific-token dropping.Shuffling disregards sentence boundaries; replacement uses a random sequence from another validation book.Source: §4 · p. 6
  • Shuffling or replacing distant context minimally affects overall perplexity when at least 2K tokens remain unperturbed; effects grow nearer the target, especially for infrequent tokens.Random replacement consistently raises perplexity more than shuffling, while shuffling distant context can slightly outperform an unperturbed prefix.Source: §4 · p. 7
  • At 6K-token perturbation, copied-token perplexity is 172 after shuffling and 174 after random replacement; subword-cluster perplexity is 3.8 and 3.7, respectively.Both operations increase perplexity for these token groups, with shuffling more detrimental for subword-cluster tokens.Source: Figure 10 · p. 6, §4 · p. 7, Figure 11 · p. 7
  • Dropping distant occurrences raises perplexity for target tokens found only in long-range context, but not for tokens also seen within 2K tokens.The flat curve before 6K indicates reliance on the most recent occurrences for locally repeated tokens.Source: §4 · p. 7, Figure 12 · p. 7

5 Sequence-level analysis

The sequence-level experiments test whether long-range context supports exact sequence copying and suffix selection beyond token-level perplexity. Neither task benefits from context beyond 2K tokens, although nearby duplicates can help sequence copying.

5 Sequence-level analysis

  • Beyond 2K tokens, added context does not improve suffix identification, and LT and RT perform almost identically and poorly.The figure reports the suffix-identification comparison; the task selects the gold 128-token suffix over five same-book distractors.Source: Figure 13 · p. 8, §5 · p. 8
  • Both models assign low perplexity to a target sequence duplicated within the previous 512 tokens, but cannot exploit it when it occurs more than 2K tokens away.This pattern supports the conclusion that sequence order is generally not encoded in long-range context.Source: §5 · p. 8
  • Overall, distant context helps a subset of tokens superficially but does not help sequence-level prediction tasks.These experiments move beyond token-level perplexity, whose prefixes consist only of ground-truth text, toward sequence-level behavior.Source: §5 · pp. 7–8

6 Related work

The paper situates its analysis among efficient long-context Transformers, methods for incorporating distant context, and studies of long-range context use. It also notes that a newly released LT checkpoint preserves the suffix-identification trend.

6 Related work

  • Longer effective Transformer contexts have been pursued through sparse attention, recurrence, and cached memory, alongside methods incorporating long context and document-level tasks.The cited examples include sparse attention, recurrence, and cached memory [Weston et al., 2015; Rae et al., 2020].Source: §6 · p. 8
  • A newly released LT checkpoint performs slightly worse, but adding context beyond 2K tokens still does not keep improving suffix-identification accuracy.The authors direct readers to Appendix F for further details.Source: §6 · p. 8
  • Prior analyses include smaller LSTM LMs, Transformer-XL, Compressive Transformer, and pretrained masked LMs; the paper extends this line with larger models and finer-grained analysis.The cited work includes Khandelwal et al. (2018), Rae and Razavi (2020), Rae et al. (2020), and Lai et al. (2020); Press et al. (2020) also observed negligible long-term-context benefits.Source: §6 · p. 9

7 Conclusion

The paper analyzes token- and sequence-level effects of long-range context in two Transformer language models on PG-19. It finds that the models rarely use distant context, and mostly in superficial ways such as copying rare tokens.

7 Conclusion

  • The study analyzes long-range context effects on token- and sequence-level improvements in two long-range Transformer LMs using PG-19.The authors hope the analysis will lead to more meaningful progress on integrating discourse information into these models.Source: §7 · p. 9
  • The models rarely take advantage of long-term context, and when they do, use it mostly in superficial ways such as copying rare tokens from far away.Source: §7 · p. 9
  • The conclusion frames more meaningful integration of discourse information as a hoped-for direction for future progress.Source: §7 · p. 9

Ethical concerns

The authors flag misuse risks and evaluation energy costs, while noting that their results exclude generation. They also adjust the target-token range to avoid an RT-specific end-sequence artifact and keep model comparisons fair.

Ethical concerns

  • The reported results include no generation, so they do not evaluate the malicious use of these LMs to produce unfaithful, hallucinated, or biased outputs.Source: Ethical concerns · p. 9
  • Each 8K-token example takes about 1.3–1.4 seconds for one RT forward pass on RTX8000 GPUs, and evaluation energy costs should not be ignored.Source: Ethical concerns · p. 9

A Routing Transformer and end-sequence degradation

  • To avoid RT-specific end-sequence degradation, the analysis uses tokens 50 to 40 from the end, a range unaffected by the issue and comparable across RT and LT.The degradation disappears when RT clustering heads are replaced with local attention.Source: A Routing Transformer and end-sequence… · p. 11

B Effect of longer context

Aggregate perplexity plateaus beyond 2K tokens, but selected token types benefit from longer context, particularly in fictional and continuous books. The reported gains vary by token type and book category.

B Effect of longer context

  • About 8% of infrequent tokens are also tokens found only in distant context, as reported in Table 2.Source: Table 2 · p. 11
  • Infrequent tokens improve through around 5K context in fictional, continuous, and single-authored books; other book types rely more on local context.The author breakdown is not shown in the main text because 10 of the 11 multi-authored books are nonfiction.Source: B Effect of longer context · p. 11, Figure 15 · p. 12
  • Within-subword-cluster tokens improve through around 4K in fictional and continuous books, while nonfictional and discontinuous books show no long-range benefit.Source: B Effect of longer context · p. 11, Figure 16 · p. 12
  • For tokens found only in distant context, perplexity improves through 8K in fictional, continuous, and single-authored books; the single-authored set also includes discontinuous books.Across infrequent, within-cluster, and distant-context-only targets, the reported gains are concentrated in continuous and fictional books.Source: B Effect of longer context · p. 11, Figure 17 · p. 13
  • Perplexity for all target tokens plateaus after 2K prefix tokens for both the Routing Transformer and Local Transformer.The analysis avoids the final 15 tokens because the Routing Transformer’s clustering mechanism raises perplexity there; the main analysis uses tokens 50 to 40 from the sequence end.Source: Figure 14 · p. 12, Figure 18 · p. 13

C Token overlaps

The token categories that benefit from longer context largely overlap only in limited ways. Perturbation results for the Local Transformer also show little sensitivity to changes in the distant prefix.

C Token overlaps

  • Except for infrequent and within-subword-cluster tokens, overlap ratios among the three token categories are below 0.1.The categories are infrequent tokens, tokens within subword clusters, and tokens found only in context more than 2K tokens away.Source: C Token overlaps · p. 13

D Perturbation

  • Local Transformer perplexity hardly changes when its prefix is perturbed out to around 6K tokens, consistent with its limited use of long-range context.The authors present Routing Transformer perturbation results in the main text instead.Source: D Perturbation · p. 13

E Suffix Identification

Suffix identification accuracy depends on suffix length, and an example shows the Routing Transformer assigning lower perplexity to obviously wrong suffixes. The authors interpret this as evidence that it does not properly use distant context for sequence-level prediction.

E Suffix Identification

  • Suffix identification accuracy rises with suffix length, peaks around 10 to 20 tokens, and then falls as the suffix grows longer.The authors suggest the decline occurs because including more local context makes the sequence more probable.Source: E Suffix Identification · p. 13
  • Lower perplexity for obviously wrong suffixes indicates that the Routing Transformer is not properly using distant context for sequence-level prediction.The example evaluates 128-token suffixes after a 1,024-token prefix.Source: E Suffix Identification · p. 13

F Local Transformer checkpoint results

The released Local Transformer checkpoint improves token-level performance relative to the earlier Local Transformer but performs worse on suffix identification. Across the reported analyses, the trend remains that context beyond 2K tokens is not properly used for sequence-level predictions.

F Local Transformer checkpoint results

  • Perturbing context beyond the local 2K tokens does not affect the released Local Transformer, while local random replacement affects both models more than shuffling.The released checkpoint has better overall perplexity than the RT-derived Local Transformer.Source: F Local Transformer checkpoint results · p. 14
  • The released checkpoint improves on sequence-copying but still does not properly encode token order beyond 2K; it is slightly worse on suffix identification than the earlier released model.The RT-derived Local Transformer has almost identical suffix-identification performance to the Routing Transformer.Source: F Local Transformer checkpoint results · p. 14, Figure 22 · p. 14
  • The released Local Transformer performs better on token-level tasks but worse on suffix identification than the earlier checkpoint, leaving the paper’s conclusions unchanged.Source: F Local Transformer checkpoint results · p. 14
Loading 2109.09115v1…