PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis

Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky, Peter J. Ramadge

arXiv:2212.10356v2cs.CL

TL;DR

Transformer length extrapolation asks whether models trained on short sequences can preserve perplexity on much longer ones, yet the reason ALiBi works remains poorly understood. The paper analyzes receptive fields with a cumulative normalized gradient tool and uses that analysis to develop Sandwich. Sandwich uses information beyond training length and performs competitively across the reported language-modeling evaluations, though current relative positional embeddings retain a bias toward recent tokens.

  • Problem

    Transformer length extrapolation seeks to preserve or improve perplexity on longer evaluation sequences, while the reason for ALiBi’s empirical success remains insufficiently understood.

    Source: §1 · p. 1

  • Method

    The paper analyzes ALiBi and other positional designs through receptive fields using a cumulative normalized gradient tool, then modifies Sinusoidal positional embeddings to create Sandwich.

    Source: §1 · p. 2, §5.1 · p. 5, §7 · p. 9

  • Results

    Sandwich is consistently better than all baselines on ArXiv and GitHub, and marginally worse than ALiBi on OpenWebText2 when Lex ≥4096.

    Source: §6.2 · p. 7

  • Takeaways & Limitations

    Sandwich is a parameter-free relative positional embedding that uses information beyond Ltr = 512, with distant tokens contributing substantially despite about 60% of cumulative gradient falling on recent tokens.

    Source: §6.2 · pp. 7–8

    Although Sandwich, KERPLE, and T5 use information beyond training length, their receptive fields still favor recent tokens, and current RPEs struggle on parity prediction where every bit matters equally.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Length extrapolation permits training a transformer language model on short sequences that preserves perplexities when tested on substantially longer sequences. A relative positional embedding design, ALiBi, has had the widest usage to date. We dissect ALiBi via the lens of receptive field analysis empowered by a novel cumulative normalized gradient tool. The concept of receptive field further allows us to modify the vanilla Sinusoidal positional embedding to create ~\textbf{Sandwich}, the first parameter-free relative positional embedding design that truly length information uses longer than the training sequence. Sandwich shares with KERPLE and T5 the same logarithmic decaying temporal bias pattern with learnable relative positional embeddings; these elucidate future extrapolatable positional embedding design.

1 Introduction

Length extrapolation asks whether transformer language models can preserve perplexity when tested on sequences much longer than those used in training, where increasing training length incurs quadratic cost. The paper analyzes ALiBi through receptive fields and introduces Sandwich, a parameter-free relative positional embedding designed to use information beyond the training length.

Motivation

  • The quadratic training cost of longer sequences motivates asking whether transformers preserve or improve perplexity when evaluated at lengths substantially beyond training.This gap between training and evaluation lengths defines length extrapolation.Source: §1 · p. 1
  • Despite ALiBi’s empirical success and adoption in Bloom, the paper identifies a lack of fundamental understanding of why it works.Prior work hypothesized that relative positional embeddings are critical to extrapolation.Source: §1 · p. 1

Receptive-field analysis

  • A model-agnostic cumulative normalized gradient tool measures empirical receptive fields, showing that training length covers most of ALiBi’s empirical receptive field.Windowed attention has theoretical receptive field wR for an R-layer model with window size w, but models can still extrapolate when training length is shorter than wR, motivating empirical measurement.Source: §1 · p. 2

Sandwich

  • Dropping Sinusoidal’s intermediate terms yields Sandwich, a parameter-free relative positional embedding that uses information beyond training length and shares a temporal-bias pattern with KERPLE and T5.The analysis also relates Sinusoidal and Rotary extrapolation failures to violation of the empirical receptive-field coverage principle.Source: §1 · p. 2

2 Related Work

The related work frames length extrapolation as a transformer challenge, reviews positional-embedding approaches and sparse attention, and motivates measuring receptive-field coverage empirically.

2.1 Length Extrapolation

  • Only transformer language models with special relative positional embeddings are described as length extrapolatable, while recurrent neural networks can easily perform the task.The desired language-model behavior is maintaining or improving token-level perplexity when evaluation sequences are much longer than training sequences.Source: §2.1 · p. 2

2.2 Positional Embeddings

  • Absolute positional embeddings attach positions to tokens, whereas relative positional embeddings add temporal-bias terms to attention to encode pairwise distance.The passage notes that absolute embeddings can struggle beyond training length because positions may be unavailable or unseen, while relative embeddings are easier to extend.Source: §2.2 · p. 2

2.3 Windowed and Sparse Attention

  • Viewing ALiBi as imposing a windowed-attention mask makes a model’s theoretical receptive field easier to calculate.The passage relates this view to prior sparse-attention transformer designs.Source: §2.3 · p. 2

2.4 Receptive Field

  • Receptive field denotes the input region contributing most to model outputs, and its empirical coverage is presented as crucial to transformer length extrapolation.The paper borrows the concept from receptive-field analysis in convolutional models.Source: §2.4 · p. 2

3 Background and Notations

This section establishes transformer notation and describes ALiBi, windowed attention, and an evaluation protocol that varies available context while holding evaluated tokens fixed.

3.1 Transformer Language Model

  • An R-layer transformer with H attention heads forms head-specific attention outputs, concatenates them, applies an output transform, and uses layer normalization before the next layer.The experiments use R = 12, H = 12, d = 768, and Ltr = 512.Source: §3.1 · p. 3

3.2 ALiBi

  • ALiBi modifies the attention logits with a positional bias whose coefficient is called the slope.The figure caption describes the head-index range, but the supplied passages do not give a complete formula.Source: §3.2 · p. 3

3.3 Windowed Attention

  • Windowed attention limits the context using window size w, with its formulation introduced for that setting.The supplied passage gives the condition for the windowed-attention design but not a complete equation.Source: §3.3 · p. 3

3.4 Evaluation of Length Extrapolation

  • The evaluation varies how many preceding tokens are available while calculating perplexity for the same evaluated tokens.The protocol uses evaluation sequences longer than training sequences and varies prior context for each segment’s last token.Source: §3.4 · p. 3, Figure 3 · p. 3

4 ALiBi and Windowed Attention

The experiments vary ALiBi slopes and window size to examine length extrapolation. ALiBi perplexities increase beyond the training length, while windowed attention extrapolates up to w = 100.

4 ALiBi and Windowed Attention

  • Three experiments across ArXiv and OpenWebText2 assess whether the observations hold across text domains.The results are shown in Tables 1 and 4.Source: §4 · p. 3
  • The nonoverlapping-subsequence evaluation protocol used in ALiBi’s main experiment tables is less suitable for length-extrapolation evaluation because of the “early token” curse.The authors refer readers to Appendix B of ALiBi (Press et al., 2022) for details.Source: §4 · p. 3

4.1 Slope Shift (Shift all h by ∆)

  • Shifting all slopes by a fixed ∆ causes failure to extrapolate beyond a certain quantity, suggesting slope magnitude matters more than slope diversity.The tested slope diversity means each attention head has one slope.Source: §4.1 · p. 4

4.2 Slope Equalization (Same h for all heads)

  • Only large slopes, equivalently small h, allow good extrapolation; large slopes impose a narrow attention bias that limits distant-token interaction.The study equalizes slopes across heads and steadily increases their value from 0 to 8.Source: §4.2 · p. 4

4.3 Windowed Attention (Size w)

  • Windowed attention underperforms at small w and diverges on long Lex at large w, matching the trend when h is too small or too large.The implicit window effect is made explicit using windowed attention; the text notes this formulation is also adopted by Longformer (Beltagy et al., 2020).Source: §4.3 · p. 4

4.4 Other Observations

  • Beyond Ltr, ALiBi perplexities increase, whereas windowed attention models extrapolate up to w = 100.Once Lex passes a threshold, perplexity either stays the same or explodes, which the authors interpret as ignoring additional tokens or not using them properly.Source: §4.4 · p. 4

5 Receptive Field Measurement

Receptive-field analysis distinguishes theoretical capacity from the tokens a model actually uses, and links extrapolation behavior to empirical receptive-field coverage. The analysis also motivates extending training length for failed settings and identifies receptive-field issues in Sinusoidal and Rotary embeddings.

5 Receptive Field Measurement

  • An R-layer model has theoretical receptive field wR, yet with R = 12 and Ltr = 512, windowed attention extrapolates at w = 100, whose theoretical receptive field is 1200 tokens.This exceeds Ltr, motivating measurement of the empirical receptive field, which can be smaller than the theoretical one.Source: §5 · p. 4

5.1 Quantifying Empirical Receptive Field

  • The empirical receptive field is defined as the smallest token count m for which cumulative normalized gradient c_m exceeds 0.99.The analysis derives c_m by accumulating normalized gradients for input tokens with respect to the next-token prediction.Source: §5.1 · pp. 4–5
  • For window sizes 40, 80, and 100, the most recent 512 tokens contain over 99% of normalized gradient; larger windows have empirical receptive fields of 768, 1024, and 1536 tokens and extrapolate poorly.The reported 512-token training length does not fully cover those larger empirical receptive fields.Source: §5.1 · p. 5, Figure 4 · p. 5
  • For configurations that do not reach the 0.99 cumulative-gradient threshold within Ltr, ∆ = −3 has ERF = 768 and plateaus earliest, while other models need ERF = 2048 to stabilize; ∆ values 6 and 8 have exploded perplexities.The authors relate the mismatch between Ltr and ERF to these perplexity patterns.Source: §5.1 · p. 5, Figure 5 · p. 5

5.2 Fixing Failed Cases

  • With longer training sequences, windowed attention at w = 160 (Ltr = 1024) and shifted ALiBi at ∆ = 6 (Ltr = 2048) maintain stable perplexities.These settings increase Ltr to cover their empirical receptive fields.Source: §5.2 · p. 5, Table 2 · p. 5

5.3 Analyses of Sinusoidal and Rotary

  • Sinusoidal and Rotary do not ensure a receptive field bounded by Ltr; Sinusoidal’s gradient spikes at the Ltr-th position, consistent with overfitting to the positional embeddings seen during training.The authors connect this observation to why relative positional embeddings cannot overfit on positional embeddings.Source: §5.3 · p. 6, Figure 6 · p. 6, Figure 7 · p. 6

6 A New RPE for Length Extrapolation

The section introduces Sandwich by reshaping Sinusoidal positional information into a distance-decaying bias, then evaluates its length-extrapolation performance and relates its bias pattern to KERPLE and T5.

6.1 Introduction to Sandwich

  • Sandwich drops Sinusoidal’s cross terms and retains positional inner products, transforming them into temporal biases with a decay-with-distance pattern like ALiBi.The bias is added after scaled softmax; d̄ controls its shape, and the authors use d̄ = 128 after finding smaller and larger values performed worse.Source: §6.1 · pp. 6–7

6.2 Experiments and Discussion

  • About 60% of Sandwich’s cumulative gradient falls on the most recent 512 tokens, but more distant tokens still contribute to predictions.Its receptive-field pattern differs from ALiBi and windowed attention.Source: §6.2 · pp. 7–8
  • On ArXiv and GitHub, Sandwich consistently beats all baselines at longer evaluation lengths, while on OpenWebText2 it is marginally worse than ALiBi when Lex ≥4096.At Lex = Ltr = 512, it is comparable to other models except that Rotary performs a bit better on OpenWebText2.Source: §6.2 · p. 7, Table 3 · p. 8
  • A fitted logarithmic bias plateaus at larger distances, giving distant tokens similar temporal biases that may support averaging and denoising.The authors relate this pattern to perplexities rising after some evaluation lengths, as averaging over too many historical tokens may reintroduce noise.Source: §6.2 · p. 8

6.3 Connection to KERPLE and T5

  • Sandwich’s fitted logarithmic bias is a special case of KERPLE, while T5’s log-binning treats distant tokens similarly and shares a similar empirical receptive-field pattern.The comparison identifies a common pattern across Sandwich, KERPLE, and T5.Source: §6.3 · pp. 8–9, Figure 11 · p. 9

7 Conclusion

The conclusion connects ALiBi with windowed attention and introduces cumulative normalized gradients for measuring empirical receptive fields. It presents Sandwich as a way to use information beyond the training length and identifies its log-decaying bias pattern as shared with KERPLE and T5.

7 Conclusion

  • The paper connects ALiBi to windowed attention through their constructions and language-modeling performance.It develops a cumulative normalized gradient tool to measure empirical receptive fields.Source: §7 · p. 9
  • The analysis finds that ALiBi and windowed attention can extrapolate when training length covers empirical receptive fields, but they do not use information beyond training length.Sandwich, simplified from Sinusoidal, overcomes that limitation.Source: §7 · p. 9
  • Sandwich’s log-decaying temporal bias resembles patterns in KERPLE and T5, which the paper identifies as a clue for future extrapolatable positional-embedding design.Source: §7 · p. 9

Limitations

The positional embeddings discussed still favor recent tokens, a bias that may suit human-written text but is problematic when inputs matter equally, as in parity prediction.

Limitations

  • Sandwich, KERPLE, and T5 use information beyond training length, yet their receptive fields still strongly favor recent tokens.The authors describe this recency bias as beneficial for modeling human-written text but problematic in other scenarios.Source: Limitations · p. 9
  • Transformer language models with current relative positional embeddings still struggle with parity prediction, where every bit is equally important.The paper identifies addressing this problem as a direction for future work.Source: Limitations · p. 9

Ethics Statement

The ethics statement notes potential societal harms from inappropriate use of the technique and says the cited ethical challenges are common to NLP research.

Ethics Statement

  • Inappropriate use could pose text-generation and data-privacy challenges, which the authors say are not unique to this work.Source: Ethics Statement · p. 9

A Results on OpenWebText2

On OpenWebText2, the paper reports that ALiBi perplexities remain constant for evaluation lengths of at least 2048 and that ALiBi and Sandwich support efficient inference with a cache window of 2048. It also describes the experiments, implementation resources, and configurations used for these results.

A Results on OpenWebText2

  • The three OpenWebText2 experiments from §4 use default hyperparameters without tuning, and their receptive-field plots appear in Figures 12 and 13.Table 4 lists the experiments; the plots examine cumulative normalized gradients for the last token and a shifted h setting.Source: A Results on OpenWebText2 · p. 12, Figure 12 · p. 13, Figure 13 · p. 13

B Efficient Inference

  • ALiBi perplexities stay constant when Lex ≥2048, and ALiBi and Sandwich permit efficient inference with a cache window of 2048.Sandwich performs a bit worse at Lex = 4096 than in Table 3, which the authors describe as further evidence that it uses token information beyond Ltr.Source: B Efficient Inference · p. 12, Table 5 · p. 13
  • With cache window w̄ = 2048, ALiBi inference has complexity O(w̄ × Lex), compared with O(Lex^2) without a window and O(w̄^2 × Lex) for APE.For each generated token after the first w̄, the process discards the oldest cached q, k, and v vectors and attends over the remaining window plus the new token.Source: B Efficient Inference · p. 12

C Scientific Artifacts

  • The code uses gpt-neox under Apache-2.0 and author-released datasets; code and datasets are publicly released for research purposes.The passage points to Gao et al. (2020) sections 5–7 for dataset statistics, distribution, privacy, and anonymization.Source: C Scientific Artifacts · p. 12

D Implementation Details

E Python Implementation of Sandwich

The Python implementation constructs sinusoidal positional embeddings, computes their pairwise dot products, and scales the resulting bias across attention heads.

E Python Implementation of Sandwich

  • The implementation sets base = 1e4, heads = 12, seq_len = 8192, and bar_d = 128 as the Sandwich hyperparameter.Positions are generated from 0 to seq_len − 1, and i ranges over half the embedding dimension.Source: import numpy as np · p. 14
  • Each position is encoded by concatenating sine and cosine values at frequencies determined by base^(2i/bar_d).The sine and cosine terms use the position divided by the corresponding frequency scale.Source: import numpy as np · p. 14
  • The code computes a pairwise positional dot-product matrix, then creates a multi-head representation by dividing it by ratios 8/heads, 16/heads, …, 8, where heads is the configured count.Source: import numpy as np · p. 14

Related papers

Loading 2212.10356v2…