PaperlayerView exact revision on arXiv

Source-linked AI summary

Functional Interpolation for Relative Positions Improves Long Context Transformers

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli

arXiv:2310.04418v2cs.LG

TL;DR

Transformers can lose performance on inputs longer than those used in training, and position encoding choices can limit long-context generalization. FIRE learns relative-position biases with progressive interpolation, and the paper reports stronger generalization on zero-shot language modeling and long-text benchmarks.

  • Problem

    Transformer accuracy usually drops on inputs longer than training sequences, limiting efforts to extend context length.

    Source: §1 · p. 1

  • Method

    FIRE learns relative-position biases with a functional mapping and progressively interpolates positions into a bounded domain.

    Source: §1 · p. 1, §3.1 · p. 4

  • Results

    FIRE improves length generalization across zero-shot language modeling and long-text benchmarks; on C4 it beats the best baseline by 2.28 perplexity points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FIRE unifies many additive relative-position encodings while learning diverse biases in long-context settings.

    Source: §5 · p. 9

    The study covers decoder models and does not analyze other Transformer components or training choices; existing additive RPEs also have limitations in distance representation or locality bias.

    Source: §2.2 · p. 3, §5 · p. 9

Abstract

from arXiv · show

Preventing the performance decay of Transformers on inputs longer than those used for training has been an important challenge in extending the context length of these models. Though the Transformer architecture has fundamentally no limits on the input sequence lengths it can process, the choice of position encoding used during training can limit the performance of these models on longer inputs. We propose a novel functional relative position encoding with progressive interpolation, FIRE, to improve Transformer generalization to longer contexts. We theoretically prove that this can represent some of the popular relative position encodings, such as T5's RPE, Alibi, and Kerple. We next empirically show that FIRE models have better generalization to longer contexts on both zero-shot language modeling and long text benchmarks.

1 INTRODUCTION

Transformers often lose accuracy on sequences longer than their training inputs, while existing position encodings can limit long-context performance. FIRE learns relative-position biases with progressive interpolation; the paper reports stronger long-context results and theoretical coverage of popular encodings.

1 INTRODUCTION

  • FIRE uses a learnable function for relative-position biases and progressively normalizes distances into a bounded input domain.For causal attention, normalization yields inputs in [0, 1], with longer sequences forming progressively finer grids over that domain.Source: §1 · pp. 1–2
  • At evaluation length 8192 after training at 2048, FIRE reaches 21.24 vs. Kerple’s 23.5 perplexity on C4.The introduction also reports that FIRE surpasses competing methods by over 1 point on average on SCROLLS.Source: Figure 1 · p. 2, §1 · p. 2
  • FIRE can theoretically represent T5’s RPE, Alibi, and Kerple, unifying several popular relative position encodings.The paper describes this representation as parameter-efficient and uses a tiny MLP with hidden size 32 in its experiments.Source: §1 · p. 2
  • FIRE improves length generalization across zero-shot and finetuning settings and is evaluated on language modeling, long-text, question-answering, and NLU benchmarks.The paper also reports that FIRE is competitive on short-sequence NLU tasks.Source: §1 · p. 2
  • FIRE’s learned position biases can include both local and anti-local patterns.Some attention heads learn to attend more to far-away contexts.Source: §1 · p. 2

2 POSITIONAL ENCODINGS AND LENGTH GENERALIZATION

Length generalization requires Transformers to handle positions not seen during training. This section reviews absolute, relative, and absent position encodings and their reported behavior on longer sequences.

2 POSITIONAL ENCODINGS AND LENGTH GENERALIZATION

  • Length generalization means applying a model trained on length Ltrain directly to Ltest > Ltrain without performance degradation.It requires generalization to positions unseen during training.Source: §2 · p. 2

2.1 ABSOLUTE POSITIONAL ENCODING

  • Absolute positional encoding adds a position embedding to each input, but APE-based Transformers usually generalize poorly to longer sequences.The embeddings may be learnable or fixed sinusoidal vectors.Source: §2.1 · p. 3

2.2 RELATIVE POSITIONAL ENCODING

  • Relative positional encoding represents position through query-key relations; additive variants place scalar biases in pre-softmax attention logits.T5’s RPE maps out-of-distribution lengths to the same position, while Alibi, Kerple, and Sandwich offer other additive formulations supporting arbitrary sequence lengths.Source: §2.2 · p. 3
  • T5’s RPE shares one bias across distances beyond K and is relatively slow on accelerators, while Alibi, Kerple, and Sandwich favor local attention.That locality bias can hinder attention to distant pairs and degrade performance on some tasks; T5 also buckets relative distances logarithmically in practice.Source: §2.2 · p. 3
  • RoPE makes attention depend on relative token distance, but RoPE models have shown poor length generalization; position interpolation can improve it.The reviewed interpolation approaches may require knowing the target sequence length and finetuning at that length, unlike FIRE’s progressive interpolation.Source: §2.2 · pp. 3–4

2.3 NO POSITIONAL ENCODING

  • Decoder-only Transformers can learn positional information from causal masks without explicit position encodings, and NoPE has shown strong length generalization on small synthetic tasks..Source: §2.3 · p. 4

3 METHOD

FIRE replaces fixed relative-position biases with a learnable continuous function and progressively interpolates normalized distances; additional transformations address local-position resolution and short-context modeling.

3.1 FUNCTIONAL POSITION ENCODING WITH PROGRESSIVE INTERPOLATION

  • FIRE maps normalized query–key distances through a learnable MLP, keeping causal inputs in [0, 1] so longer contexts refine the grid instead of entering an unseen input range.The progressively finer grids interpolate the position-encoding function over the same bounded domain.Source: §3.1 · p. 4
  • Using query position as the normalizer avoids unfamiliar large distances that can arise when inputs exceed a predefined maximum context length.For autoregressive generation, the query position corresponds to the current context length.Source: §3.1 · pp. 4–5

3.2 ADDITIONAL TRANSFORMATIONS

  • A logarithmic distance transform amplifies differences among local positions while preserving bounded MLP inputs for any sequence length.The experiments use ψ(x) = log(cx + 1), with learnable c > 0, on both the relative distance and the normalizer.Source: §3.2 · p. 5
  • A learnable threshold L applies progressive interpolation only when query position i > L, using ψ(L) to normalize distances in shorter sequences.This addresses preliminary evidence of marginal short-sequence degradation, which the authors attribute to normalization obscuring actual relative distances.Source: §3.2 · p. 5

3.3 EXPRESSIVENESS OF FIRE

  • For any finite L0, FIRE can exactly represent T5 RPE, Alibi, Kerple, and Sandwich; experiments use a 32-hidden-unit MLP for strong performance.The constructive representation uses nearly the same number of parameters as T5 RPE, Alibi, and Kerple, up to a constant factor.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments evaluate FIRE across zero-shot language modeling and long-context tasks, short-context benchmarks, learned-bias visualizations, and a layer-shared variant. Results show stronger long-context performance, with FIRE-S offering a speed–accuracy trade-off.

4 EXPERIMENTS

  • Experiments use C4-pretrained models at sequence length 2,048, evaluate language modeling from 512 to 8,192 tokens, and include base and large sizes.The study also covers SCROLLS fine-tuning at length 8,192, NarrativeQA contexts up to 32,768, and GLUE/SuperGLUE at length 1,024.Source: §4 · p. 6, §4.1 · p. 6, §4.2 · p. 7

4.1 LANGUAGE MODELING WITH LENGTH GENERALIZATION

  • For base models trained at sequence length 2,048 and evaluated at 8,192, FIRE achieves 21.24 perplexity versus Kerple’s 23.52 and consistently lowers perplexity across model sizes, validation lengths, and datasets.Source: §4.1 · p. 6

4.2 FINETUNING ON LONG TEXT BENCHMARK

  • FIRE achieves the best average SCROLLS score by over 1.0 point for both model sizes, with best results on 4/5 of seven tasks.RoPE-PI improves on RoPE but requires the maximum input length to be specified beforehand.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 ZERO-SHOT LENGTH GENERALIZATION ON NARRATIVEQA

  • On NarrativeQA, FIRE generalizes across context lengths, while RoPE-PI’s performance depends on its maximum-length setting.An 8,192 setting drops steeply beyond that length; 32,768 avoids the steep drop but performs worse across all tested lengths.Source: Figure 3 · p. 8, §4.3 · p. 8

4.4 FINETUNING ON GLUE/SUPERGLUE

  • FIRE is on par with RoPE, Kerple, and T5’s RPE on GLUE and SuperGLUE, showing no short-context accuracy sacrifice.The reported benchmark metric is average accuracy across tasks.Source: Figure 3 · p. 8, §4.4 · p. 8

4.5 VISUALIZATION OF FIRE

  • For a query at position 128, FIRE learns both local and anti-local attention patterns across heads and selected layers.Anti-local patterns emphasize more distant keys rather than imposing only a fixed local bias.Source: Figure 4 · p. 8, §4.5 · p. 8

4.6 LAYERWISE SHARING

  • FIRE-S shares the position-encoding function across layers, slightly degrades quality, yet retains strong long-context results.On C4 at length 8,192, FIRE-S records 3.10 log perplexity versus Kerple’s 3.16; its average SCROLLS score also exceeds strong baselines.Source: Table 2 · p. 9, §4.6 · p. 9
  • FIRE and FIRE-S are faster than T5’s RPE; FIRE-S is faster than every baseline except NoPE.Figure 5 reports inference-time comparisons averaged over 10 runs per method.Source: Figure 5 · p. 9, §4.6 · p. 9

5 CONCLUSION

The conclusion presents FIRE as a functional relative-position method for longer-context generalization, supported by theoretical and empirical results. It also identifies decoder-only models and unexamined Transformer and training components as limits of the study.

5 CONCLUSION

  • FIRE combines functional relative-position encoding with interpolation to improve Transformer generalization to longer contexts.The conclusion characterizes the evidence as both theoretical and empirical.Source: §5 · p. 9
  • The paper reports that FIRE unifies existing additive relative-position methods while learning diverse position biases in long-context settings.The conclusion frames this as supporting train-short, test-long use.Source: §5 · p. 9
  • The study covers decoder models only and does not analyze other Transformer components or training choices such as data and optimizer.The conclusion names these as directions for future exploration.Source: §5 · p. 9

A OMITTED PROOF

The appendix introduces a more general formulation of T5’s positional encoding and states that the proof of Theorem 3.1 uses it. It also describes T5’s distance buckets and log-binning formulation.

A OMITTED PROOF

  • The appendix section presents a general formulation of T5’s positional encoding before giving the proof of Theorem 3.1.The proof is based on the general formulation so it applies to specific instances.Source: A OMITTED PROOF · p. 15, §A.1 · p. 15

A.1 T5’S RPE WITH BUCKETING

  • T5’s RPE partitions relative distances into buckets, assigning all distances in a bucket the same learnable attention bias.The formulation sets bucket boundaries over relative distances.Source: §A.1 · p. 15
  • The simplified formulation is a special case of the general one; the official T5 implementation uses log-binned buckets.The appendix states this for a predefined distance L1 and K + 1 buckets.Source: §A.1 · p. 15

A.2 PROOF OF THEOREM 3.1

The proof constructs FIRE configurations that exactly reproduce several target relative positional encodings for positions within the threshold range. The constructions cover T5’s RPE, Alibi, Kerple variants, and Sandwich.

A.2 PROOF OF THEOREM 3.1

  • FIRE can represent T5’s RPE, Alibi, Kerple, and Sandwich for positions within the target range.The proof provides constructions for each variant and establishes equality with the target encoding for 0 < j ≤ i ≤ L0.Source: §A.2 · pp. 15–17
  • For T5’s RPE, the construction uses a thresholded MLP whose step activations encode bucket boundaries and whose output weights encode bucket increments.The construction sets the transform to the identity and the threshold to L0.Source: §A.2 · pp. 15–16
  • For Alibi, a linear MLP with identity transform and threshold L0 reproduces its relative-distance bias within the target range.The construction uses a one-layer MLP with identity activation and no bias.Source: §A.2 · p. 16
  • For Kerple’s power variant, a one-hidden-neuron MLP with power activation reproduces the target bias within the threshold range.The construction uses the identity position transform and sets the threshold to L0.Source: §A.2 · p. 16

B ABLATION STUDY

FIRE’s positional encoding composes a position transformation with a function approximator to turn relative-position inputs into attention biases. The ablations examine both components and length generalization across training sequence lengths.

B ABLATION STUDY

  • FIRE maps relative distance and query position to normalized distances, then maps those distances to attention biases.The position transformation and function approximator together define b(i, j) = fθ(g(i−j, i)).Source: B ABLATION STUDY · p. 17
  • The position transformation’s form depends on the choice of ψ, while the function approximator is an expressive function class parameterized by θ.The default function approximator is a two-hidden-layer MLP with 32 neurons per layer and ReLU activation.Source: B ABLATION STUDY · p. 17
  • The ablations test position transformations, function-approximator choices, and length generalization under different training sequence lengths using base-sized models.The section covers both design choices and training-length settings.Source: B ABLATION STUDY · p. 17

B.1 THE LOG TRANSFORM AND THRESHOLDING IN POSITION TRANSFORMATIONS

Ablations find that log transformation and thresholding improve FIRE’s language-modeling quality, while the standard FIRE variant performs best among the tested variants. Thresholding also improves short-sequence GLUE/SuperGLUE accuracy.

B.1 THE LOG TRANSFORM AND THRESHOLDING IN POSITION TRANSFORMATIONS

  • Both log transformation and thresholding improve language-modeling quality across lengths, and standard FIRE is the best tested variant.Table 3 compares variants with and without the two transformations.Source: §B.1 · p. 17, Table 3 · p. 18
  • 0.72 point gain in average GLUE/SuperGLUE accuracy follows from thresholding in the short-sequence evaluation.The passage presents this as evidence of improved short-sequence modeling.Source: §B.1 · p. 18
  • FIRE without thresholding still outperforms all baselines, including RoPE and T5’s RPE, at every evaluated C4 sequence length.Detailed comparisons are identified as Table 4, which compares baselines with FIRE variants with and without thresholding.Source: §B.1 · p. 18, Table 4 · p. 18
  • A learnable threshold performs better than a fixed threshold in preliminary exploration, while the fixed option adds a hyperparameter that may require tuning.The default FIRE configuration therefore uses a learnable threshold.Source: §B.1 · p. 18

B.2 EFFECTS OF THE FUNCTION APPROXIMATOR CAPACITY ON THE PERFORMANCES

Across language-modeling evaluations, FIRE shows strong length generalization, including with different training lengths and datasets. Ablations indicate that MLP capacity, activation choice, and final-output activation affect performance in distinct ways.

B.2 EFFECTS OF THE FUNCTION APPROXIMATOR CAPACITY ON THE PERFORMANCES

  • A one-hidden-layer MLP outperforms a linear function approximator by 0.24 point log perplexity at sequence length 8192; a two-hidden-layer MLP can further improve performance.Source: §B.2 · p. 19

B.3 CHOICE OF THE MLP ACTIVATION FUNCTION

  • ReLU and GeLU perform similarly in length generalization, while ReLU works better at normal sequence lengths.The paper uses ReLU as its default activation.Source: §B.3 · p. 19, Table 6 · p. 19

B.4 CHOICE OF FINAL ACTIVATION OF MLP OUTPUT

  • Omitting the MLP’s final activation performs better on long sequences than applying a final activation.The comparison uses one-hidden-layer MLPs with 32 hidden neurons and ReLU.Source: §B.4 · p. 19, Table 7 · p. 20

B.5 FIRE IS STILL STRONG WHEN TRAINED ON SEQUENCE LENGTH 512

  • With training sequence length 512, FIRE achieves the strongest overall performance among the compared baselines.The evaluation varies sequence length, and the paper reports the trend as robust across training sequence lengths.Source: §B.5 · p. 20, Figure 6 · p. 20

C.1 LANGUAGE MODELING WITH LENGTH GENERALIZATION

  • FIRE consistently demonstrates strong length generalization on C4, arXiv, and Github across base and large models.Figure 7 shows base-model results on arXiv and Github; the other reported results show similar trends.Source: §C.1 · p. 21, Figure 7 · p. 21

C.2 FINETUNING ON LONG TEXT BENCHMARK

FIRE consistently outperforms baselines across NarrativeQA context lengths and maintains performance on short-sequence and standard tasks. The appendix also visualizes learned biases, describes evaluation configurations, and situates FIRE among related approaches.

C.2 FINETUNING ON LONG TEXT BENCHMARK

  • SCROLLS evaluation covers question answering, natural language inference, and summarization, using dataset-specific metrics and an average score across datasets.The models are C4-pretrained checkpoints, and the same hyperparameters are used across models and tasks.Source: §C.2 · pp. 21–22, Table 10 · p. 22

C.3 ZERO-SHOT LENGTH GENERALIZATION ON NARRATIVEQA

  • FIRE consistently outperforms all baselines across NarrativeQA context lengths from 512 to 32768, measured with unigram-overlap F1.The models were pretrained on C4 at sequence length 2048 and finetuned on NarrativeQA at length 8192.Source: §C.3 · p. 22, Table 11 · p. 22

C.4 FINETUNING ON GLUE/SUPERGLUE

  • FIRE achieves decent performance on GLUE/SuperGLUE without sacrificing model quality on short sequences and standard tasks.The evaluation uses macro-average accuracy/exact match across tasks, with exact match for MultiRC and ReCoRD and accuracy for the remaining tasks.Source: §C.4 · p. 22, Table 13 · p. 23

C.5 VISUALIZATION

  • For a query at position 8192, FIRE learns both local and anti-local position patterns.The learned biases are visualized over key positions 1 through 8192.Source: Figure 8 · p. 24, §C.5 · p. 24

C.6 EFFICIENCY AND FIRE-SHARED

  • FIRE-S is evaluated with a 125M-parameter base model, while inference-time forward passes are timed at sequence length 2048 on four TPUv2 chips over ten runs.The supplied passages describe these configurations and timing procedure but give no efficiency results.Source: §C.6 · p. 24

D RELATED WORKS

  • Unlike Chen et al. (2023)’s RoPE Position Interpolation, FIRE needs no predefined maximum sequence length and can be applied without tuning for length generalization.The cited Position Interpolation approach requires additional finetuning on longer-sequence data.Source: D RELATED WORKS · p. 24
  • FIRE builds on additive relative positional encoding approaches, within a field that also studies positional information, representational power, efficiency, and length generalization.The related-work discussion traces relative positional encoding to Shaw et al. (2018).Source: D RELATED WORKS · p. 25
  • Unlike 2D interpolation of pretrained absolute positional encodings for higher-resolution vision inputs, FIRE interpolates relative-position encoding functions in causal attention with unknown total sequence length.The vision approach is attributed to Dosovitskiy et al. (2021).Source: D RELATED WORKS · p. 25

E IMPLEMENTATION

The FIRE implementation computes relative-position attention biases by normalizing relative distances, applying progressive interpolation, and mapping the result through an MLP.

E IMPLEMENTATION

  • A two-layer MLP maps each scalar input to per-head FIRE attention biases.The module defaults to 12 heads and an MLP width of 32.Source: E IMPLEMENTATION · p. 25
  • The implementation learns a log-transformation parameter and a multiplier for a fixed initial threshold.The threshold is initialized to 512, while the multiplier is trainable.Source: E IMPLEMENTATION · pp. 25–26
  • The forward pass computes relative distances, applies log transformation and threshold-based normalization, then maps normalized distances to attention biases.The returned bias has shape [1, num_heads, seq_len, seq_len].Source: E IMPLEMENTATION · p. 26

Related papers

Loading 2310.04418v2…