PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov

arXiv:1901.02860v3cs.LGcs.CLstat.ML

TL;DR

Fixed-length contexts limit language models’ ability to capture long-term dependencies. Transformer-XL combines segment-level recurrence with relative positional encoding, achieving strong results across language-modeling benchmarks and generating coherent articles thousands of tokens long.

  • Problem

    Fixed-length contexts cap dependency length and prevent language models from fully exploiting self-attention for long-term dependency modeling.

  • Method

    Transformer-XL combines segment-level recurrence with state reuse and relative positional encodings to extend context across segments.

  • Results

    18.3 perplexity on WikiText-103 improves the previous state of the art from 20.5, while Transformer-XL also performs strongly across four other datasets.

  • Takeaways & Limitations

    Transformer-XL models longer-term dependencies than RNNs and Transformers, speeds evaluation substantially, and generates coherent text articles with thousands of tokens.

Abstract

from arXiv · show

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.

1 Introduction

Transformer-XL addresses the difficulty of modeling long-term dependencies in language modeling by combining recurrence in a self-attention architecture with a novel positional encoding scheme. It achieves strong results across character- and word-level benchmarks and generates coherent articles with thousands of tokens.

  • Motivation: RNNs remain difficult to optimize because of gradient vanishing and explosion, and gating plus gradient clipping may not fully resolve the issue.RNNs, particularly LSTMs, are described as a standard language-modeling solution despite these optimization challenges.
  • Contributions: Transformer-XL introduces recurrence into a purely self-attentive model and derives a novel positional encoding scheme to address fixed-length contexts.The paper presents these techniques as a complete solution: neither alone resolves the fixed-length-context problem.
  • Architecture: Reusing hidden states from previous segments creates recurrent connections that propagate information and enable modeling very long-term dependencies.The reused states serve as memory for the current segment instead of recomputing every segment from scratch.
  • Results: Transformer-XL achieves substantially better results than RNNs on both character-level and word-level language modeling.Its evaluation covers five datasets spanning word-level and character-level language modeling.
  • Results: Thousands of tokens: Transformer-XL generates relatively coherent long text articles when trained on only 100M tokens.The examples are reported in Appendix E.

2 Related Work

Prior work has advanced language modeling through improved context encoding, optimization, computation, and output distributions, while long-range dependency has been addressed with wider-context inputs and modified sequence-model architectures.

  • Language modeling advances: Recent language-modeling advances include novel context-encoding architectures, improved regularization and optimization, faster Softmax computation, and richer output distributions.These directions are represented by Bengio et al., Mikolov et al., Merity et al., Al-Rfou et al., Gal and Ghahramani, Grave et al., and Yang et al.
  • Long-range context: A line of language-modeling work captures long-range context by feeding a wider-context representation into the network as an additional input.The approach explicitly supplies broader contextual information to the model.
  • Long-range context: Existing wider-context methods use either manually defined context representations or document-level topics learned from data.The cited examples span manually specified context features and data-driven topic representations.
  • Sequence modeling: In generic sequence modeling, long-term dependency remains a longstanding problem, motivating efforts to mitigate vanishing gradients in LSTM-based models.These efforts include better initialization, additional loss signals, augmented memory structures, and internal RNN architectural modifications.

3 Model

Transformer-XL addresses fixed-context limitations with segment-level recurrence and a relative positional encoding scheme. The recurrence extends usable context and accelerates evaluation, while relative positions preserve temporal distinctions when cached hidden states are reused.

  • Fixed-Length Context: Fixed-length training prevents information flow across segments and bounds the largest possible dependency length by the segment length.Processing the entire context is usually infeasible because of limited computational resources.
  • Segment-Level Recurrence: Transformer-XL caches the previous segment’s hidden states, stops their gradients, and reuses them as extended context for the next segment.Applying recurrence across consecutive segments creates segment-level recurrence while keeping gradients within each segment.
  • Segment-Level Recurrence: O(N × L) is the largest possible dependency length, growing linearly with the number of layers N and segment length L.Unlike truncated BPTT, the method caches a sequence of hidden states rather than only the last one.
  • Evaluation Efficiency: 1,800+ times faster is Transformer-XL’s maximum evaluation speedup over the vanilla model on enwiki8.Previous-segment representations are reused instead of recomputed from scratch.
  • Relative Positional Encoding: Relative positional information prevents reused states from assigning the same positional encoding to corresponding positions in consecutive segments.The method replaces absolute key-position embeddings Uj with non-learned sinusoidal relative encodings Ri−j and adds a trainable parameter v for the query-position term.

4 Experiments

Experiments across word- and character-level benchmarks show that Transformer-XL achieves state-of-the-art performance, generalizes across dataset sizes, and evaluates substantially faster than vanilla Transformers. Ablations further attribute its long-context improvements to the recurrence mechanism and positional encoding scheme.

  • Benchmark results: Transformer-XL achieves new state-of-the-art results on WikiText-103, enwik8, and text8.On enwik8, the 12-layer model outperforms a same-depth vanilla Transformer by 0.05; on text8, it achieves a clear margin without further tuning.
  • Benchmark results: 17% of the parameter budget matches the result of a 64-layer vanilla Transformer on enwik8.Larger 18-layer and 24-layer Transformer-XL models produce another new state-of-the-art result.
  • Benchmark results: 21.8 improves the single-model state of the art from 23.7 on One Billion Word, despite the dataset testing mainly short-term dependency.The dataset shuffles sentences and does not preserve long-term dependency.
  • Benchmark results: Transformer-XL achieves a new Penn Treebank state-of-the-art result without two-step finetuning and generalizes on only 1M training tokens.The experiment uses variational dropout and weight averaging, with proper regularization.
  • Ablation studies: Both recurrence and the proposed positional encoding are necessary for best WikiText-103 performance and longer evaluation attention sequences.Although training backpropagation length is only 128, the two techniques allow increasing attention length during evaluation to 6…
  • Efficiency and context length: 1,874 times is the maximum evaluation speedup over the vanilla Transformer, enabled by state reuse.Relative effective context length reaches 900 on average at r = 0.1 and is 80% longer than recurrent networks and 450% longer than Transformers.

5 Conclusions

Transformer-XL achieves strong perplexity, models longer-term dependencies, accelerates evaluation, and generates coherent text articles. The authors envision applications in text generation, unsupervised feature learning, and image and speech modeling.

  • Transformer-XL obtains strong perplexity results and models longer-term dependency than RNNs and Transformer.
  • Transformer-XL achieves substantial speedup during evaluation and generates coherent text articles.
  • The authors envision applications in text generation, unsupervised feature learning, and image and speech modeling.

A Ablation Study with Memory Constraints

Under the same GPU memory constraints, Transformer-XL outperforms the baseline even when trained with a shorter backpropagation length.

  • Experimental setup: The ablation study evaluates Transformer-XL and a baseline on WikiText-103 under identical GPU memory constraints.Table 10 compares the models using the same memory budget.
  • Results: Transformer-XL remains superior to the baseline under the matched memory budget.The comparison is conducted with both models subject to the same memory constraints.
  • Results: Transformer-XL still outperforms the baseline when using a shorter backprop length.This result shows that the advantage persists despite the shorter backpropagation setting.

B Efficient Computation of the Attention with Relative Positional Embedding

The method reduces relative-positional attention computation from a quadratic pairwise procedure to linear-cost operations by exploiting the finite range of relative distances and structured matrix shifts. Both attention terms are computed through matrix multiplication or matrix-vector multiplication followed by left-shifts.

  • Linear-cost construction: The relative distance i − j ranges only from 0 to M + L − 1, enabling a linear-cost construction instead of evaluating every pair quadratically.M and L denote the memory length and segment length, respectively.
  • Term (b): The matrix B is obtained by computing eB = qQ⊤ and then applying left-shifts to its rows.Each row of B is a left-shifted version of the corresponding row of eB.
  • Term (d): The matrix D is formed analogously, with each row given by a left-shifted version of ed.The construction collects term (d) for all possible i, j into an L × (M + L) matrix.
  • Term (d): The main computation for D is the matrix-vector multiplication ed = [Qv]⊤, which is described as no longer expensive.This replaces the costly direct computation with a matrix-vector operation followed by the same shift structure.

C Details About RECL

RECL measures how much models reduce perplexity when extending context beyond a shared short-context baseline. It is computed by thresholding relative gains as context length increases, revealing longer effective context for Transformer-XL than RNNs and other baselines.

  • Metric definition: RECL defines each model’s token loss at short and long context lengths, then compares long-context performance against a shared baseline.The baseline uses the minimum short-context loss across the model group.
  • Metric definition: The shared baseline ensures fair comparison because every model must improve over the same short-context reference.Only losses below that baseline contribute to relative loss.
  • Metric definition: With ratio r, RECL can focus on positions where short-context modeling is insufficient.These are positions where the baseline performs poorly, indicating that short-term dependency is not enough.
  • RECL computation: Relative gain is defined as relative perplexity reduction, and RECL is found by increasing context in steps until the gain falls below 0.01.Starting from c and c′ = c + ∆, the procedure returns RECL = c when gi(c, c′) < 0.01.
  • Empirical comparison: Transformer-XL has longer RECL than RNNs and other baselines because its relative gains are substantially larger.Figure 3 visualizes unnormalized relative perplexity gains for context-length pairs with r = 0.1.
  • Empirical comparison: Figure 4 reports normal perplexity across varying context lengths without baseline calibration.The y-axis shows the uncalibrated perplexity values.

D Attention Visualization

Attention visualizations show that Transformer-XL generally emphasizes nearby tokens while retaining heads that attend broadly or selectively across its 640-token memory. Decomposing attention indicates that content-dependent positional bias drives the local focus, while another term favors a flatter, longer context.

  • Overall attention trend: The visualized model has 160 attention heads across 16 10-head transformer layers and uses a memory of length 640.Figure 5 averages attention over the previous 640 tokens, with each group of 10 heads corresponding to one layer.
  • Overall attention trend: Attention generally concentrates on nearby tokens, although some heads distribute attention broadly across the entire memory span.Notable wide-span behavior appears in head 8 of layer 1 and other selected heads.
  • Wide attention heads: Head 8 in layer 1 attends almost uniformly across the memory, head 78 in layer 8 attends sparsely across ranges, and head 158 in layer 16 has target-specific sparse foci.The final-layer pattern differs from head 78 because target locations do not largely share one memory location, and from head 8 because attention is not broadly uniform.
  • Attention-score decomposition: Term (a) is nearly uniform across context positions, aside from a few very close words likely semantically similar to the target.This follows from term (a) being fully content-based addressing when averaged over target words.
  • Attention-score decomposition: Term (b) largely produces the overall nearby-context focus, whereas term (d) is flatter and biased toward a longer context.Term (b) resembles the full attention distribution, while term (d) still favors nearby words but with a broader bias.

E Generated Text

Transformer-XL generates long, structurally coherent text from short WikiText-103 contexts, maintaining topics, section organization, long-range references, and chronological structure while also producing novel content. Its main limitation is occasional irrelevance or factual hallucination, attributed to training on only 100M tokens.

  • Generation setup: Using at most 512 seed tokens and top-40 sampling, Transformer-XL generates 500- or 1,000-token continuations from WikiText-103.The model was trained on the WikiText-103 dataset, with examples randomly sampled without cherry-picking.
  • Generation quality: Transformer-XL preserves Wikipedia’s sectional arrangement, stays semantically on topic, uses long-range references, and often generates novel content.These observations characterize the model’s long-text generation ability after training on only 100M tokens.
  • Limitations: Because it relies on limited 100M-token training knowledge, the model sometimes produces relevant but insufficiently precise text or factual hallucinations.The authors suggest that a larger training set could alleviate this limitation.
  • Example 1: The model maintains an imagined topic across generations, tracks section chronology from 2011 to 2013, and reuses “another back injury” consistently across seasons.The examples demonstrate topic persistence, chronological tracking, and long-term dependency capture.
  • Example 2: Transformer-XL identifies topic boundaries, starts a new page, and combines copied book information with hallucinated content, indicating generalization instead of memorization.The newly started page concerns “The Tale of Mrs. Tittlemouse,” whose title and related information are retained while novel content is generated.
  • Example 3: In a 1,000-token example, Transformer-XL stays on the Napoleon-wars topic and follows month-and-day chronology except for one 1804-versus-1805 error.The generated text makes up non-existing stories while preserving fine-grained chronological ordering over much of the passage.
Loading 1901.02860v3…