PaperlayerView exact revision on arXiv

Source-linked AI summary

Longformer: The Long-Document Transformer

Iz Beltagy, Matthew E. Peters, Arman Cohan

arXiv:2004.05150v2cs.CL

TL;DR

Long sequences make standard Transformer self-attention infeasible because its cost grows quadratically with sequence length. Longformer replaces it with linearly scaling local and task-motivated global attention, and its pretrained model consistently outperforms RoBERTa on long-document tasks while achieving state-of-the-art results on WikiHop, TriviaQA, text8, enwik8, and arXiv summarization.

  • Problem

    Standard Transformer self-attention scales quadratically with sequence length, limiting efficient processing of long documents and leaving document-level transfer-learning applications relatively unexplored.

  • Method

    Longformer replaces full self-attention with linearly scaling windowed local attention combined with task-motivated global attention.

  • Results

    Pretrained Longformer consistently outperforms RoBERTa on long-document tasks and achieves state-of-the-art results on WikiHop, TriviaQA, text8, enwik8, and arXiv summarization.

  • Takeaways & Limitations

    Longformer supports broad document-level NLP and long-document sequence-to-sequence tasks without chunking or shortening inputs or requiring complex chunk-combination architectures.

  • Takeaways & Limitations

    On WikiHop, later BigBird results improved the leaderboard, and comparisons may be confounded because BigBird used 16X more pretraining compute than Longformer.

Abstract

from arXiv · show

Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

1 Introduction

The introduction presents Longformer as a linear-scaling Transformer for long documents, addressing the quadratic cost of self-attention. It combines local and task-motivated global attention, extends evaluation to document-level transfer learning, and introduces LED for long-document sequence-to-sequence tasks.

  • Motivation: Self-attention becomes infeasible or very expensive on long sequences because its memory and computational requirements grow quadratically with sequence length.
  • Longformer: Longformer modifies Transformer self-attention so it scales linearly with sequence length, enabling versatile processing of long documents.The architecture reduces the need for task-specific architectures by building contextual representations of the entire context through multiple attention layers.
  • Attention mechanism: Longformer combines windowed local-context attention with task-motivated global attention, and ablations find both types essential.Local attention primarily builds contextual representations, while global attention builds full-sequence representations for prediction.
  • Evaluation: 32K characters: Longformer processes sequences of up to 32K characters on modern GPUs and achieves state-of-the-art results on text8 and enwik8.This evaluation uses windowed attention combined with a new dilated attention pattern for autoregressive character-level language modeling.
  • Evaluation: Longformer is pretrained from a released RoBERTa checkpoint with masked language modeling and consistently outperforms RoBERTa on document-level tasks including classification, question answering, and coreference.The introduction frames this work as addressing the limited exploration of long-document Transformers for document-level NLP tasks in transfer learning.
  • LED: Longformer-Encoder-Decoder (LED) applies Longformer’s efficient attention pattern to the encoder for long-document sequence-to-sequence tasks and is effective on arXiv summarization.

2 Related Work

Prior long-document Transformers either process text left-to-right in chunks or use sparse attention, while task-specific methods work around BERT’s 512-token limit through truncation, chunking, or aggregation. Longformer belongs to the sparse-attention approach and combines local and global attention, alongside contemporaneous models exploring similar mechanisms for long-document tasks.

  • Long-Document Transformers: Left-to-right chunked models support autoregressive language modeling but are unsuitable for transfer learning tasks requiring bidirectional context.
  • Long-Document Transformers: Sparse-attention models avoid computing the full quadratic attention matrix, with Sparse Transformer using dilated sliding windows over 8x8 blocks.
  • Long-Document Transformers: Prior evaluations beyond autoregressive language modeling remained incomplete: BP-Transformer evaluated machine translation without pretrain-finetune, while Blockwise attention omitted language modeling and used relatively short QA documents.These limitations left effectiveness on long-document tasks unexplored.
  • Task-specific Models for Long Documents: Task-specific methods address BERT’s 512-token limit by truncating documents, processing length-512 chunks separately, or combining chunk activations with task-specific models.
  • Long-Document Transformers: Contemporaneous models including ETC, GMAT, and BigBird explored local-plus-global attention and pretraining for long-document natural-language tasks, with ETC using relative position embeddings and an additional CPC loss.ETC configures global attention somewhat differently from Longformer and reports strong results in reading comprehension and classification.

3 Longformer

Longformer replaces quadratic full self-attention with a sparse attention pattern that scales linearly with sequence length. It combines local sliding-window or dilated attention with task-specific global attention to capture both local and long-range information.

  • Attention pattern: Longformer sparsifies the full self-attention matrix into an attention pattern whose computation scales linearly with input sequence length.The original Transformer self-attention has O(n^2) time and memory complexity, whereas Longformer’s sparse pattern is linear in n.
  • Sliding Window: Each token attends within a fixed-size sliding window, and stacked layers expand the receptive field to ℓ × w.With window size w, the attention computation is O(n × w), while the top-layer receptive field is ℓ × w when w is fixed.
  • Dilated Sliding Window: Dilated sliding-window attention expands the receptive field to ℓ × d × w without increasing computation.For fixed dilation d and window size w, the receptive field can reach tens of thousands of tokens even for small d.
  • Dilated Sliding Window: Different dilation configurations across attention heads improve performance by combining local-context and longer-context modeling.Some heads omit dilation to focus locally, while others use dilation to capture longer context.
  • Global Attention: Longformer adds symmetric global attention to a few pre-selected locations, enabling task-specific representations and inductive bias.A globally attended token attends to all sequence tokens, and all tokens attend to it.
  • Global Attention: Separate projection sets for sliding-window and global attention provide flexibility and are critical for best downstream-task performance.The global projections Qg, Kg, Vg are initialized to match the sliding-window projections Qs, Ks, Vs.

4 Autoregressive Language Modeling

Longformer applies dilated sliding-window attention to autoregressive character-level language modeling, using layer-dependent windows and limited dilation to combine local and distant context. It achieves state-of-the-art small-model results on text8 and enwik8, while large models outperform Transformer-XL and match Sparse Transformer performance on enwik8.

  • Attention design: Longformer uses smaller attention windows in lower layers and progressively larger windows in higher layers, assigning local modeling to lower layers and sequence-level representation to upper layers.This balances efficiency with the ability to represent the full sequence.
  • Attention design: Higher layers apply increasing dilation to only two heads, while lower layers remain undilated so the model can attend distantly without sacrificing immediate local context.The design preserves lower-layer capacity for local information while exposing higher layers to distant tokens.
  • Results: 1.10 BPC on text8 and 1.00 BPC on enwik8 establish new state-of-the-art results for Longformer small models.The comparison uses character-level language modeling on text8 and enwik8.
  • Training: Training proceeds through five phases, increasing attention-window size and sequence length from 2,048 to 23,040 tokens as the model first learns local context.The staged schedule addresses the need for many updates before longer-context use is learned.
  • Evaluation: Evaluation uses overlapping sequences of length 32,256 with a step size of 512, reporting performance on the final 512 tokens.The setup follows the evaluation procedure of Dai et al. (2019).
  • Results: On enwik8, large Longformer models outperform comparable Transformer-XL, match comparable Sparse Transformer, and match or slightly underperform newer models with more than twice the parameters.These large-model experiments were limited to enwik8 because of their expense.

5 Pretraining and Finetuning

Longformer is pretrained by continuing RoBERTa’s masked-language-modeling training with minimal changes for its attention mechanism, then finetuned for long-document tasks. It supports sequences up to 4,096 tokens while preserving compatibility with pretrained transformer architectures.

  • Finetuning scope: The resulting model processes sequences up to 4,096 tokens, eight times longer than BERT, and is finetuned on six tasks including classification, question answering, and coreference resolution.Sequences up to 16K are possible on current GPUs.
  • Pretraining method: Longformer is pretrained with masked language modeling by continuing from RoBERTa’s released checkpoint and changing only what is necessary for Longformer attention.The attention pattern can be plugged into any pretrained transformer without changing its architecture.
  • Attention pattern: Longformer uses sliding-window attention with a window size of 512, matching RoBERTa’s computation.The method is designed as a drop-in replacement for standard pretrained transformer attention.
  • Position embeddings: To extend RoBERTa’s 512-position embeddings to 4,096 positions, Longformer adds embeddings initialized by repeatedly copying RoBERTa’s pretrained embeddings.Copy initialization substantially outperforms random initialization, indicating its importance.
  • Pretraining results: Freezing RoBERTa’s weights yields 1.850 BPC, compared with 1.705 BPC when all weights are trainable.The frozen configuration is intended to preserve RoBERTa’s short-document performance.

6 Tasks

Longformer is evaluated on long-document question answering, coreference resolution, and classification as a drop-in replacement for RoBERTa self-attention. It consistently outperforms RoBERTa, with especially clear gains on tasks requiring long context, and achieves large-margin state-of-the-art results on WikiHop and TriviaQA.

  • Task setup: The evaluation covers question answering, coreference resolution, and document classification on datasets whose contexts are significantly longer than 512 wordpieces.The tasks test whether Longformer can replace BERT-style self-attention and simplify models that otherwise require task-specific handling of limited context.
  • Task setup: The baseline processes each context in the longest possible RoBERTa segment, concatenates activations, and adds the question to every segment for question answering.Longformer instead replaces RoBERTa self-attention with windowed attention plus task-motivated global attention.
  • Main Result: Longformer consistently outperforms RoBERTa, with especially obvious gains on long-context tasks such as WikiHop and Hyperpartisan.Gains are more modest on TriviaQA and HotpotQA when local context or supporting-fact supervision helps identify relevant information, while gains on IMDB and OntoNotes are small.
  • Ablations: WikiHop ablations show benefits from longer sequences, global attention, separate global-attention projections, MLM pretraining, and longer training.Configured like RoBERTa-base with sequence length 512 and n2 attention, Longformer performs slightly worse than RoBERTa-base.

7 Longformer-Encoder-Decoder (LED)

Longformer-Encoder-Decoder (LED) adapts Longformer’s efficient encoder attention to sequence-to-sequence tasks while retaining a full-attention decoder. On long-document arXiv summarization, LED processes inputs up to 16K tokens and slightly outperforms BigBird without task-specific pretraining.

  • Architecture: LED combines encoder and decoder Transformer stacks, replacing encoder full self-attention with Longformer’s local+global attention while retaining full decoder attention.This design scales linearly with input length.
  • Evaluation: The arXiv summarization dataset is a long-document testbed whose 90th-percentile document length is 14.5K tokens.LED’s encoder reads the document and its decoder generates the summary.
  • Evaluation: The LED encoder uses a 1,024-token local-attention window and global attention on the first <s> token.The decoder uses full attention.
  • Results: LED-large 16K achieves state-of-the-art results on arXiv and slightly outperforms BigBird.The reported results use an arXiv summarization evaluation with ROUGE-1, ROUGE-2, and ROUGE-L metrics.
  • Results: LED achieves this result without additional pretraining or task-specific initialization, while processing longer inputs than BigBird’s 4K-token sequence length.BigBird starts from and continues pretraining Pegasus, a summarization-specific model.

8 Conclusion and Future Work

Longformer scales to long documents through linear-complexity attention combining local and global information, enabling document-level NLP without input chunking or complex cross-chunk architectures. It achieves state-of-the-art results across language modeling, long-document understanding, and summarization, while LED extends the approach to sequence-to-sequence tasks.

  • Contributions: Longformer processes long documents without chunking or shortening inputs and without complex architectures for combining information across chunks.Its attention combines local and global information while scaling linearly with sequence length.
  • Results: Longformer achieves state-of-the-art results on character-level language modeling tasks text8 and enwik8.
  • Results: When pretrained, Longformer consistently outperforms RoBERTa on long-document tasks and sets new state-of-the-art results on WikiHop and TriviaQA.
  • Results and Future Work: LED, Longformer’s encoder-decoder variant, achieves state-of-the-art results on the arXiv long-document summarization task.Future work includes studying other pretraining objectives, especially for LED, increasing sequence length, and exploring other tasks.

A Implementation Details

Longformer’s dilated sliding-window attention requires specialized banded matrix multiplication not directly supported by standard deep learning libraries. The authors implement it through memory- or compute-oriented strategies, including a custom TVM-generated CUDA kernel that is practical for current use.

  • Attention implementation: Banded matrix multiplication is needed for Longformer’s dilated sliding-window attention but is not directly supported by PyTorch or TensorFlow.The operation produces outputs that are zero except on selected diagonals.
  • Attention implementation: The block-based implementation uses one PyTorch matrix multiplication, making it compute efficient but consuming 2x the memory of a perfectly optimized implementation.It computes some zero values and is most suitable for pretraining and fine-tuning, where the extra memory was not problematic.
  • Attention implementation: Longformer-cuda uses a custom CUDA kernel generated and compiled for GPUs by TVM from a high-level Python description.TVM compiles the banded matrix multiplication into optimized device-specific code.
  • Limitations: A perfectly optimized Longformer-cuda implementation could theoretically outperform O(n^2) computation, but the current version is already sufficiently fast and practical.Achieving the theoretical advantage requires specialized low-level GPU programming knowledge comparable to highly optimized matrix multiplication.

B Character LM Hyperparameters

The character-level language-modeling experiments use Wikipedia-derived text8 and enwik8 datasets and a Transformer-XL-based implementation with modified self-attention. Experiments compare model-design and optimization choices through targeted hyperparameter searches, while avoiding fp16 attention because it caused numerical failures.

  • text8 and enwik8 each contain 100M Wikipedia characters, split into 90M, 5M, and 5M for training, development, and testing.
  • The implementation is based on Transformer-XL with its memory mechanism disabled and uses sinusoidal relative position embeddings.
  • Small-model experiments used 4 RTX8000 GPUs for 16 days, while large-model experiments used 8 RTX8000 GPUs for 13 days.
  • Hyperparameter search included position-embedding types, dropout values, layer-normalization placement, and learning-rate settings, with configurations run for 150K steps on text8.
  • Using fp16 in the attention operation caused floating-point overflow and NaNs during later training stages.

C Pretraining Data

The pretraining corpus was compiled from long documents to help the model learn long dependencies. It combined RoBERTa-era sources with selected long-document subsets from Realnews and Stories.

  • C Pretraining Data: The corpus was compiled from long documents so the model could learn long dependencies during pretraining.Its sources included Books and English Wikipedia, both also used in original RoBERTa pretraining.
  • C Pretraining Data: One third of a Realnews subset containing documents longer than 1,200 tokens and one third of the Stories corpus were additionally included.The corpus was designed to include a mix of long-document data sources.

D Task specific model details

The paper specifies task-specific input construction, chunking, global-attention patterns, and prediction heads for WikiHop, TriviaQA, and HotpotQA, alongside a Longformer adaptation for coreference resolution. HotpotQA uses a two-stage model because paragraph filtering reduces distraction, improving joint F1 by about 4.2 points.

  • WikiHop: WikiHop concatenates tokenized questions, answer candidates, and support contexts, processes context chunks up to 4,096 tokens with candidate-focused global attention, and averages candidate logits across chunks.RoBERTa uses 512-token chunks; Longformer uses 4,096-token chunks, except Longformer-large, which uses only the first 4,096 tokens because of memory requirements.
  • TriviaQA: TriviaQA truncates documents to 4,096 wordpieces, applies global attention to question tokens, and predicts answer-span beginnings and ends with a distant-supervision loss requiring one correct span.Answers are named entities in Wikipedia documents, and answer spans are identified through text matching rather than gold annotations.
  • HotpotQA: HotpotQA jointly extracts answer spans and evidence sentences from ten paragraphs, but a two-stage Longformer first identifies relevant paragraphs before final prediction.The input concatenates the question and all paragraphs; global attention covers question, paragraph-title, and sentence tokens.
  • HotpotQA: 4.2 points is the reported improvement of the two-stage model over the single-stage model on joint-F1.The improvement is attributed to removing distracting paragraphs before evidence and span detection.
  • Coreference model details: The coreference model splits documents into non-overlapping segments, concatenates their activations for coarse-to-fine clustering, and uses sequence lengths of 384 for RoBERTa-base and 4,096 for Longformer-base.Its implementation exchanges activations and gradients between PyTorch and TensorFlow to reuse the original coarse-to-fine logic.
Loading 2004.05150v2…