PaperlayerView exact revision on arXiv

Source-linked AI summary

LongT5: Efficient Text-To-Text Transformer for Long Sequences

Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, Yinfei Yang

arXiv:2112.07916v2cs.CL

TL;DR

LongT5 addresses the underexplored problem of scaling T5 input length during training alongside model size, despite full attention’s quadratic cost for long sequences. It combines ETC-inspired TGlobal attention and PEGASUS-style pre-training within T5, achieving state-of-the-art results on several datasets while leaving some efficiency directions for future work.

  • Problem

    Scaling T5 to longer input sequences during training remains underexplored, while full attention has quadratic computation growth that increases memory consumption and training time.

  • Method

    LongT5 integrates long-input attention and PEGASUS-style Principle Sentences Generation pre-training into the scalable T5 architecture, using TGlobal as a drop-in attention replacement without additional side inputs.

  • Results

    LongT5 achieves state-of-the-art results on arXiv, PubMed, BigPatent, MediaSum, and TriviaQA.

  • Takeaways & Limitations

    Jointly scaling input length and model size supports strong performance across several challenging summarization and question answering datasets.

  • Takeaways & Limitations

    LongT5 does not achieve state-of-the-art results on Multi-News, where PRIMER performs slightly better after pre-training on news-related documents.

Abstract

from arXiv · show

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call {\em Transient Global} (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on question answering tasks.

1 Introduction

LongT5 extends scalable T5 to jointly scale input length and model size by integrating long-input attention and PEGASUS-style pre-training. Its TGlobal mechanism adapts ETC-like local/global attention without additional side inputs, supporting strong results on long-sequence tasks.

  • Contributions: LongT5 scales both input length and model size within the scalable T5 architecture.The paper analyzes performance and computation trade-offs as both dimensions increase.
  • Attention mechanism: TGlobal is a drop-in attention replacement that mimics ETC’s local/global mechanism without requiring additional side inputs.It synthesizes global tokens from input groups at each attention layer.
  • Pre-training: LongT5 adopts PEGASUS-style pre-training that masks key sentences and trains the model to reproduce them as a single string.The authors report that this strategy also improves performance on question answering tasks.
  • Results: The model achieves state-of-the-art results on arXiv, PubMed, BigPatent, MediaSum, and TriviaQA.The paper evaluates summarization and question answering tasks, while making no state-of-the-art claim for Natural Questions because its formulation differs from the original task.
  • Motivation: T5’s full attention becomes costly for long inputs because computation grows quadratically with sequence length.This creates larger memory consumption and longer training time, while scaling T5 input length during training remains underexplored.

3 LongT5

LongT5 extends T5 with sparse encoder attention for long inputs and replaces ETC’s side-input global attention with dynamically constructed transient global tokens. It also adopts PEGASUS-style sentence-generation pre-training.

  • Architecture: LongT5 extends the T5 encoder with global-local attention sparsity while retaining the standard T5 decoder for short-output tasks.The architecture preserves relative position representations, example packing, and compatibility with T5 checkpoints.
  • Local Attention: Local Attention restricts each token to a sliding neighborhood of r tokens on both sides, with r = 127 found sufficient in practice.Its attention complexity is linear in input length for a fixed radius: O(l × r).
  • Transient Global Attention: TGlobal divides the input into fixed blocks and lets each token attend to nearby tokens plus a dynamically normalized global token for every block.These transient global tokens are constructed and discarded within each attention operation.
  • Transient Global Attention: TGlobal adds relative-position biases and layer-normalization parameters for global-token embeddings, while sharing the remaining parameters with T5.The base, large, and xl models add 10k, 25k, and 50k parameters, respectively.
  • Transient Global Attention: With block size k = 16, TGlobal has complexity O(l(r + l/k)) because each input token attends to local and block-level global representations.Sequence packing masks attention between global tokens belonging to different examples.
  • Pre-training: LongT5 adopts PEGASUS Principle Sentences Generation, masking key document sentences and training the model to reproduce them as one string.Sentences are selected using independently scored ROUGE-F1 overlap with the remaining document sentences.

4.1 Configurations

LongT5 uses three scalable model sizes, long pre-training sequences, and PEGASUS-style sentence-generation pre-training. Fine-tuning varies input lengths substantially across summarization and question-answering tasks.

  • Model and pre-training configuration: LongT5 considers base, large, and xl models containing approximately 220M, 770M, and 3B parameters, respectively.The models use T5.1.1’s cased English SentencePiece vocabulary with 32,000 sentence pieces.
  • Fine-tuning configuration: The implementation uses batch size 128, Adafactor, and greedy decoding for all experiments.The authors note that beam search could potentially improve the reported results.
  • Model and pre-training configuration: LongT5 is pre-trained for 1M steps with 4096 input tokens and 910 output tokens on C4 without dropout.The PEGASUS objective masks 0.2 of sentences, with an inverse-square-root learning-rate schedule and 10,000 warm-up steps.
  • Fine-tuning configuration: Fine-tuning uses learning rate 0.001 and dropout 0.1 across tasks.These settings are held constant for all fine-tuning experiments.
  • Fine-tuning configuration: Summarization experiments use input lengths of 4096, 8192, and 16384 with output length 512.Question-answering inputs range from 512 to 36864 tokens with output length 128.

4.2 Evaluation on Summarization Tasks

LongT5 is evaluated on summarization datasets spanning varied input lengths and compared with established long-document summarization systems using ROUGE metrics. It achieves state-of-the-art scores on four datasets, while ranking second on Multi-News and remaining comparable to HAT-BART on CNN / Daily Mail.

  • Datasets: LongT5 is benchmarked on six summarization datasets covering news, scientific documents, patents, interviews, and multi-document news.The datasets are CNN / Daily Mail, PubMed, arXiv, BigPatent, MediaSum, and Multi-News.
  • Datasets: The datasets include long inputs, so the evaluation varies context length and includes CNN / Daily Mail as a shorter common benchmark.The authors use these tasks to assess long-context understanding and generative summarization.
  • Evaluation: LongT5 is compared with BigBird-PEGASUS, HAT-BART, DANCER PEGASUS, PRIMER, TG-MultiSum, LED, and BART using ROUGE-1, ROUGE-2, and ROUGE-L.The comparison includes several leading long-document and summarization approaches.
  • Results: LongT5 achieves state-of-the-art ROUGE scores on arXiv, PubMed, BigPatent, and MediaSum.For arXiv and PubMed, scaling to 16k input length supports strong results on their longer inputs.
  • Results: LongT5 ranks second on Multi-News, slightly below PRIMER rather than achieving state-of-the-art results.The authors relate PRIMER’s advantage to pre-training on a large corpus of news-event documents similar to Multi-News.
  • Results: On CNN / Daily Mail, LongT5 is comparable with HAT-BART despite not using full attention and has stronger ROUGE-2 scores.This dataset has shorter inputs than the other long-context benchmarks.

4.3 Evaluation on QA Tasks

LongT5 is evaluated on long-context Natural Questions and TriviaQA tasks, comparing attention variants, input lengths, and model sizes. Longer inputs generally improve NQ performance, while LongT5 reaches state-of-the-art on TriviaQA by scaling to 16k tokens.

  • Natural Questions uses a seq2seq short-answer formulation, ignoring long answers and evaluating generated answer texts rather than answer spans.
  • NQ evaluation compares T5.1.1 and LongT5 Local and TGlobal models at 512 tokens and each model’s largest memory-feasible input length.
  • Longer input lengths generally provide significant NQ benefits, with larger-input models outperforming smaller-input models in most cases.
  • TGlobal large models reach 6k tokens versus 3k for T5.1.1 and outperform T5.1.1 at 4k tokens, although the 6k result dips under single-run variance.
  • TriviaQA compares LongT5 with BigBird-ETC, Fusion-in-Decoder, and ReadTwice; scaling model size and input length to 16k supports state-of-the-art performance.

5 Analysis

The analysis examines how input length affects training speed, memory limits, NQ performance, and pre-training benefits. LongT5 becomes faster and handles longer inputs than T5.1.1, while TGlobal and PEGASUS-style pre-training improve the overall trade-offs.

  • Input Length vs Performance: Figure 4 plots large-model speed against NQ short-answer F1 across input lengths, with each curve point labeled by sequence length.
  • Input Length vs Speed: At 2048 tokens, T5.1.1 base processes 479 sequences per second, versus 765 for LongT5 base TGlobal and 860 for base Local.
  • Input Length vs Speed: T5.1.1 reaches memory limits earlier: base handles 6k tokens, while LongT5 base Local reaches 36k and base TGlobal 12k.
  • Input Length vs Performance: Increasing input length significantly improves NQ performance, but Local Attention can substantially underperform TGlobal and T5.1.1 even at long lengths.
  • Pre-training Ablations: Principle Sentences Generation is compared with Span Corruption and their combination on NQ and arXiv using TGlobal base models at 4096-token fine-tuning length.
  • Pre-training Ablations: With equal pre-training input length, Principle Sentences Generation outperforms Span Corruption; longer LongT5 inputs yield stronger results despite full attention performing better at matched lengths.

6 Related Work

Related work established powerful Transformer pre-training and increasingly efficient approaches for long documents. These approaches include generative corruption objectives, hierarchical modeling, and sparse attention patterns that reduce computation.

  • BERT introduced masked language modeling, while T5 and BART use generative pre-training objectives suited to text generation.
  • Long-document modeling includes document embeddings and hierarchical attention networks for classification and translation.
  • Efficient Transformer methods restrict attention fields to reduce complexity from O(n^2) toward O(nlogn) or O(n), including Longformer, ETC, and BigBird.

7 Conclusion

LongT5 combines scalable attention and PEGASUS-style pre-training to study simultaneous scaling of input length and model size. It achieves state-of-the-art results on several datasets, while efficient decoder attention remains future work.

  • LongT5 studies scaling input length and model size simultaneously with a Transformer architecture based on T5.1.1.
  • TGlobal is a drop-in replacement for standard T5 attention that requires neither additional side inputs nor modified model inputs.
  • PEGASUS-style Principle Sentences Generation is the second main difference between LongT5 and T5.1.1.
  • LongT5 achieves state-of-the-art results on arXiv, PubMed, BigPatent, MediaSum, and TriviaQA across summarization and question answering evaluations.
  • Efficient attention for the decoder and decoder-to-encoder components remains future work because current Local and TGlobal attention are encoder-only.

A Summarization Results

Summarization results improve when models use longer inputs, PEGASUS-style pre-training, or both. LongT5 combines reduced TGlobal attention with larger inputs and Gap Sentences Generation pre-training while maintaining strong performance.

  • Longer input sizes improve summarization performance metrics, although T5 fine-tuning at 4k inputs can take many days on TPUv3.The computational burden limits practical scaling for standard T5 models.
  • T5.1.1 with PEGASUS Principle Sentences Generation pre-training outperforms regular T5.1.1, with results improving as input size increases.The passage attributes the best results to combining the pre-training objective with input scaling.
  • LongT5 achieves strong summarization performance by combining larger inputs with Gap Sentences Generation pre-training despite reduced attention from TGlobal.The results table includes standard T5, PEGASUS-pre-trained T5, and LongT5 models.

B QA Results

The QA evaluation compares T5.1.1 and LongT5 across NQ and TriviaQA at different sequence lengths. Training uses held-out development data for tuning and hardware-specific maximum input lengths.

  • QA results compare T5.1.1 and LongT5 models at different sequence lengths on NQ and TriviaQA.The comparison reports numbers on the official development sets.
  • For NQ and TriviaQA, 90% of the official training data is used for training and 10% as a hold-out development set for tuning.The official development set is reserved for reporting results.
  • Models are evaluated at the largest input length allowed before running out of memory on the specified TPUv3 configurations.Base and large models use 4x8 TPUv3 without partitioning, while xl models use 8x16 TPUv3 with eight partitions.
Loading 2112.07916v2…