Long Range Arena: A Benchmark for Efficient Transformers

Yi Tay; Mostafa Dehghani; Samira Abnar; Yikang Shen; Dara Bahri; Philip Pham; Jinfeng Rao; Liu Yang; Sebastian Ruder; Donald Metzler

Source-linked AI summary

Long Range Arena: A Benchmark for Efficient Transformers

Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, Donald Metzler

arXiv:2011.04006v1Nov 8, 2020cs.LGcs.AIcs.CLcs.CVcs.IR

TL;DR

Efficient Transformers lack a consistent way to evaluate long-context quality, while their quadratic self-attention limits longer sequences. LRA addresses this gap with a unified, capability-focused benchmark spanning 1K–16K-token tasks and a side-by-side evaluation of ten models. The benchmark finds challenging tasks and no one-size-fits-all solution, with quality–speed–memory trade-offs.

Problem
Efficient Transformer research lacks a unified long-context test bed, and inconsistent tasks and datasets make relative model quality difficult to assess.
Method
LRA combines synthetic and real-world tasks spanning diverse data types and structures, evaluates ten efficient Transformer models, and analyzes quality, efficiency, and memory use.
Results
Across tasks, the benchmark is challenging and shows no one-size-fits-all solution, requiring trade-offs among model quality, speed, and memory.
Takeaways & Limitations
LRA provides a unified, open benchmark for comparing efficient and long-range sequence models and for guiding future research.
Takeaways & Limitations
Reported model rankings are not authoritative because aggressive hyperparameter tuning could change both best performance and relative ordering.

Abstract

from arXiv · show

Transformers do not scale very well to long sequence lengths largely because of quadratic self-attention complexity. In the recent months, a wide spectrum of efficient, fast Transformers have been proposed to tackle this problem, more often than not claiming superior or comparable model quality to vanilla Transformer models. To this date, there is no well-established consensus on how to evaluate this class of models. Moreover, inconsistent benchmarking on a wide spectrum of tasks and datasets makes it difficult to assess relative model quality amongst many models. This paper proposes a systematic and unified benchmark, LRA, specifically focused on evaluating model quality under long-context scenarios. Our benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens, encompassing a wide range of data types and modalities such as text, natural, synthetic images, and mathematical expressions requiring similarity, structural, and visual-spatial reasoning. We systematically evaluate ten well-established long-range Transformer models (Reformers, Linformers, Linear Transformers, Sinkhorn Transformers, Performers, Synthesizers, Sparse Transformers, and Longformers) on our newly proposed benchmark suite. LRA paves the way towards better understanding this class of efficient Transformer models, facilitates more research in this direction, and presents new challenging tasks to tackle. Our benchmark code will be released at https://github.com/google-research/long-range-arena.

1 INTRODUCTION

Long-Range Arena addresses inconsistent evaluation of efficient Transformers by providing a unified, capability-focused benchmark and side-by-side analysis. It separates architectural inductive biases from costly pretraining and examines quality, efficiency, and memory use.

Motivation: Efficient Transformers lack a unified test bed, and differing tasks and datasets make relative strengths and weaknesses difficult to assess.Existing benchmarks are also often chosen without considering whether they evaluate long-range modeling.
Motivation: Pretraining can obscure the value of an architecture and is computationally expensive, motivating its separation from inductive-bias research.The paper argues that decoupling these factors would make efficient-Transformer research more accessible.
Benchmark contribution: LRA combines synthetic probing and real-world tasks to compare ten recently proposed efficient Transformer models under long-context conditions.The benchmark is presented as a comprehensive side-by-side evaluation of this model class.
Benchmark contribution: The benchmark probes how architectures handle diverse data types and structures, including hierarchical and spatial organization.It is designed to compare model capabilities across varied circumstances rather than only measure aggregate quality.
Benchmark contribution: LRA also analyzes model efficiency and memory usage to provide practical side-by-side comparisons.The framework is open sourced in JAX/FLAX to support extension and reuse.

2 LONG-RANGE ARENA (LRA)

LRA is an open, lightweight benchmark spanning long inputs and diverse reasoning capabilities, from hierarchical text and similarity matching to visual-spatial tasks. Its task designs explicitly probe long-range dependencies, including an extreme 16K-pixel Pathfinder variant, and use required attention span to characterize task demands.

Benchmark design: LRA is implemented in Python and JAX/Flax and open sourced to make the benchmark easy to extend.
Benchmark design: The benchmark prioritizes generality, simplicity, challenge, long inputs, diverse capability probing, and accessibility without industry-grade computing resources.It excludes pretraining and particular data augmentation to simplify comparisons.
Long ListOps: Long ListOps uses sequences up to 2K tokens to test hierarchical reasoning that requires accessing all tokens and modeling logical structure.It is a challenging ten-way classification task.
Byte-level text classification: Byte-level IMDb classification uses unsegmented sequences up to 4K characters, requiring composition from characters to words and higher-level phrases.The task uses binary classification on real-world document data.
Document matching: The document-matching task tests whether two 4K-token documents can be compressed into representations suitable for similarity-based matching without cross attention.The total text length is 8K, using citation links in the ACL Anthology Network as the binary target.
Image classification: Image classification flattens N × N grayscale images into pixel sequences, requiring models to learn two-dimensional spatial relations from one-dimensional symbols.CIFAR-10 images use 8-bit pixel intensities without an additional CNN stem.
Required attention span: Required attention span measures the attention-weighted mean distance between query tokens and attended tokens, averaged across attention modules and validation samples.Across LRA tasks, the span is high, indicating that models must combine more than local information; the metric is treated as a proxy for task difficulty.

3 EXPERIMENTAL RESULTS

LRA evaluates diverse efficient Transformer models under challenging long-context tasks, revealing substantial task-dependent performance differences alongside speed and memory trade-offs. The results also show that fixed-hyperparameter comparisons are informative but not definitive rankings.

Models and setup: LRA compares ten efficient Transformer models, plus vanilla Transformer and local attention baselines, across a unified benchmark.The evaluation is intended to represent a diverse cross-section of recent efficient Transformer models.
Models and setup: The evaluation fixes major hyperparameters across models because a perfectly fair comparison is impractical, so rankings may change with aggressive tuning.The authors release implementation details and frame the results as a starting point rather than an authoritative ranking.
Task results: LRA tasks are challenging overall, and model performance differs meaningfully across architectures and task types.The benchmark is designed to probe capabilities such as handling hierarchical and spatial structure.
Task results: 37% is the best ListOps 10-way classification result, with roughly half the models exceeding 30% and kernel-based models performing less effectively on hierarchical data.The authors suggest that efficiency-inspired inductive biases may affect performance on hierarchically structured data.
Task results: 55% is the average Retrieval score, with Sparse Transformer best and BigBird second, while fixed sparse patterns outperform low-rank and kernel-based approaches.The vanilla Transformer reaches 57.46% accuracy, and some variants score near random chance.
Efficiency and overall results: At 4K tokens, Performer is 5.7× faster than vanilla Transformer, while Linformer uses 0.99GB versus 9.48GB per TPU device for vanilla Transformer.The authors report these efficiency measurements using batch size 32 on 4×4 TPU V3 chips; BigBird has the best integrated LRA score, while kernel-based variants offer a stronger speed-performance-memory trade-off.

4 RELATED WORK

Prior evaluations of long-range sequence models use heterogeneous tasks and datasets, while common benchmarks often do not test genuinely long-context reasoning. The paper motivates broader evaluations spanning long-range dependencies and diverse task structures.

Efficient Transformer studies lack a unified evaluation consensus, with each model often tested on different tasks and datasets.
Prior work ranges from fixed local-window attention to combined and learned attention patterns for reducing Transformer costs.
Generative modeling tasks predict the next character, word, or pixel, but their long-range signal has been questioned.
Question-answering benchmarks can require long contexts, including entire documents or books, and multiple reasoning hops.
Many natural-language understanding benchmarks use single-sentence examples averaging fewer than 100 tokens.

5 CONCLUSION

LRA is a challenging benchmark for efficient Transformers, covering diverse data types, structures, and sequence lengths. Results show that model quality, speed, and memory involve trade-offs rather than a single best solution.

LRA evaluates efficient Transformers on text, mathematics, and visual data using tasks ranging from 1K to 16K tokens.
An extensive side-by-side comparison evaluates ten recently proposed efficient Transformer models.
The tasks remain challenging even for long-range Transformer models.
Overall results show no one-size-fits-all solution, requiring trade-offs among model quality, speed, and memory.
The authors plan to open-source the code and benchmarks to support future benchmarking, research, and model development.

A.1 LRA TASKS

The LRA task suite uses long sequences and standardized model configurations across document, language, matching, and image-classification settings. Tasks employ encoder representations and classification heads tailored to each setup.

The benchmark implementation includes task, evaluator, and model configuration files intended to reproduce reported results.
A generated long-version task uses 2K-token sequences, six layers, eight heads, 512-dimensional embeddings, and a 10-class classifier.
IMDb document classification evaluates sequence lengths from 1K to 4K tokens and uses [cls] representations passed to a two-layer MLP.
Related-article matching encodes two 4K-token documents separately, sharing encoder parameters and producing a score from their [cls] embeddings.
The matching setup uses a two-layer ReLU MLP, batch size 32, 128-dimensional embeddings, four heads, four layers, and 5K training steps.
Grayscale CIFAR10 images are flattened into sequences of 1024 pixels and classified with a two-layer ReLU MLP using softmax cross-entropy.

A.2.1 GENERALIZATION GAP

Transformer models show a substantial train–test generalization gap on the image-classification task and find the setup difficult despite conventional models performing well. Architectural changes can improve performance but alter the benchmark setting.

Almost all evaluated models have a considerably high gap between training and test accuracy on image classification.
The task is difficult for Transformer models even though wide-resnet achieves 89.21 accuracy on grayscale CIFAR10 without data augmentation.
Treating pixel intensities as independent symbols leaves models uninformed about their ordinal structure, making suitable embeddings difficult to learn.
Replacing the embedding layer with a conventional CNN stem raises vanilla Transformer accuracy to 75.32 in the reported setup.
Adding 2D relative positional embeddings gives vanilla Transformer accuracy of 61.72, but such changes depart from the benchmark’s original task design.

A.2.2 VISUALIZATIONS OF LEANED EMBEDDING BY A VANILLA TRANSFORMER

The visualizations examine how a vanilla Transformer embeds pixel intensities and positions for image inputs. Learned intensity embeddings only partly reflect intensity order, while positional embeddings preserve spatial proximity unevenly across axes.

Embedding visualizations: Figure 4 compares cosine similarities among learned pixel-intensity embeddings and among positional embeddings for image pixels.The left visualization concerns intensity values; the right uses each indicated pixel position as a reference against all others.
Pixel-intensity embeddings: Learned pixel-intensity embeddings are more similar for close values but do not perfectly encode intensity ordinality.This concerns the pairwise similarities shown on the left side of Figure 4.
Positional embeddings: Positional embeddings become more similar as pixels become spatially closer.The right-side visualization compares the position embedding of each indicated pixel with those of all other pixels.
Positional embeddings: Spatial closeness along the y axis is better preserved in learned positional embeddings than distance along the x axis.The comparison concerns the directional structure of the learned positional representation.
Task context: Pathfinder requires models to detect long-range spatial dependencies by identifying and tracing a target contour across flattened image pixels.The task uses 32 × 32 images represented as sequences of 1024 pixels.

A.3.1 VISUALIZATION OF THE ATTENTION MAPS FROM A VANILLA TRANSFORMER

The attention visualization examines how a vanilla Transformer uses the final-layer CLS query on Pathfinder examples. The attention distribution collects information from multiple input positions to trace the target path, while Pathfinder-X tests a much longer 16K-pixel setting.

Attention behavior: The attention module collects information from different input positions to trace the target path.This reflects the global-receptive-field design of Transformer attention for the Pathfinder task.
Attention-map visualization: Figure 5 visualizes Pathfinder attention distributions from the final-layer CLS token, averaged across heads in a vanilla Transformer.Attention-rollout is used for more precise input attribution in the visualization.
Pathfinder-X: Pathfinder-X extends the task to 16K-pixel inputs, and none of the evaluated models handles or generalizes to that input length.This holds both when models are trained from scratch and when they are evaluated after training on Pathfinder.

B MODELS AND IMPLEMENTATION

The implementation uses JAX and FLAX and documents model-specific settings for the diverse efficient-Transformer implementations. Several low-level choices differ across architectures, including attention blocking, projection sharing, and Performer’s FAVOR+ implementation.

Implementation framework: The implementation is primarily written in JAX and FLAX, with some model-specific details documented for the evaluated xformers.Additional fine-grained details and hyperparameters were planned for release in code or accompanying scripts.
Attention settings: Local attention uses non-overlapping blocks, including within Sinkhorn Transformer blocks.The implementation explicitly avoids overlapping windows in both cases.
Projection settings: Linformer projections are shared between keys and values but not across multiple layers.This specifies the scope of projection sharing in the implementation.
Kernel-based models: Performer models use FAVOR+, described as the more recent version in the implementation overview.The passage identifies this as the implementation choice for Performer models.

B.2 SPECIAL CASES OF OUR IMPLEMENTATION

The implementation includes special cases arising from hardware and operator constraints, which complicate direct comparisons. Sparse Transformer and Longformer speed are excluded because equivalent masked implementations emulate unavailable CUDA kernels, while Reformer uses a distinct batching strategy.

Implementation constraints: Hardware support and implementation details create trade-offs and edge cases when comparing the evaluated Transformer models.The paper characterizes hardware support as part of whether a model can realize its technical design in practice.
Speed comparison: Sparse Transformer and Longformer are not benchmarked for speed because their CUDA kernels are emulated with equivalent masks.The stated reason is that CUDA kernels are cumbersome, hardware-specific, and difficult to use on TPU pods.
Reformer implementation: Reformer computes attention on N × d tensors before parallelizing over batch and head dimensions with VMAP.This differs from computing tensors with batch and head dimensions directly.
Comparison conditions: Re-evaluation and copied LRA comparisons retain fixed model size, prohibit pretraining, and preserve fundamental task setups.The paper gives these conditions for maintaining fair comparison when proposing or evaluating new models.