Source-linked AI summary
SCROLLS: Standardized CompaRison Over Long Language Sequences
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, Omer Levy
TL;DR
Existing NLP benchmarks mainly target short texts, leaving limited standardized evaluation for reasoning over naturally long language. SCROLLS curates seven diverse datasets requiring long-range contextualization, unifies them in a text-to-text benchmark, and finds that initial baselines still face a formidable challenge. The benchmark is intended to support research on architectures, pretraining, and alternatives for long-text processing.
Problem
Standard NLP benchmarks focus largely on short texts, while existing long-sequence evaluations do not adequately test naturally long language and long-range dependencies.
Method
SCROLLS handpicks 7 naturally long datasets across summarization, question answering, and natural language inference, then converts them into a unified text-to-text format.
Results
Initial BART and Longformer Encoder-Decoder baselines show that SCROLLS poses a formidable challenge, leaving substantial room for improvement.
Takeaways & Limitations
SCROLLS provides a standardized evaluation platform for future long-text architectures, pretraining schemes, and retrieval-based methods.
Takeaways & Limitations
SCROLLS evaluates only English and relies on ROUGE for summarization, which may undervalue semantically equivalent paraphrases.
Abstract
from arXiv · showhide
NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild. We introduce SCROLLS, a suite of tasks that require reasoning over long texts. We examine existing long-text datasets, and handpick ones where the text is naturally long, while prioritizing tasks that involve synthesizing information across the input. SCROLLS contains summarization, question answering, and natural language inference tasks, covering multiple domains, including literature, science, business, and entertainment. Initial baselines, including Longformer Encoder-Decoder, indicate that there is ample room for improvement on SCROLLS. We make all datasets available in a unified text-to-text format and host a live leaderboard to facilitate research on model architecture and pretraining methods.
1 Introduction
SCROLLS addresses the lack of a standard evaluation scheme for models that process naturally long language and reason across distant parts of documents. It curates diverse long-text tasks and provides a unified benchmark with initial baselines that remain far from solved.
- Motivation: SCROLLS targets long-discourse processing, whereas established NLP benchmarks have primarily focused on sentences and paragraphs.The motivation includes books, articles, and meeting transcripts as naturally long language contexts.
- Motivation: Existing evaluations based on language-model perplexity and Long Range Arena do not adequately test natural-language long-range reasoning.Perplexity mainly captures local patterns, while LRA contains few natural-language datasets and artificially elongated sequences.
- Benchmark: SCROLLS handpicks 7 datasets spanning summarization, question answering, and natural language inference across multiple domains.The benchmark covers literature, science, business, entertainment, and meeting transcripts.
- Benchmark: SCROLLS converts the datasets into a unified text-to-text format for evaluating a single model across tasks.Its inputs typically contain thousands of words, and critical information is distributed across longer distances within documents.
- Baselines and evaluation: Initial BART and Longformer Encoder-Decoder baselines indicate that SCROLLS remains a formidable challenge with substantial room for improvement.The benchmark also supports aggregate and dataset-level evaluation through a live leaderboard.
2 Background: Contemporary Evaluation of Long-Text Models
Long-sequence evaluation has expanded beyond language modeling, but existing approaches often fail to represent naturally long language or long-range dependencies. SCROLLS responds by emphasizing diverse, naturally long summarization tasks alongside other language tasks.
- Evaluation context: Transformer self-attention creates a computational challenge for long sequences because its complexity is O(n^2) in sequence length.This challenge has motivated many efficient transformer alternatives, making comparable evaluation important.
- Language modeling: Perplexity evaluates next-token prediction, but next-token prediction is mostly local and may not require modeling long-range dependencies.Masking or down-weighting distant tokens can even improve perplexity.
- Long Range Arena: Long Range Arena includes only two natural-language datasets, artificially inflates sequences through byte tokenization, and truncates examples below 1,000 words.Its multimodal composition also limits its relevance as a testbed for pretrained language models.
- Summarization: Long-document summarization evaluations have often emphasized academic abstracts, while SCROLLS adds datasets from several domains.SCROLLS includes summarization as one of its main tasks to increase domain diversity.
3 The SCROLLS Benchmark
SCROLLS curates seven naturally long datasets requiring contextualization and synthesis across distant text, then standardizes them into a common sequence-to-sequence format. Its task and domain coverage includes summarization, question answering, and natural language inference.
- Curation: SCROLLS selects datasets that challenge models through both long inputs and dependencies spanning different sections.The curation also preserves task diversity, including summarization, query-based summarization, open-ended and multiple-choice question answering, and natural language inference.
- Curation: The benchmark handpicks 7 datasets and processes them into a uniform text-to-text format.Examples include GovReport, SummScreenFD, QMSum, Qasper, NarrativeQA, QuALITY, and Contract NLI.
- Summarization datasets: GovReport pairs long national-policy reports with expert-written executive summaries, while SummScreenFD maps TV episode transcripts to recaps.The selected sources extend summarization beyond academic domains.
- Question-based summarization: QMSum uses meeting transcripts and queries whose relevant text spans at least 200 words or 10 turns.Its meetings cover academic, industrial, and parliamentary domains.
- Question answering and inference: Qasper and NarrativeQA require answering questions over entire NLP papers, books, or movie scripts, while QuALITY requires multiple-choice reasoning over large portions of stories or articles.Contract NLI evaluates whether legal hypotheses are entailed, neutral, or contradicted by non-disclosure agreements.
- Unified format: Every dataset is reformulated as sequence-to-sequence input-output pairs, with queries prepended to raw text when applicable.QuALITY also includes all four answer candidates in its query.
4 Quantitative Analysis
SCROLLS is designed to test whether correct outputs require integrating information distributed across long inputs, rather than merely processing longer sequences. Its quantitative analysis finds substantially greater output-information spread than in standard comparison datasets.
- Methodology: SCROLLS output production typically requires fusing input parts separated by hundreds or thousands of words.The analysis measures spread using the standard deviation of output-bigram locations in the input.
- Summarization datasets: SCROLLS summarization reference bigrams are spread across much larger distances than in CNN/DM and 1.5 to 2 times farther than in arXiv on average.Figure 4a compares the three SCROLLS summarization datasets with CNN/DM and arXiv.
- QA and NLI datasets: SCROLLS question-answering outputs are usually separated by hundreds of words, compared with under 5 words in SQuAD and Natural Questions.NarrativeQA contains both tightly clustered answers and a substantial subset with widely dispersed answer bigrams.
5 Experiments
SCROLLS experiments compare heuristic and transformer baselines under different input-length limits, showing that additional context generally improves scores while current models remain well below estimated human performance.
- 5.1 Baselines: The experiments fine-tune BART and LED alongside naive heuristics to establish baseline performance on SCROLLS.BART truncates inputs to at most 1,024 tokens, while LED uses sliding-window attention and tests sequences up to 16,384 tokens.
- 5.1 Baselines: SCROLLS measures whether models can process naturally long inputs whose critical information is distributed across distant portions of documents.The benchmark includes multiple domains and tasks, while its analysis compares the spread of reference-text bigrams across datasets.
- 5.2 Results: 2.66 points: BART’s SCROLLS score increases when its input limit grows from 256 to 1,024 tokens; LED gains 2.1 points from 1,024 to 16,384 tokens.The improvement is relatively consistent across BART datasets but less consistent for LED, including QMSum and ContractNLI.
- 5.2 Results: The experiments show that evaluation must assess both efficient long-sequence processing and effective semantic modeling.This is the evaluation capability SCROLLS is designed to provide through aggregate and dataset-level scores.
- 5.2 Results: 7 to 10 points: model baselines improve over the heuristic lower-bound average score of 19.35, but contemporary off-the-shelf models still struggle relative to human-agreement indicators.NarrativeQA provides one comparison: approximately 58.7% F1 inter-annotator agreement versus 18.5% F1 for the best baseline.
6 Conclusion
SCROLLS introduces a benchmark centered on naturally long texts and the challenges of processing and reasoning over longer discourses. It targets evaluation of efficient architectures, alternatives, and pretraining schemes for long language sequences.
- SCROLLS places naturally long texts and their intricacies at the center of benchmark evaluation.
- The benchmark addresses a gap in evaluating efficient transformer architectures and their alternatives on natural language tasks.
- SCROLLS provides a testing ground for pretraining schemes targeting long language sequences.
7 Limitations
SCROLLS has limitations in evaluating long summarization outputs and in language coverage. Its summarization evaluation may undervalue semantically equivalent paraphrases, and the benchmark is monolingual.
- SCROLLS’s main limitation is evaluating long output texts, specifically in summarization.ROUGE accounts for n-gram overlap and may downvalue paraphrases with the same semantic content.
- Unbiased automated metrics for long generations that correlate well with human judgments remain an emerging research area.The authors may replace or complement ROUGE with model-based evaluation in the future.
- SCROLLS is monolingual, limiting its coverage to one language.The paper identifies benchmarks focused on other languages as a natural future direction.
B Dataset Splits
Table 3 reports the number of examples in SCROLLS training, validation, and test sets. The table organizes dataset sizes by the original split categories.
- Table 3 reports the number of examples in each dataset split.
- The reported split categories are training, validation, and test.
- The table’s counts distinguish dataset examples by their train, validation, and test allocation.
C Original Datasets Results
Table 4 reports dataset-author results from sequence-to-sequence baselines when available. These results are mostly not directly comparable with SCROLLS results because the data was cleaned, filtered, and reformatted.
- Table 4 reports results achieved by the original datasets’ authors.
- The reported baselines are sequence-to-sequence models when applicable.
- Most Table 4 results are incomparable with SCROLLS results after data cleaning, filtering, and reformatting.
D Hyperparameters
The baselines are fine-tuned separately on each dataset with large-batch mixed-precision training, scheduled learning rates, dropout, and greedy decoding.
- Training setup: Each baseline is fine-tuned separately on every dataset using AdamW, mixed precision, and gradient checkpointing.Training uses β = (0.9, 0.98) and ε = 1e-6.
- Training setup: 131,072 tokens form the effective batch size, using 16,384 tokens per GPU across 8 NVIDIA V100 GPUs.The GPUs operate either in parallel or through gradient accumulation.
- Optimization: The learning rate warms up over 10% of training and linearly decays over the remaining 90%, with 0.1 dropout throughout each network.Summarization datasets train for 10 epochs, while Qasper, QuALITY, and ContractNLI train for 20.
- Inference: Inference uses greedy decoding to generate outputs.
E Qualitative Analysis
The qualitative analysis presents SCROLLS examples in which answering or summarizing requires locating and synthesizing information distributed across long documents. The examples span government reports, meetings, scientific papers, narratives, and NDAs, and are shown in the benchmark’s text-to-text format.
- Analysis scope: The analysis manually examines examples from each dataset to demonstrate long-range contextualization and information synthesis.Figures 5–9 show references, relevant document parts, and queries when applicable.
- Presentation: The benchmark’s text-to-text illustration truncates dataset inputs and outputs for brevity.
- Summarization: GovReport summaries can require information from the first and last sentences of a long government report.
- Summarization: QMSum answers may depend on relevant information spread across different locations in meeting transcripts.
- Question answering: Qasper questions can combine evidence from the introduction with a later results-table description of neural-model experiments.
- Question answering and inference: NarrativeQA and ContractNLI add ambiguity: narrative answers may be preceded by distractors, while NDA evidence can be implicit and located mid-document.The ContractNLI example concerns an implicitly contradicted hypothesis.