Source-linked AI summary
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, Omer Levy
TL;DR
ZeroSCROLLS addresses the limited evaluation of zero-shot language understanding over long texts by introducing a benchmark without training data and with new information-fusing tasks. Its evaluation finds GPT-4 and Claude strongest overall, while aggregation tasks remain difficult and output-format discrepancies complicate automatic scoring.
Problem
Existing long-text benchmarks rely on task-specific fine-tuning, while zero-shot benchmarks mostly evaluate short sequences.
Method
ZeroSCROLLS extends SCROLLS with four tasks and evaluates open and closed language models using test sets, small validation sets, private references, and no training data.
Results
GPT-4 achieves the highest average score at 41.7, with Claude closest at 39.1; aggregation tasks remain exceptionally challenging, with only GPT-4 surpassing the naive baseline on each.
Takeaways & Limitations
ZeroSCROLLS enables systematic comparison of LLMs on naturally long inputs and tasks requiring contextualizing and aggregating information across documents.
Takeaways & Limitations
Automatic ROUGE and F1 scores can undervalue semantically equivalent outputs, and common prompts may not yield the best performance for every model.
Abstract
from arXiv · showhide
We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data. We adapt six tasks from the SCROLLS benchmark, and add four new datasets, including two novel information fusing tasks, such as aggregating the percentage of positive reviews. Using ZeroSCROLLS, we conduct a comprehensive evaluation of both open-source and closed large language models, finding that Claude outperforms ChatGPT, and that GPT-4 achieves the highest average score. However, there is still room for improvement on multiple open challenges in ZeroSCROLLS, such as aggregation tasks, where models struggle to pass the naive baseline. As the state of the art is a moving target, we invite researchers to evaluate their ideas on the live ZeroSCROLLS leaderboard.
1 Introduction
ZeroSCROLLS fills a gap in zero-shot evaluation by benchmarking long-text understanding and comparing open and closed language models across diverse tasks. It introduces new tasks for aggregating and comparing information over long sequences, while revealing strong performance on some question-answering tasks and persistent challenges in summarization and aggregation.
- Motivation: ZeroSCROLLS benchmarks zero-shot natural language understanding over long sequences, addressing the short-input focus of existing zero-shot benchmarks.BigBench, for example, averages 77 words per input.
- Benchmark design: The benchmark extends SCROLLS with four tasks, including query-based summarization, multi-hop question answering, sentiment aggregation, and sorting book chapter summaries.The latter two tasks test aggregation and comparison across long sequences.
- Benchmark design: ZeroSCROLLS uses test sets with natural prompts and private gold references, small validation sets, no training data, and a live leaderboard.These design choices support zero-shot evaluation and transparent, dynamic progress.
- Evaluation findings: GPT-4 nearly reaches human performance on the challenging QuALITY question-answering task, while zero-shot LLMs bridge the gap with task-specific fine-tuned models on question answering.The passage reports this as a category-level finding from the benchmark evaluation.
- Evaluation findings: Only GPT-4 surpasses the naive baseline on both sentiment aggregation and book-chapter-summary sorting, which are exceptionally challenging for all evaluated LLMs.LLMs also struggle on summarization without training data, although GPT-4 approaches the fine-tuned state of the art on two of three datasets.
- Evaluation considerations: ZeroSCROLLS also highlights output-format discrepancies: models can produce correct answers that automatic metrics penalize for not matching requested formats.The paper encourages using the benchmark to study instruction understanding, prompt engineering, and evaluation of generated text.
2 Background: SCROLLS
SCROLLS established a reusable benchmark for long-text understanding with single-document input-output tasks and enabled progress in specialized long-sequence modeling. ZeroSCROLLS responds to the continued reliance on task-specific fine-tuning by introducing a zero-shot evaluation setup for general-purpose reasoning models.
- SCROLLS: SCROLLS provides curated, cleaned, and reformatted long-text datasets in a common single-input, single-output format.Examples contain one long document, such as a scientific paper or book.
- SCROLLS: SCROLLS has supported new pretraining objectives, adaptations of short-text models to long sequences, and dedicated long-sequence models pretrained from scratch.These developments span multiple approaches to long-context modeling.
- Motivation for a new setting: Prior SCROLLS methods eventually fine-tune a specialized model for each task, whereas modern general-purpose zero-shot reasoning LLMs motivate a setup that reduces dependence on task-specific fine-tuning.The paper frames this as the need for a new evaluation setting.
3 The ZeroSCROLLS Benchmark
ZeroSCROLLS is a zero-shot benchmark of ten long-text reasoning tasks, combining six adapted SCROLLS datasets with four new tasks and no training data. It uses natural-language prompts, capped test sets, and task-specific evaluations, including two tasks designed to aggregate information across long inputs.
- ZeroSCROLLS contains ten zero-shot tasks, each requiring reasoning over a different type of long text, with at most 500 examples per task.The benchmark includes test sets, small validation sets, and no training data.
- Six tasks are adapted from SCROLLS, while four new tasks add query-based summarization, multi-hop question answering, sentiment aggregation, and chapter-summary sorting.The adapted summarization and question-answering datasets include GovReport, SummScreenFD, QMSum, SQuALITY, Qasper, NarrativeQA, and QuALITY; MuSiQue is added for multi-hop question answering.
- The benchmark covers full-document and query-focused summarization, question answering over papers, stories, scripts, and Wikipedia paragraphs, plus aggregation and ordering tasks.QMSum focuses summaries on meeting topics, while SQuALITY uses guiding questions over stories and MuSiQue requires multiple hops between paragraphs.
- Aggregation: SpaceDigest asks models to estimate the percentage of positive reviews among 50 hotel reviews, using only strictly positive or negative ratings during construction.Reviews with an ambivalent rating of 3 are discarded, and human annotators achieved approximately 98.4% single-review classification accuracy.
- Aggregation: BookSumSort asks models to restore the original order of shuffled chapter summaries from manually selected books, with instances containing 3 to 86 summaries.The dataset contains 500 instances created from four random permutations of each selected summary list.
- Prompting: Each instance receives a task instruction and desired output format without demonstrations, using prompts structured around instruction, context, query, and response.Contexts may be trimmed when they exceed a model’s window, with an explicit notice appended; prompts are adapted for chat models such as Claude.
4 Evaluating State-of-the-Art LLMs
ZeroSCROLLS evaluates open-source, closed, and task-specific models across ten long-text understanding tasks, revealing strong overall performance from GPT-4 but persistent weaknesses in summarization and information aggregation.
- Evaluation Setup: The benchmark compares open-source and closed models using greedy decoding, task-specific baselines, naive baselines, and human performance where available.Open-source models use a maximum input length of 8,192 tokens, while CoLT5 uses up to 16,384 tokens.
- GPT-4 achieves the highest overall average score at 41.7, followed by Claude at 39.1.
- Summarization: GPT-4 reaches the highest ROUGE scores on all four summarization datasets, but remains below fine-tuned CoLT5, especially on GovReport.On GovReport, CoLT5 scores 41.0 compared with GPT-4's 26.3.
- Question Answering: 89.2 on QuALITY places GPT-4 close to human performance at 93.5, while other models lead on Qasper, MuSiQue, and NarrativeQA.Flan-UL2 leads Qasper and MuSiQue, while Claude's NarrativeQA F1 is five points above GPT-4's.
- Aggregation: Only Claude and GPT-4 significantly outperform the naive baseline on SpaceDigest, while only GPT-4 surpasses it on BookSumSort.These tasks require aggregating or comparing information across long sequences.
- Model Size and Input Length: Increasing model size generally improves performance across almost all tasks, and longer inputs generally help both Flan-T5 and Claude.Claude gains almost 3 average-score points when increasing input length from 4k to 8k tokens.
5 Analysis
The analysis shows that automatic scores can understate answer quality when models violate requested formats. Format discrepancy affects multiple models and tasks, while GPT-4 remains the top leaderboard model overall.
- GPT-4 has the highest ZeroSCROLLS leaderboard score, although other models score higher on several question answering tasks.
- Manual annotation finds GPT-4 performs better than Claude on NarrativeQA and better than Flan-UL2 on Qasper, despite lower F1 scores in the reported comparisons.
- 191 of 200 Claude answers matched the correct format, compared with 71 of 200 GPT-4 answers, causing GPT-4 to receive harsher F1 penalties.
- Format discrepancy is widespread: Claude produces long QMSum answers, Flan-UL2 produces long SummScreenFD summaries, and all models produce short GovReport summaries.
6 Conclusion
ZeroSCROLLS benchmarks zero-shot understanding of long texts, including tasks that require contextualizing and aggregating information across documents. Evaluation identifies strong long-context models but leaves aggregation as a major open challenge.
- ZeroSCROLLS is a benchmark for zero-shot natural language understanding over long texts.
- The benchmark compares LLMs on naturally long inputs and tasks requiring contextualization and information aggregation across multiple documents.
- 30 of 200 NarrativeQA cases led GPT-4 to say the trimmed context lacked an answer, compared with 5 such cases for Claude.
- GPT-4 and Claude are the best-performing models overall, while Flan-UL2 is powerful on long-context question answering.
- The two new aggregation tasks are particularly difficult for contemporary LLMs.
7 Limitations
The paper identifies evaluation and prompting limitations in zero-shot long-text benchmarking. Automatic metrics can undervalue semantically equivalent outputs, while prompt and model changes may affect results.
- ROUGE and F1 can assign low scores to semantically equivalent zero-shot generations with different wording or answer lengths.
- The benchmark uses common prompt templates across models, although model-specific prompts and chain-of-thought prompting may improve performance.
- Long-range models, alignment methods, decoding algorithms, and prompting techniques can change the state of the art over time.
A Prompts
The appendix provides simple, natural, explicit prompts for summarization, question answering, and aggregation tasks. Prompts include task text placeholders and chat-model-specific formatting adjustments.
- Summarization: The summarization prompts are designed for simple, natural, and explicit task instructions.
- Prompt templates use braces as placeholders for the text of every example.
- Question answering and aggregation: Question answering and aggregation prompts are presented separately and include an additional instruction for chat models.