Source-linked AI summary
QuALITY: Question Answering with Long Input Texts, Yes!
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, Samuel R. Bowman
TL;DR
Long-document comprehension requires benchmarks whose contexts exceed the short inputs handled by many NLU models. QuALITY addresses this gap with full-passage, multiple-choice crowdsourcing designed to make questions difficult to answer by skimming, but baseline models still trail humans substantially.
Problem
Most NLU models process only a few hundred words, while existing QA datasets often use shorter contexts or rely on excerpts and summaries rather than holistic passage understanding.
Method
QuALITY collects English multiple-choice QA from full articles, using speed and untimed validation to enforce challenging, answerable, and unambiguous questions.
Results
38.1 percentage points separate human annotators from the best model on QuALITY, widening to 42.3 points on QuALITY-HARD.
Takeaways & Limitations
QuALITY provides a long-document QA benchmark where skimming and simple search do not yield high performance, leaving a substantial human–model gap.
Takeaways & Limitations
Performance on QuALITY demonstrates effectiveness on mainstream US English and should not be presumed to generalize to other languages or language varieties.
Abstract
from arXiv · showhide
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).
1 Introduction
QuALITY is introduced to evaluate long-document comprehension with English multiple-choice questions built from full passages. Its passages are substantially longer than typical QA contexts, and its crowdsourcing process targets questions that require broader understanding than skimming or search.
- QuALITY is a multiple-choice QA dataset using English source articles of 2k–8k tokens.
- Writers read full articles and create questions requiring unambiguous answers and information consolidated across multiple text parts.
- 6,737 questions comprise QuALITY, including 3,360 questions in the difficult QuALITY-HARD subset.
- 55.4% is the best reported model accuracy, while human accuracy reaches 93.5% on the full dataset.
2 Data Collection
QuALITY uses iterative crowdsourcing, speed validation, and untimed validation to produce difficult questions that remain answerable and unambiguous. The process combines writer feedback and incentives with checks of difficulty, correctness, and agreement.
- Writing Task: Writers construct 10 four-option questions per passage over multiple rounds, receiving validation feedback and bonuses for questions meeting HARD criteria.
- Speed Validation: Questions become HARD when annotators are collectively unable to select the correct answer under strict time constraints despite requiring full-text understanding.
- Speed Validation: Speed validation tests whether annotators can answer questions while limited to 40 seconds of skimming or keyword search after viewing the question and options.
- Untimed Validation: Untimed validation uses three training annotations and five dev or test annotations to verify that questions are answerable and unambiguous.
- Untimed Validation: Q1 determines dataset inclusion, while Q2 and Q3 provide writer feedback about question quality.
- Annotator Performance: 0.71 is the median Krippendorff’s alpha for reading-comprehension agreement, while individual agreement with the Q1 majority vote is 92.6%.
3 Dataset Information and Analysis
QuALITY contains 6,737 validated questions, with 49.9% in the harder subset defined by failure under speed validation. Human annotators achieve 93.5% accuracy overall, while the dataset includes varied question types and broader-context reasoning demands.
- Dataset construction: 49.9% of the final dataset belongs to QuALITY-HARD, where most annotators answered incorrectly under speed validation.The hard subset is intended to exclude questions solvable by rapid skimming or phrase search.
- Human accuracy: 93.5% human accuracy relative to the gold label falls to 89.1% on HARD and rises to 97.0% on EASY.Annotators marked 98.5% of questions as answerable and unambiguous.
- Question types: QuALITY includes many “how” and “why” questions, but measured difficulty does not vary by question type.The question-type analysis compares HARD and EASY subsets using automatically assigned categories.
- Reasoning strategies: Many questions require best-description reasoning, explanations, or interpretation and symbolism, which rely on broader passage context than factual questions.The qualitative analysis manually annotates a random subset of 500 questions, with disagreements resolved by discussion.
4 Baseline Experiments
The experiments compare long-context models, extraction-based systems, and training regimes for multiple-choice QA on QuALITY. DeBERTaV3-large with DPR extraction reaches 55.4%, but the best model remains 38.1 points below humans on the full test set and 42.4 points below on HARD.
- Results: 38.1 points separate the best model from human performance on the full test set, increasing to 42.4 points on QuALITY-HARD.All results in Table 6 fall below human performance.
- Training data: RACE→QuALITY training outperforms RACE-only training in most cases, while both substantially outperform training on QuALITY alone.The authors attribute this pattern likely to QuALITY’s small training set and suggest useful knowledge transfer from RACE.
- Models and extraction: DPR-based extraction almost always performs best, while DeBERTaV3-large consistently outperforms the other tested model configurations.Extraction selects relevant sentences for short-sequence QA models, trading broader context for compatibility with stronger pretrained models.
- Baselines: 43.3% accuracy is achieved by the best question-only baseline, only 12.1 points below the corresponding DPR-excerpt model.The authors note that this small margin may indicate ineffective use of input contexts.
- Results: 78.3% is reached with oracle-answer extraction, yet performance remains 15 points below humans.This result indicates that extracting relevant excerpts alone is insufficient for solving QuALITY questions.
5 Related Work
Prior QA datasets either use short contexts, retrieve supporting documents after question writing, or rely on summaries and free-form generation, limiting direct evaluation of long-context comprehension.
- TriviaQA and SearchQA add documents after writing question-answer pairs, so most questions can be answered from short retrieved contexts.
- HotpotQA, QAngaroo, and ComplexWebQuestions target multi-hop reasoning across multiple paragraphs.
- NarrativeQA uses entire books and scripts but crowdsources questions from Wikipedia summaries and uses free-form generation.
- Existing multiple-choice QA datasets generally use contexts shorter than 500 tokens, unlike QuALITY.
- High-quality long-document QA requires a tractable crowdsourcing pipeline, with training, qualification, and iterative feedback identified as effective strategies.
6 Conclusion
QuALITY is a human-crowdsourced and validated long-document QA dataset designed to make questions answerable, unambiguous, and resistant to skimming. Baseline models remain substantially below human performance, leaving a large gap for long-context NLU research.
- The dataset is crowdsourced and validated to ensure questions are answerable, unambiguous, and challenging.
- QuALITY-HARD comprises half the dataset and contains questions unanswerable under tight time constraints, limiting the usefulness of skimming and simple search.
- 38.1 percentage points separate human annotators from the best baseline model on QuALITY.
- 42.3 points separate human annotators from the best baseline model on QuALITY-HARD.
- Research targeting this gap may expand the scope of texts on which effective NLU systems can be applied.
Ethical Considerations
The dataset’s contributors were recruited primarily from US-based and relatively privileged, educated populations, constraining the demonstrated scope of system effectiveness.
- QuALITY primarily represents mainstream US English rather than other languages or language varieties.
- Question writers were recruited through Upwork, including writers, teachers, and people with college-level degrees in relevant fields.
- The project used a qualification task and paid interview process to select writers and annotators with reading-comprehension expertise.
- Of 22 hired writers, 15 had relevant college degrees and 11 had taught high-school or college-level English or literature.
A.1.2 Writing Task
The writing and validation process combines incentives, feedback, and qualification checks to produce difficult but answerable multiple-choice questions grounded in whole passages.
- Writers earn $12.50 per passage plus a $1.20 bonus for questions meeting answerability, agreement, and quality criteria.
- Validation feedback tells writers how much of a passage was needed to answer each question and encourages difficult distractors.
- Writers are excluded from future rounds when too few questions meet the project’s bonus or validation criteria.
- Writers create 10 multiple-choice questions with four answer options per page and receive prompts covering motivations, causes, definitions, summaries, and counterfactuals.
- Writers may skip passages that are difficult to read, highly technical, culturally unfamiliar, or uninteresting.
- Annotators qualify through a roughly 3000-word passage task requiring accuracy on unambiguous questions and identification of ambiguous ones.
A.2.2 Speed Validation
Speed validation tests whether annotators can answer long-passage questions within 45 seconds, encouraging skimming and keyword search rather than full reading. Performance data also supports filtering questions for the HARD subset.
- Catch questions: Catch questions are designed to be answerable quickly and detect random guessing rather than measure long-document comprehension.Examples can be solved with a single ctrl+F search or by rejecting clearly improbable options.
- Task procedure: Annotators receive 10 questions and have 45 seconds to answer each after viewing the article.The article is hidden until the annotator finishes reading the question and options; the page hides it again with five seconds remaining.
- Annotator performance: 83.8% average accuracy on catch questions indicated that most workers could find a correct answer when one was readily available.Annotators consistently scored above the 25% chance rate, and workers below 50% accuracy in a round were removed from future rounds.
- Annotator performance: 48.2% overall accuracy on writers’ questions rose from 39.5% in the first round to 58.4% in the final round.The authors attribute the improvement as likely reflecting new strategies for searching for answers.
- HARD subset: Questions answerable by at least three of five annotators in the speeded setting were excluded from QuALITY-HARD.This criterion uses speed-validation performance to identify questions that remain difficult under time pressure.
- Untimed validation: Main-data annotator agreement with gold labels was 91.2%, with workers required to maintain at least 75% accuracy per round.The process also re-posted passages when an annotator disagreed with the writer’s labels on at least 40% of questions.
B Data
The dataset analysis broadens coverage beyond simple fact retrieval by examining question forms and defining reasoning categories adapted to QuALITY’s diverse questions. Some source types were excluded when writers could not reliably create difficult questions.
- Source data: Switchboard conversations were discarded because writers found them difficult to use for creating challenging questions.Writers cited short conversations, simple everyday topics, and fewer complex details or plot twists than other sources.
- Question types: QuALITY questions are analyzed by question type and split between HARD and EASY subsets.The analysis includes examples, with most “other” questions using finish-the-phrase formats and many yes/no questions requiring multi-part reasoning.
- Reasoning types: The analysis defines reasoning type by the category of content needed for the answer or the strategy required to select it.Examples include person questions and symbolism/interpretation questions that require extrapolating beyond explicit statements.
- Reasoning types: QuALITY uses 15 reasoning categories adapted from NarrativeQA, and nearly a third of questions receive two or more categories.The categories are not mutually exclusive because questions can require multiple kinds of reasoning.
- Reasoning types: The category definitions cover forms including description, why/reason, symbolism/interpretation, how/method, event, person, relation, and finish the phrase.These categories distinguish the information or reasoning strategy needed to select the correct option.
C More Details on Analysis
The analysis tests whether lexical overlap can identify correct answers in QuALITY. Although correct options sometimes fully overlap with the article, competing incorrect options limit this heuristic.
- Lexical overlap: 26.6% accuracy is achieved by a baseline that selects answers using lexical overlap with the article.The heuristic compares correct-option overlap with the maximum overlap among incorrect options.
- Lexical overlap: Correct options have a higher fraction of complete lexical overlap with the article, but incorrect options for the same question may also completely overlap.Figure 7 compares the correct option with the incorrect option having the highest overlap, while Figure 8 shows normalized distributions for all options.
D.2 Training
The training setup uses standard baseline tuning and trains the main models on QuALITY, with optional intermediate training on RACE. Warmup covers 10% of the full training steps.
- Training setup: Hyperparameters for baseline tuning are reported in Tables 12 and 13.The supplied passage identifies the tables as the location of the full tuning settings.
- Training setup: Warmup is set to 10% of the full training steps.This setting applies to the described training procedure.
D.3 Results
This section presents development-set results, oracle accuracy results, and fine-tuning hyperparameters for QuALITY and RACE.
- Table 10 reports results on the QuALITY development set.
- Table 11 reports oracle accuracy on the full QuALITY development set and the QuALITY-HARD subset.The table uses correct answers as queries to retrieve relevant excerpts.
- Tables 12 and 13 list hyperparameters for fine-tuning models on QuALITY and RACE, respectively.