Source-linked AI summary
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner
TL;DR
Existing information-seeking QA datasets largely emphasize generic factoids, while paper-grounded systems need data for questions requiring evidence distributed across full research papers. QASPER provides such a dataset and evaluates document-level models, finding substantial gaps from human performance. Its scope is computational linguistics and English-language research papers.
Problem
Existing information-seeking QA benchmarks focus on generic factoid questions rather than the broader, paper-specific information needs of readers.
Method
QASPER collects 5,049 questions over 1,585 NLP papers, written from titles and abstracts and answered from full texts with supporting evidence.
Results
The best models trail humans by 27 F1 points on answering questions from entire papers and by 32 F1 points on selecting evidence paragraphs.
Takeaways & Limitations
QASPER provides a test bed for evaluating document-grounded, information-seeking QA research.
Takeaways & Limitations
The dataset is limited to computational linguistics papers, and systems built on it are not expected to work well on non-English research papers.
Abstract
from arXiv · showhide
Readers of academic research papers often read with the goal of answering specific questions. Question Answering systems that can answer those questions can make consumption of the content much more efficient. However, building such tools requires data that reflect the difficulty of the task arising from complex reasoning about claims made in multiple parts of a paper. In contrast, existing information-seeking question answering datasets usually contain questions about generic factoid-type information. We therefore present QASPER, a dataset of 5,049 questions over 1,585 Natural Language Processing papers. Each question is written by an NLP practitioner who read only the title and abstract of the corresponding paper, and the question seeks information present in the full text. The questions are then answered by a separate set of NLP practitioners who also provide supporting evidence to answers. We find that existing models that do well on other QA tasks do not perform well on answering these questions, underperforming humans by at least 27 F1 points when answering them from entire papers, motivating further research in document-grounded, information-seeking QA, which our dataset is designed to facilitate.
1 Introduction
QASPER targets information-seeking questions grounded in academic papers rather than generic factoids. Its questions require document-level reasoning over evidence distributed across full texts, and current models substantially trail humans.
- Existing information-seeking benchmarks emphasize generic factoid questions rather than the full range of information needs.
- QASPER asks follow-up questions about papers after readers see only their titles and abstracts, with answers and supporting evidence drawn from the remaining full text.
- Figure 1 illustrates that answering a question can require combining evidence from three different paper sections.
- 55.5% of questions require evidence from multiple paragraphs, while 13% require tables or figures.
- 27 F1 points separate the best models from humans on answering questions from entire papers, and 32 F1 points separate them on selecting evidence paragraphs.
2 Building the QASPER Dataset
The dataset was built from open-access computational linguistics papers through separate interfaces and annotation stages for question writing, evidence selection, and answer generation.
- Answers: Separate annotators received the full paper, figures, tables, and associated questions before judging answerability and annotating answers.
- The authors filtered S2ORC for open-access arXiv papers with LaTeX sources in computational linguistics and converted them to full text while preserving structure and equations.
- Answers: Answering workers received tutorials and qualification exams because the answering task was considered significantly more complex than question writing.
- Questions: NLP practitioners wrote questions after viewing only paper titles and abstracts, selecting papers through search or receiving random papers.
- Answers: For answerable questions, annotators selected the minimal text, figure, or table evidence and wrote a concise answer with its answer type.
3 QASPER Analysis
QASPER contains mostly paper-specific questions and evidence that often spans multiple paragraphs or modalities. Multiple annotations show strong agreement on answerability, though answer correctness is not uniform.
- Question types: 67% of manually categorized questions are specific to the papers they address, supporting QA over fixed-schema information extraction.
- Answer types: Most answers are extractive, averaging 14.4 words, while abstractive answers average 15.6 words.
- Evidence types: 55.5% of answerable questions with text-only evidence require multiple paragraphs, and evidence may also combine paragraphs with figures or tables.
- Distribution of evidence paragraphs: No frequently used paper section contains a majority of evidence spans, indicating an approximately uniform distribution across sections.
- Inter-annotator agreement: 90% of annotators agree on answerability, and 84.0% agree on whether evidence is text or a figure.
- Correctness: In a sample of multiple-answer questions, 207 answers, or 75.8%, were correct, while 98% of questions had at least one correct answer.
4 Modeling QASPER
QASPER models answer generation and evidence selection over long research papers, using sequence-to-sequence generation with LED and auxiliary evidence supervision. The evaluation defines separate answer and evidence metrics and estimates human performance as a lower bound.
- QASPER asks systems to determine answerability and generate answers from full-paper text, with evidence selection as an auxiliary task.
- Answer-F1 measures answer correctness, while Evidence-F1 measures overlap between selected paragraphs, figures, or tables and labeled evidence.
- Human performance is reported as a lower bound because estimates compare limited annotations and use NLP practitioners rather than expert researchers.
- The model converts extractive, abstractive, yes/no, and unanswerable responses into one sequence-to-sequence answer-generation task.
- LED processes long inputs with local and global attention, supporting sequences up to 16K tokens and encoding 99% of QASPER papers without truncation.
- Evidence extraction adds paragraph classification and global attention over paragraph markers, training answer generation and evidence prediction jointly.
5 Experiments
Experiments evaluate answer generation and evidence selection against human lower bounds using long-document LED models and multiple input settings. Results show that full-paper context matters, performance remains below human estimates, and gold evidence substantially improves answer prediction.
- Experiments evaluate question answering and evidence selection against estimated human performance using multiple annotations.
- Question answering: ∆= +9.5: encoding the entire paper significantly outperforms the best heuristic input, the introduction, on overall Answer-F1.
- Question answering: The LED-large model performs much worse than the LED-base model on the question-answering task.
- Question answering: Table 2 reports LED-base and lower-bound human performance on Answer-F1, including heuristic baselines and evidence-scaffolding variants.
- Answer prediction from gold evidence: Gold-evidence experiments show that large models exceed the human lower bound for span answers and that better evidence selection can close most non-oracle headroom.
- Varying amounts of training: Figure 2 varies training data size and tracks dev-set Answer-F1 and Evidence-F1; performance has not reached a plateau.
- Error analysis: Error analysis of 63 incorrect sampled predictions finds 31 extractive, 24 abstractive, 3 yes, 3 no, and 2 unanswerable gold answers.
6 Related Work
QASPER follows the information-seeking QA paradigm but grounds questions in individual research papers and requires reasoning over their full text. This distinguishes it from datasets built around known answers, open-domain questions, or abstract-level biomedical QA.
- Information-Verifying QA: Information-verifying QA datasets typically have question writers who already know the answers, unlike genuinely information-seeking datasets.Academic-paper QA datasets often automatically construct questions from entities, relations, or structured knowledge resources.
- Information-Seeking QA in General Domain: Natural Questions and TyDiQA are not grounded in contexts, whereas QASPER questions follow from a paper’s title and abstract.The QASPER setup primes readers to ask detailed, paper-specific questions requiring deeper understanding of the full context.
- Information-Seeking QA in General Domain: QASPER differs from QuAC by not imposing an entity-focused question-collection procedure.Both datasets use a similar information-seeking collection method, but QASPER targets questions about research papers more broadly.
- Domain-Specific Information-seeking QA: QASPER is the first information-seeking QA dataset in computer science and uses full papers rather than only abstracts.It contains over 5,000 questions, compared with 1,000 for PubMedQA and 3,200 for BioASQ.
7 Conclusion
The paper concludes that QASPER turns natural follow-up questions about NLP papers into a challenging full-document QA benchmark. Its evidence often spans paragraphs, figures, and tables, while model performance remains substantially below estimated human performance.
- 7 Conclusion: QASPER is designed to evaluate document-grounded QA research using natural questions asked after reading paper titles and abstracts.The collection setup connects question formulation with readers’ initial exposure to each paper.
- 7 Conclusion: QASPER questions require evidence from multiple paragraphs and/or figures and tables within papers’ full text.This structure makes the dataset a test-bed for document-grounded QA research.
- 7 Conclusion: QASPER’s empirical results leave substantial room for improvement relative to estimated human performance.The conclusion presents this gap as evidence that the benchmark remains challenging for current systems.
Ethical Considerations
The dataset was constructed from openly licensed arXiv papers and collected annotator background information without releasing personally identifiable information. Its arXiv-based source makes it almost entirely English-language, limiting expected performance on non-English research papers.
- Ethical Considerations: QASPER uses arXiv papers released under CC-BY-* licenses to adhere to copyright requirements.The papers were identified through Unpaywall as part of the S2ORC dataset construction.
- Ethical Considerations: QASPER is almost entirely English-language because its papers come from arXiv.This source choice defines an important coverage boundary for the dataset.
- Ethical Considerations: Systems built on QASPER would not be expected to work well on non-English research papers.The stated limitation follows directly from the language distribution induced by the paper source.
- Ethical Considerations: Annotator background information was collected, but personally identifiable information was not released except as needed for payment.The paper states that such information was not collected without permission for other purposes.