Source-linked AI summary
ASQA: Factoid Questions Meet Long-Form Answers
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, Ming-Wei Chang
TL;DR
Long-form QA lacks high-quality data and reliable notions of answer quality, especially when questions require detailed explanations. The paper introduces ASQA, a dataset and task for answering ambiguous factoid questions with multi-source summaries, and proposes DR for evaluation. DR agrees well with human judgments, while strong baselines remain far below human performance.
Problem
Long-form QA has less high-quality data and less reliable evaluation than factoid QA, whose questions typically have clearer correctness criteria.
Method
ASQA pairs ambiguous factoid questions with crowdsourced long-form answers, requiring explanations of ambiguity and connections among valid short answers, while DR combines correctness and fluency metrics.
Results
DR correlates well with human judgments, and strong baselines show a large gap from human performance.
Takeaways & Limitations
ASQA provides a long-form QA task with a clear correctness notion and identifies retrieval and summarization as important directions for future work.
Takeaways & Limitations
ASQA has a much smaller sample size than MS-NLG and ELI5, and its disambiguation metrics measure recall of required information.
Abstract
from arXiv · showhide
An abundance of datasets and availability of reliable evaluation metrics have resulted in strong progress in factoid question answering (QA). This progress, however, does not easily transfer to the task of long-form QA, where the goal is to answer questions that require in-depth explanations. The hurdles include (i) a lack of high-quality data, and (ii) the absence of a well-defined notion of the answer's quality. In this work, we address these problems by (i) releasing a novel dataset and a task that we call ASQA (Answer Summaries for Questions which are Ambiguous); and (ii) proposing a reliable metric for measuring performance on ASQA. Our task focuses on factoid questions that are ambiguous, that is, have different correct answers depending on interpretation. Answers to ambiguous questions should synthesize factual information from multiple sources into a long-form summary that resolves the ambiguity. In contrast to existing long-form QA tasks (such as ELI5), ASQA admits a clear notion of correctness: a user faced with a good summary should be able to answer different interpretations of the original ambiguous question. We use this notion of correctness to define an automated metric of performance for ASQA. Our analysis demonstrates an agreement between this metric and human judgments, and reveals a considerable gap between human performance and strong baselines.
1 Introduction
ASQA addresses the data and evaluation challenges of long-form QA by targeting ambiguous factoid questions that require coherent summaries resolving multiple interpretations. It combines a clear correctness notion with long-form retrieval and summarization complexity, and its baselines remain substantially below human performance.
- Motivation: Long-form QA requires detailed explanations, but available data are limited and objective evaluation is difficult for open-ended questions.ELI5 questions can be answered in many ways, contributing to unreliable evaluation criteria.
- ASQA Task: ASQA defines correctness through whether an answer supports users in answering different interpretations of an ambiguous question.This extends factoid-style correctness to long-form answers while preserving a clear evaluation target.
- ASQA Task: The task requires retrieving diverse documents and synthesizing them into coherent summaries that explain ambiguity and relationships among answers.Additional knowledge from Wikipedia helps resolve ambiguity beyond the short answers supplied by AMBIGQA.
- ASQA Dataset: ASQA pairs 6,316 ambiguous factoid questions with crowdsourced long-form answers.The answers explain why questions are ambiguous and connect valid short answers into coherent passages.
- Evaluation: DR combines correctness and fluency-oriented evaluation, and a human study finds high agreement between DR and human judgments.The metric combines disambiguation accuracy with ROUGE-based evaluation.
- Baselines: Strong baselines combine joint passage retrieval with T5-large, yet extensive evaluations reveal a large gap between baseline and human performance.The results identify substantial room for improvement on ASQA.
2 Related Work
Related work spans extractive, generative, and summarization-based QA, but ASQA targets ambiguous questions requiring multi-source information synthesis into one long-form response. Its evaluation uses question-answering-based correctness alongside long-form answer quality.
- Extractive QA: Most extractive QA tasks assume questions have unique answers, whereas AMBIGQA identifies multiple valid short answers for ambiguous questions.ASQA builds on AMBIGQA by adding long-form answer generation.
- Generative QA: In ambiguous factoid QA, systems must combine information from unknown sources rather than extract an answer readily available on a webpage.This motivates ASQA’s focus on generating textual answers.
- Generative QA: Existing generative QA datasets often contain short answers, while ELI5 addresses more challenging open-ended questions with paragraph-length responses.NARRATIVEQA and COQA average 4.7 and 2.7 tokens per answer, respectively.
- Generative QA: ASQA differs from ELI5-style human-in-the-loop approaches by supplying relevant passages and using DR as a potential automated evaluation alternative to reward modeling.The paper presents replacing the reward model with DR as a future direction.
- Summarization: Long-form answer generation can be viewed as query-based multi-document summarization when relevant documents are available.Prior datasets emphasized single-document summarization or small-scale multi-document summarization.
- QA-Based Evaluation: Unlike prior QA-based evaluation methods that generate questions from system text, ASQA evaluates answers to ambiguous questions using their required disambiguations.ASQA also provides relevant passages and reports that DR correlates well with human judgments.
3 ASQA Task and Data
ASQA turns ambiguous factoid questions into a long-form QA task whose answers must cover multiple interpretations with contextual, fluent, attributable explanations. The dataset is built from AMBIGQA and uses structured annotation, training, and support evidence to improve answer quality and evaluation reliability.
- ASQA Task: ASQA asks for a comprehensive paragraph-long answer to an ambiguous question.The task is built on AMBIGQA questions paired with disambiguated questions and unique short answers.
- Annotation Criteria: Good answers must be complete, comprehensive, fluent, and attributable to Wikipedia sources.Completeness covers all valid short answers, while comprehensiveness explains the ambiguity and relationships among answers.
- Annotation Pipeline: Annotators receive the ambiguous question, disambiguations, visited Wikipedia pages, and relevant context passages, then attach additional evidence passages to their answers.The annotation interface supports any number of additional knowledge fields for attribution.
- Instructions and Training: 27 annotators completed self-study, in-person instruction, and an exam before full-time annotation, with in-person training producing consistently high-quality exam answers.The training was designed to reduce annotation noise and address mistakes such as omitting disambiguations.
- ASQA Dataset: ASQA averages 64.8 answer tokens, supports both ROUGE and accuracy evaluation, captures 92% of answer tokens in support documents, and reaches 49.6 inter-annotator ROUGE-L F1 versus 16.9 for ELI5.Excluding AMBIGQA disambiguations, support documents still capture 78% of answer tokens.
- Caveats: ASQA’s agreement is partly contingent on agreement in AMBIGQA because shared disambiguations can inflate agreement, and its smaller sample size limits direct comparison with MS-NLG and ELI5.The authors suggest those larger datasets may serve as pre-training counterparts, leaving that exploration for future work.
4 ASQA Metrics
ASQA combines text-overlap and disambiguation-sensitive metrics because fluent summaries can still provide different information. Its overall DR score uses a geometric mean, while human evaluation separately measures answer coverage and comparative quality.
- ROUGE: ROUGE-L evaluates generated-answer quality by comparing each prediction with both references and retaining the higher score.The metric uses a multireference setup for examples with two annotations.
- Disambiguation Metrics: ROUGE-L alone may miss whether fluent answers provide the required information, motivating metrics for completeness and comprehensiveness.The disambiguation metrics complement text-overlap evaluation.
- Disambiguation Metrics: STR-EM measures the fraction of disambiguations whose corresponding short answers appear exactly in the long answer.The fraction is computed within each question and then averaged across questions.
- Disambiguation Metrics: Disambig-F1 uses a SQUADV2-pretrained Roberta model to estimate how many disambiguated questions can be answered from the generated summary.The model predicts a short answer for each disambiguation from the generated long-form answer, then compares it with the ground truth using token-level F1.
- Caveats: The disambiguation metrics primarily measure recall and can be affected by distractors or hallucinated information, while ROUGE-L penalizes unnecessary extra content.Distractors may reduce the accuracy of the Roberta evaluator and effectively penalize low precision.
- Overall DR Score: The DR score combines Disambig-F1 and ROUGE-L with a geometric mean to penalize maximizing one metric while substantially reducing the other.STR-EM is excluded because it measures the same aspect as Disambig-F1.
- Human Evaluation: Human evaluation measures disambiguation accuracy and pairwise judgments of comprehensiveness, fluency, and overall impression in blind comparisons.Pairwise wins count one point and ties count half a point.
5 Experimental setup
The experimental setup compares naïve, retrieval-only, generative, oracle, and human-generated answers. The baselines vary access to retrieved Wikipedia context, while human references provide contextual upper and lower comparison bounds.
- Baseline Models: The QUESTION model repeats the ambiguous question eight times to establish a lower bound with output length comparable to other systems.
- Retrieval-Only Models: DPR@1 and JPR@1 retrieve one Wikipedia passage, with JPR adding a reranker trained for multiple-answer AMBIGQA questions.JPR was state of the art on AMBIGQA at the time of writing.
- Generative Models: T5 Closed Book answers without Wikipedia passages, whereas T5 Open Book receives top-K paragraphs retrieved by JPR.Both regimes use T5-large, and open-book variants are denoted T5-O-K.
- Oracle Model: ORACLE gives T5-large all disambiguations and additional annotator-identified knowledge, representing a generative model with access to a perfect retriever.Its ROUGE-L is compared against the annotator answer whose additional knowledge was withheld from the input.
- Human Comparisons: Human performance with context uses two reference answers as an upper bound, while human performance without context annotates 200 development questions as a conservative lower bound.The no-context annotators search for disambiguations and required information themselves.
6 Results
ASQA results show that retrieval and concise summarization are both essential: retrieved-passage T5 models outperform closed-book T5, while top-passage retrieval alone is insufficient. The DR metric aligns closely with human judgments, yet substantial headroom remains between the strongest model and human performance.
- Importance of Retrieval: Retrieved-passage T5 models outperform the closed-book setup across automated and human evaluations.T5-O-1 exceeds T5-C by 20.0 HO points and 13.5 DR points; T5-O-5 exceeds it by 15.6 HO points and 18.0 DR points.
- Importance of Retrieval: High-quality retrieval is necessary for strong ASQA performance, unlike in ELI5.Providing oracle passages gives T5 a DR score more than three times that of T5-C.
- Importance of Summarization: Top-passage retrieval alone is insufficient because JPR@1’s stronger disambiguation metrics do not translate into a higher human overall impression than T5-O-1.JPR@1’s STR-EM and Disambig-F1 are higher by 11.4 and 4.6 points, respectively, but its HO is similar to T5-O-1.
- Importance of Summarization: ASQA answers must balance correct information with conciseness: JPR@1 averages 196.8 words, whereas human references average 65 words.Disambiguation metrics do not evaluate conciseness, making summarization important in addition to retrieval.
- Correlation with Human Judgments: DR has the highest correlation with the overall human judgment HO among the automated metrics.The result supports combining ROUGE-L and Disambig-F1 to capture both answer quality dimensions, although its advantage over Disambig-F1 is not statistically significant.
- Remaining Headroom: Human performance remains substantially above the best model: the lower human bound is 42.3 DR and 74.4 HO, versus 33.7 DR and 36.7 HO for T5-O-5.The upper human bound reaches 61.8 DR and 88.9 HO.
7 Analysis
The analysis finds substantial headroom in both retrieval and summarization, while qualitative errors show that retrieved knowledge alone does not ensure high-quality answers.
- Quantitative Analysis: Retrieval-based Disambig-F1 increases considerably as the number of retrieved passages increases.Figure 3 varies the number of retrieved passages and shows this trend for retrieval-based methods.
- Quantitative Analysis: T5 has a large Disambig-F1 gap relative to JPR despite using JPR’s output passages as input.The analysis attributes this gap to information loss or outputs inconsistent with the input.
- Quantitative Analysis: JPR@5 exceeds the lower bound on human Disambig-F1, suggesting summarization improvements alone could reach that performance level.This conclusion concerns the lower bound on human performance, not human-level performance overall.
- Quantitative Analysis: JPR@5 lags the ORACLE model by 14.4 and the human upper bound by 32.6 on the reported retrieval comparison.The analysis also reports DR of 52.5 for ORACLE versus 33.7 for T5-O-5.
- Qualitative Analysis: T5-generated answers exhibit hallucinations, question misunderstanding, and repetitions.Examples include fabricated election facts, incorrect disambiguation, answers omitting the target phrase, and repeated text.
- Qualitative Analysis: Summarization remains an important component requiring substantial improvement to approach human-level performance on ASQA.The analysis identifies summarization as a major source of remaining performance headroom.
8 Conclusion
The paper releases ASQA and uses its correctness notion to define DR, finding that DR agrees with human judgments while strong baselines remain far below human performance. The authors identify retrieval and summarization as future-work priorities and note the scalability challenge of expert evaluation.
- ASQA is a long-form QA dataset with a clear notion of correctness used to define the overall DR performance metric.
- DR correlates well with human judgment, while a large performance gap remains between human performance and strong baselines.
- Strong ASQA performance depends on both high-quality retrieval and summarization, which the paper identifies as important future-work directions.
- Scaling human evaluation remains an open issue because the study uses an expert-based approach.The authors propose transferring the annotation interface to public crowdsourcing while ensuring reliable worker evaluations.
Appendix
The appendix introduces an additional discussion section but provides no substantive content in the supplied passages.
- The appendix begins an additional discussion of several aspects of the work.
A Additional Details on the Annotation Procedure
The appendix describes how context passages were constructed and how annotator quality, feedback, questions, and workload were managed.
- Construction of Context Paragraphs: Context construction supplements each AMBIGQA disambiguation with a context passage when available.
- Construction of Context Paragraphs: The procedure selects Wikipedia paragraphs containing the short answer, computes TF-IDF similarity, and applies an empirical threshold.
- Construction of Context Paragraphs: 45% of disambiguations received non-empty additional context passages.
- Quality Control and Feedback: An automated interface warning checks whether short answers are missing from annotators’ long-form responses.
- Quality Control and Feedback: Continuous monitoring and constructive feedback were used to maintain annotation quality and discourage reduced effort.
- Quality Control and Feedback: Annotators could park uncertain tasks while awaiting answers through an FAQ document.
- Annotators’ Well-Being: Annotators were given up to one hour per question and recommended to answer at least ten questions daily.Average completion time was 15 minutes per question and decreased slightly with familiarity.
B Additional Details on Modeling
This section describes the input formats and parameter choices used for the T5 and ORACLE models. T5-O-K receives an ambiguous question with retrieved passages, while ORACLE incorporates additional annotator-provided knowledge.
- Input Format: Figures 4 and 5 schematize the T5-O-K and ORACLE inputs, whose tags separate conceptual parts and whose blue text denotes instance-specific content.
- Input Format: T5-O-K takes an ambiguous question and K retrieved passages separated by the context tag.
- Input Format: Each T5-O-K retrieved passage includes an info field containing the passage and a wikipage field containing its Wikipedia source title, with passages separated by “|”.
- Input Format: ORACLE has five input parts, including an ambiguous question and annotator-provided additional knowledge pieces.
- Parameter Choice: The evaluations use context lengths of 512, 1024, and 2048 for T5-O-1, T5-O-3, and T5-O-5, respectively, with batch size 8.
- Parameter Choice: T5-C instead uses batch size 16 and context length 128.