Source-linked AI summary
AmbigQA: Answering Ambiguous Open-domain Questions
Sewon Min, Julian Michael, Hannaneh Hajishirzi, Luke Zettlemoyer
TL;DR
Open-domain QA often receives ambiguous questions whose interpretations depend on corpus evidence, while conventional systems assume a single answer. The paper introduces AMBIGQA, which returns all plausible answers with disambiguated rewrites, and constructs AMBIGNQ to study the task. The dataset contains frequent, diverse ambiguity, and the authors provide baseline models that benefit from weak supervision while exposing challenges in answer recall and disambiguation.
Problem
Open-domain QA questions are frequently ambiguous even though existing methods assume a single well-defined answer.
Method
The paper introduces AMBIGQA and AMBIGNQ, pairing each plausible answer with a minimally edited question rewrite and training baseline models with evidence and NQ-OPEN supervision.
Results
AMBIGNQ averages 2.1 distinct answers per question with 89.0 F1 agreement on valid answers, while baseline experiments show effectiveness in learning from the data.
Takeaways & Limitations
AMBIGQA establishes a benchmark for returning multiple answers paired with distinct interpretations of ambiguous open-domain questions.
Takeaways & Limitations
Answer recall and question disambiguation remain difficult, and the evaluation framework can separate answer accuracy from rewrite quality imperfectly.
Abstract
from arXiv · showhide
Ambiguity is inherent to open-domain question answering; especially when exploring new topics, it can be difficult to ask questions that have a single, unambiguous answer. In this paper, we introduce AmbigQA, a new open-domain question answering task which involves finding every plausible answer, and then rewriting the question for each one to resolve the ambiguity. To study this task, we construct AmbigNQ, a dataset covering 14,042 questions from NQ-open, an existing open-domain QA benchmark. We find that over half of the questions in NQ-open are ambiguous, with diverse sources of ambiguity such as event and entity references. We also present strong baseline models for AmbigQA which we show benefit from weakly supervised learning that incorporates NQ-open, strongly suggesting our new task and data will support significant future research effort. Our data and baselines are available at https://nlp.cs.washington.edu/ambigqa.
1 Introduction
AMBIGQA reframes open-domain QA around questions whose interpretations depend on corpus evidence, requiring all plausible answers paired with minimally disambiguated rewrites. The authors introduce AMBIGNQ and baseline models to study this problem.
- Motivation: Over 50% of sampled Google search questions are ambiguous, and ambiguity depends on both the question and corpus evidence.Ambiguity can involve entities, events, or answer types and may emerge only after reading Wikipedia.
- Task: AMBIGQA requires finding distinct, equally plausible answers and generating minimal rewrites that clarify the interpretation behind each answer.The task produces question-answer pairs rather than a single answer.
- Models: The paper presents baseline models for producing multiple answers and disambiguated questions, including components for set-based QA, rewriting, and weak supervision from NQ-OPEN.The baseline work includes an ablation study and qualitative analysis indicating room for future research.
2 Related Work
The paper addresses ambiguity in real-user open-domain factoid questions, which existing benchmarks generally treat as having one answer. It distinguishes AMBIGQA from clarification-question research by returning complete answer sets with unambiguous rewrites.
- Open-domain QA: Existing open-domain QA benchmarks generally assume each question has a single clear answer, despite debatable answers and 49.2% average agreement on NQ-OPEN test data.The paper treats ambiguity as inherent to information-seeking questions posed by real users.
- Clarification Questions: Unlike community-QA clarification work, AMBIGQA answers the original factoid question while handling naturally occurring ambiguity in entity, event, and other references.The task focuses on real-user questions with clear information needs rather than intentionally pre-specified ambiguities.
- Task distinction: Instead of extending an information-seeking session with clarification questions, AMBIGQA provides a complete immediate solution through unambiguous rewrites paired with answers.This formulation differentiates interpretations without requiring another user interaction.
- Question Rewriting: Question rewriting is proposed as a novel task for differentiating distinct answers and resolving ambiguity.The paper contrasts this use with prior rewriting work focused on converting conversational questions into self-contained questions.
3 Task: AMBIGQA
AMBIGQA treats open-domain QA as requiring all plausible answers and minimally edited questions that make each interpretation unambiguous. It also evaluates answer correctness together with the quality of question disambiguation.
- 3.1 AMBIGQA Setup: AMBIGQA outputs distinct, equally plausible answers and minimally edited questions that make each answer uniquely correct.
- 3.1 AMBIGQA Setup: Multiple Answer Prediction requires producing an unknown-sized set of semantically distinct and equally plausible answers.
- 3.1 AMBIGQA Setup: Question Disambiguation generates one minimal question edit per answer, excluding all alternative answers from that interpretation.
- 3.2 Evaluation Metrics: Gold answers are represented as disjoint sets of acceptable strings to accommodate alternative references to the same answer.
- 3.2 Evaluation Metrics: Evaluation assigns each predicted pair a score combining answer correctness with similarity between predicted and reference questions.
- 3.2 Evaluation Metrics: F1ans scores answers alone, F1BLEU uses question-string similarity, and F1EDIT-F1 scores semantic edits relative to the prompt.
4 Data: AMBIGNQ
AMBIGNQ uses Wikipedia-based generation and validation to collect high-recall annotations for ambiguous questions. The resulting 14,042-example dataset reveals frequent, diverse ambiguity and retrieval-intensive annotation behavior.
- 4.1 Data Collection: The two-stage pipeline combines answer generation with validation to address the central challenge of maximizing recall.
- 4.1 Data Collection: Workers search English Wikipedia to find all plausible answers and pair multiple answers with minimal edits distinguishing their interpretations.
- 4.1 Data Collection: 89.0 F1ans agreement between co-authors and workers indicates substantial human agreement despite task difficulty and subjectivity.
- 4.1 Data Collection: 14,042 annotated examples form AMBIGNQ, with over 50% of development and test examples containing multiple question-answer pairs.
- 4.2 Data Analysis: Ambiguity spans entity references, event references, properties, and answer types, with relatively uniform distribution across categories.
- 4.2 Data Analysis: Workers commonly used multiple queries and Wikipedia pages, capturing ambiguity in retrieval rather than relying on a pre-specified evidence document.
- 4.2 Data Analysis: 36% of unigram edits are covered by the 100 most frequent edits, while 69% are covered by the top 1,000.
5 Model
The baseline AMBIGQA model combines multi-passage retrieval, sequence-to-sequence answer generation, question disambiguation, and weakly supervised co-training. Its answer-prediction component improves performance on NQ-OPEN.
- The baseline predicts multiple answers and corresponding disambiguated questions while conditioning on the prompt, answers, and evidence passages.
- SPANSEQGEN retrieves 100 passages, reranks them, and uses a BART-based sequence-to-sequence model instead of predicting from only the top passage.
- The question-disambiguation model generates each edited question using the prompt, target answer, alternative answers, and top evidence passages.
- Weakly supervised co-training iteratively expands fully annotated AMBIGNQ training data with majority-voted silver examples from NQ-OPEN.
6 Experiments
Experiments show that evidence-aware models outperform disambiguation without context, while multiple-answer prediction and question rewriting remain difficult. Co-training improves performance across metrics, but overall results are limited by under-generation, scarce supervision, and distributional differences.
- Main results: 67% classification accuracy, close to the 60% majority baseline, makes DISAMBIG-FIRST significantly worse than other models.Its rewrites can appear reasonable while contradicting the evidence, showing that reading evidence documents is crucial for identifying ambiguities.
- Main results: 49.6 precision and 25.3 recall yield an F1ans of 31.7 on questions with multiple answers.The model outputs shorter sequences on average than the 6.7-token gold sequences, contributing to low recall.
- Main results: SPANSEQGEN achieves reasonable F1ans, but its multi-question performance is lower because predicting all plausible answers is more challenging than predicting one.It also achieves the best F1BLEU and F1EDIT-F1, although both metrics have low absolute values.
- Effect of co-training: Co-training achieves the best performance on all metrics, exceeding the gains from ensembling alone.This result demonstrates the potential of jointly using AMBIGNQ and partial supervision from NQ-OPEN.
- Question disambiguation and limitations: Performance is low overall even with gold answers, and development-test differences are substantial, likely reflecting distributional differences in the original questions.The authors attribute the low performance to sequence-likelihood objectives, limited question-disambiguation annotations, and metrics missing semantically correct alternative phrasings.
7 Conclusion & Future Work
The paper introduces AmbigQA for producing multiple plausible answers and corresponding disambiguated questions, supported by the 14,042-annotation AmbigNQ dataset. Its baseline demonstrates learning from the data, while future work targets ambiguity modeling, answer recall, and question disambiguation.
- Conclusion: AmbigQA produces multiple possible answers to an ambiguous open-domain question and a disambiguated question for each answer.
- Conclusion: AmbigNQ contains 14,042 annotations on NQ-OPEN questions with diverse ambiguity types, many of which are not visible from the prompt alone.
- Future Work: The development setting allows tuning threshold γ, making the evaluation not strictly zero-shot.
- Conclusion: The baseline experiments show effectiveness in learning from AmbigNQ while identifying areas for improvement.
- Future Work: Future work includes modeling event, entity, and retrieval ambiguity and improving answer recall and question disambiguation.
A Data Collection Details
The dataset was crowdsourced through a Wikipedia-based interface with feedback and validation procedures designed to support high-quality question-answer annotations. Workers could skip questions they could not answer appropriately, and human agreement was measured through independent annotation and co-author checks.
- Crowdsourcing details: The collection used Amazon Mechanical Turk and Spacro during February and March 2020, with Google Search API support.
- Crowdsourcing details: Workers used a mobile-view Wikipedia interface and received error or warning messages while generating questions and answers.
- Crowdsourcing details: Workers copied free-text answers from Wikipedia, while generators could skip questions lacking an answer or being ill-formed, subjective, or overly ambiguous.
- Inter-annotator agreement: 76% of questions had annotations that passed validation without changes, while the remaining 24% involved a missed or invalid question-answer pair.
- Inter-annotator agreement: 89.0% average F1ans agreement was obtained between co-authors and workers on 50 sampled validations.
B Discrepancy between development and test in NQ-OPEN
The NQ-OPEN test set differs from development because it was drawn from questions with five annotators rather than one, creating sampling bias toward harder questions. This produces a substantial development-test performance gap under consistent evaluation, although AmbigNQ retains the original split for comparability.
- Dataset construction: NQ-OPEN training and development use questions with one annotator, whereas the test set comes from development data with five annotators per question.
- Sampling bias: A question with a 50% per-annotator answer probability has a 50% chance of entering development but a 97% chance of entering test.
- Evaluation discrepancy: Evaluating only the first reference answer yields a 5–8% development-test performance gap across all models.
- Implications for AMBIGNQ: AMBIGNQ preserves the NQ-OPEN split for consistency with prior work, leaving test-question distribution shift as an ongoing challenge.
C Data Analysis Details
The analysis examines mismatches between AMBIGNQ and NQ-OPEN, model implementation choices, and evaluation considerations. AMBIGNQ answers were correct in most sampled mismatches, while the implementation uses standard retrieval, generation, ensembling, and co-training settings.
- Mismatches with NQ-OPEN: 29.4% of AMBIGNQ development examples omit the NQ-OPEN answer.
- Mismatches with NQ-OPEN: In a sample of 50 mismatches, AMBIGNQ answers were correct in 92% of cases; disagreements included span mismatches, incorrect NQ-OPEN answers, and time dependence.
- Evidence corpus: The evidence corpus consists of English Wikipedia dumps from 2018-12-20 and 2020-01-20, with passages split into chunks of up to 100 words.
- Model implementation: Models use BERTBASE and BARTLARGE, with a dual-encoder retriever and standard PyTorch, PyTorch-Transformers, and fairseq implementations.
- Training and evaluation: Models are trained for 4 epochs on NQ-OPEN or pseudo-labelled data and 15 epochs on AMBIGNQ, selecting checkpoints by development performance.
- Ensemble and co-training: The ensemble selects answers by voting, using C = 5 models before co-training and C = 4 afterward; co-training uses N = 2 iterations and C = 6 models.
E Error Analysis of SPANSEQGEN
This section analyzes SPANSEQGEN’s multiple-answer predictions using 50 development samples, with examples reported in Table 10. It also provides a Table 9 breakdown of cases where the NQ-OPEN answer is absent from AMBIGNQ answers and shows the crowdsourcing interface.
- Figure 3 presents the interface used for crowdsourcing.
- Table 9 breaks down cases where the NQ-OPEN answer is not included among AMBIGNQ answers.
- Table 10 analyzes multiple-answer predictions made by SPANSEQGEN with co-training on 50 development samples.
- The prediction examples omit rewrites and mark differentiation between multiple answers with italicized keywords.