IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions

• IfQA fills an open-domain QA evaluation gap with over 3,800 questions whose counterfactual presuppositions are expressed through “if” clauses.The questions combine causal-inference questions with factual text sources comprehensible to lay readers.Source: §1 · p. 1
• Answering a counterfactual question can require retrieving a current fact, updating it under the presupposition, finding a relevant alternative, and comparing the outcomes.For the Mount Everest example, the process lowers its height from 8,848 to 8,548 metres, retrieves K2’s 8,611-metre height, and compares them.Source: §1 · pp. 1–2
• The paper evaluates both closed-book models, which answer without external evidence, and open-book retriever-reader models that fetch documents before answering.The examples include chain-of-thought prompting with GPT-3, RAG, and FiD.Source: §1 · p. 2
• Nearly 35% of examples lack retrieved gold passages; FiD scores 50% F1 even with a gold passage, while retrieval plus a large-model reasoner reaches 51% F1.The combined system achieves the best results reported in the introduction.Source: §1 · p. 2

2.1 Open-domain Question Answering

• IfQA combines a counterfactual presupposition with retrieved factual knowledge, unlike existing open-domain QA formulations that assume factual presuppositions.The distinction is framed against benchmarks for single-hop, multi-hop, ambiguous, multi-answer, multimodal, and real-time questions.Source: §2.1 · pp. 2–3
• Mainstream open-domain QA uses retriever-reader architectures, with research improving retrieval or reading components (Zhu et al., 2021b; Ju et al., 2022).Approaches described include sparse and dense retrieval, reranking, and generative readers.Source: §2.1 · p. 3

2.2 Counterfactual Thinking and Causality

• Prior counterfactual NLP work includes training with counterfactual samples and tasks such as revising stories or predicting procedural perturbation effects, but not open-domain QA.Counterfactual-sample training is described as orthogonal to incorporating counterfactual presuppositions into model decision-making.Source: §2.2 · p. 3

3.1 Dataset Collection

• The dataset was crowdsourced from 188 workers, who were paid $15–$20 per hour and limited to 30 questions each; its protocol used passage selection, question-answer annotation, and validation.The three phases are described as extracting candidate Wikipedia passages, collecting question-answer pairs, and validating their correctness and quality.Source: §3.1 · p. 3

3.1.1 Question and Answer Annotation

• Workers selected at least 10 passages from a random sample of 20 and annotated questions on passages screened for causal events, especially past-tense events.Workers supplied answers and could add other valid answers; 20.6% of questions came from free-form annotation.Source: §3.1.1 · pp. 3–4

3.1.2 Question and Answer Verification

• Reviewers checked readability and passage relevance, whether questions were understandable without the passage, and answer correctness, revising errors and voting when answers differed.A third worker was added when workers submitted different answers.Source: §3.1.2 · p. 5

3.1.3 Answer Post-processing

• Answer post-processing adds canonical Wikipedia entity titles and normalizes number and date forms alongside the original passage spans.Number normalization follows the unit used in the passage, while dates also receive an ISO 8601 format.Source: §3.1.3 · p. 5

3.2 Dataset Analysis

• Answers are mostly entities (49.7%) or passage spans (75.1%); questions average 22.2 words versus 1.8-word answers, and 11.2% have multiple valid answers.The question length is attributed mainly to the counterfactual presupposition clause.Source: §3.2 · pp. 5–6
• On IfQA-S, BM25 performs better than DPR for retrieval, while FiD-l performs best for end-QA among the retrieve-then-read models.Source: Figure 2 · p. 6

3.3 Dataset Splits

• IfQA-S has 2,401 training examples and 701 each for validation and test; IfQA-F has 200 training examples and divides the remainder evenly between validation and test.Source: §3.3 · p. 6

4.1 Retrieval Corpus

• Wikipedia serves as the retrieval corpus; pages from the dump dated 2022-05-013 were split into disjoint 100-word passages, yielding 27,572,699 million passages across 6,394,490 pages.Source: §4.1 · p. 6

4.2 Comparison Systems

• The comparison includes closed-book GPT-3 variants and open-book systems that retrieve evidence before a reader predicts an answer.The described open-book readers include FiD and RAG; the closed-book systems include CoT and GENREAD variants.Source: §4.2 · p. 6

4.3 Evaluation Metrics

• Retrieval is measured with Recall@K, while end-QA performance is measured with exact match and F1.F1 is averaged over questions using the maximum score across each question’s ground-truth answers.Source: §4.3 · p. 7

4.4 Results and Discussion

• IfQA retrieval is challenged by longer questions and counterfactual–factual mismatches; sparse and dense retrievers reach only about 60% Recall@20.Retrieving more passages increases reader memory costs; GPT-3 variants have a 4096-token input limit.Source: §4.4 · p. 7
• With golden passages in the retrieved set, FiD answered 225 of 540 questions correctly (41.7%), and scored 31.5% on questions requiring complex reasoning.The 540-example subset was drawn under the supervised data split.Source: §4.4 · pp. 7–8
• Chain-of-thought improves closed-book LLM performance on IfQA, but those models still lag behind retrieve-then-read methods.The paper attributes the gap to closed-book models’ lack of non-parametric knowledge.Source: §4.4 · p. 8
• Combining BM25 retrieval with GPT-3 as reader achieves the best performance on IfQA.The pipeline combines retrieved factual evidence with the large model’s reasoning ability.Source: Table 3 · p. 7, §4.4 · p. 8

4.5 Case Study

• In the case study, GPT-3 and chain-of-thought hallucinate release dates, BM25+FiD gives a wrong answer, and retrieval plus an LLM gives the correct answer.The combined system uses factual evidence from retrieval and stronger reasoning ability.Source: §4.5 · p. 8, Table 4 · p. 8

5 Conclusion

• IfQA contains over 3,800 questions, each based on an “if” clause expressing a counterfactual presupposition.Source: §5 · p. 8
• The authors report that existing open-domain QA methods find both retrieval and reasoning on IfQA highly challenging.Source: §5 · p. 8
• The authors conclude that IfQA’s challenges would push open-domain QA research on retrieval and counterfactual reasoning.Source: §5 · p. 8

6 Limitations

• IfQA covers only event-based questions and is not intended for training general open-domain QA models or evaluating their capabilities.This scope limit follows from how the counterfactual presuppositions are created.Source: §6 · p. 9
• Human annotators and heuristic selection of Wikipedia passages may introduce annotator and pattern-oriented biases into the dataset.The authors say their instructions, examples, and diverse annotator sampling cannot completely remove annotator bias.Source: §6 · p. 9
• Large language models evaluated on IfQA may retain biases learned from web text or make biased judgments as a result.Source: §6 · p. 9

Ethics Statement

• IfQA may reflect social, ethical, and regional biases of the workers who created and validated its questions.Source: Ethics Statement · p. 9