WIQA: A dataset for "What if..." reasoning over procedural text

1 Introduction

• WIQA addresses the lack of a dataset for testing how perturbations propagate through a procedure’s chain of influences.The task asks whether a change in one part of a process affects another.Source: §1 · p. 1
• WIQA contains 40.7K questions covering 379 process paragraphs, built from 2,107 crowdsourced influence graphs.Source: §1 · p. 1
• Questions are multiple-choice judgments of whether one qualitative change causes a target change, has the opposite effect, or has no effect.The task provides the paragraph but not its influence graph; changes typically concern paragraph entities and events.Source: §1 · p. 2
• The strongest model remains 23% behind human performance, and the gap widens for indirect and out-of-paragraph effects.The paper presents WIQA as a challenge for counterfactual textual reasoning over procedural text.Source: §1 · p. 2

2 Related Work

• Procedural-text datasets have targeted tracking entity states, whereas WIQA asks how states might change when a perturbation is introduced.Source: §2 · p. 2
• Qualitative reasoning has studied effects of qualitative change primarily through formal models, and counterfactual reasoning has likewise used formal frameworks.Source: §2 · p. 2
• WIQA treats what-if reasoning over text as a mixture of reading comprehension and commonsense reasoning, creating a new NLP challenge.Source: §2 · p. 2

3 Dataset Construction

• An influence graph is a directed graph whose nodes describe changes and whose edges mark positive or negative influence.Nodes can represent changes to events inside or outside the paragraph, and graph traversal captures indirect effects.Source: §3 · p. 2
• Workers populated a five-question template for paragraphs drawn from ProPara’s 377 training paragraphs, yielding 2,107 influence graphs.Later questions were automatically constructed from earlier answers.Source: §3 · pp. 2–3
• Graph paths generate correct-or-opposite questions from the product of traversed edge polarities; out-of-paragraph changes from other graphs supply no-effect questions.Questions are labeled by path length and by whether their starting node is in- or out-of-paragraph.Source: §3 · p. 3
• Five workers answered each test question, and retaining questions with at least three matching answers kept 88% of them.Worker agreement measured by Krippendorff’s alpha was 0.6.Source: §3 · p. 3
• The 40,695-question dataset approximately balances question categories and answers, and its partitions separate paragraphs by topic.Source: §3 · p. 3
• The authors also generated path-based answer explanations for correct and opposite answers, but not for no-effect answers.They removed explanations with contradictory worker annotations or no majority decision and made the database available for possible future explanation tasks.Source: §3 · p. 3

4.1 Models

• The baselines include majority-label prediction, a rule-based polarity heuristic, AdaBoost over change/effect features, and Decomposable Attention recast as entailment.The polarity heuristic assigns more/less directionality using a small lexicon and defaults to a random class when it cannot assign polarities.Source: §4.1 · pp. 3–4
• BERT is fine-tuned on paragraph, question, and answer-option inputs, with an ablation that omits the paragraph at training and test time.Human performance was estimated from three independent answers to 100 test questions; agreement was Krippendorff’s alpha 0.908.Source: §4.1 · p. 4
• Baseline accuracy declines as hop count increases, and declines faster for the no-paragraph version.Figure 3 reports this pattern for the best baselines.Source: Figure 3 · p. 4

5.1 Prediction Accuracy

• 73.8 accuracy for the strongest model is over 20 points below human performance, and out-of-paragraph effects are 37 points behind human performance.Providing the paragraph improves the score by 10 points over the version without it.Source: §5.1 · p. 4

5.2 Predicting Indirect Effects

• Indirect 2/3-hop effects are harder to predict than direct 1-hop effects, suggesting that reliable prediction may require reasoning along influence chains.The text notes that indirect effects are less likely to be explicitly stated in corpora or embedded in pretrained language models.Source: §5.2 · p. 5

5.3 Consistency

• Models are far from transitive and disjunctive consistency, whereas humans make substantially more consistent predictions.The paper suggests global consistency constraints may improve results.Source: §5.3 · p. 5, Figure 4 · p. 5

5.4 Linguistic and Semantic Phenomena

• ≈65% of changes use qualitative language and ≈45% involve commonsense; lexical mismatches, negation, and juxtaposed polarities add further challenges.The reported phenomena overlap; lexical matching occurs in ≈15% of in-paragraph changes, negation in ≈6%, and juxtaposed polarities in ≈3%.Source: §5.4 · p. 5

• WIQA introduces a large-scale dataset for testing whether perturbation effects in procedural text can be predicted.Source: §6 · p. 5
• Language models’ predictions are over 20 points worse than humans and particularly erroneous about indirect, multi-hop effects.Source: §6 · p. 5
• The dataset is presented as a new challenge and resource for the research community.Source: §6 · p. 5

• Paragraph context improves BERT accuracy more on Pollination, blood, mountain, and evaporation than on Igneous rocks, plant crops, solar eclipse, and DNA replication.Source: Appendix A · p. 6
• Some topics are harder for BERT because answering their questions requires additional paragraph context.Source: Appendix A · p. 6
• The topic-wise comparison reports better accuracy as the paragraph-context BERT model reads more paragraphs in context.Source: Table 3 · p. 6

• Workers construct influence graphs progressively by answering five questions, with later questions built from earlier answers.Source: Appendix B · p. 6
• After the first question is answered, two nodes of the partial graph are filled.Source: Appendix B · pp. 6–7
• The annotation starts with a blank influence graph showing its basic structure.Source: Figure 5 · p. 7
• Annotators can validate their answers by examining the graph that emerges as they answer questions.Source: Figure 6 · p. 7

• The displayed influence graphs are examples from evaporation, flashlight, lungs, and minerals, spanning natural, human-body, and mechanical processes.Source: Appendix C · p. 7, Figure 8 · p. 8, Figure 9 · p. 9
• The Mechanical Turk interface presents a paragraph and five questions whose answers produce an influence graph.Source: Figure 7 · p. 8