Source-linked AI summary
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, Jonathan Berant
TL;DR
Existing multi-hop QA benchmarks often state the reasoning steps explicitly, leaving a gap for evaluating implicit strategy inference and creating diverse questions without shortcuts. StrategyQA addresses this with a crowdsourced benchmark, an elicitation and filtering pipeline, and decomposition-plus-evidence annotations. It contains 2,780 examples spanning diverse strategies; humans perform well, while strong systems struggle and retrieval remains a central challenge.
Problem
Existing multi-hop QA benchmarks commonly make reasoning steps explicit, while eliciting creative, diverse implicit questions without context-based shortcuts remains difficult.
Method
StrategyQA combines random Wikipedia-term priming, controlled annotator participation, adversarial model filtering, and per-question decomposition and Wikipedia evidence annotations.
Results
StrategyQA contains 2,780 questions covering diverse domains and reasoning operations; humans perform well, but strong systems struggle with its combined retrieval and QA challenge.
Takeaways & Limitations
The benchmark provides decompositions and evidence annotations for studying implicit multi-hop reasoning, where retrieval and decomposition quality remain important to model performance.
Takeaways & Limitations
The benchmark assumes Wikipedia provides correct content and uses it as the answering corpus.
Abstract
from arXiv · showhide
A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of $\sim$66%.
1 Introduction
StrategyQA targets short, implicit multi-step questions whose reasoning strategy must be inferred rather than read directly from the question. Its collection pipeline promotes creative, diverse questions and filters reasoning shortcuts, while annotations provide decompositions and evidence; humans perform well, but strong systems struggle.
- Motivation: StrategyQA defines strategy questions as requiring implicit multi-step reasoning, unlike benchmarks whose questions explicitly specify the reasoning process.For example, “Did Aristotle use a laptop?” requires inferring temporal comparison.
- Challenges: Implicit questions make context retrieval difficult because little lexical overlap connects the question to its supporting evidence.Their short wording also reduces opportunities to exploit language shortcuts.
- Data collection challenge: Crowdsourcing must elicit creative questions independently of context while covering diverse reasoning strategies rather than repetitive patterns.Showing workers full contexts can limit creativity and create lexical overlap that enables shortcuts.
- Data collection: The annotation pipeline primes workers with random Wikipedia terms, limits each annotator’s output, and continuously adversarially filters questions during collection.These techniques aim to increase diversity, creativity, and difficulty while preventing recurring patterns.
- Findings: STRATEGYQA spans diverse knowledge domains and logical operations, combining retrieval and QA challenges that strong systems still struggle to solve despite good human performance.The analysis specifically mentions domains such as physics and geography and operations such as number comparison.
- Benchmark contribution: 2,780 questions form the STRATEGYQA benchmark, with each question annotated by a reasoning decomposition and per-step evidence paragraphs.The benchmark is designed as the first dataset for implicit multi-step QA and its dataset and code are publicly available.
2 Strategy Questions
Strategy questions are definitive Boolean multi-step questions whose reasoning steps are implicit and answerable from a document corpus. StrategyQA represents those steps as decompositions with evidence, distinguishing retrieval from logical operations and allowing reasoning beyond predefined patterns.
- Structure: Each strategy question comprises at least two single-step questions, which may retrieve facts from text or apply logical operations to earlier answers.A single-step question can be answered from a short corpus fragment or compute over previous results.
- Implicitness: Implicit questions require introducing new content words to describe their reasoning process, whereas explicit decompositions can be written largely from the question’s vocabulary.This vocabulary-based rule is a practical characterization rather than a precise lexical-overlap definition.
- Definition: Strategy questions are multi-step Boolean questions with implicit reasoning and a definitive answer reachable from a corpus.The paper assumes Wikipedia is the corpus and limits outputs to yes/no answers.
- Decomposition: The decomposition is a sequence of steps whose final step returns the answer, with special references serving as placeholders for earlier results.Each step corresponds to a single-step question needed to compute the answer.
- Decomposition: Unlike explicit multi-step QA, strategy decompositions may use any tokens needed to describe implicit reasoning, making decomposition substantially harder.The paper contrasts this flexibility with prior decompositions restricted to a small vocabulary derived from the original question.
- Annotations: Each dataset example contains a strategy question, its decomposition, and supporting evidence for each decomposition step.Evidence paragraphs verify retrieval steps, while operation steps are annotated separately.
3 Data Collection Pipeline
STRATEGYQA uses a multi-stage crowdsourcing pipeline to create implicit strategy questions, decompose them into answerable steps, and match each step with Wikipedia evidence. The design combines creativity prompts, annotator-population controls, adversarial filtering, and quality checks.
- Pipeline overview: The pipeline separates question writing, question decomposition, evidence matching, and quality assurance into distinct tasks.Workers write a question and required facts, decompose it into steps with expected Wikipedia pages, and then match evidence paragraphs to those steps.
- Diversity and filtering: The collection process limits context overlap, recruits many annotators, and uses model-in-the-loop adversarial filtering to reduce recurring patterns and biases.The approach addresses the risk that long contexts or repeated annotator patterns produce lexical overlap and reasoning shortcuts.
- Creative Question Writing (CQW): Workers receive a Wikipedia term, a short description, and an expected yes/no answer, then write a strategy question and its supporting facts.Term-based priming provides minimal context while encouraging workers to use background knowledge and web-search skills.
- Strategy Question Decomposition (SQD): Decomposers write the steps needed to answer each question and identify expected Wikipedia pages, helping avoid explicit or infeasible decompositions.Facts from question writing provide soft guidance because multiple valid strategies and phrasings may exist.
- Evidence matching and quality assurance: Evidence matchers retrieve Wikipedia paragraphs for each decomposition step, while indexed paragraph selection and worker qualifications constrain annotation quality.The pipeline also uses automatic checks, author review, and inter-task feedback to identify invalid, non-definitive, or unsupported examples.
4 The STRATEGYQA Dataset
STRATEGYQA contains 2,780 verified questions with short, diverse language and decompositions centered around multiple reasoning steps and retrieved evidence. Evaluation indicates broad strategy coverage, substantial evidence support, and strong but imperfect human performance.
- Dataset statistics: 2,780 verified strategy questions remain after 55 of 2,835 collected questions are marked incorrect during decomposition.The dataset uses an annotator-based split, and the question-writer train and test populations are disjoint.
- Dataset statistics: Questions are shorter than 10 words, have a roughly balanced yes/no answer distribution, and use diverse language whose most common trigram appears in roughly 1% of examples.Compared with HOTPOTQA and COMPLEXWებულQUESTIONS, STRATEGYQA has shorter questions and lower top-trigram concentration.
- Reasoning complexity: Most questions require at least three reasoning steps, with decomposition and evidence distributions centered around three steps and two evidence paragraphs.A considerable portion requires more steps and paragraphs, while about two steps typically involve retrieving external knowledge.
- Question quality: Expert review found 81% of sampled questions valid multi-step implicit questions, with 82% implicit and 95.5% multi-step.Experts also agreed with the provided answers in 94% of sampled cases, suggesting that the questions generally have clear answers.
- Evidence quality: 88.3% of questions are fully covered by matched evidence, and aggregating three workers yields sufficient evidence for all retrieval steps in 88% of questions.At the step level, 93% of matched paragraphs are valid; evidence matching can also succeed when one worker’s match is compensated by another’s.
- Reasoning skills and human performance: STRATEGYQA spans diverse domain-related and logical strategies, while human analysis reports high accuracy and identifies evidence-finding difficulty as the main source of failure.The benchmark’s category combinations are broader than those reported for HOTPOTQA in the cited comparison.
5 Experimental Evaluation
Experiments evaluate pretrained models, retrieval, and decomposition generation on STRATEGYQA, finding that retrieval and implicit decomposition remain challenging while gold information improves performance.
- Strategy QA performance: 53.9 → 63.6 accuracy shows that training without context helps over the majority baseline, but remains far below human performance.The model can answer some questions using pretrained knowledge and auxiliary reasoning datasets without retrieval.
- Strategy QA performance: 70.7 accuracy with gold paragraphs and 72.0 with gold decompositions plus retrieval show that high-quality context and decomposition support improve reasoning.The combined oracle setting provides the strongest reported performance in this comparison.
- Strategy QA performance: 63.6 accuracy for question-based retrieval exceeds gold-decomposition retrieval at 62.0 and predicted-decomposition retrieval at 61.7.Retrieval with predicted decompositions performs even worse than question-based retrieval.
- Retrieval evaluation: Low retrieval performance partially explains why retrieval models do not improve over ROBERTA*∅, while gold decomposition retrieval substantially outperforms question-based retrieval.Predicted decomposition retrieval does not improve retrieval over question-based retrieval, indicating that better decomposition models are needed.
- Retrieval evaluation: BM25 often fails on synonyms, missing entities represented by reference tokens, and complex decomposition language, reflecting limited word overlap with evidence.The authors suggest neural retrieval may benefit STRATEGYQA.
- Human retrieval performance: Human experts achieve Recall@10 of 0.586 with decompositions and 0.513 without, outperforming the IR baselines while still missing many gold paragraphs.Decompositions improve human retrieval, but evidence coverage remains incomplete because multiple valid evidence paragraphs exist and retrieval is difficult.
- Predicted decompositions: BARTDECOMP generates grammatical, well-structured decompositions but often applies an appropriate strategy incorrectly.The lifeboat example illustrates a coherent decomposition built around the wrong assumption that lifeboats dive.
6 Related Work
STRATEGYQA differs from prior multi-hop datasets by eliciting questions from minimal information and requiring implicit decompositions across diverse reasoning strategies.
- Question elicitation: Annotators are primed with minimal information and their own wording rather than an entire context, aiming to support creative questions with less lexical overlap.The approach contrasts with prior context-based question authoring and resembles a recent first-100-characters priming method.
- Implicit reasoning: STRATEGYQA requires implicit decompositions and seeks broader strategy diversity than datasets limited to specific domain strategies.The benchmark targets diversity in reasoning strategies rather than restricting questions to a narrow strategy family.
- Decomposition annotation: Unlike most multi-hop datasets, STRATEGYQA annotates question decompositions during data collection rather than adding them only to existing datasets.Prior work has also trained models to generate decompositions, but generally did not fully annotate them in the original datasets.
7 Conclusion
The paper presents STRATEGYQA as a benchmark for implicit multi-step questions requiring diverse reasoning skills, built with a pipeline that elicits creative questions and annotates their reasoning support.
- Conclusion: STRATEGYQA is presented as the first dataset of implicit multi-step questions requiring a wide range of reasoning skills.Its questions use simple language while covering diverse strategies.
- Conclusion: The annotation pipeline elicits creative questions and provides decompositions into reasoning steps plus evidence paragraphs.These annotations are intended to guide research on implicit multi-hop reasoning.