Source-linked AI summary
Break It Down: A Question Understanding Benchmark
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, Jonathan Berant
TL;DR
Complex questions require multiple reasoning steps, but question understanding has largely been handled separately within each task. This paper introduces QDMR and BREAK, showing high-quality decomposition annotation at scale and improved multi-hop question answering.
Problem
Complex questions require operations such as fact chaining and counting, while language understanding has largely been addressed separately within individual tasks.
Method
The paper introduces QDMR, an ordered natural-language decomposition formalism, and collects the BREAK dataset through a crowdsourcing pipeline with constrained step language.
Results
43.3 to 52.4 F1: combining BREAK QDMR structures with an RC model improves open-domain HotpotQA performance, while 97.4% of annotated structures are correct.
Takeaways & Limitations
QDMR provides a question-understanding representation that is agnostic to the information source and can alleviate logical-form annotation in semantic parsing.
Takeaways & Limitations
The paper omits the exact description of how INTERSECTION steps are handled in its open-domain QA procedure.
Abstract
from arXiv · showhide
Understanding natural language questions entails the ability to break down a question into the requisite steps for computing its answer. In this work, we introduce a Question Decomposition Meaning Representation (QDMR) for questions. QDMR constitutes the ordered list of steps, expressed through natural language, that are necessary for answering a question. We develop a crowdsourcing pipeline, showing that quality QDMRs can be annotated at scale, and release the Break dataset, containing over 83K pairs of questions and their QDMRs. We demonstrate the utility of QDMR by showing that (a) it can be used to improve open-domain question answering on the HotpotQA dataset, (b) it can be deterministically converted to a pseudo-SQL formal language, which can alleviate annotation in semantic parsing applications. Last, we use Break to train a sequence-to-sequence model with copying that parses questions into QDMR structures, and show that it substantially outperforms several natural baselines.
1 Introduction
The paper frames question understanding as a standalone task: decomposing complex questions into executable natural-language steps that generalize across information sources. It introduces QDMR and BREAK, and demonstrates their value for multi-hop QA, semantic parsing, and automatic decomposition.
- Problem: Question understanding is presented as a standalone task for decomposing complex questions across information sources.The motivation is that complex questions often require operations such as fact chaining and counting, while humans can decompose them without knowing the answer or source.
- QDMR formalism: QDMR represents complex questions as ordered atomic questions executable in sequence, using operations for selection, attribute retrieval, and aggregation.Its natural-language formulation abstracts away the context needed to answer a question and supports modalities beyond knowledge bases.
- Dataset: BREAK contains 83,978 question decompositions from ten datasets and three modalities, collected through crowdsourcing.The annotation interface trains workers and presents only the question, keeping them agnostic to its original modality.
- Dataset: 97.4% of annotated QDMR structures were validated as correct.The paper also reports high annotation consistency, supporting the quality of the collected representations.
- Applications: Combining BREAK QDMRs with a reading-comprehension model improved HotpotQA multi-hop QA F1 from 43.3 to 52.4.The paper also discusses using QDMRs as a proxy for full logical forms in semantic parsing.
- Applications: A sequence-to-sequence parser with copying achieved 54% accuracy in manual analysis, demonstrating automatic QDMR parsing but remaining below human performance.The parser maps questions into QDMR representations trained on BREAK.
2 Question Decomposition Formalism
QDMR represents a question as an ordered sequence of natural-language decomposition steps, each corresponding to formal query operations and linked as a directed acyclic graph. Its design supports both granular and high-level decompositions across modalities while abstracting away from a particular information source.
- Domain-Agnostic Formalism: By assuming an idealized knowledge base, QDMR abstracts from SQL and can in principle execute over text, images, and other information sources.The abstraction contains the entities and relations expressed in the question rather than restricting the formalism to relational databases.
- QDMR Definition: QDMR defines a question's meaning as an ordered sequence of steps, each associated with one query operator, with the final step returning the answer.Steps may use words from a predefined lexicon or references to earlier results.
- Decomposition Graph: QDMR structures form a directed acyclic graph whose edges encode references from a step to results of previous steps.The graph representation is used to evaluate QDMR parsing models.
- QDMR Operators: The 13 QDMR operators cover selection, attribute retrieval, aggregation, filtering, grouping, comparison, sorting, set operations, Boolean tests, and arithmetic.Each operator is expressed through a natural-language template, signature, and example.
- High-level Decompositions: High-level QDMRs merge adjacent operator nodes, allowing less granular representations such as combining SELECT with PROJECT or FILTER with GROUP and COMPARATIVE.The paper provides both granular and high-level QDMRs for a random subset of reading-comprehension questions and discusses their utility for open-domain QA.
3 Data Collection
BREAK was built through a three-phase pipeline that collected questions, crowdsourced QDMR annotations, and validated worker output across multiple QA tasks.
- Data Collection: The pipeline collected complex questions from existing QA benchmarks, crowdsourced QDMR annotations, and validated the resulting worker annotations.Questions were sampled from ten datasets spanning semantic parsing, reading comprehension, and visual question answering.
- QDMR Annotation: Workers received explanations and examples, then decomposed questions into sequential steps using a restricted lexicon of question words, function words, and reference tokens.The interface showed only the question, keeping workers agnostic to its original modality.
- Question Collection: One exception was COMPLEXWEBQUESTIONS, where annotators paraphrased automatically generated questions.This dataset-specific collection procedure differs from the general sampling process.
- QDMR Annotation: The lexicon contained question words or inflections, 66 predefined function words, and tokens referring to previous-step results.This design aimed to keep annotation language consistent while preserving expressive decompositions.
- QDMR Annotation: Crowdsourcing produced 83,978 examples from 64 distinct workers, with train, development, and test partitions aligned to the source datasets.Development and test samples were constructed without sharing the same context.
- Worker Validation: Qualification tasks required workers to decompose ten examples, and 64 workers qualified by correctly decomposing at least eight.Random validation reviews were then conducted on more than 9K annotations.
4 Dataset Analysis
BREAK contains nearly 84K QDMR decompositions across modalities, with most structures spanning several steps and expert review finding high correctness and mapping accuracy.
- Dataset Composition: BREAK contains 83,978 decompositions, including 60,150 QDMRs and 23,828 high-level QDMRs exclusive to text modalities.Its data are proportionately distributed between structured databases and unstructured text and image modalities.
- Operator Analysis: SELECT and PROJECT are the most common operators, while at least 10% of QDMRs contain complex-reasoning operators such as GROUP and COMPARATIVE.These operators are rare in high-level QDMRs.
- Sequence Length: Most QDMR decompositions contain 3–6 steps, whereas high-level QDMRs are much shorter.A single SELECT can identify an entity described by a long noun phrase in high-level decompositions.
- Quality Analysis: 97.4% of a random sample of 500 QDMRs were judged correct, including 93.8% granular-correct and 3.6% correct decompositions.Experts reported more difficulty decomposing superlatives, while correctness estimates were similar across modalities.
- Pseudo-logical Mapping: 99.5% of QDMRs had all steps mapped to pseudo-logical forms, and 93.1% of sampled examples produced fully accurate logical forms.Among 350 sampled logical forms, 99.4% had every step correctly mapped to its corresponding operator.
5 QDMR for Open-domain QA
QDMR decompositions support multi-step retrieval and answering for open-domain HotpotQA, improving retrieval and end-to-end performance while exposing operator-coverage limitations.
- Experimental Setup: BREAKRC answers multi-hop questions by processing high-level QDMR steps sequentially with information retrieval and reading comprehension models.SELECT runs answering directly; PROJECT substitutes prior answers, FILTER intersects probabilities, and COMPARISON compares returned numbers.
- Results: QDMR-based retrieval raises IR from 46.3 to 59.2 with gold QDMRs and to 52.5 with predicted QDMRs.The reported IR metric measures whether both gold paragraphs needed for the multi-hop answer were retrieved.
- Results: 43.3 to 52.4 is the reported COMBINEDG EM/F1 gain, while COMBINEDP reaches 49.3 from 43.3 using predicted QDMRs.COMBINED uses retrieved contexts with end-to-end answering rather than separate procedures for every QDMR operator.
- Limitations: QDMR-based systems remain limited by operators that BREAKRC does not implement, including BOOLEAN steps appearing in 9.4% of examples.Consequently, BREAKRC’s EM and F1 are only slightly higher than BERTQA in the cited comparison.
- Results: BREAKRC notably outperforms BERTQA on PROJECT and COMPARISON questions, which require multi-step reasoning and comprise 48% and 7% of examples.These question types are identified as less susceptible to reasoning shortcuts.
- Ablations: COMBINED substantially outperforms IR-NP, indicating that QDMR structure, rather than merely issuing multiple noun-phrase queries, drives the improvement.IR-NP issues one retrieval query for each noun phrase and feeds the union of retrieved paragraphs to BERTQA.
6 QDMR for Semantic Parsing
QDMR provides a domain-agnostic intermediate representation between natural-language questions and executable queries, reducing the need to annotate fully grounded logical forms.
- Scope: Pseudo-logical forms omit grounding of entities and relations in knowledge-base constants, because QDMR assumes an idealized knowledge base.This preserves domain and modality independence while leaving schema-specific grounding for later processing.
- Intermediate Representation: QDMR functions as an intermediate representation between a natural-language question and an executable query.This positioning parallels prior approaches using underspecified logical forms or intermediate text-to-SQL representations.
7 QDMR Parsing
The paper evaluates QDMR parsing with sequence- and graph-based metrics, comparing rule-based and neural models. COPYNET performs best overall, while manual analysis shows substantial semantic equivalence beyond exact matching, especially for standard QDMRs.
- Task Definition: QDMR parsing maps each question to an ordered sequence of decomposition steps separated by a special token.The target sequence concatenates step tokens with separator markers.
- Evaluation Metrics: GED evaluates predicted and gold QDMR graphs using normalized node and edge edit costs.Insertions and deletions cost 1, while node substitutions depend on token alignment.
- Evaluation Metrics: GED+ additionally models graph-node splitting and merging, but is computed only for graphs with up to five nodes.This restriction covers 75.2% of development examples according to the complexity caveat.
- Results: Neural models outperform RULEBASED, with COPYNET obtaining the best scores across all reported metrics.The authors attribute this result to the large proportion of QDMR tokens copied from the input question.
- Error Analysis: For standard QDMRs, 24% of COPYNET predictions exactly match the gold decomposition and another 30% are fully decomposed and semantically equivalent.For high-level examples, exact matches fall to 8%, while 46% are semantically equivalent and 46% erroneous.
8 Related Work
The related work positions QDMR as a domain-agnostic, natural-language question representation that supports scalable annotation and can connect to task-specific formalisms. It differs from prior decomposition and semantic-representation approaches in scope and annotation purpose.
- Question decomposition: Prior question-decomposition research largely focused on single modalities, while BREAK covers a broader set of questions across text, images, and databases.The paper contrasts QDMR with earlier text decomposition work conducted on a much more limited question set.
- Question decomposition: QDMR reverses an earlier annotation cycle: humans compose questions in BREAK and then decompose them into QDMRs.Cheng et al. instead generated simple-question sequences that crowd-workers paraphrased into compositional questions.
- Semantic formalism annotation: Unlike expert-oriented semantic formalisms, QDMR expresses question meaning in natural language to facilitate annotation by non-experts.This aligns QDMR with recent efforts to obtain quality semantic annotations through crowdsourcing.
- Relation to other formalisms: QDMR is designed for annotation, whereas DCS centers on paralleling syntax, and QDMR represents questions rather than general language.The formalism can potentially be converted to other representations depending on the execution task.
9 Conclusion
The paper concludes that QDMR supports scalable question-understanding annotation, improves open-domain QA, assists semantic parsing, and enables a reasonably effective automatic parser. BREAK provides over 83K decompositions spanning ten datasets and three modalities.
- Contributions: The authors report high-quality crowd-worker annotations at scale and introduce BREAK with over 83K decompositions from ten datasets and three modalities.The modalities are databases, images, and text.
- Implications: QDMR is useful for open-domain question answering and semantic parsing, while the proposed parser achieves reasonable performance.The conclusion presents these as applications and outcomes of the formalism and dataset.
- Implications: The paper characterizes QDMR as a promising direction for modeling question understanding across tasks that probe reasoning through questions.This stated scope extends beyond the specific evaluations presented in the paper.