Source-linked AI summary
Unsupervised Question Decomposition for Question Answering
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, Douwe Kiela
TL;DR
Multi-hop QA is difficult because answers require combining information scattered across sources, while supervised question decompositions are costly to obtain. The paper proposes ONUS, which mines pseudo-decompositions and learns to generate multiple simple sub-questions, answers them with an off-the-shelf QA model, and recomposes the answers. On HotpotQA, decomposition improves QA across original, out-of-domain, and multi-hop development sets, with performance matching supervised and heuristic methods.
Problem
Multi-hop QA requires combining scattered evidence, but existing decomposition methods rely on costly supervision, heuristics, or rule-based handling.
Method
ONUS mines pseudo-decompositions from simple-question corpora, learns unsupervised question-to-sub-question mappings, answers sub-questions, and recomposes their answers.
Results
ONUS decompositions improve F1 over a strong baseline by 3.1 points on original, 10 points on out-of-domain, and 11 points on multi-hop HotpotQA development sets.
Takeaways & Limitations
ONUS matches supervised and heuristic decomposition utility while automatically generating fluent decompositions across all four HotpotQA question types.
Takeaways & Limitations
FastText representations outperformed tested TFIDF and BERT representations, with no improvements observed from those alternatives.
Abstract
from arXiv · showhide
We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering. Since labeling questions with decompositions is cumbersome, we take an unsupervised approach to produce sub-questions, also enabling us to leverage millions of questions from the internet. Specifically, we propose an algorithm for One-to-N Unsupervised Sequence transduction (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We answer sub-questions with an off-the-shelf QA model and give the resulting answers to a recomposition model that combines them into a final answer. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets. ONUS automatically learns to decompose different kinds of questions, while matching the utility of supervised and heuristic decomposition methods for QA and exceeding those methods in fluency. Qualitatively, we find that using sub-questions is promising for shedding light on why a QA system makes a prediction.
1 Introduction
The paper addresses multi-hop QA by decomposing hard questions into simpler sub-questions without annotated decompositions. ONUS learns such decompositions unsupervised, then uses single-hop QA and recomposition to answer the original question.
- Multi-hop QA is difficult because required information can be scattered across different places.
- A divide-and-conquer pipeline decomposes hard questions, answers simpler sub-questions, and recomposes their answers.
- Existing heuristic, rule-based, and supervised decomposition methods require substantial human effort and high-quality annotations.
- ONUS mines pseudo-decompositions from millions of web questions and trains an unsupervised sequence-to-sequence model to generate fluent sub-questions.
- The system answers generated sub-questions with an off-the-shelf single-hop QA model and gives the answers to a recomposition model.
- ONUS improves HotpotQA over a strong baseline across original, out-of-domain, and multi-hop development sets while matching supervised and heuristic methods.
2 Method
The method creates noisy pseudo-decompositions from simple-question retrieval, learns to generate multiple sub-questions without trusting noisy pairings, and recomposes their predicted answers into a final answer.
- The goal is to use a single-hop QA model to help a multi-hop QA model answer hard questions without annotated decompositions.
- The pipeline decomposes a question, obtains sub-answers from a single-hop QA model, and predicts the final answer from the sub-questions and sub-answers.
- Creating Pseudo-Decompositions: Pseudo-decompositions retrieve simple questions that are similar to the hard question while encouraging diversity among retrieved sub-questions.
- Learning to Decompose: PseudoD uses retrieved sub-questions directly, whereas Seq2Seq learns to maximize the likelihood of pseudo-decompositions given the original question.
- Learning to Decompose: ONUS maps one question to N sub-questions with unsupervised sequence-to-sequence learning because the initial question–pseudo-decomposition pairing is noisy.
- The recomposition model receives the original input together with each decomposed sub-question and its predicted answer.
3 Experimental Setup
The experiments evaluate ONUS on HotpotQA using retrieval-based pseudo-decompositions, unsupervised decomposition models, single-hop QA, and recomposition. The benchmark requires reasoning over two Wikipedia paragraphs and reports F1 and Exact Match.
- Benchmark: HotpotQA questions require information from two distinct Wikipedia paragraphs, with ten context paragraphs including supporting facts and distractors.
- Benchmark: HotpotQA answers are yes, no, or text spans, and performance is measured by word-overlap F1 and Exact Match.
- Pseudo-Decompositions: The method fixes the number of retrieved sub-questions at N = 2 after finding similar QA results with variable N.
- Pseudo-Decompositions: Pseudo-decompositions use FastText embeddings and cosine similarity to retrieve relevant simple questions.
- Pseudo-Decompositions: The retrieval search is approximated because exact optimization requires O(|S|^2) comparisons when the simple-question corpus exceeds 10M items.
- Single-Hop QA: The single-hop QA ensemble is trained on SQuAD 2 and easy HotpotQA examples, then used as a black-box model for sub-question answering.
- Recomposition Model: The recomposition model appends sub-question and sub-answer pairs to the original question and averages results across five training runs.
4 Results on Question Answering
ONUS-based decompositions improve HotpotQA across original, out-of-domain, multi-hop, and question-type dev sets, while approaching supervised and heuristic methods. Sub-answers and supporting evidence are central to these gains, and the decompositions also generalize beyond standard QA.
- Main Results: ONUS decompositions improve QA by 3.1 F1 on original, 11 F1 on multi-hop, and 10 F1 on out-of-domain dev sets.The approach reaches 80.1 versus 79.8 F1 for supervised and heuristic DECOMPRC on the original dev set.
- Main Results: ONUS trained on FastText pseudo-decompositions achieves 79.34 test F1 and 66.33 EM, competitive with strongly supervised SAE and HGN.SAE and HGN use supporting-fact supervision, unlike this approach.
- Question Type Breakdown: Decompositions improve QA across all four question types, including single-hop questions, where performance rises from 73.9 to 76.9 F1.A single decomposition model handles all types without question-type-specific tailoring.
- Answers to Sub-Questions are Crucial: Sub-answers are necessary for improvement: random or absent answers stay near the 77.0 F1 baseline, whereas sentence-level answers reach 80.1 F1.Providing the sentence containing the predicted span is better than providing the span alone, 80.1 versus 77.8 F1.
- How Do Decompositions Help?: ONUS decompositions produce relevant natural-language sub-questions and sub-answers that can make otherwise black-box QA predictions more interpretable.Examples include imperfect or under-specified sub-questions whose predicted answers still support the final recomposed answer.
5 Analysis
Analysis shows that decomposition quality and sub-answer confidence are related to downstream QA performance, while the approach remains robust across QA and recomposition models. Stronger recomposition models obtain larger gains from decompositions.
- Decomposition Quality: ONUS decompositions are more natural and well-formed than DECOMPRC decompositions and remain close in edit distance and length to the original question.This matches the observation that ONUS decompositions are largely extractive.
- Unsupervised Decomposition Model: Lower-probability beam hypotheses reduce F1 only from 80.1 to 79.3, indicating that several top decompositions remain useful.The experiment uses five beam-search hypotheses and evaluates each ranked decomposition with a recomposition model.
- Sub-Answer Confidence: Higher single-hop sub-answer confidence correlates with better downstream multi-hop QA accuracy across all dev sets.Low confidence may indicate an unanswerable or ill-formed sub-question or an incorrect sub-answer, reducing useful evidence retrieval.
- Single-hop Question Answering Model: A weaker BERTBASE single-hop model yields nearly identical full-system performance to ROBERTALARGE, 79.9 versus 80.1 F1.Although the models score 56.3 versus 66.7 F1 on HotpotQA itself, their sub-question answers work similarly within the larger system.
- Recomposition Model: Decomposition gains increase with recomposition-model strength: 1.2 F1 for BERTBASE, 2.6 F1 for BERTLARGE, and 3.1 F1 for ROBERTALARGE.The comparison measures the improvement from adding decompositions to each pretrained base model.
6 Related Work
Prior decomposition methods rely on supervision, heuristics, special-case handling, or formal programs, whereas ONUS is fully automated and unsupervised. The approach can use unlabeled question collections and complement supervised or weakly supervised methods.
- Heuristic and Supervised Decomposition: Earlier systems decompose questions with hand-crafted heuristics, rule-based algorithms, or supervised decomposition annotations, requiring substantial human effort.DECOMPRC combines supervision with heuristic handling based on part-of-speech tags and more than 50 keywords.
- Unsupervised Decomposition: ONUS differs by providing fully automated decomposition with little hand-engineering instead of question-type-specific special-case handling.This contrasts with Watson and DECOMPRC, which decompose questions in multiple ways or use heuristic exceptions.
- Semantic Parsing: Traditional semantic-parsing approaches represent decompositions as compositional programs but usually require strong program-level supervision.Other systems use strong supervision over sentences needed to answer questions.
- Relation to Supervision: Unsupervised decomposition can exploit millions of otherwise unusable questions and can be combined with labeled examples in semi-supervised learning.The paper frames this as complementary to strongly and weakly supervised decomposition approaches.
7 Conclusion
The paper presents an unsupervised decomposition-based QA system and reports that it improves HotpotQA while matching strongly supervised methods. It also finds that large question corpora and suitable pseudo-decomposition objectives improve the approach.
- ONUS decomposes questions, answers sub-questions with an off-the-shelf QA system, and recomposes their answers into a final answer.
- The approach matches the effectiveness of state-of-the-art methods using stronger supervision, including supporting-fact labels or example decompositions.
- For N = 3 synthetic questions, the alternate objective reaches 30% mean reciprocal rank, while Eq. 1 reaches approximately 0%.
- 80.1 QA F1 is achieved with Common Crawl pseudo-decompositions, compared with 79.2 using only SQUAD 2 questions.
A.4 Pseudo-Decomposition Retrieval Method
The retrieval method compares representation choices and training variants for constructing pseudo-decompositions, while training on a much larger question corpus improves QA F1.
- For one ONUS training run with FastText pseudo-decompositions, the unsupervised stopping criterion selects epoch 3, which performs roughly as well as the best checkpoint at epoch 5.
- Pseudo-decompositions are retrieved using FastText, TFIDF, or BERTLARGE first-layer hidden-state representations across several learning methods.
B.1 Training Procedure
ONUS training uses unsupervised stopping based on scaled round-trip BLEU, while downstream QA evaluation shows this criterion does not significantly reduce performance. The system also includes unsupervised supporting-fact retrieval.
- Scaled round-trip BLEU evaluates generated decompositions against the original question while rewarding decompositions with two suitable sub-questions.
- The stopping criterion prevents copying the full multi-hop question from achieving perfect round-trip BLEU by requiring two sub-questions and excluding overly long or all-word sub-questions.
- The unsupervised stopping criterion does not significantly hurt downstream QA compared with weakly supervised stopping based on multi-hop QA accuracy.
- ONUS and Seq2Seq decomposition models are trained with distributed computation over eight 32GB NVIDIA V100 GPUs for roughly eight hours.
- 15.7 EM and 55.2 F1 are achieved for retrieving gold supporting facts, approaching the 59.0 F1 of early supervised methods.
B.4 Decomposing Questions in Other Tasks
The paper applies a decomposition model trained on HotpotQA and Common Crawl to questions and claims from several other tasks. Its downstream single-hop QA model is trained on SQUAD 2 and easy HotpotQA examples with distractor paragraphs.
- The trained decomposition model generates sub-questions for ComplexWebQuestions, CLEVR, and FEVER 1.0 inputs.
- The single-hop QA ensemble uses SQUAD 2 and HotpotQA easy questions, with two TFIDF-retrieved Wikipedia distractor paragraphs added for one model.
- Figure 6 varies training-data quantity and tests the effect of removing SQUAD 2.0 and HotpotQA easy single-hop training data on recomposition-model QA F1.
D.1 Varying Training Set Size
Decompositions improve QA when sufficient training data is available, with gains varying across question types and answer forms.
- D.1 Varying Training Set Size: Decompositions improve QA as the number of multi-hop training examples varies, provided the recomposition model has adequate training support.The experiments vary multi-hop training examples and whether easy or SQUAD 2 data augment recomposition training.
- D.1 Varying Training Set Size: Decompositions help yes/no and span-answer questions roughly equally across the evaluation sets.
- D.1 Varying Training Set Size: No consistent decomposition benefit emerges across question types grouped by wh-question starting words.
- D.1 Varying Training Set Size: Larger gains occur when the subanswer sentence contains the gold final answer.
D.3 Training Hyperparameters
The supplied passages specify fixed training settings and selected hyperparameters, alongside examples and tables illustrating ONUS decompositions.
- D.3 Training Hyperparameters: ROBERTALARGE training uses two epochs because longer training did not improve performance.
- D.3 Training Hyperparameters: The baseline QA model uses batch size 64, learning rate 1.5 × 10^-5, and weight decay 0.01.These hyperparameters were selected using development-set performance.
- D.3 Training Hyperparameters: The hyperparameter search considers learning rates from 1×10^-5 to 3×10^-5 and weight decay values from 0 to 0.1.
- D.3 Training Hyperparameters: ONUS examples decompose complex questions into simpler subquestions, including two-question decompositions for geography, historical construction, and language questions.
- D.3 Training Hyperparameters: The Variable-Length ONUS model produces three subquestions for one singer-identification question rather than two.
- D.3 Training Hyperparameters: The supplied tables compare decomposition methods on age-comparison and shared-country questions, and report zero-shot decompositions from other datasets.