Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

Motivation and framework

• TMNs learn to generate sub-questions answerable by existing neural or symbolic sub-models rather than using decompositions independent of those models.The framework can introduce phrases beyond standard question spans and target modules with different input languages.Source: §1 · pp. 1–2
• A next-question generator sequentially chooses each sub-question and its appropriate sub-model, with sub-question-answer sequences providing interpretable reasoning traces learned from distant supervision.The approach does not require explicit human decomposition annotation.Source: §1 · p. 2

MODULARQA

• MODULARQA decomposes questions for a neural factoid single-span QA model and a symbolic calculator across five reasoning classes: composition, conjunction, comparison, difference, and complementation.These classes support multi-hop and discrete numeric reasoning.Source: §1 · p. 2

Evaluation

• MODULARQA evaluates across DROP and HotpotQA, establishing a cross-dataset decomposition-based interpretable QA system.This scope supports reuse beyond a single task-specific dataset.Source: §1 · p. 2
• MODULARQA trails NumNet+V2 by only 3.7% F1, exceeds it by 2% F1 in limited-data settings, and shows +7% F1 robustness.It is also competitive with task-specific neural module networks and its explanations were preferred, understandable, and trustworthy in 67%-78% of human comparisons.Source: §1 · p. 2

Modular and decomposition-based QA

• Earlier modular QA systems commonly used predetermined composition structures or task-specific neural modules, limiting flexibility and reuse.Neural module networks for HotpotQA and DROP target one dataset and do not reuse existing QA systems.Source: §2 · pp. 2–3

Question decomposition

• Span-based decomposition methods for ComplexWebQuestions and HotpotQA are unsuitable for DROP questions requiring free-form sub-questions.TMNs address this by generating sub-questions in existing models’ language.Source: §2 · p. 3
• QDMR-style approaches can require costly human decomposition annotation and produce model-agnostic questions that still need specialized systems for symbolic operations.TMNs begin with predetermined models and learn decompositions compatible with them.Source: §2 · p. 3

Explainability

• Many multi-hop systems focus on one dataset, while post-hoc explanations are available only where supporting-sentence annotations exist and may be unfaithful or gameable.TMNs provide explanations across multiple datasets without such annotations.Source: §2 · p. 3

TMN architecture

• TMNs solve a complex question by alternating next-question generation with answering that question through the selected sub-model.The process ends when the generator emits [EOQ], returning the final sub-model answer.Source: §3 · p. 3
• The next-question generator receives the complex question and prior question-answer pairs, then outputs the next sub-model and sub-question.This makes module selection and textual communication part of one iterative process.Source: §3 · p. 3, §3.2 · p. 4

Model language

• Sub-task question models learn each sub-model’s question language by generating plausible questions conditioned on context, answer, and vocabulary hints.A SQuAD-trained text-to-text model, for example, captures the factoid questions answerable by that sub-model.Source: §3 · p. 4, §3.1 · p. 4

Training decompositions

• Distant-supervision hints provide intermediate answers, contexts, and vocabulary for each reasoning step, allowing training decompositions without explicit annotations.Hints can trade extraction ease against decomposition quality and need not be perfectly accurate.Source: §3.2 · p. 4
• Generated decompositions become training examples mapping the complex question and previous answers to the next model-question pair.The system samples multiple candidate questions per reasoning step and filters noisy decompositions before training the next-question generator.Source: §3.2 · p. 4

4.1 QA Sub-models

• MODULARQA combines a SQuAD-trained neural QA model with a symbolic Python calculator covering difference, complement, and conditional operations.The system also defines sub-task question models corresponding to the QA sub-models.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Sub-task Question Models, G

• SQuAD sub-questions are generated by BART from passages, answers, and vocabulary hints, then filtered to retain questions answered correctly by the neural QA model.The math sub-task question model instead enumerates numeric questions and filters them with the calculator.Source: §4.2 · p. 5

4.3 Generating Training Decompositions

• Distant-supervision hints encode question classes such as difference, comparison, complementation, composition, and conjunction to construct candidate reasoning steps.For example, difference hints pair numbers with a calculator operation, while composition hints connect supporting paragraphs through an intermediate entity.Source: §4.3 · pp. 5–6
• The hint-based procedure covers 89% of questions and finds gold-answer decompositions for 50%, while the next-question generator generalizes to decompositions leading to the gold answer for 81%.The reported 81% result reflects generalization beyond the directly usable hint-derived decompositions.Source: §4.3 · pp. 6–7

4.4 Inference

• At inference, nucleus sampling proposes five questions per reasoning step, and best-first search scores partial chains to select the best decomposition and final answer.Each generated question is answered by its associated sub-model, and the resulting partial chain conditions subsequent generation.Source: §4.3 · p. 7, §4.4 · p. 7

Comparison to Baselines

• MODULARQA is competitive with prior approaches on DROP and mainly trails HotpotQA systems that can exploit dataset artifacts.Table 2 covers cross-dataset variants, targeted interpretable systems, and targeted black-box systems across aggregate and reasoning-class F1 scores.Source: Table 2 · p. 8, §5.4 · p. 9

Explanation and Interpretability

• Crowdworkers understood MODULARQA explanations better in 78% of cases and generally trusted and preferred them over DecompRC explanations.The comparison used 155 in-scope questions and five annotations per question across trust, understanding, and preference dimensions.Source: §5.1 · pp. 7–8, Table 3 · p. 8

Comparison to MODULARQA Variants

• MODULARQA outperforms a word-overlap decomposition by 7 F1 points on DROP and 4 on HotpotQA, showing that coverage alone is insufficient.The word-overlap baseline removes the RoBERTa classifier, while MODULARQA uses learned scoring and search.Source: §5.2 · p. 8
• Removing search reduces performance by 18 points on DROP and 19 on HotpotQA because the most likely sub-question may be unanswerable while an alternative decomposition succeeds.Search allows the system to retain alternate decompositions that can lead to the correct answer.Source: §5.2 · p. 8

Comparison to Dataset-Specific Models

• MODULARQA outperforms the DROP-specific NMN-D on the shared subset and is comparable to S-NMN but below DecompRC on HotpotQA.DecompRC can use single-hop reasoning and potentially exploit HotpotQA artifacts, while MODULARQA produces textual explanations.Source: §5.3 · p. 8, §5.4 · p. 8

Error Analysis

• HotpotQA errors mainly involve unsupported intermediate answer types: 65% arise from intermediate questions with multiple or yes/no answers.The current modules do not handle these answer types, identifying a concrete improvement path.Source: §5.4 · p. 9

Additional Benefits of TMNs

• On DROP’s contrast set, MODULARQA outperforms NumNet+V2 by 7%-10% on in-scope questions and surpasses it when both systems train on 20% of the data.The robustness evaluation uses 239 of 947 contrast-set questions within MODULARQA’s scope.Source: §5.5 · p. 9

Conclusion

• Text Modular Networks cast complex tasks as textual interaction between existing, simpler QA modules, instantiated here as MODULARQA for multi-hop and discrete numeric reasoning.MODULARQA decomposes DROP questions into textual sub-questions and applies to both DROP and HotpotQA.Source: §6 · p. 9

Future Work

• Future work includes extending the model to counting and Boolean conjunction questions, whose decomposition or answer-generation challenges remain unresolved.The paper also mentions end-to-end training with REINFORCE as a possible improvement for scoring and greedy inference.Source: §6 · p. 9

A.1 Training SQuAD Question Generator

• SQuAD 2.0 answerable questions provide training data, while content-word vocabulary hints and sampled distractors simulate noisy generation conditions.Hints use nouns, verbs, numbers, proper nouns, adjectives, and adverbs after stop-word filtering; 2-7 distractors are sampled per question.Source: §A.1 · p. 12
• At each reasoning step, five sampled questions are retained when the corresponding sub-model answers correctly, and successful complete chains are selected.Each retained sub-question is expanded by five questions at the next step.Source: §A.2 · p. 12

A.4 Inference Parameters

• Inference samples more questions early and fewer later, while QA models choose the most likely answer independently for each paragraph when multiple paragraphs are available.The sampling schedule uses n_i = N ∗ r^i with N=15 and r = 1/2; greedy inference uses beam search with width=4.Source: §A.4 · pp. 12–13
• The search uses θ to guide exploration but relies primarily on the chain score δ to select complete decompositions, with score(u) = θ(u) + λδ(u) and λ=10.Higher scores are worse, and partial chains with sufficiently high scores can be pruned because the score is monotonic under continuation.Source: §A.4 · p. 13
• The chain scorer is trained from positive and negative complete chains labeled by final-answer F1, treating chains above an F1 threshold of 0.2 as positive.A RoBERTa model is trained to classify the collected chains.Source: §A.5 · p. 13

A.6 Less Data Training

• When trained on 20% of the data, or 2600 questions, the system samples 15 questions per step and increases N to 10 for chain-score generation.The additional sampling is used to generate more training data for both the next-question generator and chain scorer.Source: §A.6 · p. 13

Additional Results

• Table 4 reports both F1 and exact match scores for each evaluated setting, alongside whether models explain reasoning, generalize across datasets, and reuse existing QA models.The table expands the quantitative results previously presented in Table 2.Source: B Additional Results · p. 13
• Table 5 lists the operations handled by the symbolic calculator AC and the corresponding question-generation approach in GC.The supplied passage identifies the table’s scope but does not provide its individual operations or results.Source: Table 5 · p. 14

• ModularQA had only a 2-point F1 gap from DecompRC on 253 in-scope questions.Source: C Human Evaluation · p. 14
• ModularQA produced a reasoning chain on 79 questions where DecompRC did not, while failing on only 12 questions where DecompRC produced a chain.Source: C Human Evaluation · p. 14
• Crowdworkers evaluated explanations on 155 questions where both systems produced chains, measuring trust, understanding, and overall preference.Source: C Human Evaluation · p. 14
• The evaluation compared which system’s answer participants trusted, which explanation they understood better, and which explanation they preferred overall.Source: C Human Evaluation · p. 14

HotpotQA bridge questions

• HotpotQA bridge questions use document-specific hints to identify an intermediate entity in one document and target the final answer in the other.Source: §D.1.1 · p. 14

HotpotQA conjunction questions

• HotpotQA conjunction questions use two sub-questions with the same answer when that answer appears in both supporting paragraphs.Source: §D.1.1 · p. 14

Hint resources

• The generated hints and decompositions are organized in tables covering sample DROP and HotpotQA cases, difference-question regexes, and DROP error types.Source: Table 6 · p. 15, Table 7 · p. 16, Table 8 · p. 16

HotpotQA comparison questions

• For HotpotQA comparison questions, hints combine two document numbers or dates with candidate entities for symbolic calculator questions.Source: §D.2 · p. 15

DROP hints

• DROP questions are classified using lexical patterns, filtered for false positives, and paired with numbers or dates whose arithmetic can produce the answer.Source: §D.3 · p. 15, §D.3.1 · p. 15

• Because the search does not explore all decompositions, other valid or invalid alternatives may not be considered.Source: E Drop Error Analysis · p. 16
• Sampling may explain errors labeled as sampling, incomplete decomposition, or no valid decomposition when exhaustive search could produce different candidates.Source: E Drop Error Analysis · p. 16