PaperlayerView exact revision on arXiv

Source-linked AI summary

Successive Prompting for Decomposing Complex Questions

Dheeru Dua, Shivanshu Gupta, Sameer Singh, Matt Gardner

arXiv:2212.04092v1cs.CL

TL;DR

Complex question answering requires intermediate reasoning supervision, but prior approaches couple decomposition and answering. Successive Prompting separates these steps and improves few-shot DROP performance, including further gains from a fine-tuned QA module.

  • Problem

    Prior approaches couple supervision for question decomposition with intermediate question answering, while reasoning-step annotations can be expensive to collect.

  • Method

    Successive Prompting iteratively decomposes a complex question, answers each simple question, and alternates these steps until producing the final answer.

  • Results

    Successive prompting improves performance by 4.6 F1 over chain-of-thought prompting, while replacing the in-context QA module with a fine-tuned module adds a 9.5 F1 improvement.

  • Takeaways & Limitations

    Modular systems that delegate subtasks to the most appropriate model can be more effective for complex tasks than relying on a large LM alone.

  • Takeaways & Limitations

    The approach requires decomposition data, and some complex questions or domains are difficult to decompose or cover with synthetic data generators.

Abstract

from arXiv · show

Answering complex questions that require making latent decisions is a challenging task, especially when limited supervision is available. Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting by demonstrating how to output intermediate rationalizations while solving the complex question in a single pass. We introduce ``Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution. Successive prompting decouples the supervision for decomposing complex questions from the supervision for answering simple questions, allowing us to (1) have multiple opportunities to query in-context examples at each reasoning step (2) learn question decomposition separately from question answering, including using synthetic data, and (3) use bespoke (fine-tuned) components for reasoning steps where a large LM does not perform well. The intermediate supervision is typically manually written, which can be expensive to collect. We introduce a way to generate a synthetic dataset which can be used to bootstrap a model's ability to decompose and answer intermediate questions. Our best model (with successive prompting) achieves an improvement of ~5% absolute F1 on a few-shot version of the DROP dataset when compared with a state-of-the-art model with the same supervision.

1 Introduction

Successive Prompting addresses limitations of single-pass reasoning by iteratively decomposing complex questions, answering simpler subquestions, and separating supervision for decomposition from question answering.

  • Motivation: Single-pass prompting couples decomposition and intermediate-step supervision, selects only one example set, and cannot readily substitute symbolic reasoning engines.The selected examples may not provide optimal supervision for the required intermediate steps.
  • Approach: Successive Prompting alternates between generating a simple question and answering it until the complex question is resolved.Each step uses a separate LM query.
  • Motivation: Separate decomposition and answering enable step-specific in-context examples and independently trained models for each task.This modularity also permits specialized components for reasoning steps where large LMs are weaker.
  • Data: Synthetic data generated from semi-structured Wikipedia tables supplies complex-question and intermediate-step supervision alongside 300 manually decomposed DROP examples.The synthetic data is intended to address poor performance in the few-shot setting.
  • Results: ∼5% improvement in F1 was achieved over a state-of-the-art model on few-shot DROP with the best successive prompting model.The comparison is reported for models using the described few-shot supervision.

2 Decomposing Complex Questions

The method represents latent reasoning as successive simple question-answer pairs, alternating question decomposition and question answering while selecting demonstrations separately for each stage.

  • Successive prompting: Successive prompting represents each latent step as a simple question-answer pair rather than a declarative Chain-of-Thought sentence.The model alternates intermediate-question and intermediate-answer outputs, treating final-answer prediction as a stopping decomposition step.
  • In-context learning: At each decomposition step, a QD index retrieves demonstrations using the complex question and current step, while a QA index retrieves examples using the generated simple question.This gives the QA stage access to subquestions that may not resemble the original complex question.
  • Training paradigm: Fine-tuning uses T5 sequence-to-sequence models with control codes to train question decomposition and question answering as separate tasks.Synthetic data extends the operations beyond the first-order reasoning handled by prior text modular networks.
  • Specialized modules: Successive prompting supports specialized QA modules, including a mathematical submodule for operations such as counting, difference, and sorting.The module parses generated simple questions for symbolic operation types and arguments before execution.

3 Synthetic Dataset

The paper generates synthetic decompositions from semi-structured Wikipedia tables to reduce reliance on costly manual intermediate-step annotations and broaden training coverage.

  • Motivation: Manual supervision for intermediate reasoning steps is expensive, expert-dependent, and insufficient when only a small collection of decompositions is available.The paper reports poor performance from such small collections on varied DROP questions.
  • Data representation: Table 1 supplies Wikipedia rows as sentence-level context and columns as the basis for generating questions used in the synthetic-data examples.The table is the source context for the COUNT decompositions illustrated in Figure 3.
  • Generation process: Synthetic data converts semi-structured Wikipedia tables into paragraphs and questions, using columns for first-order operations and combinations for higher-order questions.The generator uses curated templates and supports operations including COUNT, TOP(k), FILTER, SUM, DIFFERENCE, and INTERSECTION.
  • Generation process: Approximately 141K synthetic examples are generated with natural-language or symbolic operation representations depending on whether an LM or symbolic engine performs arithmetic.Higher-order combinations are generated wherever possible.

4 Experiments and Results

Experiments on few-shot DROP evaluate successive prompting with synthetic data, in-domain annotations, symbolic reasoning, and modular QD/QA components. Successive prompting improves over comparable baselines, while results expose strengths of fine-tuning and remaining decomposition challenges.

  • 4.1 In-context Learning: On DROP in-context prompting, successive prompting beats Chain-of-Thought by 3.5% with synthetic data and 4.3% with synthetic data plus 300 DROP annotations.The best Synthetic+DROP model reaches 30.6% F1 on the test set, while replacing the symbolic calculator with a language model reduces performance by 1.5% F1.
  • 4.2 Model Fine-tuning: Synthetic data universally improves performance, with gains nearing 20% absolute for TASE; adding supervision from complex QA pairs to decompositions also consistently helps.These comparisons are reported across 0-shot, complex-QA, and decomposition-supervision settings.
  • 4.2 Model Fine-tuning: 50.2 F1 is achieved by fine-tuned successive prompting with a symbolic reasoning engine, versus 45.1 F1 for a state-of-the-art model with similar supervision.The paper reports a 5.1% F1 advantage on the test set.
  • 4.3 In-context vs Fine-Tuning: Fine-tuning the QA module improves F1 by about 10% with an in-context QD model, whereas improving QD alone helps little unless QA can handle its decompositions.In-context QA struggles particularly with list-answer questions needed for symbolic aggregation.
  • 4.4 Qualitative Examples: Error analysis attributes incorrect predictions to QA errors, next-question prediction errors, and reasoning types missing from synthetic annotations.QA errors occur 40% of the time in-context and 22% when fine-tuned; uncovered compositional types account for 28% and 46%, respectively.
  • 4.4 Qualitative Examples: Successive prompting can improve supervision by decomposing questions, but implicit passage reasoning can make decomposition difficult.The qualitative examples include correctly restricting field-goal counting to the first half and difficulty defining a comparison for the Chemulpo-versus-Veracruz question.

5 Related Work

Prior work approaches compositional question answering with prompting, symbolic decomposition, or contextual models paired with calculators. Successive Prompting differs by separating decomposition from answering and supporting broader reasoning compositions.

  • Prompting methods: Prior prompting methods use few demonstrations to guide large language models in generating reasoning steps for complex questions.Chain-of-Thought-style approaches prompt the model once using retrieved examples containing questions, reasoning, and answers.
  • Symbolic methods: Symbolic approaches parse complex questions into executable structures, while DROP systems often combine contextual models with calculators for discrete operations.
  • Relation to prior work: Successive Prompting separates question decomposition from intermediate answering, enabling tailored examples and alternative reasoning components.Unlike closely related Text Modular Networks and MRKL, it supports more varied simple questions and multi-order reasoning without a chain scorer.

6 Conclusion

The paper presents Successive Prompting as a modular approach that decomposes complex questions into simple QA pairs and independently trains or queries the components. It reports gains over chain-of-thought prompting and further gains from a fine-tuned QA module.

  • Conclusion: The approach decomposes complex questions into simple QA pairs, allowing question-decomposition and question-answering systems to be trained and queried independently.
  • Conclusion: Successive prompting improves in-context performance by 4.6 F1 over chain-of-thought prompting.
  • Conclusion: Replacing the in-context QA module with a fine-tuned module improves overall performance by an additional 9.5 F1.The fine-tuned module is described as adept at handling list-type questions.
  • Conclusion: The authors argue that delegating subtasks to the most appropriate model can be more effective than having one large language model solve the entire task.Successive prompting is presented as one implementation of this decomposition-and-delegation strategy.

Limitations

Successive Prompting requires decomposition data and incurs multiple queries, while its effectiveness depends on choosing an appropriate decomposition granularity. The demonstrated application covers only one setting and may not extend to reasoning requiring commonsense or causal knowledge.

  • Scope and data: Successive Prompting requires decomposition data, which may be difficult or impossible to obtain for some complex questions.
  • Scope and data: Synthetic data covered most DROP reasoning types but not questions requiring commonsense or causal reasoning.
  • Scope and data: The paper demonstrates only one specific version of iterative information-need decomposition in one specific setting.
  • Granularity: Choosing decomposition granularity is difficult because the appropriate level depends on the underlying model's capabilities.A model that can answer a complex question directly may not need further decomposition.
  • Computation: Successive prompting increases computational requirements by making many large-language-model queries for one complex question.

Ethics Statement

The work focuses on improving complex question answering with limited data using existing training data and conventional model-performance evaluation. It does not address social impacts or biases in natural language processing systems.

  • Ethics Statement: The work focuses on complex question answering with limited data and conventional methods for testing model performance.
  • Ethics Statement: The work does not address social impacts or biases in natural language processing systems.

A.1 Control codes for Model Fine-tuning

The model is fine-tuned to alternate between generating a simple question and answering it, using control codes to mark each stage. This alternation continues until the decomposition reaches the end-of-question marker.

  • “QS:” prompts generation of a simple question from the complex question and previously generated latent steps.
  • “A:” prompts the model to answer the current simple question.
  • The question-generation and answer stages alternate until the “EOQ” decomposition marker is reached.

A.2 Synthetic Dataset Statistics

The synthetic dataset illustrates decompositions that convert structured tables into sequential simple questions and answers. Its examples cover filtering, counting, sorting, comparison, difference, summation, and combinations of these operations.

  • Synthetic examples decompose complex questions into sequential reasoning steps over wikitable rows and columns.Rows become sentences, while columns support question generation.
  • Filter: Filter-and-count decompositions first select opponents by date and attendance, then count the qualifying opponents.The example filters dates later than 21 January 1991 and attendance below 20000 before counting.
  • Comparison: Comparison decompositions retrieve two attendances and use a conditional comparison to select the round with the higher attendance.The example compares 34,669 for SF 2nd Leg with 33,861 for QFR and selects SF 2nd Leg.
  • Difference: Difference and sum decompositions retrieve values associated with selected opponents before applying arithmetic operations.Examples compute 34,669 − 10,037 = 24,632 and sum 5,666, 33,861, and 34,669 to obtain 74,196.
  • Count: The examples also represent basic list construction and counting, such as enumerating opponents and returning a list cardinality.One decomposition counts five opponents after listing them.
  • Higher-order combinations: Higher-order combinations include sorting, filtering, and difference operations applied to intermediate answer lists.Examples identify ranked attendances and compute differences between values selected by opponent or round.
Loading 2212.04092v1…