Source-linked AI summary
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi
TL;DR
Large language models often generate noisy, logically inconsistent explanations, limiting explanation-based reasoning. Maieutic Prompting recursively generates abductive explanations, resolves their logical relations with symbolic satisfiability, and achieves stronger and more robust inference on challenging binary QA benchmarks.
Problem
Language-model explanations can be noisy and logically inconsistent, while explanation-based prompting depends on their correctness for reliable inference.
Method
Maieutic Prompting generates abductive explanations for alternative answers, recursively expands them into a tree, and uses weighted MAX-SAT to infer consistent truth assignments.
Results
Up to 20% better performance than other prompting methods was reported across three commonsense reasoning and fact verification benchmarks, with performance on par with or better than supervised models.
Takeaways & Limitations
The method provides robust inference under perturbations and an interpretable interface for examining the rationale behind predictions.
Takeaways & Limitations
Experiments are limited to validating given statements, while broader task formats such as multiple-choice QA remain future work.
Abstract
from arXiv · showhide
Despite their impressive capabilities, large pre-trained language models (LMs) struggle with consistent reasoning; recently, prompting LMs to generate explanations that self-guide the inference has emerged as a promising direction to amend this. However, these approaches are fundamentally bounded by the correctness of explanations, which themselves are often noisy and inconsistent. In this work, we develop Maieutic Prompting, which infers a correct answer to a question even from the noisy and inconsistent generations of LM. Maieutic Prompting induces a tree of explanations abductively (e.g. X is true, because ...) and recursively, then frames the inference as a satisfiability problem over these explanations and their logical relations. We test Maieutic Prompting for true/false QA on three challenging benchmarks that require complex commonsense reasoning. Maieutic Prompting achieves up to 20% better accuracy than state-of-the-art prompting methods, and as a fully unsupervised approach, performs competitively with supervised models. We also show that Maieutic Prompting improves robustness in inference while providing interpretable rationales.
1 Introduction
Explanation-based prompting can produce logically inconsistent and unreliable reasoning, motivating Maieutic Prompting, which organizes noisy explanations into a recursively generated structure and resolves them symbolically.
- Motivation: Explanation-based prompting uses model-generated explanations to elicit knowledge and guide inference before producing an answer.Prior work applies this idea to commonsense knowledge, mathematics, and program execution.
- Motivation: Manual inspection found explanations that fail to entail answers, ignore negation, or contradict the model’s own conclusions.The inspection covered 100 QA samples using 175B GPT-3.
- Method: Maieutic Prompting generates abductive explanations for both True and False hypotheses and recursively tests explanations by prompting the model with its own generations.This process forms a tree of propositions linked by logical relations.
- Method: Weighted MAX-SAT jointly assigns truth values to the explanations and original question by selecting generations that best satisfy their observed logical relations.The method can run completely unsupervised with few-shot promptable language models.
- Results: Up to 20% better performance than other prompting methods was achieved across three commonsense reasoning and fact verification benchmarks.The method also performed on par with or better than supervised models, while showing robustness and interpretable rationales.
2 Problem Setup and Background
The task is to infer whether a statement is True or False, using standard prompting or explanation-based prompting as alternative inference procedures.
- Problem Setup: The problem setup asks an LM to infer the truth value of a statement Q, with the answer restricted to True or False.The inferred answer is denoted Â.
- Standard Prompting: Standard few-shot prompting conditions the model on k in-context question–answer examples C.C contains paired examples (q_i, a_i).
- Explanation-based Prompting: Explanation-based prompting factorizes inference into generating an explanation E and then predicting the answer A conditioned on Q, E, and the examples.Its context contains question, explanation, and answer triples.
- Explanation-based Prompting: Because marginalizing over all explanations is intractable, prior methods approximate the inference process by sampling explanations.The supplied setup introduces this as a sampling-based approximation.
3 Maieutic Prompting
Maieutic Prompting addresses noisy and inconsistent LM explanations by recursively generating propositions, evaluating their logical relations, and inferring a consistent answer through weighted satisfiability.
- 3 Maieutic Prompting: Logical integrity identifies propositions whose truth values remain consistent under negation, making them more credible than non-integral propositions.A proposition is integral when the LM assigns opposite truth values to it and its negation.
- 3.1.1 Abductive Explanation Generation: Abductive explanation generation prompts the LM to rationalize both True and False labels, exposing alternative explanations and eliciting more specific reasoning.The method represents these explanations as ET and EF generated conditional on the corresponding answer labels.
- 3.1.2 Depth-wise Knowledge Spanning: Recursive prompting treats each generated explanation as a new question, producing a multi-depth maieutic tree whose propositions support one another logically.The tree spans knowledge depth-wise, while repeated sampling can broaden explanation diversity and coverage.
- 3.1.3 When to Stop Generating: Generation stops on branches reaching logically integral propositions and prunes branches that remain non-integral at the maximum depth.This stopping rule limits the exponential cost of generating a full tree while retaining paths ending in propositions validated by the LM.
- 3.2 Defining the Relations: The method models each explanation’s belief and its consistency with the question-answer pair as weighted logical relations.Belief compares the LM’s probabilities for a proposition and its negation, while consistency compares generation likelihoods under the given and opposite answers.
- 3.3 Inference: Belief constraints encode the truth values of logically integral leaf nodes, whose credibility supplies unary constraints for inference.For example, EF and ETF contribute belief constraints in the maieutic tree.
- 3.3 Inference: Consistency constraints encode logical implications along tree edges, linking parent propositions to explanations generated under True or False labels.These binary constraints complement the unary belief constraints.
- 3.3 Inference: Weighted MAX-SAT assigns truth values to all tree propositions and the original question by maximizing the weight of satisfied constraints.An off-the-shelf solver performs this symbolic inference over the generated relations.
4 Experiments
Experiments evaluate Maieutic Prompting against prompting and supervised baselines on three challenging binary commonsense and fact-verification benchmarks. Results show stronger prompting performance, robustness to semantic and prompt perturbations, and contributions from abductive generation, depth-adaptive decoding, and verifier-based consistency.
- Benchmark Performance: The evaluation covers Com2Sense, CSQA 2.0, and CREAK, which use binary QA formats but require substantial knowledge and robust reasoning.The method is compared with few-shot prompting methods and supervised models.
- Benchmark Performance: MAIEUTIC PROMPTING significantly outperforms all prompting methods across all three benchmarks.The comparison uses GPT-3 prompting baselines and supervised systems, including T5, UnifiedQA, and Unicorn.
- Benchmark Performance: MAIEUTIC PROMPTING is the only prompting method that exceeds the smallest supervised baseline on Com2Sense and CREAK.The result shows that an off-the-shelf LM can achieve performance comparable to a large fine-tuned LM by adding the inference algorithm.
- Robustness Analysis: The performance gap widens on pairwise and contrast-set accuracy, indicating robustness against semantic perturbations.These metrics pair similar-surface questions whose answers should be opposite.
- Robustness Analysis: Across few-shot example sets and permutations, Maieutic Prompting is more stable than other methods and has slightly less variance.Figure 4 compares mean and standard deviation of Com2Sense dev accuracy across three example sets and five permutations.
- Ablation Study: Removing abductive generation reduces performance by 4%, while replacing verifier-based consistency with likelihood-based consistency reduces accuracy by about 7%.The verifier supplies more accurate relations, although the method remains competitive without it.
- Robustness Analysis: Accuracy saturates beyond a threshold of maieutic-tree depth and width, attributed to topic drift and overlapping generated knowledge.The analysis varies maximal depth and width on the Com2Sense dev set.
- Human Evaluation: Human evaluation found that over 99% of true explanations were grammatically perfect, while many incorrect-answer cases still contained factual or helpful explanations.Among incorrectly answered examples, 42% of true explanations were perfectly factual and 23% were completely helpful.
5 Related Work
Related work uses explanations to promote reasoning and interpretability, but language-model explanations can be logically inconsistent and factually unreliable. Maieutic Prompting instead makes explanations intrinsic to inference and applies MAX-SAT to noisy neural generations in an unsupervised setup.
- Explanation-Based Prompting: Prior work trains models to generate natural-language explanations or produces templated-query answers and distantly supervised rationales.These approaches support model reasoning and explanation generation.
- Explanation-Based Prompting: Explanation-based prompting incorporates generated explanations into in-context learning to promote reasoning.Related methods include Chain of Thought, Self-Consistency, and Generated Knowledge Prompting.
- Interpretability: Unlike post-hoc or parallel explanations, Maieutic Prompting uses intrinsic explanations that explicitly participate in inference.The explanations are part of the reasoning process rather than merely accompanying the answer.
- Motivation: Language-model explanations are often logically inconsistent and not factually grounded, reflecting broader generative-model limitations and sensitivity to semantic perturbations.These observations motivate methods that address explanation reliability.
- Neuro-Symbolic Methods: Maieutic Prompting generalizes neuro-symbolic approaches in an unsupervised setup by using MAX-SAT to select a true subset from noisy neural generations.This combines symbolic consistency reasoning with language-model generation.
6 Conclusion
Maieutic Prompting generates a logically related tree of explanations and finds truth values that maximize satisfied relations. The paper reports competitive and robust performance while providing intrinsic interpretations of inference.
- Conclusion: Maieutic Prompting systematically generates a tree of logically related explanations and finds truth values that max-satisfy their relations.The method is inspired by the Socratic way of conversation.
- Conclusion: Empirical results show that Maieutic Prompting is competitive and robust compared with diverse baselines while providing intrinsic interpretations of inference.The conclusion summarizes the method’s reported evaluation outcomes.
Limitations
The paper identifies limits in its current relationship modeling and task coverage, while pointing to broader relational extensions as future work.
- The experiments are limited to validating a given statement, with multiple-choice QA proposed as a future extension.
- MAIEUTIC PROMPTING currently models relationships among nodes within a single maieutic tree.
- The scope could be generalized so knowledge generated for one question serves as evidence for another.
A Tree Generation Algorithm
The evaluation covers three commonsense reasoning and fact-verification benchmarks with adversarial or long-tail questions.
- MAIEUTIC PROMPTING is evaluated on Com2Sense, CSQA 2.0, and CREAK.
- Com2Sense and CSQA 2.0 contain adversarial commonsense questions designed to mislead a proxy model.
- CREAK combines commonsense reasoning with accurate fact retrieval and includes long-tail questions.
C Multi-hop Reasoning on StrategyQA
The StrategyQA evaluation adapts both Chain-of-Thought and Maieutic Prompting for multi-hop reasoning by decomposing questions into smaller subquestions.
- The StrategyQA evaluation uses the dev split to assess generalizability to multi-hop reasoning.
- Both Chain-of-Thought and Maieutic Prompting first decompose each original question into 2-3 minor questions.
- The adapted procedures generate an explanation and answer after decomposing the original question, and are denoted Multi-hop in Table 5.
- Maieutic Prompting yields promising improvement over standard and Chain-of-Thought prompting in this multi-hop setting.
E ET0 ET1 ET1T0 ET2 EF0T0 EF1 EF2F0 E ET1F0 EF0 EF1F0 EF2
The example presents a maieutic tree with proposition truth-values and reports an inferred answer matching the ground truth.
- The example lists generated propositions associated with True and False branches.
- The example reports True as the inferred answer and True as the ground truth.
- Figure 7 shows a generated maieutic tree together with assigned truth-values for its propositions.
E ET0T0 ET1 ET2 ET2T0 EF0 EF0T0 EF0F0 EF1 E ET0 ET2F0 EF1T0
The figures present examples of correct and incorrect inference by Maieutic Prompting, including cases where the inferred answer matches or conflicts with the ground truth.
- One example reports an inferred answer of False matching the False ground truth.
- Figures 8, 9, and 10 are labeled as examples of correct or incorrect inference by Maieutic Prompting.
- Another example reports an inferred answer of True despite a False ground truth.
- A further example reports an inferred answer of False despite a True ground truth.