Source-linked AI summary
Explaining Answers with Entailment Trees
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pipatanangkura, Peter Clark
TL;DR
The paper addresses the limited ability of QA explanation systems to show how evidence leads to an answer. It represents explanations as multistep entailment trees, creates ENTAILMENTBANK, and trains generative models across three input settings. Results show partial success, especially with relevant facts included, alongside indications of cross-domain generalization and clear error modes.
Problem
QA explanation systems rarely show the reasoning chain from known evidence to an answer, limiting understanding and error diagnosis.
Method
The paper builds ENTAILMENTBANK, defines three entailment-tree generation tasks, and trains EntailmentWriters to produce multistep, multi-premise derivations.
Results
The models partially solve the tasks, with 35% of Task 1 trees perfectly matching the gold tree when gold leaves are provided, and show indications of generalization to other domains.
Takeaways & Limitations
Multistep entailment trees and ENTAILMENTBANK provide a resource for developing richer, more systematic explanations that can help users understand and debug answers.
Takeaways & Limitations
Approximately 20% of analyzed cases selected correct leaves but produced invalid entailment steps, motivating more reliable entailment procedures or validation modules.
Abstract
from arXiv · showhide
Our goal, in the context of open-domain textual question-answering (QA), is to explain answers by showing the line of reasoning from what is known to the answer, rather than simply showing a fragment of textual evidence (a "rationale'"). If this could be done, new opportunities for understanding and debugging the system's reasoning become possible. Our approach is to generate explanations in the form of entailment trees, namely a tree of multipremise entailment steps from facts that are known, through intermediate conclusions, to the hypothesis of interest (namely the question + answer). To train a model with this skill, we created ENTAILMENTBANK, the first dataset to contain multistep entailment trees. Given a hypothesis (question + answer), we define three increasingly difficult explanation tasks: generate a valid entailment tree given (a) all relevant sentences (b) all relevant and some irrelevant sentences, or (c) a corpus. We show that a strong language model can partially solve these tasks, in particular when the relevant sentences are included in the input (e.g., 35% of trees for (a) are perfect), and with indications of generalization to other domains. This work is significant as it provides a new type of dataset (multistep entailments) and baselines, offering a new avenue for the community to generate richer, more systematic explanations.
1 Introduction
The paper frames explanation as reconstructing how evidence leads to an answer, then introduces entailment trees, ENTAILMENTBANK, three tasks, and generative baselines for this goal.
- Motivation: Explanation systems often provide supporting rationales but rarely show the reasoning chain from known evidence to an answer.Without that chain, understanding responses and locating errors is difficult.
- Approach: The proposed explanations are multistep entailment trees composed of individual multi-premise textual-entailment steps.The trees connect known facts through intermediate conclusions to a question-answer hypothesis.
- Dataset: ENTAILMENTBANK contains 1,840 expert-annotated multistep entailment trees for QA and is presented as the first dataset of its kind.The dataset includes trees averaging 6.6 nodes and 2.7 entailment steps.
- Tasks: The dataset defines three tasks using gold leaves, leaves plus distractors, or a full corpus as input.These settings progressively increase the challenge of generating a valid entailment tree.
- Models: EntailmentWriters are generative models trained for the tasks, with reasonable trees generated especially when necessary raw facts are provided.The formulation separates derivation correctness from decisions about which explanation content to show users.
2 Related Work
Prior QA explanations mainly support lookup answers or simplify inference chains; this paper instead makes multi-step reasoning and entailment provenance explicit.
- Prior explanations: Existing QA explanations include answer-bearing sentences, attention maps, synthesized connecting phrases, or answer-location patterns.These methods primarily address lookup questions by explaining where or how an answer was found.
- Inference explanations: For inference questions, prior datasets often collect supporting sentences without specifying how they combine or focus on one-hop chains.The paper generalizes this setting to multi-step entailment trees.
- Entailment structure: ENTAILMENTBANK uses multi-premise entailments, where two or more sentences entail a hypothesis, rather than mainly single-sentence paraphrase entailments.Its trees also record which sentences support each entailment.
- Contribution: The dataset chains multiple entailments into a hypothesis-directed tree, making reasoning steps explicit rather than presenting separate single-step entailments.The approach applies generative techniques previously used for deductive proofs.
3 The ENTAILMENTBANK Dataset
ENTAILMENTBANK is a QA dataset of expert-authored multistep entailment trees, built from ARC questions and WorldTree knowledge to represent detailed, diverse reasoning.
- Dataset composition: ENTAILMENTBANK contains 1,840 entailment trees for QA pairs and a general corpus of domain-specific and general knowledge.The trees show how QA pairs are entailed from relevant sentences; the corpus supports both simplified and full-corpus generation tasks.
- Dataset composition: The dataset uses ARC multiple-choice science questions and a science-and-general-knowledge corpus derived from WorldTree V2.WorldTree’s grade-school science focus made it the source for the corpus.
- Annotation guidelines: Annotators were trained to construct entailment trees with entailed, fine-grained, explicit, compositional, and QA-relevant steps.Each step should encode a single inference, while complex conclusions are composed from simpler facts.
- Tool and authoring procedure: The authoring workflow combines a worksheet, a pool of top-ranked corpus facts, interactive tree construction, and separate review of entailment steps.Authors could add missing science or scenario facts; completed trees took approximately 20 minutes per question on average.
- Overall dataset: The dataset includes 1,840 randomly selected ARC questions, 5,881 discrete entailment steps, and approximately 600 paid work hours of annotation.Each tree averages 7.6 nodes across 3.2 entailment steps, with a broad range of problem sizes.
- Dataset analysis: Approximately 33% of sampled entailments require domain-specific rules, while the full analysis identifies six diverse reasoning categories.Other categories include substitution, further specification or conjunction, and additional property-, class-, and order-based inferences.
4 Task Definitions
The paper defines entailment-tree generation as producing a valid tree from a hypothesis and varying amounts of relevant or irrelevant knowledge, while approximating validity through gold-tree comparison.
- Task formulation: Each task takes a hypothesis H formed from a question and answer, plus sentences S expressing relevant or irrelevant knowledge.The desired tree has selected input sentences as leaves, new intermediate conclusions, and H as its root.
- Task formulation: A tree is valid when every node is entailed by its children, and the three tasks differ by the size and composition of S.The tasks progress from relevant leaf sentences to distractors and then a full corpus.
- Validity approximation: Automatic evaluation assumes the input contains all gold leaf sentences and treats the gold tree, including valid reorderings, as the only constructable tree.This approximation is considered more reasonable for Tasks 1 and 2 than for Task 3.
- Three tasks: Task 1 supplies H, QA, and gold leaf sentences; Task 2 adds 15–20 distractor sentences; Task 3 supplies H, QA, and the corpus C.The full-corpus setting uses WorldTree plus additional annotator-created science facts.
- Scope and limitation: Task 3 may require question-specific scenario facts not present in the corpus, and its baseline does not derive those facts.The paper explicitly identifies this as a limitation of the Task 3 baseline.
5 Model
EntailmentWriter uses T5-based generative models to output linearized entailment trees, with retrieval added for the full-corpus task because the corpus exceeds T5’s token limit.
- Model: The paper trains three T5-based EntailmentWriter models, one for each task.The models are inspired by the All-at-once sequence-to-sequence model in ProofWriter.
- Entailment tree encoding: Trees are linearized from leaf facts toward the hypothesis using identifiers for input sentences, intermediate conclusions, and the root.The encoding uses “&” for conjunction and “->” for entailment, while newly generated intermediate sentences follow their identifiers.
- Full-corpus model: For Task 3, the model retrieves 25 corpus sentences using the hypothesis as a query before generating the encoded tree.Retrieval is added because the full corpus exceeds T5’s token limit.
- Training: The largest T5-11B model is fine-tuned for 40k steps with batch size 8, using the checkpoint with the highest development score.The experiments use default hyperparameters from the T5 library.
6 Experiments
EntailmentWriter models are evaluated on generating entailment trees across three increasingly difficult input settings, using aligned tree-level and component-level metrics. Performance is strongest when relevant facts are supplied and declines for full-corpus generation, with retrieval becoming an additional constraint.
- Evaluation Metrics: The evaluation aligns predicted and gold trees, then scores leaf selection, entailment-step structure, intermediate correctness, and strict overall perfection.F1 and BLEURT measure element overlap, while AllCorrect requires every relevant element to be correct.
- Results: Over one-third of Task 1 trees perfectly match the gold tree when only gold leaves are provided.Manual analysis suggests approximately 20% of additional low-scoring trees are valid but differently structured, so the metric underestimates validity.
- Results: 89% leaf F1 and nearly half of trees with perfectly selected leaves show that Task 2 models identify relevant facts despite 30-sentence inputs.Task 2 step F1 is 41%, compared with 51% for Task 1, indicating only a modest structural decline despite added distractors.
- Results: Task 3 results are lower, indicating that full-corpus generation is difficult and that retrieval can constrain tree construction.The baseline retrieves 66.1% of required gold sentences on average, and the authors identify retrieval as critical when the full corpus cannot be ingested.
- Error Analysis: Task 2 scores drop significantly as the number of entailment steps in the gold tree increases.The analysis groups results by proof length, showing that longer multistep proofs remain more difficult.
7 Generality
The paper tests whether models trained on ENTAILMENTBANK can generate entailment-tree explanations beyond the original setting. Results show partial transfer to other domains and some success when generating without context or interactively.
- Out-of-domain transfer: The eQASC experiment applied Task 2 EntailmentWriter to questions with ten retrieved sentences and a hypothesis.eQASC supplies valid one-step entailments from pairs of retrieved sentences to the hypothesis.
- Out-of-domain examples: Figure 5 presents three zero-shot, out-of-domain explanations in which both leaves and intermediate sentences were generated.The examples illustrate fully generated explanations rather than trees built from supplied sentences.
- Out-of-domain transfer: 67% leaf F1 and 26% Overall AllCorrect were achieved on eQASC without fine-tuning, indicating positive out-of-domain transfer.The comparison used the best-matching gold tree, and eQASC does not list all valid entailments.
- No-context generation: Approximately 35% of no-context trees generated for Challenge300 were valid and non-vacuous.The no-context model received only a QA pair and generated both leaves and intermediate sentences.
- Interactive generation: Interactive generation was trained by shredding trees into one-deep trees whose intermediate nodes became new hypotheses to prove.Users can select premises to drill down into recursively during explanation generation.
8 Summary and Conclusion
The paper frames explanations as multistep entailment trees and introduces ENTAILMENTBANK to support this formulation. Baselines indicate that generation is possible, especially with raw facts supplied, with signs of cross-domain generalization and potential support for understanding and debugging answers.
- Summary and Conclusion: The paper aims to enable richer, more systematic machine-generated explanations through multistep entailment trees and ENTAILMENTBANK.ENTAILMENTBANK is described as the first large dataset of such trees.
- Summary and Conclusion: Baseline results show entailment-tree generation is possible, particularly when necessary raw facts are included in the model input.The models were trained on ENTAILMENTBANK for science-question answering.
- Summary and Conclusion: The paper reports indications that models trained on ENTAILMENTBANK can generalize to other domains.The evidence is presented as an indication rather than a full investigation.
- Summary and Conclusion: ENTAILMENTBANK offers a resource for developing richer, more systematic explanations that may help users understand and debug system answers.The conclusion also identifies future opportunities for dialogs exploring a machine’s line of reasoning.
- Dataset construction: The dataset construction uses pools of potentially relevant WorldTree facts selected with relevant-sentence classifiers.BERT and RoBERTa classifiers were trained using additional WorldTree annotations.
- Evaluation: Predicted trees are evaluated after alignment based on shared ancestor leaf sentences and Jaccard similarity between intermediate nodes.Predictions with zero similarity to all gold nodes are aligned to dummy blank-conclusion nodes.
C Training and Model Selection
The paper trains large T5 models for entailment-tree generation and selects checkpoints using development-set Overall AllCorrect scores. The examples illustrate the target reasoning structure, while training uses distant supervision for relevant-fact selection.
- Training: Task 1 and Task 2 use T5 11B models trained with batch size 8 and 512-token input and output block sizes.Training follows the procedure of Khashabi et al. with default hyperparameters except for training steps.
- Relevant-fact selection: Relevant-sentence classifiers use BERT and RoBERTa with WorldTree annotations and random negative facts as distant supervision.WorldTree annotations identify supporting evidence rows, though they do not identify all relevant sentences.
- Entailment-tree example: The chocolate example represents a hypothesis that chocolate shape changes as melting moves material from solid to liquid.The tree connects definite solid-state shape with liquid-phase properties and melting.
- Entailment-tree example: The target trees express reasoning from premises to a hypothesis through intermediate conclusions.The example’s displayed fragments include solid-to-liquid melting and substance-state properties.
- Tree representation: Figure A2 contrasts a gold tree with a valid model-generated tree having different structure and intermediate conclusions.Both trees use H for the hypothesis root and blue nodes for intermediate conclusions.
- Model selection: Checkpoint selection evaluates models trained for up to 40K steps and chooses the best development-set Overall AllCorrect score.The selected checkpoints were 16K steps for Task 1 and 32K steps for Task 2.
- Computational cost: Each Task required 16 hours of training, while inference on 340 test questions took 12 minutes.A large fraction of the reported time was spent saving or loading model checkpoints.
D Tree Structure Variation
The evaluation can penalize valid predicted trees when their structure differs from the gold tree. This structural mismatch causes step scores to underestimate true prediction quality, while larger trees also receive lower scores.
- Evaluation limitation: The authors state that structural scoring cases can underestimate true prediction quality by as much as 20%.The metric accounts for different node ordering and intermediate wording but not all valid structural variations.
- Evaluation comparisons: Tables A2 and A4 provide development results and T5-large test results alongside the main T5-11B evaluation.Both tables are described as analogous baseline-result reports.
- Tree size: Table A3 reports that Task 2 scores drop rapidly as the number of entailment steps increases.The breakdown is organized by the number of steps in the gold tree.
- Tree Structure Variation: A valid prediction can receive Step F1 = 0% despite F1 = 100% for leaves when its entailment structure differs from the gold tree.The example reorganizes three premises into an intermediate conclusion before reaching the hypothesis.
- Tree Structure Variation: Figure A2 shows valid gold and predicted trees with identical leaves but different organization, producing leaf F1 = 100% and Step F1 = 0%.The mismatch concerns tree organization rather than the selected leaf sentences.
E Additional Results: T5-large baseline
The T5-large baseline used default training hyperparameters, with model checkpoints selected by Dev-set AllCorrect performance for the two tasks.
- T5-large was trained with default hyperparameters, batch size 64, and 512-token input and output block sizes.Training followed Khashabi et al. (2020).
- Models were evaluated at training checkpoints through 80K steps, sampled every 8K steps.
- Checkpoint selection used the best Dev-set Overall AllCorrect score.
- The selected checkpoints were 48K steps for Task 1 and 32K steps for Task 2.Table A4 reports the corresponding test-set model scores.