Source-linked AI summary
Prompting Contrastive Explanations for Commonsense Reasoning Tasks
Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Luke Zettlemoyer, Hannaneh Hajishirzi
TL;DR
Commonsense reasoning models often provide little interpretable evidence for decisions based on implicit knowledge. The paper uses specialized prompts to generate contrastive explanations and conditions predictions on them. This improves performance on two benchmarks, earns higher human relevance judgments, and enables faithfulness evaluation, while the reported experiments do not use GPT-3 as an explainer.
Problem
PLMs perform well on commonsense reasoning but provide little human-interpretable evidence of the implicit knowledge and reasoning behind their predictions.
Method
Specialized contrastive prompts elicit explanations comparing answer alternatives, and a task model conditions its prediction on the original input and generated explanation.
Results
Contrastive explanations improve performance on two commonsense benchmarks, are judged more relevant by humans, and support faithfulness evaluation through contrast perturbations.
Takeaways & Limitations
Contrastive explanations provide useful evidence for commonsense decisions and can reveal whether models rely on the intended contrast when making predictions.
Takeaways & Limitations
The evaluation did not use GPT-3 as an explainer because the authors did not have access to it.
Abstract
from arXiv · showhide
Many commonsense reasoning NLP tasks involve choosing between one or more possible answers to a question or prompt based on knowledge that is often implicit. Large pretrained language models (PLMs) can achieve near-human performance on such tasks, while providing little human-interpretable evidence of the underlying reasoning they use. In this work, we show how to use these same models to generate such evidence: inspired by the contrastive nature of human explanations, we use PLMs to complete explanation prompts which contrast alternatives according to the key attribute(s) required to justify the correct answer (for example, peanuts are usually salty while raisins are sweet). Conditioning model decisions on these explanations improves performance on two commonsense reasoning benchmarks, as compared to previous non-contrastive alternatives. These explanations are also judged by humans to be more relevant for solving the task, and facilitate a novel method to evaluate explanation faithfulfness.
1 Introduction
PLMs perform well on commonsense reasoning but provide little interpretable evidence, motivating contrastive prompts that compare plausible answers and condition predictions on generated explanations. The approach improves two benchmarks, human-rated relevance, and faithfulness evaluation.
- PLMs achieve strong commonsense reasoning performance but provide little human-interpretable evidence of the knowledge or reasoning underlying their predictions.
- Contrastive explanations focus on the distinguishing attributes that make one plausible answer more likely than an alternative, rather than exhaustively explaining one answer.
- The method prompts an explainer PLM to generate contrastive text, then conditions another PLM on the original input and explanation to predict the answer.
- The approach improves performance over two existing explainability methods on WSC and PIQA, with zero-shot gains of up to 11% over Shwartz et al. (2020).Human evaluations find the explanations more useful, and contrast-flipping supports faithfulness evaluation.
2 Related Work
Related work includes free-form and prompt-based explanations, counterfactual reasoning, and extractive rationales. The paper distinguishes its unsupervised specialized prompts from these approaches by generating well-formed contrastive explanations for commonsense tasks.
- Extractive rationales are inadequate for tasks requiring implicit commonsense knowledge because such tasks require reliance on PLMs’ embedded parametric knowledge.
- Prior work uses free-form explanations for commonsense tasks, but these explanations are ambiguous, incomplete, and expensive to collect and evaluate.
- The proposed method differs from Latcinnik and Berant (2020) by using specialized prompts to generate well-formed human-interpretable explanations without additional supervision.
- Unlike Shwartz et al. (2020), which uses fixed generic clarification questions, this work elicits instance-specific contrastive information relevant to the classification decision.
- Contrastive reasoning is related to counterfactual explanations used for model evaluation and training, although much prior work explicitly constructs counterfactual cases.
3 Contrastive Explanations
Contrastive explanations answer why one fact occurred rather than a foil by identifying differences in relevant attributes. Multiple-choice commonsense tasks support this framing, and annotators frequently use explicit fact–foil contrasts.
- A contrastive explanation answers “Why P rather than Q?” by explaining why a fact occurred instead of a possible foil.
- Good explanations identify differences between fact and foil along relevant attributes, rather than merely stating that the fact has an attribute.
- Contrastive explanations are computationally efficient because they focus on causes distinguishing P from Q instead of exhaustively listing causes for P.
- WSC tests pronoun coreference using everyday knowledge, while PIQA asks which of two solutions better achieves a physical goal; both expose explicit fact–foil alternatives.
4 Our Approach
The approach generates contrastive explanations from customized prompts, then uses those explanations to score competing answers. It combines template-based prompting, language-model infilling, and explanation-marginalized task prediction.
- 4.1 Contrastive Templates: Contrastive templates were abstracted from annotator explanations of 250 Winogrande and PIQA instances, retaining patterns used at least 10 times.Templates contain placeholders for the fact and foil, plus possible contrasting properties; a separate evaluation found over 82% coverage on 100 examples.
- 4 Our Approach: For each instance, the method generates explanations from a list of templates and feeds the resulting candidates into the task model, aggregating scores over templates.Figure 1 depicts the conversion of (c, a1, a2) into a prompt for the explainer and the subsequent scoring of each answer with generated explanations.
- 4.2 Generating Explanations: The pipeline first constructs a neutral context and customized contrastive template, then an explainer PLM infills the template to produce candidate explanations.Templates are customized with the two answer choices, while the remaining gaps are filled using maximum-likelihood candidate phrases from top-K decoding.
- 4.3 Task Model: The second stage scores each answer-explanation pair with a task PLM and marginalizes over multiple explanations before selecting between the two answers.The task model uses φ(c, a, e) as input to a logistic regression classifier; during zero-shot inference, answer scores are aggregated across explanations.
5 Experimental Setup
The experiments compare the approach with context-only, unconstrained-generation, and self-talk baselines using several PLM configurations and commonsense reasoning datasets. The setup covers Winograd-style pronoun resolution and physical commonsense question answering.
- 5.1 Baselines: Baselines include a context-only model without explanations, unconstrained free-form explanation generation, and Self-Talk clarification prompts.The unconstrained-generation comparison uses a beam-decoded hypothesis and does not fine-tune the explainer PLM; Self-Talk uses a PLM as an answer scorer and another as a knowledge source.
- 5.2 Implementation details: The study uses BART-Large and T5 as explainer PLMs and GPT2-XL as the task model for fair comparison across models.GPT2-XL was selected because it was the best-performing PLM used by Shwartz et al. for WSC and PIQA.
- 5.2 Implementation details: The Winograd experiments include SuperGLUE WSC, Winogrande, and WINOGENDER, covering pronoun resolution and gender-bias diagnostics.SuperGLUE WSC contains 285 anaphora-resolution examples; answer choices are used directly, with dataset-specific construction of neutral and substituted contexts.
- 5.2 Implementation details: PIQA supplies two answer choices that usually differ in a substring, which are converted into contrasting answer components and question-answer contexts.The neutral answer combines the alternatives with “or,” while the differing substrings serve as the two answer choices.
6 Experimental Results
The experiments show that contrastive explanations improve commonsense task performance, receive stronger human judgments than self-talk explanations, and encode task-relevant knowledge that affects predictions. Flipping or abstracting answer identities further tests whether models use the explanatory contrast rather than only the original input.
- 6.1 Task Performance: Contrastive explanations outperform clarification-question and unconstrained-generation baselines, with T5-Large already surpassing self-talk despite using a smaller explainer model.
- 6.1 Task Performance: T5-11B achieves the highest reported zero-shot results for an open-sourced model on Winogrande, PIQA, and WSC.
- 6.1 Task Performance: Training-data scarcity widens the performance gap between the contrastive approach and the Context-Only baseline, whereas fine-tuning produces smaller improvements overall.
- 6.2 Human Evaluation: Human judges overwhelmingly prefer contrastive explanations to self-talk explanations for relevance, factual correctness, and helpfulness, although infilling noise may reduce grammaticality.
- 6.3 Analysis: Humans judge flipped explanations to imply the opposite label in a majority of cases, suggesting that the explanations often capture properties on which labels genuinely depend.
- 6.3 Analysis: Flipping the contrastive explanation can reduce accuracy by up to 8%, indicating that models use some contrastive knowledge when making predictions.
- 6.3 Analysis: With abstracted answers, conditioning on both context and explanation raises Winogrande accuracy from 63.2 to 70.4, compared with 79.1 for the original non-abstracted model.
- 6.3 Analysis: On CommonsenseQA, contrastive explanations are useful and approach the zero-shot performance of the ConceptNet-based state of the art, while self-talk significantly hurts performance.
7 Conclusion
The paper shows that PLMs can generate human-inspired contrastive explanations and that conditioning decisions on them improves commonsense reasoning. These explanations also support human relevance judgments and faithfulness evaluation, while broader open-ended applications remain future work.
- Contrastive explanations improve performance on two commonsense reasoning benchmarks when used to condition model decisions.The approach is flexible enough for multiple-choice tasks with many foils.
- Humans judge the generated explanations highly relevant and helpful compared with prior work.
- Flipping or abstracting facts and foils shows that explanations encode substantial information relevant to classification decisions.In many cases, models rely on the contrast in the expected way.
- Applying contrastive reasoning to a wider variety of open-ended tasks remains an open challenge.
A Generating Contrastive Templates
The template-generation procedure uses contrastive patterns tailored to dataset entities and grammatical properties. Filtering removes prompts that do not fit the example’s entity type, number, or commonsense domain.
- Generating Contrastive Templates: The method detects fact and foil entities in WSC and applies dataset-specific transformations for Winogrande and PIQA.
- Generating Contrastive Templates: Case filtering removes templates that become ungrammatical for singular or plural fact and foil entities.
- Generating Contrastive Templates: WSC templates about personal characteristics are excluded when examples lack PERSON entities, while some temporal, use-case, and spatial patterns are omitted when PERSON entities occur.
- Generating Contrastive Templates: PIQA removes all templates about personal characteristics because the dataset concerns physical commonsense.
B Human Evaluation
Human annotations were collected from qualified crowdsourced workers under geographic, experience, and compensation requirements, then aggregated by majority vote. Agreement was moderate.
- Human Evaluation: Annotations were collected from US, UK, or Canadian workers with 99% approval across at least 5,000 prior tasks.Workers were paid $0.30 per HIT, targeting approximately $15 per hour.
- Human Evaluation: Each annotation was aggregated from three workers using majority vote, yielding moderate Fleiss kappa agreement.
C Hyperparameters
The appendix specifies explainer and task-model settings, comparison procedures, generation baselines, and example tables for contrastive prompting and evaluation.
- Hyperparameters: T5 uses special blank tokens and BART uses four MASK tokens to encourage multi-word contrastive explanation generation.Top-K decoding uses beam size 200, with maximum output lengths of 20 tokens for T5 and 100 for BART.
- Hyperparameters: The task PLM is fine-tuned for 20 epochs with BertAdam, learning rate 2e −5, batch size 8, and dropout 0.1.
- Hyperparameters: Self-Talk generates clarification questions and answers with nucleus sampling, then scores each answer using the clarification pair that maximizes language-model likelihood.It conditions prediction on one clarification question-answer pair per answer choice.
- Hyperparameters: Tables document Winogrande and PIQA substitutions, abstractive evaluation inputs, and the complete contrastive-pattern inventory.
- Hyperparameters: The unconstrained explanation baseline uses beam size 200, maximum output length 20, and early stopping.