Source-linked AI summary
Explaining Question Answering Models through Text Generation
Veronica Latcinnik, Jonathan Berant
TL;DR
End-to-end language-model QA can answer knowledge-intensive questions, but it is difficult to see which stored knowledge supports each prediction. The paper generates a textual hypothesis for a classifier, using training constraints to keep that hypothesis meaningful and natural, and achieves performance comparable to end-to-end models while exposing useful information for inspection.
Problem
End-to-end QA models make it difficult to identify the knowledge in a language model that supports a correct prediction.
Method
A generator produces a textual hypothesis from the question, and a classifier uses that hypothesis to select the answer with weak downstream-task supervision.
Results
Across several tasks, the approach reaches performance comparable to end-to-end models while producing hypotheses that reveal knowledge used for answering.
Takeaways & Limitations
Inspectable hypotheses help users and practitioners understand model behavior and distinguish missing world knowledge from language-understanding difficulties.
Takeaways & Limitations
Jointly training the supervised generator with the LM-based classifier resulted in poor performance because of optimization issues.
Abstract
from arXiv · showhide
Large pre-trained language models (LMs) have been shown to perform surprisingly well when fine-tuned on tasks that require commonsense and world knowledge. However, in end-to-end architectures, it is difficult to explain what is the knowledge in the LM that allows it to make a correct prediction. In this work, we propose a model for multi-choice question answering, where a LM-based generator generates a textual hypothesis that is later used by a classifier to answer the question. The hypothesis provides a window into the information used by the fine-tuned LM that can be inspected by humans. A key challenge in this setup is how to constrain the model to generate hypotheses that are meaningful to humans. We tackle this by (a) joint training with a simple similarity classifier that encourages meaningful hypotheses, and (b) by adding loss functions that encourage natural text without repetitions. We show on several tasks that our model reaches performance that is comparable to end-to-end architectures, while producing hypotheses that elucidate the knowledge used by the LM for answering the question.
1 Introduction
The paper addresses the difficulty of explaining what knowledge language models use in multi-choice question answering by generating an inspectable textual hypothesis before classification. It evaluates weakly supervised generation across synthetic hypernym extraction, CSQA, and zero-shot QASC, finding comparable performance to end-to-end models while revealing model knowledge and errors.
- End-to-end QA models can perform well yet leave the knowledge behind their predictions difficult to inspect, motivating explainable intermediate representations.The paper notes that models may rely on shortcuts rather than language understanding.
- The proposed architecture generates a short textual hypothesis from each question, then uses that hypothesis with a classifier to select the answer.The hypothesis is intended to expose relevant knowledge extracted from the language model.
- Training uses weak downstream-task supervision rather than target explanations, framing the approach as controlled text generation for hypotheses useful to question answering.The paper contrasts this with methods trained against explicitly provided explanation targets.
- The model addresses discrete-generation optimization with the Gumbel-Softmax straight-through estimator and constrains the classifier to encourage meaningful hypotheses.These choices respond to nondifferentiability and to the risk that the classifier ignores or repurposes generated text.
- Across synthetic hypernym extraction, CSQA, and zero-shot QASC, the approach reaches performance comparable to end-to-end models while producing useful hypotheses.Generated hypotheses are shown to help analyze errors such as missing world knowledge versus language-understanding difficulties.
2 Background
The background defines multi-choice question answering as selecting the correct candidate from a list and describes the standard transformer-based end-to-end solution. Although effective on commonsense tasks, that setup keeps the knowledge used for prediction internal to the model, motivating a language-based intermediate representation.
- Multi-choice question answering selects the correct answer from a candidate list using question-candidate-answer training triples.
- A standard end-to-end model encodes each question-candidate pair with a pretrained transformer, scores candidates, and applies softmax training against the correct answer.The score is computed from a summarized contextual representation and a learned weight vector.
- This architecture has succeeded on commonsense reasoning tasks, but its reasoning remains internal, making the knowledge used for prediction difficult to determine.
- The paper instead exposes LM knowledge by having the model generate it in language, without assuming beforehand which words should be produced.The generated information is intended as an inspectable representation rather than primarily as a performance improvement.
3 Model
The model generates an inspectable textual hypothesis from a question, then uses it with answer candidates to predict the answer. Training combines weak downstream supervision with mechanisms that encourage hypotheses to remain useful, meaningful, natural, and non-repetitive.
- 3 Model: The generator outputs a textual hypothesis c from question q, and the classifier uses q, c, and answer candidates to predict the answer.The hypothesis is an inspectable intermediate representation rather than internal reasoning hidden inside an end-to-end model.
- 3 Model: Straight-through Gumbel-softmax makes optimization possible despite discrete generator outputs by providing a differentiable relaxation during training.The estimator creates a mismatch between training and test-time decoding, and generator and classifier vocabularies must therefore be tied.
- 3.2 Classifier Expressivity: A simple similarity classifier encourages hypotheses to resemble correct answers more than distractors, while a stronger LM classifier preserves performance.The similarity classifier does not use the question, forcing the hypothesis to carry information for answering; the LM-based classifier can still receive the question and hypothesis.
- 3.3 Explainability: KLD regularization limits drift from the original language model, and an unlikelihood repetition penalty discourages tokens that already appeared in the generated prefix.These objectives target meaning preservation and repetitive text, respectively.
- 3.3 Explainability: Top-K sampling passes multiple distinct likely tokens to the classifier, widening the information channel without evaluating a full beam of hypotheses.The method outputs top-K tokens in the straight-through estimator's forward pass and performs one classifier step.
- 3.4 Supervised Generator: The supervised generator variant uses the correct answer as a token-level target, but joint training with the LM classifier produced poor performance because of optimization issues.Unlike weak supervision, this variant receives supervision at every decoding step.
4 Experiments
Across synthetic hypernym extraction, CSQA, and zero-shot QASC transfer, the hypothesis-generating models achieve competitive accuracy while exposing inspectable information used for prediction. Experiments also reveal trade-offs between accuracy, naturalness, repetition, and the kinds of errors the hypotheses can diagnose.
- Synthetic hypernym extraction: 84.0 accuracy versus 86.5 for END2END shows that the model converts LM knowledge into language tokens on synthetic hypernym extraction.The task contains 7,625 questions with six hypernym categories, and the generator produces one token.
- Similarity classifier experiments: 63.7 accuracy for INTERACTION establishes an upper bound for the similarity classifier, while the best TOP-K = 3 ST model reaches 58.0 accuracy.The best model is 5.7 points below NOINTERACTION, attributed to the narrow word-based channel and difficult optimization.
- Similarity classifier experiments: Adding KLD and especially repetition loss dramatically reduces repeated words but also decreases accuracy, while KLD does not improve phrase coherence.The similarity classifier’s preference for words resembling the correct answer contributes to repetitions in multi-step decoding.
- Supervised generation comparison: SUPGEN produces more natural text but lowers accuracy by 7.2 points, demonstrating a trade-off between QA optimization and textual naturalness.The QA-trained models tend to produce list-like text, whereas independently supervised generation yields more natural hypotheses.
- Explainability analysis: Hypotheses support error analysis: both classifiers are correct in 49% of cases with a 0.94 reasonableness score, whereas both are wrong in 23% with a 0.21 score.Unreasonable hypotheses expose missing knowledge and semantic errors, which together account for 54% of unreasonable hypotheses.
- Explainability analysis: When the LM-based classifier is correct but the similarity classifier is wrong, hypotheses score 0.55 and are often reasonable, with two-thirds of errors involving distractor handling.The model can generate a reasonable hypothesis that is more similar to a distractor than to the gold answer.
- Overall results: Jointly trained models reach performance comparable to end-to-end models while providing hypotheses that expose information used for prediction.On CSQA, the hypotheses strongly affect predictions, and the analysis uses them to inspect model knowledge and errors.
5 Discussion and Related Work
Prior work studies human-generated explanations and explanation faithfulness, while MC-QA remains vulnerable to distractor-based shortcuts. This paper positions its hypothesis layer as an explanation whose influence on prediction can be experimentally tested.
- Explanation datasets: Human-generated explanation datasets support explanation generation or selection, but their explanations are expensive to collect.
- Explainability: Table 5 organizes bad hypotheses into missing knowledge and semantic errors, plus reasonable hypotheses that fail to fit particular distractors.
- Explainability: The paper distinguishes explanations that influence predictions from explanations that merely describe them, relating its similarity-based hypotheses to attention-like explanations.
- Multi-choice QA: MC-QA simplifies evaluation but may allow models to exploit weaknesses in distractors rather than demonstrate genuine question answering.
6 Conclusion
The paper proposes an LM-based MC-QA model that generates natural-language hypotheses as inspectable intermediate representations. Its performance is on par with end-to-end models, while downstream supervision makes the approach generalizable beyond MC-QA.
- The model generates natural-language text as an intermediate step for understanding knowledge extracted from the LM.
- Performance is on par with end-to-end models while providing practitioners and users an inspectable layer.
- Because training uses downstream application signal only, the approach can generalize to scenarios where LM text is trained for a downstream application.
A Examples of the Impact of the Hypothesis on Prediction
Tables 8 and 9 provide examples of how generated hypotheses affect prediction, contrasting cases where hypotheses help with cases where removing them fixes errors.
- Table 8 presents examples where TOP-K = 3 ST hypotheses are useful and zeroing them out causes an error.
- Table 9 presents examples where TOP-K = 3 ST hypotheses hurt the model and zeroing them out fixes an error.
B Hypernym Extraction
The hypernym-extraction appendix evaluates Gumbel-Softmax and the straight-through estimator against an end-to-end model. The straight-through estimator is crucial on this synthetic task, while Gumbel-Softmax helps mainly without it.
- The straight-through estimator is crucial even on the simple hypernym-extraction task, consistent with prior observations about train-test discrepancies [Havrylov and Titov, 2017].
- Gumbel-Softmax improves results without the straight-through estimator but has little effect when the straight-through estimator is used.
- Table 10 compares END2END with the model's GS and ST variants on hypernym extraction.