PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Explanations: How much do explanations from the teacher aid students?

Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen

arXiv:2012.00893v2cs.CLcs.LG

TL;DR

The paper addresses the lack of clear criteria for evaluating prediction explanations. It measures whether explanations help a student model simulate a teacher, with explanations available during training but not testing. Across text classification and question answering, attribution methods differ quantitatively, and rankings show moderate-to-high agreement across student architectures and learning strategies.

  • Problem

    There is little agreement on what makes a prediction explanation good, and attribution methods often disagree about which features to highlight.

    Source: §1 · p. 1

  • Method

    The framework measures explanation utility by whether teacher-provided explanations improve a student’s ability to simulate the teacher on unseen examples, without test-time explanation access.

    Source: §1 · pp. 1–2, §8 · p. 12

  • Results

    Across text classification and question answering, explanation methods show quantitative differences, with rankings consistent to a moderate-to-high degree across student architectures and learning strategies.

    Source: §1 · p. 2, §8 · p. 12

  • Takeaways & Limitations

    The framework supports automatic, model-agnostic comparison of attribution methods through their utility for student-teacher simulation.

    Source: §1 · p. 2, §8 · p. 12

    The experiments compare explanations only within the same format, leaving cross-format comparisons for future work.

    Source: §6 · p. 11

Abstract

from arXiv · show

While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated. In this work, we introduce a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model. Crucially, the explanations are available to the student during training, but are not available at test time. Compared to prior proposals, our approach is less easily gamed, enabling principled, automatic, model-agnostic evaluation of attributions. Using our framework, we compare numerous attribution methods for text classification and question answering, and observe quantitative differences that are consistent (to a moderate to high degree) across different student model architectures and learning strategies.

1 Introduction

Attribution methods highlight features to explain predictions, but there is little agreement about what makes an explanation good, and methods often select different tokens. The paper proposes evaluating explanations by whether they help students simulate teachers, then compares attribution methods on classification and question answering.

1 Introduction

  • Attribution methods often disagree about which input tokens to highlight, while the literature has little agreement on what constitutes a good explanation.Table 1 measures overlap among top-10% tokens selected by different sentiment-analysis techniques.Source: §1 · p. 1, Table 1 · p. 2
  • Existing evaluations often rely on visual inspection, while some quantitative methods can be gamed, use out-of-distribution examples, or rely on specifically crafted tasks.Source: §1 · p. 1
  • The proposed framework measures explanation utility by whether explanations help a student model better simulate a teacher on future examples.Explanations serve as side information during training; they are unavailable at test time.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • Across text classification, attention-based explanations and integrated gradients are most effective, while vanilla gradient saliency and LIME are least effective.Rankings show moderate-to-high agreement across student architectures and learning strategies.Source: §1 · p. 2
  • For question answering, the authors evaluate student learners using both human-produced explanations and explanations generated automatically by SpanBERT.Source: §1 · p. 2

2 Explanation as Communication

The framework treats explanations as information that helps a student anticipate a teacher’s decisions, measuring their value by gains in student-teacher agreement on unseen examples. Automated evaluation uses machine students, but results depend on student design and require safeguards against explanations that simply reveal the teacher’s weights or outputs.

2.1 An Illustrative Example

  • In the admissions example, explanations are useful when they improve a student’s accuracy in predicting the committee’s decisions on future applications.Generic explanations that add no information about the decision are unlikely to improve simulation.Source: §2.1 · pp. 2–3

2.2 Quantifying Explanations

  • The framework compares student-teacher agreement on test examples after learning from teacher-labeled data with versus without explanations.Students can use explanations during training, but the classifiers predict from the input alone at test time.Source: §2.2 · p. 3

2.3 Automated Teachers and Students

  • Automated evaluation can use machine learners and standard evaluation, but its outcomes depend on student models, learning procedures, and training-set size.The authors experiment with diverse students and different values of n to reduce reliance on any single student.Source: §2.3 · p. 3, §2.2 · p. 3

2.4 Discussion

  • The framework’s measurements remain contingent on the student and how explanations are incorporated, motivating evaluation across varied student designs.Source: §2.3 · p. 3, §2.2 · p. 3

2.3 Automated Teachers and Students

  • To prevent perfect simulation through weight-copying, the authors restrict explanations to human-comprehensible formats and use diverse student architectures.The framework also avoids test-time explanation access, which can let explanations leak the teacher output; machine students permit training on thousands of examples.Source: §2.3 · p. 3, §2.4 · p. 4

3 Learning with Explanations

The section presents two ways to train students with teacher explanations: align student attention with explanations or jointly predict labels and generate explanations. For question answering, it adapts these methods to QED annotations, with an additional loss to use their phrase alignments.

Classification Tasks

  • The student’s training data pairs documents with teacher outputs and explanations, and both training approaches improve performance with human explanations on classification tasks.The authors also explore variants of the approaches for question answering.Source: §3 · p. 4
  • Attention regularization aligns the student’s attention distribution with a distribution over the teacher-highlighted tokens during training.The explanation distribution is uniform over highlighted tokens and assigns a small ϵ elsewhere; continuous importance scores can be converted by selecting the top-k% tokens.Source: §3 · pp. 4–5
  • Multitask learning trains prediction and explanation generation jointly, with shared BERT parameters in the classification experiments.The explanation-generation task is sequence labeling; the authors expect the tasks to benefit each other when they are complementary.Source: §3 · p. 5

Question Answering

  • For question answering, QED explanations pair question phrases with passage phrases, but the two basic approaches use a binary token vector that discards this alignment information.The vector marks tokens appearing in one or more referential equalities; neither attention regularization nor multitask learning uses the phrase alignment after this lossy transformation.Source: §3 · p. 5
  • An added attention-alignment loss uses QED phrase pairs to increase attention from question phrases to their aligned passage phrases.The loss is appended to the standard loss and uses last-layer average attention across tokens in the question phrase and attention heads.Source: §3 · p. 5

4 Human Experts as Teachers

With human-provided explanations, student models improve on sentiment analysis and question answering, though the gains depend on the training approach and number of examples. Corrupting explanations is associated with lower student performance, supporting a relationship between simulation performance and explanation quality.

Results

  • 6 absolute accuracy points for 600 examples: attention regularization improves sentiment-analysis student performance, with a 2.9-point gain for 1200 examples.Multitask learning shows no improvement with 600 examples, but gains 1 point with 900 and 1.4 points with 1200.Source: Table 3 · p. 6, §4 · p. 6

Setup

  • The experiments use expert movie-review rationales and QED annotations, with pretrained BERT-base students fine-tuned on human outputs and explanations.The QED annotations are from Natural Questions and link salient question entities to referential mentions in passages.Source: §4 · pp. 5–6

Results

  • 8.4 F1 points for 500 examples: attention alignment loss improves question-answering student performance; attention regularization gains 2.3 F1 points.The gains from attention alignment loss decrease to 5 points for 2500 examples, while multitask learning is not effective on this task.Source: Table 4 · p. 6, §4 · p. 6
  • Corrupted explanations are associated with lower student performance as noise increases (Pearson correlation ρ = −0.72).The authors describe this as correlation with an admittedly simple notion of explanation quality.Source: §4 · p. 6

5 Automated Evaluation of Attributions

The framework evaluates explanations by whether they improve a student model’s ability to simulate a teacher, with explanations available during training. Experiments compare attribution methods across sentiment analysis and question answering, and examine how results vary across students, strategies, and benchmarks.

5.2 Main Results

  • Attention explanations improve sentiment-model simulation accuracy over the no-explanation control, with gains of 1.4–2.6 points for transformer students and up to 7 points for BiLSTM students.Integrated gradients and layer conductance are also effective; the reported rankings are largely consistent across transformer and BiLSTM students.Source: §5.2 · pp. 7–8, Table 5 · p. 7, Table 6 · p. 8
  • Trivial explanations do not outperform the no-explanation control, although other test-time protocols can reward label-leaking explanations.The authors use this result as evidence that their framework resists this form of gamification.Source: §5.2 · p. 8
  • For question answering, explanation benefits decrease as student training data increases: attention alignment gains 7.9 points with 250 examples, 2.9 with 4,000, and none with 16,000.The table reports student simulation performance as F1 across the learning strategies.Source: §5.2 · p. 8, Table 7 · p. 8

5.3 Analysis

  • Attribution rankings show high agreement across student models and learning strategies: mean Kendall τ is 0.64 across strategies for the same student, and 0.65 with AR versus 0.47 with MTL across students.Across all 21 student-and-strategy combinations, average τ is 0.52.Source: §5.3 · p. 9
  • Across five BiLSTM hyperparameter configurations, attribution rankings have an average Kendall correlation of 0.95.The configurations vary batch size, learning rate, embedding dimension, and hidden size.Source: §5.3 · p. 9, Table 8 · p. 9
  • Across top-k token settings, attention remains the most effective explanation method, followed by integrated gradients; using the full attention vector adds 0.9 points over top-10% attention at 500 examples, but only 0.1 at 2,000.The 0.9-point improvement is statistically significant (p = 0.03).Source: §5.3 · p. 10

5.4 Comparison With Other Benchmarks

  • Rankings from this framework correlate moderately with sufficiency and comprehensiveness rankings (τ = 0.39), while LIME ranks best on both of those metrics.The authors attribute LIME’s performance on those metrics to its surrogate model being trained on perturbed sentences, like the inputs used by the metrics.Source: §5.4 · p. 10, Table 9 · p. 10

5.5 Human Students

  • In a crowdworker study, simulation accuracy was 68.0% with LIME, 69.0% with attention, and 75.0% with random explanations, with no statistically significant differences.The study concludes that explanations do not definitively help crowdworkers simulate text-classification models.Source: §5.5 · pp. 10–11

6 Limitations & Future Directions

The paper identifies limits in comparing explanation formats and validating automated scores with human judgment, and notes that rankings may depend on student design. It also proposes learning explanations to improve student performance.

6 Limitations & Future Directions

  • Experiments compare only explanations of the same format; cross-format comparisons require explanations to convey comparable information and suitable student-learning strategies.The authors suggest comparing natural-language explanations with highlighted tokens, or comparing natural-language generation methods.Source: §6 · p. 11
  • Human validation of automated explanation evaluations remains difficult because people can judge plausibility more readily than whether highlighted inputs drive a model’s prediction.The authors suggest that granular simulatability and counterfactual access to models might help people assess explanation roles.Source: §6 · p. 11
  • Rankings may depend on student-model choice, despite moderate-to-high agreement across tested architectures and learning schemes.The authors recommend averaging over diverse student architectures, training examples, and learning strategies, and validating quality with intended users where possible.Source: §6 · p. 11
  • A future direction is to train explanation modules to optimize the metric using students’ few-shot test performance as a meta-learning signal.One proposed starting point is transforming teacher attention heads; implicit gradients could update the explanation module.Source: §6 · p. 11

7 Related Work

Prior work evaluates explanations through human simulation studies, communication games, and rationale-based benchmarks. The paper contrasts its training-only student setup with these approaches and their evaluation challenges.

7 Related Work

  • Human forward-simulation studies compare users’ predictions with and without explanations, but reported differences were not significant and large studies are costly and logistically challenging.The paper says automating students mitigates these challenges and yields quantitative differences among methods.Source: §7 · p. 11
  • Treviso and Martins (2020) assess communication success, but test-time explanations can leak labels or enable trivial protocols; this framework withholds explanations at test time.The paper says constraints on explainer and explainee only partially address the issue.Source: §7 · p. 12
  • Hase et al. (2020) adjust simulatability for leakage by averaging results across examples categorized by whether a discriminator can infer the answer from the explanation.The paper questions whether averaging resolves leakage when trivial explanations reveal labels.Source: §7 · p. 12
  • ERASER measures rationale sufficiency and comprehensiveness, while ROAR and hybrid-document evaluation also face concerns about test-time gaming or out-of-distribution inputs.The paper argues ERASER’s rationale-removal comparisons may reflect distribution shift, and that hybrid documents are task-specific and out of distribution.Source: §7 · p. 12

8 Conclusion

The conclusion defines explanation value by how much training-time explanations improve a student’s simulation of a teacher. Experiments find method differences and favor attention-based explanations, with rankings generally consistent across tested settings.

8 Conclusion

  • Attention is the most effective explanation method among those evaluated, and method rankings are consistent to a moderate-to-high degree across student choices.The reported comparisons cover numerous previously proposed explanation schemes.Source: §8 · p. 12
  • Explanation utility is measured by how much teacher-provided side information during training improves a student’s ability to simulate the teacher.Explanations are unavailable at test time, preventing leakage between explanations and output labels.Source: §8 · p. 12
  • The evaluation confirms the value of human-provided explanations and correlates with a simplistic notion of explanation quality.Source: §8 · p. 12
  • Multitask and attention-regularized student learners are effective, while attention-based learners perform better, especially in low-resource settings.Source: §8 · p. 12

9 Explanation Types

The section describes attribution methods that supply explanations for student-teacher evaluation, including perturbation, gradient, neuron-conductance, and attention approaches. It also reports robustness of rankings across settings and a no-explanation baseline for a student model.

9 Explanation Types

  • LIME perturbs an input string, fits a local interpretable linear model, and selects top words to explain the predicted class.The procedure draws twice as many samples as input tokens and sets the classifier’s feature count to 2k when selecting k words.Source: §9 · p. 15
  • Gradient saliency uses the predicted-label log-likelihood gradient, either its token-representation ℓ2 norm or its dot product with that representation.Source: §9 · p. 15
  • Integrated Gradients integrates gradients along the path from a reference point to the input, using 50 points in these experiments.Layer Conductance computes neuron-level importance at the teacher’s first encoder layer and aggregates scores into input-token attributions.Source: §9 · p. 15
  • DeepLIFT measures input-feature contributions relative to a reference and uses an all-zero embedding reference in these experiments.It is described as addressing gradient-attribution limitations with zero or discontinuous gradients; attention coefficients are sometimes interpreted as indicating which encoded tokens receive focus.Source: §9 · p. 15
  • Table 10 reports highly correlated attribution rankings across student models and learning strategies, including comparisons with ERASER metrics.The table uses Kendall’s τ, ranging from -1.0 for perfect disagreement to 1.0 for perfect agreement.Source: Table 10 · p. 16
  • Table 11 examines selecting top-k% explanatory tokens; the BERT-base student without explanations has 92.6 simulation accuracy.Source: Table 11 · p. 16
Loading 2012.00893v2…