PaperlayerView exact revision on arXiv

Source-linked AI summary

Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?

Peter Hase, Shiyue Zhang, Harry Xie, Mohit Bansal

arXiv:2010.04119v1cs.CLcs.AIcs.LG

TL;DR

Similarity to human explanations does not show whether explanations support model behavior. The paper introduces leakage-adjusted simulatability and tests it with model and human observers. ST-RA reaches similar LAS scores to human explanations, while optimizing explanations for simulatability can improve LAS.

  • Problem

    Common similarity-based evaluations of natural-language explanations do not establish whether explanations support model behavior rather than resemble what humans would say.

    Source: §1 · p. 2

  • Method

    The paper introduces LAS to measure observer prediction of model outputs while controlling for direct label leakage, and validates a model observer with two human evaluations.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    ST-RA attains similar LAS scores to human explanations, rationalizing methods do better than reasoning methods, and simulatability optimization can improve LAS scores.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    LAS offers an evaluation of explanation support that accounts for direct label leakage, and the multi-agent optimization improves LAS in some settings.

    Source: §1 · p. 2

    LAS scores across datasets are not directly comparable because they depend on underlying task difficulty.

    Source: §6.1 · p. 8

Abstract

from arXiv · show

Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores. We provide code for the experiments in this paper at https://github.com/peterbhase/LAS-NL-Explanations

1 Introduction

The paper argues that similarity to human explanations measures plausibility, not whether explanations support model behavior. It introduces leakage-adjusted simulatability (LAS), validates a model observer with human experiments, and explores optimizing explanations for simulatability.

1 Introduction

  • Similarity metrics such as BLEU and human judgments of resemblance to human explanations do not establish whether explanations reflect model behavior.The paper frames this as a gap between plausibility and faithfulness.Source: §1 · p. 2
  • LAS measures how explanations help an observer predict model outputs while controlling for direct output leakage, using a pretrained language model as a proxy observer.The authors evaluate model-generated and human explanations on CQA and SNLI.Source: §1 · p. 2
  • Two human evaluations assess the model-based approach: expert simulation of the LAS procedure and crowdsourced subjective explanation ratings.The subjective ratings were collected from Mechanical Turk workers.Source: §1 · p. 2
  • A multi-agent training objective rewards simulator prediction of the task model’s output from explanations while penalizing explanations that merely restate outputs.The paper presents this as a proxy objective because LAS itself is not differentiable.Source: §1 · p. 2
  • The authors report that ST-RA achieves nearly human-level LAS, rationalizing models outperform reasoning models, and a multi-agent objective can improve LAS in some settings.They also report no statistically significant relationship between simulatability and accuracy.Source: §1 · p. 2

2 Related Work

Prior work generated explanations for language-understanding tasks and commonly evaluated them by similarity to human explanations. The paper distinguishes its work through leakage-controlled simulatability and comparison of several graphical models.

2 Related Work

  • Prior language-understanding approaches include e-SNLI with a joint label-and-explanation model and CoS-E with the CAGE framework [Camburu et al., 2018; Rajani et al., 2019].The paper evaluates their corresponding MT-RA and ST-RE structures alongside ST-RA and MT-RE.Source: §2 · p. 3
  • Earlier evaluations commonly used BLEU or human judgments of similarity, which assess plausibility rather than whether explanations support model behavior [Hendricks et al., 2016; Park et al., 2018; Kim et al., 2018; Camburu et al., 2018; Rajani et al., 2019].The authors introduce LAS to address this distinction.Source: §2 · p. 3
  • The paper presents its novelty as evaluating natural-language explanations through simulatability while controlling for explanation label leakage and comparing distinct graphical models.It contrasts this joint focus with contemporaneous work on explanation support or recovery of ground-truth labels alone.Source: §2 · p. 3
  • The paper’s multi-agent approach optimizes full explanations of model outputs, unlike related image-reference games that optimize descriptions and extractive simulatability work [Lazaridou et al., 2017; Lazaridou et al., 2020].The paper notes that its approach shares the premise that language use is goal-oriented.Source: §2 · p. 3

3 Modeling With Explanations

The paper compares explanation-generation models that differ in whether explanations are conditioned on predictions and whether they serve as task inputs or multitask supervision. The approaches are implemented with pretrained T5 models.

3 Modeling With Explanations

  • The models generally start from T5-Base, a pretrained Transformer sequence-to-sequence model, while MT-RA and the baseline use a task mixing weight of α = .5 on both datasets.The paper also presents the graphical models and baseline in Table 1.Source: §3 · pp. 4–5, Table 1 · p. 4
  • The baseline predicts a task output from the input by selecting the most likely answer choice; SNLI uses its three relations as choices.The model is fine-tuned by maximizing the likelihood of correct answer tokens conditioned on task inputs.Source: §3 · p. 4
  • ST-RE generates explanations from task inputs in a first training phase, then supplies those explanations with inputs to a classifier in a second phase.The authors implement the framework with two separate T5 models.Source: §3 · p. 4
  • ST-RA generates an explanation for each possible task output and pairs each answer choice with its supporting explanation before classification.The classifier scores each input-and-output sequence.Source: §3 · p. 4
  • MT-RE uses explanations as multitask supervision conditioned only on the task input, mixing task-label prediction and explanation language-modeling losses.A task-specific prepended word tailors encoder states to task prediction or explanation generation.Source: §3 · p. 4
  • MT-RA is a multitask model whose explanations are conditioned on the model output.The paper identifies it as the approach originating in Camburu et al. (2018).Source: Figure 2 · p. 4, §3 · p. 5

4 LAS: Leakage-Adjusted Simulatability

Simulatability asks whether an observer can predict a model’s outputs, but raw simulator accuracy cannot distinguish explanations’ semantic contribution from input-based guessing or label leakage. LAS estimates the leakage-controlled effect of explanations, using a learned simulator as a proxy for a human observer.

4 LAS: Leakage-Adjusted Simulatability

  • Raw simulator accuracy conflates explanations’ semantic contribution with correct guesses from the input or direct output leakage.Simulatability is the extent to which an observer can predict model outputs.Source: §4 · p. 5
  • LAS subtracts input-only simulator correctness, groups examples by whether explanations leak the label, and averages the explanation-related gains across both groups.The leakage proxy tests whether the simulator can predict the output from the explanation alone.Source: §4 · pp. 5–6
  • A pretrained T5-Base simulator predicts task-model outputs from inputs and explanations, with input or explanation dropout used during training.The authors validate using a model as a human-simulator proxy with two human-subject experiments.Source: §4 · p. 6
  • LAS estimates typically vary by less than 1 point across leakage-probability bin counts from 2 to 100.The sensitivity analysis calibrates simulator probabilities with Platt scaling.Source: §4 · p. 6

5 Multi-Agent Explanation Optimization

The paper frames explanation optimization as a two-agent game: a task model generates explanations to help a simulator predict its output, while label leakage is penalized. It describes approximate SGD and REINFORCE approaches, with task, language-modeling, and explanation losses in the task model’s objective.

5 Multi-Agent Explanation Optimization

  • A jointly trained task model and simulator form a game that optimizes explanations for simulatability while penalizing direct label leakage.The task model’s predictions are not optimized to agree with the simulator; its explanations are optimized instead.Source: §5 · p. 6
  • Table 3 reports LAS, overall simulator accuracy, and BLEU for human and model-generated explanations, with bootstrap confidence intervals.The caption specifies 95% confidence intervals.Source: Table 3 · p. 7
  • Approximate SGD relaxes greedy argmax decoding to softmax during backpropagation while retaining argmax in the forward pass.The relaxation uses a temperature of 1.Source: §5 · p. 7
  • REINFORCE uses simulator output probabilities as a reward, balancing prediction from input-plus-explanation against prediction from explanation alone.The reward is defined as α times the former probability minus (1−α) times the latter.Source: §5 · p. 7
  • The task model’s full objective combines task loss, language-modeling loss, and explanation loss with separate weights.The tuning procedure and mixing-weight values are given in Appendix A.5.Source: §5 · p. 7

6 Experimental Results

Experiments on CoS-E/CQA and e-SNLI show that some models achieve human-level simulatability, while the relationship between simulatability, human ratings, and accuracy varies. The experiments also test accuracy trade-offs and multi-agent optimization.

6 Experimental Results

  • Experiments used CoS-E/CQA and e-SNLI; main-paper SNLI results use 10% of training data, and CQA results use dev data because human test explanations are unavailable.CQA Version 1.0 was selected for its higher-quality explanations than Version 1.1.Source: §6 · p. 7

6.1 Automatic Explanation Evaluation

  • ST-RA performed similarly to human explanations on LAS for both datasets, and rationalizing models outperformed reasoning models.LAS did not correlate with BLEU or overall simulator accuracy.Source: §6.1 · pp. 7–8

6.2 Human Validation of LAS

  • Expert simulation found moderate proxy-human LAS alignment (ρ = 0.29) and stronger leakage alignment (ρ = 0.53); human ratings tracked explanation-only leakage best, while LAS aligned poorly.Humans simulated human explanations better, while models better predicted model outputs; the average simulator-accuracy difference was 12.83 percentage points.Source: §6.2 · pp. 8–9

6.3 Accuracy-Interpretability Trade-off

  • Accuracy changes across four models remained small and nonsignificant; using human explanations in an MLM objective also yielded no significant accuracy improvement.The largest reported changes were −0.47 on SNLI (p = .3124) and −2.10 on CQA (p = .3272).Source: §6.3 · p. 9

6.4 Multi-Agent Game

  • SGD cut CQA MT-RA leakage from 85.58% to 75.21% and slightly improved LAS; only SNLI MT-RE was significant, while BLEU fell, accuracy changes were nonsignificant, and RL was unstable.The largest accuracy change was a 3.37-point CQA drop (p = .1287).Source: §6.4 · p. 9

7 Conclusion

The paper introduces LAS to evaluate how explanations support prediction of model outputs while controlling for label leakage. Experiments validate the metric and report human-level LAS for ST-RA, alongside results on model comparisons and multi-agent optimization.

7 Conclusion

  • LAS measures how natural-language explanations affect model simulatability while controlling for explanations leaking model outputs.The authors validate the metric with two human-subject experiments.Source: §7 · p. 9
  • ST-RA attains similar LAS scores to human explanations, rationalizing methods outperform reasoning methods, and no statistically significant relationship emerges between simulatability and accuracy.The automatic metric correlates with expert simulation results, and explanation-only leakage is the strongest predictor of crowdsourced ratings.Source: §7 · p. 9
  • Optimizing explanations for simulatability can improve LAS scores.Source: §7 · p. 9

A.1 Datasets and Examples

The appendix documents the datasets, testing procedures, model selection and training, simulator training, and hyperparameter tuning used in the experiments.

A.1 Datasets and Examples

  • CoS-E Version 1.0 has 7,610/950/940 train/dev/test examples; e-SNLI has 549,339/9,842/9,824, with the first of three e-SNLI test explanations used for BLEU.CQA Version 1.0 was chosen because its explanations were higher quality than Version 1.1.Source: §A.1 · p. 11

A.2 Hypothesis Testing

  • Statistical significance uses p < .05; LAS uses bootstrap tests, accuracy uses binomial-means tests, and rating associations use linear regression with i.i.d. normal noise.Expert-simulation correlations use Spearman rank correlation.Source: §A.2 · p. 12

A.3 Model Selection and Training Details

  • Standard task models were trained with five seeds and selected by dev performance; two-agent runs used one seed, selected with T5-Small LAS, then retrained with T5-Base.Source: §A.3 · p. 12

A.4 Training Simulator Models

  • Simulator training uses token-level dropout on the input or explanation subsequence so the model can approximate the three LAS prediction distributions.Without this technique, simulators rely too heavily on explanations and underperform input-only baselines when conditioned only on the input.Source: §A.4 · p. 12

A.5 Hyperparameter Tuning

  • For multi-agent models, task, language-model, and simulatability weights were tuned for simulator accuracy, then α for LAS; SGD used α = .8/.9 for CQA/SNLI, while RL used .8 for both.Source: §A.5 · p. 13

B.1 Continuous Leakage Scores and LAS Metric

The section describes continuous leakage scoring and sensitivity checks for LAS, alongside the ST-RA scoring setup and task-accuracy context.

B.1 Continuous Leakage Scores and LAS Metric

  • A continuous leakage measure can be obtained from p(ŷ|ê), and in the limit of infinitely many bins LAS becomes an integral over leakage probability.The described implementation binarizes a leakage proxy and averages explanation effects across two leakage bins.Source: §B.1 · pp. 13–14
  • ST-RA generates one explanation per answer choice conditioned on that choice, then scores the sequences and answers with a separate T5 task model.The framework uses separate T5 models for explanation generation and task scoring.Source: Figure 5 · p. 14
  • Generative models perform as well as non-generative baselines on the CQA and SNLI task-accuracy results.The caption specifies CQA development data and SNLI test data.Source: Table 8 · p. 14
  • Across 2–100 leakage bins, LAS estimates typically vary by less than 1 point and method ranking is almost always preserved.The authors recommend sensitivity analysis, especially when explanation effectiveness is not linearly related to leakage probability.Source: §B.1 · p. 14

B.2 Robustness to Seed and Model Choice

The robustness checks examine variation across random seeds and simulator architectures. Seed-based rankings are typically preserved, while simulator choice can change LAS scores and rankings.

B.2 Robustness to Seed and Model Choice

  • Across three random seeds, score rankings are typically preserved and most scores show relatively low variance, though some are outliers.The check is motivated by the influence random seeds can have on statistics from pretrained language models.Source: §B.2 · p. 14
  • On SNLI, RoBERTa produces lower LAS scores than T5, and their rankings are not necessarily the same; ST-RA is highest on average with both.The comparison uses T5-Base and RoBERTa-Large; MT conditions are not evaluated with RoBERTa because it is not generative.Source: §B.2 · p. 14
  • The authors note that simulator differences could reflect pretraining, architecture, fine-tuning sample efficiency, or another cause.The comparison does not establish which factor accounts for the differences.Source: §B.2 · p. 14

C Alternative Computational Models and Language Modeling Objectives

Alternative computational models and language-modeling objectives are tested for task accuracy, while the supplied table captions identify additional accuracy and explanation-evaluation results.

C Alternative Computational Models and Language Modeling Objectives

  • Substituting GPT2 and BERT or using masked rather than causal language-modeling objectives yields no task-accuracy differences from the baselines.The tested alternatives include masked language modeling for MT models and GPT2/BERT substitutions for generators or task models.Source: C Alternative Computational Models and… · p. 14
  • The alternative-model task-results table covers methods using GPT2 and BERT, which the authors tag as ENC methods.The table caption identifies the scope but provides no cell values in the supplied passage.Source: C Alternative Computational Models and… · p. 14
  • The opposite data split is reported for a table evaluating human and model-generated explanations by LAS, simulator accuracy, and BLEU, with bootstrap confidence intervals.The caption notes wider intervals when the nonleaking subset is very small.Source: Table 9 · p. 15
  • The seed-check table retrains the task model, simulator, and ST generators for two additional seeds beyond the main-body seed.The caption says these are tests of the primary experiments.Source: Table 10 · p. 15

D Human Quality Rating Collection

The authors collect human ratings of explanation quality using shuffled model and human explanations, then filter responses with a check based on explanations contradicting model outputs.

D Human Quality Rating Collection

  • For each dataset, the study samples 200 examples and presents five shuffled explanations per example: four model-generated and one human.Turkers rate each explanation separately on a 5-point Likert scale and are instructed to judge how it supports the answer choice, not whether it is literally true.Source: D Human Quality Rating Collection · p. 15
  • The collection materials include full CQA rating instructions and an example question with answer choices, a model choice, and its explanation.SNLI used a similar interface, with five choice–explanation pairs on each page.Source: D Human Quality Rating Collection · p. 15, Figure 6 · p. 17, Figure 7 · p. 17
  • Each example receives three responses, totaling 600 responses per dataset before filtering.The researchers manually selected ten explanations per dataset that contradicted their model outputs as quality-control items.Source: D Human Quality Rating Collection · p. 15
  • Responses were filtered when turkers rated known contradictory explanations highly, using thresholds above 2 for CQA and above 3 for SNLI.The thresholds differ because SNLI had a higher average rating; the resulting totals were 466 CQA responses and 436 SNLI responses.Source: D Human Quality Rating Collection · p. 16

Related papers

Loading 2010.04119v1…