PaperlayerView exact revision on arXiv

Source-linked AI summary

A Diagnostic Study of Explainability Techniques for Text Classification

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein

arXiv:2009.13295v1cs.CLcs.LG

TL;DR

The paper addresses the lack of a general way to select and evaluate explainability techniques across application tasks and model architectures. It proposes diagnostic properties and applies them to diverse techniques, comparing their behavior with human salient-token annotations. Gradient-based explanations perform best across the considered tasks and architectures, while several alternatives are slower, less faithful, and less consistent with model rationales and dataset similarities.

  • Problem

    Existing explainability techniques can yield differing rationales, but there is no definitive guide for selecting them across application tasks and model architectures.

  • Method

    The paper develops diagnostic properties and compares diverse explainability techniques across three text classification tasks, CNN, LSTM, and Transformer architectures, and human salient-token annotations.

  • Results

    Gradient-based explanations perform best across all three considered models and downstream text classification tasks; ShapSampl, LIME, and Occlusion take more time and are less faithful and consistent.

  • Takeaways & Limitations

    The diagnostic properties support comparative assessment of explainability techniques across method groups, tasks, and architectures.

  • Takeaways & Limitations

    The study evaluates word-level human annotations and leaves sentence-level annotation datasets for future work because they require other model architectures.

Abstract

from arXiv · show

Recent developments in machine learning have introduced models that approach human performance at the cost of increased architectural complexity. Efforts to make the rationales behind the models' predictions transparent have inspired an abundance of new explainability techniques. Provided with an already trained model, they compute saliency scores for the words of an input instance. However, there exists no definitive guide on (i) how to choose such a technique given a particular application task and model architecture, and (ii) the benefits and drawbacks of using each such technique. In this paper, we develop a comprehensive list of diagnostic properties for evaluating existing explainability techniques. We then employ the proposed list to compare a set of diverse explainability techniques on downstream text classification tasks and neural network architectures. We also compare the saliency scores assigned by the explainability techniques with human annotations of salient input regions to find relations between a model's performance and the agreement of its rationales with human ones. Overall, we find that the gradient-based explanations perform best across tasks and model architectures, and we present further insights into the properties of the reviewed explainability techniques.

1 Introduction

Explainability methods make individual model predictions more transparent by assigning saliency scores to input words, but their outputs can conflict and lack a general selection framework. This study proposes diagnostic properties and compares diverse techniques across tasks, architectures, and human rationales.

  • Motivation: Greater insight into model rationales is important for identifying potential flaws or biases in applications such as medical diagnosis and judicial sentencing.The introduction also connects explainability with legal demands for explanations of decisions.
  • Motivation: Explainability methods provide post-hoc word-level reasons for individual predictions, including techniques specialized to one model and techniques applicable across models.They include gradient-based, perturbation-based, and simplification-based approaches.
  • Research gap: Different methods can produce varying or contradictory explanations, motivating an automated way to choose techniques for particular architectures and application tasks.Existing property definitions remain difficult to evaluate in practice.
  • Research gap: A thorough evaluation should cover method groups, downstream tasks, and model architectures, whereas prior studies usually examine at most two of these dimensions.Consequently, prior insights are often tied to a specific experimental setup.
  • Contributions: The paper compiles diagnostic properties, compares explainability groups across three text classification tasks and three architectures, and relates model attributions to human salient-token annotations.The architectures are CNN, LSTM, and Transformer.

2 Related Work

Prior explainability evaluations vary in scope and often provide setup-specific evidence, while human judgments and isolated counterexamples have limited diagnostic value. This work unifies comparisons across method groups, tasks, and architectures using measurable diagnostic properties.

  • Method landscape: Explainability methods span simplification, gradient-based, perturbation-based, and generated-text approaches.LIME exemplifies simplification, while gradient and perturbation methods assign or derive feature importance differently.
  • Evaluation limitations: Counterexamples can reveal differing explanations for the same output, but isolated examples do not measure explainability techniques systematically.Human evaluations also show low interannotator agreement and may reflect perceived reasonableness rather than method properties.
  • Proposed evaluation: The paper turns diagnostic properties into practical measures and examines instance activations to probe model reasoning beyond agreement in final predictions.This extends consistency and stability-oriented property definitions with measures intended for empirical assessment.
  • Diagnostic measures: For sufficiency, the study measures model-performance decrease while varying the proportion of excluded salient tokens instead of fixing one removal threshold.AUC-TP is used as an approximation of whether the most salient words are also most important for prediction.
  • Human agreement: The study compares human–machine saliency agreement across model architectures, downstream tasks, and explainability methods.Its human annotations are at the word level rather than the sentence level.
  • Scope: Existing work commonly focuses on one architecture, dataset, or explainability group, whereas this study presents a unified comparison across three tasks and three architectures.The paper characterizes this unified scope as novel relative to prior evaluations.

3 Evaluating Attribution Maps

The paper defines diagnostic properties for evaluating attribution maps, covering predictive confidence, faithfulness, consistency across models and datasets, and agreement with human rationales.

  • The framework evaluates explainability techniques through diagnostic properties intended to extend prior notions to downstream tasks.
  • Agreement with human rationales: Agreement with human rationales measures overlap between annotator saliency scores and technique-generated scores, averaging instance-level Average Precision into Mean AP.Saliency scores are computed for each instance using its gold label.
  • Confidence Indication: Confidence Indication estimates model confidence from differences between predicted-class and other-class saliency scores, using logistic regression and Mean Absolute Error.For multiclass tasks, the differences are represented by their maximum, minimum, and average; lower MAE indicates easier confidence identification.
  • Faithfulness: Faithfulness uses masking of highly salient words and the area under the threshold-performance curve to assess whether salient words are important for predictions.The AUC-TP is compared with a random saliency map, while prior work shows models can remain confident after most tokens are removed.
  • Rationale Consistency: Rationale Consistency compares similarities between models’ reasoning paths with similarities between their explanation scores using Spearman’s ρ.Reasoning-path similarity is approximated through activation-map distances, and higher positive correlation indicates greater consistency.
  • Dataset Consistency: Dataset Consistency tests whether instances with similar rationales receive similar explanations on the same model, again using activation and saliency-score similarities.The relationship between the two similarity measures is quantified with Spearman’s ρ and absolute differences.

4 Experiments

The experiments evaluate explainability techniques across three text-classification datasets and diverse neural architectures, using diagnostic properties summarized across tasks and models. The study includes human-annotated saliency data, multiple model families, and representative gradient, perturbation, and simplification methods.

  • Datasets: The datasets cover natural-language inference and sentiment classification, with human-annotated salient tokens available for evaluation.e-SNLI targets contradiction, neutral, or entailment; Movie Reviews and TSE target positive, negative, or neutral sentiment.
  • Models: The experiments compare CNN, LSTM, and Transformer-based BERT architectures across the selected text-classification tasks.These architectures provide a comparison across diverse model types.
  • Explainability Techniques: The selected explainability methods represent gradient, perturbation, and simplification-based groups.The study includes Saliency, InputX-Gradient, Guided Backpropagation, Occlusion, Shapley Value Sampling, and LIME.
  • Explainability Techniques: Occlusion replaces each token with a zero baseline and measures the resulting change in model output.Shapley Value Sampling instead approximates average marginal word contributions using a fixed number of random perturbations.
  • Explainability Techniques: Gradient-based methods require aggregating embedding-level saliency scores into one score per input token.The experiments use L2 norm and averaging as the two aggregation methods.

5 Results and Discussion

Across tasks and architectures, gradient-based explainability methods generally perform best on the diagnostic properties, although ShapSampl and LIME excel for Confidence Indication. Agreement with human rationales varies by architecture, and faithfulness and human agreement measure distinct properties.

  • Overall results: Gradient-based explainability techniques achieve the best or second-best diagnostic-property performance across all three architectures and tasks.LIME is the strongest non-gradient method, followed by ShapSampl and Occlusion.
  • Dataset and architecture effects: Explainability methods perform better on e-SNLI and TSE than on IMDB, which the authors hypothesize is related to IMDB’s longer input texts.Figures 2 and 3 show diagnostic-property performance for e-SNLI and TSE; IMDB exhibits similar tendencies in supplementary results.
  • Agreement with human rationales: Transformer explanations achieve the highest MAP scores against human rationales, while mean-aggregated gradient methods score close to or below randomly initialized models.CNN and LSTM MAP scores are consistently lower than Transformer scores, and CNN scores exceed LSTM scores despite the LSTM’s higher e-SNLI F1.
  • Faithfulness: InputXGradℓ2 performs particularly well for Faithfulness, with gradient-based techniques leading this property overall.The CNN has the highest Faithfulness scores for seven of nine explainability methods.
  • Confidence Indication: ShapSampl and LIME most accurately predict model confidence, while their computation takes considerably more time than gradient-based alternatives.The Confidence Indication result is attributed to perturbation-based changes in model confidence.
  • Rationale Consistency: There is no universal winner for Rationale Consistency: LIME and gradient-based methods perform strongly, whereas Occlusion is best for LSTM models across tasks.The paper relates Occlusion’s LSTM performance to its simple perturbation-based design, which does not inspect model internals.
  • Dataset Consistency: Dataset Consistency shows low-to-moderate correlations, particularly for LIME and gradient-based methods with L2 aggregation.The reported correlations concern similarities across instances in the dataset.

6 Conclusion

The paper proposes diagnostic properties to evaluate explainability techniques from multiple perspectives and compares diverse methods across three tasks and architectures. Gradient-based explanations perform best overall, while ShapSampl, LIME, and Occlusion are slower and weaker on most properties.

  • The study evaluates explainability techniques using a comprehensive list of diagnostic properties.The properties assess techniques from different perspectives.
  • The comparison covers different technique groups, three downstream tasks, and three diverse model architectures.
  • Gradient-based explanations perform best across all three evaluated models and text-classification tasks.
  • ShapSampl, LIME, and Occlusion require more computation time and perform worse on faithfulness, consistency, and related diagnostic properties.

A.1 Experimental Setup

The experimental setup tunes CNN, LSTM, and Transformer models and evaluates explainability-related measures using task-specific procedures. The experiments use macro F1 for model performance, sampled instance pairs for consistency, and confidence up-sampling where needed.

  • CNN and LSTM models are tuned with grid searches, early stopping, validation-based selection, and a maximum of 100 training epochs.Early stopping uses validation accuracy with a patience of five.
  • The Transformer experiment fine-tunes a 110M-parameter pretrained uncased language model with a 512-token maximum input size.The model has 12 layers with hidden size 768, and training uses learning-rate warm-up.
  • Model performance is evaluated with macro F1 score.
  • Confidence Indication uses logistic regression over five splits and reports mean absolute error across test splits.Instances with lower confidence are randomly up-sampled to balance confidence intervals.
  • Rationale and Dataset Consistency use Spearman’s ρ, with Dataset Consistency estimated from selected similar and different instance pairs.The procedure selects 2,000 high-overlap pairs and samples 2,000 remaining pairs.
  • For LSTM consistency comparisons, activations are restricted to prediction outputs and differences are min-max scaled.This handles varying sequence lengths and small saliency scores.

A.3 Detailed explainability techniques evaluation results.

Figure 4 evaluates diagnostic properties for all explainability techniques on the IMDB dataset. Arrows indicate whether higher or lower property values are preferable.

  • Figure 4 compares all explainability techniques across diagnostic properties on IMDB.
  • The ↗ and ↙ markers indicate whether higher or lower values of each property measure are better.

MAP MAP RI FLOPs MAP MAP RI FLOPs MAP MAP RI FLOPs

The detailed evaluation reports human agreement, faithfulness, confidence indication, and rationale consistency for explainability techniques across datasets and models. Results are averaged across model runs or random initializations, with variability reported in brackets where specified.

  • The reported technique rows include Random, GuidedBP, ShapSampl, LIME, Occlusion, Saliency, and InputXGrad variants.
  • Human Agreement is measured with MAP against gold human annotations, alongside MAP for a randomly initialized model and computation cost in FLOPs.Table 5 averages results over five models and reports standard deviations.
  • Faithfulness is evaluated with Faithfulness-AUC over thresholds from 0 to 100, where lower scores indicate better assignment of saliency to prediction-responsible words.
  • Confidence Indication reports MAE and MAX for predicting model confidence, with and without confidence up-sampling.Results are averaged over models trained from different random seeds.
  • Rationale Consistency is reported as Spearman’s ρ correlation, with p-values provided in brackets.Best results are identified per dataset and model, and correlations below random saliency are marked.
Loading 2009.13295v1…