PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Deep Learning Interpretability by Saliency Guided Training

Aya Abdelsalam Ismail, Héctor Corrada Bravo, Soheil Feizi

arXiv:2111.14338v1cs.CVcs.AIcs.LG

TL;DR

Noisy gradients can make saliency-based feature attributions unfaithful. The paper trains models by masking low-gradient features while encouraging similar outputs for masked and original inputs, and reports improved explanations across image, language, and time-series tasks. The training procedure also carries computational and memory costs.

  • Problem

    Saliency methods use gradients to highlight influential input features, but noisy gradients can produce unfaithful attributions.

    Source: §1 · p. 1

  • Method

    Saliency guided training iteratively masks low-gradient features and combines task loss with KL divergence to encourage similar outputs for masked and original inputs.

    Source: §1 · p. 2, §4 · pp. 3–4

  • Results

    Across image, language, and multivariate time-series tasks, saliency guided training improves explanation quality, including reduced visual noise and increased time-series saliency precision and recall.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The approach trains explanations without requiring them as input and is presented as an unsupervised, regularization-based route to improved model faithfulness.

    Source: §6 · p. 9

    Training is more computationally expensive, with doubled memory use and slower convergence than traditional training.

    Source: §4 · p. 4

Abstract

from arXiv · show

Saliency methods have been widely used to highlight important input features in model predictions. Most existing methods use backpropagation on a modified gradient function to generate saliency maps. Thus, noisy gradients can result in unfaithful feature attributions. In this paper, we tackle this issue and introduce a {\it saliency guided training}procedure for neural networks to reduce noisy gradients used in predictions while retaining the predictive performance of the model. Our saliency guided training procedure iteratively masks features with small and potentially noisy gradients while maximizing the similarity of model outputs for both masked and unmasked inputs. We apply the saliency guided training procedure to various synthetic and real data sets from computer vision, natural language processing, and time series across diverse neural architectures, including Recurrent Neural Networks, Convolutional Networks, and Transformers. Through qualitative and quantitative evaluations, we show that saliency guided training procedure significantly improves model interpretability across various domains while preserving its predictive performance.

1 Introduction

Saliency maps use gradients to identify influential input features, but their noise can undermine explanation faithfulness. The paper introduces saliency guided training and reports improved explanations across image, language, and time-series tasks.

1 Introduction

  • Saliency maps assign importance scores to input features based on their influence on a model prediction, but their gradients can be noisy.Prior approaches include SmoothGrad [53], which averages maps from noisy inputs, and methods that use modified or higher-order gradients.Source: §1 · p. 1
  • Saliency guided training masks low-gradient input features and minimizes a loss combining KL divergence between original and masked outputs with the task loss.The procedure aims to reduce gradient noise without sacrificing predictive performance.Source: §1 · p. 2
  • Across image, language, and multivariate time-series classification, the approach is evaluated with CNNs, RNNs, and Transformers.Image tasks show reduced visual saliency noise and sparser maps; sentiment analysis and fact extraction show improved explanation comprehensiveness.Source: §1 · p. 2
  • In multivariate time-series classification, saliency guided training increases saliency-map precision and recall and reduces vanishing saliency in RNNs.The reported explanation improvements also extend to several gradient-based saliency methods beyond the vanilla gradient used for masking.Source: §1 · p. 2

2 Background and Related Work

Related work spans post-hoc explanation methods, reliability evaluation, interpretability-oriented architectures, and learning procedures that incorporate explanations. This paper positions its training approach as self-supervised and distinct from perturbation-based training techniques.

2 Background and Related Work

  • Post-hoc explanation methods include gradient-based approaches [53] and perturbation-based approaches [55] [58].Other strands evaluate reliability with benchmarks [21] [11] [42] or debugging [2], and modify architectures [12] [45].Source: §2 · p. 2
  • Ghaeini et al. [14] rely on ground-truth explanations and Ross et al. [45] on incorrect-explanation annotations, whereas this approach does not require such annotations.The paper notes that most datasets only have ground-truth labels.Source: §2 · p. 2
  • Prior input-perturbation training uses attention maps for weakly supervised segmentation [19] or classification accuracy, while DeVries and Taylor [10] mask square regions for robustness and performance.The paper distinguishes its own goal as increasing interpretability through self-supervised training.Source: §2 · p. 2
  • The evaluated saliency methods include gradients, Integrated Gradients, DeepLIFT, SmoothGrad [53], and Gradient SHAP.They use, respectively, input gradients, a reference-to-input gradient path integral, activation differences from a reference, noisy-input averaging, and gradients at sampled path points.Source: §2 · p. 2
  • The evaluated architectures include VGG-16 [49], ResNet, and sequence-handling TCNs [39] [7], alongside LSTMs, attention-equipped LSTMs, and Transformers.The passage describes CNNs, RNNs, and Transformers as the architecture families used.Source: §2 · pp. 2–3

3 Notation

The notation treats each classification example as an input with features and a label, and describes standard training as minimizing cross-entropy. It also defines gradient-based feature ordering and masking, with input structure adapted for language and time series.

3 Notation

  • Standard classification training minimizes cross-entropy over examples (X_i, y_i), where each input X has N features and y is its label.The network is denoted f_θ and parameterized by θ.Source: §3 · p. 3
  • The mask M_k removes the k lowest-ranked input features according to a sorting function applied to input gradients.For language, features are word embeddings and sorting and masking operate at the word level; for time series, they operate on each feature at each time step.Source: §3 · p. 3
  • The Kullback–Leibler divergence is introduced as the relative entropy between two discrete probability distributions defined on the same space.The passage names the divergence from Q to P.Source: §3 · p. 3

4 Saliency Guided Training

Saliency guided training reduces potentially noisy gradients by masking low-gradient input features and training the model to preserve its outputs on masked inputs. The procedure combines classification with an output-consistency objective, with masking choices adapted to the data and additional computational cost.

4 Saliency Guided Training

  • The method masks low-gradient input features to reduce gradients for irrelevant features and encourage sparse, more faithful explanations.It aims to do so without sacrificing model performance.Source: §4 · pp. 3–4
  • Training minimizes classification loss alongside KL divergence between outputs for the original and masked inputs.The KL term encourages similar output distributions; λ balances it against classification loss.Source: §4 · p. 4
  • For images and time series, low-gradient features are replaced with random in-range values; for language, they are replaced with the previous high-salience word.The dataset-dependent k controls how many features are masked.Source: §4 · p. 4
  • The procedure costs more than traditional training: it doubles memory use and increases the CIFAR-10 example’s convergence time from 118 epochs at roughly 24 seconds each to 124 epochs at roughly 75 seconds each.It also requires choosing k and λ; the authors report λ = 1 worked well in their experiments.Source: §4 · p. 4
  • The training loop sorts input gradients, masks the bottom k features, computes the loss, and updates network parameters.The loss is computed for the masked and original inputs.Source: §4 · p. 4

5 Experiments

Across image, language, and time-series experiments, saliency guided training improves explanation quality and faithfulness measures. Results also show a boundary for modification-based evaluation: it may be unreliable when the uninformative feature distribution is unknown.

5 Experiments

  • Experiments repeated each setting five times and report the average across runs.Hyperparameters, standard error bars, and computational-resource details are in supplementary materials.Source: §5 · p. 5

Quality of Saliency Maps for Images

  • On MNIST, CIFAR10, and BIRD, saliency guided training produced more precise maps, concentrating gradients near zero while separating highly salient features from the mean.The image examples were correctly classified by both models; maps more often highlighted objects rather than backgrounds.Source: Quality of Saliency Maps for Images · p. 5, Figure 2 · p. 5, §5.1 · p. 5

Model Accuracy Drop

  • For image explanations, saliency guided training yielded steeper accuracy drops after highly salient features were removed, across gradient-based methods.The modification-based evaluation replaces ranked features and measures accuracy degradation; a steeper drop indicates better performance.Source: Model Accuracy Drop · p. 6, Figure 3 · p. 6
  • Modification-based evaluation is limited to settings such as MNIST where the uninformative feature distribution is known; OOD replacement and ROAR differed from IID replacement.The authors argue these methods may be unreliable unless the uninformative IID distribution is known, and leave further exploration for future work.Source: Model Accuracy Drop · p. 6, Figure 4 · p. 6

5.2 Saliency Guided Training for Language

  • Saliency guided training improved language-task comprehensiveness on Movie Reviews, FEVER, and e-SNLI, while sufficiency improved for Movie Reviews but not clearly for FEVER or e-SNLI.The experiments used Glove embeddings and a bidirectional LSTM; the authors suggest shrinking sentences may explain the sufficiency pattern on FEVER and e-SNLI.Source: §5.2 · p. 7

Saliency Maps Quality for Multivariate Time Series

Quantitative Analysis on Synthetic Data

  • Across 10 synthetic time-series datasets, the best mean weighted AUP and AUR performance came from TCN with Integrated Gradients and saliency guided training.Higher metric differences indicate better performance; negative values indicate performance similar to random feature assignment.Source: Quantitative Analysis on Synthetic Data · p. 8, Table 2 · p. 8

Saliency Guided Training reduces vanishing saliency of recurrent neural networks

  • Saliency guided training let LSTMs identify informative features regardless of time location, unlike traditional training, whose AUP and AUR favored later time steps.The comparison used three synthetic datasets with informative features at varied time intervals.Source: Saliency Guided Training reduces… · p. 9, Figure 6 · p. 9

6 Summary and Conclusion

Saliency guided training improves explanations from existing gradient-based saliency methods by optimizing gradients for irrelevant features. The approach masks low-gradient inputs and minimizes KL divergence between outputs for original and masked inputs, and is demonstrated on images, language, and multivariate time series.

6 Summary and Conclusion

  • The method masks low-gradient features and minimizes KL divergence between model outputs for original and masked inputs, alongside the main loss.It is presented as a training procedure to improve explanations from existing gradient-based saliency methods.Source: §6 · p. 9
  • The training procedure is demonstrated on image, language, and multivariate time-series tasks.Source: §6 · p. 9
  • The authors frame the method as unsupervised explanation training that sharpens gradient-based explanations without requiring explanations as input.They describe feature sparsity driven by gradient-based attribution as a regularizer and report improved model faithfulness.Source: §6 · p. 9

Checklist

The checklist reports that the paper’s main claims and scope are accurately represented and that limitations are described in section 4. It also records where experimental and asset-related details are supplied or deemed not applicable.

Checklist

  • The checklist marks the main claims and scope as accurately represented and says limitations are described in section 4.Source: Checklist · p. 13
  • Training details, including data splits and hyperparameters, and error bars are reported in the supplemental material.Source: Checklist · p. 13
  • The checklist reports citing creators of existing assets, while asset licensing and new-asset release are marked not applicable.Source: Checklist · p. 14

Experimental Details

Across gradient-based saliency methods, saliency guided training improves feature-removal evaluation, while image maps are reported as less noisy and more focused on objects. Fine-tuning improves MNIST accuracy-drop performance and reduces CIFAR10 training epochs relative to training from scratch.

Experimental Details

Datasets and Classifiers

  • The image experiments include MNIST with a two-convolutional-layer classifier, CIFAR10 with ResNet18, and BIRD with VGG16.Table 3 describes k as the percentage of overall features masked during saliency guided training.Source: Datasets and Classifiers · p. 14, Table 3 · p. 14

Saliency Map Quality for Images

  • Image saliency maps are reported as less noisy and more object-focused after saliency guided training, with high-saliency features more separated from near-zero gradients.The compared examples were correctly classified by both models, and gradients were scaled per sample to values between -1 and 1.Source: Saliency Map Quality for Images · p. 15, Figure 7 · p. 16, Figure 8 · p. 17

Model Accuracy Drop

  • Saliency guided training improves model-degradation performance across gradient-based saliency methods, regardless of the method used.Figure 10 reports mean and standard error; a steeper accuracy drop after removing highly salient features indicates better performance.Source: Figure 10 · p. 15, Model Accuracy Drop · p. 15

Fine-tuning with Saliency guided Training

  • Fine-tuning improves MNIST area-under-accuracy-drop performance over traditional training; on CIFAR10 it takes 70 epochs versus 118 for traditional and 124 for saliency training.The MNIST comparison uses AUC, where a smaller value indicates better performance; the passage does not provide the table values.Source: Fine-tuning with Saliency guided… · p. 15, Table 4 · p. 15

Saliency Guided Training for Language

For language tasks, the paper evaluates explanations with ERASER and finds that replacing low-saliency words with the previous high-saliency word performs best across three datasets. It also reports evaluating saliency guided training on multivariate time-series data using time-series MNIST and synthetic data.

Saliency Guided Training for Language

  • Language experiments use ERASER, with GloVe embeddings and a bidirectional LSTM classifier across the listed datasets.The dataset overview is in Table 5.Source: Saliency Guided Training for Language · p. 19, Table 5 · p. 19
  • Across three language datasets, replacing each low-saliency word with the previous high-saliency word gives the best results among the tested masking functions.The authors note that masking choice may depend on the dataset and leave finding optimal functions for different tasks to future work.Source: Saliency Guided Training for Language · p. 19
  • ERASER evaluation discretizes continuous token importance scores into rationales using dataset-specific top-k thresholds and reports comprehensiveness and sufficiency measures.The passage describes AOPC aggregation over token-importance bins and comparison with random scores.Source: Saliency Guided Training for Language · pp. 19–20

Saliency Guided Training for Time Series

Saliency Maps Quality for Multivariate Time Series

On MNIST treated as a multivariate time series, saliency-guided training visibly improves map quality across networks, most notably TCNs. On synthetic time-series benchmarks, its effects vary by architecture and saliency method, with improved recall but not precision for Transformers.

Saliency Maps Quality for Multivariate Time Series

Quantitative Analysis on Synthetic Data

  • The synthetic benchmark comprises 10 datasets that examine different time-series design aspects, and compares four neural architectures trained traditionally or with saliency-guided training.The experiments report test accuracy and assess saliency-map precision and recall through feature replacement and accuracy-drop measurements.Source: Quantitative Analysis on Synthetic Data · pp. 20, 22, 23, Table 6 · p. 22, Table 7 · p. 22
  • For LSTMs, Integrated Gradients with saliency-guided training achieves the best preference; for LSTMs with Input-Cell Attention, guided training improves most methods, with DeepSHAP best for precision and DeepLIFT best for recall.The table reports differences in weighted AUP and AUR.Source: Table 8 · p. 23, Table 9 · p. 23, Quantitative Analysis on Synthetic Data · p. 23
  • For TCNs, saliency-guided training improves most saliency methods; Integrated Gradients and DeepSHAP produce the best precision, while several methods perform similarly for recall.The best recall performance is achieved with saliency-guided training.Source: Quantitative Analysis on Synthetic Data · p. 23, Table 10 · p. 24
  • For synthetic time-series benchmarks, saliency-guided training improves recall but not precision for Transformers; traditional training with Gradient SHAP has the best precision, while guided training with Integrated Gradients has the best recall.The benchmark comparison is reported for different Transformer and saliency-method pairs.Source: Quantitative Analysis on Synthetic Data · p. 23, Table 11 · p. 24

Related papers

Loading 2111.14338v1…