PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretations are useful: penalizing explanations to align neural networks with prior knowledge

Laura Rieger, Chandan Singh, W. James Murdoch, Bin Yu

arXiv:1909.13584v4cs.LGcs.CVstat.ML

TL;DR

Explanation methods can reveal what deep neural networks learn, but often offer insight without a way to act on it. This paper introduces CDEP, which directly penalizes feature importances using existing explanations; across toy and real data, it removes bias and improves predictive accuracy.

  • Problem

    Existing explanation methods can reveal relationships learned by deep neural networks but often provide insight without a way to act on the explanation.

    Source: §1 · p. 1

  • Method

    CDEP uses existing explanations to let users directly penalize feature or interaction importances, incorporating prior knowledge into a neural network.

    Source: §1 · p. 1

  • Results

    Across toy and real datasets, CDEP removes bias and improves predictive accuracy.

    Source: §5 · p. 8

  • Takeaways & Limitations

    CDEP can incorporate domain knowledge, with its effectiveness depending on the quality of prior knowledge used to set explanation targets.

    Source: §5 · p. 8

    Future work includes extending CDEP to more complex settings and incorporating finer-grained explanations and interaction penalizations.

    Source: §5 · p. 8

Abstract

from arXiv · show

For an explanation of a deep learning model to be effective, it must provide both insight into a model and suggest a corresponding action in order to achieve some objective. Too often, the litany of proposed explainable deep learning methods stop at the first step, providing practitioners with insight into a model, but no way to act on it. In this paper, we propose contextual decomposition explanation penalization (CDEP), a method which enables practitioners to leverage existing explanation methods in order to increase the predictive accuracy of deep learning models. In particular, when shown that a model has incorrectly assigned importance to some features, CDEP enables practitioners to correct these errors by directly regularizing the provided explanations. Using explanations provided by contextual decomposition (CD) (Murdoch et al., 2018), we demonstrate the ability of our method to increase performance on an array of toy and real datasets.

1. Introduction · 2. Background

CDEP lets practitioners use prior importance scores to penalize feature or interaction explanations, guiding neural networks toward both correct predictions and explanations. The paper presents it as adaptable to differentiable explanation methods and evaluates it across tasks involving spurious features and fairness.

1. Introduction

  • DNNs can learn spurious correlations from dataset bias or overfitting and exploit sensitive-attribute biases; CDEP aims to penalize such learned relationships.Source: §1 · p. 1
  • CDEP uses prior importance scores to penalize selected feature or interaction explanations, training a network to produce both the correct prediction and explanation.Source: §1 · p. 1
  • CDEP focuses on contextual decomposition (Murdoch et al., 2018; Singh et al., 2018), but can adapt to differentiable explanation methods and arbitrary neural architectures.The paper states that it is often orders of magnitude faster and more memory efficient than recent gradient-based methods.Source: §1 · p. 1
  • CDEP improves skin-cancer prediction by teaching a classifier to ignore spurious confounders, and examples apply it to misleading color cues and fairness issues.The examples cover MNIST digit classification, text classification, and risk prediction.Source: §1 · pp. 1–2

2. Background

  • DNN explanation methods commonly assign local importance to individual features; approaches include gradient-based, decomposition-based, and other methods, while interaction-based work can expose feature hierarchies.Evaluation concerns have been raised about many such explanation methods (Adebayo et al., 2018; Nie et al., 2018).Source: §2 · p. 2
  • Prior work used attention or penalized gradients and attributions to steer models, reporting gains in performance or adversarial robustness (Ross et al., 2017; Bao et al., 2018; Du et al., 2019; Ross & Doshi-Velez, 2018).Attention is not necessarily equivalent to explanation (Jain & Wallace, 2019).Source: §2 · p. 2
  • Dataset augmentation and adversarial training are alternative constraints, but can be costly or dataset-specific and do not clearly connect explanation-based discovery of learned relationships to correcting them.Source: §2 · p. 2

3. Methods

CDEP augments prediction training with penalties that align model explanations to practitioner-supplied prior knowledge. It uses contextual decomposition to regularize feature groups and interactions, while allowing other differentiable explanation methods.

3.1. Augmenting the loss function

  • The augmented objective combines standard prediction loss with explanation error, weighted by λ; the explanation error compares model explanations with user-provided targets.The prediction loss penalizes wrong predictions, while the explanation error penalizes wrong explanations.Source: §3.1 · pp. 2–3

3.2. Contextual decomposition (CD)

  • CD decomposes logits into feature-group importance β and remaining contribution γ, and captures interactions beyond individual feature importance.This makes CD suited to regularizing complex feature importance.Source: §3.2 · p. 3

3.3. CDEP objective function

  • CDEP applies SoftMax to CD scores and L1-penalizes user-targeted feature-group explanations; practitioners define the correct reasons, so biased targets can bias the model.The objective indexes examples and penalized feature subsets, summing across classes.Source: §3.3 · p. 3

3.4. Encoding domain knowledge as explanations

  • Practitioners can encode domain knowledge as human explanations, programmatic region or word rules, or interaction-level targets, including zeroing isolated-pixel importance to prevent local cues.Human explanations can be costly to collect.Source: §3.4 · pp. 3–4

3.5. Computational considerations

  • CD penalties use standard single forward and backward passes, enable finetuning only unfrozen final layers, and add only constant memory overhead versus standard training.Gradient-based attribution optimization requires twice-differentiable components and can be expensive; even simple gradient-based methods can more than double memory use.Source: §3.5 · p. 4

4. Results

The results evaluate CDEP on skin images, color- and decoy-biased MNIST, COMPAS fairness, and sentiment text. Reported findings include improved skin-test metrics, partial resistance to color bias, and lower, more balanced wrongful-conviction rates on COMPAS.

4.1. Ignoring spurious signals in skin cancer diagnosis

  • CDEP improves AUC and F1 on both ISIC test sets and achieves the best F1, surpassing both unpenalized versions.The test sets include images without patches and the full, biased set; the reported AUC and F1 standard deviations are below 0.006 and 0.012, respectively.Source: Table 1 · p. 5, §4.1 · p. 5
  • On ISIC, RRR does not improve base AUC and decreases performance in all considered metrics.The comparison used batch size 16 and learning rate 10^-5 because of memory constraints.Source: §4.1 · pp. 5–6

4.2. Combating inductive bias on variants of the MNIST dataset

  • On ColorMNIST’s inverted-color test, CDEP reaches 31.0% accuracy, while RRR and EG variants fail to exceed random accuracy.CDEP penalizes sampled single-pixel CD contributions to encourage reliance on groups of pixels that can represent digit shapes.Source: §4.2 · p. 6, Table 2 · p. 7
  • On DecoyMNIST, all methods recover roughly the base accuracy; with early layers frozen, CDEP becomes more efficient in memory and training time.The decoy patch is unentangled from other features and always appears in a corner; the best penalization parameter is selected by validation-set cross-validation.Source: §4.2 · pp. 6–7

4.3. Fixing bias in COMPAS

  • On COMPAS, hiding race does not remove the disparity; CDEP lowers wrongful-conviction rates for both races, narrows their gap, and keeps test accuracy relatively fixed.The unregularized model misclassifies black defendants as high-risk at more than twice the rate of white defendants; CDEP regularizes importance on race and its interactions.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4. Fixing bias in text data

  • The SST text experiment injects spurious signals into training reviews and uses a standard LSTM for binary sentiment classification.Source: §4.4 · p. 8

Positive

On SST sentiment data, CDEP improves test accuracy by helping models ignore injected spurious signals. The experiments examine three biased training variants, including one using gendered pronouns.

Positive

  • The examples praise Pacino, Keener, and Davies in positive snippets, while negative snippets criticize Arnold and Sandler.Source: Positive · p. 8
  • The three SST variants inject class-indicator words, semantically similar articles, or gendered pronouns; the indicator-word variant drives an unpenalized DNN to nearly random accuracy on unbiased test data.For the pronoun variant, accuracy is reported only on test sentences containing pronouns, which occur in about 2% of training data; performance on test sentences without pronouns remains unchanged.Source: Positive · p. 8
  • CDEP substantially improves predictive accuracy on unbiased test sets after training on biased data, successfully ignoring injected spurious signals across all scenarios.Source: Table 4 · p. 8, Positive · p. 8

5. Conclusion

CDEP penalizes neural networks to align with prior knowledge, including complex features and interactions, and can remove bias and improve predictive accuracy on toy and real data. Its effectiveness depends on the quality of the prior knowledge used to set explanation targets.

  • CDEP is the first method described here to penalize complex features and feature interactions, and is more computationally efficient than previous work.This efficiency enables use with more complex neural networks.Source: §5 · p. 8
  • CDEP can remove bias and improve predictive accuracy on toy and real data, with effectiveness depending on the quality of prior knowledge used to determine explanation targets.The method may incorporate domain knowledge from areas such as biology or physics.Source: §5 · p. 8
  • Future work includes extending CDEP to more complex settings and incorporating finer-grained explanations and interaction penalizations.The authors hope the work advances more rigorous use of interpretability methods as the field develops (Doshi-Velez & Kim, 2017; Murdoch et al., 2019).Source: §5 · p. 8

Supplement · S1. Additional details on MNIST Variants

The supplement details how ColorMNIST pixels are sampled and reports that Expected Gradients variants fail to exceed random accuracy, while CDEP captures and removes color bias. It also describes the setup and measurement conditions for DecoyMNIST runtime and memory comparisons.

S1. Additional details on MNIST Variants

S1.1. Runtime and memory requirements of different algorithms

S2. Image segmentation for ISIC skin cancer

The ISIC skin-cancer images were segmented into patches, then segment color statistics were used to identify regions differing from typical Caucasian skin tone and create binary maps.

S2. Image segmentation for ISIC skin cancer

S3. Additional heatmap examples for ISIC

Additional ISIC heatmaps show that CDEP preserves similar importance maps on cancerous and benign images without patches while ignoring patches. The networks also exploited a weaker spurious correlation between cancer and rulers in images, highlighting CDEP’s relevance in medical settings.

S3. Additional heatmap examples for ISIC

S4. Additional details about the COMPAS task · S5. Additional details about SST task · Positive

The supplemental sections detail the COMPAS model and data choices, then describe three SST bias variants that vary the decoy words, how they are introduced, and how prevalent they are. The SST variants include all-sentence decoys, semantically similar replacements in 27% of sentences, and gendered words appearing in 2%.

S4. Additional details about the COMPAS task

  • For COMPAS, a two-hidden-layer network with five neurons per layer used SGD, 0.1 dropout, and training until loss failed to improve for ten epochs.SGD used a learning rate of 0.01 and momentum of 0.9.Source: S4. Additional details about the… · p. 16
  • From 7,214 full-dataset samples, the analysis excluded 1,042 for missing recidivism information, as in ProPublica’s report, focused parity on Black and Caucasian groups, and omitted youth felony count because positive samples were very few.Source: S4. Additional details about the… · p. 16
  • COMPAS inputs included categorized age, sex, race, crime severity, prior felony count, and crime-description categories such as drugs, driving, violence, robbery, and other.The crime-description categories followed Madras et al. (2018).Source: S4. Additional details about the… · p. 16

S5. Additional details about SST task

  • In SST task 1, two decoy words were added to every sentence: “text” for positive examples and “video” for negative examples; neither word otherwise appeared in the dataset.Both decoy words had Word2Vec representations.Source: S5. Additional details about SST task · p. 16

Positive

  • SST task 2 replaced “the” and “a” in 27% of sentences, assigning one word exclusively to each class while using semantically similar replacements to preserve sentence structure.The words were replaced only where they appeared.Source: Positive · p. 17, Figure S9

S6. Network architectures and training

The experiments used task-specific network architectures and training setups for ISIC, MNIST, and SST. Explanation penalties added a fixed-rate λ hyperparameter tested over a wide range.

Loading 1909.13584v4…