PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical interpretations for neural network predictions

Chandan Singh, W. James Murdoch, Bin Yu

arXiv:1806.05337v2cs.LGcs.AIcs.CLcs.CVstat.ML

TL;DR

DNNs are difficult to interpret because their nonlinear relationships are hard to visualize. The paper introduces ACD, which hierarchically clusters predictive feature groups and assigns their contributions, and reports improved human model selection and trust alongside robustness to adversarial perturbations.

  • Problem

    DNNs’ complex nonlinear relationships are difficult to visualize, while existing methods do not provide a principled way to search and display important feature groups.

  • Method

    ACD generalizes contextual decomposition and uses its group-level interaction scores as the joining metric in agglomerative hierarchical clustering.

  • Results

    ACD helps users identify the more accurate model and is rated more trustworthy overall; its hierarchy is also largely robust to adversarial perturbations.

  • Takeaways & Limitations

    Hierarchical interpretations expose nonlinear contributions in individual DNN predictions while supporting diagnosis of incorrect predictions and dataset bias.

Abstract

from arXiv · show

Deep neural networks (DNNs) have achieved impressive predictive performance due to their ability to learn complex, non-linear relationships between variables. However, the inability to effectively visualize these relationships has led to DNNs being characterized as black boxes and consequently limited their applications. To ameliorate this problem, we introduce the use of hierarchical interpretations to explain DNN predictions through our proposed method, agglomerative contextual decomposition (ACD). Given a prediction from a trained DNN, ACD produces a hierarchical clustering of the input features, along with the contribution of each cluster to the final prediction. This hierarchy is optimized to identify clusters of features that the DNN learned are predictive. Using examples from Stanford Sentiment Treebank and ImageNet, we show that ACD is effective at diagnosing incorrect predictions and identifying dataset bias. Through human experiments, we demonstrate that ACD enables users both to identify the more accurate of two DNNs and to better trust a DNN's outputs. We also find that ACD's hierarchy is largely robust to adversarial perturbations, implying that it captures fundamental aspects of the input and ignores spurious noise.

1 INTRODUCTION

ACD addresses the black-box nature of DNNs by organizing feature contributions hierarchically, helping users interpret nonlinear relationships and compare model reliability. The method extends group-level importance scoring across architectures and data types, with demonstrations spanning language and vision tasks.

  • Motivation: DNNs learn complex nonlinear relationships but remain difficult to visualize, limiting their use in medicine, policy, science, and scrutinized industrial settings.The passage also connects black-box use with concerns about fairness.
  • ACD: ACD produces a hierarchy of input-feature clusters and each cluster’s contribution to a trained DNN prediction.The hierarchy is optimized to identify feature groups the network learned are predictive.
  • ACD: ACD combines generalized contextual decomposition with hierarchical saliency, using group importance to guide agglomerative clustering.The approach extends contextual decomposition beyond LSTMs, including convolutional neural networks.
  • Evaluation: The method is evaluated on LSTMs for Stanford Sentiment Treebank and CNNs for MNIST and ImageNet, alongside human judgments of interpretability and trust.The experiments also examine whether ACD can help users select the more accurate model and whether its hierarchy withstands adversarial perturbations.
  • Background: Hierarchical interpretations address the difficulty of searching among many possible feature groups while preserving interactions that feature-level explanations miss.Prior methods either emphasize individual features or provide group scores without a principled search and display procedure.

3 METHOD

ACD extends contextual decomposition to general DNN architectures and combines group-level importance scores with hierarchical clustering. It builds feature hierarchies by iteratively selecting and merging groups according to CD-based interaction scores.

  • 3.1 CONTEXTUAL DECOMPOSITION (CD) IMPORTANCE SCORES FOR GENERAL DNNS: For a feature group, generalized CD decomposes the network logits into β(x), the group’s importance, and γ(x), contributions from the remaining input.Layer-wise decompositions preserve β_i + γ_i = g_i(x) before composing the scores across the network.
  • 3.1 CONTEXTUAL DECOMPOSITION (CD) IMPORTANCE SCORES FOR GENERAL DNNS: ACD generalizes contextual decomposition from LSTMs to a wide range of neural architectures, including CNNs with residual and recurrent architectures.The generalized method introduces layer-wise decompositions for convolutional, max-pooling, ReLU, and dropout layers.
  • 3.1 CONTEXTUAL DECOMPOSITION (CD) IMPORTANCE SCORES FOR GENERAL DNNS: The method applies architecture-specific CD updates for max-pooling, ReLU, dropout, convolutional, and fully connected layers.For max-pooling, it uses the selected indices; for ReLU, it updates β and γ from separate activations; dropout is applied separately to both terms.
  • 3.2 AGGLOMERATIVE CONTEXTUAL DECOMPOSITION (ACD): ACD uses CD interaction scores as the joining metric in agglomerative hierarchical clustering, starting from individual features and iteratively combining them.The procedure combines the feature-group scoring framework with a hierarchy-building algorithm.
  • 3.2 AGGLOMERATIVE CONTEXTUAL DECOMPOSITION (ACD): At each step, ACD selects groups within k% of the highest-scoring group and ranks candidate expansions by the score difference between candidate and original groups.The hyperparameter k is fixed at 95 for images and 90 for text; candidates add adjacent words or pixels.
  • 3.2 AGGLOMERATIVE CONTEXTUAL DECOMPOSITION (ACD): ACD termination depends on the application: sentiment analysis stops when all words are selected, whereas image interpretations use a predefined number of iterations before merging remaining groups.The remaining image groups are merged using the same selection criteria.

4 RESULTS

ACD is evaluated qualitatively and quantitatively across LSTMs and CNNs, showing how hierarchical interpretations diagnose model behavior and support human reasoning about predictions. Human studies find benefits for model selection and trust, while additional experiments show stability under adversarial perturbations.

  • Evaluation: ACD is evaluated on LSTMs trained on SST and CNNs trained on MNIST and ImageNet, with qualitative and quantitative analyses of its interpretive benefits.The evaluation includes visualization, human experiments, and adversarial-perturbation tests.
  • Qualitative analyses: ACD identifies sentiment-bearing phrases and exposes an incorrect positive interaction between two phrases in an SST LSTM prediction.The phrases are individually associated with positive and negative sentiment, but their combination leads to the incorrect prediction.
  • Qualitative analyses: ACD reveals that an ImageNet VGG model uses hockey skates as evidence for the class “puck,” exposing a dataset-related bias.The model distinguishes “puck” from other top classes only when skate and puck patches merge, suggesting that skates act as corroborating features.
  • Human experiments: ACD outperforms prior non-hierarchical methods for identifying the more accurate model on ImageNet, while its advantage is smaller or inconclusive on MNIST and partly suggestive statistically.On SST, only ACD and CD outperform random selection; on ImageNet, ACD is the only method above random chance, with gaps from other methods statistically suggestive.
  • Human experiments: ACD receives an average trust rank of 3.5 out of 4 on ImageNet and significantly exceeds all other methods on SST and ImageNet.On MNIST, results are inconclusive except for a significant difference between ACD and occlusion.
  • Adversarial robustness: ACD hierarchies are largely robust to adversarial perturbations, with substantially higher stability than hierarchies produced by replacing CD with occlusion.The comparison uses correlations across five attack types and 100 randomly chosen predictions per attack.

5 CONCLUSION

The paper introduces ACD, a hierarchical interpretation algorithm that captures non-linear feature contributions in individual DNN predictions. Experiments show benefits for human evaluation, diagnosing errors and bias, and robustness to adversarial perturbations.

  • ACD is the first method presented to use a hierarchy for interpreting individual neural network predictions.
  • ACD automatically detects and displays non-linear contributions to individual DNN predictions.
  • Human experiments and examples demonstrate benefits for diagnosing incorrect predictions and identifying dataset bias.
  • ACD’s hierarchy is robust to adversarial perturbations in CNNs, implying that it captures fundamental input aspects and ignores spurious noise.

S1 CD SCORE COMPARISONS

The supplementary comparisons examine how CD represents feature contributions relative to perturbation-based baselines. CD tracks blob and non-blob contributions through the network, with differences from baselines emerging in later layers.

  • CD decomposes a DNN feedforward pass into contributions from the blob of interest and everything else.
  • CD tracks blob and non-blob contributions throughout the network, unlike perturbation-based comparisons that alter the input.
  • The build-up baseline sets everything except the patch of interest to a reference value, often zero.
  • CD, occlusion, and build-up representations are similar in early layers, but differences emerge in later layers.
  • CD scores better identify information relevant to predicting the correct class in four compared images.

S2 TOP SCORING ACD PHRASES

The supplementary material extends the SST phrase analysis by listing the highest-scoring positive and negative phrases selected by ACD across phrase lengths.

  • The extended table contains the top five positive and negative phrases of each length extracted by ACD from an LSTM trained on SST.

S3 ACD EXAMPLES

The supplementary material provides additional ACD visualizations across SST, MNIST, and ImageNet, including top- and lowest-predicted examples and SST phrase results.

  • Additional ACD visualizations were automatically selected using the same criteria as the human experiments.
  • SST supplementary examples include both top-predicted and lowest-predicted cases corresponding to the main SST visualization.
  • MNIST supplementary examples include top-predicted and lowest-predicted cases using the model and figure interpretation from the MNIST section.
  • ImageNet supplementary examples include top-predicted and lowest-predicted cases corresponding to the ImageNet visualization.
  • The SST supplementary table reports top-scoring phrases of different lengths extracted by ACD, with identified phrases described as matching their positive or negative polarity.

S4 HUMAN EXPERIMENTS EXPERIMENTAL SETUP

The human experiments compare whether participants can identify the more accurate of two models and which interpretation method makes a model’s predictions most trustworthy, across text and vision tasks.

  • Choosing the better model: Participants compared visualizations from two models and selected which model they believed had higher predictive accuracy.The text task used positive-versus-negative movie-review classifiers, while the vision tasks used image classifiers.
  • Interpretation visualizations: The visualizations represented contributions from individual words or pixels, or from groups of words or pixels.For vision examples, group visualizations could show multiple levels of granularity.
  • Interpretation visualizations: Blue indicated positive contributions, while red indicated negative contributions in the text and vision visualizations.For image heat maps, colors represented positive or negative signals for each class.
  • Trust evaluation: Participants ranked four visualization methods by how much they trusted the predictions from the more accurate model.They assigned ranks from 1 to 4, with 1 representing the most trustworthy interpretation.
  • Vision task: The vision instructions noted that the network’s displayed predicted class was wrong half the time in the examples.Each example showed the raw image followed by five class-specific heat maps.

S5 ACD ON ADVERSARIAL EXAMPLES

ACD produced substantially similar hierarchies for an original image and an adversarially perturbed image whose predicted class changed from 0 to 6.

  • Hierarchy robustness: ACD hierarchies for an image predicted as 0 were substantially similar before and after an adversarial perturbation that changed the prediction to 6.The example used a DeepFool attack.

S6 ADVERSARIAL ATTACK EXAMPLES

The attack examples show prediction changes after adversarial perturbations, while modifications to the CNN decomposition reduced noise and extreme score magnitudes.

  • Adversarial attacks: Adversarial perturbations altered the predicted class for the displayed image examples.The original image in each example was correctly predicted as class 0, whereas the perturbed image received a different prediction.
  • CNN decomposition: Partitioning the bias between γ_i and β_i qualitatively reduced noise in the CNN heat maps.This was one of two modifications made to the naive extension of contextual decomposition to CNNs.
  • CNN decomposition: Changing the bias partition and ReLU decomposition qualitatively improved scores and avoided extremely large magnitudes.The figure compares unit-level contextual decomposition scores with scores from the naive CNN extension.
Loading 1806.05337v2…