PaperlayerView exact revision on arXiv

Source-linked AI summary

Explaining Explanations: Axiomatic Feature Interactions for Deep Networks

Joseph D. Janizek, Pascal Sturmfels, Su-In Lee

arXiv:2002.04138v3cs.LGstat.ML

TL;DR

Feature attributions reveal which inputs influence neural-network predictions but not why features matter or how they interact. The paper introduces Integrated Hessians, an architecture-independent extension of Integrated Gradients for pairwise interactions, and reports that it satisfies common-sense axioms and outperforms prior methods on simulated interaction benchmarks.

  • Problem

    Feature attributions identify important features but do not explain why they are important or how features interact in neural networks.

  • Method

    Integrated Hessians extends Integrated Gradients to quantify pairwise feature interactions across neural-network architectures while satisfying common-sense interaction axioms.

  • Results

    Integrated Hessians outperforms previous methods at identifying known interactions on simulated data and demonstrates interaction analyses across model classes and high-dimensional data.

  • Takeaways & Limitations

    Feature interactions provide information about model behavior that feature attributions alone do not reveal.

  • Takeaways & Limitations

    When neural networks learn confounded relationships, the reported feature attributions and interactions reflect those confounders rather than necessarily causally meaningful features.

Abstract

from arXiv · show

Recent work has shown great promise in explaining neural network behavior. In particular, feature attribution methods explain which features were most important to a model's prediction on a given input. However, for many tasks, simply knowing which features were important to a model's prediction may not provide enough insight to understand model behavior. The interactions between features within the model may better help us understand not only the model, but also why certain features are more important than others. In this work, we present Integrated Hessians, an extension of Integrated Gradients that explains pairwise feature interactions in neural networks. Integrated Hessians overcomes several theoretical limitations of previous methods to explain interactions, and unlike such previous methods is not limited to a specific architecture or class of neural network. Additionally, we find that our method is faster than existing methods when the number of features is large, and outperforms previous methods on existing quantitative benchmarks. Code available at https://github.com/suinleelab/path_explain

1 Introduction and Prior Work

Feature attributions identify important input features but not why they matter or how they interact. The paper motivates interaction explanations that apply broadly across neural architectures and satisfy common-sense axioms.

  • Feature attribution methods assign credit to input features according to their influence on a model prediction.
  • Feature attributions do not explain why features are important or how features interact within a model.
  • Feature interactions can help distinguish why more expressive neural networks outperform simpler models on language tasks.
  • Existing interaction methods are often restricted to particular architectures, such as feed-forward, Bayesian, recurrent, convolutional, or attention-based networks.
  • Most existing interaction methods do not satisfy common-sense axioms, while axiom-satisfying Shapley-based methods are computationally inefficient.
  • The paper proposes architecture-independent pairwise interaction quantification, axioms for interactions, ReLU-network computation, and evaluation on simulated data.

2 Explaining Explanations with Integrated Hessians

Integrated Hessians applies Integrated Gradients recursively so feature interactions quantify how one feature affects another feature’s importance. Its interaction values also satisfy completeness properties that connect interactions to model output differences.

  • The baseline x′ and input x define the integration path, and multiple training-set baselines can be averaged when one baseline is difficult to choose.
  • Integrated Hessians treats each Integrated Gradients attribution as a differentiable function and applies Integrated Gradients to explain feature importance through another feature.
  • The method explains pairwise interactions as the combined contribution of two features to f(x) − f(x′).
  • 2.1 Fundamental Axioms for Interaction Values: Interaction completeness requires the sum of Γi,j(x) terms to equal the model output difference between x and baseline x′.
  • 2.1 Fundamental Axioms for Interaction Values: Self-completeness interprets Γi,i(x) as feature i’s main effect after subtracting its interactions with all other features.
  • 2.1 Fundamental Axioms for Interaction Values: Integrated Hessians is also described as satisfying generalized sensitivity, linearity, and symmetry properties for interaction values.

3 Smoothing ReLU Networks

ReLU networks have zero second derivatives almost everywhere, preventing direct Hessian methods from detecting interactions. Replacing ReLU with smooth SoftPlus activations enables interaction computation and can reduce the gradient evaluations needed for Integrated Hessians.

  • ReLU networks are piecewise linear with zero second partial derivatives, so previous Hessian-based methods fail to detect their interactions.
  • SoftPlus provides a smooth ReLU approximation with well-defined higher-order derivatives, enabling Integrated Hessians to calculate interactions in ReLU networks.
  • Smoothing a network with SoftPlus allows Integrated Hessians values to be accurately approximated with fewer gradient calls.
  • For a one-layer SoftPlus network, the paper bounds interpolation points needed for a target error tolerance using the input dimension and SoftPlus parameter.
  • SoftPlus smoothing makes gradients more similar in direction along the path from reference to foreground sample.

4 Explanation of XOR function

In an XOR network, Integrated Gradients assigns identical attributions to samples with both features off or on. Integrated Hessians distinguishes them by identifying the negative interaction that cancels the features’ individual positive effects.

  • An XOR network produces low output when both binary features are off or both are on, despite high output when either feature is on alone.
  • Integrated Gradients gives identical attributions to the both-off and both-on samples when using a zeros baseline.
  • Integrated Hessians identifies a negative interaction when both features are on, distinguishing that sample from the baseline-equivalent both-off sample.
  • Path integration lets Integrated Hessians detect the XOR interaction even though the input Hessian is zero on the saturated data manifold.

5 Empirical Evaluation

The empirical evaluation compares Integrated Hessians with four existing interaction methods on computational cost, interaction identification, rank correlation, and sanity checks. Integrated Hessians becomes more tractable as feature count grows and consistently performs best across the reported synthetic interaction benchmarks.

  • The evaluation compares Integrated Hessians with the Shapley Interaction Index, Generalized Contextual Decomposition, Neural Interaction Detection, and the input Hessian.
  • Computation time: Estimated 1000 hours were required for the 500-feature Monte Carlo Shapley Interaction Index case, which did not finish with 200 samples.
  • Computation time: As feature count grows, Integrated Hessians becomes more tractable because Hessian backpropagation parallelizes on GPUs while alternatives require at least O(d^2) forward passes.
  • Quantitative comparison: Integrated Hessians most quickly identifies the largest interactions on gtanhsum and outperforms all existing methods across five tested interaction types.
  • Quantitative comparison: Integrated Hessians consistently outperforms other methods in rank correlation with true interactions on multiplicative and pairwise min/max synthetic datasets, and passes both sanity checks.

6 Applications of Integrated Hessians

Applications show that Integrated Hessians reveals interactions unavailable from feature attributions alone in sentiment models and drug-combination response prediction. These interactions expose language patterns, model performance differences, and biologically grounded variation in drug effects.

  • NLP: Integrated Hessians reveals sentiment-model interactions that feature attributions alone do not, including positive interaction between “painfully” and “funny.”
  • NLP: DistilBERT captures positive interactions between “not” and “bad” that a CNN does not fully capture, helping explain their validation accuracy gap.
  • Drug combination response prediction: In drug-response prediction, Venetoclax is the most important first-order feature, but its predicted response varies across samples.
  • Drug combination response prediction: Venetoclax and Artemisinin exhibit a strong negative interaction whose magnitude is associated with cancer-pathway expression and matches external ground truth.
  • NLP: Saturation appears when multiple negative adjectives describe one noun: the adjectives interact positively, while each individual adjective contributes less.

7 Conclusion

The paper concludes that Integrated Hessians explains pairwise feature effects, satisfies common-sense axioms, and performs well in identifying known interactions. It also extends interaction analysis to ReLU networks and demonstrates applications across model classes and high-dimensional data.

  • Integrated Hessians interprets pairwise interactions as either combined feature effects or one feature’s importance explained through another.
  • The method provably satisfies common-sense axioms that previous methods do not and outperforms them on simulated interaction-identification tasks.
  • The paper explains how to obtain interactions from ReLU networks despite their zero second derivatives and uses softplus at explanation time as an intuitive, efficient alternative.
  • Experiments demonstrate utility for understanding model-class performance gaps and discovering learned patterns in high-dimensional data.

Broader Impact

Integrated Hessians is presented as technically sound and broadly applicable, with potential benefits for stakeholders, but its interpretation remains bounded by model confounding, possible manipulation, and limited interaction order.

  • Broader Impact: Integrated Hessians could help practitioners debug deployed neural networks and help consumers understand algorithmic decisions affecting them.The paper frames these as potential benefits for a wide variety of stakeholders when the method is used properly.
  • Broader Impact: In biological applications, interactions are causal with respect to the model but not necessarily the true data-generating process.Correlated features, noise, batch effects, and confounding can make the model depend on spurious correlates rather than causally meaningful features.
  • Broader Impact: Loan explanations may support GDPR-related explanation requirements, but the method may be vulnerable to adversarial manipulation by stakeholders hiding unfair dependencies.The paper recommends trustworthy deployers such as impartial auditors or combining explanation tools with tools that modify underlying models.
  • Derivation and Assumptions: The method’s derivation assumes sufficient continuity for integration and differentiation to be interchangeable, including continuous first and second derivatives over the integration region.The appendix states these are conditions for applying the Leibniz Integral Rule and notes common activation functions satisfy relevant conditions.
  • Approximation in Practice: Around 50 samples often suffice to approximately satisfy interaction completeness, although larger models and higher-dimensional data generally require more.The paper reports that fewer than 300 samples are generally required, with the precise number depending on model and data complexity.
  • Theoretical and Practical Scope: Integrated Hessians does not generate higher-order interactions, although recursive application could theoretically generate kth-order interactions.The paper does not discuss that recursive extension and contrasts completeness with the Shapley Interaction Index’s recursive axioms.

B.3.2 Rank Correlation and Sanity Checks

The paper evaluates Integrated Hessians on synthetic interaction datasets and tests whether its attributions respond to network and data randomization. It reports stronger identification of true interactions and zero correlation under randomized sanity checks.

  • Synthetic interaction benchmarks: The experiments use multiplicative and Min/Max synthetic interactions with known ground truth for quantitative comparison.The datasets contain five independent Gaussian features, and neural networks are trained to validation R^2 > 0.99.
  • Synthetic interaction benchmarks: Integrated Hessians outperforms all existing methods across the interaction types in the ablation comparison.Figure 7 reports means and standard deviations across five retrainings as interactions are ablated.
  • Synthetic interaction benchmarks: Table 3 compares local and global rank correlation with true interactions for multiplicative and non-multiplicative interaction types.The comparison includes both interaction scales and multiple detection methods.
  • Sanity checks: The method passes the randomization sanity check: Spearman correlation between true and randomized interactions is 0 in both tested settings.The test compares explanations from the fitted network with explanations from randomized network weights or randomized data labels.

C.1 Proof of Theorem 1

The proof analyzes how activation smoothing affects numerical convergence for Integrated Gradients and Integrated Hessians, then extends the analysis to deeper networks. It concludes that decreasing the softplus β parameter reduces the interpolation points needed for a target error.

  • Theoretical convergence: Theorem 2 bounds the interpolation points needed to approximate Integrated Hessians for a one-layer softplus network with d input features and error tolerance ϵ.The bound depends on the softplus parameter β and network weights.
  • Theoretical convergence: For fixed network weights and path length, the required interpolation points decrease as β decreases because smoothing reduces derivative magnitudes.The proof derives this relationship for Integrated Gradients and extends it to Integrated Hessians.
  • Theoretical convergence: Diagonal Integrated Hessian terms incur an additional β-dependent error factor, while aggregating all terms adds a d^2 factor to the completeness bound.The diagonal terms include main effects, and the total convergence error sums individual interaction terms.
  • Deep-network experiments: Empirical five-layer and ten-layer networks show faster decreases in interaction completeness error as β is reduced.The experiments use networks with 50 nodes per hidden layer and 100 features; the deeper network shows a more dramatic effect.

D.4 Additional Examples of Interactions

Additional sentiment examples show that Integrated Hessians captures saturation, contextual reversals, intuitive phrase interactions, and non-adjacent word relationships. Some less intuitive pairwise interactions may reflect limited data or higher-order effects.

  • Saturation and context: As negative adjectives accumulate, their pairwise interactions become positive because each adjective contributes less to an already saturated negative output.DistilBERT confidence changes only slightly from 0.9981 to 0.9984 as four negative adjectives are added.
  • Interpretive caveats: Some interactions are less intuitive, potentially indicating insufficient training data or higher-order effects beyond word pairs.The paper gives “a good” as a possible saturation, higher-order, or learned-pattern example and identifies “better suited” as strongly negative.
  • Intuitive phrase patterns: Interactions identify intuitive sentiment patterns, including strongly positive “beautifully observed,” “miraculously unsentimental,” and “quite compelling.”The examples come from the Stanford Sentiment Analysis Treebank validation set.
  • Saturation and context: The word “nonprofessional” has a negative main effect, but surrounding context makes its overall attribution positive.This example separates a feature’s standalone contribution from contextual interaction effects.
  • Intuitive phrase patterns: The strongest interaction pairs need not be adjacent words, as illustrated by “artfully” and “technical.”The interaction matrix therefore captures relationships beyond neighboring tokens.

E.1 Heart disease prediction

The paper aggregates Expected Hessians interactions on clinical and physics datasets to inspect global model behavior. The examples recover clinically and physically meaningful patterns while also exposing unintuitive trends and feature-specific interactions.

  • Heart disease prediction: The Cleveland heart disease dataset contains 298 patients and 13 features for predicting coronary artery disease.The features include demographics, clinical measurements, symptoms, electrocardiographic information, and calcium-vessel counts.
  • Heart disease prediction: The trained heart disease network achieves held-out accuracy 0.8667, true positive rate 0.8214, and true negative rate 0.9062.The model is a two-layer softplus network trained for 200 epochs.
  • Heart disease interactions: The model learns that increased calcium accumulation across major coronary vessels indicates increased disease risk.The number of affected vessels ranges from 0 to 3 and is described as a strong predictor of coronary artery disease.
  • Heart disease interactions: Expected Hessians reveals that higher maximum exercise heart rate indicates lower disease risk more strongly for men than for women.This pattern is represented as an interaction between maximum achieved heart rate and patient gender.
  • Pulsar prediction: On the pulsar task, kurtosis of the integrated profile and standard deviation of the DM-SNR curve interact when kurtosis is roughly between 0 and 2.The second feature helps distinguish negative samples with standard deviation below 40, and the model recovers a peak positive interaction in the relevant range.

F.1 Data description

The study models anti-cancer drug-combination responses using RNA-seq gene expression, drug indicators, and pathway embeddings in a feed-forward network. The dataset and preprocessing support attribution and interaction analyses, including biological synergy evaluation.

  • 12,362 samples measured responses to two-drug combinations tested in cancer cells from patients.
  • RNA-seq measurements were converted to FPKM and restricted to protein-coding transcripts.FPKM normalizes counts by transcript length and total sample reads.
  • The model used 15,377 RNA-seq features sparsely connected to 1,077 pathway nodes before concatenation with drug identity and target indicators.Pathways came from KEGG, BioCarta, or Reactome, with nonlinear Tanh pathway embeddings.
  • The feed-forward network had hidden layers of 500 and 250 Tanh-activated nodes and was trained with mean squared error using Adam.Training stopped after validation error failed to improve for 10 epochs, with an optimum at 200 epochs.
  • Integrated Gradients and Integrated Hessians explanations used the all-zeros reference and more than 256 interpolation points.
  • Biological drug synergy was evaluated using additional single-drug response data unavailable to the model during training.Combination responses were compared with expected additive responses to assess interactions.
Loading 2002.04138v3…