PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Robust Interpretability with Self-Explaining Neural Networks

David Alvarez-Melis, Tommi S. Jaakkola

arXiv:1806.07538v2cs.LGstat.ML

TL;DR

The paper addresses the limited attention given to interpretability during model learning compared with post-hoc explanation. It designs self-explaining models with architecturally explicit explanations and regularized faithfulness and stability, reporting robust explanations while retaining complex modeling capacity.

  • Problem

    Post-hoc explanations for already-trained complex models are common, while models designed to explain themselves during learning have received less attention.

  • Method

    The framework progressively generalizes linear models using input-dependent coefficients and interpretable basis concepts, while regularization enforces explanation faithfulness and stability.

  • Results

    The proposed self-explaining models consistently and substantially outperform other methods on robustness across various datasets.

  • Takeaways & Limitations

    The framework offers a direction for reconciling model complexity with interpretability through intrinsic, robust explanations.

  • Takeaways & Limitations

    Interpretable basis concepts remain challenging and ill-defined when expert knowledge is unavailable, requiring fidelity, diversity, and grounding desiderata.

Abstract

from arXiv · show

Most recent work on interpretability of complex machine learning models has focused on estimating $\textit{a posteriori}$ explanations for previously trained models around specific predictions. $\textit{Self-explaining}$ models where interpretability plays a key role already during learning have received much less attention. We propose three desiderata for explanations in general -- explicitness, faithfulness, and stability -- and show that existing methods do not satisfy them. In response, we design self-explaining models in stages, progressively generalizing linear classifiers to complex yet architecturally explicit models. Faithfulness and stability are enforced via regularization specifically tailored to such models. Experimental results across various benchmark datasets show that our framework offers a promising direction for reconciling model complexity and interpretability.

1 Introduction

The paper argues that interpretability should be built into complex models during learning rather than estimated afterward, and proposes explicitness, faithfulness, and stability as core desiderata.

  • Interpretability can limit machine-learning adoption in decision-critical domains such as medicine and law.
  • Self-explaining models incorporate interpretability architecturally and enforce it through regularization.
  • The paper defines explicitness, faithfulness, and stability as desiderata, with stability requiring similar inputs to yield similar explanations.
  • Post-hoc methods explain predictions locally using limited model access or oracle queries, but their locality and identifiability present known challenges.
  • The proposed framework provides intrinsic explanations, optimization procedures for the desiderata, and quantitative evaluation metrics.

2 Interpretability: linear and beyond

The framework progressively generalizes linear models by making coefficients input-dependent, replacing raw features with interpretable concepts, and constraining aggregation to preserve feature-level meaning.

  • Linear models are interpretable because features are anchored to observations, coefficients quantify contributions, and additive terms preserve feature-level effects.
  • 2.1 Generalized coefficients: Input-dependent coefficients enrich linear models while remaining locally interpretable when their variation is constrained around nearby inputs.
  • 2.2 Beyond raw features – feature basis: Interpretable basis concepts replace raw variables with higher-order features such as aggregates, expert-designed extractors, prototypes, or constrained learned representations.
  • 2.2 Beyond raw features – feature basis: Each concept remains a scalar feature-presence degree, while its corresponding input-dependent coefficient acts as a local coefficient.
  • 2.3 Further generalization: A generalized aggregation function should be permutation invariant, isolate individual concept effects, and preserve relevance signs and relative magnitudes.
  • 2.3 Further generalization: The framework extends to multiple outputs by making each concept relevance a vector, with classification focusing on the predicted class.

3 Self-explaining models

Self-explaining models constrain concept-dependent relevance functions locally and combine them through structured aggregators, yielding high-capacity architectures with explanations defined by concepts and influence scores.

  • 3 Self-explaining models: The model class bounds relevance variation relative to changes in interpretable concepts, using a local rather than global stability condition.
  • 3 Self-explaining models: Local difference-boundedness permits the stability constants and neighborhoods to vary with the reference input.
  • 3 Self-explaining models: A self-explaining model uses an input-dependent relevance function, concept representation, and aggregation function to produce predictions.
  • 3 Self-explaining models: Its properties require a monotone, additively separable aggregator, nonnegative partial effects, locally stable relevance, interpretable concepts, and a small concept count.
  • 3 Self-explaining models: For an input, the explanation consists of basis concepts paired with their influence scores.
  • 3 Self-explaining models: When relevance functions are neural networks, the resulting SENNs retain high modeling capacity and can be trained end-to-end with back-propagation.
  • 3 Self-explaining models: Gradient regularization adds λLθ(f) to the classification loss, trading prediction performance against relevance stability and interpretability.

4 Learning interpretable basis concepts

SENN learns a small set of interpretable basis concepts instead of relying only on raw input features. These concepts are trained for fidelity, diversity, and grounding, then represented through prototypes while the model preserves concept-level interpretability.

  • Raw pixels can produce noisy, incoherent, and fragile explanations, motivating higher-level basis concepts for high-dimensional inputs.The paper notes that raw-feature explanations are sensitive to imperceptible artifacts and simple transformations.
  • Interpretable concepts should preserve relevant information, remain few and non-overlapping, and have immediate human-understandable meaning.
  • SENN learns concepts end-to-end by combining autoencoder training, sparsity for diversity, and prototypes formed from maximally activating training examples.The encoder is trained jointly with the rest of the model, while prototypes provide concrete interpretations of learned concepts.
  • The prototype-based grounding strategy is used in the experiments, while synthetic inputs and learned weights are left for future work.
  • A SENN combines a concept encoder, an input-dependent parametrizer, and an aggregation function to produce predictions with interpretable concepts and relevances.Robustness regularization encourages local linear behavior on the learned concept representation h(x).

5 Experiments

The experiments evaluate SENN on explicitness, faithfulness, and stability across image, tabular, and criminal-justice classification settings. SENN retains competitive prediction accuracy while producing more robust and concept-based explanations than post-hoc alternatives.

  • Evaluation criteria: The evaluation asks whether explanations are immediate and understandable, faithful to true importance, and stable across similar examples.Explicitness is assessed qualitatively, while faithfulness and stability receive quantitative evaluation.
  • Datasets and methods: Experiments cover MNIST, UCI benchmark datasets, and COMPAS, with additional qualitative CIFAR10 results in the supplement.
  • Datasets and methods: SENN is compared with LIME, SHAP, occlusion sensitivity, gradient×input, saliency maps, Integrated Gradients, and E-LRP.
  • Explicitness/Intelligibility: SENN explanations use a small number of higher-level concepts represented by prototypes, yielding more parsimonious explanations than pixel-based methods.On MNIST, shared diagonal-stroke concepts receive positive influence, while a stylized-2 concept can negatively influence one prediction.
  • Faithfulness and stability: A prediction-probability change of ≤10^-4 can accompany considerable variation in post-hoc explanations after minimal Gaussian-noise perturbations.
  • Faithfulness and stability: SENN’s gradient regularization creates a stability–accuracy tradeoff, while SENN consistently and substantially outperforms other methods on robustness across datasets.The paper also reports that the gradient penalty often boosts performance, possibly because of its additional regularization.
  • Faithfulness and stability: Adversarial examples show that minimal perturbations can drastically alter many explanations, particularly those from LIME and SHAP, even when the explained model is robust.

6 Related Work

The paper contrasts post-hoc interpretability methods with self-explaining architectures that use intrinsic relevance scores and general interpretable concepts. It also positions the approach as extending prototype-based explanations beyond similarity to prototypes.

  • Post-hoc neural-network methods leave existing architectures unchanged and reverse-engineer input importance or sensitivities after prediction.The paper distinguishes gradient- and perturbation-based approaches from methods that query or approximate local behavior.
  • The proposed models explain predictions through general concepts and use their relevance scores intrinsically, avoiding additional explanatory computation.This differs from methods whose explanations are computed separately from the predictive model.
  • Token-selection models reveal which text elements matter but not how the downstream complex network uses them.The paper therefore characterizes such representations as transparent about what is used, but not how it is used.
  • The approach generalizes prototype-based interpretability from similarities to prototypes toward more general interpretable concepts.The distinction concerns both the explanatory units and how higher-level input representations are used.

7 Discussion and future work

The discussion argues that interpretability and performance need not remain in conflict when interpretability principles guide architecture and training. It identifies robust explanations as a key outcome and points to broader-domain evaluation as future work.

  • The framework challenges the apparent conflict between interpretability and performance by designing complex architectures around explicitness, faithfulness, and stability.These desiderata are enforced during training rather than added only after a model is learned.
  • The resulting rich models produce robust explanations, a property the paper reports as missing from various popular interpretability frameworks.The discussion presents robustness of explanations as central to the framework’s contribution.
  • Future work includes evaluating interpretable models on larger image datasets, speech recognition, and natural language processing tasks.The paper also identifies extensions involving alternative interpretable basis concepts.

A.1 Data Processing

The data-processing procedures apply dataset-specific normalization, splitting, and filtering choices for MNIST/CIFAR10, UCI, and COMPAS.

  • MNIST and CIFAR10 use original datasets with standard mean-and-variance normalization and 10% of training data for validation.
  • UCI datasets use standard mean-and-variance scaling with 80%, 10%, and 10% train, validation, and test splits.
  • COMPAS rescales Number_of_priors to [0, 1] and filters examples whose labels differ from a strong 80% majority.

A.2 Architectures

The SENN architectures vary by task and distinguish concept encoders from more complex parametrizers. Training uses Adam with a fixed initial learning rate and, when learning h(·), a sparsity parameter.

  • A.2 Architectures: The task-specific SENN architectures use convolutional or fully connected layers, with c denoting the number of concepts.The architecture summary uses CL for convolutional layers and FC for fully connected layers.
  • A.2 Architectures: Each architecture uses a more complex parametrizer than concept encoder.
  • A.2 Architectures: The listed prediction functions combine concept representations through summation.The architecture table represents h(·), θ(·), and g(·), with g(·) shown as sum operations.
  • A.2 Architectures: Training uses Adam with initial learning rate l = 2 × 10^-4 and sparsity strength ξ = 2 × 10^-5 whenever h(·) is learned.

A.3 Predictive Performance of SENN

SENN achieves competitive predictive performance across MNIST, UCI, COMPAS, and CIFAR10 while supporting comparisons of explanation robustness and adversarial perturbations. Its performance remains strong under regularization, with COMPAS and MNIST results reported explicitly.

  • MNIST: 99.11% test accuracy is achieved by the unregularized SENN model on MNIST, while λ = 1 yields 98.7%.The model used in Figure 2 achieves 99.03% accuracy, slightly below the unregularized model.
  • COMPAS: 82.02% test accuracy is achieved by SENN on COMPAS, compared with 78.54% for a baseline logistic classification model.Both methods have relatively low performance because of inconsistent examples.
  • UCI: SENN models achieve competitive performance across UCI datasets for most parameter configurations.
  • CIFAR10: 78.56% test accuracy is achieved by SENN on CIFAR10, described as on par with models of comparable size.
  • Stability and robustness: Gradient regularization creates a tradeoff between explanation stability and prediction accuracy, while sometimes improving performance through additional model regularization.This pattern is reported on COMPAS, BREAST-CANCER, and MNIST.
Loading 1806.07538v2…