PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Tilman Räuker, Anson Ho, Stephen Casper, Dylan Hadfield-Menell

arXiv:2207.13243v6cs.LGcs.AIcs.CLcs.CV

TL;DR

Interpretability research needs more rigorous, practically useful tools for understanding and engineering deep neural networks. The paper surveys inner-interpretability methods and argues that future work should emphasize scalable diagnostics, adversarial evaluation, and benchmarking.

  • Problem

    Interpretability research produces few tools and insights useful in real-world evaluation or engineering, while hypotheses are too often treated as conclusions.

  • Method

    The survey examines inner-interpretability approaches including dataset-based analysis of neuron responses and methods for characterizing, modifying, or pruning neural components.

  • Results

    Some interpretability methods perform well on only a fraction of examples, and Shapley values have limited ability to provide useful causal explanations.

  • Takeaways & Limitations

    Future interpretability methods should scale to large models and undergo rigorous testing on randomly or adversarially sampled tasks rather than only favorable cases.

  • Takeaways & Limitations

    Interpretability methods may be poorly equipped for complex subprocesses, and techniques demonstrated on small networks may not remain tractable for large models.

Abstract

from arXiv · show

The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, "inner" interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications.

I. INTRODUCTION

As DNNs grow more capable and enter real-world deployment, test performance alone may miss harmful biases, adversarial vulnerabilities, and other inadequacies. The survey presents interpretability as a broader toolbox for understanding, evaluating, and addressing such problems.

  • Training compute for machine learning systems grew by ten orders of magnitude from 2010 to 2022 as DNN scale and capabilities increased.
  • Strong test-set performance does not establish that a learned solution is adequate because test sets may not capture the full deployment distribution.
  • Black-box evaluation can miss adversarial inputs, harmful biases, and other problems unrelated to test performance, while making fixes difficult.
  • The survey defines interpretability as characterizing AI computations in human-understandable terms and focuses on internal structures and representations.
  • Interpretability can support open-ended evaluation, failure analysis, bug fixing, accountability, and basic understanding of models and algorithms.
  • Understanding how systems accomplish tasks may provide domain knowledge and enable reverse engineering of more understandable or verifiable solutions, including for superhuman systems.

B. Scope

The survey focuses on inner interpretability for DNNs rather than black-box, input-attribution, adversarial, neurosymbolic, or other out-of-scope approaches. It emphasizes internal methods because they support goals such as manual modification, reverse engineering, and detecting latent knowledge.

  • B. Scope: Inner interpretability methods explain internal DNN structures and representations, while several other interpretability approaches remain outside this survey’s scope.
  • B. Scope: The survey focuses on inner methods because they are well-equipped for guiding manual modifications, reverse engineering solutions, and detecting inner latent knowledge.
  • B. Scope: The paper surveys existing work extensively, unlike cited critique papers and surveys focused mainly on non-DNNs, feature attribution, or language models.
  • B. Scope: Weight-focused inner interpretability includes continual-learning methods that specialize weights and masking methods that identify task-relevant weights.

C. Taxonomy

The taxonomy organizes inner interpretability methods by the network component they explain and by whether they operate during or after training. It also highlights conceptual connections among method families and hazards such as unimportant weights.

  • C. Taxonomy: The taxonomy classifies methods by whether they explain weights, neurons, subnetworks, or latent representations.
  • C. Taxonomy: Methods are also divided into intrinsic approaches used during training and post hoc approaches used after training.
  • C. Taxonomy: The taxonomy sometimes separates conceptually similar methods, such as continual learning for weights and neurons, while noting their connections.
  • C. Taxonomy: Continual-learning methods specialize weights for particular input types, offering a way to characterize weights by the tasks or classes they serve.
  • C. Taxonomy: Weight masks can identify subsets of weights and corresponding subnetworks that specialize in particular subtasks.
  • C. Taxonomy: Many network weights can be pruned to a very small fraction of their original number with little or no performance loss, sometimes after fine-tuning.

A. Continual Learning (Intrinsic):

Inner interpretability methods examine neurons through task specialization, dataset responses, synthesized inputs, perturbations, and gradients. These approaches support both descriptive characterization and tests of how neural components affect behavior.

  • A. Continual Learning (Intrinsic):: Neuron-based continual learning typically adds new neurons for new tasks, discouraging unrelated feature sharing and supporting task-specific interpretations.
  • E. Gradient-Based Attribution (Post Hoc):: Neuron-focused interpretability also includes gradient-based attribution methods that analyze partial derivatives of outputs with respect to neural activations.
  • B. Dataset-Based (Post Hoc):: Dataset-based methods characterize neurons by finding inputs that maximally excite them or by aligning activations with labeled semantic concepts.
  • C. Feature Synthesis (Post Hoc):: Feature-synthesis methods optimize or generate inputs that maximally or minimally activate neurons, without being limited to a particular dataset.
  • D. Neural Perturbation and Ablation (Post Hoc):: Perturbation and ablation methods test counterfactuals by examining how changing a neuron alters network behavior, helping distinguish causal from correlational relationships.
  • D. Neural Perturbation and Ablation (Post Hoc):: Identifying neurons that can be removed or modified has been shown to reduce bias or improve robustness, although Shapley values have limited causal explanatory power.

E. Gradient-Based Attribution (Post Hoc):

Gradient-based attribution methods have been widely used to study influential features, including features attributed to internal neurons. Related neuron-level analyses face interpretability challenges from polysemantic and frivolous neurons, while sparsity can simplify subnetwork analysis without consistently clarifying individual neurons.

  • E. Gradient-Based Attribution (Post Hoc):: Gradient-based feature attribution studies which features influence neural responses or model outputs, with related approaches applied to internal neurons.
  • E. Gradient-Based Attribution (Post Hoc):: Polysemantic neurons respond to multiple unrelated features, making interpretations more likely to be incorrect or incomplete and enabling adversarial exploitation.
  • E. Gradient-Based Attribution (Post Hoc):: Frivolous neurons are unimportant and may be prunable or redundant, making their contributions difficult to detect with some interpretability methods.
  • E. Gradient-Based Attribution (Post Hoc):: After compression, remaining neurons were more interpretable with only marginal performance change, and interpretability proxies have been used to guide neuron pruning.
  • E. Gradient-Based Attribution (Post Hoc):: Sparsification can reduce weights by almost two orders of magnitude with little to no performance tradeoff, but does not necessarily improve individual-neuron interpretability.

B. Modularity (Intrinsic):

Modularity methods seek to make subnetworks easier to understand by organizing neurons into specialized modules or analyzing task-specific circuits. Intrinsic methods impose modular structure during training, while post hoc methods partition neurons after training, with limited success in producing highly composite modules.

  • B. Modularity (Intrinsic):: Modular DNNs support analysis of parts separately, using hard modular architectures or softer designs with sparse information flow between competing modules.
  • B. Modularity (Intrinsic):: Figure 4 groups subnetwork methods into sparsifying computational graphs, enforcing or discovering neuron modules, and analyzing circuits that perform specific tasks.
  • B. Modularity (Intrinsic):: Post hoc methods partition neurons using weight graphs or neuron associations with inputs and outputs, sometimes estimating cluster interpretability without human evaluation.
  • B. Modularity (Intrinsic):: Post hoc modular partitioning has had very limited success in finding highly-composite partitions, motivating combinations with intrinsic modularity methods.
  • B. Modularity (Intrinsic):: Circuit analysis studies small subnetworks, but many successes involve toy models and intensive human effort; practical applications will likely require automation.

V. INTERNAL REPRESENTATIONS

Methods for interpreting internal representations include self-explanations, adversarial training, disentanglement, transformer analyses, concept vectors, probing, and cross-network comparisons. These approaches offer multiple ways to study or expose internal reasoning, but explanation faithfulness and robustness remain open challenges.

  • A. Self-Explaining Models (Intrinsic):: Self-explaining systems aim to expose internal reasoning through model-generated explanations, analogous to asking a person to explain their reasoning.
  • A. Self-Explaining Models (Intrinsic):: Prototype-based classifiers explain outputs by attributing them to exemplary datapoints that resemble the input.
  • A. Self-Explaining Models (Intrinsic):: Human-supervised explanations have been trained from the same internal representations for vision and language tasks.
  • A. Self-Explaining Models (Intrinsic):: Explanations are evaluated by explicitness, faithfulness, and stability, but reported explanations can be unfaithful or vulnerable to adversarial examples.
  • V. INTERNAL REPRESENTATIONS: Figure 5 organizes representation methods around self-explanation, adversarial training, disentanglement, transformer analyses, concept vectors, probing, and representation comparison.
  • V. INTERNAL REPRESENTATIONS: Adversarially trained classifiers have shown improvements in several interpretability-related properties and transfer-oriented representations, although robustness may conflict with accuracy.

C. Disentanglement (Intrinsic):

Disentanglement seeks latent representations in which neurons align more closely with interpretable concepts. It can be pursued through supervised bottlenecks or transformations, and through unsupervised mechanisms such as competition, while leakage remains a challenge.

  • C. Disentanglement (Intrinsic):: Disentanglement encourages a more bijective relationship between latent-space neurons and interpretable concepts.
  • C. Disentanglement (Intrinsic):: Supervised approaches use whitening and orthogonal transformations, bottleneck layers, or sparse interpretable embeddings to separate features.
  • C. Disentanglement (Intrinsic):: Bottleneck-based models face leakage, in which undesired information still passes through the bottleneck.
  • C. Disentanglement (Intrinsic):: Unsupervised disentanglement methods include dropout, lateral inhibition, capsule architectures, and activation alignment.
  • C. Disentanglement (Intrinsic):: Transformer representations can be studied through token representations, feed-forward processing, and key-query products that encode token-to-token attention.

E. Concept Vectors (Post Hoc):

Concept-vector methods are post hoc approaches that associate meaningful concepts with directions in a model’s latent space. Probing tests whether an embedding contains information about a target quality by training a model on embedded examples.

  • Concept-vector methods associate directions in latent space with meaningful concepts using post hoc analyses of activations.
  • Probing tests whether an embedding captures target information by transferring it to a supervised prediction task.
  • Probing consists of collecting varied examples, embedding them, and training a model on those embeddings to predict the quality of interest.

G. Representation Comparison (Post Hoc):

Representation-comparison methods estimate similarity between the internal representations of different DNNs, but nonlinear and complex representations make this difficult to quantify. A benchmark found that canonical correlation analysis and centered kernel alignment each fail in one relevant respect.

  • Representation comparison estimates similarity between internal representations of different DNNs.The approach is challenging because nonlinear networks may encode concepts in complex ways that do not align reliably with neurons or activation directions.
  • A benchmark evaluating similarity measures found that canonical correlation analysis and centered kernel alignment each fail in one respect.

VI. DISCUSSION

The discussion connects interpretability with robustness, continual learning, modularity, compression, and visual-system research, while emphasizing scalable, validated, non-cherry-picked evaluation. It argues that interpretability should prioritize safety and practical usefulness without sacrificing competitiveness or increasing risky capabilities.

  • Connections: Interpretability and adversarial robustness are linked in both directions, and interpretability tools can also be used to design adversaries.The survey reports that more interpretable networks can be more robust, more robust networks can be more interpretable, and tools can help construct adversarial examples.
  • Connections: Continual-learning methods using parameter isolation or regularization can make weights and neurons intrinsically interpretable through partial task memberships.
  • Scalability: Interpretability techniques should scale beyond small networks and simple tasks because scalability strongly affects their practical usefulness.The survey urges researchers to report computational requirements and test methods on large models.
  • Evaluation: Interpretability methods generate hypotheses rather than conclusions, so interpretations require validity tests, uncertainty estimates, and calibrated confidence.Suggested checks include sanity tests, counterexamples, multiple trials, random baselines, and comparisons with simple methods.
  • Evaluation: Cherry-picked best cases can overestimate method value, especially because some techniques work well on only a fraction of examples.
  • Evaluation: Attributing features to combinations of neurons is more general than attributing them to single neurons, which matters for complex subprocesses.
  • Evaluation: Interpretability methods should be evaluated on randomly or adversarially sampled tasks rather than only on circuits selected for being easy to interpret.
  • Safety and usefulness: Interpretability should preserve competitive performance and avoid increasing bias, compute demands, or risky capabilities that could outpace oversight.

VII. FUTURE WORK

The survey calls for interpretability research to become more rigorous, scalable, interdisciplinary, and useful for engineering practice. It prioritizes discovering novel failures, benchmarking methods, combining techniques, and pursuing mechanistic understanding while addressing deception and latent knowledge.

  • Scaling interpretability to large systems requires making human oversight more efficient through methods such as active learning, weak supervision, and proxy models.Current analyses can require many hours of expert effort even for simple tasks; the survey proposes using humans more for screening than generating interpretations.
  • Interpretability methods should discover novel behaviors and failures beyond fixed datasets or failure modes specified in advance.Dataset-based methods are limited to the distributions they characterize, while abstract subnetwork understanding may help reveal unexpected failures.
  • Mechanistic interpretability and microscope AI aim to explain DNN computations algorithmically and extract domain insights from high-performing systems.Early progress has relied on small models, simple tasks, and meticulous human effort, with program synthesis and analysis suggested for automating hypothesis generation and validation.
  • Rigorous benchmarks should measure whether interpretability methods produce useful insights relevant to engineers.The survey notes that widely accepted benchmarks do not yet exist because methods pursue different goals and the field is developing rapidly.
  • Combining intrinsic and post hoc techniques may improve results, but their interactions remain relatively unexplored.The survey reports useful synergies in some studies while noting that no work has thoroughly examined interactions between different methods.
  • Interpretability research should focus on tools useful to engineers for diagnostics, debugging, adversaries, and benchmarking.The survey describes the current field as producing few tools and insights useful in real-world engineering applications.
Loading 2207.13243v6…