PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks

Tilman Räuker, Anson Ho, Stephen Casper, Dylan Hadfield-Menell

arXiv:2207.13243v6cs.LGcs.AIcs.CLcs.CV

TL;DR

Interpretability research needs more rigorous, practically useful tools for understanding and engineering deep neural networks. The paper surveys inner-interpretability methods and argues that future work should emphasize scalable diagnostics, adversarial evaluation, and benchmarking.

  • Problem

    Interpretability research produces few tools and insights useful in real-world evaluation or engineering, while hypotheses are too often treated as conclusions.

    Source: §VII · p. 12

  • Method

    The survey examines inner-interpretability approaches including dataset-based analysis of neuron responses and methods for characterizing, modifying, or pruning neural components.

    Source: B. Dataset-Based (Post Hoc) · p. 4, D. Neural Perturbation and Ablation… · p. 5, G. Frivolous Neurons (Hazard) · p. 5

  • Results

    Some interpretability methods perform well on only a fraction of examples, and Shapley values have limited ability to provide useful causal explanations.

    Source: D. Neural Perturbation and Ablation… · p. 5, §VI · p. 10

  • Takeaways & Limitations

    Future interpretability methods should scale to large models and undergo rigorous testing on randomly or adversarially sampled tasks rather than only favorable cases.

    Source: §VI · pp. 9–10

    Interpretability methods may be poorly equipped for complex subprocesses, and techniques demonstrated on small networks may not remain tractable for large models.

    Source: §VI · pp. 9–10

Abstract

from arXiv · show

The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, they are difficult to analyze, raising concerns about using them without a rigorous understanding of how they function. Effective tools for interpreting them will be important for building more trustworthy AI by helping to identify problems, fix bugs, and improve basic understanding. In particular, "inner" interpretability techniques, which focus on explaining the internal components of DNNs, are well-suited for developing a mechanistic understanding, guiding manual modifications, and reverse engineering solutions. Much recent work has focused on DNN interpretability, and rapid progress has thus far made a thorough systematization of methods difficult. In this survey, we review over 300 works with a focus on inner interpretability tools. We introduce a taxonomy that classifies methods by what part of the network they help to explain (weights, neurons, subnetworks, or latent representations) and whether they are implemented during (intrinsic) or after (post hoc) training. To our knowledge, we are also the first to survey a number of connections between interpretability research and work in adversarial robustness, continual learning, modularity, network compression, and studying the human visual system. We discuss key challenges and argue that the status quo in interpretability research is largely unproductive. Finally, we highlight the importance of future work that emphasizes diagnostics, debugging, adversaries, and benchmarking in order to make interpretability tools more useful to engineers in practical applications.

I. INTRODUCTION

As DNNs grow more capable and enter real-world deployment, test performance alone may miss harmful biases, adversarial vulnerabilities, and other inadequacies. The survey presents interpretability as a broader toolbox for understanding, evaluating, and addressing such problems.

  • Training compute for machine learning systems grew by ten orders of magnitude from 2010 to 2022 as DNN scale and capabilities increased.Source: §I · p. 1
  • Strong test-set performance does not establish that a learned solution is adequate because test sets may not capture the full deployment distribution.Source: §I · p. 1
  • Black-box evaluation can miss adversarial inputs, harmful biases, and other problems unrelated to test performance, while making fixes difficult.Source: MIT CSAIL · p. 1
  • The survey defines interpretability as characterizing AI computations in human-understandable terms and focuses on internal structures and representations.Source: MIT CSAIL · p. 1
  • Interpretability can support open-ended evaluation, failure analysis, bug fixing, accountability, and basic understanding of models and algorithms.Source: A. The Importance of Interpretable AI · pp. 1–2
  • Understanding how systems accomplish tasks may provide domain knowledge and enable reverse engineering of more understandable or verifiable solutions, including for superhuman systems.Source: A. The Importance of Interpretable AI · p. 2

B. Scope

The survey focuses on inner interpretability for DNNs rather than black-box, input-attribution, adversarial, neurosymbolic, or other out-of-scope approaches. It emphasizes internal methods because they support goals such as manual modification, reverse engineering, and detecting latent knowledge.

B. Scope

  • Inner interpretability methods explain internal DNN structures and representations, while several other interpretability approaches remain outside this survey’s scope.Source: B. Scope · p. 2
  • The survey focuses on inner methods because they are well-equipped for guiding manual modifications, reverse engineering solutions, and detecting inner latent knowledge.Source: B. Scope · p. 2
  • The paper surveys existing work extensively, unlike cited critique papers and surveys focused mainly on non-DNNs, feature attribution, or language models.Source: B. Scope · p. 2
  • Weight-focused inner interpretability includes continual-learning methods that specialize weights and masking methods that identify task-relevant weights.Source: Figure 2 · p. 3

C. Taxonomy

The taxonomy organizes inner interpretability methods by the network component they explain and by whether they operate during or after training. It also highlights conceptual connections among method families and hazards such as unimportant weights.

C. Taxonomy

  • The taxonomy classifies methods by whether they explain weights, neurons, subnetworks, or latent representations.Source: C. Taxonomy · p. 3
  • Methods are also divided into intrinsic approaches used during training and post hoc approaches used after training.Source: C. Taxonomy · p. 3
  • The taxonomy sometimes separates conceptually similar methods, such as continual learning for weights and neurons, while noting their connections.Source: C. Taxonomy · p. 3
  • Continual-learning methods specialize weights for particular input types, offering a way to characterize weights by the tasks or classes they serve.Source: A. Continual Learning (Intrinsic) · p. 3
  • Weight masks can identify subsets of weights and corresponding subnetworks that specialize in particular subtasks.Source: B. Weight-Masking (Post Hoc) · p. 3
  • Many network weights can be pruned to a very small fraction of their original number with little or no performance loss, sometimes after fine-tuning.Source: C. Frivolous Weights (Hazard) · p. 3

A. Continual Learning (Intrinsic):

Inner interpretability methods examine neurons through task specialization, dataset responses, synthesized inputs, perturbations, and gradients. These approaches support both descriptive characterization and tests of how neural components affect behavior.

A. Continual Learning (Intrinsic):

  • Neuron-based continual learning typically adds new neurons for new tasks, discouraging unrelated feature sharing and supporting task-specific interpretations.Source: A. Continual Learning (Intrinsic) · p. 3

E. Gradient-Based Attribution (Post Hoc):

  • Neuron-focused interpretability also includes gradient-based attribution methods that analyze partial derivatives of outputs with respect to neural activations.Source: Figure 3 · p. 4

B. Dataset-Based (Post Hoc):

  • Dataset-based methods characterize neurons by finding inputs that maximally excite them or by aligning activations with labeled semantic concepts.Source: B. Dataset-Based (Post Hoc) · p. 4

C. Feature Synthesis (Post Hoc):

D. Neural Perturbation and Ablation (Post Hoc):

E. Gradient-Based Attribution (Post Hoc):

Gradient-based attribution methods have been widely used to study influential features, including features attributed to internal neurons. Related neuron-level analyses face interpretability challenges from polysemantic and frivolous neurons, while sparsity can simplify subnetwork analysis without consistently clarifying individual neurons.

E. Gradient-Based Attribution (Post Hoc):

  • Gradient-based feature attribution studies which features influence neural responses or model outputs, with related approaches applied to internal neurons.Source: E. Gradient-Based Attribution (Post Hoc) · p. 5
  • Polysemantic neurons respond to multiple unrelated features, making interpretations more likely to be incorrect or incomplete and enabling adversarial exploitation.Source: F. Polysemantic Neurons (Hazard) · p. 5
  • Frivolous neurons are unimportant and may be prunable or redundant, making their contributions difficult to detect with some interpretability methods.Source: G. Frivolous Neurons (Hazard) · p. 5
  • After compression, remaining neurons were more interpretable with only marginal performance change, and interpretability proxies have been used to guide neuron pruning.Source: G. Frivolous Neurons (Hazard) · p. 5
  • Sparsification can reduce weights by almost two orders of magnitude with little to no performance tradeoff, but does not necessarily improve individual-neuron interpretability.Source: A. Sparsity (Intrinsic) · p. 5

B. Modularity (Intrinsic):

Modularity methods seek to make subnetworks easier to understand by organizing neurons into specialized modules or analyzing task-specific circuits. Intrinsic methods impose modular structure during training, while post hoc methods partition neurons after training, with limited success in producing highly composite modules.

B. Modularity (Intrinsic):

  • Modular DNNs support analysis of parts separately, using hard modular architectures or softer designs with sparse information flow between competing modules.Source: B. Modularity (Intrinsic) · pp. 5–6
  • Figure 4 groups subnetwork methods into sparsifying computational graphs, enforcing or discovering neuron modules, and analyzing circuits that perform specific tasks.Source: Figure 4 · p. 6
  • Post hoc methods partition neurons using weight graphs or neuron associations with inputs and outputs, sometimes estimating cluster interpretability without human evaluation.Source: C. Modular Partitionings (Post Hoc) · p. 6
  • Post hoc modular partitioning has had very limited success in finding highly-composite partitions, motivating combinations with intrinsic modularity methods.Source: C. Modular Partitionings (Post Hoc) · p. 6
  • Circuit analysis studies small subnetworks, but many successes involve toy models and intensive human effort; practical applications will likely require automation.Source: D. Circuits Analysis (Post Hoc) · p. 6

V. INTERNAL REPRESENTATIONS

Methods for interpreting internal representations include self-explanations, adversarial training, disentanglement, transformer analyses, concept vectors, probing, and cross-network comparisons. These approaches offer multiple ways to study or expose internal reasoning, but explanation faithfulness and robustness remain open challenges.

A. Self-Explaining Models (Intrinsic):

V. INTERNAL REPRESENTATIONS

  • Figure 5 organizes representation methods around self-explanation, adversarial training, disentanglement, transformer analyses, concept vectors, probing, and representation comparison.Source: Figure 5 · p. 7
  • Adversarially trained classifiers have shown improvements in several interpretability-related properties and transfer-oriented representations, although robustness may conflict with accuracy.Source: B. Adversarial Training (Intrinsic) · p. 7

C. Disentanglement (Intrinsic):

Disentanglement seeks latent representations in which neurons align more closely with interpretable concepts. It can be pursued through supervised bottlenecks or transformations, and through unsupervised mechanisms such as competition, while leakage remains a challenge.

C. Disentanglement (Intrinsic):

E. Concept Vectors (Post Hoc):

Concept-vector methods are post hoc approaches that associate meaningful concepts with directions in a model’s latent space. Probing tests whether an embedding contains information about a target quality by training a model on embedded examples.

  • Concept-vector methods associate directions in latent space with meaningful concepts using post hoc analyses of activations.Source: E. Concept Vectors (Post Hoc) · p. 8
  • Probing tests whether an embedding captures target information by transferring it to a supervised prediction task.Source: F. Probing (Post Hoc) · p. 8
  • Probing consists of collecting varied examples, embedding them, and training a model on those embeddings to predict the quality of interest.Source: F. Probing (Post Hoc) · p. 8

G. Representation Comparison (Post Hoc):

Representation-comparison methods estimate similarity between the internal representations of different DNNs, but nonlinear and complex representations make this difficult to quantify. A benchmark found that canonical correlation analysis and centered kernel alignment each fail in one relevant respect.

  • Representation comparison estimates similarity between internal representations of different DNNs.The approach is challenging because nonlinear networks may encode concepts in complex ways that do not align reliably with neurons or activation directions.Source: G. Representation Comparison (Post Hoc) · p. 8
  • A benchmark evaluating similarity measures found that canonical correlation analysis and centered kernel alignment each fail in one respect.Source: G. Representation Comparison (Post Hoc) · p. 9

VI. DISCUSSION

The discussion connects interpretability with robustness, continual learning, modularity, compression, and visual-system research, while emphasizing scalable, validated, non-cherry-picked evaluation. It argues that interpretability should prioritize safety and practical usefulness without sacrificing competitiveness or increasing risky capabilities.

Connections

  • Interpretability and adversarial robustness are linked in both directions, and interpretability tools can also be used to design adversaries.The survey reports that more interpretable networks can be more robust, more robust networks can be more interpretable, and tools can help construct adversarial examples.Source: §VI · p. 9
  • Continual-learning methods using parameter isolation or regularization can make weights and neurons intrinsically interpretable through partial task memberships.Source: §VI · p. 9

Scalability

  • Interpretability techniques should scale beyond small networks and simple tasks because scalability strongly affects their practical usefulness.The survey urges researchers to report computational requirements and test methods on large models.Source: §VI · p. 9

Evaluation

  • Interpretability methods generate hypotheses rather than conclusions, so interpretations require validity tests, uncertainty estimates, and calibrated confidence.Suggested checks include sanity tests, counterexamples, multiple trials, random baselines, and comparisons with simple methods.Source: §VI · p. 9
  • Cherry-picked best cases can overestimate method value, especially because some techniques work well on only a fraction of examples.Source: §VI · pp. 9–10
  • Attributing features to combinations of neurons is more general than attributing them to single neurons, which matters for complex subprocesses.Source: §VI · p. 10
  • Interpretability methods should be evaluated on randomly or adversarially sampled tasks rather than only on circuits selected for being easy to interpret.Source: §VI · p. 10

Safety and usefulness

  • Interpretability should preserve competitive performance and avoid increasing bias, compute demands, or risky capabilities that could outpace oversight.Source: §VI · p. 10

VII. FUTURE WORK

The survey calls for interpretability research to become more rigorous, scalable, interdisciplinary, and useful for engineering practice. It prioritizes discovering novel failures, benchmarking methods, combining techniques, and pursuing mechanistic understanding while addressing deception and latent knowledge.

  • Scaling interpretability to large systems requires making human oversight more efficient through methods such as active learning, weak supervision, and proxy models.Current analyses can require many hours of expert effort even for simple tasks; the survey proposes using humans more for screening than generating interpretations.Source: §VII · p. 10
  • Interpretability methods should discover novel behaviors and failures beyond fixed datasets or failure modes specified in advance.Dataset-based methods are limited to the distributions they characterize, while abstract subnetwork understanding may help reveal unexpected failures.Source: §VII · p. 10
  • Mechanistic interpretability and microscope AI aim to explain DNN computations algorithmically and extract domain insights from high-performing systems.Early progress has relied on small models, simple tasks, and meticulous human effort, with program synthesis and analysis suggested for automating hypothesis generation and validation.Source: §VII · p. 11
  • Rigorous benchmarks should measure whether interpretability methods produce useful insights relevant to engineers.The survey notes that widely accepted benchmarks do not yet exist because methods pursue different goals and the field is developing rapidly.Source: §VII · p. 11
  • Combining intrinsic and post hoc techniques may improve results, but their interactions remain relatively unexplored.The survey reports useful synergies in some studies while noting that no work has thoroughly examined interactions between different methods.Source: §VII · pp. 11–12
  • Interpretability research should focus on tools useful to engineers for diagnostics, debugging, adversaries, and benchmarking.The survey describes the current field as producing few tools and insights useful in real-world engineering applications.Source: §VII · p. 12

Related papers

Loading 2207.13243v6…