Source-linked AI summary
Summit: Scaling Deep Learning Interpretability by Visualizing Activation and Attribution Summarizations
Fred Hohman, Haekyu Park, Caleb Robinson, Duen Horng Chau
TL;DR
Existing image-interpretability methods often focus on single images or neurons, making it difficult to understand class-level representations in large models. SUMMIT combines scalable activation and neuron-influence aggregation into interactive attribution graphs. Exploration scenarios reveal surprising learned representations and inform future architecture design.
Problem
Single-image or single-neuron explanations can miss how large neural networks represent entire classes and produce predictions.
Method
SUMMIT aggregates important activations and influential neuron relationships into attribution graphs, using feature visualizations and dataset examples for interactive exploration.
Results
SUMMIT reveals surprising learned representations in a prevalent large-scale image classifier through neural network exploration scenarios.
Takeaways & Limitations
Class-level summarization provides a higher-level explanation of what features models learn and how those features interact to make predictions.
Takeaways & Limitations
The authors note that single important channels may reflect INCEPTIONV1’s use of multiple softmax heads, requiring comparison with other networks.
Abstract
from arXiv · showhide
Deep learning is increasingly used in decision-making tasks. However, understanding how neural networks produce final predictions remains a fundamental challenge. Existing work on interpreting neural network predictions for images often focuses on explaining predictions for single images or neurons. As predictions are often computed from millions of weights that are optimized over millions of images, such explanations can easily miss a bigger picture. We present Summit, an interactive system that scalably and systematically summarizes and visualizes what features a deep learning model has learned and how those features interact to make predictions. Summit introduces two new scalable summarization techniques: (1) activation aggregation discovers important neurons, and (2) neuron-influence aggregation identifies relationships among such neurons. Summit combines these techniques to create the novel attribution graph that reveals and summarizes crucial neuron associations and substructures that contribute to a model's outcomes. Summit scales to large data, such as the ImageNet dataset with 1.2M images, and leverages neural network feature visualization and dataset examples to help users distill large, complex neural network models into compact, interactive visualizations. We present neural network exploration scenarios where Summit helps us discover multiple surprising insights into a prevalent, large-scale image classifier's learned representations and informs future neural network architecture design. The Summit visualization runs in modern web browsers and is open-sourced.
1 INTRODUCTION
SUMMIT addresses the difficulty of understanding large neural networks by summarizing learned class representations and the feature relationships contributing to predictions. Its interactive attribution graphs support exploration of surprising learned associations and architecture design.
- Motivation: Existing image-interpretability work often explains single images or neurons, which can miss how large models represent entire classes.Such models use millions of weights optimized over millions of images.
- Contributions: Activation aggregation discovers important neurons, while neuron-influence aggregation identifies relationships among those neurons.Together, these techniques summarize crucial neuron associations and substructures contributing to model outcomes.
- Scalability: SUMMIT scales its summarization techniques to large datasets such as ImageNet ILSVRC 2012 with 1.2M images.It also uses feature visualization and dataset examples to distill complex models into compact, interactive visualizations.
- Findings: Exploration scenarios reveal surprising learned representations in a prevalent large-scale image classifier and inform future neural network architecture design.The visualization runs in modern web browsers and is open-sourced.
- Approach: SUMMIT combines top activations and attributions from thousands of class images into attribution graphs linking lower-level and higher-level features.The illustration uses white-wolf images, with features such as “legs” contributing to “white fur” and the final outcome.
2 BACKGROUND FOR NEURAL NETWORK INTERPRETABILITY
Prior interpretability methods visualize individual activations, concepts, or connections, but large neural networks require scalable explanations that also capture learned relationships. SUMMIT frames this as a coordinated challenge of scalability, influence, visualization, interaction, and access.
- Existing methods: Neurons detect features inside neural networks, and activation-based methods use those detections to interpret model decisions.Feature visualization generates synthetic images that maximize individual neurons, helping users inspect learned features and hierarchical representations.
- Limitations: Existing visualizations may treat activations independently, rely on randomized sampling, or become computationally expensive.SUMMIT instead combines activations with relationships between network layers, leverages entire datasets, and integrates scalable techniques.
- Higher-level interpretation: SUMMIT targets the gap between understanding important neurons and understanding how highly contributing neurons combine inside a network.CNNVis derives neuron connections for diagnosis and refinement but did not scale to large datasets.
- Scalability: The design challenges include scaling explanations to entire image classes and datasets rather than inspecting top activations for only a handful of images.Different images may contain different objects, complicating identification of class-representative concepts.
- Design challenges: Additional challenges are discovering influential connections, synthesizing holistic visualizations, supporting interactive exploration of many classes, and reducing the research-access barrier.These challenges span influence, visualization, interaction, and access to large-scale neural-network understanding.
4 DESIGN GOALS
SUMMIT’s design goals are to summarize learned features by identifying strongly activated channels across images and to support class-level interpretation through aggregated activation information.
- G1. Aggregating activations: The resulting activation summaries provide feature information that can be visualized for an entire image class.This goal addresses the challenge of scaling interpretation beyond individual images.
- G1. Aggregating activations: SUMMIT identifies the strongest channels for each image and records top activated channels across images in aggregate.The aggregate indicates which channels most commonly produce strong activations for a class.
G2. Aggregating influences by counting previous top influential
The influence-aggregation goal is to identify which channels in earlier layers most affect later channels for a given class of images. SUMMIT does this by aggregating influential paths across images.
- G2. Aggregating influences: SUMMIT aggregates intermediate convolutional outputs across images to identify the most influential paths through a network.These paths indicate which earlier channels have the greatest impact on future channels for a class.
G3. Finding what neural networks look for, and how they inter-
Summit aims to visualize entire class attribution graphs so users can inspect learned features and how they relate within a large neural network classifier.
- G3. Finding what neural networks look for, and how they inter-: Aggregated influences provide graph edges, while aggregated activations provide vertex values for representing the network.This graph construction supports extracting important subgraphs from the entire neural network.
- G3. Finding what neural networks look for, and how they inter-: Summit visualizes entire attribution graphs for classes in large neural network classifiers.The interface is designed for inspecting any class and understanding learned features and their relationships.
- G3. Finding what neural networks look for, and how they inter-: The system pairs feature visualization with dataset examples to make channel representations more interpretable.These techniques help users inspect what individual channels detect within the attribution graph.
G5. Deployment using cross-platform, lightweight web technolo-
Summit uses modern web browsers to make attribution-graph visualization accessible without specialized computational resources and open-sources its code for reproducible research.
- G5. Deployment using cross-platform, lightweight web technolo-: Summit uses modern web browsers to visualize attribution graphs without specialized computational resources.The deployment approach targets accessible visualization across platforms.
- G5. Deployment using cross-platform, lightweight web technolo-: Summit open-sources its code to support reproducible research.
5 MODEL CHOICE AND BACKGROUND
The paper demonstrates Summit on InceptionV1, a large-scale CNN evaluated on ImageNet, while noting that the techniques can apply to other architectures and domains.
- 5 MODEL CHOICE AND BACKGROUND: InceptionV1 has nine mixed layers treated as the network’s primary layers for interpretability analysis.These include mixed3a,b, mixed4a,b,c,d,e, and mixed5a,b.
- 5 MODEL CHOICE AND BACKGROUND: Although the study uses InceptionV1, the proposed summarization and visualization techniques can apply to other neural network architectures and domains.
6 CREATING ATTRIBUTION GRAPHS BY AGGREGATION
Summit constructs attribution graphs by aggregating class-level channel activations and inter-layer channel influences, then using graph algorithms to extract important subgraphs.
- 6.1 Aggregating Neural Network Activations: Activation aggregation summarizes which channels are important for representing each class across images.For each image and layer, Summit computes per-channel maxima, filters images by class, and aggregates the selected channels into A_l.
- 6.1 Aggregating Neural Network Activations: Maximum channel activation is used as an indicator of concept strength because mean aggregation may dampen relevant channels.The aggregation process requires only a forward pass through the network.
- 6.1 Aggregating Neural Network Activations: 3% of channels are recorded using an adaptive cumulative-weight threshold to form the aggregated activation matrix.This method selects channels until their cumulative probability weight exceeds k_M2, rather than fixing the same count for every image.
- 6.2 Aggregating Inter-layer Influences: Influence aggregation quantifies how a channel in layer l−1 affects a channel in layer l and stores these relationships in I_l.For convolutional layers, each prior-layer channel is convolved with a corresponding kernel slice, and the resulting map is reduced to a scalar influence value.
- 6.3 Combining Aggregated Activations and Influences to Generate Attribution Graphs: Attribution graphs combine A_l and I_l to represent detected features and their relationships.Aggregated influences act as an edge list, with edge weights counting images whose top influential paths connect two channels; graph algorithms then extract class-specific subgraphs.
7 THE SUMMIT USER INTERFACE
SUMMIT combines linked interactive views to explore class representations, similarities, and attribution graphs across a large image classifier. Its interface supports both broad class-level browsing and detailed inspection of important channels, connections, and filtered graph structure.
- Interface overview: SUMMIT presents an interactive interface for scalable summarization and interpretation of entire learned classes in large-scale image classifiers.The system combines aggregation methods with interactive graph visualizations to distill complex models.
- Embedding View: The Embedding View uses UMAP to position each model class according to similarity in its layer-specific feature matrix.Users can zoom, pan, inspect labels, and select a class to update the other views.
- Embedding View: A selectable network minimap lets users animate the Embedding View across layers to compare how class representations change through the network.This supports high-level model debugging and inspection of class representations at different depths.
- Class Sidebar: The Class Sidebar lists classes with performance statistics and sorts them by cosine similarity to the selected class.Users can hover to connect sidebar entries with embedding points, select classes, search directly, and apply additional sorting criteria.
- Attribution Graph View: The Attribution Graph View lays out important class channels by layer, replacing vertices with feature visualizations and connecting channels by convolutional influence.The graph is vertically ordered from later mixed5b layers at the top to earlier mixed3a layers at the bottom; feature visualizations scale with class activation magnitude.
- Attribution Graph View: Attribution graphs expose hierarchical feature construction and variation in class complexity, such as few large strawberry detectors versus many smaller drum detectors.Hover interactions highlight incoming and outgoing connections, while importance filtering reduces large graphs to more manageable views.
- Scalability and access: SUMMIT generated the required representations for all 1000 ImageNet classes using a workstation with 8 GPUs and 504GB of RAM.The system is web-based and runs in modern browsers.
8 NEURAL NETWORK EXPLORATION SCENARIOS
SUMMIT reveals how learned features combine within and across classes, exposing unexpected semantic cues, shifting class associations, discriminative features, and non-semantic channels. These findings also motivate possible changes to classification design while highlighting assumptions requiring validation beyond INCEPTIONV1.
- 8.1 Unexpected Semantics Within a Class: Tench classification unexpectedly relies on people detectors combined with brown fish-body and scale detectors because many training images show people holding fish.The attribution graph shows fish and scale detectors only in middle layers, focused on the body rather than eyes, faces, or fins.
- 8.1 Unexpected Semantics Within a Class: Lionfish representations contain few fish-part detectors but use stripe and quill textures with a final-layer orange-fish-in-water feature.The attribution graph indicates that the orange-fish feature is influenced by stripe and quill detectors.
- 8.2 Mixed Class Association Throughout Layers: Horsecart shifts from mechanical neighbors in early layers to animal neighbors in middle layers, then returns to mechanical associations at the output.Its attribution graph includes people, spoke wheels, horse hips, horse bodies with saddles, and mechanical gear.
- 8.3 Discriminable Features in Similar Classes: Black bear and brown bear share general bear features early, then diverge through fur-color and face-color detectors toward the network output.This discrimination aligns with the color distinction humans might use when classifying the two bears.
- 8.4 Finding Non-semantic Channels: SUMMIT identified five non-semantic channels that activate on irrelevant features across inputs or classes, including mixed3a channel 67 responding to image frames.The authors excluded these channels from subsequent activation and influence aggregation, while noting that SUMMIT lacks automated semantic-quality measurement.
- 8.5 Informing Future Algorithm Design: SUMMIT observations motivate testing classifiers that make decisions at intermediate layers using the most relevant channels, though the finding may depend on INCEPTIONV1’s multiple softmax heads.The proposed design applies MaxPooling to selected channels followed by a Dense-layer classifier.
9 DISCUSSION AND FUTURE WORK
Summit’s discussion identifies limits in attribution-graph comparison, model coverage, attribution generation, hyperparameter choices, and long-term impact evaluation, while outlining concrete future directions.
- Interactive visual comparison of attribution graphs: Automatic comparison of multiple attribution graphs remains future work, including graph differences, unions, and intersections.Summit currently visualizes single attribution graphs interactively.
- Mining attribution graphs for subgraph motifs: Mining attribution graphs for recurring subgraph motifs could reveal how neural networks arrange hierarchical concepts.Examples include shared triangular channel structures among mammal classes or common paths among car classes.
- Visualizing other neural network models: Applying Summit to other neural architectures requires adaptation for models such as ResNets with skip connections.Simpler models like VGG can be adapted easily, while GAN components can be explored when they provide activation information.
- Better attribution graph generation: Attribution graphs depend on an active research area without consensus on the best attribution method.Summit initializes personalized PageRank using activation aggregation and aggregated influences, while alternative relevance-based seeds could be explored.
- Hyperparameter selections: Summit includes hyperparameters for channel recording and PageRank thresholds, although aggregation and visualization support arbitrary dataset sizes.The authors report few differences as image counts increase in their tests.
- Longitudinal evaluation of impacts in practice: Long-term studies are still planned to evaluate Summit’s practical effects on model design, data collection, and deployed-model analysis.The authors presented Summit to ML researchers and scientists and discussed testing it on their models.
10 CONCLUSION
SUMMIT addresses the challenge of understanding neural networks’ learned representations by interactively summarizing and visualizing features and their interactions. The system runs in modern web browsers, is open-sourced, and is presented as a step toward higher-level neural-network explanations.
- SUMMIT scalably and systematically summarizes and visualizes learned features and how they interact to make predictions.
- The system supports interactive exploration of entire class representations in large-scale image-classifier models.
- SUMMIT runs in modern web browsers and is open-sourced for further development.