Source-linked AI summary
Towards Better Analysis of Deep Convolutional Neural Networks
Mengchen Liu, Jiaxin Shi, Zhen Li, Chongxuan Li, Jun Zhu, Shixia Liu
TL;DR
Deep CNNs are powerful but difficult to understand and develop because their functions, component roles, and interactions are unclear, making model construction dependent on trial-and-error. The paper introduces CNNVis, a DAG-based hybrid visual analytics system using rectangle packing, matrix reordering, and biclustering-based edge bundling. Case studies found that experts could better explore CNNs, diagnose issues, and refine architectures, while the system remains focused on offline snapshot analysis.
Problem
Deep CNNs are difficult to understand because their component roles and interactions are unclear, so developing high-quality models often relies on substantial trial-and-error.
Method
CNNVis represents a deep CNN as a directed acyclic graph and combines rectangle packing, matrix reordering, and biclustering-based edge bundling for visual analysis.
Results
Three case studies showed that experts could explore and understand CNNs, diagnose model issues, and refine CNN architectures to improve performance.
Takeaways & Limitations
CNNVis supports comprehensive analysis of neuron roles, neuron connections, and CNN architecture during model construction.
Takeaways & Limitations
CNNVis analyzes snapshots of CNNs during training and does not yet continuously update with the online training process.
Abstract
from arXiv · showhide
Deep convolutional neural networks (CNNs) have achieved breakthrough performance in many pattern recognition tasks such as image classification. However, the development of high-quality deep models typically relies on a substantial amount of trial-and-error, as there is still no clear understanding of when and why a deep model works. In this paper, we present a visual analytics approach for better understanding, diagnosing, and refining deep CNNs. We formulate a deep CNN as a directed acyclic graph. Based on this formulation, a hybrid visualization is developed to disclose the multiple facets of each neuron and the interactions between them. In particular, we introduce a hierarchical rectangle packing algorithm and a matrix reordering algorithm to show the derived features of a neuron cluster. We also propose a biclustering-based edge bundling method to reduce visual clutter caused by a large number of connections between neurons. We evaluated our method on a set of CNNs and the results are generally favorable.
1 INTRODUCTION
CNNVis is an interactive visual analytics system for understanding, diagnosing, and refining deep CNNs, which are difficult to analyze because of their scale and unclear component roles. It represents CNNs as DAGs and combines multiple visual techniques to expose neuron features and interactions.
- Motivation: Deep CNN development often requires substantial trial-and-error because experts lack a clear understanding of when and why models work.Training a single model on a large dataset may take several days or weeks.
- Challenges: Large CNNs are difficult to study because they contain many layers, neurons, and connections, while component roles and interactions remain poorly understood.The paper identifies scale and unclear functional relationships as two technical challenges.
- Approach: CNNVis formulates a deep CNN as a directed acyclic graph whose nodes are neurons and edges are connections.The system clusters layers and neurons and selects representative elements to visualize large networks.
- Approach: Its hybrid visualization combines hierarchical rectangle packing, matrix reordering, and biclustering-based edge bundling.These techniques show cluster features, activation patterns, and neuron connections while reducing visual clutter.
- Evaluation: Three image-classification case studies examine model structure, failed training diagnosis, and CNN refinement.The reported studies indicate that experts can explore neuron roles and connections and refine architectures for faster iteration and convergence.
2 RELATED WORK
Prior neuron-visualization methods reconstruct or optimize images but do not reveal CNN inner workings, especially neuron roles and interactions. This work instead uses a DAG-based hybrid visualization and a weighted biclustering method to analyze large, weighted neural networks.
- Neuron Feature Visualization: Existing neuron-feature methods use code inversion or activation maximization to display reconstructed or synthesized images.Both families rely on natural-image priors or regularization to obtain realistic visualizations.
- Neuron Feature Visualization: Grid-based feature visualizations do not disclose each neuron's role for different image types or interactions between neurons.The paper identifies this as a limitation of prior approaches to understanding CNN mechanisms.
- DAG-Based Visualization: The proposed DAG-based hybrid visualization combines rectangle packing, matrix ordering, and biclustering-based edge bundling to expose neuron facets and interactions.The combined representation targets the inner working mechanism of deep CNNs.
- Weighted Edge Bundling: Unlike BiSet's unweighted edges, CNN edges carry weights, so the paper develops weighted biclustering based on Apriori to preserve important biclusters.Converting CNNs to unweighted graphs may lose information represented by edge weights.
- DAG-Based Visualization: Existing DAG visualizations can suffer serious visual clutter on large neural networks, motivating layer and neuron clustering with representative elements.The method selects representative layers and neurons to make large networks more tractable to visualize.
3 BACKGROUND
CNNs use alternating convolutional and pooling layers, followed by fully connected layers, to transform grid-like data into increasingly abstract representations. Convolution captures shift-invariant patterns, while pooling aggregates features and reduces computational cost.
- Architecture: CNNs typically combine alternating convolutional and pooling layers with one or more fully connected layers.Local connectivity links each neuron to a subset of neurons in the previous layer.
- Convolution: Convolution slides shared weights across an image, producing weighted sums that capture shift-invariant visual patterns.The same window weights are reused throughout scanning.
- Softmax: Softmax converts output-layer values into class probabilities that sum to 1 for probability-based classification losses.The output-layer input x is the result of a linear transformation through the weights.
- Pooling: Pooling aggregates small input regions, provides some translation invariance, and downsamples features to reduce computational cost.Max-pooling outputs the maximum over each region.
- Loss Functions: Hinge loss measures the difference between correct-class and predicted-class scores and is introduced here for two classes.The binary class label is t ∈ {−1,+1}, and o is the network's real-valued class score.
4 CNNVIS
CNNVis was developed with deep-learning experts to address the difficulty of interpreting large CNNs and understanding how their components and neurons work together. It combines a DAG-based overview, neuron-cluster visualization, edge bundling, and interactive analysis to support understanding, diagnosis, and refinement.
- Requirement Analysis: CNNVis was designed collaboratively with six deep-learning experts over twelve months to make CNN mechanisms more understandable.The team held discussions every two weeks, and three paper co-authors participated.
- Requirement Analysis: Existing deep-learning frameworks expose statistics and debugging information but do not reveal neuron roles across image categories or neuron cooperation.This limitation makes failed training cases difficult for experts to diagnose.
- Requirement Analysis: CNNVis requirements include overviews of learned neuron features and explanations of how low-level features aggregate into high-level concepts.Lower layers detect simple features, middle layers object parts, and higher layers concepts.
- System Overview: CNNVis converts CNNs into DAGs and aggregates neurons and layers to provide an overview of large networks.The DAG formulation represents neurons as nodes and connections as edges.
- System Overview: Its modules visualize neuron facets, bundle connections through biclustering, and support interactive clustering changes and on-demand debugging information.The primary goal is to help experts understand, diagnose, and refine CNNs.
5 DAG FORMULATION
The DAG formulation makes large CNNs tractable by aggregating layers and clustering neurons according to class-based activation patterns. CNNVis represents each neuron with an average activation vector and selects representative neurons for cluster visualization.
- Layer Aggregation: CNNVis represents a CNN as a DAG and aggregates adjacent layers into groups to present networks with many layers and neurons.The current implementation divides layers into groups at pooling layers.
- Neuron Clustering: Neurons are clustered by similar roles, using the assumption that similar activations indicate similar roles.This avoids directly clustering activations across potentially millions of training images.
- Activation Aggregation: For each class, CNNVis averages a neuron's activations across that class's training samples before clustering.The training samples are partitioned into m classes, with N_i samples in class c_i.
- Activation Aggregation: The class averages form an m-dimensional real-valued activation vector for each neuron.The vector contains one average activation for each training-set class.
- Neuron Clustering: CNNVis supports K-Means and MeanShift clustering, then selects neurons near each cluster centroid as representatives.MeanShift does not require the cluster count in advance.
6 VISUALIZATION
CNNVis represents a CNN as a DAG and combines cluster-level nodes with rectangle packing, matrix visualization, and biclustering-based edge bundling. These views expose neuron features, activations, cluster patterns, and weighted connections while addressing scale and visual clutter.
- 6.1 Overview: CNNVis represents neuron clusters as nodes and their connections as edges in a DAG-based hybrid visualization.The overview uses large rectangles for neuron clusters and adds an in-between layer for biclusters.
- 6.3 Biclustering-based Edge Bundling: Biclustering-based edge bundling aggregates weighted connections into biclusters to reduce clutter from dense edges and their crossings.The method seeks biclusters involving input and output neuron clusters while retaining distinctions between important positive and negative weights.
- 6.2.1 Learned Features as Rectangle Packing: Rectangle packing places learned neuron features into rectangles so feature size can encode neuron importance.The approach was developed because a uniform grid fails to emphasize important neurons; importance may reflect activation or contribution measures.
- 6.2.1 Learned Features as Rectangle Packing: A hierarchical rectangle packing algorithm divides large packing problems into smaller sub-problems to handle neuron clusters containing hundreds or thousands of neurons.Existing methods become impractical as the number of rectangles increases, motivating the hierarchical strategy.
- 6.2.2 Activations as Matrix Visualization: The activation matrix places neurons in rows and image classes in columns, with cell color representing average activation.Users can switch between feature rectangles and matrix visualization; matrix reordering reveals activation clusters by placing similar neurons adjacently.
- 6.2.2 Activations as Matrix Visualization: Matrix reordering preserves the class-column order and reorders only neuron rows to reveal cluster patterns while supporting comparisons across neuron clusters.The ordering maximizes similarities between adjacent neurons, using cosine similarity; direct optimization has time complexity O(2^NC · N_C^2).
7 APPLICATION
Three case-study applications show CNNVis helping experts inspect learned features, relate architecture to performance, diagnose failed training, and refine models. The reported analyses identify redundancy, underfitting, and training stagnation, while an improved model reaches 9.43% CIFAR-10 error.
- Overview: CNNVis exposes simple lower-layer features and helps experts compare neuron features and activations across CNN variants.Experts identified corners, color patches, stripes, and similar activations among redundant neurons.
- Network Depth: Insufficient depth produced more impure top-layer clusters, indicating difficulty distinguishing similar classes and decreased performance.The comparison involved ShallowCNN and BaseCNN variants.
- Network Width: Wider BaseCNN×4 showed overfitting and redundant lower-layer neurons, while narrower variants showed underfitting and reduced performance.The wider model had much lower training than testing loss; narrower networks had comparable losses but worse performance than BaseCNN.
- Architecture Analysis: CNNVis helped experts locate candidate layers for architectural changes and use cluster purity and feature patterns to assess model quality.Experts reported that quantitative criteria alone provided insufficient intuition and clear guidance for deciding which layers to narrow or remove.
- Training Diagnosis: In a failed training case, mostly negative weights led to mostly zero ReLU outputs and therefore zero stochastic-gradient updates to weights.The experts used edge visibility and weight coloring to identify the issue.
- Training Diagnosis: The improved model achieved an average error of 9.43% on CIFAR-10 after the diagnosed training issue was addressed.Expert E3 reported satisfaction with the result and the system’s training-statistics views.
8 CONCLUSION
The paper presents CNNVis, a hybrid visual analytics system for understanding, diagnosing, and refining CNNs through multiple views of network structure and learned features. The authors demonstrate its usefulness through case studies, while noting limits in online training integration and applicability beyond DAG-formulated models.
- 8 CONCLUSION: CNNVis combines rectangle packing, matrix ordering, and biclustering-based edge bundling to support comprehensive CNN analysis.The system represents CNNs as directed acyclic graphs and provides multiple perspectives on neurons and their interactions.
- 8 CONCLUSION: Three case studies demonstrate CNNVis for understanding, diagnosing, and refining CNNs.The reported application focuses on expert analysis of CNN models.
- Future Work: CNNVis currently analyzes snapshots offline rather than continuously updating training status during online training.Selecting representative snapshots and comparing them effectively remains difficult.
- Future Work: Applying CNNVis to models such as RNNs remains future work because they cannot be formulated as directed acyclic graphs.RNN feedback loops create a data-flow visualization challenge.