Source-linked AI summary
Visual Interpretability for Deep Learning: a Survey
Quanshi Zhang, Song-Chun Zhu
TL;DR
CNNs deliver strong performance but their black-box representations remain difficult to interpret. This survey synthesizes methods for visualizing, diagnosing, disentangling, and learning interpretable CNN representations, including semantic-level middle-to-end learning. It concludes that these representations support knowledge transfer, weakly supervised learning, and learning with very few annotations, while interpretability evaluation remains underdeveloped.
Problem
CNNs achieve superior visual-task performance, but end-to-end learning makes their internal representations difficult to understand and evaluate reliably.
Method
The paper surveys studies on understanding CNN representations and learning interpretable or disentangled representations across visualization, diagnosis, model construction, and semantic-level learning.
Results
The reviewed approaches include explanatory graphs and semantic-level middle-to-end learning, with reported benefits for knowledge transfer, weakly supervised learning, and learning from very few annotations.
Takeaways & Limitations
Interpretable representations provide a basis for human-guided learning and for reducing annotation requirements in supported weakly supervised settings.
Takeaways & Limitations
Only very few studies have evaluated network interpretability, and proposing a promising evaluation metric remains a major challenge.
Abstract
from arXiv · showhide
This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks, the interpretability is always the Achilles' heel of deep neural networks. At present, deep neural networks obtain high discrimination power at the cost of low interpretability of their black-box representations. We believe that high model interpretability may help people to break several bottlenecks of deep learning, e.g., learning from very few annotations, learning via human-computer communications at the semantic level, and semantically debugging network representations. We focus on convolutional neural networks (CNNs), and we revisit the visualization of CNN representations, methods of diagnosing representations of pre-trained CNNs, approaches for disentangling pre-trained CNN representations, learning of CNNs with disentangled representations, and middle-to-end learning based on model interpretability. Finally, we discuss prospective trends in explainable artificial intelligence.
1 Introduction
The survey examines CNN interpretability as a response to black-box representations, reviewing visualization, diagnosis, disentanglement, explainable models, and semantic-level learning approaches.
- CNN end-to-end learning achieves strong visual-task performance but leaves the logic of intermediate predictions difficult to understand.
- The survey covers six directions: visualization, diagnosis, disentanglement, explainable models, and semantic-level middle-to-end learning, alongside related interpretability research.The listed directions include visualization of intermediate layers, diagnosis of feature spaces, disentanglement of filter patterns, building explainable models, and human-computer interaction for learning.
- Interpretable or disentangled representations may support weakly supervised middle-to-end learning with fewer human annotations and semantic-level human-computer interaction.
- Visualization is presented as the most direct way to explore representations and as a foundation for diagnosing CNNs.
- Disentangling complex convolutional representations and learning explainable representations remain challenging for state-of-the-art algorithms.
2 Visualization of CNN representations
CNN visualization methods expose patterns encoded by intermediate units through optimization, inversion, or receptive-field estimation, while differing in how directly they reflect actual representations.
- Filter visualization directly explores visual patterns hidden inside CNN units.
- Gradient-based methods estimate input appearances that maximize a chosen CNN unit’s score by computing gradients with respect to the input image.
- Up-convolutional networks invert CNN feature maps into images to illustrate the appearance corresponding to a feature map.
- Unlike gradient-based methods, up-convolutional visualizations cannot mathematically ensure that their results exactly reflect the CNN’s actual representations.
- Accurate receptive-field estimation identifies the image-resolution field of neural activations and helps explain filter representations.The actual receptive field may be smaller than the theoretical field computed from filter size.
3 Diagnosis of CNN representations
The survey groups CNN diagnosis into analyses of feature spaces, output-contributing regions, vulnerabilities, representation blind spots, and biased representations.
- Global analyses examine filter semantics, intermediate-layer transferability, and feature distributions across object categories or attributes.
- Region-based methods identify image areas contributing directly to a label or attribute by propagating feature-map gradients back to the image plane.
- Vulnerability analyses estimate the minimum noisy input perturbation capable of changing a CNN’s final prediction.
- Feature-space refinement can discover knowledge blind spots by grouping samples into thousands of pseudo-categories and examining the learned subspaces.
- Biased-representation analysis mines convolutional inference patterns and compares encoded attribute relationships with annotated ground-truth relationships.Unrelated co-appearing features can be used as contexts for predicting an attribute, so unchanged scores after mouth manipulation reveal a lipstick-prediction bias.
4 Disentangling CNN representations into explanatory graphs & decision trees
The survey describes explanatory graphs that disentangle mixed filter patterns into semantic part nodes and relationships, then decision trees that explain individual CNN predictions using those representations.
- 4.1 Disentangling CNN representations into explanatory graphs: Pre-trained CNN filters often mix multiple object-part patterns, motivating explanatory graphs that clarify hidden semantic organization.
- 4.1 Disentangling CNN representations into explanatory graphs: An explanatory graph disentangles each filter’s mixture of part patterns and represents each part with a node across layers corresponding to CNN convolutional layers.
- 4.1 Disentangling CNN representations into explanatory graphs: Graph nodes consistently represent object parts across images, support localization, and show robustness to shape deformation and pose variation.
- 4.1 Disentangling CNN representations into explanatory graphs: Edges encode co-activation and spatial relationships, while the graph compresses feature-map information into transferable part patterns shared across many images.
- Application: multi-shot part localization: With three annotations per object part, retrieving related graph nodes reduced localization errors by about 1/3 versus the second-best baseline.
- 4.2 Disentangling CNN representations into decision trees: Decision trees organize fully connected-layer decision modes coarsely to finely and quantify which disentangled object parts contribute to each CNN prediction.
5 Learning neural networks with interpretable/disentangled representations
The survey reviews methods for learning interpretable middle-layer representations, including CNNs with part-oriented filters, compositional detection models, capsule networks, and InfoGAN. These approaches associate internal representations with object parts or semantic factors, sometimes without direct part annotations.
- 5 Learning neural networks with interpretable/disentangled representations: Learning disentangled middle-layer representations is more challenging than interpreting pre-trained networks, and only a few studies had been published in this direction.
- 5.1 Interpretable convolutional neural networks: Interpretable CNNs add filter-level losses that regularize feature maps toward specific object-part representations during end-to-end learning.
- 5.1 Interpretable convolutional neural networks: Interpretable CNN training requires no object-part or texture annotations because the loss automatically assigns an object part to each filter.
- 5.1 Interpretable convolutional neural networks: Each filter in an interpretable CNN consistently represents the same object part across different images.
- 5.2 Qualitatively interpretable object detection: AOG-based detection replaces RCNN RoI pooling with parsing to unfold latent object-part configurations and produce bounding boxes with parse-tree rationales.
- 5.3 Capsule networks: Capsule networks use activity-vector length for activation strength and orientation for instantiation parameters; on MNIST, dimensions encoded semantic features such as scale, thickness, and translation.
- 5.4 Information maximizing generative adversarial nets: InfoGAN maximizes mutual information between latent-code dimensions and images to learn semantic concepts without supervision, encoding factors such as digit type, rotation, lighting, and hairstyle.
6 Evaluation metrics for network interpretability
The survey highlights that interpretability metrics remain difficult to define and evaluate, and introduces filter interpretability and location instability as two CNN-focused measures.
- Interpretability metrics help define network interpretability and guide the development of interpretable representations.
- Filter interpretability: Filter interpretability measures how well a filter’s activation receptive fields fit pixel-level semantic annotations.The method evaluates concepts including objects, parts, scenes, textures, materials, and colors using intersection-over-union compatibility.
- Filter interpretability: Top activations are selected using a filter-specific threshold, then their low-resolution regions are scaled to image resolution for receptive-field evaluation.The threshold is set so p(xij > Tf) = 0.005.
- Location instability: Location instability evaluates whether a filter consistently represents the same object part across different objects.It uses the highest-activation feature-map location and compares its image position with object landmarks.
- Location instability: Location instability is computed from deviations in normalized landmark distances, using top-M activations and averaging across filters and landmarks.The resulting average measures the CNN’s location instability.
7 Network interpretability for middle-to-end learning
Interpretable representations support middle-to-end learning by combining semantic structures with limited human supervision. The reviewed approaches use active question-answering or interactive pattern selection to transfer CNN knowledge to object-part models.
- Interpretable or disentangled representations can enable semantic-level middle-to-end learning without strong supervision.
- Active question-answering: Active question-answering semanticizes convolutional-layer patterns and builds a hierarchical object-understanding model from a pre-trained CNN.
- Active question-answering: The And-Or graph represents semantic parts, part templates, latent patterns, and feature-map neural units in four layers.AND nodes encode compositional regions, while OR nodes encode alternative templates or patterns.
- Active question-answering: Active questioning selects unexplained objects whose neural patterns do not fit the current And-Or graph and estimates the value of asking about them.
- Interactive part learning: The reviewed method is reported as having better performance than fast-RCNN methods.
- Interactive part learning: Interactive diagnosis transfers CNN patterns to object parts using very few annotated examples, including scenarios with three object images.People inspect mined latent patterns and remove patterns unrelated to the target part.
8 Prospective trends and conclusions
The survey concludes that interpretable representations are an emerging route for opening CNN black boxes and supporting knowledge transfer and weakly supervised learning. It identifies end-to-end interpretable networks, semantic middle-to-end learning, and semantic-level debugging as prospective directions.
- Disentangling convolutional representations into graphical models or symbolic logic has emerged as a route toward opening neural-network black boxes.
- Transforming pre-trained CNNs into explanatory graphs has shown efficiency in knowledge transfer and weakly supervised learning.
- End-to-end interpretable networks aim for intermediate layers that encode comprehensible patterns, with high-level filters representing specific object parts.
- Semantic-level middle-to-end learning uses interpretable CNN patterns to speed learning and can incorporate human interaction with very few annotations.
- Semantic hierarchies in interpretable networks may support debugging CNN representations at the semantic level and create new visual applications.