Source-linked AI summary
Interpreting CNN Knowledge via an Explanatory Graph
Quanshi Zhang, Ruiming Cao, Feng Shi, Ying Nian Wu, Song-Chun Zhu
TL;DR
CNN filters can mix multiple object-part patterns, making the knowledge encoded by a pre-trained CNN difficult to interpret. The paper learns an unsupervised explanatory graph whose nodes disentangle these patterns and whose edges encode their relationships. The graph yields stable, transferable part representations and outperforms supervised part-representation methods in part localization, while remaining only a rough approximation of CNN knowledge.
Problem
The paper addresses how to reveal and organize visual knowledge in pre-trained CNNs when individual filters may encode mixtures of object parts.
Method
The method learns an unsupervised explanatory graph with part-pattern nodes and edges encoding co-activation and spatial relationships across CNN convolutional layers.
Results
The graph nodes consistently represent the same object parts across images, and transferred patterns outperform supervised part representations in part localization.
Takeaways & Limitations
The explanatory graph provides a concise, transparent representation that supports transferring CNN part patterns to object-part tasks.
Takeaways & Limitations
The explanatory graph remains a rough representation rather than an accurate reconstruction of CNN knowledge, and its spatial model assumes Gaussian relationships between connected patterns.
Abstract
from arXiv · showhide
This paper learns a graphical model, namely an explanatory graph, which reveals the knowledge hierarchy hidden inside a pre-trained CNN. Considering that each filter in a conv-layer of a pre-trained CNN usually represents a mixture of object parts, we propose a simple yet efficient method to automatically disentangles different part patterns from each filter, and construct an explanatory graph. In the explanatory graph, each node represents a part pattern, and each edge encodes co-activation relationships and spatial relationships between patterns. More importantly, we learn the explanatory graph for a pre-trained CNN in an unsupervised manner, i.e., without a need of annotating object parts. Experiments show that each graph node consistently represents the same object part through different images. We transfer part patterns in the explanatory graph to the task of part localization, and our method significantly outperforms other approaches.
Introduction
CNNs achieve strong object recognition performance, but their end-to-end learning makes the network a black box. This work seeks a global view of how visual knowledge is organized and how many pattern types each convolutional filter memorizes.
- CNNs have achieved superior performance in object classification and detection, while their end-to-end learning strategy makes them difficult to interpret globally.
- The study treats convolutional layers as encoding implicit patterns such as object parts and textures.
- A central question is how many types of patterns each convolutional filter memorizes.
3 What is the spatial relationship between two patterns?
The paper organizes CNN part patterns into an explanatory graph whose nodes disentangle mixed filter activations and whose structure represents the hierarchy and relationships among patterns. The graph is learned without part annotations and supports interpretable transfer to part localization.
- Representing knowledge hierarchy: The explanatory graph uses multiple layers corresponding to CNN convolutional layers, with nodes summarizing knowledge in their feature maps.A graph node represents a part pattern rather than a fixed neural-unit location.
- Pattern relationships: Graph nodes encode co-activation and spatial relationships, allowing patterns such as horse faces and ears to occur at varying feature-map positions while preserving their relationships.
- Disentangling object parts: The method disentangles mixtures of object-part patterns within each filter’s feature map and learns them without ground-truth part annotations.It also filters out noisy activations and aims for consistent part meaning across images.
- Transferable graph nodes: Given a testing image, the graph indicates whether a part node is triggered and where that part appears on the feature map.
- Transferable graph nodes: The graph is presented as a concise representation that clarifies chaotic CNN knowledge and enables transfer of part patterns to other tasks.
- Experiments: Experiments evaluate visualization, semantic interpretability across images, and transferability through multi-shot part localization across several CNN architectures.
Related work
Prior work visualized or retrieved CNN features, but often treated filters as encoding single concepts or selectively displayed one pattern from a mixture. The explanatory graph instead separates part patterns without annotations and represents them more interpretably for transfer.
- Visualization & interpretability of CNN filters: CNN interpretability research has used statistical analyses and visualization methods to explore semantics hidden in learned features.
- Visualization & interpretability of CNN filters: Gradient-based visualization may selectively show one local minimum when a high-layer filter represents multiple patterns, while other methods analyze only the highest 0.5% activations.
- Pattern retrieval: Pattern-retrieval methods have selected feature-map units for scenes, objects, or parts, with some approaches learning part representations in a supervised or weakly supervised manner.
- Pattern retrieval: Many existing methods assume that each filter mainly encodes one visual concept and overlook mixtures of patterns in high convolutional layers.
- Explanatory graph: The explanatory graph disentangles patterns from different parts without part annotations and makes them more interpretable than raw feature maps.
- Knowledge transfer: The paper motivates the graph as a transparent representation for transferring part knowledge to other applications.
Algorithm
The algorithm discovers noisy activation peaks that correspond to part patterns, then organizes them layer by layer using stable spatial relationships and co-activation with upper-layer patterns. It models each filter’s activations as a mixture of graph nodes, learns node positions and neighbors top-down, and infers pattern presence from feature-map responses.
- Pattern discovery: Activation peaks are treated as candidate part patterns when corresponding patterns in other filters activate at similar map positions, while remaining peaks represent background noise.The method connects lower-layer patterns to neighboring upper-layer patterns that preserve spatial relationships across images.
- Pattern hierarchy: High-layer patterns filter noise and disentangle lower-layer patterns, while lower-layer patterns serve as components of larger high-layer parts.
- Top-down learning: The explanatory graph is constructed top-down: upper-layer inference results guide the disentangling of patterns in neighboring lower layers.Learning proceeds layer by layer over the CNN’s convolutional layers.
- Mixture model: Each filter’s activation distribution is modeled as a mixture whose hidden components are graph nodes, with a dummy Vnone component accounting for unexplained noisy activations.Node compatibility depends on spatial relationships with neighboring upper-layer nodes.
- Parameter estimation: For each node, EM updates its prior position, while a greedy strategy selects upper-layer neighbors that maximize the learning objective.The learned parameters include node positions and neighboring-pattern sets.
- Inference: At inference time, each node is assigned the highest-scoring unit in its filter’s feature map to determine whether the corresponding part appears and where it is located.The inferred position is the position of the selected unit.
Experiments
Experiments evaluate explanatory graphs across CNN architectures and animal-part datasets through visualization, semantic interpretability, and part-localization transfer. The graph nodes provide more meaningful part representations than raw filters, and the AOG outperforms all baselines in multi-shot part localization.
- Experimental settings: The method was applied to VGG-16, 50-layer and 152-layer Residual Networks, and a VAE-GAN encoder.Graphs were built from CNNs trained or fine-tuned on animal categories from three benchmark datasets.
- Experimental settings: Three experiments assessed graph visualization, semantic interpretability of part patterns, and multi-shot learning for part localization.The evaluation used thirteen baselines across these experiments.
- Pattern visualization: Heat maps and synthesized images visualize inferred patterns, their spatial distributions, and the foreground appearance encoded by graph nodes.Heat maps show top-50% patterns with the highest inference scores, while synthesis filters out responses not assigned to valid patterns.
- Semantic interpretability: Human evaluations measured whether each pattern consistently represented the same object region across images using top-K inference results selected by inference energy.The selected results were required to account for about 30% of inference energy, and raters judged part-semantic purity.
- Semantic interpretability: Graph nodes encoded much more meaningful part representations than raw filter maps and filter peaks.Figure 8 compares image regions associated with explanatory-graph nodes against regions from feature-map baselines.
- Part transfer: The AOG outperformed all baselines, including methods that learned part features with supervision.The AOG linked implicit explanatory-graph patterns to explicit part names for multi-shot semantic-part localization.
Conclusion and discussions
The paper concludes that explanatory graphs provide a concise, transparent representation of CNN knowledge that supports stable part patterns and transfer to object-part localization. However, the graph remains a rough representation rather than an information-preserving reconstruction of CNN knowledge.
- Conclusion: The proposed explanatory graph reveals knowledge hierarchy inside pretrained CNN convolutional layers.It filters noisy activations, disentangles reliable part patterns from individual filters, and encodes co-activation and spatial relationships.
- Conclusion: Experiments showed that the learned patterns had significantly higher stability than baselines.The conclusion presents stability as an empirical property of the learned patterns.
- Transferability: The graph’s transparent representation supports transferring CNN patterns to object parts, with part-localization experiments demonstrating good transferability.The method even outperformed supervised learning of part representations.
- Limitation: The explanatory graph is a rough representation of the CNN rather than an accurate reconstruction of its knowledge.This defines the principal scope boundary of the representation.