Source-linked AI summary
Interpretable Convolutional Neural Networks
Quanshi Zhang, Ying Nian Wu, Song-Chun Zhu
TL;DR
CNNs achieve strong visual performance but lack clear high-layer knowledge representations, making their decision patterns difficult to inspect. The paper adds filter-level losses during end-to-end training so high-layer filters represent object parts without extra annotations, and reports more semantically meaningful representations than traditional CNNs. The approach is designed to apply across CNN structures, though interpretability may slightly reduce discrimination power.
Problem
CNNs have strong visual performance but their high-layer representations remain difficult to interpret, motivating interpretable knowledge representations without additional human supervision.
Method
The method adds a filter-level loss that encourages each high-layer filter to represent a category-specific object part and match spatial activation templates without part or texture annotations.
Results
Experiments showed that interpretable CNNs encoded more semantically meaningful high-layer knowledge and improved object-part interpretability over traditional CNNs.
Takeaways & Limitations
Clear object-part semantics let people inspect which memorized parts support CNN classification decisions.
Takeaways & Limitations
The exploratory interpretability design may slightly decrease discrimination power, and future work targets discriminative textures and parts shared across categories for greater flexibility.
Abstract
from arXiv · showhide
This paper proposes a method to modify traditional convolutional neural networks (CNNs) into interpretable CNNs, in order to clarify knowledge representations in high conv-layers of CNNs. In an interpretable CNN, each filter in a high conv-layer represents a certain object part. We do not need any annotations of object parts or textures to supervise the learning process. Instead, the interpretable CNN automatically assigns each filter in a high conv-layer with an object part during the learning process. Our method can be applied to different types of CNNs with different structures. The clear knowledge representation in an interpretable CNN can help people understand the logics inside a CNN, i.e., based on which patterns the CNN makes the decision. Experiments showed that filters in an interpretable CNN were more semantically meaningful than those in traditional CNNs.
1. Introduction
CNNs perform strongly on visual tasks, but their high-layer representations are difficult to interpret. The paper modifies CNNs so filters learn clearer object-part semantics without additional part or texture annotations.
- CNN interpretability remains a major challenge despite strong performance in visual tasks.
- The paper targets end-to-end learning of interpretable convolutional representations without additional human supervision.
- Traditional high-layer filters may mix multiple object-part patterns, whereas interpretable filters are intended to activate for a specific part.
- The method revises CNNs across different structures and automatically pushes each filter toward representing an object part.
- The interpretable CNN preserves the original top-layer loss and training samples while adding filter-level interpretability objectives.
- Clear object-part semantics can help people inspect the patterns memorized for classification and better understand network decisions.
- The authors report significantly improved object-part interpretability, while acknowledging that interpretability-oriented design may slightly reduce discrimination power.
2. Related work
Prior work mainly visualizes, retrieves, or diagnoses CNN representations, while newer approaches learn more meaningful representations. The paper is positioned as learning explicit part-level semantics directly during CNN training.
- Statistical methods analyze semantics hidden in CNN features, but do not constitute the paper’s proposed interpretable training approach.
- Network visualization: Visualization methods inspect patterns that maximize neural-unit scores or invert feature maps into images.
- Pattern retrieval: Pattern-retrieval studies select feature-map units to describe scenes, objects, or semantic parts, sometimes using supervision.
- Model diagnosis: Model-diagnosis methods identify image regions or relationships associated with network outputs, often requiring manual inspection of accountable regions.
- Learning a better representation: Representation-learning approaches pursue more meaningful features, but cited methods lack explicit part-level or texture-level semantics or require additional supervision.
3. Algorithm
The algorithm trains high-layer filters to respond to a category-specific object part at one spatial location, using template-based masking and an auxiliary filter loss during end-to-end learning.
- Each target-layer filter is expected to activate for one category-specific object part and remain inactive for other categories.
- For each filter, n^2 templates model possible spatial locations of the target part in its ReLU feature map.Each template is an n × n matrix representing the ideal activation distribution when the part triggers one feature-map unit.
- During forward propagation, the CNN selects the template at the feature-map maximum and masks the feature map by an element-wise product.The resulting masked map filters noisy activations, and different images may select different templates.
- The method supports L-1-based templates for faster computation and can assign each filter a target category based on its strongest mean activation.
- The filter loss fits each feature map to one positive spatial template or a negative template for other categories.The loss is formulated as mutual information between feature maps and the template set, with template likelihoods measuring fitness.
- End-to-end training combines gradients from the final task loss and the local filter loss, weighted by λ.The local loss gradients are computed with respect to each feature-map element and use approximations updated during training.
4. Understanding of the loss
The loss decomposes interpretability into category separation and spatial concentration, encouraging each filter to identify one category and one region of activation.
- The loss includes a conditional entropy term that encourages each filter to distinguish its target category from all other categories.Positive templates are grouped as one label, while the negative template represents other categories.
- The spatial entropy term encourages a target-category feature map to activate in one region rather than repeatedly across different locations.
5. Experiments
Experiments evaluated interpretable CNNs across multiple architectures and animal-part datasets using part interpretability and location stability. The interpretable models generally produced clearer, more stable filter semantics, while classification effects depended on the task.
- Experimental setup: Experiments used three benchmark datasets with landmark or part annotations across single-category and multi-category classification settings.The datasets were ILSVRC 2013 DET Animal-Part, CUB200-2011, and Pascal VOC Part.
- Evaluation metrics: Filters were evaluated using object-part interpretability and location stability, which measure semantic clarity and localization consistency.Part interpretability was based on filter-part association, while location stability used relative location deviations.
- Experimental setup: The method was applied to four CNN structures—AlexNet, VGG-M, VGG-S, and VGG-16—to demonstrate broad applicability.
- Experimental results: Ordinary CNNs performed better in single-category classification, whereas interpretable CNNs performed better in multi-category classification.The authors suggest that clarified filter semantics may reduce later filter-learning difficulty in multi-category training.
6. Conclusion and discussions
The paper concludes that a general loss-based modification can make high-layer CNN representations more interpretable without additional part annotations. It also identifies broader semantic coverage as a direction for future work.
- Conclusion: The proposed method modifies traditional CNNs so high conv-layer filters are pushed toward object-part representations without additional supervision annotations.
- Conclusion: Experiments showed that interpretable CNNs encoded more semantically meaningful knowledge in high conv-layers than traditional CNNs.
- Future work: Future work will add filters for discriminative category textures and object parts shared across multiple categories to increase model flexibility.
Proof of equations
The supplied equations define a mutual-information-based loss construction involving variables T and T′. The excerpt does not provide enough prose to explain the full derivation or every symbol.
- Equation construction: The displayed construction swaps the roles of T and T′ while forming terms involving p(T′) and p(x|T′).
- Objective: The loss is written as Loss = −MI(X; T), using mutual information between X and T as the optimization objective.
Visualization of CNN filters
The figures visualize filters in the top interpretable conv-layer and compare them with filters from an ordinary CNN. Each row represents feature maps for a category-classifying filter.
- Interpretable CNN filters: Figures 7 and 8 visualize filters in the top interpretable conv-layer, with each row corresponding to feature maps of a category-classifying filter.
- Comparison: The comparison is between the row-wise feature-map visualizations of interpretable and ordinary top-layer filters.
- Ordinary CNN filters: Figure 9 provides the corresponding visualization for filters in the top conv-layer of an ordinary CNN.