Source-linked AI summary
Interpreting CNNs via Decision Trees
Quanshi Zhang, Yu Yang, Haotian Ma, Ying Nian Wu
TL;DR
CNNs are powerful but difficult to interpret because middle-layer features are not readily connected to semantic concepts or prediction rationales. The paper learns disentangled filters and a coarse-to-fine decision tree to explain which object parts support each prediction and by how much, reporting effectiveness across benchmark datasets while noting that explanations are approximate and focused on classification.
Problem
The paper addresses the limited connection between CNN middle-layer features and semantic concepts, as well as the need to quantify how those features produce prediction scores.
Method
The method learns disentangled high-layer filters without object-part or texture annotations, assigns filters semantic part meanings, and mines their shared rationales into a coarse-to-fine decision tree.
Results
Experiments on different benchmark datasets demonstrated the effectiveness of the proposed approach for encoding CNN prediction rationales as semantic and quantitative decision modes.
Takeaways & Limitations
The decision tree provides semantic and quantitative explanations of CNN predictions by identifying contributing object parts or filters and organizing common rationales at multiple abstraction levels.
Takeaways & Limitations
The decision tree provides approximate rather than exact explanations, because unsupervised filter learning can produce incorrect activations and decision modes omit insignificant filters.
Abstract
from arXiv · showhide
This paper aims to quantitatively explain rationales of each prediction that is made by a pre-trained convolutional neural network (CNN). We propose to learn a decision tree, which clarifies the specific reason for each prediction made by the CNN at the semantic level. I.e., the decision tree decomposes feature representations in high conv-layers of the CNN into elementary concepts of object parts. In this way, the decision tree tells people which object parts activate which filters for the prediction and how much they contribute to the prediction score. Such semantic and quantitative explanations for CNN predictions have specific values beyond the traditional pixel-level analysis of CNNs. More specifically, our method mines all potential decision modes of the CNN, where each mode represents a common case of how the CNN uses object parts for prediction. The decision tree organizes all potential decision modes in a coarse-to-fine manner to explain CNN predictions at different fine-grained levels. Experiments have demonstrated the effectiveness of the proposed method.
1. Introduction
The paper addresses the difficulty of connecting CNN middle-layer features to semantic concepts and quantitatively explaining how those features produce predictions. It proposes a decision tree that organizes shared prediction rationales into coarse-to-fine decision modes.
- CNN interpretability remains challenging despite strong task performance and prior efforts to visualize or analyze hidden feature representations.
- The paper targets the gap between middle-layer CNN features and semantic explanations, including associating neural activations with object-part concepts.
- The method learns disentangled high-layer filters that consistently activate on the same object region across images without part or texture annotations.
- For each prediction, it identifies activated object-part filters and quantitatively measures each part’s contribution to the prediction score.The paper illustrates this with contributions such as a head filter contributing 2.32% and a feet filter contributing 0.72%.
- The decision tree mines common prediction rationales as decision modes and organizes them hierarchically from common coarse modes near the root to fine-grained modes near the leaves.Each node represents a mode shared by multiple images, while leaf-level modes correspond to more specific cases.
- The approach aims to bridge CNN representations with semantic visual concepts and provide semantic, quantitative explanations without strong explanatory supervision.The authors report successful application to benchmark CNNs and experimental evidence of effectiveness.
2. Related work
Prior work has explored CNN interpretability through visualization, diagnosis, semanticization, and decision-tree distillation, but existing tree methods do not provide semantic explanations of individual CNN predictions. This paper distinguishes its approach by using a gradient-based tree to represent prediction rationales more directly.
- CNN interpretability: CNN interpretability research includes feature visualization, network diagnosis, semantic representation learning, and decision-tree knowledge distillation.These approaches examine hidden patterns, feature behavior, semantic units, or output spaces from different perspectives.
- CNN visualization: CNN visualization methods estimate activating image patterns or invert feature maps, but their guarantees about correspondence to actual neural representations differ.Gradient-based visualization estimates maximizing inputs, whereas up-convolutional methods invert feature maps without mathematically ensuring that the result reflects actual neural representations.
- Network diagnosis: CNN diagnosis methods analyze filters, feature distributions, gradients, image regions, adversarial samples, or blind spots to obtain insight into representations.The reviewed methods include joint semantic analysis, transferability evaluation, feature-space distributions, gradient propagation, region attribution, network attacks, and weakly supervised blind-spot discovery.
- Decision trees for neural networks: Existing decision-tree methods for neural networks mainly distill knowledge for classification or representation regularization rather than semantic-level explanations.The cited studies use trees or additive models but do not explain the network’s knowledge at a human-interpretable semantic level.
- Novelty: The proposed research differs by semantically explaining each prediction from a pre-trained CNN and summarizing gradient-derived decision modes as rationales.The gradient-based tree is described as reflecting CNN predictions more directly and strictly than distillation-based alternatives.
3. Image-specific rationale of a CNN prediction
The method learns disentangled top-layer filters that represent object parts, then expresses an image-specific CNN prediction rationale through feature signals and contribution weights. It uses a piecewise-linear formulation to identify which parts matter and how much they contribute to a category score.
- 3.1. Learning disentangled filters: The study focuses on classification CNNs and uses top-convolutional-layer features because high-layer filters are more prone to represent object parts.The fully connected layers are analyzed for how they use these part features to produce final predictions.
- 3.1. Learning disentangled filters: The method revises a benchmark CNN so top-layer filters represent specific object parts without additional part annotations.A filter loss encourages each filter to activate on the same part across different images.
- 3.1. Learning disentangled filters: The filter loss uses negative mutual information between feature-map distributions and part-location distributions to promote disentangled filters.It encourages a feature map to match one location candidate, producing a single activation peak when the part appears and no activation otherwise.
- 3.2. Quantitative rationales of CNN predictions: For an input image, x denotes the top-layer feature map and y the pre-softmax category score whose rationale is represented.The feature map has spatial dimensions L × L and D filter channels after ReLU.
- 3.2. Quantitative rationales of CNN predictions: The CNN’s cascaded fully connected and ReLU layers are represented piecewise linearly, with g(h,w,d)x(h,w,d) measuring each activation’s contribution to y.The weights are computed as g = ∂y/∂x, while b accounts for the remaining score component.
- 3.2. Quantitative rationales of CNN predictions: Different images receive different rationale weights because their signals follow different routes through ReLU layers.Because interpretable filters have a single activation peak, the representation can be further reduced to filter-level vectors x and g.
- 3.2. Quantitative rationales of CNN predictions: The vectors x and g represent which filters or parts support prediction and the strength of their rationale signals.The dimensions of x measure object-part signal strength, while g selects the parts used for the prediction.
4. Learning a decision tree
The method learns a hierarchical decision tree whose nodes summarize shared CNN prediction rationales at progressively finer levels. It represents these rationales with selected filters and uses them to estimate filter- and object-part contributions.
- Tree construction: The method learns a decision tree for each category using a CNN with disentangled filters and positive and negative training images.The tree interprets the target category’s pre-softmax classification score.
- Decision modes: Each tree node represents a decision mode shared by a group of positive images, with child nodes dividing it into finer-grained modes.Leaf nodes encode image-specific rationales.
- Mode parameterization: Each decision mode uses a unit rationale vector and a binary filter-selection vector, with sparsity encouraged to produce concise explanations.The rationale parameters are w = α ◦ g, where g captures common rationales and α selects filters.
- Tree construction: It initializes image-specific modes, then greedily merges pairs of root children into nodes representing shared decision modes.Merged nodes combine image sets and learn their filter-selection and rationale parameters.
- Optimization: The learning objective preserves discriminative power while encouraging a small set of generic explanations for CNN predictions.Positive likelihood is computed from the prediction of the best child in the second tree layer.
- Inference and contributions: During inference, a top-down parse tree selects progressively finer modes, and the selected node yields filter- and object-part contribution vectors.Positive and negative filter contributions are distinguished by the sign of the filter contribution.
5. Experiments
Experiments evaluate rationale accuracy, contribution-distribution fitness, and information loss across benchmark CNNs and datasets. Fine-grained modes generally reduce prediction errors, but do not necessarily improve classification accuracy.
- Experimental setup: The method is evaluated with AlexNet, VGG-M, VGG-S, and VGG-16-based disentangled CNNs, excluding residual networks because the filter loss is incompatible with skip connections.Experiments use PASCAL-Part, CUB200-2011, and ILSVRC 2013 DET Animal-Part datasets.
- Evaluation metrics: Rationale accuracy is measured by object-part contribution error and the fitness of estimated versus ground-truth filter contribution distributions.The ground-truth contribution for a part is obtained by comparing the original CNN output with the output after removing that part’s activations.
- Evaluation metrics: Information loss is evaluated through average classification accuracy and average prediction error when the decision tree represents the CNN.The evaluation is extended to nodes at multiple tree layers by reconnecting those nodes directly to the root.
- Explanations: Figure 5 visualizes typical images associated with decision modes at the second tree layer, while Figure 6 estimates object-part contribution distributions from those modes.Figure 6 combines contribution pie charts, activation heat maps, and filter receptive fields used for part assignment.
- Results: Fine-grained decision modes usually yield lower prediction error rates because they more closely match image-specific rationales, but they do not show higher classification accuracy.The method mines common category-level modes and ignores random or negative images, unlike discriminative classifier learning.
6. Conclusion and discussions
The method couples a revised CNN with a decision tree to encode CNN decision modes as quantitative rationales without object-part or texture annotations. These explanations are theoretically approximate because filters may activate incorrectly and decision modes omit insignificant part filters.
- The method revises a CNN and tightly couples it with a decision tree that encodes CNN decision modes as quantitative rationales.It does not require annotations of object parts or textures in training images.
- The approach explains CNN predictions at the semantic level and was tested on different benchmark datasets, where experiments proved its effectiveness.
- The decision tree provides an approximate explanation rather than an accurate reconstruction of CNN representation details.
- Approximation arises because unsupervised filter loss may produce incorrect activations in challenging images, while each decision mode omits insignificant object-part filters for sparsity.