Source-linked AI summary
Neural Prototype Trees for Interpretable Fine-grained Image Recognition
Meike Nauta, Ron van Bree, Christin Seifert
TL;DR
Deep neural networks offer strong computer-vision performance but are difficult to interpret, while traditional decision trees are transparent yet not competitive for these tasks. ProtoTree combines neural prototype learning with a hierarchical binary tree to provide intrinsic global and local explanations. An ensemble of 5 ProtoTrees achieves competitive performance on CUB-200-2011 and Stanford Cars while using roughly 10% as many prototypes as ProtoPNet.
Problem
Deep neural networks perform strongly in computer vision but lack transparency, whereas decision trees are interpretable yet have noncompetitive computer-vision performance.
Method
ProtoTree combines a CNN, trainable prototypical parts, and a binary decision-tree structure trained end-to-end with prototype-based routing.
Results
An ensemble of 5 interpretable ProtoTrees achieves competitive performance on CUB-200-2011 and Stanford Cars while using roughly 10% as many prototypes as ProtoPNet.
Takeaways & Limitations
ProtoTree organizes interpretable prototypes hierarchically, enabling faithful global explanations and traceable local decision paths.
Takeaways & Limitations
Prototype visualizations can focus on background and reveal learned biases, motivating possible human-in-the-loop correction.
Abstract
from arXiv · showhide
Prototype-based methods use interpretable representations to address the black-box nature of deep learning models, in contrast to post-hoc explanation methods that only approximate such models. We propose the Neural Prototype Tree (ProtoTree), an intrinsically interpretable deep learning method for fine-grained image recognition. ProtoTree combines prototype learning with decision trees, and thus results in a globally interpretable model by design. Additionally, ProtoTree can locally explain a single prediction by outlining a decision path through the tree. Each node in our binary tree contains a trainable prototypical part. The presence or absence of this learned prototype in an image determines the routing through a node. Decision making is therefore similar to human reasoning: Does the bird have a red throat? And an elongated beak? Then it's a hummingbird! We tune the accuracy-interpretability trade-off using ensemble methods, pruning and binarizing. We apply pruning without sacrificing accuracy, resulting in a small tree with only 8 learned prototypes along a path to classify a bird from 200 species. An ensemble of 5 ProtoTrees achieves competitive accuracy on the CUB-200- 2011 and Stanford Cars data sets. Code is available at https://github.com/M-Nauta/ProtoTree
1 University of Twente, the Netherlands 2 University of Duisburg-Essen, Germany
A pruned ProtoTree uses roughly 200 prototypes for CUB while requiring only 8 local decisions on average for each test image.
- Roughly 200 prototypes support CUB classification, with only 8 local decisions on average for one test image.
1. Introduction
ProtoTree addresses the accuracy–interpretability trade-off by combining neural representations, trainable prototypes, and a hierarchical binary decision tree. It provides globally faithful reasoning and locally traceable predictions while reducing prototype and decision complexity.
- The architecture targets the accuracy–interpretability trade-off by combining deep learning’s expressiveness with decision trees’ hierarchical transparency.
- ProtoTree combines a CNN with a built-in binary tree and trainable prototypical parts for fine-grained image recognition.
- Prototype presence determines routing at each node, while root-to-leaf paths encode classification rules and leaves learn class distributions.
- ProtoTree provides faithful global explanations and locally traceable decision paths, enabling model simulation and error analysis.
- An ensemble of 5 interpretable ProtoTrees achieves competitive performance on CUB-200-2011 and Stanford Cars.
- Compared with ProtoPNet, ProtoTree uses roughly 10% as many prototypes while organizing them hierarchically.
2. Related Work
Prior work includes post-hoc visual explanations, prototype-based intrinsic interpretability, and neural soft decision trees. ProtoTree extends these approaches by embedding prototypes in a self-explanatory tree without requiring hierarchical labels or a predefined taxonomy.
- Interpretability with Prototypes: Computer-vision explainability methods include local explanations, internal visualizations, exemplars, and concept-based analyses.
- Interpretability with Prototypes: Prototype-based methods incorporate visual explanations directly into models for intrinsic interpretability, often using prototypical parts.
- Interpretability with Prototypes: ProtoPNet classifies images by comparing image patches with learned class-specific prototypes and aggregating weighted similarity scores.
- Neural Soft Decision Trees: Neural soft decision trees use learned routing functions over latent representations, but prior approaches can lose decision-tree interpretability or require limited model components.
- Neural Soft Decision Trees: ProtoTree extends prior tree approaches by placing prototypes in a tree, yielding global intrinsic explanations with one decision per node.
- Neural Soft Decision Trees: Unlike hierarchical image-classification methods, ProtoTree does not require hierarchical labels or a predefined taxonomy.
3. Neural Prototype Tree
ProtoTree combines a CNN with a soft binary decision tree that routes images according to prototype similarity and aggregates class distributions across leaves. This structure supports hierarchical, prototype-based image recognition with trainable routing and prediction components.
- Architecture: A ProtoTree combines a convolutional neural network with a soft neural binary decision tree for supervised image classification.The CNN produces a latent representation that serves as input to the tree.
- Prototype routing: Each internal node contains a trainable prototype whose similarity to a latent image patch determines routing toward its children.Prototype similarity measures the extent to which the prototype is present anywhere in the input image.
- Prototype routing: Soft routing sends an image through both child branches with probabilistic weights rather than selecting only one branch.The edge probabilities are complementary, so the sample traverses all edges and contributes to multiple leaves.
- Prediction: The probability of reaching a leaf is the product of the edge probabilities along that root-to-leaf path.Each leaf path therefore receives a probability determined by the routing decisions at its internal nodes.
- Prediction: Each leaf learns a class distribution, and the final prediction combines these distributions weighted by the corresponding leaf path probabilities.The CNN representation is traversed through the tree, allowing all leaves to contribute to the predicted class distribution.
4. Training a ProtoTree
Training learns the CNN, prototypes, and leaf distributions, while tree size controls the number of prototypes and computational cost. A derivative-free leaf update is interleaved with mini-batch optimization, and prototype replacement and pruning support interpretable trees.
- Training setup: Training learns CNN parameters, node prototypes, and leaf class-distribution logits, with a tree of height h containing 2^h leaves and 2^h − 1 prototypes.Because the number of prototypes depends on tree size, learning prototype parameters grows exponentially with h.
- Training setup: ProtoTree training uses a pre-trained CNN, mini-batches, cross-entropy loss, and simultaneous backpropagation updates for the CNN and prototypes.The training algorithm initializes a tree and processes randomly split mini-batches before computing the loss.
- Prototype interpretation: Prototype replacement substitutes each prototype with its nearest latent training-image patch, making the learned prototype representable as an image patch.This replacement supports visualization of prototypes as prototypical parts.
- Leaf learning: Leaf distributions are learned with a derivative-free update because jointly optimizing them with CNN and prototype parameters produced inferior classification results.The authors hypothesize that including leaf parameters in the loss creates an overly complex optimization problem.
- Leaf learning: Computing the prediction once per mini-batch roughly halves training time compared with the original scheme, which computes it twice.The revised procedure intertwines mini-batch gradient descent for CNN and prototypes with the leaf update and incorporates each mini-batch update into leaf learning.
- Pruning: Pruning removes leaves with nearly uniform class distributions and can remove their entire subtrees, reducing explanation size while barely changing prediction accuracy.The pruning threshold is chosen so retained leaves are nearly deterministic and interpretable.
5. Interpretability and Visualization
ProtoTree interpretability is built through prototype visualization, leaf pruning, and conversion from soft to deterministic routing. These mechanisms expose faithful global and local reasoning while reducing explanation complexity.
- Pruning: Pruning removes leaves with nearly uniform class distributions, retaining leaves with discriminative class assignments.The pruning threshold is defined using the maximum leaf probability and the number of classes.
- Prototype Visualization: ProtoTree prototypes are visualized by replacing each learned prototype with its nearest latent training patch.The nearest latent patch is mapped back to pixel space through an upsampled similarity map.
- Prototype Visualization: Using the corresponding latent patch for routing makes the visualized ProtoTree a faithful model explanation.
- Deterministic Reasoning: Hard test-time reasoning can select the highest-probability path or greedily route according to node probabilities.These strategies convert the soft training tree into a deterministic decision path.
6. Experiments
Experiments evaluate ProtoTree on fine-grained bird and car recognition, showing competitive accuracy, effective pruning, and substantially smaller explanations. Deterministic routing preserves nearly the same predictions while visualizations expose learned prototypes and biases.
- Experimental Setup: ProtoTree is evaluated on CUB-200-2011 with 200 bird species and Stanford Cars with 196 car types.The datasets were selected because both were used by ProtoPNet.
- Accuracy: An ensemble of ProtoTrees outperforms a ProtoPNet ensemble and approximates uninterpretable or attention-based methods while retaining global and local explanations.
- Tree Height: For CUB, accuracy increases through height h = 9 and then plateaus, while greater height has a larger effect on CARS.Ensembling substantially increases prediction accuracy but enlarges explanations.
- Pruning: Pruning barely changes prediction accuracy and leaves roughly 1 prototype per class, making the tree almost 90% smaller than ProtoPNet on CUB.ProtoPNet uses 10 prototypes per class, or 2000 prototypes for CUB.
- Deterministic Reasoning: Highest-path-probability routing has nearly the same accuracy as soft routing, with fidelity practically 1.A deterministic ProtoTree with h = 9 requires at most 9 prototypes per prediction, or 45 for an ensemble of 5.
- Visualizations and Discussion: Visualized prototypes generally correspond to perceptually relevant parts and cluster similar-looking classes, but some focus on background cues that reveal learned biases.The examples include sky or water context distinguishing Gray Catbirds from Black Terns.
7. Conclusion
ProtoTree combines neural feature learning with a hierarchical prototype tree to support interpretable fine-grained recognition. The paper reports competitive performance with fewer prototypes and argues that intrinsic interpretability can coexist with predictive accuracy.
- Conclusion: ProtoTree arranges learned prototypes in a hierarchical tree, simplifying model comprehension and error analysis while reducing the number of prototypes by a factor of 10.
- Conclusion: An ensemble of 5 ProtoTrees approximates non-interpretable state-of-the-art accuracy while using fewer prototypes than ProtoPNet.
- Conclusion: The conclusion reports that most learned prototypes are semantically relevant, producing a fully simulatable model.
Supplementary Material
The supplementary material details ProtoTree training, prototype visualization, deterministic test-time conversion, pruning, and additional analyses of learned trees. These experiments examine how structural choices affect accuracy, interpretability, and computational cost.
- Prototype visualization: Class-constrained prototype replacement changes the nearest latent training patch by only 5.86 × 10^-5 on average for CUB.The constrained and unconstrained visualization methods produce similar accuracy, with constraints even slightly improving it.
- Deterministic classification and pruning: Selecting the leaf with the highest path probability nearly matches soft routing on CARS, with fidelity equal to 1, while greedy routing performs slightly worse.Pruning the height-11 CARS tree reduces the average path length to 8.6.
- Leaf behavior: Leaf distributions typically become either concentrated on one class or nearly uniform at 1/K across trained ProtoTrees.This pattern is shown for both CARS and CUB.
- Tree interpretation: Prototype visualizations show distinctive parts and class structure, including shared parts between species and hierarchical clustering of similar classes.Examples include a shared green prototype for Mallard and Ruby Throated Hummingbird and a subtree grouping Warblers.