Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes

Jon Donnelly; Alina Jade Barnett; Chaofan Chen

Source-linked AI summary

Deformable ProtoPNet: An Interpretable Image Classifier Using Deformable Prototypes

Jon Donnelly, Alina Jade Barnett, Chaofan Chen

arXiv:2111.15000v3Nov 29, 2021cs.CVcs.AIcs.LG

TL;DR

Prototype-based image classifiers provide interpretable “this looks like that” reasoning but are limited by spatially rigid prototypes that cannot explicitly model pose variation. Deformable ProtoPNet makes prototype parts move according to each input, and the paper reports improved explanation richness, state-of-the-art benchmark accuracy, and specific remaining limitations.

Problem
Existing prototype-based image classifiers use spatially rigid prototypes that cannot explicitly account for geometric transformations or pose variations.
Method
Deformable ProtoPNet uses spatially flexible prototypes composed of prototypical parts that adapt their relative positions depending on the input image.
Results
The model achieves state-of-the-art accuracy, including 87.8% with five ensembled models on CUB-200-2011 and competitive 86.5% accuracy on Stanford Dogs.
Takeaways & Limitations
Deformable ProtoPNet captures pose variations while providing richer, context-sensitive explanations than rigid prototype-based classifiers.
Takeaways & Limitations
Offsets are shared across deformable prototypes at each spatial location, and some prototypical parts exhibit semantic mismatches with image parts judged similar by the model.

Abstract

from arXiv · show

We present a deformable prototypical part network (Deformable ProtoPNet), an interpretable image classifier that integrates the power of deep learning and the interpretability of case-based reasoning. This model classifies input images by comparing them with prototypes learned during training, yielding explanations in the form of "this looks like that." However, while previous methods use spatially rigid prototypes, we address this shortcoming by proposing spatially flexible prototypes. Each prototype is made up of several prototypical parts that adaptively change their relative spatial positions depending on the input image. Consequently, a Deformable ProtoPNet can explicitly capture pose variations and context, improving both model accuracy and the richness of explanations provided. Compared to other case-based interpretable models using prototypes, our approach achieves state-of-the-art accuracy and gives an explanation with greater context. The code is available at https://github.com/jdonnelly36/Deformable-ProtoPNet.

1. Introduction

Existing prototype-based image classifiers support “this looks like that” explanations but use spatially rigid prototypes. Deformable ProtoPNet introduces spatially flexible prototypes whose parts adapt to input images, capturing pose variations while improving accuracy and explanation richness.

Motivation: Prototype-based classifiers compare input-image parts with learned training prototypes to produce “this looks like that” explanations.These models combine deep learning with case-based reasoning for interpretable image classification.
Problem: Spatially rigid prototypes cannot explicitly account for geometric transformations or object pose variations.
Approach: Deformable ProtoPNet lets each prototype’s prototypical parts adapt their relative spatial positions according to the input image.The parts move while the prototype is compared with image features, enabling spatial flexibility.
Contribution: Deformable ProtoPNet explicitly captures pose variations and improves model accuracy and explanation richness.
Contribution: The paper reports spatially flexible deformable prototypes as a first for prototypical case-based interpretable neural networks and state-of-the-art accuracy on CUB-200-2011 and Stanford Dogs.It also introduces angular margins to improve the accuracy of case-based interpretable neural networks.

3. Deformable Prototypes

Deformable ProtoPNet represents prototypes as collections of movable prototypical parts and compares them with image features at input-dependent offset locations. Fixed-length representations give this comparison a cosine-similarity interpretation, while scanning selects each prototype’s strongest image match.

Prototype representation: A deformable prototype is a ρ1 × ρ2 × d tensor whose spatial positions represent individual 1 × 1 prototypical parts.Each part is treated as a semantic concept that can move independently of the others.
Deformation: Equation (5) computes prototype similarity by comparing each prototypical part with an image feature sampled at its input-dependent deformed position.Fractional offsets are handled through feature interpolation, with interpolated features constrained to preserve the L2 length r.
Geometric interpretation: Fixed-length prototype and image-feature vectors make each part contribution a cosine similarity determined by their angle on a hypersphere.The entire deformable prototype and the stacked features at deformed positions likewise use cosine similarity.
Image-level matching: The prototype’s image-level similarity is the maximum similarity over all spatial positions scanned in the image-feature tensor.
Prototype configurations: The experiments use both 3 × 3 and 2 × 2 deformable prototypes, with the latter implemented using four spatially separated prototypical parts.

4. Deformable ProtoPNet

Deformable ProtoPNet combines a CNN, deformable prototypes, and a final classifier to perform case-based reasoning with spatially flexible prototype parts. Its training uses clustering, separation, subtractive-margin, and orthogonality objectives, followed by prototype projection and last-layer optimization.

Architecture: A CNN maps an image to normalized latent features, a deformable prototype layer produces similarity scores, and the final layer converts them into class scores.The architecture consists of backbone f, deformable prototype layer g, and fully connected layer h.
Training procedure: Training proceeds in three stages: optimizing pre-last-layer features, projecting prototypes onto similar training features, and optimizing the last-layer connections.Prototype projection uses the most similar collection of interpolated image features and keeps all parts of a prototype on one training image.
Training objectives: Cluster and separation losses encourage same-class image features to approach same-class prototypes while separating them from prototypes of other classes on a hypersphere.The cluster loss maximizes similarity to correct-class prototypes, whereas the separation loss penalizes similarity to incorrect-class prototypes.
Training objectives: Subtractive margin cross entropy increases class separation by inflating incorrect-class prototype activations during training, forcing the network to increase angular distances.The method applies a fixed margin ϕ = 0.1 to incorrect-class prototype activations.
Training objectives: Orthogonality loss discourages duplicated prototypical parts by driving pairwise inner products toward a scaled identity matrix, thereby increasing part diversity.The loss uses P^(c)P^(c)⊤ and targets r^2I^(ρL).

5. Experiments and Numerical Results

Experiments evaluate Deformable ProtoPNet across backbone architectures and fine-grained image datasets. The model generally outperforms comparable interpretable baselines, while deformations and training regularizers improve or preserve accuracy across ablations.

Experimental setup: Experiments use CUB-200-2011 and Stanford Dogs with VGG, ResNet, and DenseNet backbones, including 3 × 3 and 2 × 2 deformable prototypes.The CUB case study uses full uncropped images and commonly trains six 3 × 3 or ten 2 × 2 prototypes per class.
Cross-backbone results: Deformable ProtoPNet achieves higher accuracy than ProtoPNet and the non-interpretable baseline in all tested backbone cases.Except for VGG-16 and VGG-19, the deformed model with 2 × 2 prototypes performs best among models sharing a backbone.
Ablation results: Deformations, orthogonality loss, and subtractive margin generally improve or maintain accuracy under matched ablation settings.Deformations help most backbone architectures, subtractive margin generally improves accuracy, and orthogonality loss maintains accuracy in most cases.
CUB-200-2011: 86.4% accuracy is achieved by a single Deformable ProtoPNet with ten 2 × 2 prototypes per class on full CUB-200-2011 test images.This result is reported alongside 86.1% for six 3 × 3 prototypes per class.
Benchmark results: 87.8% accuracy is achieved by an ensemble of five Deformable ProtoPNets with 2 × 2 prototypes, exceeding the competing models reported in the comparison.On Stanford Dogs, the model achieves 86.5% accuracy, described as competitive with the state of the art.

6. Conclusion

Deformable ProtoPNet is presented as a case-based interpretable neural network with deformable prototypes and competitive accuracy and transparency. The paper also identifies shared offsets and semantic mismatches as limitations.

Deformable ProtoPNet is a case-based interpretable neural network with deformable prototypes.
Table 3 compares accuracy and interpretability across CUB-200-2011 and Stanford Dogs, including full-image, bounding-box, ensemble, and prototype-configuration variants.
The model shares offsets across all deformable prototypes at each spatial location and exhibits semantic mismatches between some prototypical and image parts.

7. Proof of Theorem 3.1

The theorem shows that ordinary bilinear interpolation can change the L2 norm even when all four input vectors have equal norm. A square-root-based interpolation instead preserves that norm.

Bilinear interpolation of four vectors with equal L2 norm does not generally preserve the interpolated vector’s L2 norm.The theorem states this for vectors with norm r and interpolation weights determined by α and β.
The proposed L2 norm-preserving interpolation takes the element-wise square root of a weighted sum of the four vectors’ element-wise squares.
For equal-norm inputs and α, β in [0, 1], this operation guarantees an interpolated vector with L2 norm r.
A counterexample uses four orthogonal vectors, each with norm r, whose equally weighted average has squared norm r^2/4 rather than r^2.
In Deformable ProtoPNet, the operation computes interpolated image features at offset spatial locations using four neighboring feature vectors and fractional offsets α and β.

8. Backpropagation through a Deformable Prototype

The model computes deformable prototype similarities from interpolated, offset-dependent image features. Because offsets are differentiable functions of the feature tensor, the similarity score supports backpropagation.

A deformable prototype similarity sums dot products between each prototypical part and its corresponding interpolated image feature.
The offsets Δ1 and Δ2 depend on the image-feature tensor and are produced by convolutional layers for each spatial position and prototypical part.
Interpolated features are obtained by taking the element-wise square root of ζ, a bilinear-weighted sum of squared neighboring feature vectors.
The gradient with respect to an input feature includes direct dependence through the feature term and indirect dependence through both offsets, requiring the chain rule.
Because the offset-producing convolutional layers are differentiable, the prototype similarity gradient is well-defined and can be back-propagated.

9. More Examples of Reasoning Processes

The examples visualize how Deformable ProtoPNet reasons by scanning deformable prototypes across test images and showing the spatial arrangements of their parts. The figures cover bird and dog classifications.

For each test image, the model compares the predicted class’s deformable prototypes while scanning them across the image-feature tensor.
Figure 6 covers three bird examples, Figure 7 covers three additional bird examples with 2 × 2 prototypes, and Figure 8 covers three Stanford Dogs examples.
Prototype parts adapt their relative spatial positions during scanning to seek image parts that are semantically similar to the prototypical parts.

10. Local Analysis: Visualizations of Most Similar Prototypes to Given Images

Local analyses visualize which deformable prototypes most closely match test images and where their prototypical parts align spatially. These visualizations expose the similarity evidence used for predictions across CUB-200-2011 and Stanford Dogs.

The analyses show the two most similar deformable prototypes for each of three CUB-200-2011 test images.
For CUB-200-2011 images, the visualizations include both 3 × 3 and 2 × 2 deformable prototypes.
The Stanford Dogs analyses show the two most similar deformable prototypes for each of three test images.
For each test image, the top row shows similar prototypes, while the bottom row shows the prototypical-part arrangement producing each similarity score.

11. Global Analysis: Visualizations of Most Similar Images to Given Prototypes

Global analyses examine which training and test images most closely match individual deformable prototypes. The reported nearest images generally belong to the prototype’s class and display corresponding visual features.

Each global analysis shows two most similar training images and two most similar test images for each deformable prototype.
The visualizations cover deformable prototypes from CUB-200-2011 using 3 × 3 and 2 × 2 prototypes, and from Stanford Dogs using the best-performing model.
Most similar training and test images for a deformable prototype generally come from the same class as the prototype.
The global analyses provide image-level context for interpreting what visual examples each deformable prototype matches.

12. Numerical Results on Stanford Dogs

On Stanford Dogs, Deformable ProtoPNet remains competitive across CNN backbones, improves or preserves accuracy over non-deformable counterparts, and reaches state-of-the-art-level performance.

Experiments evaluate VGG-19, ResNet-152, and DenseNet-161 backbones, all pretrained on ImageNet.
Deformable ProtoPNet exceeds the uninterpretable baseline in two of three backbone cases, including the highest-performing ResNet-152 model.
Deformable ProtoPNet achieves substantially higher accuracy than ProtoPNet across all evaluated backbone cases.
Accuracy with deformations is higher than or equal to the corresponding non-deformable model across all three CNN backbones.
More than 3% higher test accuracy is achieved with deformations than without them when VGG-19 is used as the backbone.
86.5% accuracy is reported for a single Deformable ProtoPNet, competitive with state-of-the-art interpretable models.

13. Experimental Setup

The experiments use multiple CNN backbones, deformable prototype configurations, and training stages, with reasoning visualized through local, global, and class-prediction analyses.

Experimental setup: Experiments use VGG, ResNet, and DenseNet CNN backbones with pretrained weights and a 14 × 14 latent spatial dimension.
Experimental setup: Offset prediction uses two convolutional layers, producing 18 channels for 3 × 3 prototypes and 8 channels for 2 × 2 prototypes.
Experimental setup: CUB-200-2011 models use 6 prototypes per class with 9 parts or 10 prototypes per class with 4 parts, while Stanford Dogs uses 10 prototypes per class with 9 parts.
Training: Models are trained for 30 epochs with warm-up stages, prototype projection, and last-layer optimization at epochs 20 and 30.
Reasoning analyses: Reasoning examples display the two highest-similarity deformable prototypes of the predicted class for bird and dog test images.
Reasoning analyses: Local analyses show similar prototypes and their part arrangements, while global analyses show nearest training and test images for each prototype.