Source-linked AI summary
ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery
Dawid Rymarczyk, Łukasz Struski, Jacek Tabor, Bartosz Zieliński
TL;DR
Prototype-based explanations can suffer from too many class-specific prototypes and semantically similar prototypes that are distant in representation space. ProtoPShare uses data-dependent merge-pruning to share prototypes across classes, and experiments report reduced prototype counts, maintained high accuracy, and improved interpretability and similarity discovery.
Problem
Prototype-based explanations are limited by large numbers of class-specific prototypes and unstable predictions when semantically similar prototypes are distant in representation space.
Method
ProtoPShare first trains with exclusive prototypes, then uses data-dependent similarity to iteratively merge similar prototypes across classes.
Results
ProtoPShare reduces the number of prototypes up to three times while maintaining high accuracy across experiments.
Takeaways & Limitations
Sharing prototypes reduces model complexity and enables discovery of similarities between classes while preserving self-explaining predictions.
Takeaways & Limitations
The theoretical guarantee applies to correctly classified inputs whose predictions have sufficient confidence before prototype merging.
Abstract
from arXiv · showhide
In this paper, we introduce ProtoPShare, a self-explained method that incorporates the paradigm of prototypical parts to explain its predictions. The main novelty of the ProtoPShare is its ability to efficiently share prototypical parts between the classes thanks to our data-dependent merge-pruning. Moreover, the prototypes are more consistent and the model is more robust to image perturbations than the state of the art method ProtoPNet. We verify our findings on two datasets, the CUB-200-2011 and the Stanford Cars.
1. Introduction
ProtoPShare addresses limitations of prototype-based explanations by sharing prototypes across classes and identifying semantic similarity beyond representation-space distance. It aims to reduce prototype counts, improve interpretability, and support class-similarity discovery.
- 1. Introduction: Prototype-based self-explaining models are motivated by the need for human-understandable decisions in applications such as medical diagnosis and autonomous systems.They explain classifications through recognizable object parts, such as tires, roofs, headlights, and horns.
- 1. Introduction: ProtoPNet assigns prototypes to individual classes, producing many prototypes and limiting interpretability because explanations become larger and more complex.Prototype-based explanations compare image parts with class-specific prototypical parts.
- 1. Introduction: ProtoPShare uses initial training with exclusive prototypes followed by iterative merge-pruning of the most similar prototypes.The pruning procedure is based on data-dependent similarity and merges prototypes across classes.
- 1. Introduction: Data-dependent similarity identifies semantically similar prototypes even when they are distant in representation space.This addresses separation caused by training prototypes of different classes away from one another.
- 1. Introduction: ProtoPShare shares prototypes between classes, reducing their number and enabling prototypical similarity discovery between classes.The method extends ProtoPNet with shared prototypes obtained through data-dependent pruning.
3. ProtoPShare
ProtoPShare extends prototype-based self-explaining classification with shared prototypes obtained through data-dependent merge-pruning. The architecture computes prototype similarities from image patches, merges similar prototypes, and provides conditions under which merging preserves predictions.
- Architecture: ProtoPShare combines convolutional layers, a prototype layer, and a fully connected classifier whose shared prototypes connect to multiple classes.Shared prototypes reduce the total number of prototypes while retaining class-specific connections.
- Data-dependent merge-pruning: The method first trains with exclusive class prototypes, then returns a smaller network by pruning through prototype merging.This two-phase design addresses instability caused by semantically similar prototypes being distant in representation space.
- Data-dependent merge-pruning: Each pruning step computes prototype similarities, removes one prototype from selected similar pairs, and aggregates its classifier weights into the retained prototype.The merged prototype is thereby reused by the classes connected to either original prototype.
- Data-dependent merge-pruning: The similarity measure uses training-set patch representations to identify semantically similar prototypes that may be distant in the representation space.The method compares prototype responses over training patches rather than relying only on prototype-space distance.
- Theoretical results: The theorem states that merging similar prototypes preserves an input’s prediction when the pre-merge top-two logits are separated by a sufficient confidence margin.Under the stated assumptions, the correct-class logit decreases and other-class logits increase only within bounded amounts.
4. Experiments
Experiments evaluate ProtoPShare on CUB-200-2011 and Stanford Cars across pruning rates, merge step sizes, finetuning strategies, and image perturbations. ProtoPShare generally maintains higher accuracy with fewer prototypes, while smaller pruning steps and last-layer finetuning are favored.
- How many prototypes are required?: ProtoPShare achieves higher accuracy than ProtoPNet and the tested variants across pruning rates, remaining reasonable with only 20% of initial prototypes in one ResNet34 setting.ProtoPNet can prune at most 30% of prototypes, whereas ProtoPShare continues through substantially higher pruning rates before critical accuracy declines.
- Accuracy vs. step size: Accuracy is higher when each pruning step merges 5% rather than 10% or 15% of prototypes on CUB-200-2011.The authors attribute this to smaller prototype-set changes making 25-iteration finetuning more effective.
- Why two training phases?: With the same final prototype count, ProtoPShare using last-layer finetuning after pruning achieves the highest accuracy among the compared training and finetuning strategies.The comparison includes no-pruning shared prototypes, ProtoPNet, and finetuning of h, P and h, or f, P and h.
- Resistance to perturbations: ProtoPShare with 600 prototypes is slightly more accurate than ProtoPNet with 1400 prototypes under stronger brightness, contrast, saturation, hue, and perspective perturbations.The comparison uses DenseNet161 and supports the conclusion that reducing prototypes does not increase susceptibility to these perturbations.
- Interpretability: ProtoPShare merges prototypes corresponding to shared visual parts, including eighteen prototypes from seventeen classes representing a brighter belly region in birds.This illustrates the method’s use for discovering similarities between classes.
5. Interpretability of ProtoPShare
ProtoPShare’s interpretability analysis examines merge-pruning, inter-class similarity discovery, and data-dependent prototype similarity. The reported evidence indicates that merging preserves meaningful prototypes while improving class-consistent activations and model-capacity use.
- Why merge-pruning instead of pruning?: ProtoPShare’s merge-pruning joins semantically similar prototypes across classes, avoiding the removal of important prototypical parts while preserving accuracy.The authors motivate merging because about 30% of prototypes pruned by ProtoPNet represented significant prototypical parts rather than background.
- Inter-class similarity discovery: The shared-prototype graph represents class similarity through edges whose strengths count shared prototypes, enabling similarity analysis and class clustering.The graph can also support visualizations of prototypes shared by selected class pairs.
- Why data-dependent similarity?: Data-dependent similarity identifies semantically similar prototypes that can differ in color distribution, unlike proximity measured by Euclidean representation distance.The authors describe this capability as especially important after merge-pruning.
- Why data-dependent similarity?: ProtoPShare is the only compared method whose correctly assigned patch percentage constantly increases during pruning, indicating more effective use of model capacity.Figure 10 evaluates five highly activated parts from each testing image and checks whether their activations correspond to the true class.
- User study: A user study compares prototype pairs selected by data-dependent similarity against pairs closest in representation space to assess consistency.The pairs were shown in random order to reduce order bias.
- Discussion: The interpretability discussion concludes that merge-pruning, data-dependent similarity, and shared prototypes support more meaningful explanations and similarity discovery.The section also frames data-dependent similarity as superior to the Euclidean norm for this purpose.
6. Conclusions
ProtoPShare extends prototypical-part explanation by sharing prototypes across classes through data-dependent pruning. The paper reports fewer prototypes, improved interpretability and similarity discovery, and high accuracy maintained across theoretical and experimental evaluation.
- 6. Conclusions: ProtoPShare shares prototypes between classes and can reduce their number up to three times while maintaining high accuracy.The method uses data-dependent pruning to merge prototypes with similar semantics.
- 6. Conclusions: The method increases interpretability and enables similarity discovery through shared prototypes.The conclusion attributes these capabilities to prototype sharing and data-dependent pruning.
7. Proof for Theorem 1
The proof analyzes how merging prototypes changes class output logits under stated assumptions. It concludes that a sufficiently large pre-merge margin preserves the correct prediction after merging.
- 7. Proof for Theorem 1: The proof models each prototype through its nearest training-patch representation and assumes bounded prototype relationships controlled by δ and ε.It also assumes class-specific prototype weights are one for assigned prototypes and zero otherwise.
- 7. Proof for Theorem 1: After merging, the correct-class output logit can decrease by at most ∆c_max, while another class’s logit can increase by at most ∆k_max.The proof derives separate bounds for the correct and incorrect classes.
- 7. Proof for Theorem 1: For classes whose prototypes are not merged into other prototypes, the corresponding output remains unchanged.This establishes the no-change case before bounding affected logits.
- 7. Proof for Theorem 1: If the correct-class logit exceeds every competing logit by at least the merge-induced bound, the input remains correctly classified after prototype merging.This is the theorem’s final prediction-preservation claim.
8. Training details
The reproducibility section supplies omitted training details, including offline augmentation and architectural modifications. Training uses cropped dataset images and specified convolutional activations.
- 8. Training details: Training applies exhaustive offline augmentation using rotation, skewing, flipping, and shearing, each with probability 0.5.The images are cropped using the datasets’ provided bounding boxes.
- 8. Training details: The architecture adds two 1 × 1 convolutional layers before the prototype layer and uses ReLU except for a final sigmoid convolution.These details are provided to support reproducibility.
9. Undesirable behaviors of ProtoPNet pruning
The authors motivate merge-pruning by finding that roughly 30% of ProtoPNet’s pruned prototypes represent significant prototypical parts rather than background.
- 9. Undesirable behaviors of ProtoPNet pruning: Around 30% of ProtoPNet’s pruned prototypes represent significant prototypical parts instead of background.This observation motivated introducing the merge-pruning algorithm and aligns with prior analysis identifying mixed-class nearest patches as background prototypes.
10. Similarity distributions
ProtoPShare’s data-dependent similarity identifies semantically similar prototypes even when they are far apart in representation space.
- 10. Similarity distributions: Data-dependent similarity finds semantically similar prototypes even when they are distant in representation space.The comparison uses normalized pairwise prototype distances for ProtoPShare and a data-independent alternative on CUB-200-2011.
11. User study questionnaire
The user study evaluates data-dependent pruning through questionnaires presenting image sets whose boxed parts should share a bird-part characteristic.
- 11. User study questionnaire: The user study used five questions across four questionnaire versions to evaluate whether boxed image parts conveyed a shared semantic meaning.The questionnaires instructed users to assess sets of yellow-boxed images representing characteristics of bird parts.
12. Extended versions of figures and tables
The extended materials provide additional prototype-similarity visualizations, an inter-class similarity graph, pruning-accuracy comparisons, pruning-rate comparisons, and Stanford Cars accuracy results.
- 12. Extended versions of figures and tables: Additional figures visualize prototype activation and nearest image parts, while a Stanford Cars table reports accuracy for models with the same final prototype count.The prototype examples are described as not representing background.
- 12. Extended versions of figures and tables: Prototype-pair figures show semantically similar prototypes that are distant in representation space but close under data-dependent similarity, alongside pairs close in representation space.Each prototype is represented by three closest image parts marked with yellow bounding boxes.
- 12. Extended versions of figures and tables: The inter-class similarity graph represents classes as nodes and connects them according to the number of shared prototypes.For example, Lincoln sparrow shares three prototypes with Savannah sparrow but none with Prairie warbler; the graph supports similarity discovery and clustering.
- 12. Extended versions of figures and tables: The extended figures compare ProtoPShare, ProtoPNet, and data-independent and random variants across pruning rates and architectures on CUB-200-2011.Accuracy is higher-is-better, and early pruning steps are zoomed because ProtoPNet prunes at most 30% of prototypes.
- 12. Extended versions of figures and tables: The extended materials include accuracy curves for merging 5%, 10%, or 15% of prototypes per pruning step across CUB-200-2011 architectures.Results for additional architectures appear in the supplementary materials.