PiPS: Post-Hoc Prototypical Explanations for Interpretable Semantic Segmentation

Miłosz Adamczyk; Tymoteusz Zapala; Piotr Borycki; Przemysław Spurek

Source-linked AI summary

PiPS: Post-Hoc Prototypical Explanations for Interpretable Semantic Segmentation

Miłosz Adamczyk, Tymoteusz Zapala, Piotr Borycki, Przemysław Spurek

arXiv:2609.16909v1Sep 15, 2026cs.CV

TL;DR

Semantic segmentation lacks scalable, prototype-based explanations despite requiring precise spatial interpretability, while existing ante-hoc approaches require architectural changes and costly retraining. PiPS projects features from frozen pre-trained segmentation models onto spatial prototypes, providing localized explanations without modifying the network. The paper reports 100% preservation of the base model’s original predictive performance and extends the approach to 3D point cloud segmentation.

Problem
Prototype-based explainability remains largely unexplored for dense semantic segmentation, despite the need for precise spatial localization and local interpretability in high-stakes applications.
Method
PiPS extracts dense features from frozen pre-trained segmentation networks and projects them onto representative spatial prototypes without architectural modification or backbone fine-tuning.
Results
100% of the base model’s original predictive performance is preserved while PiPS produces intuitive, region-localized explanations and also adapts to 3D point cloud segmentation.
Takeaways & Limitations
PiPS combines post-hoc scalability with localized prototype-based interpretability for segmentation models without requiring retraining or changing network architectures.
Takeaways & Limitations
Existing prototypical segmentation methods remain constrained by ante-hoc architectures, costly training from scratch, and potential predictive-performance degradation relative to black-box models.

Abstract

from arXiv · show

With the increasing deployment of deep neural networks in critical systems, such as medical diagnostics and autonomous vehicles, ensuring their interpretability is crucial to building trust in decision-making systems. In the field of explainable artificial intelligence, prototype-based reasoning has gained particular popularity, as it mimics human cognitive processes by explaining model decisions based on visual similarity under the looks like this paradigm. While this paradigm has been thoroughly investigated in the context of global image classification, the interpretability of dense predictions, particularly semantic segmentation, remains largely unexplored despite its immense importance in tasks requiring precise object localization. Existing prototype-based interpretable segmentation models rely on ante-hoc architectures, which entails significant limitations because they require costly training from scratch and modifications to the network structure, ultimately leading to a noticeable drop in predictive performance compared to standard black-box models. To address this issue, we propose PiPS (Post-hoc interpretable Prototypical Segmentation), the first fully post-hoc solution for generating prototypical explanations for semantic segmentation models. Our method enables the extraction of intuitive, spatially localized explanations from any pre-trained network without modification or fine-tuning, thereby preserving 100% of the model's original predictive performance. This approach opens a new avenue for the safe and cost-effective deployment of transparent systems in advanced computer vision tasks. Codebase available at https://github.com/gmum/PIPS.

Introduction

Semantic segmentation needs pixel-level, spatially localized explanations, but existing prototypical methods require costly ante-hoc redesigns that can reduce predictive performance. PiPS provides post-hoc prototype explanations from pre-trained networks while preserving the base model’s performance.

Introduction: Semantic segmentation interpretability is important because high-stakes applications require explanations for why specific pixels receive particular class assignments.This need arises in autonomous driving, medical image analysis, and robotic navigation, where class presence alone is insufficient.
Introduction: Existing prototypical segmentation methods require architectural modifications and costly training from scratch, limiting adaptation to large foundation models.Their constrained prototype-friendly latent spaces can also noticeably lower mIoU relative to black-box baselines.
Introduction: PiPS introduces the first fully post-hoc framework for prototype-based explainability in semantic segmentation.It extracts dense pixel-wise feature maps from frozen pre-trained models and projects them onto representative spatial prototypes.
Introduction: PiPS decouples explanation generation from prediction, translating latent representations into intuitive region-based similarities without changing the model’s original weights.The framework is illustrated against existing explanation methods in Figure 1 and with heatmaps in Figure 2.
Introduction: The framework eliminates architectural modifications and retraining while preserving 100% of the base model’s predictive performance across integrated pre-trained segmentation networks.The authors also report highly localized, human-understandable explanations across complex datasets.

Related Works

Prior work includes post-hoc attribution methods and ante-hoc prototype-based segmentation, but neither provides the full combination of scalable deployment and spatially bounded prototypical explanations. PiPS is positioned to fill this gap on already trained segmentation networks.

Related Works: Post-hoc segmentation methods provide attribution maps or adversarial frameworks, but lack the visually verifiable prototype reasoning desired for dense predictions.Grad-CAM is an example of attention-map-based explanation, while hybrid concept methods do not provide exact prototypical patches.
Related Works: Figure 2 compares PiPS and ScaledProtoSeg using heatmaps of spatial activation patterns and prototypical-part localization.The caption directs attention to variations in activation between the approaches without specifying a quantitative winner.
Related Works: Ante-hoc methods such as ProtoSeg and multi-scale prototype networks integrate prototypes into specialized architectures for localized pixel-wise explanations.Multi-scale grouped prototypes represent objects at different spatial granularities.
Related Works: Strictly ante-hoc prototypical segmentation requires expensive retraining and constrained latent spaces that can reduce mIoU compared with black-box models.These requirements make such methods difficult to apply to modern large-scale foundation models.
Related Works: PiPS combines post-hoc scalability with the localized interpretability of ante-hoc prototypical segmentation on already trained networks.It does so without architectural modifications or retraining while preserving 100% of the original model’s performance.

PiPS

PiPS explains pre-trained semantic segmentation models post hoc by rotating latent coordinates, optimizing localized concepts, and retrieving spatially bounded prototypes. It preserves the original prediction mapping while producing localized visual evidence, and extends the same procedure to point-cloud segmentation.

Orthogonal Transformation: PiPS applies a post-hoc orthogonal coordinate rotation to dense features while freezing the backbone and preserving the original classifier mapping.A trainable matrix parameterizes the rotation, and the classifier weights are rotated dually so the predictions remain unchanged.
Optimization Objective: The spatial purity objective aligns latent channels with compact, localized semantic regions by minimizing weighted activation entropy.Channel activations are converted into spatial probability distributions, and entropy is weighted by normalized peak activation to emphasize informative channels.
Training Strategy: Training dynamically retains the highest-scoring exemplars for each channel, reducing the candidate set from K=100 to K=5 over 20 epochs.The dataset is subset every two epochs using purity-based exemplar scores, progressively focusing on representative prototypes.
Segment Visualization: During inference, PiPS restricts feature evaluation to each predicted segment, locates the peak activation, and compares its image patch with top-scoring training exemplars.Nearest-neighbor mask resizing and segment-constrained attribution keep the visual explanation inside the predicted class boundary.
Adaptation to Point Clouds: The framework also applies to point-cloud segmentation when the model provides spatially coherent latent representations, yielding localized point-group prototypes.Point-cloud features are grouped, transformed after the final patch representations, and paired with the same purity, training, and retrieval procedures.

Experiments and Results

PiPS provides localized prototypical explanations across image and point-cloud segmentation while preserving the underlying model’s predictive outputs. User evaluations found positive ratings for feature presence, coherence, and similarity, with feature presence rated highest.

Explanation of model decision: PiPS produces disentangled, part-by-part explanations that pair critical spatial regions with semantically rich training prototypes, unlike diffuse SegGradCam maps and degraded ScaledProtoSeg explanations.The qualitative comparison is conducted on multi-object PASCAL VOC 2012 scenes.
Segmentation Performance: 100% of the original DeepLabV3 black-box accuracy is preserved by PiPS, whereas ante-hoc ProtoSeg and ScaledProtoSeg modifications noticeably degrade their DeepLabV2 baseline.The comparison uses mean Intersection over Union on the PASCAL VOC 2012 validation dataset.
Point Cloud Segmentation: Figure 5 explains each predicted point-cloud part using its most active localized patch and prototypes from other objects sharing that active channel.The first, second, and third columns show the part class, selected patch, and corresponding prototypes, respectively.
Point Cloud Segmentation: mIoU_C of 84.1 and mIoU_I of 85.6 match the original Point-BERT model on ShapeNet Parts point-cloud segmentation.The result indicates that the post-hoc adaptation does not alter segmentation performance.
User Study Results: Feature Presence received the highest user rating at Mean = 3.60 ± 1.01 and Median = 4.0, followed by Visual Coherence at Mean = 3.56 ± 1.06 and Median = 4.0.Both metrics were significantly above the neutral midpoint of 3.0.
User Study Results: Visual Similarity scored Mean = 3.15 ± 1.08 and Median = 3.0, significantly exceeding the neutral baseline, while Friedman tests found metric-dependent evaluation differences.The reported Wilcoxon test for Visual Similarity gives p = 0.0429; Friedman tests give p ≤0.003 across metrics.

Conclusions

PiPS is a fully post-hoc framework that generates localized, human-understandable prototype explanations without modifying architectures or fine-tuning backbones. It preserves 100% of base-model predictive performance and also extends to 3D point-cloud segmentation.

Conclusions: PiPS uses an orthogonal coordinate transformation and spatial purity objective on frozen pre-trained backbones to extract region-localized explanations without architectural changes or backbone fine-tuning.The framework is presented as a post-hoc approach for prototype-based interpretability in semantic segmentation.
Conclusions: 100% preservation of the base model’s original predictive performance resolves the performance-interpretability trade-off reported for ante-hoc prototypical segmentation methods.The conclusion also reports successful adaptation to 3D point-cloud segmentation and support from qualitative, quantitative, and user-study evaluations.

A.1 Explanations of model decision

Additional experiments compare PiPS with SegGradCam and ScaledProtoSeg on multi-object PASCAL VOC 2012 scenes. The results emphasize PiPS’s disentangled, part-by-part decomposition of dense predictions.

A.1 Explanations of model decision: PiPS provides disentangled, part-by-part object decompositions in multi-object PASCAL VOC 2012 scenes when compared with SegGradCam and ScaledProtoSeg.The comparison concerns explanations of model decisions rather than only segmentation outputs.

A.2 More details on user study

The user study collected 26 valid responses through randomized Google Forms questions after filtering unengaged submissions. Participants assessed explanation quality and compared PiPS with ScaledProtoSeg for segmentation quality.

A.2 More details on user study: The study retained 26 valid responses after filtering submissions with identical answers across all questions.Questions were randomized, so each participant received a subset of unique queries.
A.2 More details on user study: Participants evaluated generated explanation quality and which method produced better segmentations, comparing only PiPS with ScaledProtoSeg.SegGradCam was excluded from the segmentation comparison because its post-hoc nature yields the same results as PiPS.

A.3 Segmentation Performance

PiPS preserves the pre-trained model’s segmentation performance through its strictly post-hoc design. The reported mIoU comparison shows post-hoc methods retaining the original black-box accuracy while ante-hoc modifications reduce baseline performance.

100% of the pre-trained model’s predictive performance is preserved by PiPS, with identical categorical segmentation outputs for each image.The authors report no minor numerical errors affecting the predictions; mIoU values are provided in Table 3.
Table 3 compares PiPS and SegGradCam with ante-hoc ProtoSeg and ScaleProtoSeg on PASCAL VOC 2012 validation mIoU.The table contrasts post-hoc preservation of DeepLabV3 black-box accuracy with degraded DeepLabV2 performance after ante-hoc modifications.

A.4 Dataset

The experiments use PASCAL VOC 2012, a challenging dense-prediction benchmark containing multi-object scenes, severe occlusions, and substantial within-class spatial variation.

PASCAL VOC 2012 provides dense multi-object scenes with severe occlusions and high intra-class spatial variation for evaluating prototype-based segmentation.Only a few prior parts-based prototypical methods, including ProtoSeg and ScaleProtoSeg, had been evaluated on such complex scenes.

A.5 PACAL VOC 2012 Experiments

The experiments evaluate PiPS through lightweight post-hoc optimization, qualitative explanations, and user-study interfaces across image and point-cloud segmentation settings. Explanations are spatially localized to predicted segments and visualized with prototypes for multiple object and part classes.

Experimental setup: PiPS keeps the underlying segmentation network frozen while optimizing an orthogonal transformation and mining representative spatial prototypes.The added computation is limited to the post-hoc optimization and prototype-mining stages.
Qualitative image explanations: Figures 8–16 show image-segmentation explanations with original images, ground truth, predictions, target segments, and generated prototypes.The examples cover multiple PASCAL VOC 2012 object classes, including birds, buses, plants, dogs, chairs, and vehicles.
Point-cloud experiments: Point-cloud experiments use Point Transformer V1 from Point-BERT with frozen pre-trained weights and a post-hoc disentanglement module.Figures 17–27 present explanations for ShapeNetPart objects and parts using predicted and ground-truth segmentations alongside selected prototypes.
Explanation procedure: For dense predictions, PiPS restricts feature evaluation to each predicted class mask and extracts peak activations to obtain localized visual patches.This spatial restriction avoids background contamination in pixel-level explanations.
User study: The user study includes prototype-similarity questions and a segmentation-quality comparison between PiPS and ScaledProtoSeg.Participants rate explanations on a 5-point Likert scale, while segmentation comparisons allow choosing either method or neither.