MorphoSHAP: Rethinking the Unit of Attribution in Explanation for Deep Visual Models

• The choice of attribution units defines the explanation space, so changing those units changes the explanation itself.Source: §1 · p. 1
• Pixels, superpixels, and patches can locate evidence but provide limited information about the visual structure the model relies on.Source: §1 · pp. 1–2
• MORPHOSHAP uses Tree of Shapes structures as Shapley players, describing each by scale and geometric type and estimating its prediction contribution with Shapley values.Source: §1 · p. 2
• Each relevant shape carries spatial support, scale, geometric type, and signed contribution, enabling heatmap, SHAP-plot, textual, and dataset-level class analysis.Source: §1 · p. 2
• Experiments show MORPHOSHAP provides faithful shape-level attributions with more structured and interpretable explanations than pixel-, patch-, and superpixel-based alternatives.Source: §1 · p. 2

2.1. Gradient-Based Attribution

• Gradient-based methods produce efficient spatial heatmaps but depend on model activations and gradients, can lack precise object boundaries, and face saturation and architecture limitations.Source: §2.1 · p. 2

2.2. Hierarchical and Structured SHAP

• Hierarchical and superpixel SHAP methods structure image regions to reduce cost or improve coherence, while MORPHOSHAP derives players from a multi-scale morphological hierarchy without regular partitions or external segmentation.Source: §2.2 · p. 2

2.3. Mathematical Morphology for XAI

• The Tree of Shapes is a self-dual hierarchy of nested connected structures at multiple scales; MORPHOSHAP uses these structures directly as attribution units.Source: §2.3 · pp. 2–3

3.1. Overview and Problem Formulation

• Each explanatory unit specifies where evidence lies, its scale and geometry, and how strongly it contributes to the target class prediction.The method explains the class logit unless stated otherwise.Source: §3.1 · p. 3

3.2. Morphological Shape Decomposition

• The Tree of Shapes decomposes images into connected, nested structures that serve as explanatory units rather than flat segments or regular patches.The formulation is presented for grayscale images for simplicity, although color-image extensions exist.Source: §3.2 · p. 3

3.3. Multi-Scale Morphological Characterization

• Each shape is assigned one of eight ordered scale levels according to its relative area, from fine to coarse.The exact scale intervals are reported in Appendix A.1.Source: §3.3 · p. 3

3.4. Geometric Characterization and Shape Naming

• Each shape receives a label from a shared geometric vocabulary, using either deterministic rules or a learned classifier over geometric descriptors.The descriptors include area, perimeter, aspect ratio, circularity, solidity, and polygon-approximation vertex count.Source: §3.4 · pp. 3–4

3.5. Morphological Shape Contribution

• Shapes are players in a cooperative game, and coalition images retain active-shape pixels while replacing inactive shapes with a mid-gray reference.The residual background remains unchanged, and the coalition mask includes pixels belonging to at least one active shape.Source: §3.5 · p. 4
• Each shape’s Shapley value is its average marginal contribution across coalitions; positive values support the class and negative values oppose it.KernelSHAP estimates the values by sampling coalitions and fitting a Shapley-weighted additive surrogate.Source: §3.5 · pp. 4–5

3.6. Local Morphological Explanations

• Local explanations associate shapes with scale, geometric category, and signed contribution, presenting them through shape-level plots, spatial maps, or text templates.The spatial map accumulates evidence across nested shapes; the templates describe the most influential positive and negative shapes.Source: §3.6 · p. 5

3.7. Global Morphological Explanations

• Because scale and geometry labels are shared across images, class-level explanations can aggregate and distinguish structures that systematically support or oppose a class.The method separately aggregates positive and negative evidence for each geometric category and scale.Source: §3.7 · pp. 5–6

4.1. Experimental Setup

• The evaluation spans five domains and three architectures, comparing gradient-based, perturbation-region, and tree-based methods with insertion/deletion AUC and runtime metrics.The authors also report ablations of structural parameters and core algorithmic components.Source: §4.1 · p. 6

4.2. Quantitative Comparison

• MorphoSHAP achieved the highest Insertion AUC on ImageNet, EuroSAT, Galaxy10, and TissueMNIST, and the lowest Deletion AUC on ImageNet, EuroSAT, and Waterbirds; ShapBPT fell short on insertion scores on most datasets.The authors report stable results across hyperparameter changes.Source: §4.2 · pp. 6–7
• On ImageNet, MorphoSHAP averaged 0.13 seconds per image, over 6× faster than SLIC-SHAP (0.82s), over 12× faster than Integrated Gradients (1.64s), and more than 2× faster than ShapBPT (0.29s).The authors attribute its resolution scaling to M ≪ H · W.Source: Table 2 · p. 7, §4.2 · p. 7

4.4. Cross-Domain Global Semantic Insights

• Global aggregation links industrial-zone predictions to Complex primitives and spiral-galaxy predictions to Rectangle cores plus Complex arms; shape categories can be added for domain-relevant structures.Source: §4.4 · p. 7

4.3. Qualitative and Semantic Explanations

• Across ImageNet, EuroSAT, and Galaxy10, MorphoSHAP isolates geometric primitives and quantifies their prediction contributions in heatmaps and SHAP waterfall plots.Examples include circular clock faces, rectangular signs or agricultural parcels, and complex galactic structures, without domain-specific retuning.Source: §4.3 · p. 7, Figure 4 · p. 8

4.5. User Study

• 89.7% of forward-simulation guesses were correct with MorphoSHAP, versus Grad-CAM at 91.7%, SHAP-BPT at 85.6%, and PartitionSHAP at 81.6%; median decision time was 6.78 seconds, fastest among evaluated methods.Source: §4.5 · p. 8
• Participants preferred MorphoSHAP in 60.1% of trials, versus SHAP-BPT at 24.6% and Grad-CAM at 15.2%.The authors relate these preferences to discrete, mathematically grounded geometric primitives.Source: §4.5 · p. 8

• MorphoSHAP shifts the Shapley game from pixel or superpixel grids to morphological primitives extracted with the Tree of Shapes.The framework is model-agnostic and post-hoc, and represents spatial location, morphological scale, and geometric category.Source: §5 · p. 8
• Across diverse image domains, the authors report superior insertion and deletion performance versus existing perturbation methods and stronger alignment with human interpretability.Source: §5 · p. 8
• MorphoSHAP may be less effective for predictions relying mainly on fine textures, and explanation quality depends on the shape-tagging function.Source: §5 · p. 8

A.1. Multi-Scale Morphological Characterization

• Relative-area thresholds assign extracted shapes to eight shared scale levels, from fine to coarse, without changing or filtering the Tree-of-Shapes decomposition.Using relative rather than raw area makes the scale definition independent of image resolution.Source: §A.1 · p. 11

A.2. Rule-based geometric naming

• The rule-based tagger applies ordered geometric rules, assigning Complex when a structure cannot be faithfully summarized by an elementary primitive.The categories are mutually exclusive because the rules are evaluated in a specified order.Source: §A.2 · pp. 11–12

A.3. Learned Geometric Naming (MLP Variant)

• On the holdout test set, the trained MLP achieved near-perfect accuracy (≈99%), demonstrating that its expanded descriptor provides a highly separable feature space for identifying the five target morphological categories.The MLP variant uses an expanded 24-dimensional geometric feature vector for each shape contour.Source: §A.3 · p. 12

A.4. Comparisons of the Two Namings

• Rule-based naming is transparent, zero-shot, and computationally inexpensive, while ShapeMLP can resolve ambiguous contours but requires labeled data and is tied to its training distribution.The main experiments use the learned variant; the paper compares both naming strategies on a real dataset in Table S14.Source: §A.4 · p. 12

A.5. Cross-Domain Global Semantic Insights

• MorphoSHAP reports 46.2% for Street Signs and 34.3% for Wall Clocks; Industrial Zones are driven by Complex Polygon (64.0%), while Spiral Galaxies show Rectangle (44.0%) and Complex Polygon (36.0%) reliance.Source: §A.5 · p. 13
• The shape vocabulary also supports custom semantic taggers, including domain-specific neural networks or pretrained vision-language models.The tagging interface maps an isolated binary blob to a semantic string.Source: §A.5 · p. 13

A.6. Textual explanation

• Textual explanations are generated deterministically from Shapley values, ranking positive shape categories and describing up to three contributors by size and geometry.The pipeline avoids external vision-language or large language models, keeping the generated explanations faithful to the underlying Shapley values.Source: §A.6 · p. 13

B. Details on the Evaluation Metrics

• Insertion AUC measures recovery of the predicted-class score as top-ranked explanatory units are restored from a reference image; higher values indicate faster recovery.Units are restored from most to least important.Source: B. Details on the Evaluation Metrics · p. 14
• Deletion AUC measures score decline as ranked units are removed from the original image; lower values are better.A faithful explanation identifies regions whose removal strongly affects the prediction.Source: B. Details on the Evaluation Metrics · p. 14
• For MorphoSHAP, insertion and deletion operate on ranked morphological shapes, replacing absent shapes with the same reference value used in the Shapley perturbation process.This evaluates faithfulness in the morphological space used to define the explanation.Source: B. Details on the Evaluation Metrics · p. 14
• Runtime is reported as average seconds per image, with lower values indicating greater computational efficiency.Source: B. Details on the Evaluation Metrics · p. 14

C.1. Detailed Quantitative Results per Architecture

• MorphoSHAP maintains consistent faithfulness across ResNet-50, ViT-B/16, and ConvNeXt-Tiny; Tables S5 and S6 compare methods using insertion and deletion AUC.The table captions state that higher Insertion AUC and lower Deletion AUC are better.Source: §C.1 · p. 14, Table S5 · p. 15, Table S6 · p. 15
• On ImageNet, MorphoSHAP is reported as significantly faster than KernelSHAP and SHAP-BPT across the three architectures, including Vision Transformers.Table S8 reports average runtime in seconds per image.Source: §C.1 · p. 14

C.2. Algorithm details

• The main experiments use eight relative-area scale thresholds, a learned or rule-based tagging function, a 50% overlap cutoff, and 1,024 sampled coalitions.Shapes overlapping previously registered blobs by more than 50% are discarded to constrain players and redundant attributions.Source: §C.2 · pp. 14–15

C.3. Ablation study

• Across KernelSHAP budgets from 32 to 4096 coalitions, Insertion and Deletion AUC effectively plateau after 1024 samples while computational runtimes continue scaling, validating baseline N = 1024 as a balance of attribution stability and speed.Source: §C.3 · p. 15
• At a 50% overlap threshold, the baseline balances distinct morphological players and runtime; 10% pruning runs in 1.86s but penalizes Deletion AUC, while 90% raises runtime to 7.59s.The 90% setting marginally improves Insertion AUC but causes player count and runtime to increase sharply.Source: §C.3 · p. 15, Table S10 · p. 17
• The 8-bin scale baseline provides robust multi-scale representations without the greater-than-2.5× runtime cost of log-spaced bins; 4 bins sharply reduce Insertion AUC.The comparison also included uniform 10-bin and 15-bin configurations.Source: §C.3 · pp. 15–16
• Across background strategies, the gray baseline scores 0.875 Insertion AUC, while image-global mean scores 0.882 and pure black scores 0.786.The black-baseline drop is attributed primarily to out-of-distribution artifacts in EuroSAT imagery.Source: §C.3 · p. 16
• On extracted composite components, the feature-learned MLP reaches 77.86% global accuracy, compared with 59.83% for handcrafted rules, despite extraction noise, discretization artifacts, and occlusion.Under isolated conditions, the MLP achieves 99.60% classification accuracy.Source: §C.3 · p. 16
• 74.44% overall primitive recovery, versus 57.00% for the rule-based heuristic, was achieved by the MLP tagger on 500 test images.Recovery was especially high for Triangles (86.6%) and Rectangles (85.3%).Source: §C.3 · pp. 16–17

D.1. Study Setup

• The study compares MORPHOSHAP, PartitionSHAP, Grad-CAM, and ShapBPT using heatmaps for 150 sampled ImageNet classes, with one random image per class.Specific class labels were paired with broad categories generated to help participants identify the objects without domain expertise.Source: §D.1 · p. 17

D.2. Participant Demographics

• The study reports 43 survey participants from 7 countries across 3 continents; separately, it collected responses from 42 participants from 7 unique countries, highlighting age, gender, education, and AI expertise.Figure S6 provides a comprehensive overview of participant demographics, highlighting diversity in age, gender, education, and AI expertise.Source: §D.1 · p. 17, §D.2 · p. 18

D.1. Study Setup

• Each participant evaluated 20 images, seeing exactly five heatmaps from each method in randomized order and completing classification and subjective-preference tasks.For classification, each image had five answer options, including “Not sure.”Source: §D.1 · pp. 17–18
• For classification, masked area was matched across methods by revealing the most important MORPHOSHAP blob or the same area of salient baseline pixels.This controlled the size of the revealed region in the comparison.Source: §D.1 · p. 18

D.3. Detailed Metrics, Statistical Analysis and Subgroup Analysis

• Figure S7 provides detailed method comparisons for accuracy, decision time, and preference, supplementing the overall comparison in the main text.The caption identifies confidence and trust scores as comparison measures.Source: §D.3 · p. 18, Figure S7 · p. 20
• MORPHOSHAP’s improvements hold across participant subgroups divided by ML expertise and educational background.Figures S8 and S9 show performance metrics categorized by ML expertise and education level, respectively.Source: §D.3 · p. 18, Figure S8 · p. 20, Figure S9 · p. 21
• Method had a significant effect on user accuracy (F = 4.4, p < 0.01), but not decision time (F = 0.4, p < 0.75); preference selections were non-uniform (χ2 = 552.4, p < 0.001) and favored MORPHOSHAP.The authors interpret the accuracy result as improved target-class identification without additional decision time compared with baselines.Source: §D.3 · p. 18