PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral-Spatial Interpretation

Haotian Ma, Ruqi Yang, Philip Townsend

arXiv:2610.21114v1cs.LGcs.CV

TL;DR

Spatial heatmaps and global channel scores do not show which spectral channels support a prediction at each location, even though channel relevance can vary spatially. SSI computes observed-context Shapley attributions for channels at each position. It achieves strong joint attribution results across three datasets and outperforms MC-Owen on HLS Burn Scar using about 6% of its evaluation budget.

  • Problem

    Spatial heatmaps aggregate over channels and global channel scores average over space, leaving position-specific spectral evidence unrepresented.

    Source: §1 · p. 1

  • Method

    SSI fixes each position’s observed surrounding context and applies Shapley allocation among its channels, producing a full position-channel attribution matrix.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 5

  • Results

    Across CUB-200, EuroSAT, and HLS Burn Scar, SSI improves joint faithfulness, spectral hit rates, and channel deletion over dense, flat, and rank-1 baselines, and beats MC-Owen on HLS Burn Scar using roughly 6% of its evaluation budget.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    SSI’s channel attributions at each position aggregate exactly to the contextual contrast from replacing that position with the baseline.

    Source: §1 · p. 2, §5 · p. 10

    For hyperspectral sensors with hundreds of correlated bands, channel screening or adaptive grouping may help reduce redundancy; validating these extensions on operational missions remains future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Post-hoc image explanations usually show where a model looks using spatial heatmaps or indicate which channels matter using global channel scores. These separate views are limiting for multispectral data because channel relevance is often spatially localized and physically meaningful. We introduce Spectral-Spatial Interpretation (SSI), a full-grid attribution method over position-channel players (g,c). For each spatial position, SSI fixes the observed surrounding context, defines a conditional game over the channels at that position, and applies the Shapley value to this local game. The result is a full spectral-spatial attribution matrix rather than a single heatmap or a global channel ranking. SSI has a clear local interpretation. It satisfies Shapley fairness within each position, and aggregating its channel attributions exactly recovers the position-level contextual contrast obtained by replacing that position with the baseline. SSI also differs from context-averaged Owen-style hierarchical accounting, which uses the same position-channel grouping but averages over external spatial coalitions. On HLS Burn Scar, SSI outperforms an MC-Owen estimator across retained attribution metrics while using only about 6% of its evaluation budget. Across CUB-200, EuroSAT, and HLS Burn Scar under both per-channel mean and zero baselines, SSI achieves the best joint deletion and insertion AUC over dense, flat, and rank-1 baselines, yields the strongest spectral hit rates and channel deletion, and leads spatial localization on both annotated datasets. Code is available at https://github.com/AtheavenMa/Spectral-Spatial-Interpretation.

1 Introduction

Multispectral explanations need to show which channels support a prediction at each location, because channel relevance can vary across space. SSI assigns position-channel attributions and reports strong results against attribution baselines across three datasets.

1 Introduction

  • Existing spatial explanations aggregate over channels, while global channel scores average relevance over space, obscuring localized spectral evidence.A spatial-by-channel product further assumes the same channel profile applies everywhere, although different regions can have different spectral signatures.Source: §1 · pp. 1–2
  • Figure 1 illustrates spatially distinct channel evidence, with bird plumage colors and Sentinel-2 land-cover types aligned to different channels.SSI is designed to recover this joint spectral-spatial structure.Source: Figure 1 · p. 2
  • SSI assigns an attribution to each position-channel pair by fixing the observed surrounding context and applying Shapley allocation among channels at that position.Its output is a full joint attribution matrix rather than a single heatmap or global channel ranking.Source: §1 · p. 2, Figure 2 · p. 3
  • At each position, SSI’s channel attributions aggregate to the contextual contrast from replacing that position with the baseline.The method characterizes this as contextual aggregation consistency, alongside local Shapley fairness.Source: §1 · p. 2
  • On HLS Burn Scar, SSI outperforms an MC-Owen estimator while using about 6% of its evaluation budget.The evaluation spans CUB-200, EuroSAT, and HLS Burn Scar.Source: §1 · p. 2
  • Across CUB-200, EuroSAT, and HLS Burn Scar under both per-channel mean and zero baselines, SSI achieves the best joint deletion, rank-1 deletion, and grid-sum insertion AUC over dense, flat, and rank-1 baselines.It also leads spectral hit rates and channel deletion on EuroSAT and HLS Burn Scar, and spatial localization on both annotated datasets.Source: §1 · p. 2

2 Related Work

Prior explanations separately emphasize spatial regions, global channel relevance, or joint prediction architectures, leaving position-specific spectral attribution unaddressed. SSI instead provides game-theoretic attribution over position-channel pairs with observed context.

Post-hoc spatial attribution

  • Spatial attribution methods identify relevant regions but aggregate over channels or treat input elements independently, so they do not decompose which spectral channels matter at each location.Examples include gradient-based methods such as Integrated Gradients and Grad-CAM, and perturbation-based methods such as RISE and Extremal Perturbation.Source: §2 · p. 3

Spectral and channel attribution

  • Global spectral and channel methods average relevance over space, which can hide a band that is decisive in a small region.SSI instead assigns attribution to each position-channel pair.Source: §2 · p. 3

Joint spectral-spatial modeling

  • Joint spectral-spatial prediction architectures focus on predictive performance, while their internal representations remain opaque; some propagation-based explanations are architecture-specific.SSI provides model-agnostic, game-theoretic attribution for already-trained models.Source: §2 · p. 3

Game-theoretic and hierarchical attribution

  • Flat-player Shapley methods, interaction indices, group-level methods, and feature-dependence methods differ from SSI in attribution units or treatment of context.SSI uses position-channel groups but targets observed-context conditional attribution rather than context-averaged hierarchical accounting.Source: §2 · p. 3

3 Method

SSI applies Shapley attribution to the channels at each position while fixing the observed surrounding image. It provides local fairness and exact position-level aggregation, targeting observed-context attribution rather than context-averaged Owen accounting.

3.1 Spectral-Spatial Players

  • The player space consists of position-channel pairs; channels at one position measure the same physical region, and adding a player may increase or decrease the target logit.Source: §3.1 · p. 4

3.2 Conditional Position Games

  • The conditional position game varies only the channels at one position while holding all other positions fixed at their observed values.It is the local channel-allocation problem induced by the observed surrounding context.Source: §3.2 · p. 4

3.3 Spectral-Spatial Interpretation

  • SSI applies standard Shapley values to each conditional position game, producing a full grid-by-channel attribution matrix for the observed context.Unlike Owen-style accounting, SSI does not average a channel’s contribution over possible external spatial coalitions.Source: §3.3 · pp. 4–5

3.4 Properties

  • Within each position, SSI is the unique channel attribution satisfying local efficiency, symmetry, null-player, and additivity for the conditional game.This is the standard Shapley characterization applied locally after conditioning on observed context.Source: §3.4 · p. 5
  • SSI’s channel-aggregated attribution exactly equals the full-context contrast from replacing that position with the baseline.This follows from local Shapley efficiency and does not require monotonicity, supermodularity, or an architectural assumption.Source: §3.4 · p. 5
  • SSI gives position-wise conditional explanations, but does not aim to allocate the entire prediction value globally across positions.The full-context contrasts for different positions need not sum to the total value in interacting games.Source: §3.4 · p. 5

3.5 Computation, Outputs, and Relation to Owen-Style Accounting

  • SSI reports every position-channel score and samples channels within each position, while MC-Owen samples full block-respecting permutations and includes variance from external spatial contexts.For C=3, SSI enumerates all channel subsets exactly.Source: §3.5 · pp. 5–6

4 Experiments

The experiments evaluate SSI on RGB, multispectral land-cover, and burn-scar data using joint position-channel faithfulness, spatial and spectral aggregation, qualitative examples, and comparisons with hierarchical and gradient-based methods. Across datasets and baselines, SSI performs strongly on joint and aggregated metrics, and on HLS Burn Scar it outperforms MC-Owen with about 6% of its evaluation budget.

4.1 Experimental Setup

  • The evaluation spans CUB-200 RGB classification, EuroSAT 13-band land-cover classification, and HLS Burn Scar segmentation, using per-channel mean and zero masking baselines.The datasets use 16 × 16 grids for CUB-200 and 8 × 8 grids for EuroSAT and HLS Burn Scar.Source: §4.1 · p. 6

4.2 Joint Faithfulness

  • On HLS Burn Scar, SSI achieves the highest LeRF AUC (.657) and lowest MoRF AUC (.1143) among nine methods under ROAD.Lower deletion AUC indicates that the attribution ranking removes full-context evidence earlier.Source: §4.2 · p. 7
  • The rank-1 diagnostic shows factorized explanations perform substantially worse than SSI, with the largest joint-to-rank-1 deletion gap on 13-channel EuroSAT.The gap is smallest on CUB-200, which has three channels.Source: §4.2 · p. 7
  • Across three datasets and both mean and zero baselines, SSI achieves the lowest joint deletion AUC and highest insertion AUC against dense, flat, and rank-1 baselines.On CUB-200 under the mean baseline, insertion AUC is .7132 for SSI versus .6886 for RISE.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Spatial and Spectral Aggregation

  • Across the annotated datasets, SSI leads spatial localization; on HLS Burn Scar it reaches .389 mIoU at Top-10%, versus .277 for DeepSHAP and .191 for RISE.SSI also leads spectral Hit@K and has the lowest channel deletion AUC on EuroSAT and HLS Burn Scar.Source: §4.3 · p. 8, Figure 3 · p. 9

4.4 Qualitative Case Studies

  • Qualitative cases show localized channel evidence: RGB attributions differ across bird regions, burn-scar attributions emphasize Red and SWIR, and EuroSAT vegetation attributions favor red-edge and NIR over red.The EuroSAT pattern matches the cited characterization of vegetation reflectance [Tucker, 1979, Clevers et al., 2002].Source: §4.4 · p. 8

4.5 Observed Context versus Context-Averaged Hierarchical Attribution

  • On HLS Burn Scar, SSI outperforms MC-Owen on all retained metrics while using roughly 6% of its evaluation budget.The comparison covers joint and rank-1 deletion, channel deletion, grid-sum insertion, spatial localization, and spectral hit rates.Source: §4.5 · p. 9, Table 2 · p. 10

4.6 Additional Gradient-Based Baselines

  • On HLS Burn Scar, SSI performs best on all ten reported metrics against Expected Gradients, Guided IG, and FullGrad.The comparison uses the same position-channel player space.Source: Table 3 · p. 10, §4.6 · p. 10

5 Conclusion

SSI attributes predictions to position-channel pairs by fixing each position’s observed surrounding context and applying Shapley allocation among its channels. Across three datasets, it improves on several attribution baselines, while its use on hyperspectral data with hundreds of correlated bands may benefit from channel screening or adaptive grouping.

5 Conclusion

  • Across CUB-200, EuroSAT, and HLS Burn Scar, SSI improves joint faithfulness, spectral hit rates, and channel deletion over dense, flat, and rank-1 baselines.On HLS Burn Scar, it also outperforms MC-Owen with roughly 6% of its evaluation budget.Source: §5 · p. 10
  • At each position, SSI fixes the observed surrounding context and uses Shapley values to allocate the position’s contribution among its channels.The resulting attribution matrix aggregates across channels to recover the position-level contextual contrast.Source: §5 · p. 10
  • For hyperspectral sensors with hundreds of correlated bands, channel screening or adaptive grouping may help reduce spectral redundancy before local Shapley allocation.Validation of these extensions on operational hyperspectral missions remains future work.Source: §5 · p. 10

A Additional Method Details

The appendix records proofs and sampling details for SSI’s local Shapley characterization and related properties. At a fixed position, SSI treats the conditional game as an ordinary cooperative game over that position’s channels.

A Additional Method Details

  • The appendix covers proofs of local Shapley characterization, contextual aggregation consistency, and the conflict between contextual aggregation and global efficiency, alongside sampling details.It also discusses flat Shapley and context-averaged Owen-style accounting.Source: A Additional Method Details · p. 14

A.1 Local Shapley Axioms

  • SSI applies the standard Shapley value to the local channel game at each position.The relevant axioms are local rather than global.Source: §A.1 · p. 14
  • For each fixed position, the conditional game is an ordinary cooperative game over its finite channel set, so the standard Shapley characterization gives the unique attribution satisfying efficiency, symmetry, null-player, and additivity.The axioms apply locally to that position’s game.Source: §A.1 · p. 14

A.2 Proof of Contextual Aggregation Consistency

The appendix relates contextual aggregation to local Shapley efficiency, shows that contextual aggregation and global efficiency cannot both be guaranteed for all games, and describes permutation sampling for estimating SSI.

A.2 Proof of Contextual Aggregation Consistency

  • The contextual-aggregation proof invokes local Shapley efficiency in the conditional game at each position.The supplied proof passage identifies this as its step toward the consistency result.Source: §A.2 · p. 14

A.3 Contextual Aggregation Versus Global Efficiency

  • In general, no attribution rule can guarantee both contextual aggregation and global efficiency for every game.A two-position, one-channel counterexample has position contrasts summing to 2 while the full-game value is 1.Source: §A.3 · p. 15

A.4 Sampling Estimator and Concentration

  • For larger channel sets, SSI estimates local Shapley values by sampling channel permutations; the estimate is unbiased, with a concentration bound across position-channel players.The bound does not imply that one fixed small K suffices for every possible game.Source: §A.4 · pp. 15–16

A.5 Empirical Instantiation of the Concentration Bound

The appendix reports empirical sampling bounds, explains how flat Shapley can fail to preserve position-level sums, distinguishes SSI from context-averaged accounting, and describes a region-restricted option.

A.5 Empirical Instantiation of the Concentration Bound

  • At K=10, the reported uniform error bound is approximately 0.0067 for EuroSAT’s 832 players and 0.0043 for HLS Burn Scar’s 384 players; both bounds are below the inter-method gaps reported in Tables 1 and 2.The empirical summary states that K=10 provides sufficient estimation precision for the reported comparisons.Source: §A.5 · p. 16

A.6 Relation to Flat Shapley

  • Flat Shapley does not guarantee position-level hierarchical aggregability: summed position attribution can change even when the quotient game over positions is unchanged.The result is attributed to flat Shapley’s allocation over position-channel players.Source: §A.6 · pp. 16–17

A.7 Relation to Context-Averaged Hierarchical Accounting

  • SSI fixes the observed external context, whereas Owen-style accounting averages over external spatial contexts; SSI therefore estimates a different quantity.The MC-Owen sampling comparison includes variance from external spatial contexts, which is absent when SSI fixes that context.Source: §A.7 · pp. 17–18

A.8 Optional Region-Restricted SSI

  • Restricting SSI to a user-specified set of positions changes the evaluated support but leaves each position’s conditional game unchanged.All positions outside the evaluated position remain fixed as observed context.Source: §A.8 · p. 18

B.1 Datasets, Models, and Preprocessing

The experiments use three image datasets with dataset-specific models and response definitions. Attribution baselines are converted to the shared position-channel player space, while the RISE control factorizes spatial and global channel scores.

B.1 Datasets, Models, and Preprocessing

  • CUB-200 uses an ImageNet-pretrained ResNet-18 and ground-truth class logit, EuroSAT uses a 13-band ResNet-18 trained from scratch, and HLS uses a lightweight UNet with region-averaged burn-class logit.CUB-200 contains RGB bird images, EuroSAT contains Sentinel-2 land-cover patches, and HLS Burn Scar contains six spectral bands.Source: §B.1 · p. 18

B.2 Attribution Baseline Details

  • Integrated Gradients and DeepSHAP dense attributions are summed within each grid cell and channel; KernelSHAP directly treats each position-channel pair as a feature.KernelSHAP uses the same grid as SSI but does not impose its position-level coalition structure.Source: §B.2 · p. 19
  • The RISE rank-1 control multiplies a spatial importance map by a global channel score, so channel relevance cannot vary by position.This comparison isolates the full joint attribution matrix from a factorized spatial-spectral summary.Source: §B.2 · p. 19

B.3 Computational Cost

SSI’s evaluation cost is counted in model forward passes and grows with the number of grid cells and local channels. The section also describes its scaling and the attribution metrics used for evaluation.

B.3 Computational Cost

  • 4,480 vs. 76,800 evaluations: SSI uses about 6% of MC-Owen’s HLS Burn Scar budget under the conservative forward-pass counting convention.These are analytical counts, not measured wall-clock times; forward-pass counts are not directly comparable with gradient methods’ backward-pass costs.Source: Table 4 · p. 20, §B.3 · p. 20

B.4 Scalability to Higher Channel Counts

  • For C ≥4, SSI costs G · K(C+1) forward passes per image with K capped at 10, rather than traversing all |P|=GC players per permutation.For larger channel counts, K can rise at O(ln C); the example increases from about 10 permutations at C=13 to about 21 at C=200.Source: §B.4 · p. 20

C.1 Metric Definitions

  • Joint deletion masks position-channel players in descending signed attribution order; lower deletion AUC and higher insertion AUC indicate better results.Rank-1 projection evaluates a best rank-1 approximation of the attribution matrix.Source: §C.1 · pp. 20–21
  • Spatial localization sums attribution over channels and compares thresholded maps with spatial annotations, while spectral evaluation sums over positions and tests channel rankings.Spatial maps use mIoU; channel deletion masks whole channels and spectral hit rates compare top-ranked channels with dataset-specific priors.Source: §C.1 · p. 21

C.2 ROAD Faithfulness Evaluation

The section evaluates faithfulness, physical spectral priors, grid-resolution robustness, spatial localization, and spectral ranking. SSI performs strongly on the reported evaluations, with spatial localization unavailable for EuroSAT because it lacks a discriminative proxy.

C.2 ROAD Faithfulness Evaluation

  • SSI performs best in both ROAD removal directions among nine methods on HLS Burn Scar.ROAD reports MoRF AUC, where lower is better, and LeRF AUC, where higher is better; it is intended to mitigate some feature-removal artifacts, not eliminate every distribution-shift effect.Source: Table 5 · p. 21, §C.2 · p. 21

C.3 Spectral Band Priors

  • Spectral hit-rate priors include only bands with documented, physically grounded positive associations; classes without a clear signature are excluded from hit-rate evaluation.For HLS burn scars, Red, SWIR1, and SWIR2 are positive priors, while NIR is excluded because it is negatively correlated with the class.Source: §C.3 · pp. 21–22

C.4 Grid Resolution

  • SSI maintains leading faithfulness performance across the reported grid resolutions, suggesting robustness to grid-size choice.The evaluation varies grids from 4×4 to 8×8 where applicable.Source: §C.4 · p. 22, Table 6 · p. 22, Table 7 · p. 23

C.5 Full Spatial Localization Curves

  • SSI leads spatial mIoU at all reported coverage thresholds on CUB-200 and HLS Burn Scar; EuroSAT has no spatially discriminative ground-truth proxy.On CUB-200, IG and DeepSHAP average mIoU scores are 3 to 4 points lower, while KernelSHAP falls substantially behind.Source: §C.5 · pp. 22–23, Table 10 · p. 23, Table 11 · p. 23

C.6 Full Spectral Hit Curves

  • On HLS Burn Scar, SSI leads channel deletion and reaches a spectral hit rate of .8867 at K=5; on EuroSAT, it has the best channel deletion and leads at every hit-rate threshold.KernelSHAP is competitive at higher hit-rate thresholds on EuroSAT.Source: §C.6 · pp. 23–24, Table 12 · p. 24, Table 13 · p. 24
Loading 2610.21114v1…