Source-linked AI summary
Ten Architectures, One Error: Shared Failure Modes in Hyperspectral Classification under Spatially Disjoint Evaluation
Ehsan Faghih, Fatemeh Ashrafi, Marguerite Moore, Zahra Saki
TL;DR
Hyperspectral classification often evaluates models with random within-scene splits that place correlated training and test pixels close together. This paper introduces receptive-field-aware, leakage-free evaluation across ten architectures and finds lower Macro-F1, changed rankings, receptive-field constraints, and shared errors indicating unresolved spectral ambiguity.
Problem
Random within-scene pixel splits can place correlated test and training pixels adjacent, while prior leakage-aware studies left cross-architecture rankings under spatial separation untested.
Method
The paper evaluates ten architectures using a leakage-free protocol that links spatial separation to receptive field, with common subsets, matched training sizes, and equalized benchmark procedures.
Results
0.147 Macro-F1 is the average matched-size drop under leakage-free evaluation, while rankings shift by as many as five places across the ten-model benchmark.
Takeaways & Limitations
Shared errors across all ten architectures point to spectral ambiguity, while leakage-free evaluation limits which receptive-field configurations a benchmark can support.
Takeaways & Limitations
The Salinas training split cannot support HyperSIGMA’s required radius 16 or its published tokenization configuration.
Abstract
from arXiv · showhide
Hyperspectral image classification still relies heavily on random pixel splits within a single scene. The Salinas dataset, randomly split, is among the most widely used datasets for comparing different architectures. However, under a random split method, a large fraction of test pixels fall immediately adjacent to a training pixel, which inflates reported accuracy. This work introduces a leakage-free evaluation protocol linking spatial separation to the model's receptive field. Applying this protocol across ten different architectures, including classical, spectral, spectral-spatial, transformer, vision-backbone, and state-space families, shows that Macro-F1 drops by 0.147 on average and model rankings change by as many as five places. Furthermore, leakage-free evaluation limits which architectures can be tested on a given benchmark. Since each partition supports patches only within a finite radius, reporting this radius alongside the receptive field is essential for fair comparison. In addition, this study reveals that all ten architectures misclassify largely the same pixels, pointing to a spectral ambiguity in the data that none of them resolves.
1. Introduction
Hyperspectral classification commonly uses random within-scene pixel splits, but spatial autocorrelation and patch receptive fields can leak information between training and test data. This work proposes leakage-free, receptive-field-aware evaluation and finds substantial performance and ranking changes across ten architectures.
- Motivation: Random Salinas splits can produce near-ceiling results despite extremely limited labeled training data.Reported results include 99.05% OA with 0.5% training data and 99.31 ± 0.26% OA with 1% training data.
- Motivation: Adjacent pixels provide weak evidence of generalization because spatial autocorrelation makes nearby test and training samples non-independent.Patch-based models can additionally read pixels from the neighboring partition through their receptive fields.
- Research gap: Prior leakage-aware studies did not report achieved spatial separation or compare cross-generational architectures under equal tuning budgets.The survival of established rankings under spatially disjoint evaluation therefore remained untested.
- Contributions: The proposed protocol ties admissible spatial separation to each model’s receptive field and uses frozen, hash-verified splits with a common evaluation subset.The accompanying audit covers partition disjointness, receptive-field admissibility, mask-composition bias, and a pre-training spectral probe.
- Contributions: The benchmark evaluates ten architecture families with 90 hyperparameter trials per model and five random seeds.Equal tuning effort is intended to reduce ranking differences caused by unequal optimization effort.
- Findings: 0.147 Macro-F1 is the mean performance drop under matched training size, while rankings shift by as many as five places under spatially disjoint evaluation.SpectralFormer moves from third under random splitting to eighth under spatially disjoint evaluation.
2. Related work
Prior work established that random or boundary-adjacent evaluation can leak spatial information in hyperspectral and other dependent-data settings. Existing hyperspectral leakage-aware splits addressed overlap but left separation, boundary handling, and broad architectural comparison incompletely reported.
- Dependent-data evaluation: Ecology and geostatistics studies show that random folds underestimate predictive error when observations have spatial dependence.They recommend spatial blocks combined with buffers that discard samples near partition boundaries.
- Hyperspectral evaluation: Nalepa et al. released leakage-aware folds for Salinas, Pavia University, and Indian Pines and evaluated spectral and spectral-spatial networks.Their work demonstrated leakage in repeated random splits and provided a patch-based partitioning algorithm.
- Architecture landscape: Hyperspectral architectures have expanded from SVMs and random forests to CNNs, spectral transformers, vision backbones, and state-space models.These families differ in how they model spectral sequences and local spatial neighborhoods.
- Broader context: Evaluation revisions in other fields have shown that matched-condition comparisons can weaken previously reported advantages and expose duplicated test data.Examples include domain-generalization reruns and corrected CIFAR test sets.
- Open limitations: Existing hyperspectral leakage-aware splits prevent patch overlap but generally do not report their achieved spatial separation.One approach retains boundary pixels with zero-padding, so some test inputs contain incomplete neighborhoods.
3. Methodology
The methodology constructs a spatially disjoint Salinas benchmark and makes model admissibility depend on receptive-field support. It then evaluates architectures on a common subset while measuring masking, protocol, partition-size, and metric effects.
- Spatial partition: Protocol B uses class-aware contiguous chunks separated by an unassigned per-class buffer, while Protocol A randomly distributes pixels with matched raw partition sizes.Protocol B contains 4,465 training, 4,425 validation, and 41,542 test pixels, plus 3,697 buffer pixels.
- Spatial partition: 47.69% of Protocol A test pixels lie immediately adjacent to training pixels at Chebyshev distance 1, compared with 0.00% under Protocol B.The minimum training-test distance is 1 for Protocol A and 2 for Protocol B.
- Receptive-field admissibility: Leakage-free evaluation requires training-test separation d ≥2r + 1 when each consumed patch has radius r.Pixels at distance d share at least one input pixel if and only if d ≤2r.
- Receptive-field admissibility: The pre-registered rule fixes Reval = 2 because training availability and validation class support constrain the usable radius before model training.At radius 3, validation class 11 retains no pixels, while training retains at least half its pixels only through R = 3.
- Masking effects: Masking removes mostly interior pixels and produces a harder common test subset rather than an easier one.Only 1.1% of removed pixels lie on cross-class boundaries, while removed pixels score approximately six Macro-F1 points higher than retained pixels under spectral probes.
- Benchmark protocol: Ten architectures are compared using native or capped evaluated radii, with every model scored on identical Protocol B pixels and Macro-F1 as the primary metric.The study also reports four evaluation conditions, including matched-size Protocol A comparisons and seed-level means with standard deviations.
4. Results
Under leakage-free Protocol B, all architectures perform worse than under Protocol A, rankings shift substantially, and errors remain strongly shared across models. The protocol also exposes receptive-field constraints that prevent evaluating some large-context architectures on this split.
- 4.1. The cost of leakage-free evaluation: 0.147 Macro-F1 is the mean drop at matched training size under Protocol B, while the full-training comparison drops by 0.165.The 0.018 difference is attributed to the smaller training set.
- 4.2. Rankings do not survive the correction: Five places is the largest ranking shift: SpectralFormer falls from third under Protocol A to eighth under Protocol B, while ResNet1D rises from fifth to third.The matched comparison shows the same pattern, with Spearman correlation ρ = 0.77; full-size Protocol A versus Protocol B has ρ = 0.78.
- 4.5. Supporting analyses: 0.173 Macro-F1 separates the best and worst models under Protocol B, compared with 0.0157 average variation across seeds.The reported gap is eleven times larger than seed noise, so the observed differences mainly reflect architectural differences.
- 4.3. Architectures diverge, yet fail on the same pixels: 0.471 is the mean pairwise overlap of misclassified pixels, increasing to 0.537 among the four strongest models; 30% of pixels missed by any model are missed by at least seven.On seed 37, 1,779 pixels are misclassified by all ten models, with 83% involving class 8 predicted as class 15.
- 4.4. The protocol bounds receptive field: Radius 16 is required by HyperSIGMA, but this split cannot support it; the published tokenization also cannot use the 5×5 window allowed by Reval = 2.The partition geometry, rather than the model, limits available receptive field, creating tension with large-context architectures on Salinas.
- 4.5. Supporting analyses: pHolm ≈10^-140 can indicate a statistically significant difference without a practically meaningful Macro-F1 gap: ConvNeXt2D and S2Mamba differ by only 0.0007.Their odds ratio is 2.30 over 4,142 discordant pixels, yet the difference lies within either model’s seed variation.
- 4.5. Supporting analyses: A factor-of-two parameter difference and expected calibration error spanning 0.022 to 0.234 do not predict Protocol B accuracy.The best-calibrated neural model ranks fifth on Macro-F1, and the largest benchmark model is neither most nor least accurate.
5. Discussion
The discussion separates evaluation-induced difficulty from intrinsic spectral ambiguity, while showing that spatially disjoint evaluation imposes geometric limits on supported architectures and requires reporting receptive-field context.
- Evaluation effects: 0.147 Macro-F1 is the mean performance gap between random and spatially disjoint evaluation at matched training size.A raw-spectra nearest-neighbor classifier also scores 0.110 higher under random splitting, before architecture enters the comparison.
- Data ambiguity: The ten architectures misclassify largely the same pixels, including 1,156 errors shared across every architecture and seed.Classes 8 and 15 are never spatially adjacent within the benchmark patches, so the shared errors point to spectral ambiguity rather than partition artifacts.
- Scope: The experiments use one scene and one frozen partition, while ConvNeXt2D and S2Mamba were evaluated under smaller receptive-field-constrained inputs than their published windows.These conditions limit how broadly the reported effect sizes and top-model results should be generalized.
- Reporting requirements: Spatial separation should be reported with model receptive field because leakage-free admissibility depends on both quantities.The stated condition is reval ≤ (d − 1)/2 for a split with minimum separation d.
- Benchmark limits: Leakage-free evaluation imposes a geometric ceiling on which architectures a benchmark can support, independent of model design.Under Protocol B, the training partition retains half its pixels only through radius 3 and admits no patch beyond radius 12.
- Benchmark limits: The audit reports which architectures a benchmark can host before model selection, identifying when a different split or larger scene is required.The procedure uses split distances and each model’s patch size to expose these constraints.
6. Conclusion
The conclusion finds that Salinas results depend strongly on whether spatial adjacency leaks between training and test pixels. Leakage-free evaluation lowers performance, changes rankings, and exposes shared errors that none of the ten architectures resolves.
- Evaluation protocol: 47.69% of random-split test pixels lie within one pixel of training pixels, compared with none under the leakage-free partition.Removing this adjacency lowers nearest-neighbor Macro-F1 by 0.110 and lowers the ten-model average by 0.147 at matched training size.
- Evaluation protocol: Model rankings shift by as many as five places, with a tuned RBF-SVM ranking above six of the nine other benchmark models.The conclusion therefore treats a single accuracy figure as insufficient to characterize Salinas performance.
- Shared errors: Every architecture misclassifies the same 1,156 pixels across all seeds, mostly through one directional confusion between two nonadjacent classes.The shared errors point to spectral ambiguity rather than a split artifact, and none of the ten architectures resolves it.
- Scope and reuse: The exact drops are specific to Salinas, but the broader issue applies to spatially structured scenes with receptive-field ceilings under leakage-free splitting.The audit requires only split distances and model patch sizes, and the released protocol supports future comparisons on common pixels.