PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperSAM: A Promptable Foundation Model for Hyperspectral Remote Sensing

Li Pang, Xinqiao Wu, Jing Yao, Pedram Ghamisi, Jun Zhou, Zhengchao Chen, Deyu Meng, Xiangyong Cao

arXiv:2609.37340v1cs.CV

TL;DR

Hyperspectral foundation models face limited high-resolution, reliably annotated data and often underuse vision-foundation-model priors. HyperSAM combines synthetic object-centric spectral-mask data with SAM3-based spectral adaptation and confidence-aware learning. The paper reports strong cross-task generalization and concludes that higher-quality synthetic data can be more effective than simply scaling noisy supervision.

  • Problem

    Hyperspectral foundation models face limited high-spatial-resolution data with reliable dense annotations, while many models underuse the geometric priors of modern vision foundation models.

    Source: §I · pp. 1–2

  • Method

    HyperSAM synthesizes full-spectrum cubes from high-resolution multispectral imagery, pairs them with SAM3 pseudo-masks, and adapts SAM3 using a spectral branch and confidence-aware learning.

    Source: §I · pp. 2–3

  • Results

    HyperSAM shows strong cross-task performance, with the strongest gains where object grouping and spectral-spatial features are useful; target detection remains more scene-dependent.

    Source: 3) Cross-Task Summary · p. 14

  • Takeaways & Limitations

    The paper concludes that a compact, higher-quality synthetic corpus can be more effective than a larger weakly object-centric corpus and reports transfer to airborne oil-spill mapping.

    Source: §V · p. 16

Abstract

from arXiv · show

Hyperspectral remote sensing provides dense spectral measurements that are indispensable for material-level Earth observation, yet the construction of a general-purpose hyperspectral foundation model remains difficult. Two bottlenecks are especially limiting. First, large hyperspectral corpora rarely provide high spatial resolution together with reliable dense annotations. Second, many hyperspectral models are still trained almost from scratch, so the geometric and interactive priors learned by modern vision foundation models are not fully reused. To alleviate these issues, we \highlight{present} \textbf{HyperSAM}, a promptable hyperspectral foundation model that couples a data-centric hyperspectral synthesis pipeline with a spectral adaptation architecture based on Segment Anything Model 3 (SAM3). On the data side, HyperSAM synthesizes full-spectrum hyperspectral cubes from high-resolution SpaceNet multispectral imagery through a physics-informed abundance-transfer generator, while SAM3-derived pseudo-masks provide object-centric supervision. On the model side, the latest implementation uses a frozen SAM3 RGB image branch, a trainable hyperspectral side encoder initialized from the RGB vision transformer (ViT), ControlNet-style zero-initialized feature injection, and a lightweight mixture-of-experts mask refiner. To enhance training robustness against noisy pseudo-labels, Cross-modal Sample Selection (CromSS)-style confidence selection is incorporated for noisy-label weighting. Extensive experiments show that HyperSAM obtains strong generalization on diverse hyperspectral tasks (e.g., classification, anomaly detection, change detection, target detection, and airborne oil-spill mapping) and that high-quality synthetic hyperspectral data can be more effective than simply scaling noisy hyperspectral supervision.

I. INTRODUCTION

Hyperspectral foundation models face gaps in data quality and reuse of vision-foundation-model priors. HyperSAM addresses these gaps with synthetic spectral-mask training data, SAM3-based spectral adaptation, and confidence-aware learning.

I. INTRODUCTION

  • Hyperspectral data are high-dimensional, sensor-dependent, costly to annotate, and often modest in spatial resolution, limiting transfer across scenes and tasks.Existing corpora also tend to lack the combination of high spatial detail and reliable object-level labels needed for promptable training.Source: §I · pp. 1–2
  • Many hyperspectral models train new backbones, underusing SAM-style spatial priors relevant to coherent objects, boundaries, and regions.The paper frames the challenge as injecting spectral evidence without destroying those priors.Source: §I · pp. 1–2
  • HyperSAM synthesizes full-spectrum cubes from high-resolution multispectral imagery and pairs them with SAM3-derived pseudo-masks for object-centric supervision.The paper presents this as a data-centric response to the shortage of sharp, densely annotated hyperspectral training data.Source: §I · pp. 2–3, Figure 2 · p. 2
  • A frozen RGB branch preserves SAM3 geometric and prompt-response priors while a trainable spectral branch injects full-spectrum corrections through zero-initialized residual adapters.A lightweight mixture-of-experts decoder refines masks across object scales and residual label uncertainty.Source: §I · pp. 2–3
  • Confidence-aware training emphasizes reliable pseudo-mask regions and softly down-weights uncertain ones, using agreement across spectral views and branches.Hyperspectral-specific augmentations target geometric perturbations, spectral-response variation, and band-wise degradation.Source: §I · pp. 2–3
  • HyperSAM demonstrates cross-task effectiveness on classification, anomaly detection, change detection, and target detection with a single frozen checkpoint.The paper describes this as a unified prompt-mask-feature inference paradigm.Source: §I · p. 3

II. RELATED WORK

The related work distinguishes representation-learning models from promptable approaches and motivates HyperSAM’s combination of synthetic object-centric data, SAM3 priors, and robust learning. The method constructs physically constrained full-spectrum training samples paired with object masks.

A. Remote Sensing and Hyperspectral Foundation Models

  • Representation-learning models commonly transfer pretrained encoders through task-specific heads, probes, or fine-tuning, while promptable models offer mask or feature use without inference-time backbone updates.HyperFree is identified as the closest prior promptable hyperspectral method; HyperSAM differs in data, supervision, and spectral adaptation.Source: A. Remote Sensing and Hyperspectral… · p. 3

B. Promptable Segmentation for Remote Sensing

  • Remote-sensing promptable segmentation methods mainly target RGB or multispectral imagery; HyperSAM preserves SAM3’s geometric and prompt-response priors while adding full-spectrum evidence.The comparison includes RSPrompter, PointSAM, AnyChange, and RemoteSAM.Source: B. Promptable Segmentation for Remote… · p. 3

C. Synthetic Hyperspectral Data and Noisy-Label Learning

  • Hyperspectral training data can lack dense annotation, and pseudo-masks can be noisy; HyperSAM adapts confidence-based selection to softly weight reliable supervision.It uses spectral-view and branch-level agreement to form soft, class-balanced confidence weights.Source: C. Synthetic Hyperspectral Data and… · pp. 3–4, Figure 3 · p. 4

III. METHOD OVERVIEW

  • HyperSAM couples synthetic hyperspectral data construction, a frozen-RGB/trainable-spectral SAM3 framework, and robust learning to predict task-agnostic masks, quality scores, and transferable features.Hyperspectral-specific augmentations and confidence-aware weighting address imperfect pseudo-masks.Source: §III · p. 4

A. High-Fidelity Hyperspectral Dataset Construction

  • The corpus uses SpaceNet RGB and eight-band WorldView-3 imagery from four urban regions, with separate roles for visible-band mask extraction and full-channel spectral reconstruction.It contains 10,592 training patches and 3,527 validation patches; the paper reports clearer boundaries and richer semantic entities than existing pseudolabeled hyperspectral corpora.Source: A. High-Fidelity Hyperspectral Dataset… · p. 4
  • A physics-informed abundance-transfer process estimates shared material abundances and illumination, then transfers them through paired multispectral and hyperspectral endmember libraries to generate full-spectrum cubes.The AVIRIS library spans approximately 400–2500 nm, and the paired library contains 772 materials.Source: A. High-Fidelity Hyperspectral Dataset… · pp. 4–5
  • Synthesis is constrained by multispectral fidelity, spectral-shape and projection consistency, abundance sparsity, and spatial smoothness; sampled synthetic spectra overlap real AVIRIS spectra in a shared PCA manifold.The resulting corpus pairs physically constrained spectra with high-resolution object masks for promptable mask learning.Source: A. High-Fidelity Hyperspectral Dataset… · p. 5

B. HyperSAM Model Architecture

HyperSAM preserves SAM3’s frozen promptable RGB pathway while adding a trainable full-spectrum branch and lightweight MoE refinement. Zero-initialized adapters inject spectral residuals while retaining the inherited starting behavior.

B. HyperSAM Model Architecture

  • A frozen SAM3 pathway and trainable hyperspectral side branch combine inherited visual priors with full-spectrum material information.The side branch uses spectral-spatial patch embedding and spectral-shape extraction to align its tokens with SAM3 visual features.Source: B. HyperSAM Model Architecture · pp. 5–6
  • Zero-initialized residual adapters inject spectral features while initially keeping the adapted model equivalent to the original SAM3 RGB branch.Spectral corrections are learned during adaptation, reducing the risk of destroying inherited priors.Source: B. HyperSAM Model Architecture · pp. 5–6
  • The prompt-driven decoder accepts point, box, or mask prompts and uses a lightweight MoE refiner to adapt mask logits and quality scores.Three convolutional experts with different receptive fields predict residual mask logits, and an expert-balance loss discourages router collapse.Source: B. HyperSAM Model Architecture · p. 6
  • A single prompt-mask-feature interface returns task-agnostic masks, quality scores, and dense embeddings for classification, anomaly detection, change detection, and target detection.The embeddings support similarity matching, prototype construction, and temporal comparison.Source: B. HyperSAM Model Architecture · p. 6

C. Robust Learning With Confidence-Aware Noisy-Label Weighting

HyperSAM treats SAM3-derived masks as noisy supervision and uses confidence-aware weighting to reduce the influence of uncertain pixels. Two spectral views and class-balanced confidence selection help retain reliable supervision while limiting the effect of ambiguous regions.

C. Robust Learning With Confidence-Aware Noisy-Label Weighting

  • Ambiguous boundaries, incomplete objects, fragments, and RGB/material-boundary mismatches make SAM3 masks noisy supervision rather than hyperspectral ground truth.The mismatches can occur around shadows, thin structures, mixed pixels, roof–background transitions, and spectrally distinct but visually similar surfaces.Source: C. Robust Learning With… · p. 6
  • Confidence-aware training locally downweights ambiguous boundaries and background leakage while retaining supervision from reliable regions.The weight maps assign larger weights to stable regions and smaller weights to uncertain pseudo-label pixels.Source: C. Robust Learning With… · pp. 7–8, Figure 8 · p. 8
  • CromSS-style weighting gives larger weights to regions confident across spectral views and branches, while softly down-weighting ambiguous regions with class-balanced selection.Class balancing helps prevent frequent background regions or easy objects from dominating supervision.Source: C. Robust Learning With… · p. 7
  • The selected candidate mask minimizes confidence-weighted BCE-plus-Dice loss, while consistency and expert-balance terms supplement segmentation loss.The consistency term improves invariance across spectral windows; the balance term prevents expert collapse.Source: C. Robust Learning With… · pp. 7–8
  • Training samples two overlapping spectral windows from one cube, preserving the spatial target while varying spectral response for cross-view consistency.The windows share at least 32 bands, and a symmetric KL term encourages consistent binary mask distributions.Source: C. Robust Learning With… · p. 8

IV. EXPERIMENTS

The experiments assess HyperSAM’s shared promptable model across four hyperspectral interpretation tasks. They also examine the roles of synthetic data, spectral adaptation, MoE refinement, and confidence-aware weighting.

IV. EXPERIMENTS

  • The evaluation covers hyperspectral classification, anomaly detection, change detection, and target detection using the same trained checkpoint, with no task-specific downstream network.The main tables assess cross-task generalization under the shared checkpoint.Source: §IV · p. 8
  • Evaluation is limited to one-shot, zero-shot, or other low-label small-scene settings, not large-patch benchmarks with full downstream fine-tuning.Each task specifies its dataset, supervision, prompting rule, and decision rule.Source: §IV · p. 8
  • The analysis compares high-quality spectral-mask data with scaled weak supervision and tests whether promptable visual priors retain hyperspectral material sensitivity.Ablations separate synthetic spectral construction, MoE refinement, and confidence-aware pseudo-mask weighting.Source: §IV · p. 8

1) Datasets and Metrics:

The evaluation covers four hyperspectral tasks on representative datasets, plus oil-spill mapping on held-out scenes. Comparisons use task-specific methods and foundation-model baselines under controlled support and adaptation protocols.

1) Datasets and Metrics:

  • Classification, anomaly detection, change detection, and target detection use two representative datasets each, with task-appropriate metrics; oil-spill mapping adds two unseen HOSD scenes.The task metrics are OA, AA, and Kappa for classification; DF and ODP for anomaly and target detection; and IoU and F1 for change detection.Source: 1) Datasets and Metrics · p. 9

2) Baselines:

  • Baselines span classical spectral-statistical detectors, task-specific networks, and foundation models, with matched support and explicit limits on target-scene adaptation.The comparison is framed as low-label transfer and prompt efficiency, not maximum performance after full downstream adaptation.Source: 2) Baselines · p. 9

3) HyperSAM Inference Protocol:

  • Inference uses a common SAM3 checkpoint and a three-channel proxy; task-specific prompt densities and seeded supports standardize comparisons.Anomaly detection uses denser prompting because anomalies are usually small; classification and target detection use matched support configurations across methods.Source: 3) HyperSAM Inference Protocol · p. 9

B. Hyperspectral Classification

HyperSAM is evaluated for one-shot classification and unsupervised anomaly detection. It leads the reported classification metrics and produces coherent classification maps, while its anomaly proposals capture compact targets with fewer false alarms.

1) Experimental Settings:

  • One support pixel per class forms a prototype from the smallest generated mask covering it, and all masks are assigned by cosine similarity without fine-tuning.A single-pixel embedding is used when no generated mask covers a support pixel.Source: 1) Experimental Settings · p. 9

2) Results and Analyses:

  • 69.37% OA versus SAM3's 62.49% on Indian Pines; HyperSAM also achieves the best OA, AA, and Kappa on both classification datasets.On PaviaU, it reaches 89.12% OA and 85.81% Kappa.Source: 2) Results and Analyses · pp. 9–10, Table II · p. 10
  • HyperSAM achieves the best DF on both Beach scenes and best ODP on Beach-2; on Beach-1, HyperSIGMA and DOFA obtain higher ODP.Compact mask proposals localize anomalous objects and reduce scattered false alarms, while ODP also reflects score calibration and background ranking.Source: 2) Results and Analyses · pp. 10–11, Figure 10 · p. 11, Figure 11 · p. 11
  • Object-level priors yield more spatially coherent classification maps with sharper boundaries and less salt-and-pepper noise than the baselines.The smallest-mask support rule reduces irrelevant context when support pixels lie in mixed parcels or near boundaries.Source: 2) Results and Analyses · p. 10, Figure 9 · p. 11

D. Hyperspectral Change Detection

The section evaluates zero-shot change detection and target detection with mask-level spectral comparisons and region prototypes. HyperSAM performs strongly on Hermiston and Airport, while specialized spectral detection remains strongest on Cri.

1) Experimental Settings:

  • Zero-shot change detection compares mask-region embeddings across dates in both directions, takes the maximum response, and thresholds at a fixed quantile.Masks are generated independently for each date; no changed or unchanged target-pair pixels train or tune the model.Source: 1) Experimental Settings · p. 11

2) Results and Analyses:

  • 70.31% IoU versus HyperFree's 50.82% on Hermiston, with HyperSAM achieving the best IoU and F1 among compared methods.On BayArea/River, it is close to FC-SD and outperforms the other foundation-model baselines.Source: 2) Results and Analyses · p. 11, Figure 12 · p. 12
  • Mask-level averaging over coherent regions reduces local noise, and the two directional maps help represent pre-change and newly emerged or disappeared structures.The Hermiston comparison shows more complete changed regions and fewer fragmented artifacts than foundation-model baselines.Source: 2) Results and Analyses · p. 11, Figure 12 · p. 12

1) Experimental Settings:

  • Target detection builds a region-level prototype from the mask containing the support point, scores candidate masks by cosine similarity, then applies fixed thresholds and dilation.The support is a target spectrum or prompt, shared across methods within each test scene.Source: 1) Experimental Settings · pp. 11–12

2) Results and Analyses:

F. Ablation Studies and Further Analysis

The ablations indicate that data quality, complementary spectral and spatial branches, stable feature injection, and robust supervision each contribute distinct benefits. The MoE refiner specializes across mask scales, while confidence-aware training improves several task results but does not remove target-detection calibration sensitivity.

1) Data Construction, Architecture Components, and MoE Refinement:

  • The synthetic-data comparison indicates that sample count alone is insufficient: spectral plausibility, spatial sharpness, semantic diversity, and pseudo-mask reliability matter for promptable learning.Raw multispectral input remains competitive on PaviaU classification, while synthesized full spectra help when material shape and target-background separation matter.Source: 1) Data Construction, Architecture… · p. 12
  • The frozen RGB branch supplies objectness, boundary, and prompt-response priors, while the hyperspectral side encoder adds material-sensitive full-spectrum corrections.The ablations indicate these branches address distinct, complementary parts of adaptation.Source: 1) Data Construction, Architecture… · p. 12
  • Zero-initialized injection begins with zero residuals and preserves the frozen SAM3 representation before gradually introducing hyperspectral corrections.The controlled comparison reports consistent improvement over randomly initialized injection modules.Source: 1) Data Construction, Architecture… · pp. 12–13, Table IV · p. 13
  • Removing any reconstruction-objective term weakens its associated constraint, and the complete objective achieves the best overall transfer.The terms support radiometric fidelity, spectral-shape consistency, observed-signal agreement, parsimonious mixtures, and spatial coherence.Source: 1) Data Construction, Architecture… · p. 13, Table V · p. 13
  • Expert routing varies with mask size: Expert 3 favors small masks, Expert 1 becomes more common for large masks, and Expert 2 remains active across scales.Expert 2 routes 45.5%–50.3% of masks across ranges, without expert collapse.Source: 1) Data Construction, Architecture… · p. 13, Figure 15 · p. 13

2) Confidence-Aware Noisy-Label Weighting:

  • Confidence-aware weighting improves PaviaU classification, both change-detection scenes, and anomaly-detection ODP by selecting reliable regions and down-weighting uncertain pseudo-labels.Clear interiors can provide useful supervision, while boundaries, shadows, mixed pixels, and small objects contribute structured noise.Source: 2) Confidence-Aware Noisy-Label… · p. 13
  • Confidence maps assign higher weights to reliable semantic regions and lower weights to ambiguous boundaries and complex background fragments.The HTD results are mixed, especially on Airport, where prototype selection, score calibration, and thresholding matter.Source: 2) Confidence-Aware Noisy-Label… · pp. 13–14
  • The complete training objective gives the best overall HC, HAD, and HCD performance, while Airport target detection remains sensitive to specialized threshold calibration.BCE and Dice provide complementary supervision; confidence weighting suppresses unreliable pixels, Lcons improves spectral-window invariance, and Lbal prevents expert collapse.Source: 2) Confidence-Aware Noisy-Label… · pp. 13–14, Table VI · p. 14

3) Cross-Task Summary:

HyperSAM transfers across the evaluated hyperspectral tasks, with performance depending on whether spatial grouping complements spectral evidence. Oil-spill experiments extend that evaluation to marine scenes, while exposing boundaries of synthetic-data transfer.

3) Cross-Task Summary:

  • HC and HCD benefit from mask-level features, HAD from dense promptable proposals, and HTD remains scene-dependent, with classical spectral detectors sometimes leading for spectrum-defined targets.Across HC, HAD, and HCD, improvements generally track abundance-transfer data, prior-preserving adaptation, MoE refinement, and confidence-aware learning; HTD is more sensitive to target prototypes, calibration, and thresholding.Source: 3) Cross-Task Summary · p. 14
  • Synthetic-only training, pseudo-mask boundary disagreements, SAM3-checkpoint dependence, and small-scene protocols leave real-acquisition variability and full-fine-tuning performance unestablished.The limitations include sensor, atmosphere, resolution, geography, season, illumination, and land-cover differences, as well as mask disagreement around shadows, thin structures, mixed pixels, and visually similar surfaces.Source: 3) Cross-Task Summary · p. 14

2) Results and Analyses:

  • 91.14% OA and 90.83% AA on HOSD-GM13, and 97.39% OA and 97.65% AA on HOSD-GM17, demonstrate transfer to airborne oil-spill mapping.The unchanged checkpoint was evaluated without task-specific fine-tuning using 50 labeled pixels per class for spectral-spatial prototypes; remaining pixels were held for evaluation.Source: 2) Results and Analyses · p. 14, 1) Experimental Settings · p. 14
  • Predictions preserve large-scale oil-spill structures but err around fragmented slick boundaries and spectrally ambiguous transitions, highlighting a synthetic-urban to marine-scene gap.Adding water, coastline, and oil-like endmembers could extend abundance-transfer synthesis to application-specific spectral-mask pairs without changing the architecture.Source: 2) Results and Analyses · pp. 14–15, Figure 17 · p. 15

H. Computational Cost and Carbon Footprint

On Indian Pines, HyperSAM occupies the high-accuracy rather than minimum-energy end of the measured downstream trade-off. The conclusion emphasizes reuse across tasks and the potential value of compact, higher-quality synthetic data.

H. Computational Cost and Carbon Footprint

  • The benchmark averages three cold-start runs on one RTX 4090, measuring downstream command energy rather than upstream pretraining cost.Carbon estimates use PUE = 1.0 and Igrid = 475 gCO2e/kWh, and exclude CPU, memory, storage, networking, pretraining, checkpoint production, and embodied hardware emissions.Source: H. Computational Cost and Carbon… · p. 15
  • 66.91% OA, 74.09% AA, and 62.02% Kappa make HyperSAM highest in mean accuracy among compared methods, with 0.624 Wh and 0.296 g CO2e; DOFA-LP trails by 7.59 OA points.HyperSAM uses more GPU energy than DOFA-LP; HyperFree uses less energy but has substantially lower accuracy, while SSFTT and TGRS-ViT have low runtime and energy but lower OA and Kappa than HyperSAM and DOFA-LP.Source: H. Computational Cost and Carbon… · pp. 15–16, Table VII · p. 16, Figure 18 · p. 16

V. CONCLUSION

  • One trained HyperSAM parameter set is reusable across the evaluated HC, HAD, HCD, and HTD small-scene protocols, with gains over SAM3 and hyperspectral baselines under a common low-label budget.The conclusion frames this as a unified promptable model rather than a collection of task-specific networks.Source: §V · p. 16
  • A compact, higher-quality synthetic corpus can be more effective than a larger weakly object-centric corpus, and the HOSD experiment extends evaluation to oil-spill mapping.The paper identifies reliable object masks, physically plausible spectra, and stable promptable priors as useful ingredients for transferable hyperspectral foundation models.Source: §V · p. 16
Loading 2609.37340v1…