PaperlayerView exact revision on arXiv

Source-linked AI summary

HyperFree: A Channel-adaptive and Tuning-free Foundation Model for Hyperspectral Remote Sensing Imagery

Jingtao Li, Yingyi Liu, Xinyu Wang, Yunning Peng, Chen Sun, Shaoyu Wang, Zhendong Sun, Tian Ke, Xiao Jiang, Tangwei Lu, Anran Zhao, Yanfei Zhong

arXiv:2503.21841v1cs.CV

TL;DR

Existing remote-sensing foundation models and prompt-based systems have difficulty handling hyperspectral images with varying spectral channels without image-specific adaptation. HyperFree addresses this with channel-adaptive embeddings, prompt-mask-feature interactions, and large-scale hyperspectral pre-training, achieving comparable tuning-free accuracy to specialized models across 5 tasks and 11 datasets.

  • Problem

    Prompt engineering can process unseen hyperspectral scenes without fine-tuning but must handle sensor-dependent channel variation and tractable prompting for hyperspectral images.

  • Method

    HyperFree uses a learnable full-spectrum wavelength dictionary for dynamic embeddings and feature-space prompt-mask interactions for multiple semantic-aware masks.

  • Results

    HyperFree achieves comparable accuracy to state-of-the-art models across 11 datasets and 5 segmentation-related tasks in a tuning-free manner using one prompt.

  • Takeaways & Limitations

    HyperFree provides a tuning-free hyperspectral foundation model that directly processes unseen images with varying channel counts and prompt-based or zero-shot workflows.

Abstract

from arXiv · show

Advanced interpretation of hyperspectral remote sensing images benefits many precise Earth observation tasks. Recently, visual foundation models have promoted the remote sensing interpretation but concentrating on RGB and multispectral images. Due to the varied hyperspectral channels,existing foundation models would face image-by-image tuning situation, imposing great pressure on hardware and time resources. In this paper, we propose a tuning-free hyperspectral foundation model called HyperFree, by adapting the existing visual prompt engineering. To process varied channel numbers, we design a learned weight dictionary covering full-spectrum from $0.4 \sim 2.5 \, μ\text{m}$, supporting to build the embedding layer dynamically. To make the prompt design more tractable, HyperFree can generate multiple semantic-aware masks for one prompt by treating feature distance as semantic-similarity. After pre-training HyperFree on constructed large-scale high-resolution hyperspectral images, HyperFree (1 prompt) has shown comparable results with specialized models (5 shots) on 5 tasks and 11 datasets.Code and dataset are accessible at https://rsidea.whu.edu.cn/hyperfree.htm.

1. Introduction

Hyperspectral imagery offers rich spectral information but creates channel-variation and data-processing challenges for foundation models. HyperFree adapts visual prompt engineering with channel-adaptive embeddings, semantic-aware mask generation, and large-scale hyperspectral pre-training.

  • Motivation: Hyperspectral images provide high spectral resolution, broad wavelength coverage, and dozens to hundreds of channels for precise target identification.These properties support diverse Earth-observation applications while increasing processing complexity.
  • Existing paradigms: Existing remote-sensing foundation models primarily target RGB and multispectral imagery through pre-training and tuning or prompt engineering.Pre-training and tuning can require time-consuming backbone training and task-specific fine-tuning, while prompt engineering uses task-specific prompts with frozen models.
  • Challenges: Prompt engineering handles unseen scenes without fine-tuning but struggles with sensor-dependent spectral parameters and tractable prompting for hyperspectral images.Hyperspectral sensors vary in spectral resolution, wavelength range, and channel count, while conventional prompts typically generate a single mask.
  • HyperFree: HyperFree uses a learnable wavelength weight dictionary spanning 400 to 2500 nm to dynamically encode varying hyperspectral inputs into consistent-dimensional visual tokens.The dictionary is sampled at 10 nm intervals and supports channel-adaptive embedding generation.
  • HyperFree: HyperFree maps prompts and masks into feature space, treating feature distance as semantic similarity to generate multiple semantic-aware masks from one prompt.Prompt, mask, and feature interactions are adapted to downstream tasks in tuning-free or zero-shot workflows.
  • Data and evaluation: Hyper-Seg supplies nearly 50,000 hyperspectral-image and segmentation-mask pairs for promptable HyperFree training, while evaluations cover 11 datasets across 5 segmentation-related tasks.The constructed data address the scarcity of annotated hyperspectral datasets, and the reported evaluations show comparable accuracy to state-of-the-art models in a tuning-free manner.

2. Related Work

Hyperspectral image processing spans multiple dense pixel-level tasks, while remote-sensing foundation models and prompt engineering have mainly developed around RGB or multispectral imagery. Prompt engineering transfers frozen models with prompts, motivating its adaptation to hyperspectral data.

  • Hyperspectral image processing: Hyperspectral processing includes classification, one-class classification, target detection, anomaly detection, and change detection.These tasks discriminate pixel classes, targets, spectral anomalies, or changes between time steps.
  • Foundation models: Remote-sensing foundation models use large transformer-based models and pre-training data to transfer capabilities across downstream tasks.Existing examples such as SatMAE, SpectralEarth, and SpectralGPT focus on multispectral imagery with relatively few channels.
  • Prompt engineering: Prompt engineering completes downstream tasks directly with prompts, following the use of hand-crafted prompts in language models and point or box prompts in SAM.This paradigm differs from pre-training followed by task-specific fine-tuning.

3. Hyper-Seg Data Engine

Hyper-Seg addresses the scarcity and cost of hyperspectral segmentation annotations by automatically generating masks from spectral imagery. Its resulting dataset provides large-scale, high-resolution annotated data for promptable training.

  • Motivation: Hyperspectral segmentation datasets often contain few images because image acquisition and labeling are expensive, limiting prompt-engineering data scale.Dense hyperspectral tasks require large-scale segmentation data, unlike million-scale natural-image datasets such as SA-1B.
  • Data engine: Hyper-Seg groups spectral images into three-channel sets, segments each group with SAM-H, and combines outputs with non-maximum suppression.Nine key channels are selected using expert knowledge before segmentation results are merged into the final map.
  • Data engine: Hyper-Seg processes 41,946 AVIRIS hyperspectral images and supplements them with fMoW and SpaceNet multispectral data.The engine uses spectral information to obtain high-precision masks without manual labeling labor.
  • Dataset scale: The final Hyper-Seg dataset contains nearly 150k images and 15.44 million masks, with segmentation annotations supporting promptable training.It offers higher spatial resolution than the cited HyperSigma dataset and is described as the only compared dataset with segmentation masks.

4. HyperFree Foundation Model

HyperFree combines channel-adaptive embedding with prompt-mask-feature interaction to process varied hyperspectral inputs and support multiple tuning-free downstream tasks.

  • Channel-adaptive embedding: HyperFree dynamically constructs fixed-length tokens from hyperspectral inputs with varying channels using wavelength-dependent weight dictionaries.The embedding maps varying-channel images into a shared token representation, using dictionaries selected from input wavelengths.
  • Channel-adaptive embedding: The embedding design uses parallel branches for expert-selected key channels and spectral cubes, merging their outputs by elementwise addition.Separate dictionaries process key-channel and cube inputs before producing the final token representation.
  • Full-spectrum promptable training: HyperFree is trained promptably on automatically generated hyperspectral segmentation data with randomly selected spectral subsets, point prompts, focal loss, and Dice loss.The training procedure encourages the wavelength-aware dictionary to encode varied channel compositions into a unified token space.
  • Prompt-mask-feature interactive inferring: PMF interaction maps prompts and masks into feature space, using feature distance or cosine similarity to select multiple semantically consistent masks.This extends single-prompt segmentation to downstream tasks requiring all masks associated with a semantic category.
  • Prompt-mask-feature interactive inferring: Adaptive PMF workflows support five hyperspectral tasks, with task-specific prompt requirements and threshold meanings, including zero-shot anomaly and change detection.Classification and one-class classification use point prompts, while anomaly detection does not require a prompt; change detection compares temporal feature distances against a threshold.
  • Evaluation: HyperFree uses one prompt or zero-shot inference in tuning-free comparisons against specialized models trained with five shots.The supplied comparisons include qualitative examples and a reported five-task, 11-dataset evaluation setting.

5. Results

HyperFree is evaluated across five hyperspectral tasks and compared with trained specialized models, showing strong tuning-free performance and substantial efficiency gains. Additional analyses examine qualitative outputs, wavelength-aware encoding, tuning performance, and comparisons with SAM and HyperSigma.

  • Main Results: HyperFree processes 11 datasets across five hyperspectral tasks with fixed parameters, using one prompt or zero-shot operation.Comparison methods are trained on each dataset with five shots per class.
  • Main Results: About 12 F1 points on HC and 0.5 IoU points on HCD separate HyperFree from previous state-of-the-art results.These improvements are reported for the tuning-free setting.
  • Qualitative Comparison: Qualitative comparisons show more complete and smoother result maps for HyperFree than for specialized models.The comparison covers five exemplified images, with one state-of-the-art model visualized for each task.
  • Analysis Studies: Using the same prompt-mask-feature workflows with SAM produces metric declines of about 10–30 points on most tasks, but the comparison is described as unfair.The comparison is intended to reflect differences in hyperspectral data processing rather than establish a fair model ranking.
  • Analysis Studies: HyperFree features show more compact intra-class distances than HyperSigma features in t-SNE visualizations, supporting the tuning-free PMF workflow.
  • Analysis Studies: Randomly shuffling wavelength order causes accuracy drops of about 5–40 points, supporting the learned dictionary’s wavelength-aware behavior.The ablation evaluates HC, HOCC, and HCD tasks.
  • Analysis Studies: Tuned HyperFree surpasses previous state-of-the-art models on nearly all tasks across eight tasks and 14 datasets.

6. Conclusion

HyperFree addresses the cost-effectiveness challenges of hyperspectral foundation models by adapting visual prompt engineering to channel variation and task diversity. Its wavelength dictionary and prompt-mask-feature interaction provide channel-adaptive embeddings and semantic-aware masks without tuning.

  • HyperFree targets varied-length channels, imaging-condition sensitivity, and acquisition difficulty that make image-by-image tuning costly.
  • A learnable full-spectrum weight dictionary generates channel-adaptive embeddings for hyperspectral inputs.
  • Prompt-mask-feature interaction generates semantic-aware masks for different downstream tasks.

1. Hyper-Seg Data Engine

The Hyper-Seg data engine augments hyperspectral training data by separating images into wavelength groups informed by practical satellite bands. Its construction uses multispectral wavelength combinations and mask-generation statistics.

  • Wavelength Selection: Hyper-Seg separates original images into three groups with different wavelength combinations to use spectral information beyond RGB.The selected wavelengths cover 0.4–2.5 µm and reference Landsat-8 bands.
  • Wavelength Selection: Table 1 records selected wavelengths from classical multispectral satellites to support Hyper-Seg wavelength selection and the βk weight dictionary.
  • Dataset Statistics: After non-maximum suppression, combined masks are approximately 2–3 times as numerous as masks from each wavelength group separately.The reported density of generated masks is roughly equivalent across the three source datasets.

2. Selection of Key Channels in Weight Dictionary

Key channels in the weight dictionary are selected from successful prior satellite wavelengths and consolidated by wavelength proximity. This process creates a standardized set of representative bands for channel-adaptive embedding.

  • Key-Channel Selection: Key channels are selected from successful satellite precedents and expert knowledge listed in Table 1.
  • Key-Channel Selection: Wavelengths from different satellites separated by less than 10 nm are merged by averaging adjacent values.The 10 nm interval is treated as a common spectral resolution.

3. Overview of Experimental Datasets

The evaluation uses public hyperspectral datasets spanning eight tasks, with different channel numbers and spectral ranges. Five tasks support both tuning-free and tuning evaluation, while three tasks use tuning only because of differing output formats.

  • Five tasks support both tuning-free and tuning evaluation: HC, HOCC, HTD, HAD, and HCD.
  • HD, HU, and HOT are evaluated only in tuning mode because their output formats differ.
  • Public datasets span eight hyperspectral tasks and vary in channel numbers and spectral ranges.

4. Additional Experiments

Additional experiments assess qualitative performance, tuning behavior, execution efficiency, prompt sensitivity, and hyperparameter sensitivity across the evaluated tasks and datasets. HyperFree is reported to produce strong visual results without tuning, reduce processing time substantially, and remain usable with one prompt despite sensitivity to τ in some tasks.

  • Qualitative Results in Tuning-free Manner: HyperFree achieves the best visualization performance without tuning compared with specialized models trained with 5 shots.The comparison covers qualitative results across HC, HOCC, HTD, HAD, and HCD.
  • Further Tuning Performance: HyperFree* denotes the tuned version evaluated in additional quantitative and qualitative experiments across HC, HOCC, HTD, HAD, HCD, HD, HU, and HOT.
  • Sensitivity Analysis: One prompt is sufficient for HyperFree in HC and HOCC, whose performance is mostly insensitive to prompt number.The analysis uses mean and standard deviation over 10 repeated experiments for each prompt number.
  • Sensitivity Analysis: Metric fluctuations remain within 0.1 when varying τ, although HTD and HCD show greater sensitivity than most tasks.HC is excluded because it does not require τ.
  • Execution Efficiency Comparison Experiments: 1–2 orders of magnitude lower processing time is reported for HyperFree than other deep models without tuning.
  • Qualitative Comparisons: Figures provide qualitative comparisons for HC, HOCC, HTD, HAD, HCD, HD, HU, and HOT, with HyperFree* representing the tuning version.
Loading 2503.21841v1…