PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Hyperspectral Foundation Models for Hyperspectral Unmixing

Edgard Dabier, Christophe Kervazo, Pietro Gori, Florence Tupin

arXiv:2609.28283v1cs.CV

TL;DR

HFM performance for hyperspectral unmixing and the handling of patch-level feature resolution remain under-assessed. The paper benchmarks HFMs with a model-agnostic unmixing framework and compares feature-resolution augmentation strategies. It reports state-of-the-art HSU performance and finds linear mapping the strongest augmentation option on almost all datasets.

Abstract

from arXiv · show

Several foundation models dedicated to hyperspectral images have recently been made available. These models are trained on large unlabeled datasets and exhibit strong performance on many hyperspectral imaging tasks, such as classification or denoising. Nonetheless, their performance for hyperspectral unmixing -- the task of separating mixed spectra of overlapping materials in a hyperspectral image -- remain understudied. This might partly be due to the fact that most of them rely on vision transformer backbones, including patchification, leading to a feature resolution problem. While hyperspectral unmixing already arises from the low resolution of hyperspectral images, this patchification step potentially makes the problem even more ill-posed. Therefore, in this work, we aim to answer two questions: 1) \emph{how do foundation models perform in hyperspectral unmixing?}; 2) \emph{how to tackle the feature-level loss of resolution?} To answer the first question, we benchmark foundation models for unmixing, showing that they can reach state-of-the-art performance on four hyperspectral unmixing datasets. To answer the second question, we compare several feature upsampling approaches and empirically show that using a simple one can lead to high performance results. The code is available at https://gitlab.telecom-paris.fr/ring/hfm-hsu.git.

I. INTRODUCTION

Hyperspectral pixels often mix spectra from multiple materials because each pixel covers a large area, motivating unmixing to estimate both pure-material spectra and their proportions. The section introduces a linear mixture model for representing these observations.

I. INTRODUCTION

  • HSU estimates pure-material spectra (endmembers) and their relative concentrations (abundances) from mixed pixel spectra.The mixtures arise because each HSI pixel can cover a large area containing more than one material.Source: §I · p. 1
  • The Linear Mixture Model represents pixels as linear combinations of endmembers, with additive noise.The model is often complemented by abundance non-negativity and sum-to-one constraints.Source: §I · p. 1

A. HSU state-of-the-art and motivation

Earlier HSU methods range from pure-pixel retrieval to deep networks, while limited labeled scenes have constrained training. HFM unmixing evidence and strategies for patch-level features remain limited, motivating a multi-dataset benchmark and comparison of feature upsampling approaches.

A. HSU state-of-the-art and motivation

  • HSU research spans pure-pixel retrieval, optimization with handcrafted priors, CNNs, ViTs, Mamba models, and adversarial schemes.Pure-pixel methods can fail when the pure-pixel assumption is not met in practice.Source: A. HSU state-of-the-art and motivation · p. 1
  • HSU networks are often trained on a single image because manual labeling requires time and expertise and few scenes have annotations.This differs from computer-vision tasks that can use vast amounts of data.Source: A. HSU state-of-the-art and motivation · p. 1

B. Hyperspectral foundation models and contributions

  • HFMs learn features from large unlabeled HSI datasets, with pixel-level encoders representing each pixel and patch-level encoders losing pixel-scale detail.Examples of pixel-level models include HyperSIGMA, HyperSL, and UniverSat; patch-level examples include HyperFree, DOFA, and Panopticon.Source: B. Hyperspectral foundation models and… · pp. 1–2
  • Patch-level HFM features cannot directly provide material proportions at every pixel, and prior work leaves both their use and multi-dataset HFM assessment underexplained.The cited literature includes limited dataset testing, an unreproduced reported result, and a lack of associated code for reproducible assessment.Source: B. Hyperspectral foundation models and… · p. 2
  • The paper benchmarks HFMs for unmixing across several datasets with a model-agnostic framework and compares patch-level feature upsampling strategies.The framework performs unmixing directly from HFM features.Source: B. Hyperspectral foundation models and… · p. 2

II. METHODOLOGY

The methodology processes HFM features through resolution augmentation and a model-agnostic unmixing framework. Its unmixing block uses a CNN autoencoder, with abundances as latent vectors and a linear decoder representing the mixture model.

II. METHODOLOGY

  • The framework extracts features with a frozen HFM, augments their resolution when needed, and feeds them to a model-agnostic unmixing block.Resolution augmentation can be omitted for pixel-level HFMs.Source: §II · p. 2, Figure 1 · p. 2

A. Unmixing Auto-Encoder

  • The unmixing block uses a classical autoencoder structure, estimating abundances as latent vectors and associating linear decoder weights with endmembers.Its CNN encoder maps high-resolution features from dimension D to C, while a matrix multiplication reconstructs the HSI from C to B spectral dimensions.Source: A. Unmixing Auto-Encoder · p. 2
  • A Softmax layer enforces abundance non-negativity and sum-to-one constraints, and decoder weights are initialized from SiVM endmember estimates.The encoder also uses LeakyReLU, batch normalization, and dropout with probability 0.2.Source: A. Unmixing Auto-Encoder · p. 2

B. Resolution augmentation of the HFM features

Patch-level HFM features must be brought back to image resolution for pixel-wise abundance estimates. The paper tests interpolation, learned upsampling and feature-reconstruction approaches, as well as a learned linear mapping that performs best experimentally.

B. Resolution augmentation of the HFM features

  • Patch-level HFM features have lower spatial resolution, so their feature maps are augmented to image resolution for pixel-wise abundance estimation.The low-resolution features have shape D×α×α, while the augmented features have shape D×H×W.Source: B. Resolution augmentation of the HFM… · p. 2
  • The compared approaches include bilinear interpolation, deconvolution, Pixel Shuffle, and FiLM, which combines upsampled features with HSI details predicted by convolutions.The study also tests FeatUp and NAF feature-upsampling networks.Source: B. Resolution augmentation of the HFM… · pp. 2–3
  • The learned linear mapping transforms low- to high-resolution feature spaces and experimentally corresponds to the best tested augmentation strategy.It learns Wup in FHR = WupFLR.Source: B. Resolution augmentation of the HFM… · p. 3

C. Ensemble Learning Trick

The framework averages unmixing predictions from independent training runs; ensembling improves estimation quality and reduces variability, with a slight increase in computation time.

C. Ensemble Learning Trick

  • Independent training runs produce separate abundance and endmember estimates, whose predictions are averaged into the final unmixing result.Source: C. Ensemble Learning Trick · p. 3
  • Ensembling improves estimation quality and reduces model variability, at the cost of a slight increase in computation time.Source: C. Ensemble Learning Trick · p. 3

D. Training loss

The proposed framework is trained end-to-end without supervision by minimizing a mixture of losses. The losses assess reconstruction quality and encourage sparse abundance estimates.

D. Training loss

III. EXPERIMENTAL RESULTS

The experiments use four labelled hyperspectral datasets spanning different image sizes, spectral-band counts, and numbers of endmembers. Smaller images were upsampled and Urban was cropped to accommodate fixed-size model inputs, with the same preprocessing applied across methods.

A. Data Description

  • The datasets include Samson (95 × 95 pixels, 156 bands, 3 endmembers), Jasper Ridge (100 × 100 pixels, 198 bands, 4 endmembers), and Apex (110 × 110 pixels, 285 bands, 4 endmembers).Source: A. Data Description · p. 3
  • Urban has a spatial extent of 307 pixels, 162 spectral bands, and 6 endmembers.Source: A. Data Description · p. 3
  • For fixed model inputs, smaller images were bilinearly upsampled and Urban was cropped; this preprocessing was applied to all methods for fair comparison.Source: A. Data Description · p. 3

B. Experimental Setup

The setup assesses endmember and abundance quality with SAD and NMSE, respectively, and also compares execution time. It aggregates normalized SAD and NMSE across four datasets into a global method ranking and uses shared training and loss hyperparameters.

B. Experimental Setup

  • Endmember quality is assessed with SAD and abundance quality with NMSE, averaged over endmembers; execution time is measured on an NVIDIA V100 GPU.Source: B. Experimental Setup · p. 3
  • The global ranking aggregates SAD and NMSE after dataset-wise normalization across four datasets.Source: B. Experimental Setup · p. 3
  • Training and loss hyperparameters are shared across datasets: weight decay 0.18, learning rate 2e−3, λ1 = 0.09, λ2 = 0.6, 200 epochs, and AdamW.Source: B. Experimental Setup · p. 3

C. Choice of the Number of Trainings n for Ensembling

The paper finds that ensembling improves unmixing scores and reduces variability, with no further improvement beyond 15 trainings. Among the tested feature-resolution augmentation strategies, linear mapping performs best on almost all datasets.

C. Choice of the Number of Trainings n for Ensembling

D. Assessing of the Feature Resolution Augmentation strategy

E. HFM benchmark for HSU

Across the benchmark, hyperspectral foundation models achieve results comparable to state of the art in hyperspectral unmixing. The authors also report better performance from models with low-resolution features than from pixel-level encoders, and conclude that their model-agnostic framework achieves state-of-the-art performance.

E. HFM benchmark for HSU

  • The benchmark evaluates the HFMs within the proposed framework, with and without 15-training ensembling, against seven state-of-the-art methods.The methods include classic optimization, CNN, ViT, GAN, and Mamba approaches; available code did not yield Urban results for one comparator.Source: E. HFM benchmark for HSU · p. 4
  • Hyperspectral foundation models achieve results comparable to state of the art in hyperspectral unmixing.The benchmark compares averaged scores from 10 independent runs against seven state-of-the-art methods; the conclusion reports state-of-the-art performance for the framework.Source: E. HFM benchmark for HSU · p. 4, §IV · p. 4
  • HyperSL required cropping Urban to 128 × 128 for memory reasons, and HyperFree could not be used because its available pretrained weights lacked positional embeddings.Extracting HyperSL features from 224 × 224 images required more than 100 GB of VRAM.Source: E. HFM benchmark for HSU · p. 4
  • Models that extract low-resolution features perform better than pixel-level encoders, despite the spatial downsampling introduced by vision transformers.The conclusion describes the benchmark framework as simple to implement and model-agnostic.Source: E. HFM benchmark for HSU · p. 4, §IV · p. 4
  • Ensembling enhances prediction quality, while a single training can already achieve decent performance.The computational overhead of 15 trainings is smaller than a single unmixing-part training of HyperSIGMA or HyperSL.Source: E. HFM benchmark for HSU · p. 4
Loading 2609.28283v1…