PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Super-Resolution using Spatial-Spectral Residual Operator Networks

Seokhyun Chin

arXiv:2609.35410v1cs.CVcs.AI

TL;DR

Hyperspectral imagery provides detailed spectral information but is limited by low spatio-temporal resolution and high acquisition costs, while spectral super-resolution is ill-posed. The paper frames SSR as operator learning and proposes SSRON, which outperforms baseline models across tested metrics and demonstrates zero-shot prediction of unseen bands.

  • Problem

    Hyperspectral imagery offers detailed spectral information, but its broader use is limited by low spatio-temporal resolution and high acquisition costs, and SSR must recover hundreds of bands from limited multispectral measurements.

    Source: §I · p. 1

  • Method

    The paper formulates SSR as operator learning and proposes SSRON, a DeepONet using a spatial–spectral residual CNN as its branch network.

    Source: §I · p. 1, A. Architecture · p. 2

  • Results

    SSRON outperforms all baseline models across all tested metrics.

    Source: §IV · p. 3

  • Takeaways & Limitations

    SSRON demonstrates zero-shot prediction of unseen bands, and its continuous wavelength input suggests potential for spectral interpolation between native sensor bands.

    Source: §I · p. 1, §IV · p. 4

    Accuracy beyond native sensor bands remains unconfirmed without validation against denser spectral measurements.

    Source: §IV · p. 4

Abstract

from arXiv · show

Spectral super-resolution of multispectral satellite images can enable high temporal- and spatial-resolution hyperspectral satellite imagery at a modest cost, significantly increasing the applicability of hyperspectral remote sensing. This task is inherently ill-posed, making it well-suited for deep learning-based methods. In this study, the spectral super-resolution task is framed as an operator learning problem, and SSRON is proposed as a Deep Operator Network that effectively learns function-to-function mappings from downsampled spectra to continuous spectra. The model is trained to super-resolve Sentinel-2A-like multispectral imagery to EMIT images. Compared to baseline models, SSRON achieves superior performance across all metrics. The model also demonstrates zero-shot spectral super-resolution capability by predicting bands unseen during training. Furthermore, its continuous-output formulation suggests the potential to estimate spectra at finer wavelength intervals than the native sensor. These results suggest the potential of SSRON and establishes operator learning as a promising direction for spectral super-resolution.

I. INTRODUCTION

Hyperspectral imagery is valuable but constrained by low spatio-temporal resolution and high acquisition costs. The paper frames spectral super-resolution as operator learning, proposes SSRON, and reports improved performance and zero-shot prediction of unseen bands.

I. INTRODUCTION

  • Hyperspectral imagery provides detailed spectral information for Earth observation, but its broader use is limited by low spatio-temporal resolution and high acquisition costs.Applications include weather forecasting, precision agriculture, and water quality monitoring.Source: §I · p. 1
  • Spectral super-resolution is ill-posed because hundreds of hyperspectral bands must be recovered from only a limited number of multispectral bands.Source: §I · p. 1
  • Although neural operators have been explored for SSR, the task remains largely unformulated as an operator-learning problem.Neural operators learn mappings between infinite-dimensional function spaces and permit continuous function evaluation.Source: §I · p. 1
  • The paper formulates SSR as operator learning and proposes SSRON, a DeepONet with a spatial–spectral residual CNN branch network.The authors describe this as the first adaptation of that CNN architecture to operator learning for spectral super-resolution.Source: §I · p. 1
  • SSRON improves performance over state-of-the-art methods and demonstrates zero-shot prediction of unseen bands.The authors also identify potential for continuous-output use.Source: §I · p. 1

II. PROBLEM FORMULATION

The formulation treats sensor measurements as samples derived from continuous radiance and frames spectral super-resolution as an operator mapping multispectral observations to hyperspectral radiance.

II. PROBLEM FORMULATION

  • Sensor-calibrated radiance is modeled using solar radiation and regional reflectance, and both multispectral and hyperspectral images are downsampled from this continuous function.Source: §II · p. 2
  • The multispectral and hyperspectral instruments’ spectral response functions determine how their bands sample the continuous radiance.The hyperspectral response functions are centered around their bands’ central wavelengths.Source: §II · p. 2
  • SSR is formulated as an operator mapping multispectral measurements to radiance across wavelength and spatial position, with the multispectral response function fixed in this study.Extension to multiple multispectral response functions is described as possible.Source: §II · p. 2

A. Architecture

SSRON builds on DeepONet by encoding the input spectrum and output coordinates separately, while using a spatial–spectral residual CNN as its branch network.

A. Architecture

  • DeepONet encodes the input function at discrete sensor points with a branch network and output evaluation locations with a trunk network.The architecture is outlined in Figure 1.Source: A. Architecture · p. 2
  • SSRON replaces the standard multilayer perceptron branch with a spatial–spectral residual CNN motivated by its strong SSR performance.Source: A. Architecture · p. 2
  • The coordinates x, y, and λ are embedded independently and concatenated; the trunk is a two-layer fully connected network with 256 hidden dimensions.The embedding layer maps each coordinate value to a vector.Source: A. Architecture · p. 2

B. Dataset

The dataset uses EMIT calibrated radiance as hyperspectral imagery and constructs Sentinel-2A-like multispectral inputs by applying the satellite’s spectral response function. Atmospheric-absorption bands and one corrupted band are excluded.

B. Dataset

  • 229 EMIT bands remain after bands attributed to water-vapor and ozone absorption are dropped from the instrument’s 285 visible-to-short-infrared bands.The source comprises 10 randomly selected cloud-free scenes from December 2024.Source: B. Dataset · p. 2
  • Sentinel-2A-like multispectral images are generated by downsampling the hyperspectral data with a normalized, 229-band Sentinel-2A spectral response function, then dropping band 9.Band 9 is dropped because it samples corrupted hyperspectral bands.Source: B. Dataset · p. 2

C. Training

Training uses non-overlapping image patches and a train-validation-test split, with MAE loss and Adam optimization. SSRON’s architecture combines a branch network with a trunk network.

C. Training

  • 61,600 non-overlapping 16x16 patches were split 8:1:1 into training, validation, and test sets.Models used MAE loss, Adam at 5e-5, cosine annealing, early stopping with patience 5, and 300 training epochs.Source: C. Training · p. 2
  • Training was conducted on a single RTX 5060 Ti GPU.Source: C. Training · p. 2
  • SSRON’s branch network encodes the multispectral image into a latent representation, while its trunk network encodes spatial-spectral query positions.The architecture is shown in Figure 1.Source: Figure 1 · p. 3

1) Training SSRON:

SSRON predicts values at specified spatial and spectral coordinates, rather than producing conventional image-to-image outputs. A zero-shot experiment tests performance on spectral bands withheld during training.

1) Training SSRON:

  • For each epoch, SSRON predicts HSI label values at 2,048 selected spatial and spectral positions.The model is trained to evaluate the learned function at specified coordinates, unlike conventional image-to-image models.Source: 1) Training SSRON · p. 3

2) Zero-shot super resolution:

  • Zero-shot evaluation withholds a fixed percentage of bands during training and evaluates model performance across all bands.The excluded bands are relatively equidistant and span the same wavelength range as the data.Source: 2) Zero-shot super resolution · p. 3
  • The withheld bands are selected to be relatively equidistant while spanning the data’s wavelength range.Source: 2) Zero-shot super resolution · p. 3

D. Baselines

The study compares SSRON with spectral-super-resolution and neural-operator baselines and evaluates performance using four image-quality metrics. SSIM is calculated from image means, variances, and covariance.

D. Baselines

  • AWAN, Restormer, and SSRAN serve as spectral-super-resolution baselines, while UNO and FNO provide operator-learning comparisons.RSNO is excluded because it requires additional physics-based input unavailable and not assumed in this setting.Source: D. Baselines · p. 3

E. Metrics

  • MAE, RMSE, PSNR, and SSIM are used as evaluation metrics.Source: E. Metrics · p. 3
  • SSIM uses image means, variances, covariance, and small constants based on the data’s dynamic range.The constants stabilize division.Source: E. Metrics · p. 3

IV. RESULTS

SSRON outperforms all baseline models across tested metrics, with lower per-band errors except at band 115. Zero-shot performance varies with the bands withheld, while continuous wavelength inputs suggest interpolation potential that still requires validation against denser measurements.

IV. RESULTS

  • SSRON has lower errors across all bands except band 115, where UNO performs better; errors rise below band 5 and above band 200.The edge-band increases are attributed likely to limited Sentinel-2A spectral information in those regions.Source: §IV · p. 3
  • MAE and RMSE correlate with SRF coverage at -0.222 and -0.175, respectively, while PSNR correlates at 0.408 (p < 0.01).The reported relationships indicate lower errors at wavelengths with higher SRF values.Source: §IV · p. 3
  • SSRON outperforms all baseline models across every tested metric on the testing dataset.Table II reports testing-dataset error metrics.Source: Table II · p. 3, §IV · p. 3
  • As fewer bands are used for training, MAE increases and SSIM decreases, while RMSE and PSNR do not change monotonically.The 75% model has lower RMSE than the 80% model; the 60% model has lower RMSE and higher PSNR than the 65% model.Source: §IV · pp. 3–4, Table III · p. 4
  • Removing high-error bands can leave the model unpenalized for those errors, contributing to worse final-model performance.The authors suggest this as a likely explanation for non-monotonic zero-shot results.Source: §IV · p. 4
  • At 90% of bands used, SSRON remains competitive with state-of-the-art performance and has lower MAE than other methods.Continuous wavelength inputs also suggest potential interpolation between native bands, but denser spectral measurements are needed to validate accuracy.Source: §IV · p. 4

V. CONCLUSION AND FUTURE DIRECTIONS

The paper presents SSRON, a Deep Operator Network with a spatial-spectral residual branch, and reports state-of-the-art accuracy on simple spectral super-resolution. It identifies architecture optimization, dense spectral interpolation, and support for varying satellite SRFs as future directions.

V. CONCLUSION AND FUTURE DIRECTIONS

  • SSRON combines a Deep Operator Network with spatial-spectral convolutions and residual connections in its branch network, achieving state-of-the-art accuracy on simple SSR.The conclusion also reports potential for zero-shot prediction of additional bands.Source: §V · p. 4
  • Future work should optimize neural-operator architectures to further improve model performance.The paper frames operator learning as a new direction for spectral super-resolution.Source: §V · p. 4
  • The authors identify dense spectral interpolation for downstream applications and varying input SRFs for different satellites as directions for further study.Materials classification of imaged regions is given as one possible downstream application.Source: §V · p. 4

Related papers

Loading 2609.35410v1…