Source-linked AI summary
A convex formulation for hyperspectral image superresolution via subspace-based regularization
Miguel Simões, José Bioucas-Dias, Luis B. Almeida, Jocelyn Chanussot
TL;DR
The paper addresses fusion of HSIs and MSIs to recover images with simultaneously high spectral and spatial resolution despite the problem's scale and ill-posedness. It proposes a convex HySure formulation using subspace reduction, vector Total Variation, and SALSA/ADMM, while estimating sensor responses. Experiments on simulated and real-life data show favorable comparisons with published methods.
Problem
HSI–MSI fusion seeks jointly high spectral and spatial resolution but involves a large, ill-posed estimation problem with difficult resolution and spectral-overlap differences.
Method
HySure formulates fusion as a convex optimization problem using vector Total Variation, low-dimensional subspace representations, and SALSA, while estimating spatial and spectral sensor responses.
Results
HySure compares favorably with several published methods on simulated and real-life data, and outperformed other methods in the reported fusion tests except for one SAM comparison.
Takeaways & Limitations
The framework provides one approach for fusing hyperspectral data with multispectral or panchromatic images while jointly handling unknown sensor responses.
Abstract
from arXiv · showhide
Hyperspectral remote sensing images (HSIs) usually have high spectral resolution and low spatial resolution. Conversely, multispectral images (MSIs) usually have low spectral and high spatial resolutions. The problem of inferring images which combine the high spectral and high spatial resolutions of HSIs and MSIs, respectively, is a data fusion problem that has been the focus of recent active research due to the increasing availability of HSIs and MSIs retrieved from the same geographical area. We formulate this problem as the minimization of a convex objective function containing two quadratic data-fitting terms and an edge-preserving regularizer. The data-fitting terms account for blur, different resolutions, and additive noise. The regularizer, a form of vector Total Variation, promotes piecewise-smooth solutions with discontinuities aligned across the hyperspectral bands. The downsampling operator accounting for the different spatial resolutions, the non-quadratic and non-smooth nature of the regularizer, and the very large size of the HSI to be estimated lead to a hard optimization problem. We deal with these difficulties by exploiting the fact that HSIs generally "live" in a low-dimensional subspace and by tailoring the Split Augmented Lagrangian Shrinkage Algorithm (SALSA), which is an instance of the Alternating Direction Method of Multipliers (ADMM), to this optimization problem, by means of a convenient variable splitting. The spatial blur and the spectral linear operators linked, respectively, with the HSI and MSI acquisition processes are also estimated, and we obtain an effective algorithm that outperforms the state-of-the-art, as illustrated in a series of experiments with simulated and real-life data.
I. INTRODUCTION
HSI–MSI fusion seeks images with high spectral and spatial resolution, but its large dimensionality, ill-posedness, and uncertain sensor responses make the problem difficult. The paper proposes HySure, a convex, computationally efficient fusion framework that can estimate unknown sensor responses.
- HSIs provide high spectral resolution while MSIs provide higher spatial resolution, motivating fusion into images with both properties.
- HSI–MSI fusion is harder than pansharpening because it estimates more variables, handles larger hyperspectral dimensionality, and may lack complete spectral overlap.
- A. Contributions: HySure formulates fusion using a linear inverse problem, vector Total Variation regularization, and ADMM optimization through SALSA.
- A. Contributions: HySure reduces computational cost by exploiting data redundancy and fuses hyperspectral data with either multispectral or panchromatic images.
- A. Contributions: The blind formulation estimates spatial and spectral sensor responses from the data under limited-support and smoothness assumptions.
- A. Contributions: The paper extends prior work by detailing optimization and sensor-response estimation while adding experimental results.
B. Dimensionality reduction
The method exploits the low intrinsic dimensionality of hyperspectral data by representing the image in a smaller subspace. This reduction improves computational efficiency and can improve estimation accuracy while retaining a faithful signal representation.
- B. Dimensionality reduction: Hyperspectral spectral vectors usually lie in a subspace whose dimension is much smaller than the number of bands.
- B. Dimensionality reduction: The reduced representation makes computation more efficient and normally improves accuracy by significantly reducing the number of estimated variables.
- B. Dimensionality reduction: With 200 hyperspectral bands and Ls = 10, only 5% of the original variables need to be inferred.
- B. Dimensionality reduction: SVD provides a low-dimensional approximation by truncating small singular values, retaining a rather faithful approximation because hyperspectral data have low intrinsic dimensionality.
- C. Regularization: The vector Total Variation regularizer promotes spatial smoothness while aligning edges and other details across hyperspectral bands.
- C. Regularization: Applying regularization to reduced coefficients X indirectly regularizes the high-resolution image Z because both occupy the same, or approximately the same, subspace.
D. Optimization problem
The paper formulates HSI-MSI fusion as a convex optimization problem with data fidelity and VTV regularization, then solves it using SALSA with variable splitting.
- The objective combines two data-fitting terms with a vector Total Variation regularizer.The data terms explain the observed images, while the regularizer controls spatial and spectral structure.
- The optimization is difficult because the regularizer is non-quadratic and non-smooth, the variable is large, and downsampling prevents direct Fourier-based optimization.
- The split formulation imposes V = HXT and uses an augmented Lagrangian with a scaled dual variable and penalty parameter.
- SALSA applies ADMM through variable splitting, replacing the original problem with simpler subproblems linked by auxiliary variables.The original variable X is split into X and four auxiliary variables, V1 to V4.
- The X-subproblem and some auxiliary subproblems are quadratic and can be solved efficiently using FFTs, while others use Moreau proximity operators.
- SALSA convergence follows when H has full column rank and the cost function is closed, proper, and convex, assuming the optimization problem has a solution.
IV. ESTIMATING THE SPATIAL BLUR AND THE SPECTRAL RESPONSE FROM THE DATA
The method estimates the sensors’ spatial blur and spectral response directly from the observed HSI and MSI data by solving regularized quadratic problems under structural assumptions.
- Estimating B and R addresses scarce, inaccurate, or mismatched manufacturer response information caused by atmospheric conditions, postprocessing, and scene variability.
- B and R are inferred by minimizing a quadratic objective with regularizers and constraints on the response matrices.
- Only the component of R parallel to the HSI subspace is identifiable, while the orthogonal component has no effect on RYh and approximately no effect on RZ.
- The spatial response is modeled as a cyclic convolution with a finite-support kernel contained in a centered square window.
- The blur estimate is obtained through regularized least squares, with smoothing penalties on neighboring kernel differences and subsequent normalization to unit DC gain.
- The user-specified blur support has little experimental influence when it contains the actual blur support.
- Spectral-response estimation decouples across MSI bands, while a stronger preliminary blur reduces the unknown blur’s effect and possible image misregistration.
- Under stated spectral-coverage and contiguity assumptions, the quadratic objective has a unique solution, making the associated subproblems uniquely solvable.
V. EXPERIMENTAL STUDY
The experimental study describes datasets, evaluation indices, implementation details, and comparisons with other fusion methods.
- The experiments cover dataset descriptions, quality indices, implementation details, and comparisons with several competing fusion methods.
A. Data sets
The study uses synthetic, semi-synthetic, and real satellite datasets to evaluate fusion under controlled and real-world conditions.
- Dataset A is synthetic, using geometric shapes, five randomly selected spectral-library endmembers, and a high-resolution ground truth.
- Dataset A generates HSIs by spatial blurring and downsampling the ground truth by a factor of 4 in each direction.
- Dataset B is semi-synthetic and uses Pavia University imagery as ground truth for generating hyperspectral, multispectral, and panchromatic observations.
- Dataset C contains Hyperion and ALI images acquired over Paris, with Hyperion at 30 meters and ALI multispectral and panchromatic imagery at 30 and 10 meters.
- For real-data HSI-MSI fusion, the hyperspectral image is blurred and downsampled to make its spatial resolution lower than the MSI.
B. Quality indices
The experiments evaluate fusion quality with ERGAS, SAM, and UIQI when ground truth is available, while visual inspection is used for dataset C HSI+PAN fusion without ground truth. Preprocessing removes unreliable bands and projects data onto a 10-dimensional subspace preserving at least 99.95% of image energy.
- Quality indices: ERGAS, SAM, and UIQI were used to evaluate fusion quality when ground truth images were available.These indices were applied to datasets A and B and to HSI+MS fusion on dataset C.
- Quality indices: SAM measures spectral quality as the mean angle between estimated and ground-truth pixel spectra, reported in degrees.Lower angular differences indicate closer spectral representations.
- Quality indices: UIQI averages window-based image quality over 32 × 32 pixel windows and extends to hyperspectral images by averaging across bands.Its range is [−1, 1], with 1 attained when the estimate equals the ground truth.
- Quality indices: For dataset C HSI+PAN fusion, no ground truth was available, so results were assessed only through false-color visual inspection.The reported quality indices therefore do not apply to this case.
- Preprocessing: Uncalibrated or very noisy bands were removed before denoising the hyperspectral data with a truncated-SVD subspace of dimension Ls = 10.This dimension preserved at least 99.95% of the original image energy across all datasets.
- Parameter evaluation: For λϕ = 0 on dataset B HSI+PAN fusion, ERGAS, SAM, and UIQI were 5.046, 6.587, and 0.882, respectively.The algorithm was run for 200 iterations in every case after verifying the stopping criterion stayed below that limit.
D. Experimental results
The experiments assess sensor-response estimation and compare HySure with several fusion methods on synthetic and real-life datasets. The proposed method outperformed the alternatives in all reported cases except dataset B’s SAM index, where BT performed better.
- Sensor-response estimation: Estimated spectral and spatial blurs for dataset C appeared reasonable after applying the response-estimation procedure.The estimates are shown in Fig. 5.
- Comparison methods: The comparison included pansharpening methods originally designed for multispectral rather than hyperspectral images.These methods were extended to hyperspectral data because they do not depend on the number of bands.
- Comparison methods: Component Substitution methods transform multiband data, replace one component with a panchromatic-derived image, and invert the transformation.The tested family included GS, GSA, and FIHS-related approaches.
- Comparison methods: GSA improved over GS experimentally, although hyperspectral fusion can require inversion of a nearly singular matrix that may affect result quality.This limitation arises from adapting the method to hyperspectral images.
- Results: The proposed method outperformed the other methods in all cases except dataset B’s SAM index, where BT achieved the better result.The authors also report that many pansharpening methods handle the panchromatic spectral-range mismatch poorly.
3) Fusion of hyper- and multispectral images:
The experiments evaluate HSI+MSI and HSI+PAN fusion across several datasets, comparing the proposed method with published alternatives and examining image and error behavior. The proposed approach surpasses the other method in all reported HSI+MSI tests and also estimates sensor responses.
- Experimental setup: The HSI+MSI experiments compare the proposed method with Zhang et al.’s ZBS method on datasets B and C.Because the available ZBS implementation required equal spatial resolutions, the HSI was upsampled before comparison.
- Results: The evaluations include false-color reconstructions, per-pixel RMSE comparisons, and reflectance curves at the 10th, 50th, and 90th error percentiles.For dataset B, the RMSE analysis uses a 72 × 72 pixel section and omits the noisiest 1% of errors.
- Conclusions: HySure fuses hyperspectral images with either panchromatic or multispectral images and estimates the sensors’ relative spectral and spatial responses from the data.The method targets high resolution in both spatial and spectral domains.
APPENDIX
The appendix expands the SALSA optimization into explicit variable and multiplier updates. FFT-based computations dominate the implementation cost, while vector-soft thresholding solves the regularization-related subproblem.
- Algorithm overview: Figure 12 presents the optimization algorithm in more detail than its condensed description in Figure 1.The expanded formulation uses variables X, V1–V4 and multipliers A1–A4.
- Updates: The first minimization subproblem can be computed efficiently with the Fast Fourier Transform, with complexity O(Ls × nm log nm).Part of the computation, including an inverse-related term, can be precomputed before iteration.
- Updates: The V1 subproblem exploits the masking matrix M by separating the selected and unselected pixels.This decomposition is used to simplify the corresponding minimization problem.
- Updates: The V3 and V4 updates use a column-wise vector-soft threshold function and have complexity O(Ls × nm log nm).Their cost is dominated by FFTs.
- Complexity: The complete algorithm has per-iteration complexity O(Ls × nm log nm), dominated by FFT operations.The Lagrange multipliers are updated after the minimization steps.