Source-linked AI summary
CoSpace: Common Subspace Learning from Hyperspectral-Multispectral Correspondences
Danfeng Hong, Naoto Yokoya, Jocelyn Chanussot, Xiao Xiang Zhu
TL;DR
Multispectral imagery provides broad coverage but limited spectral discrimination, while hyperspectral imagery offers richer spectra over narrower areas. CoSpace jointly learns a label-informed shared subspace from partial HS-MS correspondences, and experiments on University of Houston and Chikusei data show superior classification performance over compared methods. Its linear modeling is limited for heterogeneous data sources, and the study assumes shared land-cover classes across HS and MS images.
Problem
Multispectral imagery has broad coverage but limited spectral information, whereas hyperspectral imagery is more discriminative but has narrower spatial coverage.
Method
CoSpace jointly learns a common latent subspace and supervised classifier by locally aligning HS-MS manifolds from partial correspondences.
Results
CoSpace achieves higher classification accuracy than baseline, P-JDR, L-USMA, and L-SMA across the evaluated classifiers and datasets.
Takeaways & Limitations
Limited HS coverage can be used to transfer rich spectral information into MS data through a learned common subspace for large-area classification.
Takeaways & Limitations
CoSpace is limited in handling heterogeneous data sources because its modeling is linear, and it assumes HS and MS images share the same land-cover classes.
Abstract
from arXiv · showhide
With a large amount of open satellite multispectral imagery (e.g., Sentinel-2 and Landsat-8), considerable attention has been paid to global multispectral land cover classification. However, its limited spectral information hinders further improving the classification performance. Hyperspectral imaging enables discrimination between spectrally similar classes but its swath width from space is narrow compared to multispectral ones. To achieve accurate land cover classification over a large coverage, we propose a cross-modality feature learning framework, called common subspace learning (CoSpace), by jointly considering subspace learning and supervised classification. By locally aligning the manifold structure of the two modalities, CoSpace linearly learns a shared latent subspace from hyperspectral-multispectral(HS-MS) correspondences. The multispectral out-of-samples can be then projected into the subspace, which are expected to take advantages of rich spectral information of the corresponding hyperspectral data used for learning, and thus leads to a better classification. Extensive experiments on two simulated HSMS datasets (University of Houston and Chikusei), where HS-MS data sets have trade-offs between coverage and spectral resolution, are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.
I. INTRODUCTION
Multispectral imagery offers broad coverage but limited spectral discrimination, while hyperspectral imagery provides richer spectra over narrower swaths. CoSpace addresses this cross-modal setting by jointly learning a label-informed common subspace from partial HS-MS correspondences.
- Motivation: Multispectral imagery supports large-scale mapping but struggles to discriminate spectrally similar classes, whereas hyperspectral imagery offers richer spectral information with narrower coverage.This coverage–spectral-resolution trade-off motivates transferring information from limited HS areas to broader MS imagery.
- Related Work: Existing multimodal methods include fusion-based joint feature learning and alignment-based shared feature learning, with fusion approaches typically requiring complete data correspondence.These approaches have been applied to classification, detection, image analysis, spectral unmixing, and HS-MS fusion.
- Problem Setting: Cross-modal learning uses one modality for testing while learning from multimodal or single-modality data, unlike conventional multimodal learning.In the MS-HS setting, limited overlapping HS-MS correspondences are used to transfer learned representations to the remaining MS data.
- Problem Gap: Prior methods may connect modalities or instances, but weak connections between the common subspace and labels can limit the discriminative power of low-dimensional features.The paper identifies this label-information gap as a central obstacle for cross-modal feature learning.
- Proposed Approach: CoSpace jointly performs subspace learning and supervised classification to bridge learned features with labels in HS-MS cross-modal learning.It learns a shared representation from partial correspondences and uses ADMM to optimize the proposed model.
2 Initializating P and Θ
The method initializes and updates its latent projections and label-connection variables through an iterative optimization procedure. CoSpace uses a shared latent subspace to transfer information between corresponding HS and MS data.
- Initialization and Optimization: The objective is solved iteratively by fixing other variables and updating P, while convergence is checked against the maximum iteration limit.The algorithm computes the objective value at each iteration before checking termination.
- Initialization and Optimization: CoSpace learns a HS-MS common subspace so data from one domain can be adaptively transferred to the other.The workflow is designed to use limited HS coverage to improve classification across the MS area.
- Initialization and Optimization: The projection matrices connect MS and HS observations to a d-dimensional latent subspace, while P connects that subspace to one-hot label information.XM and XH contain the observed MS and HS image data, respectively, and Y is the label matrix.
- Initialization and Optimization: The model duplicates the label matrix across the paired modalities and represents noise and other errors with a residual matrix E.The augmented label matrix is eY = [Y, Y], and the projection blocks are combined as [ΘM, ΘH].
1 Initialization: Θ = 0, G = 0, Λ1 = 0, Λ2 = 0,
CoSpace imposes regularization and graph structure on its constrained optimization model, with label-derived adjacency weights defining a joint manifold relationship.
- Initialization: ADMM initialization sets µ = 10^-3, µmax = 10^6, ρ = 1.5, ε = 10^-6, and t = 1 before iterative updates.
- Model formulation: CoSpace is formulated as a constrained optimization problem after introducing regularization for its ill-posed latent-subspace estimation.The model includes orthogonality and regularization constraints.
- Model formulation: The classifier-related variable P is regularized with a Frobenius norm to support reliable model generalization.
- Model formulation: The projection matrix Θ uses multi-modal manifold alignment regularization based on a joint graph structure.
- Joint graph construction: The joint Laplacian is L = D − W, with adjacency matrix W inferred directly from label information.The adjacency construction assigns class-based weights to corresponding samples.
B. Model Optimization
CoSpace optimizes its variables alternately by solving convex subproblems, using closed-form least-squares optimization for P and ADMM for Θ.
- B. Model Optimization: Because the model is nonconvex, CoSpace adopts iterative alternating optimization over convex subproblems for P and Θ.The procedures are repeated until a stopping criterion is satisfied.
- Optimization with respect to P: The P subproblem is a least-squares problem with Tikhonov regularization and has a closed-form solution.The associated projected representation is denoted Q = Θ eX.
- Optimization with respect to Θ: The Θ subproblem is reformulated with auxiliary variables J and G replacing Θ eX and Θ, respectively.
- Optimization with respect to Θ: ADMM solves the reformulated Θ problem using an augmented Lagrangian with Lagrange multipliers and a penalty parameter.Algorithm 2 summarizes the subproblem updates.
C. Convergence Analysis
The alternating optimization is a block coordinate descent method with a theoretical convergence guarantee under strict convexity and exact minimization of each subproblem.
- C. Convergence Analysis: The alternating strategy is block coordinate descent, whose convergence is theoretically guaranteed when every subproblem is strictly convex and exactly minimized.
- C. Convergence Analysis: Objective-function values are recorded across iterations to experimentally illustrate CoSpace convergence on both HS-MS datasets.The analysis is presented in Fig. 4.
III. EXPERIMENTS
The experiments evaluate CoSpace on University of Houston and Chikusei HS-MS datasets, using classification to assess transferability of learned features.
- III. EXPERIMENTS: CoSpace is quantitatively and qualitatively evaluated on HS-MS datasets from University of Houston and Chikusei.
- III. EXPERIMENTS: Classification is used as a potential application for validating transferability of features learned by CoSpace.
- III. EXPERIMENTS: The evaluation compares nearest neighbor, linear support vector machines, and canonical correlation forest classifiers.The classifiers use Euclidean distance, linear SVMs, and CCF, respectively.
A. University of Houston HS-MS Datasets
The University of Houston HS-MS dataset combines a complete Sentinel-2-simulated multispectral image with a partially overlapping hyperspectral image and redistributed labels. Performance is assessed using OA, AA, and κ across three classifiers under cross-validated parameter settings.
- Dataset construction: The University of Houston hyperspectral scene contains 349 × 1905 pixels and 144 bands from 364 nm to 1046 nm at 2.5 m GSD.Sentinel-2 spectral response functions generate a 349 × 1905 × 10 multispectral image.
- Dataset construction: Training and testing samples are redistributed to represent abundant complete multispectral data paired with limited incomplete hyperspectral data.The hyperspectral image partially covers the multispectral scene.
- Evaluation: The evaluation uses Overall Accuracy, Average Accuracy, and the Kappa Coefficient to quantify classification performance.OA measures correctly classified multispectral test samples, AA averages per-class accuracies, and κ measures agreement while accounting for chance.
- Evaluation: Three classifiers—1NN, LSVM, and CCF—are considered for the experiments.Algorithm parameters, including subspace dimension and regularization terms, are selected using 10-fold cross-validation on training data.
- Evaluation: Table III reports quantitative comparisons on the University of Houston data, with the best result identified in bold.The compared algorithms include the proposed method and prior methods under the evaluated classifiers.
3) Results and Analysis:
Across the evaluated classifiers, CoSpace generally improves classification by combining modality alignment with label information, while performance depends on training-set size and remains uneven across classes. The experiments also examine visual classification differences and data imbalance in the University of Houston and Chikusei settings.
- Classification results: ASFL approaches generally outperform the multispectral-only baseline, with L-SMA broadly stronger than P-JDR and L-USMA before CoSpace.L-USMA shows stable results across classifiers, while P-JDR is slightly below the baseline with CCF.
- Classification results: CoSpace achieves the best classification accuracy by aligning modalities while also bridging the learned latent subspace with label information.It increases OA over the baseline, P-JDR, L-USMA, and L-SMA across 1NN, LSVM, and CCF settings.
- Classification results: CoSpace improves OA over the baseline by 7.12% with 1NN, 7.26% with LSVM, and 3.96% with CCF.The cited comparisons are for the University of Houston dataset and use the baseline as the reference.
- Class-specific behavior: CoSpace performs especially well on small-scale Residential and Railway classes but fails to effectively identify Parking Lot2.The training data are highly imbalanced because samples come from a very limited area.
- Visual analysis: In a cloud-shadowed subarea, methods using hyperspectral information produce more discriminative features, while CoSpace better identifies vegetation, Residential, and Commercial materials.These classes are described as easily misclassified by traditional methods in the highlighted area.
- Sensitivity analysis: Classification accuracy increases with training-set size, improving faster initially and then largely stabilizing across classifiers.CoSpace is evaluated with training subsets ranging from 5% to 100% while the test set remains fixed.
2) Experimental Setup:
The Chikusei experiments compare CoSpace with alternative algorithms using quantitative metrics and classification maps under three classifiers. CoSpace improves recognition of challenging, spectrally similar classes, while performance depends on training-set size and the shared-class assumption.
- Experimental Setup: The Chikusei evaluation reports OA, AA, and κ for competing algorithms and visualizes full-scene classification maps under three classifiers.The quantitative results are summarized in Table IV, while corresponding maps and highlighted sub-areas appear in Fig. 8.
- Results and Analysis: 4.46% and 5.49% accuracy gains over the baseline are reported for P-JDR and L-USMA, respectively, with 1NN on Chikusei.The gains are described for the challenging Chikusei scene with inhomogeneous training-sample distribution.
- Results and Analysis: CoSpace obtains higher classification accuracy than the other algorithms by aligning modalities in a common subspace and connecting that subspace to training labels.The projected features can absorb properties from both modalities, according to the authors.
- Results and Analysis: CoSpace more effectively recognizes similar classes such as Bare Soil (Farmland) versus Row Crops and Weeds in Farmland versus Rice Field (Grown).The highlighted comparison attributes this performance to transferring hyperspectral information into multispectral data through joint subspace learning and classification.
- Limitations: The experiments assume that MS and HS images share the same land-cover classes, although large-scale MS scenes may contain more classes than the overlapped HS area.The authors identify generalizing beyond this shared-class assumption as future work.
APPENDIX A SOLUTION TO PROBLEM (8) WITH RESPECT TO Θ
The appendix solves the optimization problem with ADMM by alternating updates for the variables and multipliers. The constrained G update is handled by splitting the orthogonality constraint into two steps.
- ADMM Reformulation: ADMM reformulates problem (8) as problem (10), then successively minimizes the objective with respect to Θ, J, G, Λ1, and Λ2.The objective is convex in each variable separately when the others are fixed, although it is not jointly convex.
- Optimization with Respect to Θ: The Θ subproblem is written explicitly and has a closed-form solution.Its objective includes penalty terms involving J − Θ eX and G − Θ.
- Optimization with Respect to J: The J variable is estimated by solving its subproblem, which has an analytical solution.The appendix separately introduces the optimization problem for J before giving that solution.
- Optimization with Respect to G: The G update imposes the orthogonality constraint GGT = I and is solved by splitting the constraint into two steps using SOC.The constrained objective includes a penalty term for G − Θ.
- Multiplier Updates: Before the next iteration, the method updates Λ1 and Λ2 using the residuals J − Θ eX and G − Θ, then updates the penalty parameter µ.These updates complete the multiplier and penalty-variable steps of the iterative procedure.