Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification

Danfeng Hong; Naoto Yokoya; Nan Ge; Jocelyn Chanussot; Xiao Xiang Zhu

Source-linked AI summary

Learnable Manifold Alignment (LeMA) : A Semi-supervised Cross-modality Learning Framework for Land Cover and Land Use Classification

Danfeng Hong, Naoto Yokoya, Nan Ge, Jocelyn Chanussot, Xiao Xiang Zhu

arXiv:1901.02838v1Jan 9, 2019cs.CV

TL;DR

The paper asks whether limited high-quality hyperspectral data can improve classification using abundant, lower-quality multispectral data. It proposes LeMA, which jointly learns a common subspace and data-driven graph for semi-supervised cross-modality transfer, and reports superior knowledge-transfer effectiveness while acknowledging limitations on heterogeneous data.

Problem
Limited labeled data and the high cost of hyperspectral acquisition motivate using abundant, poorly discriminative multispectral data in cross-modality classification.
Method
LeMA jointly learns a common subspace, label connection, and adaptive graph structure, with ADMM-based optimization for the proposed model.
Results
LeMA demonstrates superiority and effectiveness in knowledge transfer across two homologous simulated datasets and one heterogeneous real dataset compared with state-of-the-art algorithms.
Takeaways & Limitations
The learned graph and common subspace support semi-supervised transfer between spectrally rich hyperspectral and spectrally poor multispectral data.
Takeaways & Limitations
Knowledge transfer remains limited for heterogeneous modalities because the listed methods, including LeMA, use a linearized model that does not fit the modality gap well.

Abstract

from arXiv · show

In this paper, we aim at tackling a general but interesting cross-modality feature learning question in remote sensing community --- can a limited amount of highly-discrimin-ative (e.g., hyperspectral) training data improve the performance of a classification task using a large amount of poorly-discriminative (e.g., multispectral) data? Traditional semi-supervised manifold alignment methods do not perform sufficiently well for such problems, since the hyperspectral data is very expensive to be largely collected in a trade-off between time and efficiency, compared to the multispectral data. To this end, we propose a novel semi-supervised cross-modality learning framework, called learnable manifold alignment (LeMA). LeMA learns a joint graph structure directly from the data instead of using a given fixed graph defined by a Gaussian kernel function. With the learned graph, we can further capture the data distribution by graph-based label propagation, which enables finding a more accurate decision boundary. Additionally, an optimization strategy based on the alternating direction method of multipliers (ADMM) is designed to solve the proposed model. Extensive experiments on two hyperspectral-multispectral datasets demonstrate the superiority and effectiveness of the proposed method in comparison with several state-of-the-art methods.

1. Introduction

The paper addresses cross-modality classification when limited, discriminative hyperspectral data must support abundant, poorly discriminative multispectral data. It proposes LeMA, which learns a data-driven graph and aligned feature subspace for semi-supervised transfer.

Multispectral imagery is widely available, but labeling large-scale, discriminative, high-quality samples is difficult and time-consuming.
Manifold alignment learns a shared subspace for joint feature representation, but conventional methods connect that subspace weakly to labels.
Existing graph-based semi-supervised methods often define unlabeled-sample topology with a fixed Gaussian kernel computed in the original space.
The framework combines a semi-supervised cross-modality model with ADMM optimization and is evaluated on two HS-MS datasets.
LeMA learns an aligned feature subspace from labeled HS-MS correspondences and partially unlabeled MS data, supporting out-of-sample identification in either modality.
LeMA learns graph structure directly from data rather than using a Gaussian kernel, strengthening transfer and generalization.

2. Learnable Manifold Alignment (LeMA)

LeMA addresses cross-modality classification by learning a common HS-MS subspace and a joint graph from labeled and unlabeled data. Its alternating optimization framework estimates projections, regression coefficients, and adjacency structure under manifold and sparsity constraints.

Problem Statement and Motivation: LeMA targets cross-modal learning where limited HS training data supports classification of large-scale MS data.The framework combines cross-modality learning with semi-supervised use of abundant unlabeled MS samples.
Problem Formulation: LeMA extends CoSpace by learning a joint Laplacian, then reformulates the problem as optimization of a symmetric, nonnegative joint adjacency matrix.The adjacency formulation uses pairwise Euclidean distances and weighted sparsity constraints.
Problem Formulation: Landmark cluster centers can represent unlabeled samples, preserving available information while reducing computational cost.The unlabeled MS samples are selected from samples outside the training set.
Problem Statement and Motivation: The method learns common projections for HS and MS data while jointly estimating regression coefficients to connect the subspace with label information.The learned common subspace is intended to absorb complementary detail and structural information from the two modalities.
Model Optimization: ADMM-based alternating optimization updates P, Θ, and fW through convex subproblems, with auxiliary variables J and G replacing ΘeX and Θ.The implementation iterates variable updates, multiplier updates, penalty adjustment, and convergence checks.

3. Experiments

The experiments evaluate LeMA on simulated HS-MS datasets from University of Houston and Chikusei and on a real multispectral-lidar/hyperspectral dataset. Classification performance is assessed with LSVM and CCF using OA, AA, and κ, against several comparison methods and the original MS baseline.

Datasets and Classifiers: Experiments cover simulated University of Houston and Chikusei HS-MS datasets plus a real DFC2018 multispectral-lidar and hyperspectral dataset.The evaluation uses classification with linear support vector machines and canonical correlation forests.
Datasets and Splits: Table 1 reports the numbers of training and testing samples for the two used MS-HS datasets.The table specifies the experimental sample allocation for those datasets.
Evaluation: Performance is quantified using overall accuracy (OA), average accuracy (AA), and kappa coefficient (κ).These indices are used for quantitative assessment of classification performance.
Baselines: LeMA is compared with GLP, SMA, S-SMA, CoSpace, Semi-supervised CoSpace, and the original MS data baseline.The comparison includes methods using label-based or Euclidean-similarity joint graphs.

3.1. The Simulated MS-HS Datasets over the University of Houston

The University of Houston experiments use simulated multispectral data derived from hyperspectral imagery, with partially overlapping modalities and reassigned training and test labels. Across visual and quantitative evaluations, manifold alignment methods outperform baseline and GLP, while LeMA achieves the strongest reported performance by learning graph connectivity, label information, and unlabeled structure jointly.

Data Description: The hyperspectral data comprise 144 bands, while Sentinel-2 spectral-response filtering generates a 10-band multispectral image with dimensions 349 × 1905 × 10.The hyperspectral image has size 349 × 1905 and a 2.5 m ground sampling distance.
Data Description: The experimental setup uses a hyperspectral image partially covering a whole multispectral image, with labels reassigned into training and testing regions containing all classes.The training region is selected from available labels, and the remaining labels form the test set.
Experimental Setup: Classification maps and quantitative assessments compare the algorithms using linear support vector machines and canonical correlation forest classifiers.The quantitative results use optimal parameters obtained through 10-fold cross-validation.
Results and Analysis: Manifold alignment methods outperform baseline and GLP across classifiers, indicating that limited hyperspectral data can guide multispectral data toward more discriminative representations.SMA performs relatively poorly compared with S-SMA because it considers only correspondences between labeled multispectral and hyperspectral data.
Results and Analysis: LeMA achieves much higher performance than the other methods by jointly modeling common-subspace connectivity, labels, and unlabeled information through a learned graph.The paper associates this learned representation with a more discriminative feature space and a better decision boundary.
Results and Analysis: Semi-supervised methods outperform their supervised counterparts because unlabeled data help address the few and highly unbalanced training samples.The paper reports particularly noticeable improvements for classes such as Stressed Grass and Water after incorporating unlabeled samples.

3.2. The Simulated MS-HS Datasets over Chikusei

The Chikusei experiments use simulated multispectral data derived from hyperspectral imagery and evaluate competing methods quantitatively and visually. The results show that poor multispectral spectral information and limited labels hinder baseline and GLP, motivating comparison with cross-modality alignment methods.

Data Description: The Chikusei multispectral data have dimensions 2517 × 2335 × 10 at 2.5 m GSD and are simulated from hyperspectral data containing 128 bands from 363 nm to 1018 nm.The hyperspectral spectral resolution is 10 nm.
Experimental Setup: The Chikusei setup selects an overlapping multispectral-hyperspectral region containing all listed classes and uses its training and test labels for evaluation.Parameters are configured by 10-fold cross-validation on the training set.
Experimental Setup: Visual and quantitative comparisons evaluate the algorithms using classification maps and the performance results reported in Table 3.The classification maps are produced with two kinds of classifiers.
Results and Analysis: The baseline fails to identify several materials because of poor multispectral spectral information and limited training samples, while GLP remains limited by low-discriminative spectral signatures.Reportedly affected classes include Plastic House, Manmade (Dark), Rice Field (Grown), Bare Soil (Farmland), and Forest.
Results and Analysis: The study additionally evaluates a heterogeneous real multispectral-Lidar and hyperspectral DFC2018 dataset after noting that Houston and Chikusei originate from similar data sources.The shared source characteristics may limit practical generalization despite enabling more effective modality transfer.

3.3. The Real Multispectral-Lidar and Hyperspectral Datasets in DFC2018

The DFC2018 experiments test whether limited hyperspectral information improves multispectral-lidar classification on heterogeneous real data. LeMA achieves superior performance among the compared methods, while linear modeling limits transfer across heterogeneous modalities.

Dataset: The DFC2018 dataset combines multispectral-lidar imagery with hyperspectral data from different modalities and spatial resolutions.The multispectral-lidar imagery has seven bands at 0.5m GSD, while the hyperspectral data has 48 bands at 1m GSD.
Experimental setup: The experiment uses 10% of labeled samples for training, with the remaining samples reserved for testing.Parameters are selected by 10-fold cross-validation on the training set.
Experimental setup: Averaged results from 10 runs provide the performance comparison, while Fig. 8 visualizes classification-map differences.Random train-test assignments motivate averaging across repeated runs.
Results and analysis: Hyperspectral information improves multispectral-lidar classification, indicating that common-subspace methods transfer knowledge between modalities to some extent.This comparison includes SMA, S-SMA, CoSpace, S-CoSpace, and LeMA.
Results and analysis: LeMA achieves superior performance by combining rich spectral information and unlabeled samples with a learned graph structure.The authors associate the learned graph with better data-distribution capture and a potential optimal decision boundary.
Limitations: Knowledge transfer remains limited on heterogeneous data because all compared methods, including LeMA, use linear modeling.The paper notes that a single linear transformation does not fit the gap between heterogeneous modalities well.

4. Conclusions

The paper addresses whether limited high-quality hyperspectral data can improve classification using abundant low-quality data. It proposes LeMA to connect the common subspace with labels and adaptively learn unlabeled-data structure, while acknowledging limitations from linear modeling on highly nonlinear heterogeneous data.

Conclusion: LeMA investigates whether limited high-quality data can improve classification using abundant low-quality data.The motivating examples are hyperspectral and multispectral data, respectively.
Conclusion: LeMA connects the common subspace with label information and embeds unlabeled information by adaptively learning a Laplacian matrix.The framework is evaluated on homologous simulated MS-HS datasets and a heterogeneous real dataset.
Conclusion: Experiments demonstrate LeMA's superiority and effectiveness in knowledge transfer compared with other state-of-the-art algorithms.The experiments cover two homologous simulated datasets and one heterogeneous multispectral-lidar and hyperspectral dataset.
Limitations and future work: LeMA's representative ability remains limited by linear modeling, especially for highly nonlinear heterogeneous data.Future work targets a nonlinear version and incorporation of spatial information.