Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model

Danfeng Hong; Jingliang Hu; Jing Yao; Jocelyn Chanussot; Xiao Xiang Zhu

Source-linked AI summary

Multimodal Remote Sensing Benchmark Datasets for Land Cover Classification with A Shared and Specific Feature Learning Model

Danfeng Hong, Jingliang Hu, Jing Yao, Jocelyn Chanussot, Xiao Xiang Zhu

arXiv:2105.10196v1May 21, 2021cs.CV

TL;DR

Multimodal remote sensing classification lacks both effective representations across heterogeneous modalities and sufficient benchmark datasets. The paper proposes S2FL, which separates shared and modality-specific features, and releases three multimodal benchmarks; experiments report superior classification performance against state-of-the-art baselines.

Problem
Heterogeneous multimodal RS data are difficult to embed into consistent, compact, accurate, and discriminative representations, while benchmark datasets remain scarce.
Method
S2FL decomposes multimodal RS data into shared and modality-specific representations and uses an ADMM-based optimization framework.
Results
S2FL obtains the best classification performance among compared advanced MFL methods across the Houston2013, Berlin, and Augsburg benchmark experiments.
Takeaways & Limitations
The three released benchmarks support multimodal land cover classification evaluation across homogeneous, heterogeneous, and three-modality RS data.
Takeaways & Limitations
In one Augsburg ablation, ℓ2-CoSpace performs slightly better with HS+SAR than with HS+SAR+DSM, indicating limited fusion as modalities increase.

Abstract

from arXiv · show

As remote sensing (RS) data obtained from different sensors become available largely and openly, multimodal data processing and analysis techniques have been garnering increasing interest in the RS and geoscience community. However, due to the gap between different modalities in terms of imaging sensors, resolutions, and contents, embedding their complementary information into a consistent, compact, accurate, and discriminative representation, to a great extent, remains challenging. To this end, we propose a shared and specific feature learning (S2FL) model. S2FL is capable of decomposing multimodal RS data into modality-shared and modality-specific components, enabling the information blending of multi-modalities more effectively, particularly for heterogeneous data sources. Moreover, to better assess multimodal baselines and the newly-proposed S2FL model, three multimodal RS benchmark datasets, i.e., Houston2013 -- hyperspectral and multispectral data, Berlin -- hyperspectral and synthetic aperture radar (SAR) data, Augsburg -- hyperspectral, SAR, and digital surface model (DSM) data, are released and used for land cover classification. Extensive experiments conducted on the three datasets demonstrate the superiority and advancement of our S2FL model in the task of land cover classification in comparison with previously-proposed state-of-the-art baselines. Furthermore, the baseline codes and datasets used in this paper will be made available freely at https://github.com/danfenghong/ISPRS_S2FL.

1. Introduction

The paper addresses limited multimodal representation capability and benchmark availability in remote sensing land cover classification. It introduces S2FL and three benchmark datasets to support more effective feature fusion and evaluation.

Challenges: Multimodal RS methods remain limited in representing heterogeneous data, constraining performance gains in land cover classification.The paper identifies inadequate multimodal representation as a central challenge.
Challenges: The scarcity of multimodal RS benchmark datasets limits methodological development and practical land cover classification applications.Different sensors, resolutions, structures, and imaging conditions complicate data acquisition and processing.
Challenges: Concatenation-based feature extraction can inadequately fuse modalities and damage original modality components, especially for heterogeneous data.The problem arises from highly coupled information across modalities.
Contributions: The paper builds three labeled multimodal RS datasets and proposes S2FL, which decomposes modalities into shared and specific representations for compact, discriminative fusion.The datasets span different resolutions, modalities, and sensors.
Optimization: An ADMM-based optimization framework is customized to obtain fast and accurate solutions for S2FL.The paper presents the optimization process alongside the model methodology.

2. Related Work

Related work distinguishes image-level fusion from feature-level multimodal learning, with existing methods organized mainly around concatenation and alignment strategies. The paper highlights limits in exploiting complementary heterogeneous-modal information.

Fusion strategies: Image-level fusion enhances spatial or spectral information in homogeneous data but does not meet heterogeneous fusion demands.Feature-level learning and fusion are therefore needed for heterogeneous remote sensing data.
Concatenation-based MFL: Concatenation-based models stack multimodal images before feature extraction or stack separately learned modality features before classification.These are the two main concatenation patterns described.
Concatenation-based MFL: Morphological operators and graph-embedding methods are established examples of concatenation-based multimodal feature extraction and classification.The related work cites applications involving hyperspectral and LiDAR data.
Concatenation-based MFL: Concatenation-based MFL has developed substantially, yet remains limited in fully exploiting diverse information, particularly from heterogeneous modalities.This limitation motivates alternative multimodal feature learning designs.
Alignment-based MFL: Alignment-based MFL learns a common feature set shared across modalities using manifold-alignment techniques.Prior work includes semi-supervised alignment and kernel-space projection approaches.

3.1. Method Overview

S2FL learns multimodal representations by separating shared information from modality-specific information. Its design aligns common components on a latent manifold while preserving modality-specific content.

Method overview: S2FL seeks discriminative features by disentangling multimodal data sources into shared and specific domains.The resulting feature space is intended to support better classification decision boundaries.
Method overview: Figure 1 illustrates the learning process for shared and specific subspaces or features in S2FL.The illustration concerns the model’s to-be-estimated projection variables.
Method overview: The model aligns shared components between modalities on a latent manifold subspace while simultaneously separating specific information.This strategy is presented as interpretable and effective for multimodal RS feature representation.

3.2. Notation

The notation defines modality-specific input matrices, labels, projection matrices, and regression parameters used by S2FL. Table 1 organizes these variables for a bi-modality example.

Variables: X_k represents the unfolded data matrix for modality k, with d_k channels and N pixels, while K is the number of modalities.The notation specifies X_k ∈ R^{d_k×N}.
Variables: Y is the C×N one-hot label matrix, where C denotes the number of categories.The same label information is used across modality groups.
Variables: Θ_0 is the shared subspace projection, Θ_k is the specific projection for modality k, and d_s is the feature dimension.These projections map modality data into shared and specific feature spaces.
Variables: P is the regression matrix connecting learned subspaces with label information Y.The regression matrix has size C×d_s.
Variables: Table 1 summarizes the S2FL variables, descriptions, and sizes using K = 2 as the bi-modality example.The notation also defines the Laplacian construction through W and D.

3.3. Problem Formulation

S2FL formulates multimodal feature learning through shared and modality-specific projections, with graph-based manifold alignment, regularization, and orthogonality constraints to support discriminative fusion.

Projection design: The model combines the shared and specific projections into a composite projection for multimodal representation learning.
Constraints and regularization: Frobenius-norm regularization on P stabilizes convergence and is intended to improve the model’s generalization ability.
Projection design: S2FL models multimodal data using one shared projection and K modality-specific projections to separate common and unique information.The shared component is learned through Θ0, while {Θk} captures modality-specific information.
Graph construction: The Laplacian regularization tr(Θ0 eXL(Θ0 eX)⊤) aligns shared components across modalities on a joint graph.The graph contains intra-modality subgraphs from a Gaussian kernel and inter-modality connections constructed from label information.
Graph construction: The adjacency matrix uses Gaussian-kernel similarities within modalities, while class information constructs the inter-modality graph structure.The Gaussian kernel has width parameter σ, and NC denotes the number of samples in class C.
Constraints and regularization: Orthogonality constraints are imposed on the shared and specific projections to reduce freedom and shrink the solution space.

3.4. Model Optimization

S2FL solves its non-convex formulation by alternating over separable subproblems for regression, shared projection, and modality-specific projections, using an ADMM-based solver for the shared subspace.

Global optimization: The non-convex S2FL objective is optimized by alternating over convex subproblems for P, Θ0, and the modality-specific projections.Algorithm 1 provides the global alternating-minimization process.
Global optimization: The P update is a least-square regression problem with common Tikhonov–Phillips regularization and an analytical solution.
Global optimization: Algorithm 1 iteratively updates P, the shared projection Θ0, and each modality-specific projection Θk until the objective-loss change falls below ζ.Initialization uses LPP, and the reported tolerance is ζ = 10^-4.
Shared projection optimization: The shared-projection subproblem introduces auxiliary variables H and G, then applies ADMM to enforce reconstruction and orthogonality-related constraints.The augmented Lagrangian uses multipliers Λ1 and Λ2 and regularization parameter µ.
Specific projection optimization: The specific projections use the same solver as the shared projection, but omit the manifold-alignment term from their optimization problem.

3.5. Convergence Analysis and Computational Cost

The paper discusses convergence of the alternating ADMM-based optimization, reports convergence curves across three benchmark datasets, and characterizes computational cost as dominated by matrix algebra and SVD operations.

Convergence: The global optimization uses alternating minimization, while the ADMM subsolver is a variant of inexact augmented Lagrange multiplier optimization.The paper notes that strict mathematical convergence proofs for the multi-block setting still require improvement.
Convergence: Relative objective loss is recorded at each iteration to produce convergence curves for S2FL on Houston2013, Berlin, and Augsburg.
Computational cost: The optimization cost is mainly dominated by classic matrix operations, including matrix multiplication and matrix inversion.
Computational cost: Updating G requires a singular value decomposition, adding an SVD-dominated cost to the shared-subproblem optimization.
Computational cost: The modality-specific projection updates have the same complexity as Algorithm 2’s shared-subproblem solver.

4. Experiments

The experiments evaluate multimodal feature learning for land cover classification using three benchmark datasets spanning homogeneous and heterogeneous sensor combinations. S2FL features are assessed with nearest-neighbor classification against established methods under quantitative metrics including OA, AA, and κ.

Datasets: Three benchmark datasets combine HS-MS Houston2013, HS-SAR Berlin, and HS-SAR-DSM Augsburg data for multimodal land cover classification.The datasets include two-modality homogeneous and heterogeneous cases, plus a three-modality heterogeneous case.
Datasets: The Houston2013 dataset uses HS and MS imagery to combine high spectral and spatial information.The original HS image covers 349 × 1905 pixels with 144 channels spanning 0.38µm to 1.05µm.
Datasets: The Augsburg dataset combines spaceborne HS, dual-Pol PolSAR, and DSM imagery after downsampling all images to a unified 30m GSD.The three sources were collected by HySpex, Sentinel-1, and the DLR-3K system, respectively.
Evaluation: Performance is measured with Overall Accuracy, Average Accuracy, and Kappa Coefficient using a nearest-neighbor classifier.The classifier is selected to attribute performance gains to learned features rather than advanced classifiers such as SVM, RF, or deep learning models.
Compared methods: The experiments compare S2FL with state-of-the-art multimodal feature-learning methods, including JDR-PCA, SMA, USMA, ℓ2-CoSpace, and ℓ1-CoSpace.Method parameters are selected using 10-fold cross-validation on the training set.

4.3. Results and Analysis on Houston2013 Datasets

On Houston2013, multimodal learning and shared-specific feature decomposition improve classification over single-modality and competing feature-learning approaches. S2FL also produces detailed classification maps, stable performance over a proper parameter range, and gains from modality-specific features and its orthogonal constraint.

Quantitative results: S2FL achieves around a 5% points gain in OA over ℓ1-CoSpace, which ranks second on the Houston2013 datasets.The model also performs best in many categories, including Soil, Commercial, Road, Highway, Railway, Tennis Court, and Running Track.
Quantitative results: Joint HS+MS use outperforms single modalities in OA, AA, and κ, while MA-based approaches perform relatively poorly under complex noise.SMA exceeds USMA by about 2% points OA through supervised information, and ℓ1-CoSpace improves on ℓ2-CoSpace by approximately 2% points.
Classification maps: S2FL classification maps retain more geometric information for man-made materials and more textural information for Grass and Tree.The visual comparison follows the same overall trend as the quantitative results in Table 5.
Parameter sensitivity: The optimal S2FL parameter combination is (q, σ, α, β, d) = (45, 100, 10−2, 100, 30), consistent with cross-validation results.Performance remains stable when the parameters are selected within a proper range.
Ablation analysis: Modality-specific features outperform modality-shared features, with OA of 83.11 versus 78.77 on Houston2013.Multimodal input exceeds single-modality input, and removing the orthogonal constraint causes dramatic performance degradation.
Cross-modality analysis: Learned features outperform original HS or MS spectral features under single-modality and cross-modality settings.Table 8 compares direct original features, single-modality S2FL, and S2FL cross-modality cases.

4.6. Results and Analysis on Berlin Datasets

The experiments compare multimodal feature-learning methods on heterogeneous Berlin data and extend the analysis to three-modality Augsburg data. Across these settings, S2FL achieves the strongest reported classification performance while producing smoother, more meaningful maps.

Berlin datasets: Berlin’s heterogeneous HS-SAR data sharply reduce classification performance relative to Houston2013, although combining HS and SAR still outperforms either single modality.The passage attributes weaker performance to the difficulty of heterogeneous data fusion.
Berlin datasets: S2FL improves over ℓ1-CoSpace by 3.39% points in OA, 3.83% in AA, and 3.06% in κ on Berlin.These are the reported gains for S2FL over ℓ1-CoSpace.
Berlin datasets: S2FL reduces noisy pixels and generates smoother classification maps by blending HS and SAR information through shared and specific feature learning.The visual improvement is especially noted for Forest and Low Plants.
Augsburg datasets: Jointly using HS, SAR, and DSM generally improves OA by around 4% over using the modalities independently on Augsburg.The three-modality evaluation also reports that S2FL exceeds ℓ1-CoSpace by approximately 2 percentage points in OA.
Augsburg datasets: S2FL produces more realistic and smoother land-cover maps, particularly for Forest, Residential Area, and Low Plants.The maps are described as showing semantically meaningful structure.
Ablation study: Across two- and three-modality combinations, S2FL’s shared-specific decoupling yields higher classification performance than ℓ2-CoSpace and ℓ1-CoSpace.The ablation findings also report that ℓ1-CoSpace outperforms ℓ2-CoSpace across different modality combinations.

5. Conclusion

The paper releases three multimodal remote-sensing benchmark datasets and proposes S2FL for learning shared and modality-specific representations. S2FL achieves the best classification performance across the three datasets compared with previous advanced feature-learning methods.

Conclusion: The paper introduces homogeneous HS-MS Houston2013, heterogeneous HS-SAR Berlin, and three-modality HS-SAR-DSM Augsburg benchmark datasets.The datasets are intended for multimodal land-cover classification and future open release.
Conclusion: S2FL learns shared-modality and specific-modality representations to produce more discriminative and compact multimodal feature blending.The model is presented as the paper’s novel multimodal feature-learning approach.
Conclusion: S2FL obtains the best classification performance on all three datasets compared with previously proposed advanced multimodal feature-learning methods.The conclusion characterizes its performance as obviously superior to other competitors.
Conclusion: The authors plan to extend the datasets to a larger scale and develop corresponding feature-learning models in future work.This identifies the stated scope of the paper’s future development.