PaperlayerView exact revision on arXiv

Source-linked AI summary

Semisupervised Manifold Alignment of Multimodal Remote Sensing Images

Devis Tuia, Michele Volpi, Maxime Trolliet, Gustau Camps-Valls

arXiv:2104.07803v1cs.CV

TL;DR

Existing alignment approaches have different limitations in handling multisensor data, strong distortions, and coregistration. SS–MA aligns manifolds using class labels while preserving their geometry; the reported examples show it compensating for deformations and outperforming an alternative in the second-spiral prediction setting.

  • Problem

    PCA fails in multisensor settings and under strong distortions, whereas (K)CCA handles those settings but requires strict coregistration.

    Source: §I · p. 4

  • Method

    SS–MA aligns remote-sensing manifolds by bringing same-class pixels together and pushing different-class pixels apart.

    Source: §V · p. 25

  • Results

    In all cases, SS–MA compensates for the present deformations, and when predicting the second spiral it always outperforms the comparison method.

    Source: A. 2D toy datasets · pp. 16–17

  • Takeaways & Limitations

    SS–MA can exploit common information from the two datasets when predicting the second spiral.

    Source: A. 2D toy datasets · p. 17

    The method requires labeled pixels in each image, a limitation the authors identify for future work.

    Source: §V · p. 26

Abstract

from arXiv · show

We introduce a method for manifold alignment of different modalities (or domains) of remote sensing images. The problem is recurrent when a set of multitemporal, multisource, multisensor and multiangular images is available. In these situations, images should ideally be spatially coregistred, corrected and compensated for differences in the image domains. Such procedures require the interaction of the user, involve tuning of many parameters and heuristics, and are usually applied separately. Changes of sensors and acquisition conditions translate into shifts, twists, warps and foldings of the image distributions (or manifolds). The proposed semisupervised manifold alignment (SS-MA) method aligns the images working directly on their manifolds, and is thus not restricted to images of the same resolutions, either spectral or spatial. SS-MA pulls close together samples of the same class while pushing those of different classes apart. At the same time, it preserves the geometry of each manifold along the transformation. The method builds a linear invertible transformation to a latent space where all images are alike, and reduces to solving a generalized eigenproblem of moderate size. We study the performance of SS-MA in toy examples and in real multiangular, multitemporal, and multisource image classification problems. The method performs well for strong deformations and leads to accurate classification for all domains.

I. INTRODUCTION

Changes in acquisition conditions and sensor properties shift image distributions, making classifiers trained on one scene unreliable on another. SS–MA addresses this portability problem by aligning image representations while preserving manifold geometry and using labeled and unlabeled samples.

I. INTRODUCTION

  • Classifier portability is difficult because acquisition, geometry, temporal, and sensor differences shift image distributions and can make direct transfer catastrophic.Analysts seek to reuse labeled samples on new scenes while minimizing new photointerpretation, including where archives lack labels.Source: §I · pp. 2–3
  • Existing alignment methods do not simultaneously handle strong distortions, different sensors, and unregistered images: PCA fails in multisensor or strongly distorted settings, while (K)CCA requires coregistration.The introduction identifies these as simultaneous requirements for a desirable alignment method.Source: §I · p. 4
  • SS–MA uses labeled samples across domains to bring manifolds closer and unlabeled-sample proximity graphs to preserve their local geometry.Its linear projection depends on both labeled and unlabeled samples, making the method semisupervised.Source: §I · p. 4
  • SS–MA aligns images without requiring coregistration, supports different dimensionalities and strong deformations, and can also perform dimensionality reduction.The method requires some labeled pixels in each domain.Source: §I · p. 5

II. SEMISUPERVISED MANIFOLD ALIGNMENT

The method projects datasets into a joint latent space designed to preserve each dataset's local geometry while bringing same-class regions together and separating different-class regions.

II. SEMISUPERVISED MANIFOLD ALIGNMENT

  • The joint latent space is designed to preserve each dataset's local geometry.Source: §II · p. 5
  • The alignment objective brings same-class regions closer and pushes different-class regions apart.Source: §II · p. 5
  • The method seeks projection functions, one per image, into a joint latent space that preserves local dataset geometry and aligns regions by class.Same-class regions are brought closer, while different-class regions are pushed apart.Source: §II · p. 5

A. Notation

The notation allows domains to differ in sample count and feature dimension, and the loss aligns them by combining manifold-geometry preservation with labeled-class attraction and repulsion. The resulting objective is a Rayleigh quotient balancing relations to minimize against relations to increase.

A. Notation

  • Each domain is represented by a data matrix containing labeled and unlabeled samples, with sample counts and feature dimensions that may differ across domains.The images need not depict the same location and may come from sensors with different spatial or spectral resolutions.Source: A. Notation · pp. 5–6

B. Semi-supervised loss function

  • The optimization is expressed as a Rayleigh quotient, balancing relations to minimize against relations to maximize.A combines G and S, while D supplies the relations to increase.Source: B. Semi-supervised loss function · pp. 6–7
  • The alignment objective minimizes within-manifold geometry and same-class distances while maximizing distances between samples of different classes.The geometry and same-class terms are combined as A = µG + S; the different-class affinity is D.Source: B. Semi-supervised loss function · p. 7
  • G preserves each domain’s local geometry by keeping graph-neighbor samples close after projection, without imposing geometric relations across domains.The domain-specific neighborhood graphs may use kNN or ε-ball similarities.Source: B. Semi-supervised loss function · p. 7
  • S pulls labeled samples of the same class together across domains, whereas D pushes labeled samples of different classes apart.The cross-domain class relations are built from labeled samples; unlabeled samples do not contribute to these class affinities.Source: B. Semi-supervised loss function · p. 8

C. Projection functions

The projection solution is obtained from a generalized eigenvalue decomposition. Its optimal matrix contains a separate domain-to-latent projection in each row block, allowing direct projection by matrix multiplication.

C. Projection functions

  • The minimization solution uses eigenvectors associated with the smallest eigenvalues of a generalized eigenvalue decomposition.The graph matrices and corresponding Laplacians are each N × N.Source: C. Projection functions · p. 9
  • Each row block of Fopt contains one domain’s projection function into the joint latent space, so a domain is projected by matrix multiplication.The row blocks correspond to the domain-specific projection functions.Source: C. Projection functions · p. 9

D. Properties

The method’s stated properties include explicit linear projections, flexible domain and band alignment, alignment without coregistered samples, and invertibility that supports cross-sensor band synthesis.

D. Properties

  • The method explicitly defines linear projection functions for projecting large datasets into a latent space optimized for joint classification.The passage identifies joint classification as the purpose of the latent-space optimization.Source: D. Properties · p. 9
  • The method imposes no restriction on the number or properties of bands to align because it uses each manifold’s geometry separately.This property concerns multisensor alignment.Source: D. Properties · p. 9
  • The method aligns an arbitrary number of domains in a common latent space without requiring a leading source domain.The passage contrasts this with PCA, TCA, and graph-based methods.Source: D. Properties · p. 10
  • The method aligns source probability distributions without requiring coregistered samples, including images from different areas or spatial resolutions.The passage distinguishes this capability from CCA.Source: D. Properties · p. 10
  • The invertible projection functions can synthesize one sensor’s bands from images acquired with another sensor.The paper states this may support the design and evaluation of new instruments.Source: D. Properties · p. 10

E. Computational complexity

The computational discussion centers on graph-Laplacian construction and a relatively small eigenproblem; the evaluation scenarios cover toy deformations and multiangular, multitemporal, and multisource images.

E. Computational complexity

  • SS–MA solves a d × d eigenproblem whose size grows linearly with the number of data sources.The graph Laplacian is computed once offline, and the authors describe eigendecomposition as noncritical for this matrix size.Source: E. Computational complexity · pp. 10–11
  • Most computational effort goes to constructing the geometry-preserving graph Laplacian Lg; the label-based Laplacians Ls and Ld are cheap to build.Their construction uses a simple convolution between label vectors.Source: E. Computational complexity · p. 10
  • Large graph Laplacians can demand substantial computation for matrix calculation and storage, as well as eigendecomposition and inversion.The text notes that memory requirements are not easily solved.Source: E. Computational complexity · p. 11

A. Datasets and evaluated scenarios

  • The toy experiment tests scaled, rotated, and translated versions of one spiral against an unchanged spiral.Each domain contains 2,000 labeled examples.Source: A. Datasets and evaluated scenarios · p. 12
  • The multiangular experiment uses five WorldView-2 views of the same Rio de Janeiro area acquired close in time, isolating angular effects as the expected source of spectral deformation.The dataset includes twelve classes.Source: A. Datasets and evaluated scenarios · p. 12
  • The other real-image scenarios assess transfer across acquisition times and places with the same sensor, and alignment across sources with different band counts.The multitemporal scenes are WorldView-2 images from Lausanne; the multisource case adds a four-band QuickBird image of Zurich to eight-band datasets.Source: A. Datasets and evaluated scenarios · p. 12

B. Experimental setup

The setup samples labeled and unlabeled pixels across domains, constructs locality and class-relationship graphs, and evaluates projections with cross-validated classification.

Experimental setup

  • The projection is learned in a domain-adaptation setting with one domain providing many labeled examples and the other domains providing limited labels.The projectors and models use labeled samples from each domain, representing all classes across the images simultaneously.Source: B. Experimental setup · p. 14
  • Unlabeled examples are selected as centroids of bisecting k-means clusters to provide uniform sampling.All domains use the same number of unlabeled samples.Source: B. Experimental setup · p. 15
  • The geometry graph uses k-nearest neighbors with k = 9, and the authors report little sensitivity to k within a reasonable range.They caution against values that over-connect or oversimplify the manifold graph.Source: B. Experimental setup · p. 15
  • A linear SVM evaluates original and projected data, with projection dimensionality and the regularization parameter cross-validated using labeled pixels.The authors use a linear classifier to assess the projection's quality and discriminative power.Source: B. Experimental setup · p. 15
  • For multisource classification, WorldView-2 is reduced to four bands so it can be evaluated alongside four-band QuickBird data.The setup otherwise cannot train a classifier using pixels from both sources simultaneously because their dimensionalities differ.Source: B. Experimental setup · p. 16
  • Reported agreement is mean Cohen’s kappa over five realizations, with one classifier applied in turn to each projected image without image-specific training.The projection is derived once and the classifier is trained on projected training samples.Source: B. Experimental setup · p. 16

IV. RESULTS AND DISCUSSION

The toy experiments test SS-MA under scaling, rotation, and translation deformations. SS-MA compensates for these deformations and aligns the two spirals, supporting subsequent linear-SVM classification.

IV. RESULTS AND DISCUSSION

  • Across the tested deformations, SS-MA compensates for the changes and correctly aligns the two spirals.The examples include scaling, scaling plus rotation, and scaling plus rotation plus translation.Source: A. 2D toy datasets · p. 16, Figure 4 · p. 17
  • After alignment, linear-SVM results show that adding samples from the other spiral does not affect first-spiral prediction, whereas unprojected data suffer a strong performance decrease.The plotted classification tests vary the second source from zero to 20 labeled pixels per class while using 20 per class from the first source.Source: A. 2D toy datasets · pp. 16–17
  • For second-spiral prediction, SS-MA efficiently exploits shared information from the two datasets and outperforms the compared alternative.The reported test varies second-spiral labels from 5 to 20 pixels per class.Source: A. 2D toy datasets · p. 17

B. Multiangular adaptation

In the multiangular experiment, SS-MA maintains accurate, stable classification across acquisition angles despite increasing off-nadir shifts. The latent space also needs more informative dimensions as the shift grows.

B. Multiangular adaptation

  • The latent projections form a discriminative space in which joint linear classification is facilitated.The visualization compares the first two and three latent dimensions for images acquired at 6.09° and −38.79°.Source: B. Multiangular adaptation · p. 18
  • Using only nadir training yields κ near 0.2 at increasing off-nadir angles; adding angle-specific labels helps, but performance still declines, especially for LDA.SS-MA instead gives accurate, stable results across acquisitions, combining graph matching's stability with the higher numerical results of the generic supervised method.Source: B. Multiangular adaptation · p. 19
  • SS-MA produces an almost flat κ surface across acquisitions, outperforming PCA alignment and matching or exceeding competing methods in stability.Its results are described as always outperforming PCA and as at least as accurate as KPCA, with greater stability than KPCA across classifiers and label counts.Source: B. Multiangular adaptation · p. 19, Figure 6 · p. 20
  • Larger shifts require more latent dimensions: the 26.75° acquisition needs 12–18, while more off-nadir acquisitions need higher dimensionality.The nadir acquisition requires the fewest dimensions; the −39.79° result at 10 pixels per class is unusually unstable, although classification remains around 0.82 κ.Source: B. Multiangular adaptation · p. 20

C. Multitemporal adaptation for the same sensor

For the tested multitemporal scenes, SS-MA yields higher κ than classification without alignment and maintains performance when auxiliary-domain labels are added. Its gains are largest when predicting other scenes.

C. Multitemporal adaptation for the same sensor

  • SS-MA uses one projection set per labeled-sample combination across the three domains, then classifies all images with the same SVM model.That model is trained on labeled pixels from the three domains.Source: C. Multitemporal adaptation for the… · pp. 21–22
  • Across all tested scenarios, SS-MA yields higher κ scores than the method without alignment.The numerical results cover tests with labeled samples from multiple WorldView-2 scenes.Source: Figure 8 · p. 22, C. Multitemporal adaptation for the… · p. 22, Figure 9 · p. 23
  • When predicting the leading training image, adding labels from other images can reduce unaligned performance, while SS-MA remains stable and generally above baseline.This contrast is particularly visible for the Montelly and Malley cases.Source: C. Multitemporal adaptation for the… · p. 22
  • When predicting another image, adding auxiliary-domain data improves results, with alignment giving κ scores 0.1–0.3 points above using only the leading domain.SS-MA gives more drastic improvements than the non-projected method and can meet the tested-domain baseline in all cases.Source: C. Multitemporal adaptation for the… · p. 23

D. Multisource adaptation

Across the tested domains, SS-MA performs well under strong deformations and classifies target domains at least as accurately as with a large target-domain labeled set. In the multisource experiment, aligned data use a joint latent space and outperform the reported comparison settings.

D. Multisource adaptation

V. CONCLUSIONS

  • SS-MA pulls same-class pixels together, pushes different-class pixels apart, and preserves each manifold's geometry during transformation.Source: §V · p. 25
  • Under strong deformations, SS-MA classifies target domains at least as accurately as when a large set of labeled pixels is available for that domain.Source: §V · p. 26
  • None of the evaluated image datasets was coregistered, a setting the conclusion identifies as relevant to SS-MA compared with methods such as CCA.Source: §V · p. 26
  • SS-MA requires some labeled pixels in each image, typically only a few.Source: §V · p. 26

Related papers

Loading 2104.07803v1…