Source-linked AI summary
Code-Aligned Autoencoders for Unsupervised Change Detection in Multimodal Remote Sensing Images
Luigi T. Luppino, Mads A. Hansen, Michael Kampffmeyer, Filippo M. Bianchi, Gabriele Moser, Robert Jenssen, Stian N. Anfinsen
TL;DR
The paper addresses unsupervised multimodal change detection when heterogeneous code spaces and changing pixels hinder image translation. It aligns autoencoder codes using input affinity relations and evaluates the approach with cycle consistency, finding performance on par with or better than state-of-the-art methods across four datasets, with degradation for limited-feature inputs.
Problem
Heterogeneous change detection lacks reliably aligned code spaces, while supervised change-area information is not always available and changing pixels can distort translation learning.
Method
Two autoencoders are aligned through an unsupervised affinity-based loss that correlates pixels with similar input-domain relations, combined with cross-domain translation and cycle consistency.
Results
The method performs on par with or better than state-of-the-art methods across four real datasets.
Takeaways & Limitations
The framework maps multispectral and multipolarisation images meaningfully across domains while reducing the contribution of changes to alignment.
Takeaways & Limitations
Performance worsens with limited input features, especially when one image has only one channel and the task becomes regression from one variable to many.
Abstract
from arXiv · showhide
Image translation with convolutional autoencoders has recently been used as an approach to multimodal change detection in bitemporal satellite images. A main challenge is the alignment of the code spaces by reducing the contribution of change pixels to the learning of the translation function. Many existing approaches train the networks by exploiting supervised information of the change areas, which, however, is not always available. We propose to extract relational pixel information captured by domain-specific affinity matrices at the input and use this to enforce alignment of the code spaces and reduce the impact of change pixels on the learning objective. A change prior is derived in an unsupervised fashion from pixel pair affinities that are comparable across domains. To achieve code space alignment we enforce that pixel with similar affinity relations in the input domains should be correlated also in code space. We demonstrate the utility of this procedure in combination with cycle consistency. The proposed approach are compared with state-of-the-art deep learning algorithms. Experiments conducted on four real datasets show the effectiveness of our methodology.
I. INTRODUCTION
Heterogeneous change detection addresses the limits of homogeneous imagery but makes direct comparison difficult. The paper proposes unsupervised code-space alignment for autoencoder-based image translation, avoiding adversarial training and supervised change labels.
- Heterogeneous imagery prevents straightforward comparison because the domains have different statistical distributions and inconsistent class signatures.
- Existing cross-domain transformations often depend on unreliable initialization, clustering, or manually selected samples that are not always available.
- Adversarial approaches can require large datasets and careful balancing while suffering from mode collapse and unstable hyperparameter behavior.
- The proposed method aligns two autoencoders’ code layers as a common latent space for reconstruction and cross-domain translation.
- Affinity matrices provide unsupervised relational information so similarly related pixels across domains receive correlated codes, and the framework includes cycle consistency.
II. METHODOLOGY
The methodology uses two domain-specific autoencoders to translate between coregistered multimodal images, aligning their latent code spaces to support change detection. It combines reconstruction, cycle consistency, weighted translation objectives, and code-alignment constraints while accounting for changed pixels.
- Problem setup: The inputs are paired, coregistered patches from the same geographical region observed by heterogeneous sensors at two times, with limited scene change assumed.The domains may have different channel counts and sensor-specific statistical distributions.
- Objective: Sensor-specific reconstruction distances are combined with weights that can normalize channel-count effects or compensate for different noise levels.The objective uses Δ = W_X · d_X(X, X̂) + W_Y · d_Y(Y, Ŷ).
- Architecture: Two encoder-decoder pairs map sensor domains X and Y into latent spaces and decode within or across domains to implement F(X) and G(Y).The encoders produce codes in Z_X and Z_Y, while opposite-domain decoders enable image translation.
- Code alignment: Latent distributions are aligned not only globally but also in land-cover arrangement, because distributional alignment alone can permit mode swapping.The method introduces loss terms to align code distributions and the locations of land-cover structures within them.
- Implementation: The framework avoids dimensionality-reduction bottlenecks by retaining image-patch dimensions through hidden layers, relying on additional code-alignment constraints for regularization.The paper reports this configuration as producing the best results in its implementation.
- Objective: A weighted patch-distance loss is defined over equal-sized patches to compare inputs with translated outputs during training.The loss is minimized with respect to the network parameters to obtain the desired cross-domain transformations.
A. Reconstruction Loss
The reconstruction loss trains each autoencoder to reproduce its input patch accurately. This preserves within-domain information while the networks learn their latent representations.
- Reconstruction requirement: Each autoencoder is required to reproduce its input patch faithfully at the output.The reconstruction targets are the original patches from the two sensor domains.
- Loss definition: The reconstruction loss is defined using mean squared error between the desired and predicted output patches.
B. Cycle-consistency Loss
Cycle consistency requires translating a patch across domains and back to recover the original input. The resulting cycle loss can use unpaired inputs, whereas translation and code-correlation losses require paired data.
- Cycle requirement: Cycle consistency requires both X→Y→X and Y→X→Y transformations to recover the original inputs.The cyclic outputs are denoted X̌ = G(Ŷ) and Y̌ = F(X̂).
- Data requirements: Cycle consistency and reconstruction can be evaluated with unpaired data because each cyclic output is computed from its corresponding input.
- Translation weighting: The translation loss weights pixels by their probability of being unchanged, reducing the contribution of likely changed pixels.The prior π is stored over the full image, initialized to zero, and updated from a scaled preliminary difference image after several epochs.
- Data requirements: Translation loss requires paired data because translated outputs from one domain are compared directly with inputs from the other domain.The code correlation loss likewise requires paired data.
D. Code Correlation Loss
The code correlation loss aligns latent codes by preserving cross-domain pixel relationships derived from domain-specific affinities. It treats comparable affinity relations as a change-sensitive signal and matches them to correlations between corresponding code entries.
- D. Code Correlation Loss: The method computes sensor-specific pixel distances, converts them into affinity representations, and defines a crossmodal distance between pixels from the two domains.Rows of the affinity matrices represent pixels in a new affinity space, enabling direct comparison across modalities.
- D. Code Correlation Loss: The crossmodal distance distinguishes pixel pairs with consistent versus inconsistent relations across domains, providing information interpretable as a probability of change.The distance is normalized to [0, 1] because the underlying affinities are normalized to that range.
- D. Code Correlation Loss: Code similarities are forced to match input-space similarities, so corresponding code entries preserve the relational structure measured by the affinity-based distance.The code correlation matrix stores the resulting code correlations, and only encoder parameters are adjusted by this loss.
- D. Code Correlation Loss: The code correlation loss is defined as the objective that enforces this relationship between crossmodal input distances and correlations in the latent codes.The loss builds on pairwise distances computed over all pixels in co-located training patches.
E. Total Loss Function
The framework minimizes a weighted sum of reconstruction, cycle-consistency, transformation, and code-correlation losses. Cycle consistency and code correlation align the code spaces, while the remaining terms preserve reconstruction and cross-domain transformation quality.
- E. Total Loss Function: The total objective is a weighted sum of reconstruction, code-correlation, transformation, and cycle-consistency losses.The coefficients λr, λc, λt, and λz balance the terms during optimization.
- E. Total Loss Function: Cycle consistency and code correlation achieve code-space alignment, while reconstruction and transformation losses maintain faithful domain outputs.The framework evaluates an optional spatially filtered change image before the final change-detection step.
A. Implementation details
The implementation uses fully convolutional autoencoders with non-strided padded layers and trains them with Adam for 100 epochs. Performance is evaluated using overall accuracy and Cohen’s kappa, with comparisons made per dataset when metrics differ across methods.
- A. Implementation details: The networks use fully convolutional 3 × 3 layers, preserve spatial dimensions with padding, and normalize outputs with Tanh.The encoders use three output filters, while domain-specific distance networks use channel counts matching their inputs.
- A. Implementation details: Adam minimizes the objective for 100 epochs with an initial learning rate of 10^-4 and scheduled exponential decay.The code-correlation loss receives a more aggressive decay rate of 0.9 than the general rate of 0.96.
- A. Implementation details: Performance is measured with overall accuracy and Cohen’s kappa, but comparisons are made dataset by dataset using whichever metrics competing papers report.High kappa generally implies high overall accuracy, but the reverse does not necessarily hold.
C. Methods compared
The evaluation compares the proposed framework with deep learning and other heterogeneous change-detection methods across real multimodal datasets. On the Texas forest-fire dataset, the proposed network achieves higher accuracy than competitors with low variance.
- Methods and datasets: The Texas experiment compares the proposed framework with four deep learning approaches and several additional heterogeneous change-detection methods.Reference methods include CAN, SCCN, ACE-Net, X-Net, TGSM, EENPPI, MIMDS, M3CD, and MLSM.
- Methods and datasets: The Texas dataset pairs Landsat 5 imagery before the forest fire with EO-1 ALI imagery after the event and provides ground truth.Both optical images contain 1534 × 808 pixels, with 7 and 10 channels respectively.
- Texas results: The proposed network produces consistently higher accuracy than competitors on Texas while maintaining low variance.The comparison is reported in Fig. 3, and prior κ values of 0.65 and 0.91 are reported for Volpi et al. and Luppino et al., respectively.
- Texas results: Training times on Texas are comparable across methods because affinity-matrix computation is time-consuming, despite the proposed method using smaller networks and fewer iterations.The average training times are listed in Table I.
E. Second dataset: Flood in California
The California experiment evaluates flood detection from heterogeneous Landsat 8 and Sentinel-1A imagery. The proposed framework outperforms state-of-the-art counterparts with high quality and low variance, while retaining comparable training time.
- Dataset: The California dataset combines Landsat 8 imagery from 5 January 2017 with Sentinel-1A VV and VH imagery from 18 February 2017, using flood ground truth.The images were resampled from 3500×2000 to 850 × 500 pixels for comparison with prior work.
- California results: The proposed framework outperforms state-of-the-art counterparts on the California flood dataset in quality and variance.The dataset metrics are summarized in Fig. 5, and the reported κ for a prior method is 0.46.
- California results: The California evaluation reports κ as the comparison metric across the proposed framework and state-of-the-art methods.Fig. 5 presents the dataset-level metric comparison, while Table II contains average training times.
- California results: The proposed approach requires training time in line with state-of-the-art algorithms on the California dataset.Average training times are reported in Table II.
- Additional dataset context: The lake-overflow dataset uses Landsat 5 NIR imagery at one time and RGB imagery at a later time, with overflow ground truth.The images contain 412×300 pixels, and Table III reports average overall accuracy for several methods.
G. Fourth dataset: Construction site in France
The proposed method produces meaningful cross-domain transformations on the construction-site datasets, with clear resemblance between transformed and original image styles. Limited features can impair code-space alignment, motivating a weighted combination of input-space difference images.
- Visual results: Meaningful cross-domain transformations preserve clear resemblance between the generated and original image styles across the evaluated datasets.Figure 8 organizes inputs, code-space transformations, translated images, filtered differences, and confusion maps for each dataset.
- Method choice: For the final datasets, computing d as a weighted sum of input-space difference images remains preferable to relying only on latent-space differences.The input-space weighting is retained despite latent-space differences remaining a valid option.
- Benchmark results: The method performs on par with or better than state-of-the-art methods across four datasets, including the construction dataset evaluated by average accuracy.Table IV reports average accuracy for several methods on the constructions dataset, with the proposed method indicated in bold.
- Limitations: Limited data and features, especially a single channel in one image, can prevent proper code-space alignment and make regression from one variable to many ill-posed.The authors identify this as a limitation affecting the last two datasets.