PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Image Translation with an Affinity-Based Change Prior for Unsupervised Multimodal Change Detection

Luigi Tommaso Luppino, Michael Kampffmeyer, Filippo Maria Bianchi, Gabriele Moser, Sebastiano Bruno Serpico, Robert Jenssen, Stian Normann Anfinsen

arXiv:2001.04271v2cs.LGcs.CVeess.IVstat.ML

TL;DR

Heterogeneous change detection needs translation between incompatible sensor domains, but supervised change information is unavailable in the unsupervised setting. The paper derives an affinity-based change prior and uses it to train X-Net and ACE-Net with translation, cycle-consistency, and adversarial objectives. Across three datasets, the proposed unsupervised methods compare favorably with reference approaches and can be in line with a supervised method.

  • Problem

    Heterogeneous sensors measure different physical quantities, while supervised change information is unavailable or costly to obtain for unsupervised change detection.

  • Method

    The method derives a self-supervised change prior from domain-specific affinity differences and incorporates it into X-Net and ACE-Net training with adversarial alignment.

  • Results

    On the benchmark dataset, traditional methods obtain κ values no higher than 0.44, whereas deep learning methods reach 0.66 and above.

  • Takeaways & Limitations

    The proposed unsupervised methodologies are reported to perform in line with the supervised AFL approach while using affinity information learned from input data.

Abstract

from arXiv · show

Image translation with convolutional neural networks has recently been used as an approach to multimodal change detection. Existing approaches train the networks by exploiting supervised information of the change areas, which, however, is not always available. A main challenge in the unsupervised problem setting is to avoid that change pixels affect the learning of the translation function. We propose two new network architectures trained with loss functions weighted by priors that reduce the impact of change pixels on the learning objective. The change prior is derived in an unsupervised fashion from relational pixel information captured by domain-specific affinity matrices. Specifically, we use the vertex degrees associated with an absolute affinity difference matrix and demonstrate their utility in combination with cycle consistency and adversarial training. The proposed neural networks are compared with state-of-the-art algorithms. Experiments conducted on three real datasets show the effectiveness of our methodology.

I. INTRODUCTION

Heterogeneous change detection compares images from different sensors, whose measurements may be statistically incompatible. The paper proposes unsupervised affinity-based priors and two neural architectures for deep image translation without supervised change information.

  • Motivation: Heterogeneous sensors measure different physical quantities, so direct comparison can be meaningless without processing and co-calibration.
  • Motivation: Unsupervised methods avoid costly expert annotations, but must infer change-related information without supervised labels.
  • Contributions: The proposed prior compares intramodal pixel relations to obtain self-supervised information about structural changes.
  • Contributions: X-Net uses two fully convolutional mappings, whereas ACE-Net aligns two autoencoder code spaces through adversarial training.
  • Contributions: The framework includes the prior, two unsupervised architectures, three benchmark datasets, and publicly available implementations and data.

A. Stacked Denoising Autoencoders

The section reviews autoencoder-based feature learning and adversarial image-translation methods used to create common representations or mappings for heterogeneous change detection. These approaches typically reconstruct, align, or compare modality-specific features before detecting changes.

  • Stacked Denoising Autoencoders: Autoencoders learn representations for feature extraction, dimensionality reduction, clustering, or denoising through input reconstruction.
  • Applications: Prior heterogeneous change-detection methods use autoencoder codes, clustering, or mapping networks to derive common-domain representations and preliminary change classes.
  • Applications: Some mapping methods initialize change-probability maps randomly and iteratively update them with network parameters.
  • Adversarial Translation: GANs train generators and discriminators competitively, while conditional GANs condition generated data on input images for image-to-image translation.
  • Adversarial Translation: Conditional GAN translation for heterogeneous change detection requires training patches without changes and can include an approximation network for direct comparison.

1) Background:

The method estimates an unsupervised change prior from differences between domain-specific affinity matrices. Averaged row-based differences, interpreted as change-graph vertex degrees, reduce the influence of likely changed pixels during translation training.

  • Background: Cycle-consistent translation uses paired generators to map X to Y and back, while coupling translation networks combine autoencoders, reconstruction, cycle consistency, and adversarial losses.
  • Prior Computation: Training on all image patches can connect different land covers, such as forests and fire scars, because changed areas distort the learned transformation.
  • Prior Computation: The unsupervised prior compares affinity matrices built from corresponding multimodal patches without ancillary information or sensor knowledge.
  • Prior Computation: Large sliding-window strides reduce computation but can create tiled artifacts and limit averaging across patches.
  • Prior Computation: Averaging rows of the absolute affinity difference matrix assigns larger values to pixels whose structural relations are more strongly perturbed.
  • Prior Computation: Interpreting the difference matrix as a change graph makes each prior a normalized vertex degree, where high degree indicates many changed pixel relations.

B. X-Net: Weighted Translation Network

X-Net learns two convolutional mappings between the domains, using a loss function composed of multiple objectives for translation. Its formulation is introduced as the weighted-translation component of the proposed framework.

  • X-Net: Weighted Translation Network: X-Net trains F(X) to map patches from domain X to Y and G(Y) to perform the reverse translation.
  • X-Net: Weighted Translation Network: The two mappings can be implemented as convolutional neural networks whose parameters minimize a weighted sum of loss terms.
  • X-Net: Weighted Translation Network: For paired patches, the translation objective is designed to enforce the desired cross-domain mapping.

1) Weighted translation loss:

The weighted translation loss uses an unsupervised affinity-based prior to reduce the influence of likely changed pixels and approximate the desired no-change conditional MSE.

  • Weighted translation loss: The translation networks use the prior-weighted objective so changed pixels do not enforce the same translation requirement as unchanged pixels.This preserves the networks’ ability to discriminate changes.
  • Weighted translation loss: Lower Ψ_i values indicate pixels likely changed, whereas values near 1/P(H0) indicate pixels likely unchanged.The same interpretation applies to the components of Φ.
  • Weighted translation loss: The method assigns each pixel pair a prior α_i measuring its chance of change from affinity reasoning.The prior is computed before training from relational information in the two domains.
  • Weighted translation loss: The weighted loss approximates the no-change conditional MSE using an unconditional, affinity-weighted objective.This reformulation avoids requiring a supervised training set containing only unchanged samples.
  • Weighted translation loss: A monotonically decreasing function Π maps change probabilities α_i into loss weights Π_i.Weights approach zero for likely changed pixels and one for likely unchanged pixels.

2) Cycle-consistency loss:

Cycle consistency requires translated data to return correctly to its original domain after being transformed across domains and back.

  • Cycle-consistency loss: Cycle consistency requires F(X) followed by G and G(Y) followed by F to reproduce the original inputs.The re-transformed outputs are denoted ˙X and ˙Y.
  • Cycle-consistency loss: The cycle-consistency loss penalizes discrepancies between the original patches and their twice-transformed versions.It complements the direct translation objective.
  • Cycle-consistency loss: Cycle-consistency training does not require paired data.This property supports its use in the unsupervised setting.

3) Total Loss Function:

The proposed architectures combine weighted translation and cycle-consistency objectives with reconstruction and regularization terms; ACE-Net additionally introduces a latent space and adversarial alignment.

  • Total loss function: The total objective is optimized with balancing weights controlling the cycle, affinity-weighted translation, and parameter-regularization terms.Optimization seeks the global minimum with respect to the network parameters.
  • Total loss function: The ACE-Net introduces a latent space Z between domains X and Y using encoders and decoders that map data into and out of the shared representation.Its transformations satisfy F(X)=D_Y(E_X(X)) and G(Y)=D_X(E_Y(Y)).
  • Total loss function: Both architectures retain weighted translation, cycle consistency, and weight-decay regularization as core loss components.The weighted translation and cycle terms operate on the cross-domain mappings.
  • Total loss function: ACE-Net adds autoencoder reconstruction so decoded codes reproduce inputs from their original domains.The reconstruction targets are ˜X≈X and ˜Y≈Y.

2) Adversarial Code Alignment Losses:

ACE-Net aligns the two latent code spaces adversarially, then extracts changes from combined translation-distance images using filtering and thresholding.

  • Adversarial Code Alignment Losses: A discriminator distinguishes codes from the two encoders, while the encoders are trained to prevent that discrimination.This targets alignment in both code distribution and feature-space class locations.
  • Adversarial Code Alignment Losses: The total ACE-Net loss combines adversarial, reconstruction, cycle-consistency, weighted translation, and regularization terms.Their relative contributions require tuning of the corresponding weights.
  • Change extraction: Image subtraction is selected after translation because original and transformed images are then in the same domain.This keeps the change-extraction operation simple while focusing evaluation on the translation methods.
  • Change extraction: The two normalized distance images are averaged so changes are highlighted and false alarms appearing in only one modality are suppressed.The resulting difference image is filtered and thresholded to form the binary change map.
  • Change extraction: Otsu’s method is used to set the threshold automatically after filtering the combined difference image.A low threshold increases false alarms, whereas a high threshold increases missed changes.

IV. EXPERIMENTAL RESULTS

The experiments use three heterogeneous remote-sensing datasets and evaluate affinity-prior computation alongside CNN-based network training. The improved prior produces more precise outputs and reduces computation time relative to the previous version.

  • Datasets: Experiments use forest-fire, flood, and construction datasets combining different satellite sensors and modalities.The Texas, California, and China datasets provide co-registered multimodal images with available ground truth.
  • Network design: The proposed networks use fully convolutional CNNs, enabling patch-based training and application to complete images.The architectures use convolutional layers without a bottleneck in the ACE-Net code layer.
  • Training setup: The training setup uses data augmentation, dropout, Adam optimization, and weighted loss terms for the ACE-Net and X-Net.Training uses randomly flipped and rotated patches, 20% dropout, 240 epochs, and a learning rate of 10^-5.
  • Prior computation: The change prior is updated from a scaled difference image so pixels likely to have changed receive smaller training weights.The prior is updated at epochs 80 and 160, with Π = 1 − d after scaling d to [0, 1].

2) SCCN and CAN:

The study compares the proposed methods with unsupervised and supervised heterogeneous change-detection approaches. The improved prior avoids the blurring and tiled patterns observed with the previous computation while reducing its computational burden.

  • Compared methods: The comparison includes SCCN and CAN, alongside several unsupervised and supervised methods evaluated on the China construction dataset.The reported evaluation uses AUC, overall accuracy, F1 score, Cohen’s Kappa, and elapsed time.
  • Comparison conditions: Supervised methods use training samples and may achieve higher change-detection performance, but require expert annotation or data collection.Unsupervised methods avoid this user-provided input, so results should be interpreted with the difference in supervision considered.
  • PC versus IPC: The improved prior produces sharper edges and smaller highlighted segments than the previous prior in visual comparisons.The previous method yields blurrier results with soft edges, while the improved method is less affected by patch-shift stride.
  • PC versus IPC: The previous prior assigns one value to an entire patch, producing tiled patterns under large strides; the improved prior is less affected by this stride.The visual comparison uses Δ = 1 and Δ = 20, with patch size k = 20.
  • Computational analysis: The improved prior is applied three times across original and half-size images, making patch count and computation time central experimental considerations.The stated applications use k_small = 10 and k = 20 at original sizes, then k = 20 after resampling.

D. Results

Across three datasets, X-Net and ACE-Net generally performed favourably against unsupervised baselines, while their architectural trade-offs differed. X-Net emphasized simplicity, stability, and speed; ACE-Net achieved the best overall performance but required more costly optimization.

  • Experimental setup: The experiments applied the proposed techniques and SCCN and CAN to the first two datasets, reporting average evaluation metrics and training times over 100 independent runs.The X-Net was the simplest framework and therefore had the fastest training procedure, whereas ACE-Net and SCCN had similar complexities and training times.
  • Dataset comparisons: Both ACE-Net and X-Net outperformed the other unsupervised methods on the China dataset, with X-Net reaching higher values.The comparison excluded supervised methods because they required supervision and user prompts for sample selection.
  • Architectural trade-offs: X-Net and CAN provided stable and consistent results on the selected datasets, with X-Net performing better.X-Net used two small CNNs, approximately 1.3 × 10^5 parameters, and converged quickly because its loss had limited terms.
  • Architectural trade-offs: CAN used approximately 3.1 × 10^5 parameters and all possible 5 × 5 patches, making training time-consuming, especially on larger datasets such as Texas.It also tended to miss changes because changed areas could align undesirably in generated and approximated images, producing many false negatives.
  • Architectural trade-offs: ACE-Net achieved the best overall performance on the three datasets, but its approximately 2.8 × 10^5 parameters and complex loss function increased convergence difficulty and variability.The architecture’s flexibility supported performance, while its complexity remained a main drawback.
  • Architectural trade-offs: SCCN’s small parameter space was associated with frequent convergence failures and poor results on the first dataset, while high accuracy on the California dataset coincided with highlighting water bodies.Using 500 training epochs helped SCCN converge more often on Texas but did not substantially improve the other two datasets.

B. Discussion of the results on the benchmark dataset

On the China dataset, deep learning methods substantially outperform traditional approaches, while ACE-Net appears more effective than CPTN and the proposed unsupervised methods remain competitive with supervised AFL. An ablation study further supports the affinity prior and core loss components, but not added image-content discriminators.

  • Benchmark comparison: Deep learning methods reach κ values of 0.66 and above, whereas traditional methods do not exceed 0.44 on the benchmark dataset.The comparison includes AFL, X-Net, ACE-Net, SCCN, CAN, and CPTN among the deep learning methods.
  • Architecture comparison: ACE-Net is more effective than CPTN on the China dataset despite their shared code spaces and adversarial losses.CPTN uses two output discriminators, while ACE-Net uses one discriminator in the code space.
  • Supervised comparison: The proposed unsupervised methodologies achieve results in line with supervised AFL, which uses patches selected from unchanged areas for training.This supports the usefulness of the affinity prior for capturing unchanged-area information without supervision.
  • Ablation study: Removing any component from the total loss is not beneficial, and replacing the proposed prior with a random one causes a considerable performance loss.Updating the prior at two milestones improves performance, whereas adding two image-content discriminators does not provide the same benefit.
  • Overall assessment: Across the evaluated methods, X-Net provides stable and consistent performance, while ACE-Net achieves the best results at the cost of greater complexity and more diligent training.Both proposed architectures consistently outperform state-of-the-art methods in the reported experiments.

APPENDIX

The appendix establishes the equivalence between a conditional mean-squared-error formulation and an unconditional weighted mean-squared-error formulation under conditional independence. It also defines the weighting behavior through likelihood ratios and states the assumptions used in the derivation.

  • Equivalence: Under the stated conditional-independence assumption, the H0-conditional MSE equals the corresponding unconditional weighted MSE.The argument is applied to the terms involving both X and Y.
  • Loss expression: The appendix expands the L2 squared distance into a pixel-wise expression involving Gi(Y), with n equal to the patch area.The expression is obtained by substituting the L2 distance into the preceding loss formulation.
  • Assumptions: The derivation assumes conditionally independent pixel sample pairs under H0 and continuous random vectors for all xi and yi.Here n = h · w, and Gi(Y) denotes the vector in G(Y) corresponding to pixel i.
  • Probability model: The unconditional joint density decomposes into H0- and H1-conditioned densities weighted by their prior probabilities.The decomposition uses the law of total probability.
  • Weight behavior: The weighting function ψ(xi, Y) decreases as the likelihood ratio Λ(xi, Y) increases and takes values in (0, 1/P(H0)].Thus, large likelihood-ratio values correspond to small weights in the loss.
Loading 2001.04271v2…