PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepSUM: Deep neural network for Super-resolution of Unregistered Multitemporal images

Andrea Bordone Molini, Diego Valsesia, Giulia Fracastoro, Enrico Magli

arXiv:1907.06490v2eess.IVcs.LG

TL;DR

Remote-sensing MISR from multitemporal, unregistered imagery has received little attention, despite the scarcity of real matched LR and HR data. DeepSUM integrates learned registration and feature fusion in a CNN, and it won the PROBA-V super-resolution challenge.

  • Problem

    Deep-learning methods for remote-sensing MISR have received little attention, and real LR observations paired with HR images from the same platform are difficult to find.

    Source: §I · p. 1, §III · p. 3

  • Method

    DeepSUM uses a CNN to combine multiple unregistered images, integrating registration within the network and optimizing the reconstruction end-to-end.

    Source: §I · p. 1

  • Results

    DeepSUM outperformed the compared methods and was the best method in the PROBA-V super-resolution challenge.

    Source: §I · p. 1, 2) Comparison to State-of-the-Art · p. 8, §VII · p. 9

  • Takeaways & Limitations

    The framework successfully handles complex degradation and temporal variation in multitemporal remote-sensing super-resolution.

    Source: §VII · p. 9

Abstract

from arXiv · show

Recently, convolutional neural networks (CNN) have been successfully applied to many remote sensing problems. However, deep learning techniques for multi-image super-resolution from multitemporal unregistered imagery have received little attention so far. This work proposes a novel CNN-based technique that exploits both spatial and temporal correlations to combine multiple images. This novel framework integrates the spatial registration task directly inside the CNN, and allows to exploit the representation learning capabilities of the network to enhance registration accuracy. The entire super-resolution process relies on a single CNN with three main stages: shared 2D convolutions to extract high-dimensional features from the input images; a subnetwork proposing registration filters derived from the high-dimensional feature representations; 3D convolutions for slow fusion of the features from multiple images. The whole network can be trained end-to-end to recover a single high resolution image from multiple unregistered low resolution images. The method presented in this paper is the winner of the PROBA-V super-resolution challenge issued by the European Space Agency.

I. INTRODUCTION

Remote-sensing MISR has received little deep-learning attention despite the need to combine complementary, often unregistered observations. DeepSUM integrates registration into a CNN for that task and won the PROBA-V super-resolution challenge.

I. INTRODUCTION

  • Deep learning has been successful for SISR, but little work has addressed remote-sensing MISR, where registration, brightness variation, scene change, and unreliable data are challenges.Remote sensing uses imagery for mapping and monitoring, while limited HR product availability motivates resolution enhancement.Source: §I · p. 1
  • DeepSUM won the PROBA-V super-resolution challenge, using real LR and HR images captured by the same spacecraft at different times.The temporal variations between acquisitions must be handled during super-resolution.Source: §I · p. 1
  • DeepSUM combines unregistered images of the same scene using spatial and temporal correlations, with in-network registration that is blind to degradation and assumes translational shifts.The authors report robustness to temporal variation and cloud occlusion.Source: §I · p. 1
  • Compared with the preliminary version, this work adds in-network registration, improves mask use, expands comparisons, and discusses variable image counts with a fixed-size network input.The earlier version addressed MISR on the PROBA-V dataset.Source: §I · pp. 1–2

II. RELATED WORK

  • SR research includes interpolation, optimization, and learning-based methods; optimization priors can improve solutions but may degrade at higher upscaling factors and incur high computational cost.CNNs have more recently achieved state-of-the-art results for SISR.Source: §II · p. 2
  • Many SR methods assume known motion, blur, and noise, with registration or blur estimation performed before reconstruction; other approaches jointly estimate motion and reconstruction.The cited examples include joint MAP estimation and unified registration with sparse coding.Source: §II · p. 2
  • Video-SR CNNs address motion and reconstruction, but remote-sensing deep MISR remains limited; the proposed approach jointly registers LR inputs and reconstructs HR output end-to-end.The reviewed remote-sensing method uses a CNN for SISR before separate fusion and is not end-to-end supervised.Source: §II · p. 3

III. THE PROBA-V SR DATASET

The PROBA-V dataset pairs real LR observations with HR images from the same platform, addressing the scarcity of such matched data. Its multitemporal acquisitions include substantial scene inconsistency and masks that identify usable pixels.

III. THE PROBA-V SR DATASET

  • Real-world LR observations paired with corresponding HR images from the same platform are difficult to find; simulated data may simplify degradation or fail to match real temporal variation.Simulated LR data are often generated by degrading and downsampling HR images under an assumed sensor model.Source: §III · p. 3
  • PROBA-V provides 300 m LR and 100 m HR imagery across 74 regions, with RED and NIR bands and HR revisits roughly every five days versus daily LR.The dataset contains 128×128 LR and 384×384 HR images.Source: §III · p. 3
  • The dataset contains 1,160 scenes, each with one HR image and 9–30 LR images acquired over as many as 30 days; masks mark pixels usable for reconstruction.Clouds, shadows, ice, water, missing regions, and human activity create inconsistencies across observations; geometric disparity is translational, including subpixel shifts.Source: §III · p. 3
  • Real same-scene LR and HR imagery makes PROBA-V a case study for learning complex degradation inversion and feature fusion under temporal variation.The authors identify enhanced resolution alongside higher temporal availability as an interesting MISR application.Source: §III · p. 3

IV. PROPOSED METHOD

DeepSUM maps multiple LR images of one scene to an HR image using a CNN with feature extraction, registration, and feature fusion. Its stages align feature maps before combining them into a residual correction and SR output.

IV. PROPOSED METHOD

  • DeepSUM maps a set of N LR images of the same scene to one HR image using learned model parameters.The inputs and output are represented as real-valued tensors, with the output scaled by factor r.Source: §IV · p. 3
  • SISRNet independently extracts features from each upsampled input using the same 2D convolutional filters across images.The LR images are bicubically interpolated before entering the CNN.Source: §IV · p. 4
  • RegNet estimates N−1 translational filters to align feature representations to a reference at integer-pixel precision on the HR-resolution grid.Integer shifts at that grid correspond to subpixel shifts in the LR images.Source: §IV · p. 4
  • FusionNet uses small-kernel 3D convolutions to merge registered features; its residual is added to the average of registered inputs to form the SR image.This is the network’s “slow” feature-fusion stage.Source: §IV · p. 4, Figure 1 · p. 4

A. SISRNet Architecture

DeepSUM extracts shared spatial features, estimates registrations between feature maps, and fuses the aligned representations to reconstruct a high-resolution image.

A. SISRNet Architecture

  • SISRNet processes each input independently with shared 2D convolutional weights and instance normalization to extract features while accounting for brightness and contrast differences.Its features are intended to exploit spatial correlations and support the subsequent network blocks.Source: A. SISRNet Architecture · p. 4

B. RegNet Architecture

  • The filters assume a shared translational shift across an image’s feature channels, with the supported correction bounded by ⌊K/2⌋ pixels.The filter is shared channel-wise.Source: B. RegNet Architecture · p. 4
  • RegNet computes image-specific registration filters from feature representations, making registration part of the network and allowing it to use feature-space information.The authors contrast this with methods that compensate for motion as a preprocessing step.Source: B. RegNet Architecture · p. 4
  • RegNet pairs each non-reference representation with the reference and estimates local shifts, then spatially averages them to produce one K×K filter per image.The reference-and-image pairing avoids ordering ambiguity; the filters are normalized with softmax before use.Source: B. RegNet Architecture · p. 5
  • Before fusion, unreliable feature regions are filled from other images when reliable values are available; if none are available, the regions remain unreliable.The per-pixel masks are shifted to approximate the registration applied to the image representations.Source: C. Mutual Inpainting · p. 5
  • FusionNet progressively combines registered features with 3D convolutions, then adds an estimated high-frequency residual to the average of the registered inputs.Registration is applied before that averaging, using the filters produced by RegNet.Source: D. FusionNet Architecture · pp. 5–6

E. Loss Function

The loss addresses brightness differences, possible image shifts, and unreliable pixels when comparing the reconstructed and target images.

E. Loss Function

  • The modified Euclidean loss equalizes average brightness and searches over shifted target patches to account for differences between reconstructed and HR images.The SR image is center-cropped by d pixels, and the minimum Euclidean distance across the candidate HR patches is used.Source: E. Loss Function · p. 6
  • The authors use a modified Euclidean objective rather than adversarial loss, which they say can hallucinate information and yield lower MSE scores and less reliable remote-sensing products.Minimizing Euclidean distance is optimal in terms of the mean-squared error metric.Source: E. Loss Function · p. 6
  • The loss uses only HR pixels marked reliable and SR pixels for which at least one input LR image was clear.Pixels hidden in the HR image or concealed in every LR image are excluded because their terrain information is unavailable for prediction.Source: E. Loss Function · p. 6

A. Pre-training

The authors pretrain SISRNet and RegNet for their separate subtasks before combining the blocks for end-to-end fine-tuning.

A. Pre-training

  • Pretraining addresses the difficulty of training all blocks from scratch, where local minima and gradients may fail to distinguish registration from feature learning.The proposed strategy is to pretrain blocks for their subtasks, then fine-tune them together end-to-end.Source: A. Pre-training · p. 6
  • SISRNet is pretrained on single-image super-resolution with a projection layer, which is removed afterward to generate feature maps for RegNet pretraining.The single-image reconstruction is compared with the scene’s HR image using the final training objective.Source: 1) SISRNet pre-training · p. 6
  • RegNet is pretrained as a K²-class shift classifier on feature maps synthetically shifted by random integer amounts relative to the first image.A cross-entropy loss trains its softmax output toward a filter with a one at the selected shift and zeros elsewhere.Source: 2) RegNet pre-training · pp. 6–7

B. Final training

The network is optimized end-to-end after block-specific initialization.

B. Final training

  • The complete network is trained end-to-end for the MISR task.FusionNet is trained from scratch, while SISRNet and RegNet use weights initialized by their pretraining procedures.Source: B. Final training · p. 7
  • FusionNet starts from scratch, whereas SISRNet and RegNet are initialized from pretraining.These initialization choices are used when the network is trained as a whole.Source: B. Final training · p. 7
  • Concurrent optimization lets SISRNet fine-tune image representations for RegNet, which in turn seeks registrations that support FusionNet.Source: B. Final training · p. 7

C. Testing phase

At test time, scenes with more than the network's fixed N low-resolution inputs can be handled through subsets and averaged SR estimates. The text also identifies estimate merging and input-count-independent design as future research.

C. Testing phase

  • With more than N LR images, testing uses multiple N-image subsets, keeps a common reference, and averages their SR estimates.The paper advises using the same reference image in each subset so the SR estimates can be registered to one another.Source: C. Testing phase · p. 7
  • Sorting images by masked-pixel count and sliding an N-image window produces subsets, while SR estimate quality declines as masked pixels increase.The suggested ordering is by increasing number of masked pixels.Source: C. Testing phase · p. 7
  • Reused images make estimates non-independent, yet the authors report consistent gains on their test set.Source: C. Testing phase · p. 7
  • Optimal merging of SR estimates and a network independent of input-image count are left for future research.Source: C. Testing phase · p. 7

VI. EXPERIMENTAL RESULTS AND DISCUSSIONS

Experiments on PROBA-V RED and NIR data compare DeepSUM with alternative super-resolution methods and evaluate its registration and sliding-window components. The results report DeepSUM outperforming the compared methods, with gains also reported for RegNet and averaging multiple estimates.

A. Experimental setting

  • Training uses 396 NIR and 415 RED scenes, with 170 NIR and 176 RED scenes held out for testing; the network uses N = 9 images.Patch extraction yields 38,400 samples, using patches accepted under the stated image-mask clarity thresholds.Source: A. Experimental setting · p. 7

B. Quantitative results

  • The full network exceeds the version without RegNet by 0.16 dB on NIR and 0.13 dB on RED.The full version is SISRNet+RegNet+FusionNet; the reduced version is SISRNet+FusionNet.Source: Table I · p. 8, 1) Ablation study · p. 8
  • Sliding-window averaging gains up to 0.3 dB on NIR and 0.2 dB on RED over a single SR estimate, then quickly saturates.Subsequent results use a sliding factor of 5.Source: 1) Ablation study · p. 8, Figure 4 · p. 8
  • The full network's end-to-end feature-space registration provides more accurate alignment and helps FusionNet merge features.The full and reduced networks were trained independently.Source: 1) Ablation study · p. 8
  • IBP [36] improves on Bicubic+Mean but is limited by its unknown degradation model; BTV [39] performs comparably and slightly worse on mPSNR.The authors attribute BTV's slight disadvantage to its L1 data-fidelity term being suboptimal for mPSNR.Source: 2) Comparison to State-of-the-Art · p. 8
  • DeepSUM outperforms the compared super-resolution methods on the reported PROBA-V evaluation.The comparison includes bicubic, SISR, IBP, BTV, and DUF approaches; the supplied passages do not give the table's individual values.Source: 2) Comparison to State-of-the-Art · p. 8, Table II · p. 9
  • DeepSUM's comparison with SISR+Mean shows the improvement associated with FusionNet's slow 3D-convolution fusion of image representations.SISR+Mean averages outputs from a pretrained single-image SR network applied to multiple images.Source: 2) Comparison to State-of-the-Art · p. 9
  • DUF performs worse than DeepSUM and depends on which LR image is selected as its central frame.The authors note that involving all LR images helps average differences in brightness, landscape, weather, and cloud cover.Source: 2) Comparison to State-of-the-Art · p. 9

C. Qualitative results

Qualitative comparisons cover RED and NIR test images, showing multitemporal variability among LR inputs and visually detailed DeepSUM reconstructions.

C. Qualitative results

  • The qualitative comparisons use images from the PROBA-V test set in both RED and NIR bands.Source: C. Qualitative results · p. 9
  • The LR images vary from one another and from the HR target in both NIR and RED bands.Figs. 5 and 8 illustrate this variability.Source: C. Qualitative results · p. 9
  • DeepSUM produces visually more detailed RED and NIR reconstructions, recovering finer texture and sharper edges than the compared methods.Absolute-difference figures compare each registered, brightness-compensated reconstruction with the HR target.Source: C. Qualitative results · p. 9

VII. CONCLUSION

DeepSUM achieves state-of-the-art performance and the best result in the PROBA-V super-resolution challenge. The conclusion points to non-local features as a possible direction for future work.

VII. CONCLUSION

  • DeepSUM provides state-of-the-art performance and is the best method in the PROBA-V super-resolution challenge.The paper reports that it handles complex degradation and temporal variation models.Source: §VII · p. 9
  • DeepSUM is presented as one of the first CNN architectures for super-resolution from multitemporal remote-sensing images.Source: §VII · p. 9
  • Future work may integrate non-local features using graph-convolutional architectures.Source: §VII · p. 9
Loading 1907.06490v2…