PaperlayerView exact revision on arXiv

Source-linked AI summary

EDiffSR: An Efficient Diffusion Probabilistic Model for Remote Sensing Image Super-Resolution

Yi Xiao, Qiangqiang Yuan, Kui Jiang, Jiang He, Xianyu Jin, Liangpei Zhang

arXiv:2310.19288v1eess.IVcs.CV

TL;DR

Remote-sensing SR is difficult because regression methods can produce over-smoothed images, GANs can generate artifacts, and diffusion SR often relies on costly denoisers while underusing LR priors. EDiffSR introduces an efficient diffusion model with enhanced LR conditioning and lightweight noise prediction, and experiments across four datasets report superior perceptual and quantitative performance, while its sampling cost and degradation coverage remain limitations.

  • Problem

    Remote-sensing SR needs to reconstruct detailed HR images from ill-posed and variably degraded LR inputs, while existing methods can over-smooth outputs, produce artifacts, or use inefficient diffusion denoisers.

  • Method

    EDiffSR combines a Conditional Prior Enhancement Module for richer LR-derived conditions with an Efficient Activation Network for lightweight diffusion noise prediction.

  • Results

    Experiments on AID, DOTA, DIOR, and NWPU-RESISC45 report that EDiffSR outperforms state-of-the-art CNN-, GAN-, and diffusion-based SR methods in data distribution and perceptual quality.

  • Takeaways & Limitations

    EDiffSR provides an efficient diffusion-based approach for perceptually pleasant remote-sensing SR on simulated and real-world images.

  • Takeaways & Limitations

    Diffusion sampling remains computationally expensive for real-time use, and EDiffSR does not model multiple degradations, limiting adaptability to real-world scenes.

Abstract

from arXiv · show

Recently, convolutional networks have achieved remarkable development in remote sensing image Super-Resoltuion (SR) by minimizing the regression objectives, e.g., MSE loss. However, despite achieving impressive performance, these methods often suffer from poor visual quality with over-smooth issues. Generative adversarial networks have the potential to infer intricate details, but they are easy to collapse, resulting in undesirable artifacts. To mitigate these issues, in this paper, we first introduce Diffusion Probabilistic Model (DPM) for efficient remote sensing image SR, dubbed EDiffSR. EDiffSR is easy to train and maintains the merits of DPM in generating perceptual-pleasant images. Specifically, different from previous works using heavy UNet for noise prediction, we develop an Efficient Activation Network (EANet) to achieve favorable noise prediction performance by simplified channel attention and simple gate operation, which dramatically reduces the computational budget. Moreover, to introduce more valuable prior knowledge into the proposed EDiffSR, a practical Conditional Prior Enhancement Module (CPEM) is developed to help extract an enriched condition. Unlike most DPM-based SR models that directly generate conditions by amplifying LR images, the proposed CPEM helps to retain more informative cues for accurate SR. Extensive experiments on four remote sensing datasets demonstrate that EDiffSR can restore visual-pleasant images on simulated and real-world remote sensing images, both quantitatively and qualitatively. The code of EDiffSR will be available at https://github.com/XY-boy/EDiffSR

I. INTRODUCTION

Remote-sensing super-resolution is ill-posed and practically important, but regression methods often produce over-smoothed images while GANs can introduce artifacts. EDiffSR addresses these issues with an efficient diffusion model that enhances LR priors and reduces denoising cost.

  • Problem: Remote-sensing SR reconstructs detailed high-resolution images from low-resolution inputs, but its ill-posedness and real-world degradations make reconstruction challenging.The problem is especially complicated by atmospheric scattering and platform tremors in large-scale earth-observation scenarios.
  • Existing methods: Regression-based CNN methods can achieve favorable PSNR but often encourage over-smoothed outputs by minimizing pixel-level MSE or MAE objectives.These models include dense, residual, attention-based, and transformer architectures.
  • Existing methods: GAN-based SR methods can recover intricate details but suffer from training instability and collapse, producing undesirable artifacts.
  • Diffusion models: Diffusion probabilistic models offer a principled reverse-diffusion process for generating complex distributions while mitigating GAN training instability.Prior DPM-based SR methods commonly use heavy UNets and rarely exploit LR-image prior knowledge critical to SR.
  • Proposed approach: EDiffSR combines Conditional Prior Enhancement with an Efficient Activation Network to enrich LR conditions and achieve favorable noise prediction at lower computational cost.The model also uses stochastic differential equations to facilitate diffusion sampling.

2) GAN-based Models:

Prior SR research includes CNN, GAN, and diffusion approaches, but diffusion-based SR remains computationally demanding and underuses remote-sensing priors. EDiffSR addresses this with efficient denoising, enhanced conditions, and an SDE-based diffusion process.

  • GAN-based Models:: CNN-based SR methods primarily optimize PSNR through regression losses, while GAN-based methods add perceptual or texture objectives to recover high-frequency details.
  • Diffusion-based Models:: Diffusion-based SR uses iterative refinement to generate high-quality results and can model image degradation through stochastic differential equations.
  • Diffusion-based Models:: Large UNets for diffusion noise estimation create inefficient inference in remote-sensing SR, where many input pixels are already known.
  • EDiffSR framework: EDiffSR uses CPEM to extract richer LR priors instead of relying only on bicubic-upsampled conditions.CPEM uses convolution, ReLU, stacked RCABs, and upscaling to extract prior information.
  • Forward Diffusion Process: The forward SDE transforms the high-resolution image toward a noisy state whose mean converges to the bicubic-upsampled LR image and whose variance approaches the stationary variance.The closed-form Gaussian state is characterized by time-dependent mean and variance.
  • EDiffSR framework: EANet uses lightweight EAB components with depth-wise convolution, simple channel attention, and gate operations for efficient noise prediction.

B. Reverse Diffusion Process

The reverse diffusion process recovers the HR image from terminal state xT by simulating a reverse-time SDE and predicting noise through a conditional score-based model.

  • Reverse diffusion recovers the HR image from terminal state xT by simulating the reverse-time SDE.
  • During inference, the model uses the ground-truth score ∇x log pt(x) to guide the reverse process.
  • During training, the available ground-truth image x0 enables use of conditional scores.
  • The training objective computes Euclidean distance between predicted noise and ground-truth noise εt, weighted by positive γt and conditioned on the original LR image v.

IV. PROPOSED METHOD

EDiffSR enriches the LR condition before noise prediction and uses EANet for efficient denoising. CPEM combines LR-derived inputs and diffusion noise to produce a more informative conditional representation.

  • CPEM takes random noise εt, LR image v, and bicubic-upsampled LR image ¯ILR to produce the enriched condition It.
  • Pixel folding downsamples εt without loss of spatial information, while a 3 × 3 convolution and channel concatenation support condition construction.
  • EANet receives the enriched condition and time t to predict pure noise, and its parameters are optimized until convergence.
  • Unlike simple bicubic conditioning, CPEM explores additional LR-image prior knowledge to retain more informative condition cues for SR.
  • CPEM uses convolution, ReLU, cascaded RCABs with a global residual connection, and PixelShuffle to generate the condition.

C. Efficient Activation Network for Noise Prediction

EANet uses lightweight Efficient Activation Blocks for multiscale noise prediction in a U-shaped encoder-decoder. Its simplified attention and gate operations reduce computational complexity while retaining denoising capability.

  • The EAB combines depth-wise convolution, simple channel attention, and simple gate operations as EANet’s lightweight core.
  • Given input X and time step t, an EAB predicts output Y while projecting t through an MLP for feature modulation.
  • Multi-scale depth-wise convolutions extract RSI features at 3 × 3, 5 × 5, and 7 × 7 scales using simple activation operations.
  • A 1 × 1 convolution aggregates the multiscale representations before layer normalization and scaling-and-shifting modulation.
  • EANet adopts a U-shaped encoder-decoder with EAB-based downsampling, pixel-shuffle upsampling, and k middle EABs.
  • The training algorithm initializes Gaussian noise with T = 100 and predicts noise using fψ(It, t).

D. Optimization and Inference

EDiffSR stabilizes EANet optimization by reformulating the diffusion objective around ideal and predicted reversed states. Inference starts from random noise and numerically solves the reverse SDE to generate the SR image.

  • The optimization objective minimizes Euclidean distance to address instability caused by predicting an instantaneous noise distribution.
  • The ideal reversed state x*_{t−1} is defined from xt through a closed-form expression.
  • Transforming noise-distance optimization into distance between ideal and predicted states helps reduce instability because most reversed-state pixels are known.
  • Inference samples xT from a Gaussian distribution and iteratively solves the reverse SDE with the pretrained EANet and Euler–Maruyama method.
  • At each inference step, fψ predicts noise, whose score is substituted into the reverse diffusion equation to produce the super-resolved image ISR.

V. EXPERIMENT AND DISCUSSION

Extensive experiments on four remote-sensing datasets evaluate EDiffSR in both simulated and real-world scenarios.

  • V. EXPERIMENT AND DISCUSSION: EDiffSR is evaluated on four remote-sensing datasets under simulated and real-world conditions.The experiments assess the method's effectiveness across both types of scenarios.

A. Dataset

The study evaluates ×4 super-resolution using four public datasets, with AID providing the training and test splits. Experiments use established image-quality metrics and fixed implementation settings.

  • A. Dataset: Four public datasets—AID, DOTA, DIOR, and NWPU-RESISC45—are used for comprehensive evaluation.AID supplies 3,000 training images and 300 non-overlapping test images.
  • B. Implementation Details: The study focuses on ×4 SR, using five RCABs in CPEM and C = 64 in EANet.The noise-prediction network has depth 4, with specified EAB allocations across its layers.
  • B. Implementation Details: Training uses 500,000 iterations with batch size 4, AdamW optimization, cosine learning-rate decay, and diffusion length T = 100.All compared methods are retrained from scratch on AID without pre-training or fine-tuning.
  • B. Implementation Details: Seven metrics assess SR quality, including full-reference metrics for simulated degradations and NIQE plus AG for real-world images.NIQE and AG provide reference-free measures of perceptual quality and high-frequency details.

D. Comparison With State-of-the-Arts

Across simulated remote-sensing benchmarks, EDiffSR delivers strong perceptual, structural, and fidelity results while producing more detailed and natural visual reconstructions than competing approaches.

  • 1) Quantitative Comparison:: EDiffSR achieves favorable FID across most of 30 AID scene categories, outperforming IRSDE by an average margin of 1.63.The results are reported against all comparative models.
  • 2) Qualitative Comparison:: On AID, EDiffSR produces more natural, visually pleasing images with richer high-frequency details than comparative approaches.CNN methods show blur, whereas GAN methods can introduce artifacts; EDiffSR avoids both effects in the cited comparisons.
  • 1) Quantitative Comparison:: Across AID, DOTA, and DIOR, EDiffSR achieves the best FID and surpasses SR3 by 0.0647 in LPIPS.On DIOR, EDiffSR reaches LPIPS 0.1898 versus IRSDE's 0.2419.
  • 1) Quantitative Comparison:: EDiffSR obtains the best DISTS scores on DOTA and DIOR, despite SPSR's emphasis on structural-detail preservation.The comparison attributes the result to effective noise prediction by EANet relative to IRSDE's UNet under the same diffusion process.
  • 1) Quantitative Comparison:: Within diffusion-based methods, EDiffSR maintains the best PSNR/SSIM performance and improves AID PSNR over SR3 from 26.24dB to 27.40dB.PSNR-oriented models can score higher on PSNR and SSIM while producing over-smoothed content and weaker perceptual metrics.
  • 2) Qualitative Comparison:: CPEM helps EDiffSR recover contextual and edge information absent from bicubic-upsampled conditions used by IRSDE and SR3.The cited AID example highlights road marks as recovered context details close to the ground truth.
  • 2) Qualitative Comparison:: On DOTA, EDiffSR restores sharper building details and realistic road-line directions while avoiding distortions and oversharpening seen in alternatives.The comparison identifies CPEM as providing additional prior information for detail recovery aligned with the ground-truth distribution.
  • 2) Qualitative Comparison:: On DIOR, EDiffSR uniquely recovers the correct direction of building-roof lines and aligns high-frequency details with ground-truth spatial distributions.The cited examples attribute this result to high-frequency prior information introduced through CPEM.

3) Real-world Comparison:

Real-world evaluation on NWPU-RESISC45 shows that EDiffSR provides strong perceptual quality and high-frequency detail recovery without simulated degradations, with component ablations also examined.

  • 3) Real-world Comparison:: Without simulated degradations, EDiffSR is evaluated on real-world NWPU-RESISC45 images using NIQE and AG.The evaluation compares EDiffSR with state-of-the-art SR methods.
  • 3) Real-world Comparison:: EDiffSR achieves the best NIQE and AG performance, indicating natural images with more high-frequency edges and textures.NIQE reflects perceptual alignment, while AG reflects high-frequency detail information.
  • 3) Real-world Comparison:: On NWPU-RESISC45, HAT-L appears blurry, GAN methods produce pseudo-details, and SR3 struggles with precise dense-line edges.The visual comparison contrasts these artifacts and limitations with EDiffSR's results.
  • Ablation Analysis: The study also analyzes EANet across different channel numbers and compares EDiffSR variants without and with CPEM.The complete model is reported to recover more high-frequency details than the version without CPEM.

1) Component Analysis of EDiffSR:

EDiffSR’s components improve perceptual quality and efficiency through enriched conditional priors and lightweight noise prediction. Ablations and comparisons show gains in FID, parameter count, inference time, and high-frequency detail recovery, while sampling cost and degradation adaptability remain limitations.

  • Component Analysis of EDiffSR: EANet improves FID over vanilla UNet while reducing model size from 137.15M to 26.31M parameters.The ablation reports FID values of 30.83 for EANet and 32.68 for vanilla UNet.
  • Component Analysis of EDiffSR: CPEM substantially improves FID and helps recover high-frequency details such as edges and boundaries.The ablation finds a significant FID improvement after adding CPEM, while visual comparisons show better detail recovery.
  • Effectiveness of EANet: The multi-scale EAB design provides a modest FID improvement over single-scale convolution designs.Table VI compares 3×3, 5×5, and 7×7 single-scale depthwise convolutions with EDiffSR’s multi-scale design.
  • Model Efficiency: 26.79M parameters, 30.83 FID, and 27.75dB PSNR make EDiffSR lighter and better-performing than IRSDE in the reported comparison.The corresponding IRSDE values are 137.15M parameters, 32.42 FID, and 27.46dB PSNR.
  • Model Efficiency: 19.26s versus 137.61s makes EDiffSR seven times faster than SR3 during diffusion sampling.The comparison concerns inference time in the diffusion sampling process.
  • Overall Results: Across AID, DOTA, DIOR, and NWPU-RESISC45, EDiffSR outperforms CNN-, GAN-, and diffusion-based SR methods in data distribution and perceptual quality.The evaluation includes quantitative and qualitative experiments on four remote sensing datasets.
  • Limitations: Diffusion sampling remains computationally expensive, and EDiffSR has limited adaptability because it does not model multiple remote-sensing degradations.The authors identify faster sampling and extension to blind SR as future directions for real-world generalization.
Loading 2310.19288v1…