PaperlayerView exact revision on arXiv

Source-linked AI summary

Stochastic Super-Resolution for Downscaling Time-Evolving Atmospheric Fields with a Generative Adversarial Network

Jussi Leinonen, Daniele Nerini, Alexis Berne

arXiv:2005.10374v4eess.IVcs.LGphysics.ao-phstat.ML

TL;DR

Super-resolution downscaling must represent uncertainty because a low-resolution input can correspond to multiple plausible high-resolution fields. The paper introduces a recurrent stochastic GAN that generates time-evolving ensembles, finding realistic and temporally consistent sequences with near-correct variability across two atmospheric datasets.

  • Problem

    Earlier super-resolution methods often produced a single output despite the inherent uncertainty of reconstructing high-resolution climate fields.

  • Method

    The paper develops a recurrent conditional GAN that generates ensembles of high-resolution atmospheric sequences from low-resolution image sequences.

  • Results

    The GAN generated realistic, temporally consistent sequences with appropriate variability for precipitation and cloud optical thickness datasets.

  • Takeaways & Limitations

    Ensemble rank statistics and CRPS provide promising tools for evaluating conditional-GAN variability and image quality.

  • Takeaways & Limitations

    The current GAN is specific to a resolution-enhancement factor of 16 and would require retraining for other factors.

Abstract

from arXiv · show

Generative adversarial networks (GANs) have been recently adopted for super-resolution, an application closely related to what is referred to as "downscaling" in the atmospheric sciences: improving the spatial resolution of low-resolution images. The ability of conditional GANs to generate an ensemble of solutions for a given input lends itself naturally to stochastic downscaling, but the stochastic nature of GANs is not usually considered in super-resolution applications. Here, we introduce a recurrent, stochastic super-resolution GAN that can generate ensembles of time-evolving high-resolution atmospheric fields for an input consisting of a low-resolution sequence of images of the same field. We test the GAN using two datasets, one consisting of radar-measured precipitation from Switzerland, the other of cloud optical thickness derived from the Geostationary Earth Observing Satellite 16 (GOES-16). We find that the GAN can generate realistic, temporally consistent super-resolution sequences for both datasets. The statistical properties of the generated ensemble are analyzed using rank statistics, a method adapted from ensemble weather forecasting; these analyses indicate that the GAN produces close to the correct amount of variability in its outputs. As the GAN generator is fully convolutional, it can be applied after training to input images larger than the images used to train it. It is also able to generate time series much longer than the training sequences, as demonstrated by applying the generator to a three-month dataset of the precipitation radar data. The source code to our GAN is available at https://github.com/jleinonen/downscaling-rnn-gan.

I. INTRODUCTION

Super-resolution enhances image resolution, while atmospheric downscaling addresses the need for finer-scale fields and uncertainty-aware predictions. This paper introduces a stochastic recurrent GAN for generating plausible, temporally evolving high-resolution atmospheric ensembles.

  • Super-resolution enhances an image’s spatial resolution beyond its original resolution.
  • Atmospheric downscaling produces finer-resolution precipitation fields from coarser data to assess impacts such as extreme rainfall.
  • Single-image super-resolution is underdetermined, yet many methods produce one output and rarely estimate reconstruction uncertainty.
  • Climate and weather applications require quantified uncertainty, motivating randomized downscaling methods and stochastic GAN-based approaches.
  • The proposed stochastic recurrent GAN generates plausible high-resolution ensembles with temporally consistent evolution from low-resolution image sequences.

A. Overview

The model combines conditional GAN processing with residual convolutional blocks, recurrent ConvGRU layers, and progressive upsampling. Its generator accepts low-resolution sequences and produces high-resolution sequences, while the discriminator evaluates paired inputs.

  • The conditional GAN gives both networks a low-resolution image condition while distinguishing real from generated high-resolution images.
  • Both generator and discriminator use deep CNNs with residual blocks that can flexibly preserve inputs through near-identity mappings.
  • ConvGRU recurrent layers learn temporal field evolution while convolutional and residual blocks learn spatial structure.
  • The generator processes low-resolution tensors containing time steps, image dimensions, and variables before recurrent modeling and decoding.
  • Four upsampling blocks increase both spatial dimensions by two, producing a resolution enhancement factor of K = 16.
  • The discriminator combines encoded high- and low-resolution sequences, evaluates temporal consistency with ConvGRU, and applies global average pooling.

1) Encoding/downsampling:

The discriminator encodes high- and low-resolution sequences separately before combining them for further spatial and temporal assessment. Design choices were selected informally because a full ablation study was computationally impractical.

  • 1) Encoding/downsampling:: The high-resolution input is encoded through residual blocks with strided convolutions that downsample each time step.
  • 1) Encoding/downsampling:: The low-resolution input is processed without strided convolutions, preserving dimensions for later combination with the high-resolution representation.
  • 1) Encoding/downsampling:: The discriminator concatenates the two encoded representations and processes the joint output through additional residual blocks.
  • 1) Encoding/downsampling:: ConvGRU layers evaluate temporal consistency, followed by global average pooling over feature-map locations.
  • 1) Encoding/downsampling:: The architecture uses spectral normalization in the discriminator and leaky ReLU activations with negative slope 0.2 in most network components.
  • 1) Encoding/downsampling:: A full ablation study was impractical with available computational resources, so design choices were assessed informally using early training metrics and image quality.

C. Training

The GAN is trained with conditional WGAN-GP objectives, alternating discriminator and generator updates. Gradient penalties constrain discriminator outputs, while augmentation increases sample diversity and approximate rotational invariance.

  • The conditional WGAN-GP objectives use high-resolution sequences as real samples, low-resolution sequences as conditions, and noise as the stochastic input.
  • The gradient penalty uses randomly weighted averages between real and generated samples and constrains the otherwise unbounded discriminator output.
  • Training alternates five discriminator batches with one generator batch using batch size 16 and Adam learning rate 10^-4 for both networks.
  • The generator used 400000 training sequences and the discriminator 2 million sequences, requiring roughly 48 hours per application on an Nvidia P100 GPU.
  • Random rotations and mirroring increase sample diversity and make the GAN approximately invariant to 90° rotations, translation, and time shifts.

D. Validation

The validation evaluates image quality, spatial structure, ensemble scoring, and output variability using complementary metrics, including rank statistics adapted from ensemble forecasting.

  • Image-quality metrics: RMSE measures pixelwise reconstruction error between real and generated images.The generated pixel is defined as xgen,i = G(y, z)i.
  • Image-quality metrics: MS-SSIM and LSD evaluate whether generated images reproduce true spatial structure, with LSD comparing power spectra in decibels.Preal and Pgen denote the real and generated image power spectra.
  • Variability assessment: GAN variability is assessed by comparing normalized ground-truth ranks among ensemble predictions with a uniform distribution.Ranks should be uniformly distributed when observations and predictions share a distribution.
  • Variability assessment: KS and Kullback–Leibler divergence quantify different discrepancies between the observed rank distribution and the uniform distribution.KS compares cumulative distribution functions, whereas DKL relates to differences in probability information content.
  • Ensemble scoring: CRPS evaluates the complete ensemble by comparing its pixelwise CDF with the observation CDF.For an entire image, CRPS is the mean of pixelwise scores and reduces to mean absolute error for one ensemble member.

III. DATA

The study uses independent precipitation-radar and satellite cloud-optical-thickness datasets, with logarithmic preprocessing, 16×16 spatial averaging for inputs, and held-out validation and test samples.

  • Datasets: The MCH-RZC dataset contains 180000 sequences of eight 128×128 images from MeteoSwiss radar observations.Each pixel represents 1 km, and successive images are separated by 10 min.
  • Datasets: The GOES-COT dataset contains daytime cloud optical thickness observations from GOES-16 collected from April–December 2019.GOES-16 Mode 6 provides scans every 10 min during the stated period.
  • Preprocessing: The GAN uses transformed versions of log(R) or log(τ), with special processing for empty non-precipitating or non-cloudy regions.The logarithmic transformation smooths both variable distributions but requires handling empty regions.
  • Preprocessing: Low-resolution inputs are produced by averaging linear R or τ values over non-overlapping 16×16 high-resolution tiles before logarithmic mapping.Averaged values between 0 and θ are truncated to 0.
  • Evaluation design: Ten percent of samples from each dataset are reserved for validation, while separate 1024-sample test datasets assess generalization.Validation samples monitor training and are excluded from training.

IV. RESULTS

Examples from both atmospheric datasets show that the GAN produces multiple plausible reconstructions with shared large-scale structure, differing details, and temporal consistency, while the displayed examples are limited and manually selected.

  • MCH-RZC examples: MCH-RZC reconstructions preserve common scene structure while varying in spatial granularity across ensemble members.In Fig. 2a, reconstructions #1 and #2 are more spatially variable than #3, consistently over time.
  • MCH-RZC examples: Figure 2 compares real high-resolution sequences, 16×16 downsampled sequences, and three GAN reconstructions.The real sequence appears on the top row, the downsampled sequence on the second row, and reconstructions on the last three rows.
  • GOES-COT examples: GOES-COT reconstructions recover approximately correct cloud pattern scales and spatial structure despite contrasts lost during downsampling.Generated cloud locations differ between reconstructions, reflecting uncertainty about the correct solution.
  • GOES-COT examples: GOES-COT examples show temporal consistency because features such as empty regions retain their locations across successive generated frames.Cloud texture and cell structure still vary among reconstructions.
  • Scope of examples: The manually selected figures are a limited, non-representative sample and show only three ensemble members.Additional randomly selected examples and more ensemble members are provided in the supplement.

B. Reconstruction quality

Image-quality metrics evolve differently during training: RMSE and MS-SSIM saturate early, while LSD and especially CRPS continue improving longer. The authors caution that these metrics incompletely capture the quality of stochastic ensemble reconstructions.

  • RMSE and MS-SSIM improve rapidly during the first 15000 training sequences, then converge near equilibrium.
  • LSD continues improving until approximately 70000 sequences, especially for the MCH-RZC dataset.
  • CRPS, which uses all ensemble members, improves longer than single-image metrics but exhibits substantially more noise.
  • After switching to SGD, noise decreases in single-image metrics but not CRPS, with little overall effect except slight GOES-COT MS-SSIM degradation.
  • Subjective quality continues increasing until at least 100000 sequences, because standard metrics poorly capture plausible stochastic reconstructions and spatially displaced structures.
  • CRPS appears promising for evaluating conditional GANs because it detects improvement much longer than the other metrics.

C. Variability

The trained GAN produces close to the appropriate amount of output variability, while outperforming alternative methods on ensemble-oriented metrics and fine-scale detail.

  • Variability metrics improved throughout training, with slower improvement continuing beyond the saturation of image-quality metrics.Improvement continued until at least 300000 sequences; switching to SGD near 350000 sequences reduced metric oscillation.
  • The rank CDF differed from uniformity by at most 0.029 for MCH-RZC and 0.059 for GOES-COT.These results indicate close to the appropriate amount of variability, although the distributions were not identical.
  • In a clear majority of cases, the real sample fell within the ensemble of predictions.
  • The GAN generated more detail and visually more accurate reconstructions than Lanczos interpolation, RMSE-trained RCNN, and RainFARM.Lanczos and RCNN produced smooth outputs; RainFARM produced detail but reproduced high-resolution structure less effectively.
  • The GAN achieved the best LSD score by far and clearly outperformed RainFARM on ensemble metrics.RMSE and MS-SSIM were similar for GAN, Lanczos, and RCNN, while RCNN achieved the best values for those metrics.
  • One eight-frame 128×128 sequence took approximately 660 ms on a CPU and 20 ms on a P100 GPU for the GAN generator.With batch size 16, Lanczos took 11 ms and RainFARM 240 ms per sequence on CPU; the GAN was relatively resource intensive.

E. Generalization to larger images and longer sequences

The fully convolutional recurrent generator extends from small training inputs to larger fields and from eight-frame training sequences to longer time series, with occasional long-run artifacts.

  • The fully convolutional generator applies to different field sizes without modification when pixel size is unchanged and dimensions are divisible by 16.
  • The recurrent structure supports sequences longer or shorter than the eight-frame training sequences when frame intervals remain unchanged.
  • The generator was applied sequentially to independent June–August 2017 MCH-RZC full-frame data at 10-minute intervals.The 710×640-pixel frames were cropped to 704 pixels wide, and the ConvGRU hidden state was propagated between frames.
  • The GAN produced realistic reconstructions for much larger images containing different precipitation modes across regions.Figure 8 shows one generated full frame; the complete June–August sequence was supplied as an accompanying animation.
  • Some generator versions produced artifacts during very long runs, although architectural adjustment and model selection sufficiently suppressed them for the demonstrated sequence.A stabilization method was described for cases where artifacts could not be avoided.

V. SUMMARY AND CONCLUSIONS

The paper presents a recurrent, fully convolutional conditional GAN for realistic stochastic atmospheric downscaling and evaluates its variability with ensemble statistics. It generalizes across data types, larger images, and longer sequences, while retaining scope limitations and open evaluation challenges.

  • The conditional GAN operates on image sequences, uses recurrence for temporal consistency, and generates high-resolution ensembles with stochastic variability.
  • Rank metrics converge more slowly than MS-SSIM and RMSE, making them useful for monitoring training after image-quality metrics saturate.The CPRS metric also appears to provide a better estimate using the entire ensemble.
  • The ensemble metrics appear promising for evaluating conditional GAN quality and variability beyond geoscience applications.
  • The GAN produced realistic high-resolution fields with appropriate variability across two distinct applications and generalized to larger images and longer sequences.Training can use relatively short sequences of small images before evaluation on larger or longer inputs.
  • The current GAN is specific to a scaling factor of 16 and does not yet support different scaling factors without further development.
  • Future work includes temporal resolution enhancement, auxiliary variables, architecture optimization, and feature-based ensemble evaluation.The proposed rank-based evaluation is currently pixelwise, motivating possible feature-based alternatives.

APPENDIX OPTIONAL STABILIZATION FOR LONG TIME SERIES

Long recurrent rollouts can produce artifacts even when the generator is stable over the eight training frames. An optional state-relaxation procedure suppresses these artifacts, but may reduce output variability.

  • Observed instability: Long recurrent runs exposed artifacts in some generator versions despite stability over the 8 training frames.The authors avoided this issue in their reported experiments but note that short-sequence stability did not guarantee long-term stability.
  • Stabilization procedure: After each update step, the evaluation procedure adjusts the ConvGRU state toward the null state produced for an all-zeros input.The adjustment uses a relaxation constant λr, with 0.01 ≤λr ≤0.2 tested for the MCH-RZC dataset.
  • Trade-off: The relaxation suppresses artifacts while preserving operation on the previous state, but appears to reduce rather than eliminate generated-image variability.The authors recommend using it only when generator improvements cannot remove the artifacts.
Loading 2005.10374v4…