PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model

Xingjian Shi, Zhihan Gao, Leonard Lausen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, Wang-chun Woo

arXiv:1706.03458v2cs.CV

TL;DR

Precipitation nowcasting needs high-resolution short-range rainfall forecasts, but existing recurrent models use location-invariant connections and the field lacks clear evaluation protocols. The paper introduces TrajGRU, which learns location-variant recurrent structures, and a benchmark with real-world data, balanced losses, and comprehensive protocols. Across the benchmark, deep learning outperforms optical flow, TrajGRU performs best among the deep-learning models, and online fine-tuning improves results over offline testing.

  • Problem

    Precipitation nowcasting uses location-invariant recurrent connections despite location-variant motion, while clear evaluation protocols for the emerging field remain unavailable.

  • Method

    The paper introduces TrajGRU, which learns recurrent connection structures, and a benchmark with HKO-7 data, offline and online protocols, and balanced losses.

  • Results

    Deep learning models outperform optical-flow models; TrajGRU has the best overall deep-learning performance, and online fine-tuning consistently outperforms offline testing.

  • Takeaways & Limitations

    The benchmark provides a comprehensive basis for evaluating precipitation-nowcasting models and evidence that online learning could potentially help video prediction more generally.

Abstract

from arXiv · show

With the goal of making high-resolution forecasts of regional rainfall, precipitation nowcasting has become an important and fundamental technology underlying various public services ranging from rainstorm warnings to flight safety. Recently, the Convolutional LSTM (ConvLSTM) model has been shown to outperform traditional optical flow based methods for precipitation nowcasting, suggesting that deep learning models have a huge potential for solving the problem. However, the convolutional recurrence structure in ConvLSTM-based models is location-invariant while natural motion and transformation (e.g., rotation) are location-variant in general. Furthermore, since deep-learning-based precipitation nowcasting is a newly emerging area, clear evaluation protocols have not yet been established. To address these problems, we propose both a new model and a benchmark for precipitation nowcasting. Specifically, we go beyond ConvLSTM and propose the Trajectory GRU (TrajGRU) model that can actively learn the location-variant structure for recurrent connections. Besides, we provide a benchmark that includes a real-world large-scale dataset from the Hong Kong Observatory, a new training loss, and a comprehensive evaluation protocol to facilitate future research and gauge the state of the art.

1 Introduction

Precipitation nowcasting supports short-range rainfall forecasts but faces challenges from atmospheric complexity, operational demands, and insufficient evaluation practices. This paper addresses these gaps with a benchmark incorporating new data, protocols, losses, and extensive model comparisons.

  • Motivation: Precipitation nowcasting forecasts local rainfall intensity over very short ranges, supporting road-condition prediction, aviation guidance, and rainfall alerts.It uses radar echo maps, rain gauges, other observations, and Numerical Weather Prediction models.
  • Existing approaches: Optical-flow systems estimate cloud motion and advect radar maps, but do not exploit the large quantities of existing radar echo data through supervised learning.The conventional pipeline first estimates motion and then predicts future radar maps using semi-Lagrangian advection.
  • Evaluation gap: 97 rainy days and a single 0.5mm/h skill-score threshold were insufficient to establish overall performance, especially for heavier rainfall events.The paper identifies unclear evaluation practices as a problem for real-world precipitation nowcasting.
  • Proposed benchmark: The HKO-7 benchmark contains radar echo data from 2009 to 2015 and evaluates both offline fixed-window forecasting and online learning from historical data.The benchmark reflects the stream-like arrival of radar observations in real-world operation.
  • Proposed benchmark: Balanced Mean Squared Error and Balanced Mean Absolute Error assign greater weight to heavier rainfall, addressing imbalanced rain-rate distributions during training and evaluation.Heavier rainfall occurs less often but has higher real-world impact.
  • Experimental scope: The benchmark evaluates seven models, including a last-frame baseline, optical-flow methods, and representative 2D CNN, 3D CNN, ConvGRU, and TrajGRU models.The evaluation is designed to compare conventional and deep-learning approaches comprehensively.
  • Main findings: Deep learning models outperform optical-flow models, TrajGRU achieves the best overall deep-learning performance, and online fine-tuning consistently outperforms offline testing.The paper presents this as its comprehensive benchmark finding for precipitation nowcasting.

2 Related Work

Related work frames precipitation nowcasting as a spatiotemporal video-prediction problem and surveys recurrent architectures that model spatial structure. It also highlights the absence of a precipitation-specific benchmark despite existing benchmarks for other video tasks.

  • Video prediction: Radar reflectivity is converted to grayscale images, making precipitation nowcasting a video-prediction problem with a fixed weather-radar camera.Methods for predicting future frames in natural videos can therefore be applied to precipitation nowcasting.
  • Video prediction: Video-prediction architectures include recurrent neural networks, 2D CNNs, and 3D CNNs, with ConvLSTM extending LSTM-based prediction to capture spatiotemporal correlations.The related work traces recurrent video prediction from convolutional recurrent models to encoder-decoder and ConvLSTM approaches.
  • Structured recurrent connections: Structured recurrent models replace fully connected recurrent connections with alternative topologies to improve spatiotemporal relationship modeling.SocialLSTM uses distances between people, S-RNN uses a given spatiotemporal graph, and TrajGRU actively learns its recurrent structure.
  • Structured recurrent connections: TrajGRU differs from SocialLSTM and S-RNN by actively learning the recurrent connection structure rather than relying on a predefined topology.Structure-evolving LSTM also learns connection structure but was designed for semantic object parsing.
  • Benchmarks: Benchmarks exist for online object tracking and video object segmentation, but precipitation nowcasting lacks one tailored to radar data and highly imbalanced rainfall.The paper positions its large-scale benchmark as filling this domain-specific gap.

3 Model

The model uses an encoding-forecasting recurrent architecture and introduces TrajGRU to learn location-variant recurrent connections through dynamically generated local neighborhoods.

  • 3.1 Encoding-forecasting Structure: The encoding network maps previous observations into recurrent states, while a reversed forecasting network generates future frames through interleaved downsampling and upsampling layers.High-level states guide low-level state updates, and low-level states further influence predictions.
  • 3.2 Convolutional GRU: ConvGRU uses convolutional recurrent connections with fixed transition structure and weights shared across spatial locations.Its neighborhood is determined by fixed convolution hyperparameters, including kernel size, dilation, and padding.
  • 3.3 Trajectory GRU: Unlike fixed convolutional neighborhoods, TrajGRU can model location-variant motion patterns such as rotation and scaling.Its number of local links can be smaller than a comparable convolutional kernel, potentially reducing parameters and improving parameter efficiency.
  • 3.1 Encoding-forecasting Structure: The encoding-forecasting example predicts two future frames from two input frames using three recurrent networks, with spatial coordinates concatenated to inputs.ConvGRU or TrajGRU can fill the recurrent-network roles, and missing input links receive zeros.
  • 3.3 Trajectory GRU: TrajGRU generates a local neighborhood for each location and timestamp from the current input and previous state.Continuous optical flows represent the discrete, non-differentiable connection indices.
  • 3.3 Trajectory GRU: TrajGRU applies learned flow fields and bilinear sampling to select locations from the previous recurrent state.The structure-generating network produces flow fields, while 1 × 1 convolutions project channels for the recurrent updates.

4 Experiments on MovingMNIST++

MovingMNIST++ evaluates models on video prediction with varied motion and appearance changes, and the experiments show that TrajGRU benefits from more learned links and captures rotation-related structures.

  • Dataset and setup: MovingMNIST++ extends MovingMNIST with random rotations, scale changes, and illumination changes in 64 × 64 frames containing three moving digits.The task uses 10 input frames to predict the next 10 frames.
  • Dataset and setup: The test set contains 10,000 sequences, and all models are trained with three seeds to report standard deviation.The MovingMNIST++ MSE is averaged over both frame size and predicted-sequence length.
  • Results: TrajGRU performance improves as the number of links increases, and TrajGRU-L13 outperforms a 7 × 7-kernel ConvGRU with fewer parameters.The results support using learned local connection structures for the dataset’s more complex motion patterns.
  • Learned structures: Visualized TrajGRU-L13 links exhibit learned behavior for rotations across the first, second, and third encoder layers.The figure displays one selected link at different frames and layers, with first-layer links subsampled for readability.

5 Benchmark for Precipitation Nowcasting

The benchmark combines the HKO-7 radar dataset, offline and online evaluation protocols, rainfall-aware metrics and balanced losses to assess precipitation nowcasting across rainfall intensities.

  • 5.1 HKO-7 Dataset: HKO-7 contains radar echo data from 2009–2015, with 812 rainy days for training, 50 for validation and 131 for testing.The dataset uses 480×480-pixel CAPPI images recorded every six minutes and covering a 512km × 512km area centered on Hong Kong.
  • 5.2 Evaluation Methodology: The benchmark evaluates offline forecasting from 5 input frames to 20 future frames and online forecasting on sequential 5-frame segments.Online learning allows algorithms to adapt to newly emerging spatiotemporal patterns.
  • 5.2 Evaluation Methodology: Skill scores are computed at rainfall thresholds of 0.5, 2, 5, 10 and 30mm/h using CSI and HSS.Thresholding converts predictions and ground truth to binary fields before calculating contingency-based scores.
  • 5.3 Training Loss: The weighted loss assigns pixel weights according to rainfall intensity to address highly imbalanced rainfall-level frequencies.Balanced B-MSE and B-MAE use these weights, while conventional MSE and MAE set weights to 1 except at masked points.
  • 5.4 Evaluation Results: Balanced losses improve heavy-rainfall nowcasting: all balanced deep models outperform optical-flow methods, with TrajGRU performing best among deep models.Online fine-tuning consistently improves performance over offline evaluation, and TrajGRU’s B-MSE and B-MAE improvements exceed three standard deviations.
  • 5.4 Evaluation Results: B-MSE and B-MAE have stronger correlations with CSI and HSS than MSE and MAE in most cases.The comparison uses Kendall’s τ coefficients across skill scores and rainfall thresholds.

6 Conclusion and Future Work

The paper concludes that its TrajGRU model and large-scale benchmark advance precipitation nowcasting by learning recurrent connection structures and evaluating models systematically.

  • Conclusion: The paper provides the first large-scale precipitation-nowcasting benchmark and proposes TrajGRU, which learns the recurrent connection structure.The authors state that TrajGRU captures spatiotemporal correlations more efficiently than ConvGRU.
  • Future Work: The authors plan to test TrajGRU on visual object tracking and video segmentation.They also plan to build an operational nowcasting system using the proposed algorithm.
  • Implementation: The model weights and biases use MSRA initialization, except that TrajGRU structure-generating-network parameters are initialized to zero.This initialization choice is specific to the structure-generating network in TrajGRUs.
  • Implementation: The structure-generating network uses two convolution layers, with the second producing 2L filters for L recurrent links.The first layer uses a 5×5 kernel, 32 filters and leaky ReLU activation.

C.1 Generation Process

The appendix describes synthetic MovingMNIST++ generation and the configurations used for CNN, recurrent and DFN baselines.

  • C.1 Generation Process: The generation process samples velocity, scaling, rotation angle and illumination factor uniformly within specified ranges.These ranges are listed in the MovingMNIST++ hyperparameter table.
  • Model Configurations: The paper illustrates the general structures of the 2D CNN, 3D CNN and DFN models and applies batch normalization in the CNNs.Detailed configurations are provided for 2D CNN, 3D CNN, ConvGRU, DFN and TrajGRU.
  • Model Configurations: The 2D CNN uses 10 input and output channels for the observed and predicted frames, while its base filter number is 70.The 2D model is derived from the 3D model by scaling channels according to the 3D kernel size.
  • Model Configurations: The 3D CNN uses base filter number 128, while the ConvGRU, DFN and TrajGRU configurations are documented in separate tables.The TrajGRU table specifies L as the number of links in the state-to-state transition.

D.1 Overall Data Statistics

The appendix reports HKO-7’s rainfall statistics and the preprocessing pipeline for identifying and removing radar-image outliers and noise.

  • D.1 Overall Data Statistics: The HKO-7 dataset’s overall statistics and monthly average rainfall distributions are reported in Figure 5 and Table 12.The rain-rate statistics are summarized in the HKO-7 dataset table.
  • D.2 Denoising Process: Ground clutter and sun spikes are detected by identifying locations with anomalous pixel-value patterns.The resulting outlier mask is illustrated in Figure 6.
  • D.2 Denoising Process: Locations whose Mahalanobis distance exceeds the mean distance plus three standard deviations are classified as outliers.The 480×480 image contains 177316 inliers, 2824 outliers and 50260 out-of-boundary points after detection.
  • D.2 Denoising Process: After outlier detection, pixels below 71 and above 0 are filtered to remove additional noise such as sea clutter.This filtering follows the outlier-detection step in the denoising pipeline.

D.3 Evaluation Protocol

The HKO-7 benchmark evaluates models in offline or online settings, with online evaluation allowing updates from previously seen sequences. The protocol also specifies model prediction, environment upload, and architecture configurations for several CNN and recurrent models.

  • Evaluation protocol: HKO-7 evaluation supports offline and online modes, with online models storing previously seen sequences and fine-tuning from sampled buffer batches.When available, the last 25 consecutive buffer frames are sampled for updates, and the buffer is then emptied.
  • Evaluation protocol: Algorithm 1 processes input frames, stores them in the model, optionally updates online, predicts future frames, uploads predictions, and saves the evaluation state.The procedure repeats until the evaluation environment ends and tracks whether each input is from a new episode.
  • Model configurations: The benchmark documents a common network structure for 2D and 3D CNNs and provides separate configurations for 2D CNN, 3D CNN, ConvGRU, and TrajGRU models.These configurations are listed in Tables 13–16.
  • Model configurations: The 2D CNN uses base filter number c = 70, derives its structure from the 3D model, and assigns its first 5 and last 20 channels to input and output frames.The table defines kernel, stride, padding, and related dimensions as height and width.
  • Model configurations: The 3D CNN uses base filter number c = 128, while the ConvGRU and TrajGRU tables reference the corresponding hyperparameter definitions from earlier model descriptions.The 3D model’s kernel, stride, padding, and related dimensions represent channel, height, and width.
Loading 1706.03458v2…