PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions

Haixu Wu, Zhiyu Yao, Jianmin Wang, Mingsheng Long

arXiv:2103.02243v3cs.CV

TL;DR

Video prediction models often focus on temporal state transitions while overlooking complex motion variations that change across space and time. MotionRNN addresses this gap by decomposing motion into transient variation and trend, modeling both with MotionGRU and a Motion Highway integrated into RNN predictors. It achieves significant improvements and state-of-the-art performance across three challenging prediction tasks.

  • Problem

    Existing video prediction methods mainly model spatiotemporal state transitions but overlook complex within-motion variations, limiting prediction of ever-changing motions.

  • Method

    MotionRNN combines a MotionGRU that captures transient variation and accumulated motion trend with a Motion Highway for flexible integration into RNN-based predictive models.

  • Results

    MotionRNN produces significant improvements and state-of-the-art performance across three challenging prediction tasks.

  • Takeaways & Limitations

    Explicitly modeling motion within state transitions and preserving it across layers makes the framework adaptable to spacetime-varying motions and existing predictive models.

Abstract

from arXiv · show

This paper tackles video prediction from a new dimension of predicting spacetime-varying motions that are incessantly changing across both space and time. Prior methods mainly capture the temporal state transitions but overlook the complex spatiotemporal variations of the motion itself, making them difficult to adapt to ever-changing motions. We observe that physical world motions can be decomposed into transient variation and motion trend, while the latter can be regarded as the accumulation of previous motions. Thus, simultaneously capturing the transient variation and the motion trend is the key to make spacetime-varying motions more predictable. Based on these observations, we propose the MotionRNN framework, which can capture the complex variations within motions and adapt to spacetime-varying scenarios. MotionRNN has two main contributions. The first is that we design the MotionGRU unit, which can model the transient variation and motion trend in a unified way. The second is that we apply the MotionGRU to RNN-based predictive models and indicate a new flexible video prediction architecture with a Motion Highway that can significantly improve the ability to predict changeable motions and avoid motion vanishing for stacked multiple-layer predictive models. With high flexibility, this framework can adapt to a series of models for deterministic spatiotemporal prediction. Our MotionRNN can yield significant improvements on three challenging benchmarks for video prediction with spacetime-varying motions.

1. Introduction

MotionRNN addresses the difficulty of predicting motions that vary across space and time by modeling both transient variation and motion trend. Its MotionGRU and Motion Highway extend flexible RNN-based prediction, achieving state-of-the-art results on three challenging benchmarks.

  • Real-world motions vary across both space and time, making accurate prediction challenging for deformation, accumulation, and dissipation processes.
  • Existing predictive models emphasize temporal state transitions but overlook internal motion variations, limiting accuracy under highly changing scenarios.Optical-flow methods additionally lack characterization of long-term motion trends and may degenerate when motions continually change.
  • Physical motions can be decomposed into transient variation and motion trend, with local changes occurring alongside persistent tendencies governed by physical processes.Examples include alternating leg movements alongside forward motion, and meteorological behavior in radar echoes.
  • MotionGRU captures transient variation from spatiotemporal information and obtains motion trend from previous accumulation within a unified unit.
  • MotionRNN combines MotionGRU with a Motion Highway to reduce motion vanishing across layers and adapt existing predictive models to spacetime-varying motions.
  • MotionRNN achieves new state-of-the-art performance on three challenging benchmarks and yields consistent improvements across a range of predictive backbones.

2. Related Work

Related work uses recurrent architectures, convolutional spatiotemporal modeling, optical flow, and stochastic future modeling for video prediction. MotionRNN instead adds an external module that explicitly models within-motion variation while complementing existing RNN-based state-transition methods.

  • RNN-based video prediction models temporal dependencies, while ConvLSTM integrates convolutions to preserve spatial information in sequence modeling.
  • PredRNN propagates memory through stacked recurrent layers, and E3D-LSTM adds 3D convolution and memory attention for long-term dynamics.
  • Optical-flow methods learn transformations or receptive-area offsets between frames to capture coherence, but prior spatiotemporal models can miss internal motion variations.
  • MotionRNN is a flexible external module for RNN-based predictors that explicitly learns transient variation and remembers motion trend without changing the original predictive framework.It complements existing methods that learn spatiotemporal state transitions.
  • Stochastic video prediction methods use adversarial training or variational autoencoders to model future uncertainty, but do not explicitly model motion variation.

3. Methods

MotionRNN inserts MotionGRU units and a Motion Highway into stacked RNN predictive models to represent both transient motion variation and accumulated motion trends. Its flexible design preserves original state-transition flows while adding motion-based hidden-state transitions and cross-layer motion context.

  • MotionRNN Architecture: MotionRNN inserts MotionGRU between stacked predictive blocks and adds a Motion Highway without changing the original state-transition flow.The framework can adapt to ConvLSTM, PredRNN, MIM, E3D-LSTM, and other RNN-based predictive models.
  • Motion Highway: The Motion Highway provides an alternative route for motion context across layers, balancing invariant content with changeable motion and mitigating motion vanishing.It reuses the predictive block’s output gate to expose unchanging content information and supplement hidden states with additional details.
  • MotionGRU State Transitions: MotionGRU learns pixel-wise offsets between adjacent hidden states and uses them to conduct motion-based state transitions.Warping the previous state with the learned motion explicitly incorporates motion variation into hidden-state transitions.
  • Motion Decomposition: MotionGRU decomposes motion into transient variation and motion trend, modeling both components in a unified recurrent unit.Transient variation captures locally changing motion, while trending momentum accumulates motion tendency from previous filters.
  • Transient Variation: The transient-variation learner adapts ConvGRU to use spatial context and temporal coherence when estimating current motion variation.The learned transient variation uses the current encoded hidden state and the previous motion filter; the filters and gates operate in offset space.
  • Trending Momentum: Trending momentum estimates motion tendency through accumulation, converging toward a weighted sum of past motion filters.This accumulation allows the model to represent longer-term tendencies such as forward motion while future frames remain unavailable.

4. Experiments

MotionRNN is evaluated on three challenging benchmarks and across diverse predictive backbones, with results showing consistent gains in quantitative and qualitative motion prediction. Its components improve accuracy while adding limited model size and computation.

  • Experimental benchmarks: The experiments cover human motions, precipitation nowcasting, and varied moving digits, using benchmarks with complex spacetime-varying dynamics.The human benchmark uses Human3.6M, precipitation uses Shanghai radar observations, and V-MNIST varies digit motion, rotation, and scale.
  • Experimental setup: MotionRNN is combined with ConvLSTM, PredRNN, MIM, and E3D-LSTM backbones to test its flexibility across predictive models.The evaluated models use four stacked blocks with 64-channel hidden states.
  • Human motion results: 29% lower MSE and 22% lower MAE are reported when MotionRNN uses PredRNN as the backbone on Human3.6M.The paper also reports improved FVD and state-of-the-art performance for the PredRNN-based model on Human3.6M.
  • Human motion results: MotionRNN produces richer, less blurry human-motion frames and more precise arm and leg pose predictions than previous models.The qualitative comparison particularly highlights sharper detail in the arms and legs.
  • Complexity analysis: MSE decreases from 48.4 to 34.2 and SSIM increases from 0.781 to 0.846 with only 9.3% additional parameters and 7.6% more computations.The added computation and model size are attributed to MotionRNN operating as an external operator for hidden states across layers.
  • Ablation study: The Motion Highway and MotionGRU jointly deliver a 29% improvement in the ablation study, while removing the highway loses arm details and causes positional skewing.The Motion Highway alone provides 12% improvement, MotionGRU alone 17%, and trending momentum and transient variation provide 9% and 13% extra promotions.
  • Precipitation nowcasting: MotionRNN improves radar-echo prediction across CSI metrics and achieves 24% improvement on GDL with E3D-LSTM.Qualitative results show more precise rotation and diffusion details than PredRNN.
  • Varied moving digits: MotionRNN consistently improves all metrics on V-MNIST, producing sharper digit predictions under simultaneous motion, rotation, and scaling.PredRNN shows especially strong gains in MSE and GDL.

5. Conclusion

MotionRNN models spacetime-varying motions by separating transient variation from motion trend and integrating MotionGRU with a motion highway. Applied across predictive models, it achieves significant improvements and state-of-the-art performance on three challenging tasks.

  • MotionRNN decomposes motion into transient variation and motion trend, then models both explicitly in state transitions.
  • MotionGRU captures transient variation and motion tendency within RNN-based predictive frameworks.
  • The motion highway helps prevent motion information from vanishing across stacked predictive layers.
  • MotionRNN achieves significant improvements and state-of-the-art performance on three challenging prediction tasks.

A. MotionGRU: Implementation Details

MotionGRU implements motion-based hidden-state transitions by learning transient and trend offsets, combining them into motion filters, warping encoded states, and decoding a gated output.

  • An encoder-decoder structure reduces memory demands while converting hidden states into lower-resolution motion representations.
  • MotionGRU represents transient variation and trending momentum as pixel-wise offsets of the hidden state.
  • The learned offsets form a motion filter that defines a motion-based state transition over encoded hidden features.
  • Warp samples encoded hidden features over offset-defined k × k neighborhoods using bilinear interpolation.
  • A sigmoid output gate combines the motion-based transited state with the decoded feature to produce the output state.

B. Visualization of Ablation Study

The ablation visualizations show that transient variation and trending momentum contribute different kinds of motion information in human-motion prediction.

  • Removing transient variation causes predictions to lose important hand and leg movement details.
  • Removing trending momentum causes the model to predict the human’s back position incorrectly.
  • Figure 11 provides additional visualized ablation cases for comparing the effects of the model components.

C. The Detail of Motion Trend Visualization

The motion-trend visualization extracts and aggregates learned offsets, then displays their directions as arrows. In the human case, the learned trend matches the global left-to-right movement.

  • The visualization splits encoded hidden features into vertical and horizontal offsets for each position and filter neighborhood.
  • Mean channel-wise offsets are regularized in length and displayed as arrows indicating motion tendency.
  • Table 6 reports quantitative Radar Echo results across network backbones, where higher CSI indicates better performance.
  • The human-case visualization shows a learned left-to-right trend corresponding to the global movement.

D. Guangzhou Benchmark

The Guangzhou benchmark evaluates forecasting in a rainy radar setting using ten observed frames to predict ten future frames, with CSI measured at three thresholds. MotionRNN improves prediction quality, particularly for severe-weather echoes and thick clouds.

  • The Guangzhou radar dataset contains echoes collected every 6 minutes from May 1 to June 30, 2014, and represents weather forecasting in rainy areas.
  • The experiment uses the previous 10 frames to generate the future 10 frames and evaluates CSI at 30, 40, and 50 dBZ thresholds.
  • MotionRNN consistently improves CSI across thresholds, including against the state-of-the-art MIM method.
  • MotionRNN particularly improves CSI50, indicating enhanced forecasts of severe weather.
  • MotionRNN produces more precise results, with better detail prediction and greater accuracy for thick-cloud echoes.

E. More Qualitative Results

Additional qualitative examples examine radar, human-motion, and synthetic-digit prediction. Across these cases, MotionRNN preserves details and produces sharper, more accurate forecasts.

  • The qualitative examples focus attention on regions highlighted in red.
  • On Human3.6M, MotionRNN preserves details of human arms that vanish in PredRNN predictions.
  • On Radar Shanghai, MotionRNN generates sharper prediction results than the comparison shown in the qualitative examples.
  • On the synthetic V-MNIST dataset, MotionRNN generates eidetic predictions of the number “3”.
  • Together, the qualitative results show improved prediction accuracy and sharpness with MotionRNN.
Loading 2103.02243v3…