PaperlayerView exact revision on arXiv

Source-linked AI summary

A Review on Deep Learning Techniques for Video Prediction

Sergiu Oprea, Pablo Martinez-Gonzalez, Alberto Garcia-Garcia, John Alejandro Castro-Vargas, Sergio Orts-Escolano, Jose Garcia-Rodriguez, Antonis Argyros

arXiv:2004.05214v2cs.CVcs.LGeess.IV

TL;DR

Video prediction matters for intelligent decision-making and self-supervised representation learning, but future frames become difficult to model because of uncertainty and high-dimensional visual variation. This review synthesizes deep-learning methods, datasets, architectures, and evaluations, finding broad design patterns while identifying persistent long-horizon and resolution limitations.

  • Problem

    Video prediction must represent future outcomes from visual sequences despite high-dimensional pixel variation and inherently uncertain long-term futures.

  • Method

    The paper reviews and classifies more than 50 deep-learning video prediction methods, their datasets, architectures, contributions, and evaluation results.

  • Results

    The review finds that recurrent connections and stochasticity generally improve performance, while autoregressive multi-step prediction accumulates errors over long horizons.

  • Takeaways & Limitations

    Video prediction is a promising framework for self-supervised learning of rich spatio-temporal correlations for intelligent decision-making systems.

  • Takeaways & Limitations

    Existing methods remain limited to short-term horizons, with long-term error accumulation and many models restricted to low-resolution predictions.

Abstract

from arXiv · show

The ability to predict, anticipate and reason about future outcomes is a key component of intelligent decision-making systems. In light of the success of deep learning in computer vision, deep-learning-based video prediction emerged as a promising research direction. Defined as a self-supervised learning task, video prediction represents a suitable framework for representation learning, as it demonstrated potential capabilities for extracting meaningful representations of the underlying patterns in natural videos. Motivated by the increasing interest in this task, we provide a review on the deep learning methods for prediction in video sequences. We firstly define the video prediction fundamentals, as well as mandatory background concepts and the most used datasets. Next, we carefully analyze existing video prediction models organized according to a proposed taxonomy, highlighting their contributions and their significance in the field. The summary of the datasets and methods is accompanied with experimental results that facilitate the assessment of the state of the art on a quantitative basis. The paper is summarized by drawing some general conclusions, identifying open research challenges and by pointing out future research directions.

1 INTRODUCTION

Video prediction supports intelligent decision-making by forecasting future frames under difficult visual conditions. This review focuses on deep-learning techniques, datasets, model taxonomies, evaluation, and open challenges.

  • Predicting future events supports intelligent decision-making but is difficult because of occlusions, camera movement, lighting, clutter, and object deformations.
  • Future video prediction has been applied to autonomous driving, robot navigation, human-machine interaction, activity anticipation, planning, segmentation, anomaly detection, and precipitation nowcasting.
  • Deep architectures learn representations from high-dimensional data end to end without hand-engineered features, enabling spatio-temporal correlation extraction from videos.
  • The review concentrates on predicting future video frames from previous-frame sequences and treats video prediction, future-frame prediction, forecasting, and generation as interchangeable terms.
  • The paper surveys terminology, background concepts, datasets, prediction methods, evaluation metrics, quantitative results, and future research directions.

2 VIDEO PREDICTION

Video prediction forecasts subsequent frames from preceding context and connects predictive coding with self-supervised representation learning. Its central difficulties are temporal variation, uncertainty, and loss-induced blurry predictions.

  • Video prediction is an intermediate step between raw video and decision making that can extract meaningful spatio-temporal representations for self-supervised learning.
  • Given n context frames, the task predicts the next m frames, with frame dimensions represented by width, height, and channel count.
  • Because future frames already exist in the input sequence, video prediction requires no additional labels or human supervision and is therefore self-supervised.
  • Temporal coherence provides local visual similarity while occlusions, lighting changes, and camera motion create larger-scale semantic and appearance variation.
  • When multiple futures are equally probable, deterministic models average them, producing blurry predictions especially over long horizons.
  • Approaches to reduce blur include improved pixel-wise losses, adversarial training, higher-level representations, probabilistic models, and regularization such as GDL and TV.

3 BACKBONE DEEP LEARNING ARCHITECTURES

Video prediction systems use convolutional, recurrent, and generative architectures as core building blocks. These architectures address spatial structure, temporal dependencies, and distribution modeling with distinct strengths and limitations.

  • Convolutional Models: CNNs model image spatial structure but are limited by receptive fields and weak explicit inter-frame modeling.
  • Convolutional Models: Long-range spatial dependencies can be addressed with deeper or dilated convolutions, larger receptive fields, multiscale combinations, pooling, and residual connections.
  • Convolutional Models: Three-dimensional convolutions capture temporal consistency as an alternative to recurrent modeling, while learned affine transforms can act as recurrent states.
  • Recurrent Models: Recurrent neural networks are designed to represent spatio-temporal structure in sequential data and have been widely used for video prediction.
  • Generative Models: Generative models represent data distributions either explicitly, as in PixelRNNs and PixelCNNs, or implicitly, as in VAEs and GANs.
  • Generative Models: GANs improved prediction quality and sharpness, but adversarial training is unstable and can suffer from mode collapse or blurry outputs when losses are poorly balanced.

4 DATASETS

Video prediction datasets span action recognition, human pose, driving, urban scenes, sports, and broad online media. They differ in motion content, annotations, sensing modalities, scale, and realism.

  • Self-supervised video prediction mainly requires sequences, while some methods additionally use segmentation maps or human poses as supervisory signals.
  • Action and Human-Pose Datasets: KTH and Weizmann provide small, static-camera action datasets with homogeneous backgrounds, and are often combined to increase available data.
  • Action and Human-Pose Datasets: UCF101 is widely used, but few sequences represent movement because many actions occur over fixed backgrounds.
  • Driving and Urban Datasets: CamVid, CalTech, Kitti, Cityscapes, Comma.ai, and Apolloscape support driving and urban-scene prediction with annotations or sensors including semantics, pedestrians, depth, stereo, pose, and steering.

5 VIDEO PREDICTION METHODS

Video prediction methods evolved from raw-pixel modeling toward higher-level representations and probabilistic approaches that address dimensionality and future uncertainty. The review organizes these methods using a taxonomy based on the prediction problems they address.

  • Early models directly predicted raw pixel intensities while implicitly modeling scene dynamics and low-level details.
  • High-level representation spaces simplify prediction because intelligent systems rarely rely on raw pixel information for decision making.
  • Probabilistic models address future uncertainty that deterministic video prediction models cannot manage effectively.
  • The review proposes a taxonomy organized by the video prediction problem addressed, allowing methods to occupy multiple categories when they solve multiple problems.

5.1 Direct Pixel Synthesis

Direct pixel synthesis methods predict future frames from visual inputs, progressing from patch-level and recurrent models to adversarial, convolutional, and context-aware architectures. Their central challenges include blurry long-term predictions, limited context, and uncertainty under complex motion.

  • Initial models predicted future pixel intensities without explicitly modeling scene dynamics, producing accurate large-object motion but tiling artifacts and weaker small-object predictions.
  • Autoencoder and LSTM approaches modeled temporal coherence using RGB, convolutional, and flow inputs, while conditioning the decoder only marginally improved accuracy.
  • Adversarial multi-scale training improved prediction sharpness, while PredNet subsequently outperformed earlier approaches in the reported comparison.
  • Probabilistic models achieved better LPIPS performance with clearer, more realistic but less ground-truth-consistent predictions.
  • Context-aware modeling aggregates per-pixel contextual information across layers and directions to address context loss during long-term prediction.
  • Raw-pixel representation learning is difficult because high dimensionality and per-pixel variability cause prediction error to grow exponentially over long horizons.

5.2 Using Explicit Transformations

Explicit-transformation methods predict how the latest observed frame should change rather than synthesizing all visual content directly. They use vector-based, kernel-based, hybrid, spatial-transformer, and object-centric mechanisms, but preserving global spatial information efficiently remains difficult.

  • Transformation-based approaches assume visual information is already present and learn transformations from the latest observed frame to the next frame.
  • Vector-based Resampling: Vector-based resampling predicts motion vectors that shift source pixels through bilinear interpolation.
  • Kernel-based Resampling: Kernel-based resampling predicts spatial kernels applied convolutionally to local patches, while hybrid SDC modules combine kernels with displaced per-pixel motion vectors.
  • Vector-based Resampling: Spatial transformer modules regress affine parameters, generate sampling grids, and produce outputs by differentiably sampling feature maps or images.
  • Vector-based Resampling: Per-pixel transformations extend spatial transformers, supporting flow-based prediction architectures and multi-step extrapolation.
  • Convolutional transformation methods struggle with complex spatially varying motion and global context, while larger kernels increase memory consumption and pooling reduces spatial resolution.
  • Object-centric transformation prediction can lose performance with multiple objects and occlusions when attention fails to distinguish them correctly.

5.3 Explicit Motion from Content Separation

Motion-content separation factorizes video prediction into temporal dynamics and spatial appearance, reducing the dimensionality of the prediction problem. Subsequent models further disentangle objects or low-dimensional temporal components, although end-to-end training can become difficult.

  • Factorizing video into content and motion predicts lower-dimensional temporal dynamics separately from spatial layout.
  • Motion-content factorization makes end-to-end training difficult despite separating temporal dynamics from spatial layout.
  • MCnet was the first end-to-end model to disentangle scene dynamics from visual appearance and reported stable long-term predictions compared with models lacking explicit factorization.
  • Two-staged models separately process flow and pixels or predict static backgrounds before moving objects, improving disoccluded-region sharpness and reducing artifacts in the reported approach.
  • DDPAE decomposes high-dimensional video into components represented by low-dimensional temporal dynamics, first separating individual digits on Moving MNIST.

5.4 Conditioned on Extra Variables

Conditioning video prediction on variables such as vehicle odometry or robot state narrows the prediction space by providing information about scene dynamics. Prior work used control inputs or robot-state conditioning, but long-term quality can still degrade because of uncertainty and blur.

  • Extra variables such as vehicle odometry or robot state narrow the prediction space and provide information about scene dynamics.The passage links camera motion in autonomous driving to wheel steering and acceleration.
  • Oh et al. produced over 100 long-term predictions conditioned on Atari control inputs, but performance dropped for small objects and stochastic scenes.The squared-error objective contributed to difficulty handling stochasticity.
  • Finn et al. predicted natural images conditioned on robot state and robot-object interactions in a controlled scenario.Their model predicted per-pixel transformations and combined them with a composition mask.
  • Finn et al. outperformed and on conditioned and unconditioned predictions, but long-term quality degraded because MSE-induced blurriness accumulated over time.The comparison covered both conditioned and unconditioned settings.
  • A sensorimotor reinforcement-learning model used raw visual inputs alongside measurements or player statistics, and multivariate data benefited decision-making over scalar rewards.Its outputs were future measurements rather than visual predictions.

5.5 In the High-level Feature Space

High-level representations such as semantic and instance segmentation, human pose, and keypoints reduce the prediction space and expose structured scene information. These approaches improve tractability or task-specific prediction, while introducing challenges involving temporal consistency, instance variability, supervision, and long-term performance.

  • Semantic Segmentation: Semantic and instance representations narrow prediction to categorical, high-level scene properties, making the prediction space more tractable than raw pixels.They represent entities such as pedestrians, vehicles, and obstacles.
  • Semantic Segmentation: PEARL used temporal features from single-frame prediction to improve future semantic segmentations, generating temporally consistent outputs.It was the first GAN-based frame-prediction model described as improving per-pixel segmentation.
  • Semantic Segmentation: LSTM-based feature-flow aggregation with an end-to-end warp layer improved short-term predictions over previous works but performed worse on mid-term predictions.The approach decomposed prediction into current-frame segmentation and future optical-flow prediction.
  • Semantic Segmentation: Chiu et al. jointly predicted semantic segmentation and future frames end-to-end from raw pixels, avoiding dependence on external signals such as flow, segmentation, or percepts.The architecture used separate student and teacher networks, with teacher-generated loss guiding forecasting.
  • Instance Segmentation: Instance-level prediction is challenging because instance labels vary in number and identity across frames, unlike fixed-size semantic representations.Fixed-sized feature pyramids used by Mask R-CNN enabled instance-segmentation prediction; CP-ConvLSTM further enriched feature-layer context.
  • Other High-level Spaces: Human pose offers a low-dimensional, interpretable supervisory signal, but pose-guided methods are limited to videos containing humans.Pose-conditioned approaches regress future frames from intermediate pose predictions.
  • Other High-level Spaces: Keypoint coordinates provide a structured, tractable representation that supports stable learning and object-level information for trajectory, action, and reward prediction.A VRNN predicted dynamics through a keypoint bottleneck; qualitative performance exceeded SVG, SAVP, and EPVA, while quantitative results were similar.

5.6 Incorporating Uncertainty

Natural videos contain multiple plausible futures, motivating models that represent uncertainty rather than assuming a single deterministic outcome. The reviewed approaches use latent variables, adversarial objectives, Bayesian formulations, and distribution-free embeddings, with evaluations extending to complex urban driving.

  • Natural video prediction is multimodal because different future outcomes can be equally probable for the same input sequence.Regression tends toward the mean, while classification discretizes a continuous high-dimensional space.
  • Table 2 summarizes model architectures, supervision signals, datasets, prediction horizons, and related task attributes using abbreviations such as variational, perceptual-loss, adversarial-loss, and multi-step prediction.The table also records predicted-frame ranges and whether models address out-of-distribution settings.
  • Some probabilistic models introduce latent variables to represent uncertainty and cover possible outcomes, but their predictions can lack realism.This creates a trade-off with GANs, which produce more realistic predictions but struggle with uncertainty.
  • SAVP combined latent-variable models with GANs to improve variability while maintaining realism in video predictions.It was presented as a compromise between variational approaches and GAN-based approaches.
  • Hu et al. jointly predicted ego-motion, static scene, and object dynamics in complex real-world urban driving, forecasting segmentation, depth, and optical flow two seconds ahead.The model outperformed existing spatio-temporal architectures, but was not compared with SVG or SAVP.

6 PERFORMANCE EVALUATION

The review evaluates video prediction across image-similarity, perceptual, distributional, and task-specific metrics, emphasizing that no protocol captures realism, plausibility, and outcome coverage simultaneously. Results vary by dataset, prediction space, horizon, and metric, with several models achieving state-of-the-art performance in specific settings.

  • 6.1 Metrics and Evaluation Protocols: No evaluation protocol simultaneously measures prediction realism, plausibility, and coverage of possible outcomes.Image similarity metrics remain widely used but do not fully assess video dynamics or temporal variability.
  • 6.1 Metrics and Evaluation Protocols: Pixel-wise metrics such as MSE, SSIM, and PSNR can favor blurry predictions over sharper, plausible alternatives that differ from the ground truth.Their scores are also influenced by the loss function used during training.
  • 6.1 Metrics and Evaluation Protocols: Probabilistic evaluation samples multiple predictions and reports the best ground-truth match, while LPIPS and FVD assess perceptual similarity and sample realism.These protocols address multimodality and distributional realism more directly than single-prediction pixel comparisons.
  • 6.2 Results: In Moving MNIST, CrevNet surpassed prior state of the art, while E3d-LSTM produced stable short- and long-term errors and strong KTH results.Moving MNIST remains restricted and quasi-deterministic, yet long-term prediction is challenging because digits can vanish against the homogeneous background.
  • 6.2 Results: RLN slightly outperformed MCnet and fRNN on short-term KTH prediction, whereas fRNN surpassed RLN on Moving MNIST and UCF101.Kwon et al., CrevNet, and Jin et al. reported state-of-the-art results on Caltech Pedestrian using different metrics; Jin et al. also led BAIR Push.
  • 6.2.1 Results on Probabilistic Approaches: A three-level latent hierarchy outperformed previous probabilistic methods across Stochastic Moving MNIST, BAIR Push, and Cityscapes.On BAIR Push, SAVP generated sharper and more realistic-looking predictions than SVG, although both were surpassed on Cityscapes.
  • 6.2.2 Results on the High-level Prediction Space: Bayes-WD-SL, Terwilliger et al., and PEARL reported the best semantic-segmentation prediction results, with Bayes-WD-SL uniquely modeling diversity through a Bayesian formulation.Evaluations covered short-, mid-, and long-term horizons on Cityscapes using IoU.
  • 6.2.2 Results on the High-level Prediction Space: Sun et al. outperformed the pioneering F2F method on short- and mid-term instance-segmentation prediction using AP50 and AP.Tang et al. also outperformed several baselines for human-pose prediction on Penn Action and J-HMDB using PSNR and SSIM.

7 DISCUSSION

The discussion identifies persistent difficulties in representing uncertainty, scaling prediction to complex videos, and evaluating plausible futures. It concludes that recurrent and stochastic components often help, but long-term, high-resolution, and low-blur prediction remain unresolved.

  • 7 DISCUSSION: Raw-pixel prediction is complicated by dimensionality and variability, often producing regression-to-the-mean blurriness, while larger models face memory limits at high resolution.Flow-based and high-level approaches introduce dependencies on estimated motion or external supervisory signals.
  • 7 DISCUSSION: Minimal inductive bias combined with increased computational scale maximized SVG performance, leaving the appropriate prediction space unresolved.Reducing prediction-space complexity does not remove multimodality in natural videos, and overly simplified representations may limit complex predictions.
  • 7 DISCUSSION: Recurrent connections and stochasticity generally improve performance, while greater capacity with low inductive bias also helps; autoregressive multi-step prediction accumulates errors.These conclusions summarize the review’s analysis of model design choices and long-term behavior.
  • 7.1 Research Challenges: Open challenges include learning spatio-temporal representations, modeling multimodal uncertainty, overcoming dimensionality, and avoiding blurry outcomes from pixel-wise losses.The review frames these issues as defining current research challenges in video prediction.
  • 7.1 Research Challenges: Current methods remain limited in long-term horizons, resolution, and sharpness because autoregressive errors accumulate, memory constrains resolution, and pixel losses average plausible outcomes.The review identifies alternative loss functions, architectures, synthetic videos, and fairer metrics as promising directions.
  • 7.2 Future Research Directions: High-level prediction can simplify the task but remains conditioned on the accuracy of external supervision such as optical flow, poses, or supervised-network features.This motivates using synthetically generated videos as a possible research direction.

8 CONCLUSION

The review synthesizes deep learning video-prediction research by examining its fundamentals, architectures, datasets, methods, metrics, and performance. It concludes that video prediction is promising for self-supervised learning of rich spatio-temporal correlations, while identifying continued room for improvement and future research.

  • The review analyzes more than 50 video-prediction methods and their datasets from descriptions, contributions, and performance results.
  • The methods are classified using a taxonomy based on their main contribution and summarized alongside datasets in tabular form.
  • The review examines video-prediction fundamentals including temporal information, stochasticity, and loss functions.
  • Video prediction is presented as a promising avenue for self-supervised learning of rich spatio-temporal correlations in intelligent decision-making systems.
  • Despite substantial progress, deep-learning-based video prediction still has room for improvement.
Loading 2004.05214v2…