PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Long-term Future via Hierarchical Prediction

Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, Honglak Lee

arXiv:1704.05831v5cs.CV

TL;DR

Long-term video prediction suffers when recursively generated pixels are fed back into the predictor, because errors accumulate over time. The paper predicts high-level structure first, then generates future frames from one observed frame and that predicted structure sequence. On Human3.6M and Penn Action, the approach produces better long-term human-action predictions than the baselines, while remaining limited by background motion and other scope constraints.

  • Problem

    Recursive pixel-level video prediction accumulates errors when predicted frames are repeatedly observed for future steps.

  • Method

    The method uses an LSTM to predict future high-level human poses and an analogy-based image generator to produce frames from one observed frame and the predicted poses.

  • Results

    The approach predicts up to 128 future frames and outperforms baselines on Human3.6M and Penn Action human-action video prediction.

  • Takeaways & Limitations

    Incorporating appropriate high-level structures into generation supports long-term pixel-level video prediction without observing generated future frames.

  • Takeaways & Limitations

    The model does not handle background motion and generates a single future trajectory, while automatically learning structures without domain knowledge remains open.

Abstract

from arXiv · show

We propose a hierarchical approach for making long-term predictions of future frames. To avoid inherent compounding errors in recursive pixel-level prediction, we propose to first estimate high-level structure in the input frames, then predict how that structure evolves in the future, and finally by observing a single frame from the past and the predicted high-level structure, we construct the future frames without having to observe any of the pixel-level predictions. Long-term video prediction is difficult to perform by recurrently observing the predicted frames because the small errors in pixel space exponentially amplify as predictions are made deeper into the future. Our approach prevents pixel-level error propagation from happening by removing the need to observe the predicted frames. Our model is built with a combination of LSTM and analogy based encoder-decoder convolutional neural networks, which independently predict the video structure and generate the future frames, respectively. In experiments, our model is evaluated on the Human3.6M and Penn Action datasets on the task of long-term pixel-level video prediction of humans performing actions and demonstrate significantly better results than the state-of-the-art.

1. Introduction

Long-term future-frame prediction is important for intelligent agents, but recursive pixel prediction accumulates errors over time. The paper proposes hierarchical prediction using high-level structure dynamics and analogy-based frame generation.

  • Motivation: Future-frame prediction supports planning and decision-making for robots and autonomous cars in real-world scenarios.The paper targets long-term predictions that provide full descriptions of the visual world.
  • Problem: Recursive pixel-level methods repeatedly observe generated frames, causing pixel noise to amplify until video context is lost.Early predictions may be reasonable, but quality can degrade dramatically at longer horizons.
  • Approach: The method first estimates high-level structures, predicts their future states, and generates future frames conditioned on those predictions.Its structure predictor uses an LSTM, while an analogy-based generator constructs frames from one observed frame and predicted structures.
  • Contribution: The paper’s contribution is a hierarchical video-prediction approach designed to avoid propagating pixel-level prediction errors through time.The authors argue that proper high-level structures can benefit the generative process.

2. Related Work

Prior video-prediction methods generally generate future frames through pixel-to-pixel processes, while the paper introduces a hierarchical alternative that predicts structure before appearance.

  • Earlier approaches: Early methods predicted small patches with simple motions, but high-resolution video contains complex motion that patchwise modeling cannot adequately represent.The aperture problem can produce blockiness as predictions advance in time.
  • Full-frame prediction: Later methods moved to full-frame prediction using architectures such as adversarial, variational, foreground-background, motion-content, and predictive-coding models.These approaches still focus on pixel-level video generation.
  • Hierarchical prediction: The proposed alternative first predicts high-level structure and then uses it to generate future video from a single input frame.This decomposes future prediction into structure prediction followed by structure-conditioned frame generation.
  • Contribution: The authors describe the architecture as the first hierarchical approach to pixel-level video prediction and attribute its long-term capability to visual-structure analogy.The image generator learns a shared embedding between image and structure inputs.

3. Overview

The pipeline predicts future high-level structures before generating pixels, avoiding recurrent observation of generated frames and thereby removing their recursive error dependency.

  • Problem formulation: Given observed structure sequence p1:t and frame x_t, the task is to predict future structures p_t+1:t+T and frames x_t+1:t+T.The overview frames prediction as a hierarchical alternative to recurrent pixel modeling.
  • Pipeline: The pipeline estimates input structures, predicts a future structure sequence, and generates future images through visual-structure analogy.Human pose is used as the high-level structure in the human-action experiments.
  • Pipeline: The method decomposes video prediction into future structure prediction and structure-conditioned frame generation.The generator transforms the last observed frame using pose relationships in feature space.
  • Benefit: Removing recurrent dependence on generated frames prevents the compound errors associated with pixel-level prediction.The approach therefore targets very long-term video prediction without observing generated future frames.

4. Architecture

The architecture predicts future human poses from observed pose dynamics, then generates future frames by transferring pose changes onto a single observed image through visual-structure analogy.

  • Architecture: The network has two modules: an encoder-decoder LSTM for pose coordinates and an image generator for visual analogy from pose heatmaps.The LSTM outputs xy-coordinates, which are converted into heatmaps for the image generator.
  • Future pose prediction: The LSTM encodes observed pose dynamics and predicts future poses after observing several inputs.It uses a memory cell to retain pose-history information and must handle noise from annotations or pose estimation.
  • Future pose prediction: The decoder does not observe previously generated poses, preventing pose-prediction errors from propagating into later predictions.This design requires the encoded representation to contain the information needed to generate the future action sequence.
  • Visual-structure analogy: The image generator synthesizes a future frame by transferring the structural transformation from an input pose to a predicted future pose.It combines pose and image encoders with a convolutional decoder.
  • Visual-structure analogy: The analogy operation adds the difference between future-pose and input-pose features to the observed image features before decoding the future image.The operation is expressed as fpose(g(ˆpt+n)) − fpose(g(pt)) + fimg(xt).

5. Training

Training separately optimizes the structure predictor and visual-structure analogy generator, using future pose supervision and a compound image-generation objective with appearance, structure, and adversarial terms.

  • Inference procedure: At test time, the Hourglass network estimates input poses, the LSTM predicts future poses, and the analogy network generates future frames from one observed frame.Predicted frames are generated one by one without observing any previously predicted frame.
  • Structure prediction training: The pose predictor is trained as a sequence-to-sequence model that observes k pose inputs before predicting future human poses.The LSTM is unrolled for k timesteps before prediction begins.
  • Structure prediction training: The pose loss uses visibility indicators so the LSTM can infer non-visible landmarks from the available landmarks and observed motion.This training strategy encourages the model to represent human structure despite missing landmarks.
  • Image generation training: The image generator is trained with random temporal jumps, mapping an observed frame and pose at time t to frame xt+n conditioned on pose pt+n.Random jumps avoid a trivial task that changes only a few pixels between nearby frames.
  • Image generation training: The visual-structure analogy network combines image-space, feature-space, and adversarial losses to train future-frame generation.Training is performed independently from the high-level structure LSTM and the modules are combined at test time.
  • Compound objective: The image loss encourages rough pixel-level predictions, while feature losses combine appearance and structure signals for accurate appearance and structure.C1 extracts mostly appearance features and C2 extracts mostly structure features.
  • Compound objective: The adversarial generator loss encourages realistic detail, while mismatch-aware discrimination strengthens feedback about the relationship between generated images and conditioning poses.The mismatch term also reduces the generator’s effective parameter search space during optimization.

6. Experiments

Experiments on Penn Action and Human3.6M evaluate perceptual realism, action representation, and pixel-level prediction against convolutional LSTM and optical-flow baselines. The hierarchical method generally performs better, though performance varies with motion complexity and exact-future matching.

  • Experimental setup: The experiments evaluate long-term pixel-level human-action video prediction on Penn Action and Human3.6M datasets.The evaluation uses pose landmarks and frames, with dataset-specific training and testing procedures.
  • Evaluation: The method is compared against convolutional LSTM and optical-flow baselines using perceptual preference and action-recognition evaluations.AMT workers compare realism, while a Two-Stream CNN assesses whether generated videos represent the correct action.
  • Penn Action results: 82.4% for ours vs 17.6% for Convolutional LSTM, and 86.1% for ours vs 13.9% for Optical Flow in Penn Action video preferences.For jump rope, the margin is smaller: 66.7% for ours versus 33.3% for Convolutional LSTM.
  • Penn Action results: The generated Penn Action videos are more representative of the correct action than all baselines in both temporal and spatial recognition streams.Combining the temporal and spatial streams gives the best quality for pixel-level prediction of the correct action.
  • Pixel-level evaluation: PSNR favors predictions generated from the exact ground-truth future, while predicted futures can deviate substantially, especially in Human3.6M.This exposes the dependence of pixel-level evaluation on matching the exact future trajectory.
  • Human3.6M results: 70.3% for ours vs 29.7% for Convolutional LSTM, and 72.3% for ours vs 27.7% for Optical Flow in Human3.6M video preferences.The method performs poorly on minimally moving actions, where static baseline predictions can appear more realistic.

7. Conclusion and Future Work

The conclusion attributes the method’s long-term prediction ability to hierarchical high-level structure, while identifying automatic structure learning, multiple futures, and background motion as open problems.

  • Conclusion: The hierarchical approach predicts up to 128 future frames, an order-of-magnitude improvement in effective temporal prediction scale.The conclusion links this success to incorporating appropriate high-level structure into the generative process.
  • Future work: Automatically learning high-level structures without domain knowledge remains an open research question.The current approach uses human action structure, motivating future work on structure discovery.
  • Future work: The model generates a single future trajectory, whereas better future estimation would require multiple generated futures.Future work proposes probabilistic sequence models for generating many futures.
  • Limitations: The model does not handle background motion, which would require hallucinating unseen background content.The authors leave background-motion prediction for future work.

A. Motion-Based Pixel-Level Evaluation, Analysis, and Control Experiments

The evaluation separates motion-dependent performance from future-structure accuracy by testing background copying and oracle pose inputs. Because future frames are strictly determined by predicted structure, the oracle isolates the effect of structure prediction on PSNR.

  • Motion-Based Evaluation: The motion-decile evaluation adds an Ours BG variant that copies background pixels from the last observed frame using predicted pose heat-maps as foreground/background masks.The section also evaluates an oracle condition, though the supplied passage truncates its description.
  • Control Experiments: Future pose oracle results measure how predicting a different future structure affects PSNR because the generated frames are strictly dictated by future structure.The authors note that multiple future trajectories can be possible from one past trajectory.

A.1. Penn Action

On Penn Action, the hierarchical method outperforms baselines across motion deciles, while oracle poses improve PSNR and expose the effect of pose deviation on pixel metrics. Qualitative results show plausible actions, but PSNR can penalize valid alternative futures.

  • Motion-Based Evaluation: The method outperforms baselines across motion deciles, and oracle future poses produce even higher PSNR scores.The evaluation is shown in Figures 6 and 7.
  • Motion-Based Analysis: Higher mean squared error in predicted pose tends to correspond to lower PSNR scores.This supports the authors’ hypothesis that pose trajectories differing from ground truth reduce pixel-level similarity even when action semantics match.
  • Evaluation Limitation: PSNR may be a poor evaluator when a single predicted trajectory differs from ground truth despite representing one of many plausible futures.The authors leave predicting multiple possible future trajectories for future work.
  • Qualitative Comparisons: Qualitative comparisons cover baseball pitch, golf swing, jumping jacks, and tennis forehand at selected time steps.Figures 9 and 10 provide quantitative and visual comparisons for these Penn Action examples.
  • Oracle Pose Comparison: Both predicted and oracle-pose videos contain the action, while the oracle-based prediction reflects the exact future more accurately.Figures 11 and 12 compare predicted-pose and ground-truth-pose image generation across several actions.

A.2. Human3.6M

On Human3.6M, the hierarchical approach is better than optical flow and comparable to convolutional LSTM in PSNR, while oracle poses yield a larger improvement. Human evaluations favor the method despite similar PSNR to convolutional LSTM.

  • Motion-Based Evaluation: The Human3.6M evaluation reports PSNR over time separately for each motion decile.This comparison is presented in Figure 13.
  • Quantitative Results: The hierarchical approach tends to outperform optical flow and achieve PSNR comparable to convolutional LSTM.The Ours BG variant is included in this comparison.
  • Oracle Control: Oracle future poses provide a larger PSNR boost on Human3.6M than in Penn Action because Human3.6M has greater action uncertainty.The authors connect plausible but different future poses with lower pixel-level scores.
  • Motion-Based Analysis: When predicted poses match ground truth, generated frames have higher PSNR and are perceptually similar; otherwise plausible pose deviations lower PSNR.Figure 14 reports the relationship between predicted-pose MSE and frame PSNR.
  • Qualitative and Human Evaluation: Optical flow and convolutional LSTM often lose foreground structure and become static, whereas the proposed method remains comparable in PSNR but strongly outperforms convolutional LSTM in human evaluation.Figures 15 and 16 show selected walking, phoning, sitting down, and walking-together examples.
  • Oracle Pose Comparison: Predicted-pose and oracle-pose videos both contain the action, but oracle-based videos reflect the exact future more accurately.Figures 17 and 18 provide qualitative comparisons across additional Human3.6M actions.
Loading 1704.05831v5…