PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Long-term Video Prediction without Supervision

Nevan Wichers, Ruben Villegas, Dumitru Erhan, Honglak Lee

arXiv:1806.04768v1cs.CV

TL;DR

Long-term video prediction has had limited success, and a prior hierarchical method requires ground-truth high-level annotations. This work jointly trains an encoder, predictor, and decoder without that supervision, and reports predictions about 20 seconds ahead plus more realistic outputs with feature-space adversarial loss.

  • Problem

    Video prediction methods have had limited success on long-term horizons, while the cited hierarchical approach requires ground-truth high-level annotations such as human landmarks.

    Source: §1 · p. 1

  • Method

    The method jointly trains an encoder, predictor, and decoder without high-level supervision, and applies adversarial loss in feature space to train the predictor.

    Source: §1 · p. 1

  • Results

    About 20 seconds into the future, the method predicts pixel-level video on Human 3.6M, and EPVA ADVERSARIAL generates more realistic predictions than the presented baselines, including Finn et al. (2016) and Denton and Fergus (2018).

    Source: §1 · p. 1, §6 · p. 8

  • Takeaways & Limitations

    The authors conclude that applying adversarial loss in higher-level feature space is beneficial for generating more realistic predictions.

    Source: §6 · p. 8

    EPVA alone uses an L2 loss on encoder outputs, which can make the predictor's encoding-space predictions blurry.

    Source: §4.2.3 · p. 4

Abstract

from arXiv · show

Much of recent research has been devoted to video prediction and generation, yet most of the previous works have demonstrated only limited success in generating videos on short-term horizons. The hierarchical video prediction method by Villegas et al. (2017) is an example of a state-of-the-art method for long-term video prediction, but their method is limited because it requires ground truth annotation of high-level structures (e.g., human joint landmarks) at training time. Our network encodes the input frame, predicts a high-level encoding into the future, and then a decoder with access to the first frame produces the predicted image from the predicted encoding. The decoder also produces a mask that outlines the predicted foreground object (e.g., person) as a by-product. Unlike Villegas et al. (2017), we develop a novel training method that jointly trains the encoder, the predictor, and the decoder together without highlevel supervision; we further improve upon this by using an adversarial loss in the feature space to train the predictor. Our method can predict about 20 seconds into the future and provides better results compared to Denton and Fergus (2018) and Finn et al. (2016) on the Human 3.6M dataset.

1. Introduction

The work targets long-term video prediction without requiring ground-truth high-level annotations. It combines unsupervised feature discovery and joint encoder–predictor–decoder training, with predictions extending about 20 seconds on Human 3.6M.

1. Introduction

  • Hierarchical prediction can degrade less quickly than pixel-space prediction, but the cited Villegas et al. (2017b) method requires ground-truth landmarks during training.The proposed work explores hierarchical prediction without ground-truth landmarks or other high-level structure annotations.Source: §1 · p. 1
  • The work proposes unsupervised discovery of high-level features needed for long-term prediction.These features are learned without ground-truth high-level structure annotations.Source: §1 · p. 1
  • The proposed training strategy jointly learns high-level features from low-level features and low-level features from high-level features.This joint strategy is presented as an alternative to training with ground-truth high-level annotations.Source: §1 · p. 1
  • The method applies adversarial training in feature space to improve high-level feature discovery and generation.The paper presents this as an additional improvement to its unsupervised approach.Source: §1 · p. 1
  • About 20 seconds into the future, the method predicts pixel-level video on the Human 3.6M dataset.The paper reports this as long-term prediction without ground-truth high-level structure annotations.Source: §1 · p. 1

2. Related Work

Prior video-prediction work ranges from patch-level synthetic-motion methods to full-frame and hierarchical approaches. Long-term prediction on realistic videos remains challenging, and the cited hierarchical method requires landmark supervision.

Patch-level prediction

  • Patch-level prediction progressed from synthetic motions to natural videos, but patch prediction encounters the aperture problem and becomes blocky over time.The cited work includes Sutskever et al. (2009), Michalski et al. (2014), Mittelman et al. (2014), Srivastava et al. (2015), and Ranzato et al. (2014).Source: §2 · p. 2

Frame-level prediction on realistic videos

Long-term prediction

  • Long-term prediction on high-resolution natural videos beyond approximately 20 frames is described as very challenging.The passage notes that Oh et al. (2015) demonstrated long-term prediction on video games, while Villegas et al. (2017b) used a hierarchical approach requiring ground-truth landmarks.Source: §2 · p. 2
  • The cited hierarchical method by Villegas et al. (2017b) requires landmark supervision, while Oh et al. (2015) had not applied its video-game method to real-world video prediction.The paper proposes techniques to address these limitations.Source: §2 · p. 2

3. Background

The background describes hierarchical prediction as forecasting high-level structure and decoding it into pixels. Its landmark-based example uses an LSTM for future poses and a visual analogy network to generate frames from the first context frame.

3. Background

  • Villegas et al. (2017b) use high-level video dynamics to relieve blurring and enable prediction many frames into the future.Their hierarchical prediction model is described as predicting in high-level feature space and decoding into pixels.Source: §3 · p. 2
  • The example estimates pose landmarks from context frames, then uses an LSTM to predict future landmark states.After C timesteps, predicted landmarks generate video frames; unlike that method, the paper's formulation feeds both the prior prediction and hidden state into the LSTM, with the prediction not being a landmark vector.Source: §3 · p. 2
  • The visual analogy network generates each frame by applying the difference between predicted and context landmark heatmaps to the first context image.The passage expresses the image as fdec(fpose(g(ˆpt)) − fpose(g(pC)) + fimg(IC)).Source: §3 · p. 2
  • Unlike Villegas et al. (2017b), the proposed method does not require landmarks or the fixed function g; it automatically discovers features for VAN input.These features locate the moving object and help the network focus on generating its pixels in future frames.Source: §3 · p. 2

4. Method

The method predicts future feature encodings with an LSTM and uses a visual analogy decoder conditioned on the first frame to generate images. It explores training the encoder, predictor, and decoder without ground-truth high-level annotations.

4.1. Network Architecture

  • The LSTM predicts future feature vectors, which the analogy decoder combines with information from the first input frame to produce predicted images.The decoder includes a gating mechanism for identifying moving objects and selecting pixels to copy or generate.Source: §4.1 · pp. 2–3

4.2. Training Objective

  • The paper explores alternative training strategies for the predictor and VAN without ground-truth annotations of high-level structures.Source: §4.2 · p. 3

4.2.1. END-TO-END PREDICTION

  • The training objective compares end-to-end predicted images with ground truth using L2 loss, a direct image-space objective that tends to produce blurry predictions.The E2E setup feeds encoded context frames to the predictor and decodes its predicted encodings into pixel-level predictions.Source: §4.2.1 · p. 3, Figure 1 · p. 3

4.2.2. ENCODER PREDICTOR WITH ANALOGY MAKING

  • EPVA jointly trains the encoder and predictor to make encodings predictable, while training the encoder and VAN to make encodings useful for image analogy.The summed losses train the VAN on ground-truth encoder outputs and the predictor to approximate those features; α is gradually increased from around 1e-7 to around 0.1.Source: §4.2.2 · pp. 3–4, Figure 2 · p. 4

4.2.3. EPVA WITH ADVERSARIAL LOSS IN PREDICTOR

  • To address blurry predictions in encoding space, the method adds an adversarial loss with an LSTM discriminator that distinguishes predicted from encoder-produced encoding sequences.It retains a weighted L2 loss, adds Gaussian noise to the predictor, and also passes VAN-encoder outputs to the discriminator.Source: §4.2.3 · p. 4

5. Experiments

Experiments test long-horizon prediction on bouncing shapes and Human3.6M, with quantitative, human, and pose-regression evaluations. The results favor EPVA variants over baselines in long-term prediction, while ablations identify limitations in decoder access and combined losses.

5.1. Long-term Prediction on a Toy Dataset

  • 97% of EPVA runs versus about 25% of CDNA runs produced a shape with the correct color in the toy dataset’s frames 1012–1022.The evaluation averaged results over 1,000 runs; EPVA sometimes repeated the shape’s location, while accurate frame-1000 location was considered unrealistic because prediction errors accumulate.Source: §5.1 · p. 5

5.2.2. HUMAN EVALUATION

5.2. Long-term Prediction on Human3.6M

  • Across 126-frame Human3.6M predictions, EPVA ADVERSARIAL produced sharper images, and the person disappeared less often and often reappeared later than with EPVA.The comparison used five context frames; the method and baselines were trained with the same number of context and prediction frames, while SVG-LP’s best-of-five protocol favored that baseline.Source: §5.2 · pp. 5–6, Figure 5 · p. 7

5.2.1. PERSON DETECTOR EVALUATION

  • For longer-term frames, the person-detector evaluation found EPVA ADVERSARIAL significantly better than the baselines, with its person score relatively constant across frames.The score is detector confidence that a predicted frame contains a person; ground-truth detector accuracy was approximately 0.4.Source: §5.2.1 · p. 6, Figure 4 · p. 6

5.2.3. POSE REGRESSION FROM LEARNED FEATURES

  • Learned encoder features achieved pose-regression error 0.0687 versus 0.0758 for VGG baseline features, an approximately 9% relative improvement.The features were input to a two-layer MLP trained to predict human pose landmarks.Source: §5.2.3 · pp. 6, 8

5.3. Ablation Studies

  • Without the VAN, the decoder often failed beyond frame 32 to mask out the person in the first frame, as it had to represent foreground and background despite not always seeing that frame.The decoder mask selects between its own output and pixels from the first frame.Source: §5.3 · p. 8
  • Combining E2E and EPVA losses produced blurrier videos than EPVA, while using 10 rather than five context frames did not improve long-term prediction.The compared E2E and EPVA variants are identified in Figure 6.Source: §5.3 · p. 8, Figure 6 · p. 8

6. Conclusion

The paper presents hierarchical long-term video prediction without ground-truth high-level annotations. EPVA ADVERSARIAL produces more realistic predictions than the presented baselines, while EPVA has occasional disappearances but generates sharper images for longer than Finn et al. (2016) and E2E.

6. Conclusion

  • EPVA ADVERSARIAL generates more realistic predictions than all presented baselines, including Finn et al. (2016) and Denton and Fergus (2018).The result suggests that applying adversarial loss in higher-level feature space is beneficial.Source: §6 · p. 8
  • EPVA generates sharper images for a longer period than Finn et al. (2016) and E2E, but its predictions occasionally disappear.Source: §6 · p. 8
  • The presented hierarchical approaches do not require ground-truth high-level structure annotations.Source: §6 · p. 8
  • Future work could apply feature-space techniques such as the variational method of Babaeizadeh et al. (2018) to enable multiple future trajectories.Source: §6 · p. 8
Loading 1806.04768v1…