Flow-Grounded Spatial-Temporal Video Prediction from Still Images

1 Introduction

• The task is to generate pixel-level future frames over multiple time steps from one still image, unlike methods conditioned on multiple historical frames.With no motion information supplied, the task has spatial and temporal uncertainty.Source: §1 · pp. 1–2
• Generative Flow2rgb synthesis addresses occlusions that make flow-based pixel copying ineffective and keeps sequences closer to the real-video manifold.The two-phase design avoids directly predicting in the high-dimensional frame-pixel space.Source: §1 · p. 2
• The method predicts consecutive future flows with a spatial-temporal 3D-cVAE and generates future-frame appearance from those flows.Different latent samples produce diverse flow predictions.Source: §1 · pp. 2–3
• The paper demonstrates its method on sequences containing both articulated objects, such as humans, and dynamic textures, such as clouds.The introduction motivates still-image prediction with surveillance and cinemagraph applications.Source: §1 · pp. 1, 3

2 Related Work

• Action-prediction approaches can identify events, trajectories, or activities, but typically provide only coarse forecasts rather than richer future information.Some early approaches require labels such as moving-object bounding boxes; later unsupervised methods rely on scene context.Source: §2 · p. 3
• Pixel-level methods generate multi-step frames using varied network structures and learning techniques, including adversarial loss, motion/content separation, and transformation parameters.The proposed architecture combines a 3D-cVAE flow predictor with Flow2rgb frame generation.Source: §2 · p. 3, Figure 2 · p. 4
• Closest still-image approaches predict poses deterministically or use pose generation requiring multiple observed frames, and focus on human dynamics; this work also targets dynamic textures.The cited approaches use recursive modules, whereas 3D convolutions encode spatial and temporal information together.Source: §2 · p. 3
• Prior uncertainty methods model only one-step prediction or dense trajectories, while this approach integrates uncertainty modeling with multi-frame prediction.Iterating a one-step model can degrade frame quality through error accumulation; trajectory supervision requires laborious human labeling.Source: §2 · pp. 3–4
• Earlier dynamic-texture methods learn a model per sequence, whereas this work collects many videos to model a general motion distribution.The resulting model can serve as an editing tool for animating static texture examples.Source: §2 · p. 4

3 Proposed Algorithm

• The full pipeline predicts a sequence of flows from the first frame, then iteratively synthesizes future frames from that frame and the corresponding flows.The flow stage is triggered by noise, and each synthesis step uses the previous frame and next flow.Source: §3 · p. 4

3.1 Flow prediction

• The 3D-cVAE predicts multiple consecutive flows probabilistically, using 3D convolutions to model spatial-temporal relationships and support continuous motion over time.Different latent samples yield diverse predictions; the model conditions the flow encoder and decoder on the starting frame.Source: §3.1 · p. 5

3.2 Frame generation

• Flow2rgb generates the next frame from the current frame and flow, using a warped estimate with frame and flow streams and pixel and feature losses.The one-step model is applied iteratively with each predicted flow and the previous generated frame.Source: §3.2 · pp. 6–7
• On chair rotation, Flow2rgb follows the ground-truth feature-manifold shape, while iterative warping progressively diverges.The comparison uses t-SNE projections of frame features; the plotted example shows the predicted sequence tracking the ground-truth trend more closely.Source: Figure 5 · p. 7, §3.2 · p. 8

Experimental settings

• Evaluation uses KTH, WavingFlag, and FloatingCloud videos, spanning articulated human motion and dynamic textures.The implementation predicts 16 future steps from a starting frame; frames are resized to 128×128, and flow prediction and Flow2rgb are trained separately.Source: §4 · p. 8

Qualitative comparisons

• Visual examples show more continuous, reasonable motion than, including flag and cloud dynamics and a walking direction missed by.The authors note that predictions lose motion and quality over successive frames, while results for dynamic textures are nearly static.Source: Figure 6 · p. 9, §4 · p. 10

Quantitative comparisons

• Our method achieves the lowest RMSE on predicted optical flows and outperforms in perceptual similarity and user votes.The quantitative comparison covers flow RMSE, perceptual dissimilarity, and a user study; the figure identifies the method as best under metrics (b)–(d).Source: Figure 7 · p. 11, §4 · pp. 11–12

Random motion

• Learned motion produces better results than random flow, whose predictions neither model the motion distribution nor account for temporal relationships.In pool5 feature space, the ground-truth sequence follows a “C” manifold that the learned prediction follows more closely than the random prediction.Source: §4 · p. 12, Figure 8 · p. 12

Diversity

• Across 10 noise samples, predictions are more diverse while remaining closer to the ground truth in perceptual comparisons than.The reported feature embeddings show the method’s predictions scattered apart from one another yet closer to the ground-truth sequence.Source: §4 · p. 13, Figure 9 · p. 13

Bringing still images to life

• The method is presented as a way to turn still images into vivid sequences for aesthetic video-editing effects.A cloud example masks the sky and applies a model trained on FloatingCloud; using reference flows makes Flow2rgb a global motion style-transfer model, while random flow sampling is currently uncontrollable.Source: §4 · pp. 13–14, Figure 11 · p. 14

• The method generates multiple likely future frames from one still image by predicting flows and then generating frames grounded in those flows.Source: §5 · p. 14
• A 3D-cVAE models future uncertainty and spatial-temporal relationships during flow prediction.Source: §5 · p. 14
• The authors demonstrate the algorithm's effectiveness on human action videos and dynamic texture videos.Source: §5 · p. 14

• The flow-prediction model consists of a 3D variational autoencoder and an image encoder.Table 1 gives the variational autoencoder architecture; Table 2 gives the image encoder architecture.Source: Network Architecture · p. 17, Table 1 · p. 17, Table 2 · p. 18
• Flow2rgb's frame and flow encoders follow VGG-19 through the Relu 4 1 layer, and its decoder is symmetrical to the encoder.Nearest-neighbor upsampling enlarges the feature maps.Source: Network Architecture · p. 17