Source-linked AI summary
SDCNet: Video Prediction Using Spatially-Displaced Convolution
Fitsum A. Reda, Guilin Liu, Kevin J. Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, Bryan Catanzaro
TL;DR
High-resolution video prediction must capture motion while preserving sharp appearances and handling disocclusions, where existing resampling and direct-generation approaches are limited. SDCNet predicts per-pixel displacement and kernels to synthesize frames, achieving state-of-the-art results while handling large motion and producing crisp, consistent predictions.
Problem
High-resolution video prediction requires sharp future frames that capture object motion and changing visibility, but flow resampling cannot handle disocclusions and direct generation can be blurry.
Method
SDCNet predicts a motion vector and kernel for each pixel, applying the kernel at the motion-displaced location in a source image while conditioning on past frames and optical flows.
Results
SDCNet achieves state-of-the-art video frame prediction, handling large motion and producing crisp future frames with motion closely matching ground-truth sequences.
Takeaways & Limitations
Combining displacement learning with small adaptive kernels preserves clean object boundaries while reducing the memory demand of large-kernel synthesis.
Takeaways & Limitations
Accuracy depends on the estimated input optical flows, leading to failures in fast-motion sequences.
Abstract
from arXiv · showhide
We present an approach for high-resolution video frame prediction by conditioning on both past frames and past optical flows. Previous approaches rely on resampling past frames, guided by a learned future optical flow, or on direct generation of pixels. Resampling based on flow is insufficient because it cannot deal with disocclusions. Generative models currently lead to blurry results. Recent approaches synthesis a pixel by convolving input patches with a predicted kernel. However, their memory requirement increases with kernel size. Here, we spatially-displaced convolution (SDC) module for video frame prediction. We learn a motion vector and a kernel for each pixel and synthesize a pixel by applying the kernel at a displaced location in the source image, defined by the predicted motion vector. Our approach inherits the merits of both vector-based and kernel-based approaches, while ameliorating their respective disadvantages. We train our model on 428K unlabelled 1080p video game frames. Our approach produces state-of-the-art results, achieving an SSIM score of 0.904 on high-definition YouTube-8M videos, 0.918 on Caltech Pedestrian videos. Our model handles large motion effectively and synthesizes crisp frames with consistent motion.
1 Introduction
SDCNet targets sharp, high-resolution future-frame prediction by combining learned motion displacement with adaptive kernels, addressing the large-motion and boundary-quality limitations of prior approaches.
- 1 Introduction: The model is trained on large amounts of unlabeled high-resolution gameplay footage, whose broad motion range improves accuracy and transfers to real-life footage.The training data consists of raw video-game frames rather than labeled optical-flow examples.
- 1 Introduction: Direct pixel-generation approaches often produce blurry frames and poorly model large object motions.The cited prior work attributes these weaknesses to the difficulty of directly generating pixels and modeling motion.
- 1 Introduction: Kernel-based methods preserve object structure but struggle with large motion because increasing kernel size is computationally expensive.Their displacement is limited by kernel size, creating a memory and inference-cost trade-off.
- 1 Introduction: SDCNet combines a learned motion vector with a per-pixel kernel, applying the kernel at a motion-displaced source location for frame synthesis.This design combines vector-based displacement with kernel-based appearance modeling.
- 1 Introduction: The paper proposes a spatially-displaced convolutional module and evaluates it against kernel-based, vector-based, and state-of-the-art alternatives.These comparisons are presented as part of the paper’s stated contributions.
2 Methods
The method predicts future frames by applying a spatially displaced, adaptive convolution to the latest frame, using past frames and optical flows to estimate pixel-wise motion and kernels. SDC combines large-motion displacement with kernel-based synthesis while addressing disocclusions and reducing kernel-size costs.
- 2 Methods: Direct vector resampling can create speckled noise, whereas kernel-based resampling produces pleasing small-motion results but requires costly large kernels for large motion.The supplied comparison motivates decoupling displacement from kernel learning.
- 2.1 Spatially Displaced Convolution: SDC predicts a motion vector and adaptive kernel per pixel, then convolves the kernel with a patch centered at the displaced source location.The displaced patch is bilinearly sampled when coordinates are non-integral.
- 2.1 Spatially Displaced Convolution: The model conditions on past frames and backwards optical flows, and a fully convolutional network outputs pixel-wise separable kernels and motion vectors for SDC.The network uses 3D convolutions and FlowNet2-derived inter-frame optical flow.
- 2.3 Optical Flow: Learned sampling vectors differ from backward optical flow because disoccluded pixels may have undefined or zero flow, requiring adaptive sampling to fill exposed regions.The disocclusion example shows that direct flow resampling duplicates a foreground border, whereas learned vectors can select suitable background pixels.
- 2.4 Loss Functions: The training objective combines RGB L1, perceptual, and style losses, while finetuning was found to remove checkerboard artifacts and sharpen predictions.The authors also found direct motion-vector supervision inferior because optical-flow targets do not properly handle disocclusions and add estimated-flow noise.
3 Experiments
Experiments show that SDC-Net achieves strong next-frame and multi-step prediction across CaltechPedestrian and high-definition YouTube videos, while producing sharper, better-aligned frames than comparison methods. Ablations indicate that spatial displacement, kernel synthesis, and style-loss fine-tuning address distinct failure modes, although inaccurate optical flow and scene transitions remain limitations.
- 3.1 Comparison on low-quality videos: L2 1.62×10^-3 and SSIM 0.918 on CaltechPedestrian outperform DualGAN, MCNet, CopyLast, and other compared models.The comparison uses the CaltechPedestrian test partition; DualGAN reports L2 2.41 × 10^-3 and SSIM 0.899, while MCNet reports 2.50 × 10^-3 and 0.879.
- 3.2 Comparison on high-definition videos: L2 2.4 × 10^-3 and SSIM 0.911 on 26 full-HD YouTube videos outperform MCNet, which reports L2 2.55 × 10^-3 and SSIM 0.895.SDC-Net frames are described as crisp and mostly aligned with ground truth, using displaced patches rather than hallucinating pixels.
- 3.4 Comparison in multi-step prediction: Across 120,725 CaltechPedestrian frames, SDC-Net consistently outperforms MCNet, BeyondMSE, and CopyLast over five predicted steps on L1, L2, PSNR, and SSIM.Qualitative sequences preserve color distribution, object shapes, and fine details.
- 3.5 Ablation results: SDC-Net combines vector-based large-motion handling with kernel-based visual quality, reducing the vector model’s speckled noise and disocclusion artifacts.The vector-based approach retains residual glove pixels and can produce speckled patterns, whereas SDC-Net reduces both effects.
- 3.5 Ablation results: Fine-tuning with perceptual and style losses reduces blurriness and preserves finer details, while L1 loss helps capture otherwise difficult large motions.The improvement is illustrated in rendered barriers, fences, and gloves.
- 3.5 Ablation results: Prediction quality is affected by inaccurate optical flow, and the model fails during scene transitions until five valid conditioning frames are available.Fast-motion cases can also produce larger misalignment, such as on the fence in Fig. 11.
4 Conclusions
The paper presents SDC-Net, combining a 3D CNN with spatially-displaced convolution for high-resolution video prediction. It handles large motion and produces crisp frames, but its accuracy depends on estimated-flow quality and can fail during fast motion.
- SDC-Net combines a 3D CNN with a spatially-displaced convolution module for video frame prediction.The method was trained on 428K high-resolution gameplay frames.
- The model handles large motion and predicts crisp future frames with motion closely matching ground-truth sequences.
- Accuracy depends on the quality of estimated input flows, causing failures in fast-motion sequences.The authors identify multi-scale architectures for fast motion as future work.