Source-linked AI summary
Video Frame Synthesis using Deep Voxel Flow
Ziwei Liu, Raymond A. Yeh, Xiaoou Tang, Yiming Liu, Aseem Agarwala
TL;DR
Video frame synthesis is difficult because complex motion and appearance challenge optical-flow methods, while direct neural hallucination often blurs outputs. Deep Voxel Flow trains an unsupervised network to copy pixels from existing frames, producing quantitatively and qualitatively improved interpolation and extrapolation results.
Problem
Complex video motion and appearance challenge optical-flow estimation, while direct RGB-synthesis methods often produce blurry frames.
Method
Deep Voxel Flow is an end-to-end differentiable network trained from frame triplets to reconstruct dropped frames by borrowing voxels from nearby video frames.
Results
Deep Voxel Flow improves upon optical-flow and CNN-based methods for video interpolation and extrapolation, including 32.7dB for extrapolation.
Takeaways & Limitations
The method provides sharper, more realistic frame synthesis without optical-flow supervision and generalizes across video types and resolutions.
Takeaways & Limitations
DVF most typically fails in scenes with repetitive patterns, where RGB differences ambiguously identify the source voxel.
Abstract
from arXiv · showhide
We address the problem of synthesizing new video frames in an existing video, either in-between existing frames (interpolation), or subsequent to them (extrapolation). This problem is challenging because video appearance and motion can be highly complex. Traditional optical-flow-based solutions often fail where flow estimation is challenging, while newer neural-network-based methods that hallucinate pixel values directly often produce blurry results. We combine the advantages of these two methods by training a deep network that learns to synthesize video frames by flowing pixel values from existing ones, which we call deep voxel flow. Our method requires no human supervision, and any video can be used as training data by dropping, and then learning to predict, existing frames. The technique is efficient, and can be applied at any video resolution. We demonstrate that our method produces results that both quantitatively and qualitatively improve upon the state-of-the-art.
1. Introduction
The paper targets interpolation and extrapolation in videos whose motion, appearance, occlusion, lighting, and camera changes make simple models and optical-flow estimation unreliable. It combines learned deep networks with pixel flow from existing frames, trained without human supervision, and reports improvements over optical-flow and CNN-based methods.
- Natural videos contain deformation, fast motion, occlusion, dis-occlusion, lighting changes, and camera movement that complicate frame synthesis.
- Optical-flow methods work when flow is accurate but create artifacts otherwise, while direct RGB hallucination often produces blurry frames.
- The proposed approach combines an end-to-end deep network with pixel copying from nearby existing frames, rather than hallucinating every pixel.
- The network is trained unsupervised by dropping frames and comparing generated pixels with the dropped ground-truth frames.
- A voxel flow layer predicts per-pixel 3D motion across the input video, and trilinear interpolation generates the output pixel.
- Training uses UCF-101, while testing spans varied videos; the fully convolutional method supports any resolution and improves over optical-flow and CNN-based methods.
2. Related Work
Related work includes optical-flow, phase-based, generative CNN, differentiable-motion, and recurrent approaches to interpolation, extrapolation, and view synthesis. The paper positions DVF as combining motion modeling and pixel copying to address flow failures and blurry hallucinated predictions.
- Optical flow is widely used for interpolation, but its quality depends on accurate flow and is challenged by large and fast motions.
- Prior interpolation methods copy pixel gradients along source-image paths or use phase-based representations, with the latter limited to smaller motions.
- CNNs have been used to predict optical flow, but supervised methods require optical-flow ground truth and related unsupervised work targets flow rather than interpolation.
- Generative CNNs for images and videos often produce blur because their losses inadequately represent multimodal natural-image distributions.
- Differentiable motion layers and LSTMs support view synthesis or video extrapolation, but reported results remain blurry or constrained to limited scenarios.
3. Our Approach
Deep Voxel Flow synthesizes interpolation and extrapolation frames by predicting a 3D voxel flow and sampling pixels from input video. Its fully convolutional, multi-scale design supports arbitrary resolutions and improves motion handling while remaining end-to-end differentiable.
- 3. Our Approach: DVF reconstructs a target frame from nearby input frames using a self-supervised, end-to-end network that borrows voxels rather than hallucinating pixels.Training uses consecutive frame triplets: two inputs and one reconstruction target.
- 3. Our Approach: The encoder-decoder predicts a per-pixel 3D voxel flow whose spatial components locate corresponding pixels and whose temporal component blends input frames.For interpolation, the corresponding locations are L0 = (x − ∆x, y − ∆y) and L1 = (x + ∆x, y + ∆y).
- 3. Our Approach: DVF uses a fully convolutional encoder-decoder with three convolution layers, three deconvolution layers, and one bottleneck layer, enabling arbitrary-sized video inputs.Skip connections are used to better maintain spatial information.
- 3. Our Approach: Trilinear volume sampling interpolates colors at the virtual voxel’s eight input-volume vertices to generate each synthesized target voxel.The temporal coordinate ranges from t = 0 for the first input frame to t = 1 for the second.
- 3. Our Approach: The training objective combines an L1 reconstruction loss with total-variation regularization on motion and temporal-selection components to reduce visual artifacts.The spatial coherence weight is set to λ1 = 0.01 and the temporal coherence weight to λ2 = 0.005.
- 3. Our Approach: Multi-scale DVF addresses large and small motions by processing frames from coarse to fine scales and fusing flow fields before final refinement.The method is motivated by the difficulty of discovering motions outside the volume-sampling kernel’s support region.
4. Experiments
Experiments evaluate DVF for frame interpolation and extrapolation, multi-step prediction, view synthesis, representation learning, and high-definition slow motion. Across these settings, the method reports stronger quantitative or qualitative results than relevant baselines, while repetitive patterns remain a failure case.
- Frame interpolation: DVF improves video interpolation over hallucination-based and optical-flow baselines, with a further 1.6dB gain over EpicFlow.Evaluation uses PSNR and SSIM on UCF-101 and THUMOS-15; higher values are better.
- Frame extrapolation: 32.7dB is DVF’s best reported performance for video extrapolation, combining flow-based synthesis with deep semantic inference.The gap between Beyond MSE and EpicFlow shrinks to 0.7dB for extrapolation.
- Multi-step prediction: DVF consistently outperforms alternatives across multi-step interpolation and extrapolation, with larger advantages for long-range extrapolation.The comparison covers steps 1, 2, and 3, and the authors attribute long-term prediction ability to large-scale unsupervised training.
- Ablations and failure cases: Removing skip connections reduces PSNR by 1.1dB, and repetitive patterns are the most typical failure case because the source voxel becomes ambiguous.The paper suggests stronger regularization as a possible remedy for repetitive-pattern failures.
- View synthesis: DVF generalizes from UCF-101 to KITTI view synthesis without retraining, outperforming Appearance Flow and other methods despite large camera motion.Without fine-tuning, the reported KITTI L1 error is 0.336 versus 0.471 for Appearance Flow and 0.492 for Recons. Views; lower is better.
- Applications: DVF representations support unsupervised flow estimation and action-recognition pretraining, while user studies find its slow-motion outputs preferred to EpicFlow.The supplied experiments report comparable performance to some supervised flow models after fine-tuning and a significant user preference over EpicFlow.
5. Discussion
Deep Voxel Flow combines pixel copying with end-to-end learning for video frame synthesis, while remaining self-supervised and enabling future extensions.
- Deep Voxel Flow is an end-to-end deep network for synthesizing video frames by copying pixels from existing frames.
- The method can be trained without supervision using any video as training data.
- The paper reports improvements over optical-flow and recent CNN techniques for video interpolation and extrapolation.