Source-linked AI summary
PhaseNet for Video Frame Interpolation
Simone Meyer, Abdelaziz Djelouah, Brian McWilliams, Alexander Sorkine-Hornung, Markus Gross, Christopher Schroers
TL;DR
Video frame interpolation methods often depend on dense correspondences and struggle with lighting changes, motion blur, or larger motion. PhaseNet combines phase-based representation with a learned neural decoder that predicts the intermediate frame's phase decomposition. The paper reports robust results in challenging scenarios, including motion blur and brightness changes, while noting remaining limitations in detail and high-frequency cases.
Problem
Existing correspondence-based and learned interpolation methods struggle with lighting changes or motion blur, while phase-based methods have limited motion range.
Method
PhaseNet predicts the phase and amplitude of the intermediate frame hierarchically from steerable-pyramid decompositions, using phase and image losses.
Results
PhaseNet produces visually preferable results over optical flow in challenging scenarios containing motion blur and brightness changes.
Takeaways & Limitations
The learned phase representation combines the advantages of phase-based and data-driven interpolation for challenging scenes.
Takeaways & Limitations
The method does not reach the detail level of explicit pixel-matching methods and can produce ringing and color artifacts around high-frequency edges.
Abstract
from arXiv · showhide
Most approaches for video frame interpolation require accurate dense correspondences to synthesize an in-between frame. Therefore, they do not perform well in challenging scenarios with e.g. lighting changes or motion blur. Recent deep learning approaches that rely on kernels to represent motion can only alleviate these problems to some extent. In those cases, methods that use a per-pixel phase-based motion representation have been shown to work well. However, they are only applicable for a limited amount of motion. We propose a new approach, PhaseNet, that is designed to robustly handle challenging scenarios while also coping with larger motion. Our approach consists of a neural network decoder that directly estimates the phase decomposition of the intermediate frame. We show that this is superior to the hand-crafted heuristics previously used in phase-based methods and also compares favorably to recent deep learning based approaches for video frame interpolation on challenging datasets.
1. Introduction
Video frame interpolation is difficult when correspondence estimation is ambiguous or when scenes contain lighting changes and motion blur. PhaseNet combines learned phase-based prediction with hierarchical reconstruction to handle larger motion and challenging scenes.
- Traditional interpolation methods estimate correspondences and warp consecutive frames, but ambiguities, occlusions, and lighting changes make this difficult.
- Phase-based methods avoid explicit correspondences but remain limited in the range of motion they can handle.
- Deep learning methods improve over optical flow but still struggle with scenes containing lighting changes and motion blur.
- PhaseNet combines phase-based representation with learning, predicting intermediate-frame phase and amplitude values level by level before reconstruction.
- The method uses a phase loss encoding motion-relevant information and trains hierarchically from coarse to fine scales for efficiency and stability.
- PhaseNet is reported to outperform existing state-of-the-art interpolation methods in challenging scenarios.
2. Related Work
Related work addresses interpolation through optical flow, learned image synthesis, and phase-based motion representations. These approaches trade off correspondence accuracy, image sharpness, robustness to appearance changes, and motion range.
- Optical-flow interpolation represents dense correspondences between images, but its accuracy is affected by ambiguities, occlusions, and brightness changes.
- Phase-based interpolation represents motion through phase differences and avoids explicit pixel correspondences, improving stability under lighting changes.
- Neural networks have been applied to optical-flow estimation and image synthesis, with supervised flow methods requiring large amounts of labeled data.
- Phase-based interpolation must resolve two possible phase solutions when estimating the middle signal.
- Learned synthesis methods predict flows or appearance transformations to warp and blend pixels, generally producing sharp images.
3. Motion Representation
Phase-based motion represents translation through phase differences, but phase ambiguity and limited receptive fields make larger-motion interpolation difficult. PhaseNet learns intermediate phase and amplitude values hierarchically from steerable-pyramid decompositions instead of relying on hand-crafted corrections.
- Motivation: Phase differences represent translation in sinusoidal signals, providing the basis for phase-based motion representation.A shift of π/3 corresponds to a phase difference of π/3.
- Motivation: The 2π ambiguity creates two valid middle-frame phase solutions, requiring the correct spatial-motion solution to be determined.The paper replaces the prior heuristic correction with direct learning of the desired intermediate phase.
- Image decomposition: Images are decomposed into complex-valued steerable-pyramid subbands spanning spatial frequencies and orientations, with cosine and sine responses defining each subband.Uncaptured frequencies are represented by real-valued high- and low-pass residuals.
- Phase prediction: Each pyramid level provides per-pixel phase values in [−π, π], while higher-level phase comparisons become unreliable when motion exceeds filter receptive fields.Lower-level phase information is used to improve predictions at higher levels under the paper’s coarse-to-fine assumption.
- Phase prediction: PhaseNet predicts the intermediate frame’s phase and amplitude decomposition level by level through a decoder that mirrors the pyramid hierarchy.The final image is reconstructed from predictions across the different levels.
4. Method
PhaseNet predicts the steerable-pyramid decomposition of an intermediate frame from two input-frame decompositions, then reconstructs the image level by level. Its decoder architecture combines image and phase losses with hierarchical training and learned amplitude mixing.
- Learning Phase-based Interpolation: PhaseNet predicts steerable-pyramid decomposition values instead of directly predicting intermediate-frame color pixels.The predicted response ˆR is reconstructed into the interpolated frame.
- Learning Phase-based Interpolation: The network uses two input-frame decompositions, R1 and R2, to predict the intermediate decomposition ˆR with a CNN trained on input-frame triplets and ground-truth interpolations.The decomposition responses are the network inputs, and training minimizes an objective over corresponding interpolation frames.
- Network Architecture: The decoder mirrors pyramid levels, predicts phase and amplitude at each resolution, reuses resized features and predictions, and reconstructs the frame from all levels.Each block applies two convolutions, predicts bounded outputs through a 1×1 convolution and hyperbolic tangent, and passes information to the next level after resizing.
- Reconstruction: Learned mixing weights combine input amplitudes and low-pass residuals, while the high-pass residual is ignored because its blur is often subtle.The learned weights are mapped to [0, 1].
- Training and Implementation Details: Hierarchical training proceeds from the lowest pyramid levels upward, using ground-truth responses for untrained finer levels during reconstruction.This procedure supports stable loss computation while progressively adding blocks.
5. Results
PhaseNet is evaluated against optical-flow, phase-based, and neural interpolation methods on challenging scenes, where it improves handling of large motion, motion blur, and lighting changes despite limitations on high-frequency detail.
- Design choices: Phase loss produces sharper results than image loss alone and also stabilizes training while reducing training time.The phase loss is computed at each resolution level and encodes motion-relevant information.
- Design choices: For high-resolution testing, reusing last-layer weights performs better than averaging lower-level phase values when extending the pyramid beyond training patches.Training uses 256×256 patches with 10 pyramid levels, while 1280×720 testing uses padded 2048×1024 images with 14 levels.
- Qualitative comparisons: On challenging pairs with extreme lighting changes, PhaseNet produces smooth, plausible results and is preferable to the pure phase-based method in the explosion scene.Optical-flow methods show artifacts when brightness constancy is violated.
- Qualitative comparisons: For motion-blurred examples, PhaseNet improves over pure phase-based interpolation and SepConv, although phase reconstruction can cause minor ringing and color artifacts near high-frequency edges.The pure phase-based method exhibits ghosting for large motion, while SepConv fails to interpolate the car under motion blur.
- Quantitative comparisons: Optical flow and SepConv generally achieve better SSIM because they introduce less blur, while PhaseNet performs strongly on motion blur and brightness-change sequences.The authors note that SSIM does not always reflect perceptual quality; for the light sequence, PhaseNet looks noticeably better.
- Overall comparison: PhaseNet combines phase-based representation with a learned decoder and outperforms previous phase-based methods on large motion and high frequencies.The method is also reported as well suited to motion blur and difficult light changes.
6. Conclusions
PhaseNet combines phase-based and data-driven interpolation by predicting an intermediate frame’s phase representation. Combining phase and image losses yields visually preferable results over optical flow in motion blur and brightness-change scenarios.
- Contribution: PhaseNet synthesizes the interpolated frame from a neural-network-predicted phase-based representation.This combines the advantages of phase-based and data-driven frame interpolation.
- Conclusion: Combining phase loss with standard ℓ1 image loss produces visually preferable results over optical flow in challenging scenes with motion blur and brightness changes.The conclusion specifically highlights these two scene conditions.
A. Error Measurements
Table 1 reports PSNR alongside the SSIM error measurements shown in Figure 8 for the evaluated interpolation sequences.
- Metrics: PSNR is reported in Table 1 in addition to the SSIM error reported in Figure 8.Example input images for the evaluated sequences are shown in Figure 11.
B. Details Network Architecture
PhaseNet is built from consecutive blocks whose convolutional layers process phase-decomposition information and predict intermediate-frame representations.
- Block structure: Each PhaseNet block contains two convolution layers, each followed by batch normalization and leaky ReLU with factor 0.2.The prediction layers use one convolution followed by a hyperbolic tangent function.
- Block structure: The network predicts phase and amplitude values level by level from steerable-pyramid decomposition information.The final image is reconstructed from predictions at different levels.
C. Details Model Training
PhaseNet is trained on augmented DAVIS frame triplets using hierarchical optimization, and its architecture and error measurements are documented through dedicated tables and example sequences.
- Training data: The training dataset contains about 10k frame triplets from DAVIS, sampled as random 256×256 patches.Horizontal and vertical flipping provide data augmentation.
- Optimization: Adam training uses β1 = 0.9, β2 = 0.999, learning rate 0.001, and batch size 32, reduced at the highest two stages because of memory limitations.The lower levels train for 12 epochs each, while the highest two train for 6 epochs to maintain approximately equal iteration counts.
- Evaluation: Error measurements compare methods using PSNR on sequences illustrated in Figure 11, where higher values are better.Figure 11 provides example images from the sequences used for the measurements.
- Architecture: PhaseNet has about 460k trainable parameters, with architecture details specified for one color channel, reused weights, and a pyramid using λ = 2 and 4 orientations.The architecture table denotes concatenated inputs with '+'.