Source-linked AI summary
Softmax Splatting for Video Frame Interpolation
Simon Niklaus, Feng Liu
TL;DR
Differentiable forward warping is difficult because multiple source pixels can map to one target location, limiting its use despite backward warping’s broad adoption. The paper introduces softmax splatting to resolve these overlaps, warps frames and feature pyramids using optical flow, and uses a synthesis network for interpolation. The resulting pipeline supports arbitrary-time interpolation, feature-pyramid and flow refinement, and achieves new state-of-the-art frame-interpolation results.
Problem
Forward warping has received less attention because overlapping source pixels require a differentiable way to resolve their conflicting contributions.
Method
Softmax splatting forward-warps two input frames and their feature pyramids using optical flow, after which a synthesis network predicts the intermediate frame.
Results
The proposed frame-interpolation pipeline compares favorably in benchmarks and achieves new state-of-the-art results.
Takeaways & Limitations
Softmax splatting enables task-specific feature pyramids and fine-tuning of the optical-flow estimator for image synthesis.
Takeaways & Limitations
Linear splatting is not invariant to translations of the importance metric Z, so equally separated depths can still be averaged under shifted values.
Abstract
from arXiv · showhide
Differentiable image sampling in the form of backward warping has seen broad adoption in tasks like depth estimation and optical flow prediction. In contrast, how to perform forward warping has seen less attention, partly due to additional challenges such as resolving the conflict of mapping multiple pixels to the same target location in a differentiable way. We propose softmax splatting to address this paradigm shift and show its effectiveness on the application of frame interpolation. Specifically, given two input frames, we forward-warp the frames and their feature pyramid representations based on an optical flow estimate using softmax splatting. In doing so, the softmax splatting seamlessly handles cases where multiple source pixels map to the same target location. We then use a synthesis network to predict the interpolation result from the warped representations. Our softmax splatting allows us to not only interpolate frames at an arbitrary time but also to fine tune the feature pyramid and the optical flow. We show that our synthesis approach, empowered by softmax splatting, achieves new state-of-the-art results for video frame interpolation.
1. Introduction
Video frame interpolation has broad applications, but flow-based methods commonly rely on backward warping, while differentiable forward warping remains difficult when source pixels overlap at target locations. Softmax splatting addresses this ambiguity and supports jointly trainable feature representations and flow refinement.
- Video frame interpolation supports display-rate conversion, video editing, compression, optical-flow estimation, motion-blur synthesis, and novel-view generation.
- Flow-based interpolation commonly estimates bidirectional optical flow from the desired intermediate frame and backward-warps both inputs.
- Forward warping avoids approximating intermediate-frame optical flow but z-buffering for overlapping sources is difficult to fully differentiate.
- Softmax splatting performs differentiable forward warping while jointly supervising all inputs that map to the same target location.
- Softmax splatting enables task-specific feature pyramids, optical-flow fine-tuning, and supervision of the metric used to resolve overlapping mappings.
2. Related Work
Prior work established backward warping and several forward-warping strategies, but overlapping source pixels remain difficult to handle robustly and differentiably. Softmax splatting provides a learned alternative while supporting feature-based synthesis and temporally consistent interpolation.
- Backward warping is widely used, whereas forward warping is less adopted partly because multiple source pixels may map to one target location.
- Z-buffering handles overlapping mappings in prior interpolation work but is not differentiable, limiting end-to-end supervision.
- Linear depth-weighted warping can separate foreground and background, but its result changes with additive bias in the depth estimate.
- The proposed approach warps multi-scale feature pyramids for synthesis, extending feature warping beyond prior applications that did not target image synthesis.
- Summation forward warping can create brightness inconsistencies, while averaging preserves appearance but blends overlapping regions.
- Softmax splatting produces temporally consistent results without requiring a specialized training dataset or cycle-consistent training.
3. Softmax Splatting for Frame Interpolation
The paper introduces softmax splatting as differentiable forward warping for frame interpolation, resolving collisions when multiple source pixels map to one target location. The framework warps input frames and multi-scale feature pyramids, then uses a synthesis network to generate the intermediate frame.
- Frame Interpolation Pipeline: Given bidirectional flow, the method forward-warps I0 by t·F0→1 and I1 by (1−t)·F1→0 to the desired temporal position.The warped color images and multi-resolution features are combined by a synthesis network to produce It.
- Forward Warping via Softmax Splatting: Softmax splatting performs differentiable forward warping while handling multiple source pixels mapping to the same target location.It is designed to address the ambiguity that makes forward warping difficult to differentiate.
- Forward Warping via Softmax Splatting: Summation splatting can create brightness inconsistencies, while average splatting preserves brightness but averages overlapping regions.Summation splatting also receives only partial contributions at some output pixels; average splatting handles these inconsistencies but blends regions such as foreground and background.
- Forward Warping via Softmax Splatting: Softmax splatting separates overlapping regions according to an importance mask Z and is invariant to translations of Z.The importance mask can relate to depth, while the paper instead uses brightness constancy as an occlusion measure and learns a scaling parameter α.
- Feature Pyramids for Image Synthesis: The complete framework estimates bidirectional flow, extracts and warps feature pyramids with the frames, and feeds the warped representations to a GridNet synthesis network.GridNet uses three rows and six columns and incorporates modifications intended to avoid checkerboard artifacts.
- Feature Pyramids for Image Synthesis: Softmax splatting enables end-to-end supervision of the feature extractor, which learns synthesis-relevant features in a multi-scale feature pyramid.The reported evaluation finds significant interpolation-quality improvements, with quality degrading when fewer feature levels are used.
4. Experiments
Experiments evaluate softmax splatting through ablations, feature and flow analyses, temporal-consistency tests, and comparisons with state-of-the-art interpolation methods.
- Evaluation setup: The evaluation uses PSNR, SSIM, and LPIPS on Vimeo-90k, Middlebury, UCF101, and Xiph datasets, including 4K clips.Higher PSNR and SSIM are better, while lower LPIPS is better.
- Softmax splatting ablation: Softmax splatting performs best among the tested forward-warping techniques in PSNR, while linear splatting behaves inconsistently across datasets.Summation splatting performs worst; the authors attribute linear splatting’s erratic behavior to lacking translational invariance.
- Importance metric: Fine-tuning the brightness-constancy importance metric with a small U-Net yields slight PSNR improvements and demonstrates that Z can be supervised.Brightness constancy works well as the importance metric for video frame interpolation.
- Feature pyramids: End-to-end training with feature pyramids improves interpolation quality as feature levels increase, although four levels overfit Vimeo-90k and three levels are selected.The feature pyramids focus on fine details, unlike optical-flow pyramids that emphasize large displacement patterns.
- Optical flow estimation: Fine-tuning PWC-Net for frame interpolation further improves results, while three evaluated optical-flow methods otherwise perform similarly.The fine-tuned PWC-Net is used in the final approach.
- Perception-distortion tradeoff: The color-loss model leads on PSNR and SSIM, whereas the perceptual-loss model leads on LPIPS and better recovers fine details in challenging cases.This comparison reflects the perception-distortion tradeoff.
- Temporal consistency: On high-frame-rate Sintel, the approach maintains temporal consistency better than DAIN, which degrades around frames 8 and 24.The benchmark interpolates frames 1 through 31 from frames 0 and 32 across 13 scenes.
- State-of-the-art comparison: On common datasets, the color-loss model outperforms all other methods in PSNR and SSIM, while the perceptual-loss model performs best in LPIPS.At 4K resolution, all methods degrade substantially across metrics, showing that high-resolution interpolation remains challenging.
5. Conclusion
The paper presents softmax splatting for differentiable forward warping and demonstrates its effectiveness for video frame interpolation. The resulting pipeline compares favorably in benchmarks and achieves new state-of-the-art results.
- Softmax splatting performs differentiable forward warping for video frame interpolation.It addresses cases where different source pixels map to the same target location.
- Feature pyramids can be used successfully for high-quality image synthesis.The paper identifies this as an unexplored aspect of feature pyramids.
- The conceptually simple frame interpolation pipeline compares favorably in benchmarks and achieves new state-of-the-art results.