Source-linked AI summary
Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution
Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P. Allebach, Chenliang Xu
TL;DR
STVSR must generate HR slow-motion video from LR, LFR input, but two-stage VFI-plus-VSR pipelines separate interrelated temporal and spatial tasks and can be inefficient. The paper proposes one-stage feature interpolation with deformable ConvLSTM and HR reconstruction, achieving stronger benchmark performance and over threefold faster processing than recent two-stage methods.
Problem
Two-stage VFI-plus-VSR pipelines cannot fully exploit the intra-relatedness of temporal interpolation and spatial super-resolution, while their large synthesis and reconstruction modules make them time-consuming.
Method
A unified one-stage network interpolates LR frame features, aligns and aggregates temporal information with deformable ConvLSTM, and reconstructs HR frames end-to-end.
Results
The method achieves state-of-the-art STVSR performance on Vid4 and Vimeo, runs three times faster than DAIN+EDVR, and has nearly 4× fewer parameters.
Takeaways & Limitations
One-stage joint learning provides a more effective and efficient alternative to existing two-stage STVSR networks while handling challenging fast-motion videos.
Takeaways & Limitations
Vanilla ConvLSTM has limited ability to handle large motions because its small receptive fields only implicitly capture motion between previous states and current features.
Abstract
from arXiv · showhide
In this paper, we explore the space-time video super-resolution task, which aims to generate a high-resolution (HR) slow-motion video from a low frame rate (LFR), low-resolution (LR) video. A simple solution is to split it into two sub-tasks: video frame interpolation (VFI) and video super-resolution (VSR). However, temporal interpolation and spatial super-resolution are intra-related in this task. Two-stage methods cannot fully take advantage of the natural property. In addition, state-of-the-art VFI or VSR networks require a large frame-synthesis or reconstruction module for predicting high-quality video frames, which makes the two-stage methods have large model sizes and thus be time-consuming. To overcome the problems, we propose a one-stage space-time video super-resolution framework, which directly synthesizes an HR slow-motion video from an LFR, LR video. Rather than synthesizing missing LR video frames as VFI networks do, we firstly temporally interpolate LR frame features in missing LR video frames capturing local temporal contexts by the proposed feature temporal interpolation network. Then, we propose a deformable ConvLSTM to align and aggregate temporal information simultaneously for better leveraging global temporal contexts. Finally, a deep reconstruction network is adopted to predict HR slow-motion video frames. Extensive experiments on benchmark datasets demonstrate that the proposed method not only achieves better quantitative and qualitative performance but also is more than three times faster than recent two-stage state-of-the-art methods, e.g., DAIN+EDVR and DAIN+RBPN.
1. Introduction
STVSR generates high-resolution, high-frame-rate video from low-resolution, low-frame-rate input, but conventional two-stage pipelines separate interrelated temporal interpolation and spatial super-resolution. The paper proposes a unified one-stage framework designed to improve effectiveness and efficiency.
- Motivation: STVSR generates photo-realistic, high-space-time-resolution video from low-resolution, low-frame-rate input for applications requiring detailed motion and imagery.HR slow-motion videos are described as visually appealing because they contain fine image details and clear motion dynamics.
- Limitations of Prior Methods: Hand-crafted regularization and strong assumptions limit prior methods’ ability to model diverse space-time patterns and can make optimization computationally expensive.One cited method requires approximately 1 hour to process 60 frames.
- Two-Stage Approaches: Two-stage pipelines first interpolate missing LR frames with VFI and then reconstruct HR frames with VSR, but temporal interpolation and spatial super-resolution are intra-related.Separating the tasks into individual procedures prevents these methods from fully exploiting their relationship.
- Proposed Framework: The proposed one-stage framework jointly learns temporal interpolation and spatial super-resolution while mapping an LR, LFR sequence directly to HR, HFR video.It interpolates intermediate LR features, aggregates temporal information with deformable ConvLSTM, and reconstructs HR frames end-to-end.
- Results: The method achieves state-of-the-art STVSR performance on Vid4 and Vimeo, runs three times faster than DAIN+EDVR, and reduces model size by nearly 4×.The paper attributes the efficiency to using one frame-reconstruction network instead of two large networks in two-stage approaches.
2. Related Work
Prior STVSR work uses handcrafted priors or computationally expensive optimization, while related VFI and VSR methods address temporal synthesis or spatial reconstruction separately. Zooming Slow-Mo instead learns feature interpolation and spatial super-resolution jointly in one stage.
- Related Topics: Related work covers video frame interpolation, video super-resolution, and space-time video super-resolution as distinct research areas.VFI synthesizes non-existent intermediate frames, whereas VSR reconstructs HR frames from reference and neighboring LR frames.
- Video Super-Resolution: VSR methods must temporally align supporting LR frames with a reference frame, using approaches such as optical flow, dynamic filters, or deformable alignment.Optical-flow estimation can be inaccurate, and flow warping can introduce artifacts.
- Framework Overview: The framework overview shows direct reconstruction of consecutive HR frames, with feature temporal interpolation and bidirectional deformable ConvLSTM for local and global temporal contexts.The figure illustrates two LR inputs from a longer sequence rather than the entire input sequence.
- Proposed Direction: Zooming Slow-Mo avoids simply chaining VFI and VSR by jointly learning temporal feature interpolation and spatial super-resolution without LR intermediate-frame supervision.The framework directly learns the mapping from partial LR observations to HR video frames.
- Space-Time Video Super-Resolution: STVSR is a highly ill-posed inverse problem because LR pixels and sometimes entire LR frames are unavailable.Earlier methods constrain reconstruction with directional smoothness, Markov random fields, local motion, or patch recurrence priors.
3. Space-Time Video Super-Resolution
Zooming Slow-Mo is a one-stage framework that jointly interpolates missing LR frame features and reconstructs HR slow-motion frames. It uses deformable temporal processing to capture local and global motion contexts before shared HR reconstruction.
- Framework overview: The framework extracts features, interpolates missing intermediate LR features, aggregates temporal information with deformable ConvLSTM, and reconstructs HR frames.Its four main components are a feature extractor, feature temporal interpolation module, deformable ConvLSTM, and HR frame reconstructor.
- Frame feature temporal interpolation: Feature temporal interpolation synthesizes intermediate LR feature maps from neighboring frame features rather than interpolating pixel-wise LR frames.The interpolated feature map is blended from deformably sampled features using learnable sampling parameters.
- Frame feature temporal interpolation: Learnable deformable offsets capture forward and backward motion information from neighboring frames, enabling interpolation to handle very large video motions.The unavailable intermediate LR frame is approximated using motion information between the two available input frames.
- Deformable ConvLSTM: Deformable ConvLSTM aligns hidden and cell states with the current input feature map before temporal aggregation, addressing mismatches caused by large motion.Unlike vanilla ConvLSTM, it explicitly performs deformable alignment and can process features bidirectionally to exploit broader temporal context.
- Frame reconstruction: A temporally shared synthesis network reconstructs HR frames from aggregated hidden states using residual blocks and PixelShuffle upscaling.The model is end-to-end trained with HR-frame supervision and a Charbonnier reconstruction loss.
4. Experiments and Analysis
Experiments evaluate Zooming SlowMo on Vid4 and Vimeo testsets using PSNR, SSIM, visual comparisons, efficiency measurements, and module ablations. The one-stage model improves reconstruction quality and efficiency, particularly for fast-motion videos.
- Experimental Setup: Vid4 and Vimeo testsets evaluate performance across slow, medium, and fast motion conditions using PSNR and SSIM.Vimeo test clips are partitioned into 1,225 fast-, 4,977 medium-, and 1,613 slow-motion clips.
- Comparison to State-of-the-art Methods: DAIN+EDVR is the best-performing two-stage approach among the 12 compared methods.
- Comparison to State-of-the-art Methods: 0.19dB, 0.25dB, 0.75dB, and 1dB PSNR gains over DAIN+EDVR are obtained on Vid4, Vimeo-Slow, Vimeo-Medium, and Vimeo-Fast, respectively.The largest improvement occurs on the fast-motion set.
- Comparison to State-of-the-art Methods: More than 4× and 3× smaller models yield more than 3× and 8× faster inference than DAIN+EDVR and DAIN+RBPN, respectively.The method remains more than 2× faster than the two-stage SuperSlowMo alternatives.
- Comparison to State-of-the-art Methods: Visual comparisons show finer details, more accurate structures, and fewer blurring artifacts, including on challenging fast-motion sequences.The paper attributes the improved handling of large motion to jointly learning temporal and spatial super-resolution.
- Ablation Study: 0.16dB on Vid4 and 0.73dB on Vimeo-Fast are the PSNR gains from deformable feature interpolation over the convolution-only baseline.Learned offsets exploit local temporal contexts and capture forward and backward motions without explicit supervision.
- Ablation Study: Vanilla ConvLSTM helps on slow motion but performs worse on fast motion, whereas DConvLSTM improves over both alternatives by learning temporal alignment.Adding bidirectional DConvLSTM further improves performance on both slow- and fast-motion testsets.
5. Conclusion
The paper concludes that one-stage STVSR jointly performs temporal feature interpolation and spatial super-resolution without synthesizing intermediate LR frames. Deformable temporal modules improve motion handling, and experiments show the framework is more effective and efficient than two-stage networks.
- Conclusion: The proposed one-stage framework directly reconstructs HR and HFR videos without synthesizing intermediate LR frames.
- Conclusion: Deformable feature interpolation and deformable ConvLSTM adaptively use local and global temporal contexts to handle motion.
- Conclusion: Extensive experiments show that the one-stage framework is more effective and efficient than existing two-stage networks.
Network Architecture
The network uses temporally shared feature extraction and reconstruction modules while processing frames independently. Its feature temporal interpolation and deformable sampling components are illustrated with implementation details including a PCD structure.
- Implementation: The implementation adopts a Pyramid, Cascading and Deformable structure to exploit multi-scale contexts with a feature pyramid.
- Feature Temporal Interpolation: The feature temporal interpolation network predicts intermediate LR frame features from neighboring LR feature maps rather than synthesizing intermediate LR pixels.
- Feature Temporal Interpolation: The deformable sampling module uses learned sampling parameters to exploit neighboring feature contexts.
- Overall Framework: Feature extraction and HR reconstruction networks are temporally shared across frames, while different frames are processed independently.