Source-linked AI summary
VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, Xiaolong Wang
TL;DR
Low-resolution, low-frame-rate storage creates a need for flexible space-time video super-resolution beyond methods limited to fixed scales. VideoINR learns a continuous implicit video representation that decodes arbitrary spatial resolutions and frame rates, achieving competitive in-distribution performance and stronger out-of-distribution results. Its main limitation is handling very large motion, which remains challenging for video interpolation.
Problem
Existing STVSR methods commonly support fixed up-sampling scales, limiting flexible interpolation across spatial resolutions and frame rates.
Method
VideoINR learns a continuous implicit neural representation that uses spatial features and temporal motion modeling to decode queried space-time coordinates into RGB values.
Results
VideoINR performs competitively with state-of-the-art STVSR methods on common scales and outperforms prior works by a large margin on out-of-distribution scales.
Takeaways & Limitations
VideoINR supports flexible video decoding at arbitrary spatial resolutions and frame rates, including scales outside its training distributions.
Takeaways & Limitations
VideoINR can perform poorly on cases involving very large motion, which remains an open challenge for video interpolation.
Abstract
from arXiv · showhide
Videos typically record the streaming and continuous visual data as discrete consecutive frames. Since the storage cost is expensive for videos of high fidelity, most of them are stored in a relatively low resolution and frame rate. Recent works of Space-Time Video Super-Resolution (STVSR) are developed to incorporate temporal interpolation and spatial super-resolution in a unified framework. However, most of them only support a fixed up-sampling scale, which limits their flexibility and applications. In this work, instead of following the discrete representations, we propose Video Implicit Neural Representation (VideoINR), and we show its applications for STVSR. The learned implicit neural representation can be decoded to videos of arbitrary spatial resolution and frame rate. We show that VideoINR achieves competitive performances with state-of-the-art STVSR methods on common up-sampling scales and significantly outperforms prior works on continuous and out-of-training-distribution scales. Our project page is at http://zeyuan-chen.com/VideoINR/ .
1. Introduction
VideoINR addresses the need to recover low-resolution, low-frame-rate videos for high-quality visualization and downstream applications. It learns a continuous video representation that supports arbitrary spatial resolutions and frame rates, with strong performance especially outside training distributions.
- Low spatial resolution and frame rate arise from the high cost of recording and storing high-fidelity video.
- Space-Time Video Super-Resolution jointly increases spatial resolution and frame rate from low-resolution, low-frame-rate input videos.
- VideoINR represents video with an implicit neural function that maps space-time coordinates to RGB values, enabling arbitrary spatial and temporal sampling.
- VideoINR uses spatial and temporal implicit representations to model continuous features and motion for RGB prediction at queried coordinates.
- VideoINR performs competitively on in-distribution scales and significantly outperforms other methods on out-of-distribution spatial and temporal scales.
2. Related Work
Implicit neural representations model signals as continuous coordinate-based neural functions, while STVSR methods increase spatial and temporal resolution together. VideoINR extends this paradigm by supporting arbitrary spatial resolutions and frame rates beyond fixed training scales.
- Implicit neural representations encode signals as neural functions mapping coordinates to quantities such as signed distance, occupancy, density, or RGB values.
- STVSR aims to simultaneously increase spatial and temporal resolutions in low-resolution, low-frame-rate videos.
- Many prior STVSR methods operate at specific pre-defined up-sampling scales or times, limiting their application scope.
- VideoINR provides a continuous video representation supporting frame interpolation at arbitrary spatial resolutions and frame rates.
3. Video Implicit Neural Representation
VideoINR constructs a continuous space-time representation by separately modeling spatial features and temporal motion, then combining them through feature warping. The representation supports querying arbitrary coordinates and decoding them into RGB values, with multiscale inputs helping mitigate limited receptive fields.
- 3. Video Implicit Neural Representation: VideoINR models a video as a continuous function over spatial and temporal coordinates, represented with MLPs and decoded to RGB values.The representation takes spatial coordinate x_s and temporal coordinate x_t as inputs; f is defined by encoded features and network parameters, and s is the predicted RGB value.
- 3.1. Continuous Spatial Representation: SpatialINR converts arbitrary 2D spatial coordinates into continuous feature vectors by querying an encoded feature map and combining the nearest feature with relative position information.The resulting continuous feature domain is designed to support subsequent temporal modeling, distinguishing SpatialINR from image-focused continuous representations such as LIIF.
- 3.2. Continuous Temporal Representation: TemporalINR extends the spatial feature domain through time by predicting a continuous motion flow from a queried space-time coordinate.Rather than directly generating target features, the method learns a motion flow field and can replace discrete input frames and spatial coordinates with continuous features from SpatialINR.
- 3.3. Space-Time Continuous Representation: The unified representation warps SpatialINR features according to TemporalINR motion, queries the displaced coordinate, and concatenates two independently generated warped features.This produces the continuous space-time feature at the queried coordinate; the two flows may implicitly capture bidirectional correspondences without explicit supervision.
- 3.4. RGB Decoding: The decoder predicts RGB values from space-time features while aggregating encoded features and input frames to enlarge the effective receptive field.For full-frame synthesis, the method queries all pixel coordinates, forms a high-resolution feature map, and generates a complete motion flow field.
4. Experiments
Experiments evaluate VideoINR on benchmark datasets, standard and out-of-distribution scales, efficiency, qualitative quality, and architectural ablations. VideoINR is competitive on common settings and shows advantages for multiple-frame interpolation and out-of-distribution generalization.
- Datasets and setup: Experiments use Vid4, GoPro, and Adobe240, with Adobe240 providing 100 training, 16 validation, and 17 test videos.Adobe240 videos are converted into image sequences, and low-resolution inputs are generated by bicubic interpolation.
- Comparison methods: VideoINR is compared with two-stage VFI-plus-VSR pipelines and one-stage STVSR methods including Zooming SlowMo and TMNet.The comparison includes SuperSloMo, QVI, DAIN, Bicubic Interpolation, EDVR, and BasicVSR.
- Standard-scale results: VideoINR achieves the best performance on GoPro-Average and Adobe-Average, indicating an advantage for modeling temporal information in multiple-frame interpolation.On Vid4, TMNet performs best, while VideoINR slightly underperforms some compared methods in another setting.
- Out-of-distribution results: VideoINR outperforms other methods by a large margin on out-of-distribution space and time scales.Against Zooming SlowMo, VideoINR is better at ×16 ×4, whereas Zooming SlowMo is slightly better for single-frame interpolation at ×4 ×2.
- Efficiency: VideoINR infers faster than other models on multi-frame interpolation because latent frames can be synthesized directly by MLPs after encoding.At up-sampling time scale ×2, the efficiency of different methods is close.
- Qualitative results: Qualitatively, VideoINR remains steady across in-distribution and out-of-distribution temporal coordinates, while comparison methods show motion-related failures.DAIN + BasicVSR degrades out of distribution, and TMNet fails to recover objects with large motion in the example shown.
- Ablation studies: Ablations show that motion flow, multi-scale feature aggregation, and separate spatial and temporal functions each contribute to performance.Removing flow correspondence or multi-scale aggregation reduces performance, and decoding space and time with one network is inferior to using two functions.
5. Discussion
VideoINR represents videos at arbitrary spatial and temporal resolutions, performing competitively on common scales and substantially better on out-of-distribution scales. The authors identify very large motion as a remaining challenge for video interpolation.
- VideoINR represents videos at arbitrary spatial and temporal resolutions for STVSR.
- VideoINR performs competitively with state-of-the-art STVSR methods on common up-sampling scales.
- VideoINR outperforms prior works by a large margin on out-of-distribution scales.
- Very large motions remain challenging cases for VideoINR and video interpolation generally.
A. Implementation Details
The experiments train competing models under matched settings while using different frame-selection procedures for VideoINR and comparison methods. The section also includes visualized failure and inference-time comparisons.
- All models are trained from scratch with the training settings specified by their original papers for fair comparison.
- Zooming SlowMo is trained to interpolate fixed time instances 0, 0.5, and 1, while TMNet uses a two-stage training setup.
- TMNet may have an experimental advantage because it is trained with more data.
- VideoINR uses the first and ninth frames from nine-frame sequences as inputs and randomly selects three target frames per iteration.
- Zooming SlowMo, TMNet, and VideoINR are optimized with the Charbonnier loss function.
- Figures 5 and 6 provide inference-time comparisons across space scales and a visual failure case, respectively.
B. Efficiency on Different Scales
The efficiency evaluation compares VideoINR against a two-stage SuperSloMo-plus-LIIF baseline across different spatial up-sampling scales.
- Inference efficiency is evaluated across different spatial up-sampling scales using SuperSloMo and LIIF as the baseline.This baseline supports arbitrary up-sampling scales in both space and time.
C. Limitations
VideoINR can fail in challenging cases involving large motion and occlusion. Errors in the motion flow field can produce blurred results with unclear boundaries.
- Large motion and occlusion can cause motion-flow errors in VideoINR.
- Motion-flow errors can produce blurred results with unclear boundaries.
D. Additional Qualitative Results
VideoINR is compared qualitatively with DAIN + BasicVSR and TMNet at in-distribution and out-of-distribution temporal scales. It produces sharper, more consistent results, especially for large motion and texture preservation beyond training scales.
- In-distribution time scale: At temporal scale 8, VideoINR produces sharper edges and clearer details than DAIN + BasicVSR and TMNet.The comparison uses a spatial scale of 4 and a temporal scale within the training distribution.
- Out-of-distribution time scale: At temporal scales 12 and 16, VideoINR better recovers objects with large motion and preserves more textural information than other methods.These temporal scales are outside the training distribution.
- Comparison with baselines: DAIN + BasicVSR and TMNet tend to generate blurry regions or artifacts in the qualitative comparisons.
- Comparison setup: Figures 7–10 provide qualitative comparisons across in-distribution and out-of-distribution temporal scales.