Source-linked AI summary
Temporal Modulation Network for Controllable Space-Time Video Super-Resolution
Gang Xu, Jun Xu, Zhen Li, Liang Wang, Xing Sun, Ming-Ming Cheng
TL;DR
STVSR raises both spatial and temporal resolution, but deformable-convolution methods were limited to intermediate moments predefined during training. TMNet uses temporal modulation and short- and long-term feature fusion for controllable interpolation, and outperforms previous STVSR methods across three benchmarks.
Problem
STVSR must increase spatial and temporal resolution together, while vanilla PCD estimates motion only for a moment predefined in training and inference.
Source: §1 · p. 1, §3.2 · p. 4Method
Results
Across three benchmarks, TMNet performs controllable interpolation at arbitrary frame rates and outperforms previous STVSR methods.
Source: §1 · p. 2Takeaways & Limitations
The reported evaluation also finds TMNet flexible for intermediate-frame interpolation and its LFC module effective for STVSR.
Source: §5 · p. 8Vanilla PCD estimates motion only for a predefined moment fixed in training and inference.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
Space-time video super-resolution (STVSR) aims to increase the spatial and temporal resolutions of low-resolution and low-frame-rate videos. Recently, deformable convolution based methods have achieved promising STVSR performance, but they could only infer the intermediate frame pre-defined in the training stage. Besides, these methods undervalued the short-term motion cues among adjacent frames. In this paper, we propose a Temporal Modulation Network (TMNet) to interpolate arbitrary intermediate frame(s) with accurate high-resolution reconstruction. Specifically, we propose a Temporal Modulation Block (TMB) to modulate deformable convolution kernels for controllable feature interpolation. To well exploit the temporal information, we propose a Locally-temporal Feature Comparison (LFC) module, along with the Bi-directional Deformable ConvLSTM, to extract short-term and long-term motion cues in videos. Experiments on three benchmark datasets demonstrate that our TMNet outperforms previous STVSR methods. The code is available at https://github.com/CS-GangXu/TMNet.
1. Introduction
STVSR must raise spatial and temporal resolution together, yet prior approaches can introduce temporal inconsistency or restrict interpolation to predefined times. TMNet combines controllable feature interpolation with short- and long-term temporal feature fusion.
1. Introduction
- The motivating broadcast gap is between commonly available FHD video at 30 FPS and UHD displays capable of 120 or 240 FPS.The introduction frames simultaneous spatial and temporal upscaling as necessary for broadcasting FHD videos on UHD TVs.Source: §1 · p. 1
- Sequential VFI-then-VSR methods ignore temporal–spatial correlation and can suffer temporal inconsistency and artifacts.STVSR instead increases spatial and temporal resolutions simultaneously.Source: §1 · pp. 1–2
- Prior STVSR networks could generate only intermediate frames predefined by their architecture, limiting flexible frame-rate applications.The paper cites commercial scenarios where users may want to adjust intermediate frames for visualization.Source: §1 · p. 2
- TMNet uses a Temporal Modulation Block to modulate deformable-convolution feature interpolation for arbitrary intermediate times.The modulation uses a temporal parameter to define the moment to interpolate.Source: §1 · p. 2
- Its two-stage temporal feature fusion combines locally-temporal comparison of adjacent-frame motion cues with globally-temporal exploration of whole-video variation.The locally-temporal module supports short-term cues, while global fusion explores long-term variation.Source: §1 · p. 2
- Across three benchmarks, TMNet supports controllable interpolation at arbitrary frame rates and outperforms state-of-the-art STVSR methods.The reported results include both arbitrary intermediate-frame generation and state-of-the-art performance.Source: §1 · p. 2
2. Related Work
TMNet combines time-controlled feature interpolation with local and global temporal fusion, followed by high-resolution reconstruction. Its training uses a two-step strategy, first training the main network and then the temporal modulation block.
2. Related Work
- Earlier STVSR approaches include space-time smoothness regularization, Markov Random Field modeling, flow-based STARnet [8], and PCD alignment with bidirectional deformable ConvLSTM [45].The paper positions TMNet as building on [45] while adding temporal control and locally temporal feature comparison.Source: §2 · pp. 2–3
3.1. Network Overview
- TMNet has three stages: controllable feature interpolation, temporal feature fusion, and high-resolution reconstruction.A temporal parameter t∈(0, 1) controls the intermediate feature moment; fusion uses LFC and BDConvLSTM, and reconstruction uses residual blocks and Pixel-Shuffle layers.Source: §3.1 · pp. 3–4
3.2. Controllable Feature Interpolation
- TMB modulates PCD offsets using a vector derived from t, enabling feature interpolation at an arbitrary moment between adjacent frames.The modulation is embedded at all three PCD levels; forward- and backward-aligned features are concatenated and passed through a 1×1 convolution.Source: §3.2 · pp. 4–5
3.3. Temporal Feature Fusion
- LFC refines each interpolated feature using motion cues from adjacent frames, while BDConvLSTM aggregates features along the video’s temporal direction.LFC supplies short-term consistency, whereas global fusion addresses LFC’s limited ability to model large or fast motions.Source: §3.1 · p. 4, §3.3 · p. 5
3.4. High-Resolution Reconstruction
- High-resolution reconstruction refines fused features with 40 residual blocks, then uses Pixel-Shuffle layers to produce the output video.The reconstruction stage adds the refined features to initial features before the Pixel-Shuffle upsampling.Source: §3.4 · p. 5
3.5. Training Details
- Training proceeds in two steps: train the main network without TMB, then freeze it and train TMB for temporal modulation.The first step uses Vimeo-90K at t=0.5; the second uses Adobe240fps with t∈{1/6, 2/6, 3/6, 4/6, 5/6}.Source: §3.5 · p. 6
4. Experiments
TMNet is evaluated on standard STVSR benchmarks and compared with one- and two-stage methods. The experiments report its benchmark performance and examine temporal modulation design choices.
4.1. Experimental Setup
- Evaluation uses Vimeo-90K septuplet training data and Vid4 and Vimeo-90K test sets; Adobe240fps trains TMB for controllable interpolation.The Vimeo-90K test set is split into Fast, Medium, and Slow motion subsets; HR frames are downsampled by a factor of four, and PSNR/SSIM are measured on the Y channel.Source: §4.1 · pp. 6–7
4.2. Comparison to State-of-the-arts
- TMNet exceeds Zooming SlowMo by 0.12dB on Vid4, 0.23dB on Vimeo-Fast, 0.19dB on Vimeo-Medium, and 0.15dB on Vimeo-Slow in PSNR.It achieves better SSIM than competitors in most cases, runs at 14.69fps, and is only slower than Zooming Slow-Mo among the compared methods.Source: §4.2 · p. 7
- Figure 5 shows five intermediate frames from Adobe240fps and reconstructions from Vimeo-Fast and Vimeo-Slow.The authors report clearer structures and textures with TMNet, including a bag texture pattern on a Vimeo-Slow clip.Source: §4.2 · p. 7, Figure 5 · p. 7
4.3. Ablation Study
- Modulating all three PCD pyramid levels outperforms single-level TMB variants, with level L1 the strongest single-level choice.The reported descending order of TMB-L1, TMB-L2, and TMB-L3 indicates L1 is more important for temporal modulation.Source: §4.3 · p. 8, Table 2 · p. 8
- The three-nonlinear-convolution TMB is 0.02dB higher in PSNR than TMB-Linear on Adobe240fps.The authors characterize nonlinear transformation as only a little better than the linear alternative.Source: §4.3 · p. 8, Table 3 · p. 8
4. How important is the proposed LFC module to our
The ablations examine how LFC contributes to temporal fusion and how initial features affect reconstruction. TMNet’s LFC→GFF variant performs best across the reported datasets.
LFC and temporal feature fusion
- Reversing the fusion order to GFF→LFC makes TMNet collapse during training, as noisy long-term information confuses LFC’s deformable-convolution learning.The tested alternatives remove LFC or concatenate LFC and GFF features before convolutional fusion.Source: §4 · p. 8
- TMNet (LFC→GFF) achieves the best PSNR results on all four datasets, exceeding GFF by 0.07dB on Vid4, 0.17dB on Vimeo-Fast, 0.15dB on Vimeo-Medium, and 0.11dB on Vimeo-Slow.The authors attribute this result to LFC exploiting short-term motion cues among adjacent frames.Source: Table 4 · p. 8, §4 · p. 8
Combining initial and high-quality features
- Adding initial features FL to the Baseline clearly improves PSNR on the evaluated Vimeo-90K and Vid4 datasets.The comparison supports combining high-quality features FH with initial features FL before the Pixel-Shuffle layers.Source: Table 5 · p. 8, §4 · p. 8
5. Conclusion
TMNet flexibly interpolates intermediate frames for STVSR, combining controllable deformable-convolution feature interpolation with short- and long-term temporal fusion.
5. Conclusion
- Experiments on three benchmarks demonstrate flexible interpolation, quantitative and qualitative advantages over previous methods, and the effectiveness of LFC for STVSR.The approach pairs TMB with short-term LFC and a Bi-directional Deformable ConvLSTM for long-term feature fusion.Source: §5 · p. 8
1. Content
The supplemental file provides additional network, training, interpolation, and comparison details for TMNet.
1. Content
- The supplement covers TMNet’s network structure and two-step training scheme.These topics are detailed in sections 2 and 3 of the supplemental file.Source: §1 · p. 11
- It reports additional evidence on arbitrary intermediate-frame interpolation and visual comparisons with prior STVSR methods.These topics appear in supplemental sections 4 and 5.Source: §1 · p. 11
- The supplement examines how one-stage rather than two-stage training affects TMNet with TMB on STVSR.This is listed as supplemental section 6.Source: §1 · p. 11
2. Detailed Network Structure of Our TMNet
TMNet combines residual feature extraction, controllable feature interpolation, temporal-cue modules, and an upsampling stage. Its CFI uses a PCD module modulated by TMB, while LFC and BDConvLSTM supply temporal information.
2. Detailed Network Structure of Our TMNet
- Five residual blocks extract the initial features, with each block using Conv-ReLU-Conv operations and a skip connection.Source: §2 · p. 11
- CFI is performed by the PCD module [41], modulated by the proposed TMB for controllable feature interpolation.Source: §2 · p. 11
- The network uses LFC and the Bi-directional Deformable ConvLSTM [45] to handle temporal features.Source: §2 · p. 11
- The upsampling stage uses two Conv-Pixel-Shuffle-LeakyReLU sequences followed by Conv-LeakyReLU-Conv.Source: §2 · p. 11
3. More Details of Two-step Training Scheme
TMNet is trained in two steps: the first learns the main network using Vimeo-90K septuplets, and the second trains TMB for temporal modulation while fixing the main-network weights.
3. More Details of Two-step Training Scheme
- Step 1 trains on Vimeo-90K septuplets, with Vid4 and Vimeo-Fast, Vimeo-Medium, and Vimeo-Slow used for evaluation.Source: §3 · p. 11
- For Step 1, t = 0.5 removes TMB, and the 1st, 3rd, 5th, and 7th LR frames train TMNet to reconstruct the corresponding seven-frame HR sequence.Training uses Kaiming initialization without pre-trained weights and runs for 600,000 iterations.Source: §3 · p. 11
- Step 2 fixes the main-network weights and trains TMB on Adobe240fps to interpolate five intermediate frames at t ∈ {1/6, 2/6, 3/6, 4/6, 5/6}.The dataset is split into 100 training, 16 validation, and 17 test videos; TMB training takes 1,500 iterations.Source: §3 · p. 11
4. Flexible STVSR with Arbitrary Number of Intermediate Frames
TMNet is evaluated for flexible intermediate-frame generation using temporal parameter t. The reported visualizations show continuously varying interpolated frames and a more consistent temporal motion trajectory than Zooming SlowMo.
4. Flexible STVSR with Arbitrary Number of Intermediate Frames
- On Vimeo-90K, TMNet interpolates nine intermediate frames between adjacent frames using t values from 0.1 to 0.9, although training uses five frames between Frames 1 and 7.Source: §4 · p. 11
- The interpolated frames vary continuously as t changes from 0.1 to 0.9, supporting generation of arbitrary numbers of intermediate frames for t ∈ (0, 1).Source: §4 · pp. 11–12
- For Clip 0277, TMNet’s nine-frame interpolation shows a more consistent temporal motion trajectory than Zooming SlowMo’s single interpolated frame, whose trajectory has clear breaking variations.The trajectory visualization concatenates 1D pixel vectors sampled along a red line across the frames.Source: §4 · p. 12
5. More Visual Comparisons on STVSR
This section presents visual comparisons of TMNet with one-stage and two-stage STVSR approaches on Vid4 and Vimeo-90K. The comparison uses methods trained on Vimeo-90K septuplets and evaluates on the Vimeo-90K test set and Vid4.
5. More Visual Comparisons on STVSR
- The visual comparison covers one-stage STVSR methods Zooming SlowMo [45] and STARnet [8], alongside two-stage approaches.Source: §5 · p. 12
- The two-stage comparisons pair SuperSloMo [15], DAIN [1], or SepConv [28] for interpolation with RCAN [47], RBPN [7], or EDVR [41] for super-resolution.Source: §5 · p. 12
- All compared methods are trained on Vimeo-90K septuplets and evaluated on the Vimeo-90K test set [46] and Vid4.TMNet uses t = 0.5 to reconstruct seven HR frames from the 1st, 3rd, 5th, and 7th LR frames.Source: §5 · p. 12
6. Training our TMNet in One-step
On Adobe240fps, the one-step variant underperformed the two-step TMNet by 1.84 dB PSNR. The authors attribute the drop to difficulty learning useful motion cues and optimizing TMB from scratch in early training.
6. Training our TMNet in One-step
- The one-step scheme optimizes the main TMNet and TMB simultaneously without pre-training, whereas the two-step scheme optimizes them separately, main TMNet first.The two-step-trained variant is TMNet-two (the original TMNet); the one-step variant is TMNet-one.Source: §6 · p. 12
- From-scratch one-step training initially fails to extract useful motion cues, hindering TMB optimization for meaningful features at arbitrary time t.The passage says this leads to failure to estimate motion cues and interpolate intermediate frames at arbitrary t ∈ (0, 1).Source: §6 · p. 12
- TMNet-one loses 1.84 dB PSNR versus TMNet-two on the Adobe240fps test set.Table 6 reports the PSNR comparison; both variants were trained on the Adobe240fps train set.Source: Table 6 · p. 12, §6 · p. 12