Source-linked AI summary
Recurrent Back-Projection Network for Video Super-Resolution
Muhammad Haris, Greg Shakhnarovich, Norimichi Ukita
TL;DR
Video super-resolution must recover missing details while handling temporal frames with different motion patterns. RBPN combines target-frame SISR with separately processed neighbor-frame MISR through recurrent back-projection, and it outperforms prior methods across reported datasets and motion regimes.
Problem
Existing VSR approaches have difficulty training with simultaneously processed frames or jointly modeling subtle and significant motion changes.
Method
RBPN integrates target-frame SISR and neighbor-frame MISR using recurrent encoder-decoder back-projection with explicit inter-frame motion representation.
Results
RBPN outperforms VSR-DUF by 1.22 dB, 1.44 dB, and 2.54 dB PSNR on slow, medium, and fast Vimeo-90k motion, respectively.
Takeaways & Limitations
RBPN preserves different temporal scales and benefits from longer context, with added frames improving performance by roughly 0.2 dB.
Takeaways & Limitations
When no other frames are available or the video is completely static, RBPN effectively ignores MISR and falls back to recurrent SISR.
Abstract
from arXiv · showhide
We proposed a novel architecture for the problem of video super-resolution. We integrate spatial and temporal contexts from continuous video frames using a recurrent encoder-decoder module, that fuses multi-frame information with the more traditional, single frame super-resolution path for the target frame. In contrast to most prior work where frames are pooled together by stacking or warping, our model, the Recurrent Back-Projection Network (RBPN) treats each context frame as a separate source of information. These sources are combined in an iterative refinement framework inspired by the idea of back-projection in multiple-image super-resolution. This is aided by explicitly representing estimated inter-frame motion with respect to the target, rather than explicitly aligning frames. We propose a new video super-resolution benchmark, allowing evaluation at a larger scale and considering videos in different motion regimes. Experimental results demonstrate that our RBPN is superior to existing methods on several datasets.
1. Introduction
Video super-resolution must recover missing target-frame details while exploiting neighboring frames across different motion regimes. RBPN addresses this by combining single-image and multi-image super-resolution through recurrent back-projection.
- Motivation: Video super-resolution enhances a low-resolution target frame by filling in missing fine details using available temporal context.The paper distinguishes SISR, MISR, and VSR, with VSR focusing on a target frame in a sequence.
- Challenges: Precise frame alignment is difficult for MISR because independently aligned frames lack temporal-smoothness cues.The paper notes that super-resolution requires highly precise, potentially sub-pixel alignment.
- Challenges: Concatenation-based VSR processes many frames simultaneously, while recurrent methods struggle to jointly model subtle and significant motion changes.Explicitly aligned methods generally outperform approaches without explicit spatial alignment, but the cited limitations remain.
- Approach: RBPN uses iterative back-projection to calculate residual differences between the target and corresponding frames, then feeds those residuals back for refinement.This design treats differences between context frames and the target as separate sources of missing detail.
- Architecture: RBPN processes the target and each target-neighbor pair separately, fusing horizontal SISR features with vertical MISR residual features through an RNN.This differs from recurrent approaches that feed all temporal frames sequentially along one path.
- Approach: RBPN integrates SISR features from the target with MISR features from neighboring frames through a recurrent encoder-decoder framework.The SISR path accepts only the target frame, whereas the MISR path also accepts neighboring frames.
2. Related Work
Deep VSR methods preserve temporal information through concatenation, aggregation, recurrence, or alignment, but each design faces limitations in representing diverse motion. RBPN is presented as a separate recurrent back-projection alternative.
- Deep SR: Deep super-resolution work includes improved upsampling, residual learning, back-projection, recursive layers, and progressive upsampling.The related-work discussion focuses on end-to-end deep neural-network approaches.
- Recurrent VSR: VSR recurrent networks use either many-to-one mapping from frame sequences to one target HR frame or synchronous many-to-many mapping to HR frame sequences.These architectures differ in whether they reconstruct one target or a sequence of outputs.
- Temporal Concatenation: Temporal concatenation extends SISR to multiple input images but can fail to represent multiple motion regimes because frames are concatenated together.VSR-DUF is cited as constructing upsampling filters and residual images within this family.
- RBPN: RBPN’s overview combines an SISR path for enlarging the target with an MISR path that extracts residual features from target-neighbor pairs and dense motion maps.Temporal connections link the projection steps, where neighbor information supplies missing target details.
- Recurrent VSR: Recurrent feedback exploits temporal smoothness, yet jointly modeling subtle and significant changes across all frames remains difficult.The related work also describes motion compensation and convolutional LSTM extensions.
3. Recurrent Back-Projection Networks
RBPN reconstructs a target HR frame through initial feature extraction, recurrent projections over neighboring frames, and final reconstruction. Its encoder-decoder chain iteratively refines features while accommodating static or absent context.
- Network Architecture: RBPN takes n + 1 LR frames ending at target frame It and outputs its HR version SRt.The target output has spatial dimensions larger than the LR input.
- Network Architecture: The network has three stages: initial feature extraction, multiple projections, and reconstruction, trained jointly end-to-end.Target features are extracted first, then neighbor-frame information is incorporated through recurrent projection modules.
- Reconstruction: The reconstruction module feeds concatenated HR feature maps from all neighboring frames into frec to produce SRt.In the experiments, frec is implemented as a single convolutional layer.
- Multiple Projection: Each shared projection module takes target LR features and neighbor features, producing next LR features and HR features for recurrent processing.The inputs and outputs are feature tensors associated with the target-neighbor projection step.
- Multiple Projection: The encoder creates a hidden HR-feature state from a neighbor projection, and the decoder converts that state into the next encoder input.This recurrent encoder-decoder mechanism supplies temporal connections across projection steps.
- Interpretation: RBPN combines target-only SISR enlargement with implicit motion absorption from target-neighbor pairs in the MISR path.Residual features refine HR features, while decoded hidden states become the next LR inputs; the process repeats across neighbors.
- Interpretation: With no neighboring frames or completely static video, RBPN effectively ignores MISR and falls back to recurrent SISR.The architecture is most useful when individual frames contribute distinct missing details to the target.
4. Experimental Results
Experiments analyze RBPN’s design choices and show strong performance across context lengths, motion regimes, and benchmark datasets. RBPN generally outperforms existing VSR methods while integrating spatial and temporal information through recurrent back-projection.
- Baseline comparisons: RBPN combines a DBPN-based SISR path with recurrent MISR processing, while baseline variants remove video context, temporal concatenation, or the decoder.The baseline study compares DBPN, DBPN-MISR, and RBPN-MISR against the full model.
- Context length: Performance improves with longer context on SPMCS-32, with additional frames increasing RBPN performance by roughly 0.2 dB.RBPN/3 also performs better than VSR-DUF despite using fewer neighboring frames.
- Context length: RBPN/3 reconstructs the brick pattern better than VSR-DUF, and increasing context length produces further visual improvements.The qualitative comparison illustrates the performance gains associated with longer temporal context.
- Temporal integration: Temporal integration remains effective across ordering choices, while removing explicit optical flow causes a smaller performance benefit than retaining it.The reported results describe order robustness and a somewhat, but not substantially, beneficial optical-flow representation.
- Benchmark comparisons: On SPMCS-11, RBPN/6-PF exceeds VSR-DUF and DRDVSR by 0.68 dB and 1.28 dB, while RBPN/4-P exceeds them by 0.33 dB and 0.93 dB.RBPN also reproduces sharper patterns and details in the qualitative SPMCS results.
- Benchmark comparisons: On Vimeo-90k, RBPN/6-PF exceeds VSR-DUF by 1.22 dB, 1.44 dB, and 2.54 dB for slow, medium, and fast motion, respectively.RBPN/3-P also exceeds prior methods using six extra frames, with a reported PSNR gain of 0.77.
A.1. Multiple scale factors
RBPN was evaluated at 2× and 8× scaling factors across Vimeo-90k, SPMCS-32, and Vid4, outperforming existing methods except for Vid4's 2× SSIM score.
- RBPN is superior to existing methods on all tested sets and scaling factors except Vid4 SSIM at 2×.The best score on that exception, from VSR-DUF, differs from RBPN by only 0.001.
A.2. Network size
The study evaluates RBPN variants with different network sizes while keeping most settings fixed. RBPN/6 achieves the best performance among the tested configurations.
- Network configurations: The original RBPN uses the same setup as in the main paper.Netsisr uses DBPN, while Netmisr, Netres, and NetD use ResNet.
- Network-size variants: RBPN-S uses three blocks for Netmisr, Netres, and NetD, whereas RBPN-L uses six-stage Netsisr.The remaining setup is unchanged between these variants.
- Results: RBPN/6 achieves the best performance among the reported network-size configurations.Its detailed performance is reported in the main paper.
A.3. Residual Learning
The residual-learning variant interpolates the target frame first and has RBPN predict only the residual image. Under the current hyperparameters, this modification does not improve RBPN.
- Residual-learning procedure: Residual learning first interpolates the target frame with Bicubic interpolation.RBPN then predicts the residual image rather than the complete super-resolved image.
- Residual-learning procedure: The interpolated frame and predicted residual are combined to produce the final super-resolved image.
- Results: The current hyperparameters show that residual learning does not improve RBPN.The comparison is reported in Table 12.
A.4. Complexity Analysis
The complexity analysis reports computational time, parameter count, and FLOPS for RBPN and competing methods in Table 13.
- Complexity metrics: Table 13 reports computational time for RBPN and competing methods.
- Complexity metrics: Table 13 reports the number of parameters for RBPN and competing methods.
- Complexity metrics: Table 13 reports FLOPS for RBPN and competing methods.
A.5. Additional Qualitative Results
RBPN/6-PF produces sharper, more visually pleasing reconstructions that appear most similar to ground truth across qualitative examples. At 8× scaling, RBPN/6 achieves the best results among the compared methods.
- RBPN/6-PF produces reconstructions that appear most similar to ground truth, more pleasing, and sharper than other methods.
- The highlighted regions identify examples where RBPN/6-PF's qualitative improvements are particularly notable.
- At 8× scaling, RBPN/6 successfully generates the best results compared with DBPN and Bicubic.