Source-linked AI summary
Investigating Tradeoffs in Real-World Video Super-Resolution
Kelvin C. K. Chan, Shangchen Zhou, Xiangyu Xu, Chen Change Loy
TL;DR
Real-world VSR must cope with diverse degradations that create artifact and computational challenges. The paper introduces input cleaning, stochastic degradation, longer-sequence training analysis, and the VideoLQ benchmark; it reports improved quality and efficiency, including up to 40% shorter training time. The authors also identify hardware-dependent speedup as a limitation.
Problem
Diverse unknown degradations make real-world VSR difficult to generalize, while propagation can amplify artifacts and training requires greater computational resources.
Method
The paper combines an image cleaning module before propagation, a stochastic degradation scheme, training comparisons across batch sizes and sequence lengths, and the VideoLQ dataset.
Results
Up to 40% reduction of training time is achieved without sacrificing output quality, while longer sequences than larger batches improve stability and RealBasicVSR improves output quality.
Takeaways & Limitations
Image cleaning and stochastic degradation provide practical solutions to real-world VSR artifact and training-efficiency tradeoffs, while VideoLQ supports common benchmarking.
Takeaways & Limitations
Speedup from the stochastic degradation scheme varies with hardware because different hardware produces different bottleneck levels.
Abstract
from arXiv · showhide
The diversity and complexity of degradations in real-world video super-resolution (VSR) pose non-trivial challenges in inference and training. First, while long-term propagation leads to improved performance in cases of mild degradations, severe in-the-wild degradations could be exaggerated through propagation, impairing output quality. To balance the tradeoff between detail synthesis and artifact suppression, we found an image pre-cleaning stage indispensable to reduce noises and artifacts prior to propagation. Equipped with a carefully designed cleaning module, our RealBasicVSR outperforms existing methods in both quality and efficiency. Second, real-world VSR models are often trained with diverse degradations to improve generalizability, requiring increased batch size to produce a stable gradient. Inevitably, the increased computational burden results in various problems, including 1) speed-performance tradeoff and 2) batch-length tradeoff. To alleviate the first tradeoff, we propose a stochastic degradation scheme that reduces up to 40\% of training time without sacrificing performance. We then analyze different training settings and suggest that employing longer sequences rather than larger batches during training allows more effective uses of temporal information, leading to more stable performance during inference. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences containing rich textures and patterns. Our dataset can serve as a common ground for benchmarking. Code, models, and the dataset will be made publicly available.
1. Introduction
Real-world VSR must handle diverse unknown degradations that complicate both inference and training. The paper addresses these challenges with artifact cleaning, training-efficiency analysis, and the VideoLQ benchmark.
- 1. Introduction: Unknown and diverse real-world degradations challenge VSR inference, training settings, and benchmark design.
- 1. Introduction: An image cleaning module before propagation removes input degradations, helping RealBasicVSR avoid artifact amplification while maintaining simplicity.A dynamic refinement scheme repeatedly applies cleaning to balance smoothness and detailedness according to a threshold or user preference.
- 1. Introduction: A stochastic degradation scheme reduces the I/O bottleneck without sacrificing output quality.
- 1. Introduction: Up to 40% reduction of training time is achieved compared with the conventional training scheme.
- 1. Introduction: Training with longer sequences rather than larger batches more effectively uses long-term information and improves inference stability.
- 1. Introduction: VideoLQ provides diverse real-world low-quality videos spanning contents, resolutions, and degradations for common benchmarking.
2. Related Work
Prior VSR methods commonly rely on synthetic or camera-specific degradations, limiting generalization to unknown real-world conditions. The paper highlights artifact amplification during propagation and motivates image cleaning for real-world VSR.
- 2. Related Work: Existing VSR methods trained with pre-defined degradations deteriorate when handling unknown degradations in reality.
- 2. Related Work: Long-term propagation improves non-blind VSR performance but can produce undesirable artifacts in real-world scenarios.L denotes sequence length in Figure 2.
- 2. Related Work: Directly extending image-level diverse-degradation augmentation to real-world VSR is not feasible because the network tends to amplify noise and artifacts.The paper instead investigates image cleaning to remedy this problem.
3. Tradeoff in Inference
Real-world VSR faces a tradeoff: long-term propagation can synthesize details but also amplify artifacts, motivating image pre-cleaning before BasicVSR propagation. The cleaning design and dynamic refinement suppress degradations while preserving details, and stochastic degradation improves training efficiency.
- Tradeoff in Inference: Long-term propagation enhances details under mild degradations but can amplify noises and artifacts in real-world VSR.This creates a tradeoff between detail synthesis and artifact suppression.
- Input Pre-Cleaning: RealBasicVSR independently cleans each input image before passing the clean sequence to BasicVSR for super-resolution.The cleaning module and VSR network are trained end-to-end.
- Cleaning Module Design: The image cleaning loss is essential: removing it causes the network to exaggerate noises and artifacts and distort original content.The authors find that a simple pixelwise loss suffices, although perceptual and adversarial losses can also be adopted.
- Cleaning Module Design: A recurrent cleaning alternative fails to remove unwanted degradations, whereas the proposed image-based design produces smoother outputs and preserves more image content.This supports assigning artifact suppression to an image-based cleaning module rather than a video-based recurrent structure.
- Stochastic Degradation: Stochastic degradation loads fewer frames per iteration and varies degradations temporally, reducing training time by 40% without sacrificing performance.The scheme improves performance compared with directly flipping the sequence while reducing the CPU workload and I/O bottleneck.
- Dynamic Refinement: Dynamic refinement halts cleaning automatically, avoiding the over-smoothing and detail loss observed with repeated fixed passes.The scheme yields fewer artifacts while preserving necessary details and obtains lower NIQE than fixed iterations.
4. Tradeoff in Training
Real-world VSR training faces a speed–performance tradeoff from I/O bottlenecks and a batch-size–sequence-length tradeoff under fixed computation. The paper proposes stochastic degradation for faster training and favors longer sequences for more stable inference.
- Training Speed vs. Performance: Stochastic degradation reduces CPU workload and training time by up to 40% without sacrificing performance.The scheme loads half the frames, temporally flips sequences, and models frame degradations as a random walk.
- Training Speed vs. Performance: Directly flipping sequences alone produces similar or degraded qualitative performance, whereas stochastic degradation improves robustness to degradation variation.The paper attributes the improvement to modeling degradations across frames rather than relying on temporal flipping alone.
- Batch Size vs. Sequence Length: With fixed computation, larger batches provide more degradation and scene diversity, while longer sequences expose the network to more long-term information.The tradeoff arises because increasing batch size decreases sequence length.
- Batch Size vs. Sequence Length: Training with B=48 and L=10 produces severe color artifacts and distorted details, which diminish as L increases to 20 and are further eliminated at L=30.The comparison suggests that short training sequences do not adapt well to longer sequences used during inference.
- Batch Size vs. Sequence Length: Under a fixed computational constraint, longer sequences are preferable to larger batches for using long-term information and maintaining stable performance.Reducing sequence length from 30 to 10 also produces color artifacts and blur when batch size is fixed at 16.
5. VideoLQ Dataset and Benchmark
VideoLQ is introduced as a diverse real-world VSR benchmark, while RealBasicVSR is evaluated against image and video methods on restoration quality and efficiency. The dataset and qualitative results emphasize varied degradations, rich visual content, and long-term temporal information.
- VideoLQ Dataset and Benchmark: VideoLQ addresses the limited diversity of existing benchmarks by covering a wide range of degradations, content, and resolutions.Existing datasets rely mainly on synthetic degradations or data captured by a single iPhone camera.
- VideoLQ Dataset and Benchmark: VideoLQ videos are collected from Flickr and YouTube under Creative Commons licenses, with 100-frame sequences containing no scene changes.Sequences are selected for varied resolutions, contents, textures, and text so long-term propagation can be assessed.
- Experimental Comparison: Experiments compare RealBasicVSR with seven state-of-the-art image and video methods using diverse image and video compression degradations during training.The training degradation pipeline includes blur, resizing, noise, JPEG compression, and randomly selected video codecs and bitrates.
- Qualitative Results: RealBasicVSR aggregates long-term information through propagation and restores fine details, including the word “hobby” in later frames.The image cleaning module is reported as enabling effective propagation and improved visual quality on VideoLQ examples.
- Comparison to State of the Arts: RealBasicVSR achieves better performance on all reported metrics with faster speed, including 17× faster speed than RealVSR.Compared with Real-ESRGAN, it also delivers superior performance with lower complexity and faster speed.
6. Discussion
The discussion identifies artifact amplification and computational cost as central real-world VSR challenges, then presents cleaning, stochastic degradation, and VideoLQ as corresponding solutions and resources.
- Discussion: Long-term information benefits real-world VSR but is accompanied by tradeoffs caused by diverse degradations and increased computational costs.The paper frames these issues as challenges requiring separate solutions for inference and training.
- Discussion: The paper addresses these challenges with an image cleaning module, a stochastic degradation scheme, and the VideoLQ dataset.The authors describe these components as easy-to-implement solutions and a foundation for future real-world VSR work.
A. Architecture and Experimental Settings
RealBasicVSR combines a cleaning module with a BasicVSR network and uses staged losses for fidelity and visual quality. Training includes diverse image and video degradations and is implemented on eight V100 GPUs.
- Losses and Training Stages: Charbonnier losses supervise output fidelity and image cleaning, while perceptual and adversarial losses target visual quality.The paper uses these four loss components across pretraining and finetuning stages.
- Losses and Training Stages: The second training stage adds perceptual and adversarial losses while keeping the cleaning-module weights fixed.The reported loss weights are λ_per=1 and λ_adv=5×10^-2.
- Training Degradations: Training degradations combine random blur, resizing, noise, JPEG compression, and randomly selected video codecs with uniformly sampled bitrates.Video compression is added after JPEG compression and models temporally and spatially varying degradations.
- Implementation: The models are implemented in PyTorch and trained using eight NVIDIA Tesla V100 GPUs.The paper states that code will be made publicly available.
B. Discussion of Baselines
The baselines combine degradation-removal designs with or without long-term temporal information. RealBasicVSR uses long-term information with designated designs and restores more details than the compared methods.
- The comparison includes four image models and three video models representing methods with promising super-resolution performance.The video models are BasicVSR++, RealVSR, and DBVSR.
- Existing degradation-removal methods remain inferior at recovering details beyond the input image or local neighbors because they do not exploit long-term video information.
- RealBasicVSR uses designated designs to exploit long-term information in real-world VSR and restores more details than the comparison methods.The qualitative improvement is shown in Figures 12 and 13.
C. Dynamic Refinement
Dynamic refinement controls repeated cleaning to remove remaining artifacts without producing over-smoothed outputs. Compared with fixed cleaning counts, it stops refinement adaptively.
- C. Dynamic Refinement: Applying cleaning once leaves unpleasant artifacts, whereas applying it five times produces unnaturally flat outputs through over-cleaning.
- C. Dynamic Refinement: The dynamic refinement scheme automatically stops cleaning to remove excessive artifacts while avoiding over-smoothing.Its qualitative effect is illustrated in Figure 14.
- C. Dynamic Refinement: The qualitative comparisons show RealBasicVSR restoring more details by effectively using long-term information.