Source-linked AI summary
Understanding Deformable Alignment in Video Super-Resolution
Kelvin C. K. Chan, Xintao Wang, Ke Yu, Chao Dong, Chen Change Loy
TL;DR
The paper asks how deformable alignment relates to classic flow-based alignment and what explains its performance. It decomposes deformable convolution into spatial warping plus convolution, studies offset diversity, and introduces optical-flow-guided offset training. The analysis links both alignments, finds benefits from diverse offsets, and reports more stable deformable alignment with up to 1.7 dB improvement from the proposed loss.
Problem
The relationship and improvement mechanism between explicit flow-based and implicit deformable alignment remain insufficiently understood.
Method
The paper decomposes deformable convolution into spatial warpings followed by convolution, analyzes offset diversity, and uses optical flow in an offset-fidelity loss.
Results
Up to 1.7 dB improvement is observed with the offset-fidelity loss, which stabilizes deformable-alignment training and avoids offset overflow.
Takeaways & Limitations
Offset diversity improves alignment and SR performance, while the formulation enables more flexible offset diversity in flow-based alignment.
Takeaways & Limitations
Offset diversity generally improves output quality but reaches a performance plateau, so its number should balance performance against computational efficiency.
Abstract
from arXiv · showhide
Deformable convolution, originally proposed for the adaptation to geometric variations of objects, has recently shown compelling performance in aligning multiple frames and is increasingly adopted for video super-resolution. Despite its remarkable performance, its underlying mechanism for alignment remains unclear. In this study, we carefully investigate the relation between deformable alignment and the classic flow-based alignment. We show that deformable convolution can be decomposed into a combination of spatial warping and convolution. This decomposition reveals the commonality of deformable alignment and flow-based alignment in formulation, but with a key difference in their offset diversity. We further demonstrate through experiments that the increased diversity in deformable alignment yields better-aligned features, and hence significantly improves the quality of video super-resolution output. Based on our observations, we propose an offset-fidelity loss that guides the offset learning with optical flow. Experiments show that our loss successfully avoids the overflow of offsets and alleviates the instability problem of deformable alignment. Aside from the contributions to deformable alignment, our formulation inspires a more flexible approach to introduce offset diversity to flow-based alignment, improving its performance.
1 Introduction
Video super-resolution must align motion-displaced neighboring frames to recover complementary details. This paper relates flow-based and deformable alignment, identifies offset diversity as important, and proposes optical-flow-guided training to stabilize deformable alignment.
- Video SR uses complementary details from adjacent frames, whose camera- and object-induced misalignment makes effective inter-frame correspondence essential.
- Flow-based and deformable alignment are formally related, despite their traditionally separate development and different apparent operations.
- Deformable alignment learns offsets with optical-flow-like patterns, while offset diversity provides complementary samples that alleviate occlusions and reduce warping errors.
- An offset-fidelity loss uses optical flow to guide deformable-convolution offsets while preserving their diversity and stabilizing training.
- The decomposition also enables a more flexible way to increase offset diversity in flow-based alignment beyond deformable convolution’s kernel-size constraint.
- Up to 1.7 dB improvement is observed with the proposed offset-fidelity loss.
2 Related Work
Video SR motion-compensation methods mainly use explicit optical-flow alignment or implicit deformable alignment. The paper connects these approaches through a decomposition of deformable convolution and uses that connection to motivate offset-fidelity training.
- Video SR differs from single-image SR by requiring alignment of multiple frames to construct accurate correspondences.
- Explicit methods estimate optical flow and warp neighboring frames, but inaccurate flow can cause failures and two-stage processing can be time-consuming.
- Implicit methods such as deformable convolution learn additional offsets that provide spatial flexibility for motion compensation.
- A 3×3 deformable convolution can be viewed as nine spatial warpings followed by one 3D convolution.
3 Unifying Deformable and Flow-Based Alignments
The paper unifies deformable and flow-based alignment by decomposing deformable convolution into spatial warping followed by convolution, revealing offset diversity as their key difference. Experiments show that diverse offsets improve alignment and restoration, while the proposed optical-flow-guided loss stabilizes deformable alignment.
- Relation between Deformable Alignment and Flow-Based Alignment: Deformable convolution can be decomposed into separate spatial warpings followed by a standard convolution, connecting it formally to flow-based alignment.Replacing the kernel-size-dependent offset count n^2 with N generalizes the formulation and allows arbitrary offset diversity.
- Relation between Deformable Alignment and Flow-Based Alignment: Flow-based and deformable alignments share the same formulation, but deformable alignment uses multiple offsets rather than a single flow offset.The special case n=1 corresponds to spatial warping followed by a 1×1 convolution, matching flow-based alignment.
- Discussion: The paper identifies offset diversity as the source of deformable alignment’s improvement over flow-based alignment and verifies this hypothesis experimentally.The claimed benefit spans both alignment accuracy and video super-resolution performance.
- Discussion: Increasing offset diversity generally improves output quality, but performance plateaus at larger offset counts, creating an efficiency trade-off.The paper recommends choosing a suitable number of offsets rather than increasing them indefinitely.
- Discussion: The decomposition also enables a more flexible flow-based alignment design because its number of offsets need not equal the square of the convolution-kernel size.Experiments show that increasing offset diversity improves the SR performance of a flow-based network.
- Offset-Fidelity Loss: An offset-fidelity loss uses optical flow to guide deformable-convolution offsets while preserving their diversity and stabilizing training.The loss is designed to avoid offset overflow, which can severely degrade large deformable-alignment models.
4 Analysis
The analysis finds that deformable alignment closely matches flow-based alignment when using one offset, while additional diverse offsets improve feature alignment and restoration. The proposed decomposition and offset-fidelity loss further improve flexibility and training stability.
- Deformable Alignment vs. Optical Flow: With G=N=1, learned offsets closely resemble optical flow, and over 80% differ from flow by less than one pixel.This setting is treated as an approximation to flow-based alignment.
- Decomposition Equivalence: DCN and the proposed decomposition achieve similar PSNR on REDS4, supporting decomposition into spatial warping and convolution.The decomposition also permits a flexible number of offsets.
- Offset Diversity: Learned offsets can differ in direction from optical flow, especially where optical flow performs poorly, while retaining similar overall motion patterns.Offset diversity is larger near image boundaries and other difficult alignment regions.
- Offset Diversity: Multiple offsets provide reciprocal warped features that reduce single-offset warping errors, improve coherence, and preserve more details during fusion.Single offsets are particularly vulnerable to unseen regions and inaccurate motion estimation.
- Increasing Offset Diversity: As offsets increase from 1 to 5, PSNR rises rapidly and then saturates near 30.23 dB, making further increases less computationally efficient.The proposed formulation allows arbitrary offset counts, unlike deformable alignment tied to the square of kernel size.
- Offset Diversity: Offset diversity correlates positively with PSNR (Pearson correlation coefficient 0.9418), and improves TDAN and flow-based models by up to 0.23 dB.The flow-based network alone gains 0.18 dB.
- Offset-fidelity Loss: The offset-fidelity loss stabilizes deformable-alignment training and adds up to 1.73 dB by penalizing offsets that deviate from optical flow.Without stabilization, offset overflow can produce a zero feature map after deformable alignment.
5 Conclusion
The study connects deformable and flow-based alignment, using this relation to motivate offset diversity and an optical-flow-guided loss for more stable deformable alignment.
- Figure 10 shows instability without the loss on REDS after about 300K iterations, while the loss reaches lower training loss on Vimeo-90K.
- The formulation reveals intrinsic connections between deformable alignment and flow-based alignment.
- Offset diversity improves flow-based alignment by relaxing deformable convolution’s constraint on the number of offsets.
- The offset-fidelity loss mitigates instability during deformable-alignment training.
A.1 Experimental Settings
Experiments use REDS and Vimeo-90K, with a two-stage training setup that follows EDVR-style optimization and augmentation choices.
- The experiments use REDS and Vimeo-90K for training and evaluation.REDS4 is used as the REDS test set, while Vimeo-90K-T is used for testing.
- Training uses Adam, cosine annealing with restarts, 6×10^5 iterations, an initial learning rate of 4×10^-4, batch size 16, and 64×64 low-resolution patches.
- Training proceeds in two stages, pretraining without TSA before initializing the full model with TSA.The pretrained weights are frozen for the first 50k iterations.
A.2 Learned Offsets in Deformable Alignment
Learned deformable offsets remain broadly similar to optical flow while gaining diversity, and this diversity improves alignment, though excessive offsets become inefficient.
- Learned offsets are highly similar to PWC-Net optical flow, although video-SR training produces a non-negligible difference.
- Increasing offsets from 1 to 25 preserves the overall optical-flow-like shape while producing complementary directions.
- Multiple diverse offsets reduce warping error by aggregating information from different locations, leading to better-aligned features.
- Increasing offset diversity indefinitely can reduce efficiency without significant performance gains.A significant improvement appears from flow-based alignment to deformable alignment with N=3, whereas increasing N from 3 to 25 adds little visual improvement.
A.3 Image Warping vs. Feature Warping
Feature-level warping preserves more detail than image-level warping, which loses information through interpolation of decimal flows.
- Image-level warping can lose high-frequency details through interpolation, whereas feature-level alignment can alleviate this effect.
- Feature warping recovers more details than image warping in the reported examples.Wall patterns recovered with feature warping are absent after image warping.
- 0.84 dB improvement is reported for feature warping over image warping on REDS4.
A.4 Deformable Groups
Deformable groups increase offset diversity by learning distinct offsets across channel groups, improving PSNR but eventually showing saturation and parameter inefficiency.
- Dividing input channels into G groups lets the model learn different offsets for each group, yielding G×N offset sets.
- PSNR improves with G by up to 0.19 dB, consistent with a positive relationship between performance and offset diversity.
- For G=1, increasing N from 1 to 5 raises PSNR by 0.25 dB with only 84K additional parameters.
- At G=64 and N=25, 1600 learned offsets produce about 30.3 dB, indicating performance saturation at high diversity.
- With N=25, G=32 and G=64 achieve nearly identical PSNR, although G=64 uses 4M more parameters.
A.5 Modulation Masks
Modulation masks downweight offsets that differ substantially from optical flow, while offset-fidelity training prevents overflow and preserves neighboring-frame information for video super-resolution.
- Modulated deformable convolution learns masks that multiply independently warped features before fusion.
- Offsets with larger differences from optical flow receive smaller mean masks and therefore contribute less during fusion.
- When many offsets are used, a substantial fraction of masks approaches zero, consistent with redundant offsets and PSNR saturation as G or N increases.
- Offset-fidelity loss constrains offsets near optical flow while allowing diversity within a threshold, enabling interpretable offsets and neighboring-frame aggregation.
- Without offset-fidelity loss, offset overflow produces zero aligned features and effectively reduces EDVR to single-image super-resolution.