Source-linked AI summary
Stripformer: Strip Transformer for Fast Image Deblurring
Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, Chia-Wen Lin
TL;DR
Dynamic-scene blur is directional, non-uniform, and region-specific, making single-image restoration difficult. Stripformer uses intra- and inter-strip attention to model blur orientations and magnitudes efficiently, and it performs state-of-the-art across three benchmarks without ImageNet-scale pre-training.
Problem
Single-image dynamic-scene deblurring must recover sharpness from globally and locally non-uniform blur using limited information.
Method
Stripformer combines local intra-strip and global inter-strip tokens with horizontal and vertical attention to model region-specific blur orientations and magnitudes.
Results
Stripformer achieves state-of-the-art performance on the GoPro, HIDE, and RealBlur benchmarks without ImageNet pre-training, while performing favorably in memory usage, model size, and inference time.
Takeaways & Limitations
The model captures diverse dynamic-scene blur while using less memory and computation than vanilla transformers and without relying on tremendous training data.
Takeaways & Limitations
Vanilla transformer pixel-wise attention can reach O(H^2W^2) space complexity, creating a memory-efficiency boundary for high-resolution processing.
Abstract
from arXiv · showhide
Images taken in dynamic scenes may contain unwanted motion blur, which significantly degrades visual quality. Such blur causes short- and long-range region-specific smoothing artifacts that are often directional and non-uniform, which is difficult to be removed. Inspired by the current success of transformers on computer vision and image processing tasks, we develop, Stripformer, a transformer-based architecture that constructs intra- and inter-strip tokens to reweight image features in the horizontal and vertical directions to catch blurred patterns with different orientations. It stacks interlaced intra-strip and inter-strip attention layers to reveal blur magnitudes. In addition to detecting region-specific blurred patterns of various orientations and magnitudes, Stripformer is also a token-efficient and parameter-efficient transformer model, demanding much less memory usage and computation cost than the vanilla transformer but works better without relying on tremendous training data. Experimental results show that Stripformer performs favorably against state-of-the-art models in dynamic scene deblurring.
1 Introduction
Dynamic-scene deblurring must recover sharp images from single inputs with globally and locally non-uniform, directional blur. Stripformer addresses this with orientation- and magnitude-aware intra/inter-strip attention designed for efficient transformer processing.
- Single-image dynamic-scene deblurring is difficult because blur is globally and locally non-uniform, directional, and constrained by limited information.
- Prior methods often assume uniform blur or rely on specific priors for non-uniform blur, while associated non-convex optimization can require high computation time.
- Horizontal and vertical intra-strip and inter-strip attention capture blur orientations, while stacked interlaced layers reveal blur magnitudes through successive local-feature extraction.
- Transformers can model blurred patterns with different magnitudes and orientations, but vanilla attention has large memory requirements and some models depend on extensive training data.
- Stripformer combines local intra-strip tokens and global inter-strip tokens to reassemble attended blur features and address the efficiency trade-off between pure and hybrid transformers.
- Stripformer uses fewer tokens than vanilla attention, reducing memory and computation costs while performing favorably against state-of-the-art deblurring models without tremendous training data.
2 Related Work
Related work spans recurrent CNN deblurring, attention-based context extraction, and efficient transformer designs. Stripformer extends strip-based representations to capture local and global blur information with lower complexity.
- CNN deblurring methods commonly use recurrent multi-scale, multi-patch, or multi-temporal architectures to restore sharp images.
- Strip pooling and criss-cross attention extract horizontal, vertical, or global dependencies, motivating Stripformer’s intra- and inter-strip attention for blurred-pattern modeling.
- Global self-attention can require O(H^2W^2) space, while patch-token transformers may require many parameters and large pre-training datasets.
- Sliding-window transformers reduce memory use through local attention but do not provide high-resolution global attention, which dynamic-scene blur may require.
- The architecture combines shallow convolution embedding with intra- and inter-strip attention blocks for image deblurring.
- Stripformer uses O(HW(H+W)) and O(H^2+W^2) space complexity for intra- and inter-strip attention, respectively, versus O(H^2W^2) for vanilla attention.
3 Proposed Method
Stripformer uses horizontal and vertical intra-strip and inter-strip attention to capture region-specific blur orientations and magnitudes. Its strip-based design reduces attention complexity while combining attended features through an encoder-decoder pipeline and contrastive training.
- Stripformer Overview: Stripformer uses horizontal and vertical strip-wise features to represent blur patterns with different orientations and magnitudes.Intra-strip tokens capture local pixel-wise features, while inter-strip tokens capture global region-wise information.
- Network Architecture: The model interlaces Intra-SA and Inter-SA blocks across feature scales to capture multi-range blur patterns.The architecture downsamples features, applies attention blocks on smaller scales, and upsamples through transposed convolution with encoder-feature concatenation.
- Intra-SA Block: Intra-SA processes parallel horizontal and vertical branches, concatenates their attended features, and applies convolution, residual, MLP, and conditional positional encoding operations.The input is normalized and projected into horizontal and vertical features before attention.
- Inter-SA Block: Inter-SA treats strip features as tokens, applying horizontal and vertical strip-wise attention with total space complexity O(W^2 + H^2).The horizontal and vertical outputs are concatenated and reshaped into three-dimensional feature tensors.
- Efficiency: Stripformer reduces attention complexity from O(H^2W^2) for a vanilla transformer to O(HW(H + W)).The reduction supports lower memory usage while the stacked attention blocks model blur orientations and magnitudes.
- Loss Function: The deblurring objective combines Charbonnier, edge, and contrastive losses, with contrastive learning bringing outputs closer to sharp targets and away from blurred inputs.The total loss is L = L_char + λ_1 L_edge + λ_2 L_con.
4 Experiments
Stripformer is evaluated on synthetic and real-world deblurring datasets using quantitative, qualitative, and ablation analyses. It performs favorably against state-of-the-art methods while combining region-specific blur modeling with efficient attention.
- Quantitative Analysis: Stripformer achieves state-of-the-art deblurring performance on the GoPro, HIDE, and RealBlur datasets.The GoPro and HIDE evaluations use the GoPro training set, while RealBlur uses its own training set.
- Qualitative Analysis: Qualitative comparisons show sharper restoration on textured regions, facial expressions, low-light motion blur, and real images without ground truth.The comparisons cover GoPro, HIDE, RealBlur, and RWBI datasets.
- Component Analysis: Combining intra-strip and inter-strip attention improves performance over either block alone, while CPE and contrastive loss provide further gains.The ablation attributes the combined benefit to pixel-wise and region-wise dependencies and reports that CPE supports arbitrary input sizes better than fixed learnable positional encoding.
- Attention Mechanism Analysis: Stripformer's strip-wise attention performs better in PSNR than the compared Swin, Twins, and CCNet attention mechanisms.The design targets both local and global blur information, unlike attention schemes with more restricted or subsampled context.
5 Conclusions
Stripformer is a token- and parameter-efficient transformer for dynamic-scene deblurring, targeting region-specific blur with diverse orientations and magnitudes. It achieves state-of-the-art performance on three benchmarks without ImageNet pre-training while remaining competitive in efficiency.
- Stripformer targets dynamic-scene blur artifacts with diverse orientations and magnitudes using intra- and inter-strip attention.These attentions reduce memory and computation costs relative to a vanilla transformer.
- Stripformer achieves state-of-the-art performance on the GoPro, HIDE, and Realblur benchmarks.
- Stripformer achieves these results without using a large dataset such as ImageNet for pre-training.
- Stripformer remains competitive in memory usage, model size, and inference time.