Source-linked AI summary
Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring
Lingshun Kong, Jiangxin Dong, Mingqiang Li, Jianjun Ge, Jinshan Pan
TL;DR
Image deblurring requires restoring clear images while balancing global-context modeling against the high cost of Transformer attention and the frequency information needed for restoration. The paper introduces frequency-domain attention and a gated frequency-domain FFN within an asymmetric encoder-decoder network, and reports favorable accuracy and efficiency against state-of-the-art methods.
Problem
Existing deblurring models rely heavily on local convolutions, while Transformer attention has quadratic complexity and efficient alternatives can lose spatial information important for restoration.
Method
The method uses FSAS for frequency-domain element-wise attention, DFFN for gated low- and high-frequency selection, and an encoder-decoder architecture applying FSAS only in the decoder.
Results
The proposed method performs favorably against state-of-the-art approaches in image-deblurring accuracy and efficiency.
Takeaways & Limitations
Frequency-domain Transformer components can support high-quality image deblurring while reducing the complexity of scaled dot-product attention.
Abstract
from arXiv · showhide
We present an effective and efficient method that explores the properties of Transformers in the frequency domain for high-quality image deblurring. Our method is motivated by the convolution theorem that the correlation or convolution of two signals in the spatial domain is equivalent to an element-wise product of them in the frequency domain. This inspires us to develop an efficient frequency domain-based self-attention solver (FSAS) to estimate the scaled dot-product attention by an element-wise product operation instead of the matrix multiplication in the spatial domain. In addition, we note that simply using the naive feed-forward network (FFN) in Transformers does not generate good deblurred results. To overcome this problem, we propose a simple yet effective discriminative frequency domain-based FFN (DFFN), where we introduce a gated mechanism in the FFN based on the Joint Photographic Experts Group (JPEG) compression algorithm to discriminatively determine which low- and high-frequency information of the features should be preserved for latent clear image restoration. We formulate the proposed FSAS and DFFN into an asymmetrical network based on an encoder and decoder architecture, where the FSAS is only used in the decoder module for better image deblurring. Experimental results show that the proposed method performs favorably against the state-of-the-art approaches. Code will be available at \url{https://github.com/kkkls/FFTformer}.
1. Introduction
Image deblurring seeks to restore high-quality images from blurred inputs, but convolutional and Transformer-based approaches face limitations in modeling spatially variant content or efficiently capturing global context. The paper addresses these issues with frequency-domain attention, frequency-selective feed-forward processing, and an asymmetric encoder-decoder design.
- Motivation: Convolutional deblurring models primarily use local, spatially invariant operations, and increasing model capacity does not always improve performance.Multi-scale, multi-stage, adversarial, and physics-inspired architectures are among the existing designs.
- Motivation: Transformer attention captures global context but has quadratic space and time complexity in the number of tokens.Reducing token count can weaken long-range modeling and produce artifacts in high-resolution images.
- Motivation: Downsampling or feature-depth attention reduces computation but can lose spatial information important for image deblurring.The paper identifies this trade-off as a limitation of existing efficient attention strategies.
- Proposed approach: FSAS estimates scaled dot-product attention through frequency-domain element-wise products rather than spatial-domain matrix multiplication.Using the convolution theorem, the method reduces per-feature-channel complexity to O(N) space and O(N log N) time, where N is the number of pixels.
- Proposed approach: DFFN uses a JPEG-motivated gated mechanism to select which low- and high-frequency feature information to preserve for clear-image restoration.The method is designed because a naive FFN does not generate good deblurred results.
- Proposed approach: The proposed FSAS and DFFN are integrated into an asymmetric encoder-decoder network that applies FSAS only in the decoder.The design reflects the observation that shallow encoder features usually contain blur effects, whereas deeper features are clearer.
2. Related Work
Prior image-deblurring research largely developed deep CNN architectures and increasingly efficient Transformer variants. The paper differs by using frequency-domain processing to avoid complex matrix multiplication while retaining spatial information relevant to restoration.
- CNN-based methods: Deep CNN deblurring methods use designs including multi-scale, recurrent, multi-patch, cross-stage fusion, and multi-input architectures.These approaches estimate clear images through increasingly structured spatial processing.
- Transformer-based methods: Transformer-based deblurring methods improve global-context modeling but include feature-depth, strip-token, and UNet-based strategies with different spatial-information trade-offs.Feature-depth attention explores channel information, while the cited discussion notes that spatial information may remain insufficiently explored.
- Frequency-domain approach: The proposed method explores the frequency domain to avoid complex matrix multiplication for scaled dot-product attention.This distinguishes it from prior efficient Transformer approaches discussed in the related work.
3. Proposed Method
The method combines frequency-domain self-attention and a discriminative frequency-domain FFN within an asymmetric encoder-decoder for image deblurring. FSAS replaces spatial attention matrix multiplication with frequency-domain element-wise products, while DFFN selectively preserves useful frequency information.
- Frequency domain-based self-attention solver: The attention pipeline begins by linearly transforming input features into Fq, Fk, and Fv, unfolding image patches, and reshaping them into query, key, and value matrices.N denotes the number of extracted patches, while R ensures the matrices have shape RN×(CHpWp).
- Frequency domain-based self-attention solver: Scaled dot-product attention has O(N^2) space and O(N^2C) time complexity, making high-resolution image processing expensive.Downsampling or extracting fewer patches reduces this cost but loses information and limits detail modeling.
- Frequency domain-based self-attention solver: FSAS estimates attention by treating query-key correlations as convolutions and replacing spatial convolution with frequency-domain element-wise multiplication.The method obtains Fq, Fk, and Fv using 1×1 point-wise and 3×3 depthwise convolutions, then applies FFT to Fq and Fk.
- Asymmetric encoder-decoder network: The proposed network uses DFFN in the encoder and both FSAS and DFFN in the decoder for image deblurring.
- Discriminative frequency domain-based FFN: DFFN uses JPEG-inspired learnable quantization and a gated mechanism to adaptively preserve low- and high-frequency information useful for latent clear-image restoration.Its patch unfolding and folding operations follow the JPEG compression formulation, with G denoting the GEGLU function.
- Asymmetric encoder-decoder network: FSAS is restricted to the decoder because shallow encoder features usually contain blur effects that can distort patch-similarity estimation.The architecture therefore uses DFFN throughout the encoder-decoder design while applying FSAS only where features are typically clearer.
4. Experimental Results
The method is evaluated on GoPro, RealBlur, and HIDE benchmarks using established protocols, with quantitative and visual comparisons against state-of-the-art methods. It achieves strong restoration quality, clearer structures, and favorable parameter efficiency across the reported evaluations.
- Datasets and parameter settings: The method is evaluated on GoPro, HIDE, and RealBlur datasets using protocols from existing methods for fair comparisons.Training uses Adam with cosine annealing, 256 × 256 patches, batch size 16, and 8 × 8 patches for JPEG-based processing and self-attention.
- GoPro dataset: The proposed method achieves the highest PSNR and SSIM values on the GoPro dataset.Its PSNR is at least 0.5dB higher than NAFNet while using one quarter of NAFNet’s parameters, and it has fewer parameters than Transformer-based competitors.
- GoPro dataset: Visual comparisons on GoPro show clearer characters, boundaries, and structures than the evaluated methods.Several CNN-based results retain significant blur, while Transformer-based results do not recover some main structures well.
- RealBlur dataset: Visual comparisons on RealBlur show clearer characters and finer structural details for the proposed method.The compared results do not recover characters or structural details well.
- HIDE dataset: On HIDE, the proposed method produces better deblurred-image quality than the evaluated methods using models trained on GoPro.The result suggests better generalization when testing on a dataset containing mainly humans without training on HIDE.
- HIDE dataset: On HIDE visual comparisons, the proposed method generates clearer faces and clothing zippers than the evaluated methods.The evaluated methods do not recover humans well in the shown examples.
5. Analysis and Discussion
The analysis evaluates FSAS, DFFN, and the asymmetric encoder-decoder design through complexity comparisons, ablations, and visual results. These components improve efficiency and deblurred-image quality, while placing FSAS only in the decoder benefits restoration.
- Visual analysis: Visual results show clearer HIDE images with the proposed method, while comparison outputs in (c)-(g) retain significant blur effects.Additional visualizations report better window recovery with DFFN and clearer images using FSAS.
- Effect of FSAS: FSAS requires O(N) space and O(NC log N) time, versus O(N^2) and O(N^2C) for original scaled dot-product attention.The comparison uses C for the number of features.
- Effect of FSAS: FSAS uses less GPU memory and runs more efficiently than the compared window-based Transformer strategy on 1280 × 720 images.The comparison is reported in Table 4 using an NVIDIA GeForce RTX 3090 GPU.
- Effect of FSAS: Adding FSAS raises PSNR by 0.42dB over the baseline using only FFN, with visual comparisons showing improved blur removal and boundary recovery.The comparison is conducted on the GoPro dataset.
- Effect of DFFN: DFFN produces a 0.36dB higher PSNR than the original FFN baseline and generates better deblurred images in the ablation comparison.The DFFN is evaluated both alone against FFN and within the proposed FSAS-based model.
- Effect of the asymmetric encoder-decoder network: The asymmetric encoder-decoder design places FSAS only in the decoder because shallow encoder features usually contain blur effects that affect FSAS estimation.The design is evaluated against placing FSAS in both encoder and decoder modules.
6. Conclusion
The paper develops a frequency-domain Transformer for high-quality image deblurring. FSAS reduces attention complexity, DFFN selectively preserves frequency information, and an asymmetric encoder-decoder achieves favorable accuracy and efficiency against state-of-the-art methods.
- Conclusion: FSAS estimates scaled dot-product attention through frequency-domain element-wise multiplication, significantly reducing spatial and computational complexity.It replaces spatial-domain matrix multiplication.
- Conclusion: DFFN discriminatively determines which low- and high-frequency feature information should be preserved for latent clear image restoration.The method is formulated as a frequency-domain feed-forward network.
- Conclusion: The asymmetric encoder-decoder uses FSAS only in the decoder and performs favorably against state-of-the-art approaches in accuracy and efficiency.The proposed network is trained end-to-end.