PaperlayerView exact revision on arXiv

Source-linked AI summary

Intriguing Findings of Frequency Selection for Image Deblurring

Xintian Mao, Yiming Liu, Fengze Liu, Qingli Li, Wei Shen, Yan Wang

arXiv:2111.11745v2cs.CV

TL;DR

The paper addresses the overlooked role of blur kernels in end-to-end, pixel-level image deblurring. It uses frequency selection inside a Res FFT-ReLU Block to combine kernel-level and pixel-level representations, achieving 33.85 dB PSNR with NAFNet on GoPro while preserving efficiency. Experiments across datasets and architectures support the block’s plug-and-play use.

  • Problem

    End-to-end deblurring networks learn blurry-to-sharp differences from pixels but overlook blur-kernel information.

  • Method

    The Res FFT-ReLU Block inserts Fourier transform, 1×1 convolutions, ReLU, and inverse Fourier transform into ResBlocks to select frequencies and combine dual-domain representations.

  • Results

    33.85 dB PSNR is achieved by FNAFNet on GoPro after plugging in the proposed block, with evaluations across three datasets and multiple architectures.

  • Takeaways & Limitations

    Frequency selection provides faithful blur-pattern information, and the proposed block improves backbone architectures with few added parameters and low computational complexity.

  • Takeaways & Limitations

    The analysis assumes a spatially invariant blur kernel in its simplified frequency-domain formulation.

Abstract

from arXiv · show

Blur was naturally analyzed in the frequency domain, by estimating the latent sharp image and the blur kernel given a blurry image. Recent progress on image deblurring always designs end-to-end architectures and aims at learning the difference between blurry and sharp image pairs from pixel-level, which inevitably overlooks the importance of blur kernels. This paper reveals an intriguing phenomenon that simply applying ReLU operation on the frequency domain of a blur image followed by inverse Fourier transform, i.e., frequency selection, provides faithful information about the blur pattern (e.g., the blur direction and blur level, implicitly shows the kernel pattern). Based on this observation, we attempt to leverage kernel-level information for image deblurring networks by inserting Fourier transform, ReLU operation, and inverse Fourier transform to the standard ResBlock. 1x1 convolution is further added to let the network modulate flexible thresholds for frequency selection. We term our newly built block as Res FFT-ReLU Block, which takes advantages of both kernel-level and pixel-level features via learning frequency-spatial dual-domain representations. Extensive experiments are conducted to acquire a thorough analysis on the insights of the method. Moreover, after plugging the proposed block into NAFNet, we can achieve 33.85 dB in PSNR on GoPro dataset. Our method noticeably improves backbone architectures without introducing many parameters, while maintaining low computational complexity. Code is available at https://github.com/DeepMed-Lab/DeepRFT-AAAI2023.

Introduction

Image deblurring must recover sharp images despite blur that harms visual quality and downstream vision tasks. The paper argues that frequency selection can expose blur-kernel information and introduces a lightweight Res FFT-ReLU Block that improves deblurring backbones.

  • Motivation: Image deblurring recovers sharp images from blur caused by camera shake, object movement, and out-of-focus optics.Blurry images reduce visual quality and hamper security, medical-imaging, and object-recognition tasks.
  • Motivation: Frequency-domain modeling expresses a blurry image as the Fourier-domain product of a sharp image and blur kernel, plus sensor noise.The formulation uses B(ω) = F(ω)G(ω) + N(ω), with Fourier transforms of the sharp image, PSF, blurry image, and noise.
  • Research gap: End-to-end deblurring networks learn blurry-to-sharp mappings with pixel-level architectures but overlook the importance of blur kernels.DeepDeblur pioneered this direction using a multi-scale CNN and modified residual blocks.
  • Proposed insight: Frequency selection via frequency-domain ReLU followed by inverse Fourier transform reveals blur direction, blur level, and implicit kernel patterns.The paper uses this observation to learn kernel-level information directly from blurry images.
  • Proposed method: The Res FFT-ReLU Block inserts Fourier transform, ReLU, inverse Fourier transform, and 1×1 convolution into a standard ResBlock.The added convolution enables flexible frequency-selection thresholds while combining kernel-level and pixel-level representations.
  • Results: 33.85 dB PSNR is achieved by FNAFNet on GoPro after plugging in the proposed block, while maintaining low computational complexity and few added parameters.FMIMO-UNet+ reaches 33.52 dB PSNR on the same dataset, and evaluations cover GoPro, HIDE, and RealBlur.

Related Works

Related work spans deep spatial-domain deblurring, efficient and global-context architectures, and Fourier-domain representations. The paper positions its contribution as a frequency-selection block rather than a new end-to-end architecture.

  • Deep Image Deblurring: Deep image-deblurring methods have achieved success, while earlier CNN work attempted to estimate spatially varying motion-blur kernels.The cited kernel-estimation approach is described as impractical for real-world use because blur characteristics are complex.
  • Deep Image Deblurring: Most deblurring networks operate CNNs in the spatial domain, while NAFNet emphasizes computational efficiency and reports 33.69 dB PSNR on GoPro.The paper instead studies frequency selection without designing a brand-new end-to-end architecture.
  • Global Context Modeling: Transformer-based restoration methods provide global context modeling, but their considerable computational complexity limits efficient image-restoration use.The cited SwinIR and Restormer models are evaluated on GoPro with reported per-image runtimes.
  • ResBlock-based Deblurring: ResBlock became a fundamental image-deblurring component after DeepDeblur, motivating subsequent modifications such as content-aware and channel-attention blocks.The paper builds on this residual-block lineage.
  • Fourier Transform Applications: Fourier-domain methods have been applied to tasks including segmentation, classification, and image inpainting by manipulating spectra or learning long-term dependencies.The paper applies Fourier operations to deblurring through frequency selection.

Method

The method uses frequency-domain ReLU selection to expose blur-pattern information, then combines it with spatial residual features in a Res FFT-ReLU Block. Learnable frequency-domain modulation and nonlinear processing provide flexible selection while preserving efficient residual computation.

  • Frequency Selection: Frequency selection applies ReLU to a blurry image’s Fourier representation and inverse-transforms it to separate a component containing blur-pattern information.ReLU is applied separately to real and imaginary parts; the resulting operation is written as F−1(σ(F(b))).
  • Frequency Selection: Inverse-transformed frequency selection reflects blur direction, blur level, and mixtures of blur, implicitly revealing the blur-kernel pattern.The observation is demonstrated on blurred images generated with motion kernels of different directions and levels.
  • Frequency Selection: A 1×1 convolution after Fourier transformation lets the network learn flexible thresholds for frequency selection instead of using a manually fixed threshold.Different thresholds produce different results, and the passage reports that T = 100(1+ j) outperforms original ReLU in one example.
  • Res FFT-ReLU Block: The proposed block adds an FFT-ReLU stream to a spatial ResBlock, combining kernel-level blur-pattern features with pixel-level features while retaining identity mapping.The FFT-ReLU stream replaces the identity mapping in a simple variant, whereas the proposed block keeps the identity mapping to assist training.
  • Res FFT-ReLU Block: The FFT-ReLU stream computes channelwise FFT, complex-domain processing with ReLU, and inverse FFT, then adds its output to the spatial residual stream and input.The final block output is Y = Yfft + Yres + Z.
  • Global Context Learning Ability: Frequency-domain linear convolution extracts features but does not provide global-context learning, whereas frequency-domain ReLU introduces global-context learning through nonlinearity.The analysis also reports that frequency-domain ReLU does not always improve performance and that its location matters.

Experiments

Experiments evaluate training setup, frequency-stream design choices, neuron behavior, and plug-in performance across deblurring architectures and datasets. Results support combining FFT-ReLU with spatial features and show gains when replacing standard residual blocks.

  • Experimental Setup: Experiments use GoPro, HIDE, and RealBlur datasets, with Group I training on 2,103 GoPro blurry-sharp pairs and evaluation across multiple test sets.The study reports PSNR and SSIM, along with parameter counts, FLOPs, and per-image testing time.
  • Position and Threshold Ablations: The proposed stream performs best when replacing the ResBlock identity mapping: 29.51 vs. 28.06 PSNR, whereas another insertion position drops to 27.78.The position study uses RSNet on GoPro and also evaluates threshold choices and alternative frequency-selection rules.
  • FFT-ReLU Stream Ablations: Adding FFT-ReLU to RSNet improves PSNR from 28.06 to 30.30, while using the stream alone reduces performance to 25.64.The ablation indicates that the frequency stream is intended as an add-on to spatial-domain CNN processing.
  • Neuron Visualizations: Neuron visualizations show that RSNet captures local neighborhoods, FFT-ReLU alone captures global context without clear localization, and the combined model captures both representations.The combined behavior is described as learning kernel-level and pixel-level representations.
  • Plug-in Evaluation: Replacing standard ResBlocks with Res FFT-ReLU Blocks produces performance gains across several architectures, including RSNet on REDS: 26.78 vs. 27.79 PSNR.The block is evaluated in DeepDeblur, U-Net, MPRNet-small, MIMO-UNet, and NAFNet; FMIMO-UNet variants and FNAFNet are compared with other methods.

Conclusion

The paper identifies frequency selection as a source of faithful blur-pattern information and proposes a plug-and-play Res FFT-ReLU Block for dual-domain deblurring. Integrated into MIMO-UNet and NAFNet, it improves performance on three public datasets.

  • Conclusion: Frequency selection provides faithful information about blur patterns, including blur direction and blur level.The operation implicitly reveals the blur-kernel pattern.
  • Conclusion: The Res FFT-ReLU Block enables image-wide receptive fields for capturing long-term interactions.
  • Conclusion: Plugging the block into MIMO-UNet and NAFNet achieves superior performance over state-of-the-art methods on GoPro, HIDE, and RealBlur.

Analysis of Remark 1

The analysis explains how applying ReLU in the frequency domain and then inverse Fourier transforming a blurry image exposes a component containing the blur pattern. In a simple motion-blur example, this component implicitly reflects the kernel structure.

  • Analysis of Remark 1: A simple motion-blur kernel can be represented as g(t) = δ(t)+δ(t+ϵ), where ϵ is a very small value.
  • Analysis of Remark 1: The analyzed operation applies Fourier transform, separate real- and imaginary-part ReLU, and inverse Fourier transform to the blurry image.
  • Analysis of Remark 1: The transformed output contains a component formed by convolving the blur kernel with decomposed peak values.The peaks arise from applying the absolute-value operation in the frequency-domain analysis.
  • Analysis of Remark 1: This component implicitly shows the kernel pattern, providing a blur-pattern signal extracted from the blurry image.

Supplementary for Experiments

The supplementary experiments analyze architecture placement, frequency selection, feature representations, visual restoration, and efficiency. Results support using both frequency bands and multiple Res FFT-ReLU layers, while showing improvements over baseline networks.

  • Supplementary for Experiments: RSNet is a simplified DeepDeblur backbone containing 16 ResBlocks.
  • Supplementary for Experiments: The NAFNet Res FFT-ReLU block uses two FFT-ReLU streams.
  • Low/High Frequency and Necessity of Multi-layer Res FFT-ReLU Block: All high- and low-frequency components are important for restoring sharp images, although LF (1/4) reaches 30.44 versus 30.30 for the full FFT-ReLU stream in RSNet.
  • Low/High Frequency and Necessity of Multi-layer Res FFT-ReLU Block: Using FFT-ReLU streams in multiple layers is necessary, with later-layer placement scoring 30.01 versus 29.69 for the first eight layers.The analysis describes the layers as progressively deblurring outputs from previous layers.
  • Other Details for Visualizing Neurons: Preferred-input visualizations indicate that RSNet captures local context, whereas FFT-ReLU variants capture global context.
  • Visualizations of MIMO-UNet w/ and w/o Res FFT-ReLU Block: FMIMO-UNet compensates more frequency information than MIMO-UNet when comparing restored-image differences and their Fourier magnitudes.
  • Supplementary for Experiments: Adding FFT-ReLU blocks to selected NAFNet encoder and skip-connected decoder blocks yields gains of 0.27 dB for FNAFNet32 and 0.16 dB for FNAFNet64 over their NAFNet counterparts.
  • Detailed Analysis for Evaluation of FMIMO-UNet and FNAFNet: FMIMO-UNet-small outperforms MIMO-UNet by 0.79 dB PSNR with 2.4M fewer parameters.

Qualitative Comparisons of FMIMO-UNet with Other Competitors

Qualitative comparisons present restored images and zoomed patches from multiple deblurring methods on GoPro, RealBlur, and HIDE. The authors report that FMIMO-UNet more successfully recovers local details and structures than the competitors.

  • Qualitative Comparisons of FMIMO-UNet with Other Competitors: FMIMO-UNet is reported to be more successful at deblurring local details and structures than the compared methods.
  • Qualitative Comparisons of FMIMO-UNet with Other Competitors: The comparisons cover GoPro, RealBlur, and HIDE test datasets using blurry inputs, competitor outputs, FMIMO-UNet variants, and ground-truth sharp images.
Loading 2111.11745v2…