Rethinking Coarse-to-Fine Approach in Single Image Deblurring

Sung-Jin Cho; Seo-Won Ji; Jun-Pyo Hong; Seung-Won Jung; Sung-Jea Ko

Source-linked AI summary

Rethinking Coarse-to-Fine Approach in Single Image Deblurring

Sung-Jin Cho, Seo-Won Ji, Jun-Pyo Hong, Seung-Won Jung, Sung-Jea Ko

arXiv:2108.05054v2Aug 11, 2021cs.CVcs.AI

TL;DR

Conventional coarse-to-fine deblurring networks are computationally costly, while lightweight designs may sacrifice state-of-the-art accuracy. MIMO-UNet uses one multi-input, multi-output U-Net with asymmetric feature fusion and outperforms conventional methods in speed and accuracy.

Problem
Conventional deblurring networks have high computational and memory costs, while lightweight architectures may not achieve comparable state-of-the-art accuracy.
Source: §1 · p. 2
Method
MIMO-UNet replaces stacked sub-networks with one U-Net using multi-scale inputs, multi-scale outputs, and asymmetric feature fusion for coarse-to-fine deblurring.
Source: §1 · p. 2, §3 · p. 3
Results
MIMO-UNet outperforms conventional methods in both deblurring speed and accuracy on the GoPro and RealBlur datasets.
Source: §1 · p. 2, §5 · p. 8
Takeaways & Limitations
A single modified U-Net provides a simpler coarse-to-fine deblurring design with reported gains in computational efficiency and restoration accuracy.
Source: §1 · p. 2, §5 · p. 8
Runtime comparisons use released test code measured on the authors’ PC alongside runtimes reported in other papers.
Source: §4.2 · p. 7

Abstract

from arXiv · show

Coarse-to-fine strategies have been extensively used for the architecture design of single image deblurring networks. Conventional methods typically stack sub-networks with multi-scale input images and gradually improve sharpness of images from the bottom sub-network to the top sub-network, yielding inevitably high computational costs. Toward a fast and accurate deblurring network design, we revisit the coarse-to-fine strategy and present a multi-input multi-output U-net (MIMO-UNet). The MIMO-UNet has three distinct features. First, the single encoder of the MIMO-UNet takes multi-scale input images to ease the difficulty of training. Second, the single decoder of the MIMO-UNet outputs multiple deblurred images with different scales to mimic multi-cascaded U-nets using a single U-shaped network. Last, asymmetric feature fusion is introduced to merge multi-scale features in an efficient manner. Extensive experiments on the GoPro and RealBlur datasets demonstrate that the proposed network outperforms the state-of-the-art methods in terms of both accuracy and computational complexity. Source code is available for research purposes at https://github.com/chosj95/MIMO-UNet.

1. Introduction

Single-image deblurring must recover sharp images despite motion-induced blur that harms visual quality and vision systems, while conventional coarse-to-fine networks incur high computational and memory costs. MIMO-UNet revisits this strategy with a single encoder-decoder network whose multi-scale inputs and multi-output decoder enable efficient coarse-to-fine restoration.

Motivation

Single-image deblurring recovers a latent sharp image from blur caused by camera or object motion, whose artifacts degrade visual quality and vision-system performance.The passage specifically identifies surveillance and autonomous driving as affected systems.Source: §1 · p. 1

Prior Work

CNN-based methods evolved from two-stage blur-kernel estimation and deconvolution frameworks toward end-to-end learning of blurry-sharp image relationships.DeepDeblur is identified as a pioneering deep multi-scale CNN for dynamic-scene deblurring.Source: §1 · p. 1

Problem

Conventional coarse-to-fine methods increase computational complexity and memory usage, limiting deployment in cost- and time-sensitive environments.Examples include mobile devices, vehicles, and robots.Source: §1 · p. 2

Proposed Method

MIMO-UNet is a single encoder-decoder U-shaped network designed to handle multi-scale blur with low computational complexity.The paper presents it as a reconsideration of the coarse-to-fine scheme.Source: §1 · p. 2
Its multi-output single decoder produces multiple deblurred images and mimics stacked sub-networks by guiding gradual coarse-to-fine recovery of latent sharp images.Experiments report superiority over state-of-the-art methods in PSNR and computational complexity.Source: §1 · p. 2

2. Related works

Related work commonly adopts coarse-to-fine deblurring, progressively recovering sharp images through multiple scales, sub-networks, or repeated U-shaped networks. DeepDeblur established this direction, followed by PSS-NSC and MT-RNN variants that modify feature connections, parameter sharing, or recurrent processing.

Conventional image deblurring methods reviewed here adopt a coarse-to-fine strategy for gradual recovery of latent sharp images.Source: §2 · p. 2, §2.1 · p. 2

DeepDeblur

DeepDeblur directly learns blurry-sharp image relations end-to-end and uses multiple stacked sub-networks receiving differently scaled inputs.Outputs from coarser sub-networks are concatenated with finer-scale inputs to enable progressive reconstruction.Source: §2.1 · p. 2

GoPro dataset

The GoPro dataset constructs blurry images by averaging successive sharp frames captured at 240 fps and uses the middle sampled frame as ground truth.Source: §2.1 · p. 2

PSS-NSC

PSS-NSC retains DeepDeblur’s multi-sub-network design but structures each sub-network as an encoder-decoder U-Net with symmetric skip connections.It additionally introduces parameter selective sharing across sub-networks.Source: §2.2 · p. 3

MT-RNN

MT-RNN repeats a single U-shaped network seven times and transfers decoder feature maps from each iteration to the next encoder.Each iteration predicts an averaged image formed with a different M, decreasing as processing proceeds.Source: §2.3 · p. 3

3. Proposed method

MIMO-UNet revisits coarse-to-fine deblurring with a single U-Net that processes multi-scale inputs, produces multi-scale outputs, and enables cross-scale feature fusion. Its design also uses multi-scale content supervision and a frequency reconstruction loss to improve restoration.

Architecture

MIMO-UNet is a single U-Net with three encoder blocks and decoder blocks, modified for efficient multi-scale deblurring through MISE, MOSD, and AFF.The architecture is organized around the paper’s three stated special features.Source: §3 · p. 3

Multi-scale input

Each encoder block receives a blurry image at a different scale and combines downsampled-image features with features from the preceding encoder level.This complementary multi-scale input is intended to help handle diverse image blurs effectively.Source: §3.1 · p. 3

Multi-scale output

Each decoder block receives intermediate supervision, allowing its multi-scale feature map to be mapped into an output image that mimics multi-stacked sub-networks.The mapping function is implemented with a single convolutional layer.Source: §3.2 · p. 4

Asymmetric feature fusion

AFF modules take outputs from all encoder blocks, align different scales through up- and down-sampling, and deliver fused features to corresponding decoder blocks.This design allows information flow from different scales within a single U-Net, so each decoder block can exploit multi-scale features.Source: §3.3 · pp. 4–5

Training objective

The network uses multi-scale L1 content loss and introduces MSFR loss, which measures L1 distance between multi-scale ground-truth and deblurred images in the frequency domain.The frequency-domain comparison uses the fast Fourier transform, and the final loss sets λ = 0.1.Source: §3.4 · pp. 5–6

4. Experiments

Experiments on GoPro and RealBlur evaluate MIMO-UNet’s accuracy, efficiency, component contributions, and utility for object detection. The study compares three model variants with state-of-the-art methods and finds consistent benefits from the proposed components and preprocessing use.

Experimental Setup

The models were trained on 2,103 GoPro and 3,758 RealBlur blurred–sharp image pairs, then tested on 1,111 and 980 pairs, respectively.GoPro testing used a model trained only on the GoPro training set.Source: §4.1 · p. 6

Comparisons

MIMO-UNet, MIMO-UNet+, and MIMO-UNet++ evaluate increasing capacity and geometric self-ensemble within a single architecture.The variants use 8 or 20 residual blocks per encoder and decoder block, while MIMO-UNet++ estimates the resultant image using geometric self-ensemble.Source: §4.2 · p. 6
MIMO-UNet was compared with state-of-the-art deblurring networks using runtime measurements from released test code and runtimes reported in the papers.Author-released models were reproduced for GoPro and RealBlur comparisons, including SRN, PSS-NSC, DMPHN, MT-RNN, MPRNet, and DeblurGAN-v2.Source: §4.2 · pp. 6–7

Ablation Studies

FAM achieved the highest performance among feature-fusion methods, while combining MISE with MOSD improved PSNR by 0.05 dB and AFF improved it by 0.17 dB over baseline.Using AFF with MISE increased the gain to 0.23 dB; MISE, MOSD, and AFF together achieved 0.30 dB higher PSNR.Source: §4.3 · pp. 7–8

Object Detection

MIMO-UNet++ was also evaluated as a deblurring preprocessing method for object detection against PSS-NSC and DMPHN using author-provided codes.The evaluation used sharp-image bounding boxes as ground truth and focused on car, person, and potted plant classes present in GoPro images.Source: §4.4 · p. 8, Figure 8 · p. 8

5. Conclusion

The paper proposes a fast, accurate image deblurring network that replaces stacked coarse-to-fine subnetworks with a single, simpler U-Net. Its encoder accepts multi-scale inputs, while its decoder outputs multi-scale deblurred images during decoding.

The proposed network targets fast and accurate image deblurring.Source: §5 · p. 8
A single U-Net replaces stacked sub-networks for coarse-to-fine deblurring, simplifying the architecture.Source: §5 · p. 8
The encoder takes multi-scale input images and combines features from different sources.Source: §5 · p. 8
The decoder outputs multi-scale deblurred images during decoding.Source: §5 · p. 8