PaperlayerView exact revision on arXiv

Source-linked AI summary

BANet: Blur-aware Attention Networks for Dynamic Scene Deblurring

Fu-Jen Tsai, Yan-Tsung Peng, Yen-Yu Lin, Chung-Chi Tsai, Chia-Wen Lin

arXiv:2101.07518v4cs.CV

TL;DR

Dynamic-scene deblurring must handle directional, non-uniform blur while avoiding the runtime cost of recurrent processing and the memory cost of conventional self-attention. BANet uses region-based blur-aware attention with multi-scale dilated convolution in a single forward pass, and experiments on GoPro and RealBlur show favorable state-of-the-art performance with real-time deblurring.

  • Problem

    Dynamic scene blur is directional and non-uniform, while self-recurrent methods increase inference time and inter-pixel or inter-channel attention can use excessive memory.

  • Method

    BANet stacks blur-aware modules that use regional attention to identify blur orientations and magnitudes and cascaded parallel dilated convolution to remove blur adaptively in one pass.

  • Results

    BANet performs favorably against state-of-the-art methods on the GoPro and RealBlur benchmarks while providing real-time deblurring.

  • Takeaways & Limitations

    BANet offers an efficient single-forward-pass approach for dynamic-scene deblurring with region-wise blur modeling and multi-scale feature restoration.

Abstract

from arXiv · show

Image motion blur results from a combination of object motions and camera shakes, and such blurring effect is generally directional and non-uniform. Previous research attempted to solve non-uniform blurs using self-recurrent multiscale, multi-patch, or multi-temporal architectures with self-attention to obtain decent results. However, using self-recurrent frameworks typically lead to a longer inference time, while inter-pixel or inter-channel self-attention may cause excessive memory usage. This paper proposes a Blur-aware Attention Network (BANet), that accomplishes accurate and efficient deblurring via a single forward pass. Our BANet utilizes region-based self-attention with multi-kernel strip pooling to disentangle blur patterns of different magnitudes and orientations and cascaded parallel dilated convolution to aggregate multi-scale content features. Extensive experimental results on the GoPro and RealBlur benchmarks demonstrate that the proposed BANet performs favorably against the state-of-the-arts in blurred image restoration and can provide deblurred results in real-time.

I. INTRODUCTION

Dynamic scene deblurring must recover detail from directional, non-uniform blur, but existing self-recurrent and self-attention approaches trade accuracy against speed or memory. BANet addresses these issues with region-wise blur-aware attention and cascaded multi-scale dilated convolutions in a single forward pass.

  • Dynamic scene blur causes information loss, motivating restoration for downstream vision applications.
  • Self-recurrent models process blur across multiple scales, patch levels, or temporal behaviors, but their progressive inference is inefficient.
  • Pixel-wise or channel-wise self-attention can require high memory and may not capture directional, local averaging artifacts from motion blur.
  • BANet performs single-forward-pass deblurring by separating blur-aware attention from cascaded parallel dilated convolution.
  • Region-wise attention captures blur orientations and magnitudes, while cascaded multi-scale dilated convolution restores blurred features.

II. RELATED WORK

Prior deblurring methods use handcrafted assumptions, recurrent processing, or self-attention, each with limitations in generalization, runtime, memory, or spatial fidelity. BANet instead uses low-memory regional averaging attention to model non-uniform blur while retaining high-resolution applicability.

  • A. Conventional Methods: Conventional methods impose handcrafted constraints on blur kernels or latent images, limiting generalization to complex real-world examples.
  • B. Learning-based Methods: Self-recurrent learning methods restore images progressively across resolutions, fields of view, or temporal characteristics, but usually require longer inference runtimes.
  • C. Self-attention: Inter-pixel self-attention requires O(H^2W^2) space, restricting its use to the smallest-scale features.
  • C. Self-attention: Regional averaging self-attention uses O(CHW) space, preserves regional blur information, and supports high-resolution real-time deblurring.

III. PROPOSED APPROACH

BANet is a one-pass residual network built from blur-aware modules that disentangle regional blur patterns and remove them efficiently. Its blur-aware attention combines multi-kernel strip pooling with attention refinement, while cascaded dilated convolutions capture multi-scale features.

  • III. PROPOSED APPROACH: BANet uses stacked blur-aware modules (BAMs) to disentangle different blur patterns and remove blurs from attended features.Each BAM contains Blur-aware Attention (BA) and Cascaded Parallel Dilated Convolution (CPDC).
  • A. Blur-aware Attention (BA): BA uses multi-kernel strip pooling (MKSP) followed by attention refinement (AR) to capture multi-scale blur patterns and refine them locally.MKSP targets blurred patterns with different magnitudes and orientations, while AR further refines the globally attended features.
  • A. Blur-aware Attention (BA): Unlike standard strip pooling, MKSP is designed to locate blurred patterns with varying orientations and magnitudes rather than only band-shaped objects.Standard strip pooling uses horizontal and vertical one-pixel-long kernels, but the paper states that this is unsuitable for directly locating diverse blur patterns.
  • A. Blur-aware Attention (BA): MKSP extends strip pooling with different kernel sizes to discover regional and directional averaging artifacts caused by dynamic blurs.It combines multi-scale horizontal and vertical results through concatenation and convolution to reveal blur orientations and magnitudes.
  • A. Blur-aware Attention (BA): The fused MKSP outputs form an attention mask, which is multiplied with the input features before local attention refinement.The mask is produced by concatenating multi-scale fused tensors and applying convolutional mapping, then AR refines the attended feature maps.
  • A. Blur-aware Attention (BA): BA requires O(HWC) memory and disentangles blurred contents with different magnitudes and orientations.The paper illustrates separation of background and foreground blur content in examples involving different object motions.

B. Cascaded Parallel Dilated Convolution (CPDC)

CPDC uses cascaded parallel dilated convolutions to aggregate multi-scale content features for deblurring. The module combines multiple dilation rates with a convolutional fusion bridge.

  • CPDC design: CPDC captures multi-scale blurred objects using multiple dilation rates without increasing convolution kernel size.Dilated convolution enlarges receptive fields while extracting features at different object scales.
  • Attention comparison: The attended-feature comparisons use AR, MKSP, and their combination MKSP + AR.These comparisons are shown for an input blurred image from the GoPro testing set.
  • Attention comparison: Blur-pattern disentanglement examples visualize attended feature maps for regions with different blur degrees.The examples are evaluated on the GoPro test set.
  • CPDC design: Unlike PDC, CPDC cascades two parallel dilated-convolution blocks through a single convolutional fusion layer.The bridge distills useful patterns before the second PDC block.
  • PDC structure: Each PDC uses three 3×3 dilated convolutions with dilation rates D = 1, 3, and 5.Each branch produces half the input channels; concatenation increases output channels by 1.5 times.

C. Loss function

BANet is optimized with a Charbonnier-based loss using restored and ground-truth images, together with a total-loss formulation and an empirically chosen weighting coefficient.

  • Loss formulation: BANet uses the Charbonnier loss for image restoration training.The loss is adopted following prior deblurring work.
  • Loss formulation: The restored image is denoted R, the ground-truth image Y, and the Charbonnier stability constant is ε = 10−3.These quantities define the loss inputs and its numerical setting.
  • Loss formulation: The total optimization objective combines the stated loss terms with λ set empirically to 0.01.The supplied passage identifies λ as the empirical weighting coefficient.

IV. EXPERIMENTS

The experiments evaluate BANet through benchmark comparisons and ablation studies after describing the experimental setup. This design assesses both overall performance and the contribution of individual components.

  • Evaluation protocol: The evaluation first introduces the experimental setup before comparing BANet with state-of-the-art methods.The comparison is part of the paper’s stated evaluation sequence.
  • Evaluation protocol: The experiments include ablation studies to analyze the effectiveness of individual components.Ablation follows the benchmark comparison in the stated evaluation plan.
  • Evaluation protocol: The evaluation sequence consists of setup description, state-of-the-art comparison, and component analysis.These are the three stages explicitly identified for the section.

A. Experimental Setup

BANet is evaluated on GoPro, HIDE, and RealBlur benchmarks, with RealBlur split into raw-image and JPEG-image subsets. Training uses Adam with cosine-annealed learning rates and standard augmentation.

  • Datasets: BANet is evaluated on GoPro, HIDE, and RealBlur image-deblurring benchmarks.GoPro and RealBlur include training and testing splits, while HIDE is used for testing.
  • Datasets: RealBlur is divided into RealBlur-R for raw images and RealBlur-J for JPEG images.The supplied setup passage identifies the two image-format subsets.
  • Experimental context: The experiments were completed by authors from universities in Taiwan.This contextual detail is stated separately from the benchmark and optimization setup.
  • Training setup: Training uses Adam with β1 = 0.9, β2 = 0.999, and a learning rate decayed from 10−4 to 10−7 by cosine annealing.The setup also applies random cropping, flipping, and rotation for augmentation.
  • Training setup: The implementation uses PyTorch on an Intel Xeon Silver 4210 CPU with an NVIDIA 2080ti GPU.These hardware and software details are reported as part of the implementation setup.

B. Experimental Results

BANet is evaluated against recent deblurring methods on GoPro, HIDE, and RealBlur, using quantitative, qualitative, and subjective comparisons. The reported results emphasize runtime efficiency, restoration of difficult details, and strong subjective preference.

  • Evaluation protocol: The evaluation compares BANet with recent deblurring methods on GoPro and HIDE, while RealBlur results and a user study assess real blurred images.The user study uses 34 subjects and 16 blurred images, producing 64 pairwise comparisons per subject.
  • Quantitative results: BANet and BANet+ are evaluated with PSNR, SSIM, runtime, parameter count, and GFLOPs on GoPro.BANet uses 128 channels and 18 million parameters, whereas BANet+ uses 192 channels and 40 million parameters.
  • Quantitative results: Self-recurrent models consume longer runtime than the non-recurrent methods, including BANet and BANet+.The comparison identifies MSCNN, SRN, DSD, MTRNN, SAPHN, and MPRNet as self-recurrent, and DeblurGAN-v2, RADN, MIMO-UNet+, and BANet as non-recurrent.
  • Qualitative results: On GoPro and HIDE, BANet better restores regions containing text, severe blur, and striped clothing than several compared methods.The qualitative comparisons include MTRNN, DSD, DMPHN, MIMO-UNet+, and MPRNet.
  • Qualitative results: On RealBlur, all compared models remove blur, while BANet performs favorably on delicate image details.The RealBlur comparisons include DeblurGAN-v2, SRN, MPRNet, and MIMO-UNet+.
  • User study: BANet receives over 95% of preference votes against every compared method in the subjective evaluation.The compared methods are MIMO-UNet+, MPRNet, SRN, and DeblurGAN-v2, with all methods trained on RealBlur-J training data.

C. Ablation Study

The ablation studies examine BAM components, the number of stacked BAMs, pooling kernels, and CPDC against alternative designs. They report gains from the proposed components and select a stack depth balancing efficiency and visual quality.

  • BAM components: Adding attention refinement to PDC improves PSNR by 0.41 dB on the GoPro test set.This comparison is between Net1 and Net2 in the BAM component ablation.
  • BAM components: Using MKSP in PDC improves PSNR by 0.72 dB over PDC with neither MKSP nor strip pooling.The comparison is between Net1 and Net4, and the reported gain exceeds the gain from strip pooling or attention refinement alone.
  • BAM components: Replacing PDC with CPDC produces a further performance gain in the proposed BAM.The component study compares the CPDC-based Net6 with the PDC-based Net5.
  • Stacked BAMs: Performance improves as more BAMs are stacked, but the improvement saturates after 12, so BANet uses 10 for efficiency and visual quality.The tested configurations are stack-4, stack-8, stack-10, and stack-12.
  • MKSP and CPDC: MKSP with kernel sizes 1, 3, 5, and 7 performs slightly better than adding kernel size 9.The ablation concludes that the size-9 kernel does not improve blur-feature capture or performance.
  • MKSP and CPDC: CPDC outperforms both PDC and the similarly sized two-stage PDC2 variant on the GoPro test set.CPDC uses a single convolution as a fusion bridge, while PDC2 stacks two PDCs in series.

D. Blur-aware Attention vs. Self-Attention

The paper compares region-based blur-aware attention with conventional self-attention for deblurring high-resolution images. Blur-aware attention uses substantially lower memory, allowing full-resolution processing where self-attention is impractical.

  • Memory and resolution: Self-attention requires O(H^2W^2) memory for 720 × 1280 images, making high-resolution processing infeasible.The comparison therefore trains with the stack-4 model and tests both methods on eight sub-images.
  • Memory and resolution: Blur-aware attention requires O(CHW) memory, with C << H × W, and can process the full-resolution image.The paper presents this lower memory usage as an advantage over self-attention for high-resolution deblurring.
  • Attention design: Region-based attention is designed to correlate regions with similar blur characteristics while avoiding the high memory demand of self-attention.The paper contrasts this design with self-attention applied at smaller network scales, where down-sampling can lose blur information.

V. CONCLUSION

BANet uses stacked blur-aware modules to separate region-wise blur contents by magnitude and orientation and aggregate multi-scale features. Experiments report real-time deblurring and favorable performance against state-of-the-art methods on GoPro and RealBlur.

  • Conclusion: BANet stacks blur-aware modules to disentangle region-wise blur contents of different magnitudes and orientations.The modules also aggregate multi-scale content features for dynamic scene deblurring.
  • Conclusion: Extensive experiments and ablations examine the attention masks, attended feature maps, design choices, and performance comparisons.The paper evaluates the proposed design through demonstrations, ablation studies, and benchmark comparisons.
  • Conclusion: BANet achieves real-time deblurring and performs favorably against state-of-the-art methods on the GoPro and RealBlur benchmarks.This is the paper's reported overall conclusion for single-image dynamic scene deblurring.
Loading 2101.07518v4…