PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Baselines for Image Restoration

Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, Jian Sun

arXiv:2204.04676v4cs.CV

TL;DR

Increasing system complexity makes SOTA image restoration methods harder to analyze and compare. The paper builds a simple baseline, then derives NAFNet by replacing or removing nonlinear activations. The resulting models achieve SOTA performance on image denoising and deblurring with reduced computational cost.

  • Problem

    Image restoration methods achieve strong SOTA performance, but their increasing system complexity hinders convenient analysis and comparison.

  • Method

    The paper decomposes SOTA methods to build a simple single-stage UNet baseline, then derives NAFNet by replacing or removing nonlinear activation functions.

  • Results

    The baseline and NAFNet achieve SOTA performance on image denoising and deblurring with only a fraction of the computation costs.

  • Takeaways & Limitations

    Nonlinear activation functions are not necessary for the proposed models to achieve SOTA computer vision performance, and the baselines may facilitate idea evaluation.

Abstract

from arXiv · show

Although there have been significant advances in the field of image restoration recently, the system complexity of the state-of-the-art (SOTA) methods is increasing as well, which may hinder the convenient analysis and comparison of methods. In this paper, we propose a simple baseline that exceeds the SOTA methods and is computationally efficient. To further simplify the baseline, we reveal that the nonlinear activation functions, e.g. Sigmoid, ReLU, GELU, Softmax, etc. are not necessary: they could be replaced by multiplication or removed. Thus, we derive a Nonlinear Activation Free Network, namely NAFNet, from the baseline. SOTA results are achieved on various challenging benchmarks, e.g. 33.69 dB PSNR on GoPro (for image deblurring), exceeding the previous SOTA 0.38 dB with only 8.4% of its computational costs; 40.30 dB PSNR on SIDD (for image denoising), exceeding the previous SOTA 0.28 dB with less than half of its computational costs. The code and the pre-trained models are released at https://github.com/megvii-research/NAFNet.

1 Introduction

Image restoration methods achieve strong performance but increasingly complex systems make clear analysis and comparison difficult. The paper develops a simpler, efficient baseline and further simplifies it into NAFNet by removing or replacing nonlinear activations.

  • 1 Introduction: High system complexity arises from both inter-block connections or stages and numerous intra-block design choices.Examples include multi-scale connections, multi-stage refinement, attention modules, gated feed-forward networks, and other block components.
  • 1 Introduction: A natural question is whether a network with low inter-block and intra-block complexity can still achieve SOTA performance.
  • 1 Introduction: The paper adopts a single-stage UNet to reduce inter-block complexity and uses ablations to identify a simple, computationally efficient baseline.The baseline is formed by starting from a plain convolution-ReLU-shortcut block and testing added or replaced components.
  • 1 Introduction: The baseline can exceed previous SOTA methods while using lower computational cost.Figure 1 presents the comparison of PSNR against computational cost for image deblurring and denoising.
  • 1 Introduction: NAFNet replaces or removes nonlinear activations, including Sigmoid, ReLU, and GELU, while matching or surpassing the simpler baseline.The simplification uses connections between GELU, Channel Attention, and GLU, including element-wise multiplication of feature maps.

2 Related Works

Image restoration models commonly use UNet variants, but performance improvements often come with greater inter-block or intra-block complexity. The paper relates gated linear units to activation-free multiplication and uses a single-stage UNet architecture as its baseline structure.

  • 2 Related Works: Image restoration methods aim to recover clean images from degraded inputs, and many deep learning approaches are UNet variants with skip connections.These variants improve performance while increasing system complexity.
  • 2 Related Works: Inter-block complexity includes serial multi-stage refinement and connections between feature maps of different sizes.Multi-stage networks refine earlier results, while multi-scale fusion methods connect differently sized features.
  • 2 Related Works: Intra-block designs include channelwise or window-based attention, gated linear units, depthwise convolution, and locally enhanced feed-forward networks.
  • 2 Related Works: Gated Linear Units can be interpreted as element-wise products of two linear transformations, with one transformation activated by a nonlinearity.
  • 2 Related Works: The paper removes the nonlinear activation from GLU without performance degradation and replaces baseline activations with feature-map multiplication.It presents this as a route to SOTA performance without nonlinear activation functions.

3 Build A Simple Baseline

The paper builds a single-stage UNet baseline by starting with common convolutional blocks and evaluating design choices for normalization, activation, and attention. The resulting baseline combines simple components while surpassing prior SOTA results at a fraction of the computational cost.

  • Architecture: The model adopts a classic single-stage U-shaped architecture with skip-connections to reduce inter-block complexity.
  • Block design: The block begins with convolution, ReLU, and a shortcut, forming the PlainNet starting point for internal-structure comparisons.
  • Activation: ReLU is replaced by GELU because denoising performance remains comparable while deblurring improves by 0.21 dB, from 31.90 dB to 32.11 dB on GoPro.
  • Attention: The design avoids window-based attention because depthwise convolution can capture local information, while channel attention provides global information efficiently.
  • Baseline: The baseline combines LayerNorm, convolution, GELU, and channel attention, and surpasses prior SOTA results on SIDD and GoPro with only a fraction of their computation costs.

4 Nonlinear Activation Free Network

The paper derives NAFNet by replacing activation-heavy components with multiplicative operations. SimpleGate replaces GELU, while Simplified Channel Attention removes nonlinearities without performance loss.

  • SimpleGate: GELU is treated as a special case of GLU, motivating multiplication-based replacement of nonlinear activation functions.
  • SimpleGate: SimpleGate divides a feature map into two channel-wise parts and multiplies them element-wise, eliminating the need for an explicit activation function.
  • SimpleGate: Replacing GELU with SimpleGate improves SIDD performance by 0.08 dB, from 39.85 dB to 39.93 dB, and GoPro performance by 0.41 dB, from 32.35 dB to 32.76 dB.
  • Simplified Channel Attention: Channel attention is reformulated as feature-map multiplication by an attention function, revealing its similarity to GLU.
  • Simplified Channel Attention: Simplified Channel Attention retains global-information aggregation and channel interaction while removing the original nonlinear components.
  • Simplified Channel Attention: Simplified Channel Attention changes performance by +0.03 dB on SIDD, from 39.93 dB to 39.96 dB, and +0.09 dB on GoPro, from 32.76 dB to 32.85 dB.
  • NAFNet: NAFNet replaces GELU with SimpleGate and channel attention with Simplified Channel Attention, leaving no nonlinear activation functions in the network.

5 Experiments

Experiments evaluate NAFNet across multiple image-restoration tasks and ablate its components, depth, and activation choices. The baseline and NAFNet achieve strong results on SIDD, GoPro, raw denoising, and REDS deblurring while reducing computational cost.

  • Experiments cover RGB denoising, image deblurring, raw denoising, and deblurring with JPEG artifacts.
  • Ablation studies: Layer Normalization stabilizes PlainNet training and improves PSNR by 0.46 dB on SIDD and 3.39 dB on GoPro.
  • Ablation studies: NAFNet’s simplification boosts PSNR by 0.11 dB on SIDD and 0.50 dB on GoPro without performance penalty.
  • Ablation studies: 36 blocks provide a better performance/latency balance than 9 or 72 blocks, with latency increases of 14.5% and 30.0% in the reported comparisons.
  • Ablation studies: Removing the SimpleGate nonlinearity leaves SIDD PSNR nearly unchanged at 39.96–39.99 dB, while GoPro PSNR drops by 0.11–0.35 dB with alternatives.
  • Applications: On SIDD, the baseline and NAFNet exceed Restormer by 0.28 dB with only a fraction of its computational cost.
  • Applications: On GoPro, the baseline and NAFNet surpass MPRNet-local by 0.09 dB and 0.38 dB, respectively, using only 8.4% of its computational costs.
  • Applications: NAFNet surpasses PMRID quantitatively and qualitatively on raw image denoising and outperforms competing methods, including HINet, on REDS-val-300.

6 Conclusions

The paper derives a simple baseline by extracting essential components from SOTA methods, then simplifies it into NAFNet by replacing or removing nonlinear activations. Despite this simplification, NAFNet matches or exceeds the baseline and supports strong restoration performance.

  • Decomposing SOTA methods yields a lower-complexity baseline that reaches SOTA performance on image denoising and deblurring.
  • NAFNet replaces or removes nonlinear activation functions, yet its performance is equal to or better than the baseline.
  • The authors suggest the findings may influence future computer-vision model design by showing nonlinear activations are not necessary for SOTA performance.

A.1 Inverted Bottleneck

The appendix analyzes the computational cost of baseline and NAFNet blocks using inverted-bottleneck channel dimensions. The resulting block costs are approximately 6 × H × W × c × c, enabling shared hyperparameters between the two models.

  • For the baseline’s first skip connection, c ≫ k × k makes its cost approximately 2 × H × W × c × c.
  • Baseline block cost is approximated as 6 × H × W × c × c after combining its two skip-connection computations.
  • NAFNet doubles the hidden dimension in its first skip connection because SimpleGate halves channel width.
  • NAFNet and baseline blocks have consistent overall computational cost, allowing them to share hyperparameters such as block count and learning rate.
  • The cost discussion omits Layer Normalization, GELU, and channel-attention computation because it is negligible relative to convolution.

A.2 Channel Attention and Simplified Channel Attention

The method uses element-wise addition to fuse encoder and decoder features, while channel attention reduces and then restores channel dimensionality. A simplified channel-attention module is matched to the original’s cost by setting r = 2.

  • Channel attention shrinks a c-width feature map by factor r and projects it back, with cost approximately c × c/r + c/r × c.
  • The simplified channel-attention module costs c × c, and r = 2 makes its cost consistent with the original for fair comparison.
  • Encoder and decoder features are fused by simple element-wise addition rather than concatenation followed by convolution.
  • Downsampling uses a 2 × 2 stride-2 convolution, while upsampling first doubles channels with a pointwise convolution and then applies pixel shuffle.

B More Visualization Results

Additional visualizations compare image denoising and deblurring methods, including raw and RGB denoising tasks. The comparisons highlight fine-detail restoration by the baselines and NAFNet.

  • Additional visualizations cover raw image denoising, image deblurring, and RGB image denoising tasks.These results are presented in Figures 1, 2, and 3.
  • 25.81 dB and 27.03 dB are reported for HINet and MPRNet-local in the qualitative deblurring comparison.
  • The additional figure comparisons include image deblurring and image denoising visual results.
  • The baselines are described as restoring more fine details than other methods, with zooming recommended for the red-box regions.
  • 24.99 dB, 38.41 dB, and 41.45 dB are reported for the noisy input, MPRNet, and NAFNet(ours), respectively.
Loading 2204.04676v4…