Source-linked AI summary
Multi-level Wavelet-CNN for Image Restoration
Pengju Liu, Hongzhi Zhang, Kai Zhang, Liang Lin, Wangmeng Zuo
TL;DR
Image restoration needs larger receptive fields without the computational cost of conventional CNN expansion or the sparse sampling caused by dilated filtering. MWCNN addresses this with a multi-level wavelet U-Net using DWT/IWT for information-preserving subsampling and reconstruction, and experiments report effectiveness across denoising, SISR, and JPEG artifact removal.
Problem
CNN receptive-field enlargement increases computational cost, while dilated filtering suffers from gridding and sparse checkerboard sampling.
Method
MWCNN inserts CNN blocks between multi-level DWT operations in a contracting network and uses IWT with CNN blocks in the expanding network.
Results
MWCNN demonstrates effectiveness and efficiency on image denoising, single image super-resolution, and JPEG image artifacts removal.
Takeaways & Limitations
Wavelet-based subsampling lets MWCNN enlarge receptive fields while preserving information and targeting a better efficiency–performance tradeoff.
Takeaways & Limitations
Higher wavelet decomposition levels deepen the network and increase computational burden, so a suitable level is needed to balance efficiency and performance.
Abstract
from arXiv · showhide
The tradeoff between receptive field size and efficiency is a crucial issue in low level vision. Plain convolutional networks (CNNs) generally enlarge the receptive field at the expense of computational cost. Recently, dilated filtering has been adopted to address this issue. But it suffers from gridding effect, and the resulting receptive field is only a sparse sampling of input image with checkerboard patterns. In this paper, we present a novel multi-level wavelet CNN (MWCNN) model for better tradeoff between receptive field size and computational efficiency. With the modified U-Net architecture, wavelet transform is introduced to reduce the size of feature maps in the contracting subnetwork. Furthermore, another convolutional layer is further used to decrease the channels of feature maps. In the expanding subnetwork, inverse wavelet transform is then deployed to reconstruct the high resolution feature maps. Our MWCNN can also be explained as the generalization of dilated filtering and subsampling, and can be applied to many image restoration tasks. The experimental results clearly show the effectiveness of MWCNN for image denoising, single image super-resolution, and JPEG image artifacts removal.
1. Introduction
Image restoration seeks to recover clean images from degraded observations, but enlarging CNN receptive fields typically increases computation, while dilated filtering introduces sparse checkerboard sampling. MWCNN uses wavelet-based U-Net processing to enlarge receptive fields with a better efficiency–performance tradeoff and achieves state-of-the-art results across three restoration tasks.
- Image restoration recovers a latent clean image from a degraded observation and is a fundamental low-level vision problem.
- Larger receptive fields provide more spatial context, but increasing depth or filter size in pooling-free CNNs raises computational cost.
- Dilated filtering enlarges receptive fields without sacrificing computational cost but suffers from gridding, producing sparse checkerboard sampling of the input.
- MWCNN replaces pooling with discrete wavelet transforms in a U-Net, uses inverse wavelet transforms for upsampling, and combines subnetworks by elementwise summation.The wavelet transforms are invertible, preserving information during downsampling; DWT also captures frequency and location information.
- MWCNN is presented as a receptive-field design with a better tradeoff between efficiency and restoration performance.
- MWCNN is reported to provide promising detail-preserving ability through the time-frequency localization of DWT.
- MWCNN achieves state-of-the-art performance on image denoising, single image super-resolution, and JPEG image artifacts removal.
2. Related work
Related work develops CNN-based approaches for denoising, super-resolution, JPEG artifact removal, and broader restoration, with recurring efforts to enlarge receptive fields or incorporate wavelets. CNN denoisers can also transfer across restoration tasks and serve as plug-and-play components.
- CNN denoising progressed from early methods to residual and symmetric-skip architectures, with DnCNN reported to outperform traditional non-CNN methods.
- SISR research enlarged receptive fields through deeper, residual, pyramid, and recursive architectures, often with increased computation or information loss.
- JPEG artifact removal methods include ARCNN deblocking, dual-domain convolutional networks, and GAN-based approaches for more realistic results.
- CNN models developed for one restoration task can often be adapted to denoising, SISR, and JPEG artifact removal by retraining.
- CNN denoisers can function as plug-and-play priors within unrolled inference frameworks for additional restoration tasks.
- Wavelet-CNN studies use wavelet subbands for denoising, super-resolution, and low-dose CT-related convolutional framelet methods.
3. Method
MWCNN extends multi-level wavelet decomposition into a U-Net-like restoration network by inserting CNN blocks between wavelet transforms. DWT/IWT replace pooling and up-convolution, while subband processing connects the architecture to subsampling and dilated filtering without their stated information-loss and gridding drawbacks.
- From multi-level WPT to MWCNN: DWT decomposes an image into four subbands and, because it is invertible, IWT can accurately reconstruct the original image.The four filters produce downsampled subband images, while biorthogonality supports exact reconstruction.
- From multi-level WPT to MWCNN: MWCNN extends multi-level WPT by inserting CNN blocks between successive levels of DWTs to learn compact subband representations.WPT is described as a special case without CNN blocks; MWCNN processes all subbands after each transform.
- Network architecture: The architecture uses contracting and expanding subnetworks, with DWT/IWT replacing conventional pooling and up-convolution.The architecture is presented as a modified U-Net with wavelet-based resolution changes.
- Network architecture: After wavelet downsampling increases feature-map channels, subsequent CNN blocks reduce them for compact representations; contracting and expanding features are combined by summation.The final network contains 24 layers, and the configuration can be duplicated to extend it to higher wavelet levels.
- Connection to subsampling: Using all subbands lets MWCNN avoid the information loss associated with conventional subsampling, whereas low-frequency-only processing makes DWT and IWT analogous to pooling and up-convolution.With Haar wavelets, the low-pass branch corresponds to sum-pooling; all subbands retain additional information.
- Connection to dilated filtering: A 3 × 3 dilation-2 convolution can be treated as a special case of a 4 × 3 × 3 convolution over wavelet subband images.The paper uses this relationship to explain MWCNN as a generalization of dilated filtering while avoiding sparse checkerboard sampling and inconsistent local information.
4. Experiments
The experiments evaluate MWCNN on image denoising, single image super-resolution, and compression artifacts removal, while also comparing MWCNN variants.
- MWCNN is evaluated on image denoising, SISR, and compression artifacts removal.
- Several MWCNN variants are compared to analyze the contribution of each component.
4.1. Experimental setting
The training setup combines images from three datasets, task-specific degradation settings, and ADAM-based optimization with augmentation.
- The training set contains 200 BSD images, 800 DIV2K images, and 4,744 WED images.Training uses N = 24 × 6,000 cropped 240 × 240 patches because MWCNN's receptive field is at least 226 × 226.
- Denoising uses Gaussian noise levels σ = 15, 25, and 50, while SISR uses scale factors ×2, ×3, and ×4.For SISR, bicubic-upsampled images are input to MWCNN.
- JPEG artifact removal evaluates four compression quality settings, while each degradation setting receives a separately learned MWCNN model.
- ADAM optimization uses α = 0.01, β1 = 0.9, β2 = 0.999, ϵ = 10−8, and mini-batches of 24.The learning rate decays exponentially from 0.001 to 0.0001 over 40 epochs, with rotation and flip augmentation.
4.2. Quantitative and qualitative evaluation
Across denoising, SISR, and JPEG artifact removal, MWCNN generally achieves favorable quantitative and visual restoration results, while offering a competitive runtime tradeoff.
- 4.2.1 Image denoising: MWCNN slightly exceeds DnCNN by about 0.1∼0.3dB PSNR on BSD68 and by 0.5dB on Set12 and 1.2dB on Urban100 at σ = 50.
- The denoising, SISR, and JPEG evaluations use average PSNR/SSIM comparisons across their respective datasets and degradation settings.
- 4.2.2 Single image super-resolution: MWCNN achieves favorable PSNR and SSIM for SISR, gaining about 0.4dB over VDSR on Set5 and Set14 and 0.9∼1.4dB on Urban100.At scale factor 4, MWCNN slightly exceeds SRResNet on Set5 and BSD100 and is comparable on Set14.
- 4.2.3 JPEG image artifacts removal: For JPEG artifact removal, MWCNN performs favorably across four quality factors on Classic5 and LIVE1, exceeding MemNet by 0.2∼0.3dB PSNR at quality factors 10 and 20.Visual examples report restored detailed textures and sharp salient edges.
- 4.2.4 Run time: MWCNN runs far faster than RED30, MemNet, and DRRN, while other methods are moderately faster but achieve lower PSNR/SSIM.The authors attribute the effectiveness to incorporating CNN and DWT rather than increasing network depth or width.
4.3. Comparison of MWCNN variants
The experiments compare wavelet choices, connection and downsampling ablations, dilated CNNs, and decomposition levels using PSNR and runtime. MWCNN (Haar) offers the strongest reported tradeoff, while deeper decomposition requires balancing performance against computational burden.
- Variant and baseline comparisons: MWCNN (Haar) achieves higher PSNR than dilated CNN and U-Net at similar runtime, supporting its performance–efficiency tradeoff.The comparison also favors Haar over DB2 and the hybrid Haar/DB2 variant in quantitative and qualitative evaluation.
- Variant and baseline comparisons: Sum connections improve efficiency without decreasing PSNR in the ablation experiments.The experiments also compare standard U-Net, sum connections, and learnable downsampling against the embedded-wavelet design.
- Variant and baseline comparisons: Independent processing of wavelet subbands harms image-restoration results, as indicated by the weaker DCF performance.MWCNN instead embeds DWT to enlarge the receptive field without information loss.
- Decomposition-level analysis: MWCNN-3 performs much better than MWCNN-1 and MWCNN-2, while MWCNN-4 provides only negligible additional PSNR improvement.MWCNN-3 also has moderate speed and is selected as the default setting.
5. Conclusion
The paper presents MWCNN, a multi-level architecture that combines DWT and IWT with CNN blocks for image restoration. It reports effectiveness and efficiency across denoising, SISR, and JPEG compression artifact removal, while identifying broader restoration tasks as future work.
- Conclusion: MWCNN uses multiple levels of DWT and CNN blocks in contraction and IWT and CNN blocks in expansion.The invertibility and frequency-location properties of DWT support subsampling without information loss and recovery of detailed textures and sharp structures.
- Conclusion: Experiments demonstrate MWCNN's effectiveness and efficiency on image denoising, SISR, and JPEG compression artifact removal.These are the three restoration tasks explicitly identified in the conclusion.
- Conclusion: Image deblurring and blind deconvolution remain identified as future extensions for MWCNN.The paper also notes possible use of MWCNN in pooling replacements for high-level vision tasks such as classification.