PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-level Wavelet Convolutional Neural Networks

Pengju Liu, Hongzhi Zhang, Wei Lian, Wangmeng Zuo

arXiv:1907.03128v1cs.CVeess.IV

TL;DR

CNNs need large receptive fields without the information loss of pooling or the sparse sampling of dilated filters. The paper introduces MWCNN, embedding invertible wavelet transforms and inverse wavelet reconstruction into CNNs; it reports state-of-the-art results across restoration and classification tasks. MWCNN is also presented as a general replacement for pooling-based downsampling.

  • Problem

    Pooling enlarges receptive fields efficiently but can lose information, whereas dilated filters can produce sparse checkerboard sampling; the paper addresses this receptive-field–efficiency trade-off.

  • Method

    MWCNN replaces pooling with discrete wavelet transforms and uses inverse wavelet transforms to reconstruct high-resolution features within a U-Net-based restoration architecture.

  • Results

    MWCNN is reported to achieve state-of-the-art performance on image denoising, single-image super-resolution, JPEG artifact removal, and object classification.

  • Takeaways & Limitations

    The method provides a general approach for embedding wavelet transforms in CNNs that employ pooling operations.

Abstract

from arXiv · show

In computer vision, convolutional networks (CNNs) often adopts pooling to enlarge receptive field which has the advantage of low computational complexity. However, pooling can cause information loss and thus is detrimental to further operations such as features extraction and analysis. Recently, dilated filter has been proposed to trade off between receptive field size and efficiency. But the accompanying gridding effect can cause a sparse sampling of input images with checkerboard patterns. To address this problem, in this paper, we propose a novel multi-level wavelet CNN (MWCNN) model to achieve better trade-off between receptive field size and computational efficiency. The core idea is to embed wavelet transform into CNN architecture to reduce the resolution of feature maps while at the same time, increasing receptive field. Specifically, MWCNN for image restoration is based on U-Net architecture, and inverse wavelet transform (IWT) is deployed to reconstruct the high resolution (HR) feature maps. The proposed MWCNN can also be viewed as an improvement of dilated filter and a generalization of average pooling, and can be applied to not only image restoration tasks, but also any CNNs requiring a pooling operation. The experimental results demonstrate effectiveness of the proposed MWCNN for tasks such as image denoising, single image super-resolution, JPEG image artifacts removal and object classification.

I. INTRODUCTION

The paper proposes MWCNN, which replaces pooling with invertible wavelet transforms to enlarge receptive fields while preserving feature information and balancing restoration performance with efficiency.

  • Motivation: Pooling enlarges receptive fields efficiently but can lose information, while fixed-factor dilated filtering sparsely samples inputs and can create checkerboard patterns.Increasing depth or filter size also enlarges receptive fields but raises computational cost.
  • Proposed approach: MWCNN replaces pooling with discrete wavelet transform, using its invertibility to avoid losing image or intermediate-feature information during downsampling.The method captures both frequency and location information through multi-frequency feature representations.
  • Proposed approach: For image restoration, MWCNN uses a U-Net backbone and inverse wavelet transform with expansion convolution to reconstruct high-resolution feature maps.Element-wise summation combines feature maps in the reconstruction process.
  • Contributions: The paper extends prior work by broadening the approach, applying it to high-level tasks, and adding further analysis and discussion.It also interprets dilated filtering as a special variant of MWCNN.
  • Contributions: The approach is presented as a general way to embed wavelet transforms in CNNs that use pooling operations.The contributions include a better trade-off among receptive field size, efficiency, and restoration performance.
  • Results: MWCNN is reported to achieve state-of-the-art performance across image denoising, single-image super-resolution, JPEG artifact removal, and classification.Figure 1 compares running time, PSNR, and receptive field on Set5 at ×4 scale using a GTX1080 GPU.

II. RELATED WORK

Related work develops CNN-based approaches for restoration and classification, including deeper, dilated, pooled, and U-Net-based architectures aimed at improving receptive fields and efficiency.

  • Scope: The reviewed literature includes CNNs for both image restoration and object classification, with the paper focusing particularly on wavelet-transform integration.The related-work discussion also introduces classification methods after reviewing restoration approaches.
  • Image restoration: Image restoration methods seek to recover a latent clean image from a degraded observation using prior modeling or discriminative learning.CNN-based methods later achieved state-of-the-art performance over traditional methods.
  • Image restoration: Early CNN restoration methods used shallow or pooling-free architectures, while later work introduced deeper networks and architectural mechanisms to enlarge receptive fields.Examples include residual learning, very deep networks, skip connections, residual units, pyramidal structures, and recursive architectures.
  • Efficient restoration networks: Dilated filtering was introduced to improve the speed–performance trade-off, while other approaches used pooling/unpooling, downsampled subimages, or U-Net-based denoisers.These methods target multi-context representation or more efficient restoration processing.
  • Cross-task restoration: Models developed for denoising, super-resolution, and JPEG artifact removal can often be extended across these restoration tasks by retraining the same network.CNN denoisers have also been used as plug-and-play priors within unrolled inference.

3) Incorporating DWT in CNNs:

MWCNN extends multi-level wavelet packet transform by inserting CNN blocks into a reversible, multi-level decomposition and reconstruction pipeline. DWT captures low- and high-frequency subbands while IWT reconstructs features without information loss.

  • Relation to prior wavelet CNNs: MWCNN uses wavelet subbands as multi-frequency feature representations, extending earlier one-level wavelet CNN methods to multi-level decomposition.Prior WavResNet and DWSR approaches are described as using only one-level wavelet decomposition.
  • Wavelet decomposition: 2D DWT uses four fixed stride-2 filters to decompose an image into one low-pass and three high-pass subband images.For Haar wavelets, the filters are fLL, fLH, fHL, and fHH.
  • From multi-level WPT to MWCNN: MWCNN inserts CNN blocks into multi-level WPT, reducing to WPT when each CNN block is an identity mapping.This makes WPT a special case of MWCNN and MWCNN a generalization of multi-level WPT.
  • Wavelet decomposition: Because DWT is invertible, IWT can accurately reconstruct the original image from its four subbands without information loss.The same decomposition and inverse reconstruction can be recursively applied across multiple WPT levels.
  • Architecture: The architecture contains contracting and expanding subnetworks, with 24 convolutional layers and extension to higher levels by duplicating the third-level configuration.Each solid box represents a multi-channel feature map.

1) Image Restoration:

For image restoration, MWCNN replaces U-Net pooling and deconvolution with DWT and IWT while inserting CNN blocks around multi-level wavelet transforms. This preserves subband information during resolution changes and supports compact feature modeling.

  • Image Restoration: MWCNN for image restoration is built on U-Net and inserts CNN blocks before or after each DWT level.Each CNN block is a three-layer FCN using 3×3 convolutions and ReLU, with convolution only in the final residual-prediction layer.
  • Image Restoration: DWT and IWT replace conventional U-Net pooling and deconvolution as MWCNN’s downsampling and upsampling operations.Convolutions reduce or increase feature-map channels around these transforms.
  • Image Restoration: DWT can replace pooling in CNNs, followed by a 1×1 compression convolution, without requiring changes to other blocks or the loss function.The paper presents DWT as a plug-in downsampling module that transmits feature information without loss.
  • Image Restoration: MWCNN retains low- and high-frequency subbands as CNN inputs, then uses IWT to restore feature-map resolution.This design is intended to preserve detailed texture while processing multi-frequency representations.

1) Connection to Pooling Operation:

MWCNN embeds wavelet decomposition and reconstruction into CNNs to reduce feature-map resolution while retaining frequency and location information. This generalizes average pooling and avoids information-loss and gridding drawbacks associated with conventional subsampling and dilated filtering.

  • Connection to Pooling Operation: Average pooling retains only the low-frequency DWT component, whereas MWCNN uses all four subbands to preserve information during downsampling.Pooling sums the four sub-images with coefficient 1/4; DWT produces four sub-images using fixed orthometric weights.
  • Connection to Dilated Filtering: Dilated filtering with factor 2 decomposes an image into four sub-images and applies a shared standard convolutional kernel to them.The operation is equivalent to sampling positions separated by one interval pixel, then convolving each sampled patch.
  • Connection to Dilated Filtering: Unlike dilated filtering, MWCNN uses distinct subband convolutions and IWT reconstruction, while dilated filtering is a variant with shared convolution weights.This relationship makes MWCNN more general than dilated filtering in the paper’s formulation.
  • Connection to Dilated Filtering: Dilated filtering can lose information through checkerboard sampling and produce neighboring outputs from non-overlapping inputs; MWCNN avoids these drawbacks.Figure 5 contrasts the sparse sampling and local inconsistency of rate-2 dilation with MWCNN’s receptive field.

IV. EXPERIMENTS

The experiments extend MWCNN beyond image restoration and examine component contributions through ablation studies before evaluating object classification.

  • IV. EXPERIMENTS: The experiments cover image restoration, ablation analysis, and an extension of MWCNN to object classification.The section sequence first describes restoration, then analyzes component contributions, and finally evaluates classification.

A. Experimental Setting for Image Restoration

Image-restoration experiments evaluate MWCNN across denoising, super-resolution, and JPEG artifact-removal settings using standard datasets and specified degradation conditions.

  • A. Experimental Setting for Image Restoration: Training uses DIV2K’s 800 training images, 100 validation images, and 100 test images, with 192 × 192 crops sampled for training.The training set contains approximately 2K-resolution images, and N = 20 × 4,000 patches are cropped.
  • A. Experimental Setting for Image Restoration: SISR uses bicubic-upsampled inputs at scale factors ×2, ×3, and ×4, evaluated on Set5, Set14, BSD100, and Urban100.The passage identifies four evaluation datasets for the three super-resolution scales.
  • A. Experimental Setting for Image Restoration: JPEG artifact removal follows compression quality settings Q = 10, 20, 30, and 40 using Classic5 and LI datasets.The settings follow the referenced prior experimental protocol.
  • A. Experimental Setting for Image Restoration: Optimization uses ADAM with β1 = 0.9, β2 = 0.999, ϵ = 10^-8, batch size 24, and an exponentially decayed learning rate from 10^-4 to 10^-5.Training runs for 200 epochs with rotation and flip augmentation on an NVIDIA GTX1080 GPU.

B. Quantitative and qualitative evaluation on Image Restoration Tasks

MWCNN achieves strong image-restoration results across denoising and single-image super-resolution, outperforming competing methods on reported PSNR/SSIM measures and recovering detailed textures.

  • B. Quantitative and qualitative evaluation on Image Restoration Tasks: A 24-layer MWCNN is evaluated on three representative image-restoration tasks using a common experimental setting.The experiments include image denoising, SISR and JPEG artifact removal.
  • 1) Image denoising:: MWCNN outperforms competing denoising methods across the reported datasets and noise levels.It exceeds DnCNN and FFDNet by about 0.3–0.5 dB on Set12 and 0.2–0.3 dB on BSD68 in PSNR.
  • 2) Single image super-resolution:: 0.4–0.8 dB: MWCNN’s PSNR gain over VDSR on Set5 and Set14 for single-image super-resolution.On Urban100, the reported gap over VDSR reaches about 1.0–1.6 dB.
  • 2) Single image super-resolution:: MWCNN outperforms other methods in both PSNR and SSIM across the reported SISR datasets and scale factors.The comparisons cover Set5, Set14, BSD100 and Urban100 with scale factors ×2, ×3 and ×4.
  • 2) Single image super-resolution:: MWCNN recovers fine textures and produces sharp edges in the BSD100 ×4 super-resolution example.The paper attributes this behavior to DWT’s frequency and location characteristics.

3) JPEG image artifacts removal:

For JPEG artifact removal, MWCNN delivers the best reported quantitative performance across quality factors and is evaluated alongside competing restoration methods and runtime comparisons.

  • 3) JPEG image artifacts removal:: The JPEG artifact-removal evaluation focuses on the luminance channel and uses quality factors Q = 10, 20, 30 and 40.MWCNN is compared with ARCNN, TNRD, DnCNN and MemNet.
  • 3) JPEG image artifacts removal:: MWCNN achieves superior PSNR/SSIM performance to competing methods for all four JPEG quality factors.The settings are Q = 10, 20, 30 and 40 on Classic5 and LIVE1.
  • 3) JPEG image artifacts removal:: 1 dB: MWCNN’s PSNR advantage over ARCNN on Classic5.It is also reported to exceed the second-best method, MemNet, by 0.2–0.3 dB.
  • 3) JPEG image artifacts removal:: MWCNN costs far less GPU time than the cited state-of-the-art methods across the three restoration tasks.Runtime measurements use cuDNNv7.0 with CUDA 9.2 on Ubuntu 16.04.
  • 3) JPEG image artifacts removal:: The ablation and related-method comparisons use 24-layer MWCNN as the baseline with matched architectures for fair comparison.The study examines performance sources and relationships to wavelet-based and dilated-filtering methods.

1) Ablation experiments:

Ablation studies attribute MWCNN’s restoration performance to invertible wavelet processing and compare wavelet choices, dilated variants, and network levels under efficiency constraints.

  • 1) Ablation experiments:: MWCNN outperforms U-Net+D, indicating that learnable stride-2 downsampling alone does not prevent information loss.The paper connects this difference to wavelet invertibility and preservation of high-frequency information.
  • 1) Ablation experiments:: Sum connections slightly improve efficiency with almost no PSNR decrease compared with concatenation.This is one of the reported ablation observations.
  • 1) Ablation experiments:: MWCNN remains slightly better than the deeper MWCNN(P+C), supporting the effectiveness of the proposed design despite its smaller depth.MWCNN(P+C) adds one CNN layer after the input and another before the output.
  • 1) Ablation experiments:: MWCNN(Haar) is the best quantitative choice among the tested Haar, DB2 and hybrid wavelet variants.The Haar variant has similar running time to dilated CNN and U-Net while achieving higher PSNR.
  • 1) Ablation experiments:: The comparisons report adverse restoration effects from gridding and from independently processing wavelet subbands.The tested alternatives include hybrid dilated CNNs, Dilated-2, WaveResNet and DCF variants.
  • 1) Ablation experiments:: Higher wavelet levels increase network depth and computational burden, motivating a selected level that balances efficiency and performance.The study reports PSNR and runtime for MWCNN levels 0 through 4.

D. Extend to Object Classification

MWCNN extends beyond restoration by replacing average pooling in CNN classifiers with DWT-based processing, improving results across several classification benchmarks and architectures.

  • D. Extend to Object Classification: The classification evaluation compares MWCNN-enabled and original CNNs across six named benchmarks and several established CNN architectures.Compared architectures include PreResNet, All-CNN, WideResNet, PyramidNet, DenseNet and ResNet.
  • D. Extend to Object Classification: DWT with a 1 × 1 convolution replaces average pooling in the classification implementation.The modified model is denoted MW, while the original CNN is denoted Base.
  • D. Extend to Object Classification: MWCNN surpasses the original CNN across the reported object-classification comparisons.The evaluation includes CIFAR-10, CIFAR-100, SVHN, MNIST, ImageNet32, ImageNet64 and Places365.
  • D. Extend to Object Classification: MWCNN is formulated as a generic plug-and-play module that directly replaces pooling without network-architecture adjustments.It takes all subbands as input to model inter-band dependency.
  • D. Extend to Object Classification: The conclusion reports effectiveness and efficiency across denoising, SISR, JPEG artifact removal and object classification.The reported design uses DWT to reduce spatial resolution and enlarge receptive field without information loss.
Loading 1907.03128v1…