Wavelet Convolutional Neural Networks

Shin Fujieda; Kohei Takayama; Toshiya Hachisuka

Source-linked AI summary

Wavelet Convolutional Neural Networks

Shin Fujieda, Kohei Takayama, Toshiya Hachisuka

arXiv:1805.08620v1May 20, 2018cs.CVcs.LG

TL;DR

Conventional CNNs primarily process spatial information and miss much of the spectral information available through multiresolution analysis. Wavelet CNNs integrate wavelet-based multiresolution components into CNNs, achieving better accuracy across texture classification and image annotation with fewer parameters than conventional CNNs. The decomposition level is coupled to network depth, limiting independent hyperparameter tuning.

Problem
Conventional CNNs capture spatial features but miss much spectral information available through multiresolution analysis, although spectral information is useful for image processing tasks.
Method
Wavelet CNNs combine CNNs with multiresolution analysis by adding wavelet-transform components that supplement the parts conventional CNNs omit.
Results
Wavelet CNNs achieve better accuracy in texture classification and image annotation with significantly fewer trainable parameters than conventional CNNs.
Takeaways & Limitations
The architecture uses spectral information within a single CNN model while reducing memory and training demands relative to conventional CNNs.
Takeaways & Limitations
The maximum decomposition level is restricted to five, and decomposition level cannot be tuned independently of network depth and trainable parameters.

Abstract

from arXiv · show

Spatial and spectral approaches are two major approaches for image processing tasks such as image classification and object recognition. Among many such algorithms, convolutional neural networks (CNNs) have recently achieved significant performance improvement in many challenging tasks. Since CNNs process images directly in the spatial domain, they are essentially spatial approaches. Given that spatial and spectral approaches are known to have different characteristics, it will be interesting to incorporate a spectral approach into CNNs. We propose a novel CNN architecture, wavelet CNNs, which combines a multiresolution analysis and CNNs into one model. Our insight is that a CNN can be viewed as a limited form of a multiresolution analysis. Based on this insight, we supplement missing parts of the multiresolution analysis via wavelet transform and integrate them as additional components in the entire architecture. Wavelet CNNs allow us to utilize spectral information which is mostly lost in conventional CNNs but useful in most image processing tasks. We evaluate the practical performance of wavelet CNNs on texture classification and image annotation. The experiments show that wavelet CNNs can achieve better accuracy in both tasks than existing models while having significantly fewer parameters than conventional CNNs.

1. Introduction

Wavelet CNNs combine CNN-based spatial processing with multiresolution spectral analysis to recover spectral information that conventional CNNs largely miss. The model is presented as an accurate and efficient approach for texture classification and image annotation.

CNNs capture spatial features, whereas spectral analyses capture scale-invariant features from spectral information.
Conventional CNNs can be viewed as limited multiresolution analyses that miss substantial spectral information.
Wavelet CNNs combine CNNs and multiresolution analysis within a single model.
The architecture reformulates CNNs as a limited form of multiresolution analysis and supplements its missing components.
Wavelet CNNs provide accurate and efficient texture classification and image annotation.

2. Related Work

Related work contrasts CNNs' strong spatial representation with spectral methods' explicit frequency selectivity and reviews parameter-efficient CNN-based alternatives. The supplied passages position wavelet CNNs as combining these perspectives while reducing parameter requirements.

Wavelet CNNs are illustrated as combining CNN processing with multiresolution decomposition and channel-wise concatenation.The architecture uses convolutional layers alongside decomposed images from multiresolution analysis.
Residual and Dense Networks use shortcut connections to address information and gradient propagation problems in deeper CNNs.DenseNet connects each layer with all previous layers, but these architectures require substantial computational resources because of their many parameters.
CNN feature extractors combined with Fisher Vectors or compact bilinear pooling improve or compress pooling and encoding, but retain substantial trainable parameters.Wavelet CNNs are reported to achieve competitive results with compact bilinear pooling while further reducing parameters in texture classification.
Spectral approaches use spatial filters to represent image statistics across scales and orientations, with explicit frequency selection that is difficult to control in CNNs.
Wavelet CNNs are reported to become equivalent to the proposed model with available datasets in practice.

3. Wavelet Convolutional Neural Networks

The paper reformulates CNN convolution and pooling as filtering and downsampling, showing that conventional CNNs retain only part of a multiresolution analysis. Wavelet CNNs restore the missing decomposition through wavelet filters and multiscale connections.

Convolutional Neural Networks: CNN convolution computes weighted sums of neighboring inputs, with shared weights reducing parameters and providing translation invariance.The convolution operation is equivalent to filtering with a shared kernel.
Convolutional Neural Networks: Average pooling performs convolution with an averaging filter followed by downsampling, reducing the number of outputs.With p = 2, pairwise averaging halves the number of outputs.
Generalized Convolution and Pooling: CNNs can be interpreted as a limited multiresolution analysis because they discard high-frequency components and retain only one kernel pathway.A full multiresolution analysis hierarchically decomposes low- and high-frequency components at each level.
Generalized Convolution and Pooling: Wavelet CNNs supplement the discarded components with an additional filter pathway, embedding multiresolution analysis inside the network.The model uses low-pass and high-pass filters with skip connections to form the wavelet-based decomposition.
Generalized Convolution and Pooling: Wavelet multiresolution analysis requires paired scaling and wavelet functions rather than arbitrary filters; the experiments use Haar wavelets.The architecture is not restricted to Haar wavelets, but the filter constraint distinguishes it from unconstrained CNN weights.
Implementation: The implementation uses VGG-like 3×3 convolutions, stride-based downsampling, dense connections, and projection shortcuts to combine feature maps across decomposition levels.Feature maps are connected to decomposed images of matching sizes, while dense connections expose each level to subsequent layers.

4. Experiments

Experiments evaluate wavelet CNNs on texture classification and image annotation, comparing accuracy with existing models and assessing parameter efficiency. Across these tasks, the model generally performs competitively or better while using substantially fewer parameters.

Experimental setup: Wavelet CNNs were evaluated on texture classification and image annotation using the kth-tips2-b, DTD, IAPR-TC12, and Microsoft COCO datasets.Texture classification used two texture benchmarks, while image annotation used two natural-image datasets.
Texture Classification: 5-level decomposition performed best on both texture datasets when wavelet CNNs were trained from scratch.The 4-level model achieved almost the same accuracy as the 5-level model.
Texture Classification: Wavelet CNNs trained from scratch performed better than AlexNet and T-CNN by a large margin on both texture datasets.The comparison used AlexNet as a conventional CNN baseline and T-CNN as a texture-specialized model.
Texture Classification: With ImageNet pre-training, the wavelet CNN achieved the best performance on both texture datasets against shearlet transform, VGG-M, T-CNN, and TS+VGG-M.The DTD improvement was marginal, at less than 1%, but used significantly fewer parameters than the other methods.
Image Annotation: In image annotation, RIA with the wavelet CNN obtained much better results than original RIA on IAPR-TC12, while improvement on MS-COCO was marginal.Evaluation included per-class and overall precision, recall, and F1 metrics.
Efficiency: The wavelet CNN used 53.9 MB of memory and about half as many parameters as T-CNN, while using more than seven times fewer parameters than VGG-16 in annotation experiments.The reported memory figures were 392 MB for VGG-M, 232 MB for AlexNet, 89.1 MB for T-CNN, and 53.9 MB for the proposed model.

5. Discussion

The discussion extends wavelet CNNs beyond texture classification and image annotation while emphasizing their parameter efficiency. It also identifies constraints linking decomposition depth to network depth and notes difficulty explaining texture-classification failures.

Application to more general tasks: 59.4% accuracy versus AlexNet’s 57.1% was achieved on ImageNet classification with a model using about four times fewer parameters.The authors describe the smaller memory footprint as suitable for image classification.
Pooling: Replacing average pooling with Lp pooling preserves the broader formulation but no longer retains all input frequency information as a multiresolution analysis.The paper focuses on average pooling because of its clear connection to multiresolution analysis.
Limitations: The maximum decomposition level is restricted to five because high-frequency parts are inserted between four CNN layers that reduce feature-map size.This couples decomposition level, network depth, and trainable-parameter count.
Texture classification: Wavelet CNNs produced the best texture-classification accuracy both when trained from scratch and when fine-tuned.The fine-tuning advantage was slight on DTD, despite significantly fewer parameters.
Limitations: Texture-classification failure cases are difficult to explain for neural networks generally, and wavelet CNNs are not an exception.The authors speculate that ImageNet pre-training may be inappropriate for texture classification.

6. Conclusion

The paper presents wavelet CNNs as a CNN architecture that incorporates spectral analysis by reformulating convolution and pooling as filtering and downsampling. It reports better accuracy with fewer trainable parameters, while leaving trainable wavelet kernels as future work.

Conclusion: Wavelet CNNs incorporate spectral analysis into CNNs by integrating multiresolution analysis as a single model.The architecture follows a reformulation of conventional CNN operations as filtering and downsampling.
Conclusion: Wavelet CNNs achieved better accuracy for texture classification and image annotation with fewer trainable parameters than existing models.The paper reports especially large margins over models with significantly more parameters when training from scratch.
Future work: Training the wavelet kernels end-to-end remains an open direction because the model uses fixed-weight Haar wavelet kernels.The authors identify learnable wavelet kernels as future work.