PaperlayerView exact revision on arXiv

Source-linked AI summary

Wavelet Convolutions for Large Receptive Fields

Shahaf E. Finder, Roy Amoyal, Eran Treister, Oren Freifeld

arXiv:2407.05848v2cs.CV

TL;DR

Large CNN kernels become over-parameterized and saturate before reaching a global receptive field. This paper introduces WTConv, which uses cascaded wavelet decomposition and small-kernel convolutions to expand receptive fields with logarithmic parameter growth. WTConv is a drop-in replacement evaluated across classification, segmentation, and detection, with reported gains in effective receptive fields, shape bias, corruption robustness, and task performance.

  • Problem

    Large CNN kernels become over-parameterized and performance saturates before achieving a global receptive field.

  • Method

    WTConv uses cascaded Wavelet Transform decomposition and small-kernel convolutions across frequency bands as a drop-in replacement for depth-wise convolutions.

  • Results

    WTConv substantially increases CNN effective receptive fields, improves shape bias and corruption robustness, and yields better performance across vision tasks.

  • Takeaways & Limitations

    WTConv provides a convolutional way to configure global-receptive-field spatial mixing while retaining applicability across CNN architectures and tasks.

  • Takeaways & Limitations

    WTConv can have relatively high running time in existing frameworks because of overhead from multiple sequential WT-conv-IWT operations.

Abstract

from arXiv · show

In recent years, there have been attempts to increase the kernel size of Convolutional Neural Nets (CNNs) to mimic the global receptive field of Vision Transformers' (ViTs) self-attention blocks. That approach, however, quickly hit an upper bound and saturated way before achieving a global receptive field. In this work, we demonstrate that by leveraging the Wavelet Transform (WT), it is, in fact, possible to obtain very large receptive fields without suffering from over-parameterization, e.g., for a $k \times k$ receptive field, the number of trainable parameters in the proposed method grows only logarithmically with $k$. The proposed layer, named WTConv, can be used as a drop-in replacement in existing architectures, results in an effective multi-frequency response, and scales gracefully with the size of the receptive field. We demonstrate the effectiveness of the WTConv layer within ConvNeXt and MobileNetV2 architectures for image classification, as well as backbones for downstream tasks, and show it yields additional properties such as robustness to image corruption and an increased response to shapes over textures. Our code is available at https://github.com/BGU-CS-VIL/WTConv.

1 Introduction

WTConv addresses the difficulty of expanding CNN receptive fields without over-parameterization by using wavelet-based multi-frequency convolutions. It is designed as a drop-in layer and is evaluated across image classification and downstream vision tasks.

  • Large CNN kernels become over-parameterized and performance saturates before reaching a global receptive field.
  • WTConv uses cascaded Wavelet Transform decomposition and small-kernel convolutions across frequency bands to expand receptive fields efficiently.The method emphasizes low frequencies while preserving spatial resolution for spatial operations.
  • For a k × k receptive field, WTConv’s trainable parameter count grows only logarithmically with k, unlike quadratic growth in some recent methods.
  • WTConv is a drop-in replacement for depth-wise convolutions that can be incorporated into existing CNN architectures without additional modifications.The evaluation includes ConvNeXt for image classification, UperNet for semantic segmentation, and Cascade Mask R-CNN for object detection.
  • The paper reports that WTConv improves CNN scalability, robustness, and shape bias across computer-vision tasks.The contributions include empirical evaluation of these properties and WTConv’s use in CNNs.

2 Related Work

Prior work enlarged CNN receptive fields through deeper networks, separable convolutions, transformers, and frequency-domain methods. WTConv builds on wavelet-based approaches while retaining local spatial information and fitting networks that use depth-wise convolution.

  • WTConv extends wavelet-based CNN methods as a depth-wise-convolution-compatible layer that is not limited to a single task.
  • CNNs traditionally increased receptive fields mainly by adding depth while retaining small kernels such as 3 × 3 convolutions.
  • Separable convolutions perform spatial operations per channel with depth-wise kernels and cross-channel operations using 1 × 1 point-wise kernels.
  • Vision Transformers renewed interest in larger CNN kernels because non-local self-attention typically outperformed local-mixing convolutions on vision tasks.
  • Fourier-based spatial mixing can achieve global receptive fields but loses local neighboring-pixel interactions and may depend on a specific input size.Wavelet Transform methods preserve some local information while decomposing inputs into frequency bands.

3 Method

WTConv uses cascaded wavelet decomposition and small-kernel convolutions across frequency bands to enlarge receptive fields efficiently. Its receptive field grows exponentially with decomposition levels while parameters grow only linearly, alongside stronger low-frequency response and favorable computational scaling.

  • 3.1 Preliminaries: The Wavelet Transform as Convolutions: The Haar wavelet transform applies four depth-wise 2D filters with stride 2, producing low-, horizontal-, vertical-, and diagonal high-frequency components at half spatial resolution.The low-pass filter produces XLL; the other filters produce XLH, XHL, and XHH.
  • 3.1 Preliminaries: The Wavelet Transform as Convolutions: Cascade wavelet decomposition recursively decomposes the low-frequency component, increasing frequency resolution while reducing lower-frequency spatial resolution.At each level, WT is applied to the preceding level’s low-frequency component.
  • 3.2 Convolution in the Wavelet Domain: WTConv filters and downsamples frequency components, applies small-kernel depth-wise convolutions, and uses the inverse wavelet transform to reconstruct the output.The operation separates frequency-specific convolutions and lets smaller kernels cover a larger area of the original input.
  • 3.2 Convolution in the Wavelet Domain: The cascade combines outputs across frequency levels through inverse-wavelet reconstruction, aggregating the convolutions from different levels.The aggregation uses the linearity of the wavelet transform and its inverse.
  • 3.3 The Benefits of Using WTConv: ℓ-level WTConv uses ℓ·4·c·k^2 parameters while its receptive field grows as 2^ℓ·k, so capacity scales linearly with levels and coverage exponentially.The layer also emphasizes low frequencies because repeated decomposition of low-frequency inputs increases their corresponding response.
  • 3.4 Computational Cost: A 3-level WTConv with a 5 × 5 kernel covers a 40 × 40 receptive field using 15.1M convolution FLOPs and 17.9M total FLOPs including WT and IWT.The total remains below the 252M FLOPs of a comparable 31 × 31 depth-wise convolution in the cited 512 × 512 single-channel example.

4 Results

WTConv is evaluated in classification, downstream vision tasks, scalability, robustness, shape bias, and effective receptive field. Across these settings, it improves results while adding relatively few parameters and achieves nearly global receptive fields.

  • ImageNet-1K Classification: WTConv achieves the best results among top-scoring methods while remaining the most parameter-efficient, reaching a global receptive field with less than half of GFNet’s parameters.The comparison uses the same ConvNeXt-T base architecture and reports parameters for depth-wise convolutions.
  • ImageNet-1K Classification: WTConvNeXt improves ConvNeXt classification accuracy with only slight increases in parameters and FLOPs.Moving to WTConvNeXt-S adds 4M parameters and 0.1 GFLOPs for a 0.5% accuracy gain, compared with 39M parameters and 6.7 GFLOPs for a 0.7% gain when moving from ConvNeXt-S to ConvNeXt-B.
  • Semantic Segmentation: WTConv improves ADE20K semantic-segmentation mIoU by 0.3-0.6% when used as a UperNet backbone.The downstream evaluation follows ConvNeXt’s configuration without parameter tuning.
  • Object Detection: WTConv increases both APbox and APmask by 0.6-0.7% for Cascade Mask R-CNN on COCO.The evaluation uses ConvNeXt’s training configuration without parameter tuning.
  • Scalability: WTConv scales better than RepLK as the receptive field increases, while GFNet suffers from over-parameterization and FFC from limited frequency mixing.The scalability analysis uses MobileNetV2 on ImageNet-50/100/200 with alternative depth-wise convolutions.
  • Robustness: WTConvNeXt gains above 1% on most robustness datasets, reaching 2.2%, despite only a 0.3-0.4% ImageNet-1K accuracy advantage.A similar trend appears in corrupted object detection, and the authors relate it to improved responses to low frequencies.
  • Shape Bias: WTConv increases shape decisions by 8-12% and makes even WTConvNeXt-T more shape-biased than larger ConvNeXt networks.The authors associate this effect with WTConv’s increased emphasis on lower frequencies.
  • Effective Receptive Field: WTConv produces a nearly global effective receptive field while using fewer parameters than RepLK and SLaK.The effective receptive-field evaluation measures pixel contributions to the final feature-map center using gradients from 50 resized ImageNet validation images.

5 Limitations

WTConv can have relatively high running time in existing frameworks despite requiring few FLOPs. The overhead comes from multiple sequential WT-conv-IWT operations.

  • Limitations: WTConv’s runtime can be relatively high in existing frameworks because multiple sequential WT-conv-IWT operations add overhead.Parallel or in-place implementations could reduce memory reads or allocations.

6 Conclusion

WTConv is presented as a drop-in depth-wise convolution replacement that enlarges receptive fields, captures low frequencies, and supports pure-convolution spatial mixing. Its algorithm initializes the final output and processes decomposition levels in reverse order.

  • WTConv enlarges receptive fields and better captures low frequencies as a drop-in replacement for depth-wise convolutions.
  • The layer can configure spatial mixing with a global receptive field using only convolutions.
  • The algorithm initializes Z^(ℓ+1) to zero before iterating through decomposition levels in reverse order.

B Training Parameters

The experiments use distinct training configurations, including SGD for one setup and AdamW with extensive augmentation and regularization for another. Both schedules use warmup followed by cosine annealing.

  • The first setup uses SGD with momentum 0.9, batch size 32 per GPU, 224×224 inputs, and 100 epochs after a 5-epoch warmup.Its initial learning rate is 0.025 and weight decay is 4·10^-5.
  • The second setup follows a 300-epoch AdamW schedule with momentum 0.9, weight decay 0.05, batch size 4096, and learning rate 4 · 10^-3.It uses a 20-epoch linear warm-up followed by cosine annealing.
  • The 300-epoch setup applies RandAugment, label smoothing, mixup, CutMix, Random Erasing, Stochastic Depth, and exponential moving average.

C Naive implementation running times

The authors characterize a naive implementation’s efficiency and compatibility across architectures. Despite lacking optimization, WTConvNeXt retains substantial throughput and WTConv is incorporated into GhostNet and EfficientNet.

  • The implementation is described as fairly naive, with Haar wavelet operations represented as FP32 convolutions involving multiplications by 1 and -1.The authors identify summation and subtraction as more efficient alternatives that could process all levels simultaneously.
  • 66-70% of the original network’s throughput is achieved by WTConvNeXt with the most naive and unoptimized layer implementation.Throughput was measured in images per second on a single RTX3090 after GPU warmup.
  • WTConv is incorporated into GhostNet and EfficientNet with levels chosen to provide a global receptive field at each stage for 224×224 inputs.

E quantitative shape-bias results

The paper reports quantitative shape-bias results for WTConv, with the detailed outcomes presented in Table 11.

  • Quantitative shape-bias results for WTConv are reported in Table 11.

F Object detection - additional results

Additional results cover COCO object detection and segmentation, ImageNet-1K training schedules, and shape-decision behavior across networks.

  • COCO evaluation: Table 12 reports the detailed COCO object detection experiment results.
  • ImageNet-1K evaluation: Additional ImageNet-1K results use a 120-epoch training schedule.
  • Shape decisions: Table 11 reports the average fraction of shape decisions for every network.
  • COCO evaluation: COCO object detection and segmentation results use Cascade Mask-RCNN with different ImageNet-1K-pretrained backbones.The experiments use 120/300-epoch pretraining schedules and 1x/3x fine-tuning schedules.

G Robustness - additional results

The robustness evaluation reports detailed ImageNet-C results and qualitative object-detection examples across increasing corruption severities. These examples indicate that WTConvNeXt loses fewer details as corruption becomes more severe.

  • ImageNet-C: Tables 13 and 14 provide detailed ImageNet-C results.
  • Corruption examples: As corruption severity increases, WTConvNeXt loses fewer details in the qualitative examples.
  • Corruption examples: Qualitative detection examples cover motion blur, pixelation, Gaussian noise, and glass blur at increasing corruption severities.
  • Motion blur: Under motion blur, WTConvNeXt detects the traffic light even at the worst corruption level.
  • Pixelation: Under pixelation, WTConvNeXt detects more zebras than ConvNeXt and classifies them more accurately.
  • Glass blur: Under glass blur, ConvNeXt misses some persons on the boat while WTConvNeXt classifies them correctly.
Loading 2407.05848v2…