Source-linked AI summary
DO-Conv: Depthwise Over-parameterized Convolutional Layer
Jinming Cao, Yangyan Li, Mingchao Sun, Ying Chen, Dani Lischinski, Daniel Cohen-Or, Baoquan Chen, Changhe Tu
TL;DR
CNNs rely on convolutional layers, but over-parameterization of these core building blocks remains underexplored as a way to improve existing architectures. The paper introduces DO-Conv by adding a depthwise convolution during training, then folding it into a conventional convolution for inference. Extensive experiments report improved CNN performance across classification, detection, and segmentation with inference computation exactly equivalent to conventional convolution.
Problem
Over-parameterization has potential for improving existing CNN architectures, but its use in convolutional layers remains underexplored.
Method
DO-Conv augments a conventional convolution with a depthwise convolution and folds the composed operation into a single convolution after training.
Results
DO-Conv boosts CNN performance on image classification, detection, and segmentation while preserving inference computation exactly equivalent to a conventional convolution.
Takeaways & Limitations
DO-Conv is presented as a simple, generic alternative to conventional convolutional layers that improves existing CNNs without extra inference computation.
Takeaways & Limitations
The paper identifies the need for theoretical understanding and broader application studies to explain and extend DO-Conv’s performance improvements.
Abstract
from arXiv · showhide
Convolutional layers are the core building blocks of Convolutional Neural Networks (CNNs). In this paper, we propose to augment a convolutional layer with an additional depthwise convolution, where each input channel is convolved with a different 2D kernel. The composition of the two convolutions constitutes an over-parameterization, since it adds learnable parameters, while the resulting linear operation can be expressed by a single convolution layer. We refer to this depthwise over-parameterized convolutional layer as DO-Conv. We show with extensive experiments that the mere replacement of conventional convolutional layers with DO-Conv layers boosts the performance of CNNs on many classical vision tasks, such as image classification, detection, and segmentation. Moreover, in the inference phase, the depthwise convolution is folded into the conventional convolution, reducing the computation to be exactly equivalent to that of a convolutional layer without over-parameterization. As DO-Conv introduces performance gains without incurring any computational complexity increase for inference, we advocate it as an alternative to the conventional convolutional layer. We open-source a reference implementation of DO-Conv in Tensorflow, PyTorch and GluonCV at https://github.com/yangyanli/DO-Conv.
1 Introduction
The paper identifies over-parameterization as an underexplored way to improve existing CNN architectures and proposes DO-Conv, which adds a depthwise convolution to conventional convolutional layers. Experiments show performance gains across vision tasks without increasing inference computation.
- Motivation: Over-parameterization can accelerate training without increasing network expressiveness, suggesting untapped potential for improving existing architectures.The paper distinguishes this use of over-parameterization from simply expanding a network’s expressive capacity.
- DO-Conv: DO-Conv augments a conventional convolution with a depthwise convolution that separately convolves each input channel.This extra operation forms the paper’s depthwise over-parameterized convolutional layer.
- DO-Conv: DO-Conv consistently boosts converged CNN performance while also accelerating training across various models.The proposed layer is intended as a drop-in modification to existing convolutional layers.
- Inference: The composed linear operations are folded into one compact layer after training, making inference computation exactly equivalent to a conventional convolutional layer.Thus, the added depthwise convolution does not remain as an extra inference computation.
- Empirical findings: DO-Conv improves CNN performance on image classification, detection, and segmentation by replacing conventional convolutions.The reported gains come from the over-parameterized training formulation rather than a different inference operation.
2 Related Work
Related work places DO-Conv within research on improving CNN building blocks and over-parameterized layers. The paper extends these ideas to convolutional kernels, whose spatial and channel dimensions permit versatile over-parameterization.
- CNN architectures: DO-Conv is orthogonal to designing novel CNN architectures and can be combined with established architectures to improve their performance.The paper positions its contribution as a modification to existing models rather than a replacement architecture.
- Convolutional layers: Alternative convolutional layers have been proposed to improve feature learning capability and/or efficiency, placing DO-Conv within this broader line of work.The paper treats convolutional layers as core CNN building blocks whose improvement can affect overall performance.
- Over-parameterization: Prior work showed that over-parameterization can accelerate training in deep linear networks and empirically accelerate training in deep non-linear networks.The paper applies this optimization-related idea to convolutional layers.
- Over-parameterization: Convolutional kernels expose both channel and spatial axes, making their over-parameterization more versatile than over-parameterizing fully connected layers.This structural distinction motivates convolution-specific over-parameterization.
- Related mechanisms: Over-parameterized layers add linear transformations without increasing expressiveness and can fold those transformations into inference-time layers.Normalization layers are described as similar in this respect, although their effectiveness remains under study.
3 Method
DO-Conv augments conventional convolution with a depthwise convolution, increasing trainable parameterization while preserving the convolutional layer’s interface and receptive field. Its feature-composition and kernel-composition forms are mathematically equivalent, with kernel composition typically preferred for training efficiency and folded weights used for inference.
- Conventional convolution: A conventional convolution applies Cout kernels to an input patch P, producing a Cout-dimensional output.The kernels are represented as W ∈ R^Cout×(M×N)×Cin.
- Depthwise convolution: Depthwise convolution independently processes each of the Cin input channels, producing Dmul features per channel.Its kernel has shape W ∈ R^(M×N)×Dmul×Cin, with Dmul as the depth multiplier.
- DO-Conv: DO-Conv composes a depthwise convolution using D with a conventional convolution using W, with Dmul ≥ M × N, while producing the same Cout-dimensional output type.The receptive field remains M × N.
- DO-Conv: Feature composition first transforms P into P′ = D ◦ P and then applies W, whereas kernel composition forms W′ = D^T ◦ W before applying it directly to P.These two realizations are mathematically equivalent.
- DO-Conv: DO-Conv over-parameterizes convolution because two trainable kernel sets represent an equivalent linear transformation, with Dmul ≥ M × N.The conventional layer uses Cout × (M × N) × Cin weights, while DO-Conv uses D and W.
- Training and inference: After training, D and W are folded into W′, so inference uses a single convolutional layer with computation exactly equivalent to a conventional layer.Kernel composition typically also uses fewer MACC operations and less memory than feature composition during training.
4 Experiments
Experiments evaluate DO-Conv across classification, segmentation, detection, training dynamics, visualizations, and ablations under controlled replacement protocols. DO-Conv generally improves performance and convergence, while effects depend on task stage, network stage, and initialization.
- Comparison protocol: DO-Conv experiments replace non-pointwise baseline convolutions without changing other settings or tuning hyperparameters, isolating the replacement’s effect across multiple vision tasks.The protocol covers image classification, semantic segmentation, and object detection on benchmark datasets.
- Image classification: DO-Conv improves classification on CIFAR, particularly for relatively shallower networks, and consistently boosts various ImageNet baselines.CIFAR results average the last five epochs over five runs; ImageNet experiments use GluonCV settings and compare independently trained baselines.
- Semantic segmentation: DO-Conv is evaluated for semantic segmentation on PASCAL VOC and Cityscapes, where it can be applied during backbone pretraining, segmentation fine-tuning, or both stages.The segmentation pipeline consists of ImageNet backbone pretraining followed by task-specific fine-tuning.
- Object detection: For COCO detection, using DO-Conv only during detection does not improve overall performance, whereas using it in both backbone and detection stages yields an obvious improvement.The reported deltas are relative to the first table row, and baseline hyperparameters are retained.
- Training dynamics: DO-Conv converges faster and to lower errors than the baseline in ResNet-v1b ImageNet training, extending prior reports of faster convergence with a lower-error outcome.The comparison uses train and validation curves over 120 epochs across different ResNet-v1b depths.
- Ablation studies: Ablations show that DO-Conv’s effectiveness varies by ResNet stage and initialization: CIFAR-100 accuracy improves with more modified stages, while ImageNet effects can be mixed and identity initialization outperforms random initialization.For Dmul > M × N, all tested choices improve over baseline, but Dmul = M × N performs best and is used by default.
5 Conclusions and Future work
DO-Conv is presented as a simple, generic way to improve existing CNNs without adding inference computation. The authors identify theoretical understanding and broader application scope as future directions.
- DO-Conv is described as a novel, simple, and generic approach for boosting CNN performance.
- Its practical benefits include improved training and final accuracy for existing CNNs without extra inference computation.
- The authors call for theoretical understanding of DO-Conv’s performance improvements and investigation across a broader range of applications.
Broader Impact
The authors frame DO-Conv’s broader impact primarily as improving existing CNN models across computer-vision tasks and potentially supporting new CNN-based applications. They report no additional ethical or social implications beyond these stated impacts.
- The work primarily aims to improve existing CNN models on a variety of computer-vision tasks.
- The authors suggest DO-Conv could assist in developing CNN-based solutions for tasks not yet tackled in this manner.
- The paper reports no ethical or social implications beyond the impacts it states.