Source-linked AI summary
Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets
Daniel Haase, Manuel Amthor
TL;DR
Depthwise separable convolutions target efficient CNNs, but their operation order implicitly exploits cross-kernel rather than dominant intra-kernel correlations. The paper introduces BSConv to exploit depth-axis correlations and reports consistent improvements across architectures and datasets without added complexity.
Problem
Efficient CNN design is needed for computationally limited mobile and automotive applications, motivating methods that exploit filter-weight redundancies.
Method
BSConv represents each filter kernel with a two-dimensional blueprint distributed along the depth axis using a weight vector, with implementations derived from CNN-kernel analyses and standard layers.
Results
BSConv consistently improves DSC-based and standard architectures at the same parameter and time complexity, including MobileNets and ResNets.
Takeaways & Limitations
BSConv provides an intra-kernel-correlation formulation that interprets and justifies DSCs while enabling efficient separation of regular convolutions.
Takeaways & Limitations
Pointwise convolutions omit activation and normalization because the method requires their weights to be able to be negative.
Abstract
from arXiv · showhide
We introduce blueprint separable convolutions (BSConv) as highly efficient building blocks for CNNs. They are motivated by quantitative analyses of kernel properties from trained models, which show the dominance of correlations along the depth axis. Based on our findings, we formulate a theoretical foundation from which we derive efficient implementations using only standard layers. Moreover, our approach provides a thorough theoretical derivation, interpretation, and justification for the application of depthwise separable convolutions (DSCs) in general, which have become the basis of many modern network architectures. Ultimately, we reveal that DSC-based architectures such as MobileNets implicitly rely on cross-kernel correlations, while our BSConv formulation is based on intra-kernel correlations and thus allows for a more efficient separation of regular convolutions. Extensive experiments on large-scale and fine-grained classification datasets show that BSConvs clearly and consistently improve MobileNets and other DSC-based architectures without introducing any further complexity. For fine-grained datasets, we achieve an improvement of up to 13.7 percentage points. In addition, if used as drop-in replacement for standard architectures such as ResNets, BSConv variants also outperform their vanilla counterparts by up to 9.5 percentage points on ImageNet. Code and models are available under https://github.com/zeiss-microscopy/BSConv.
1. Introduction
CNN efficiency research targets computational limits in mobile and automotive applications by exploiting redundancies in convolutional filter weights. The paper introduces BSConv to model intra-kernel depth-axis correlations, contrasts them with DSCs’ cross-kernel correlations, and reports consistent efficiency and accuracy gains.
- Mobile and automotive applications motivate efficient CNNs because their computational capacity is often limited.
- BSConv represents each filter kernel with a 2D blueprint distributed along the depth axis by a weight vector, exploiting intra-kernel correlations.
- DSCs implicitly rely on cross-kernel correlations because their operation order reverses the intra-kernel assumption motivating BSConv.
- Trained CNN filters exhibit strong depth-axis correlations, with many slices sharing a filter-specific 3 × 3 blueprint scaled by different factors.
- BSConv consistently outperforms DSC-based architectures at the same parameter and time complexity and can replace standard convolution layers in other architectures.
2. Related Work
Related work improves CNN efficiency through weight reduction, compressed computation, and architecture design. Depthwise separable convolutions became foundational to efficient architectures, while related approaches include pruning, quantization, low-rank approximations, grouped convolutions, and neural architecture search.
- Model pruning removes filters or connections, while quantization and compression are often combined with pruning to accelerate inference.
- Efficiency-driven CNN architecture search may be manual or automated through genetic algorithms and reinforcement-learning-based neural architecture search.
- Convolutional weight redundancies are addressed with low-rank filter approximations and grouped convolutions.
- Depthwise separable convolutions underpin recent efficient architectures, including MobileNetV1, factorized CNNs, Xception, inverted residual bottlenecks, MnasNet, MobileNetV3, and EfficientNet.
3. Blueprint Separable Convolutions (BSConv)
BSConv is derived from observed intra-kernel correlations: filter slices often share a depth-wise blueprint scaled by channel-specific weights. The formulation converts this structure into efficient standard-layer implementations, including unconstrained and low-rank subspace variants.
- 3.1. Intra-Kernel Correlations in Standard CNNs: Trained CNN filters often exhibit intra-kernel correlations, with depth slices sharing a filter-specific 2D blueprint scaled by different factors.This pattern is observed across architectures, training settings, and datasets, and is quantified using PCA on the 2D slices of each filter.
- 3.2. From Correlations to BSConv: BSConv represents each M × K × K filter using one K × K blueprint and M scalar weights distributed along the depth axis.This makes the observed approximation an integral property of the learned filters rather than merely an empirical observation.
- 3.3. Blueprint Separable Convolutions: A BSConv module has N blueprints and an M × N weight matrix, reducing the unconstrained representation from M · N · K2 kernel parameters to N · K2 blueprint parameters plus M · N weights.The weight matrix can be learned without constraints in BSConv-U and reduced further through factorization in BSConv-S.
- 3.3.1. Unconstrained BSConv (BSConv-U): The efficient BSConv-U implementation applies a 1 × 1 pointwise convolution followed by a K × K depthwise convolution.The ordering follows from splitting the regular convolution into depth-wise 2D convolutions and rearranging the blueprint-weight representation.
- 3.3.2. Subspace BSConv (BSConv-S): BSConv-S factorizes the weight matrix into W A and W B, reducing parameters to N · M′ + M′ · M while using M′ = ⌈p · M⌉.An orthonormal regularization loss encourages nonredundant basis vectors, and the implementation uses two pointwise convolutions followed by a depthwise convolution.
4. Rethinking Depthwise Separable Convolutions
This section relates BSConv to depthwise separable convolutions and linear bottlenecks, arguing that BSConv uses individual 2d blueprints while DSCs implicitly use a shared 3d blueprint. It also describes architectural substitutions and the conditions needed for BSConv-U.
- BSConv is related to depthwise separable convolutions and linear inverted residual bottlenecks as efficient mobile-model building blocks.
- DSCs enforce cross-kernel correlations through a reversed depthwise–pointwise order, whereas BSConv targets intra-kernel correlations.
- MobileNetV1 can be converted to BSConv-U by replacing DSCs, effectively switching the order of pointwise and depthwise convolutions.
- DSCs implicitly assume one 3d blueprint shared across kernels, whereas BSConv uses an individual 2d blueprint for each kernel.
- BSConv-U omits activation and normalization after pointwise convolutions so the distribution weights can be negative.
5. Experiments
The experiments evaluate BSConv variants against matched CNN baselines across CIFAR, ImageNet, and fine-grained settings. BSConv consistently improves accuracy, including substantial gains for standard architectures and models using subspace regularization.
- MobileNets: All BSConv MobileNet variants outperform their corresponding CIFAR baselines under matched parameter settings.BSConv-U is used for MobileNetV1, while BSConv-S is used for MobileNetV2/V3.
- ResNets and WideResNets: ResNet accuracy improves by up to 3.1 percentage points on CIFAR100 with slightly fewer parameters and computational costs.
- ResNets and WideResNets: WideResNet accuracy improves by up to 2.1 percentage points on CIFAR100 while using fewer parameters and computational costs.
- ImageNet MobileNets: ImageNet BSConv MobileNet variants outperform their corresponding baselines, although their relative improvements are smaller than in the CIFAR experiments.
- ImageNet ResNets: ResNet-10 versus ResNet-68+BSConv-U shows a 9.5 percentage-point accuracy gain at similar parameter counts.
- Fine-grained Recognition: BSConv-U improves fine-grained recognition accuracy by at least 2 percentage points while using up to 8.5× fewer parameters.
- Fine-grained Recognition: Orthonormal subspace regularization improves model performance by over 5 percentage points, with an optimum at α = 0.1.
6. Conclusions
The paper introduces BSConv as an efficient CNN building block and reports consistent improvements across established architectures including MobileNets, MnasNets, EfficientNets, and ResNets.
- BSConv improves established MobileNet, MnasNet, EfficientNet, and ResNet models while serving as a highly efficient CNN building block.