ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

Xiaohan Ding; Yuchen Guo; Guiguang Ding; Jungong Han

Source-linked AI summary

ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

Xiaohan Ding, Yuchen Guo, Guiguang Ding, Jungong Han

arXiv:1908.03930v3Aug 11, 2019cs.CVcs.LGcs.NE

TL;DR

CNN applications often need higher accuracy under computational constraints, while adapting architectures can require substantial human effort or GPU time. The paper introduces ACBs that add asymmetric branches during training, converts them back into the original architecture, and reports clear improvements across CIFAR and ImageNet, with associated gains in rotational robustness and kernel-skeleton strength.

Problem
CNN enhancement should improve accuracy under deployment constraints without relying on costly architecture redesign or extra inference-time computation.
Method
ACNet replaces square-kernel layers with ACBs containing square, horizontal, and vertical branches, then fuses the trained asymmetric kernels back into the original architecture.
Results
ACNet improves various plain-style, residual, and densely connected models on CIFAR and ImageNet by a clear margin.
Takeaways & Limitations
The method provides an architecture-neutral way to strengthen CNNs while preserving the original inference structure and improving robustness to rotational distortions.

Abstract

from arXiv · show

As designing appropriate Convolutional Neural Network (CNN) architecture in the context of a given application usually involves heavy human works or numerous GPU hours, the research community is soliciting the architecture-neutral CNN structures, which can be easily plugged into multiple mature architectures to improve the performance on our real-world applications. We propose Asymmetric Convolution Block (ACB), an architecture-neutral structure as a CNN building block, which uses 1D asymmetric convolutions to strengthen the square convolution kernels. For an off-the-shelf architecture, we replace the standard square-kernel convolutional layers with ACBs to construct an Asymmetric Convolutional Network (ACNet), which can be trained to reach a higher level of accuracy. After training, we equivalently convert the ACNet into the same original architecture, thus requiring no extra computations anymore. We have observed that ACNet can improve the performance of various models on CIFAR and ImageNet by a clear margin. Through further experiments, we attribute the effectiveness of ACB to its capability of enhancing the model's robustness to rotational distortions and strengthening the central skeleton parts of square convolution kernels.

1. Introduction

The paper proposes ACNet, an architecture-neutral CNN enhancement that improves square-kernel layers during training and can be converted back without extra inference cost. Experiments and analysis link its benefits to rotational robustness and strengthened kernel skeletons.

1. Introduction: ACB replaces each square-kernel layer with parallel square, horizontal, and vertical convolutions whose outputs are summed.For a d × d layer, the asymmetric branches use 1×d and d×1 kernels.
1. Introduction: ACNet converts the trained asymmetric blocks back into the original architecture by adding asymmetric kernels onto the square-kernel skeleton.The converted model produces the same outputs as the training-time ACNet through convolution additivity.
1. Introduction: Further experiments associate ACNet’s effectiveness with stronger central crisscross skeletons in square kernels and improved robustness to rotational distortions.The paper reports that models consistently learn to enhance skeletons at every layer.
1. Introduction: ACNet improves several plain-style, residual, and densely connected models on CIFAR and ImageNet by a clear margin.The method is designed as a plug-in structure for mature architectures rather than a replacement architecture.
1. Introduction: ACNet introduces no hyper-parameters and requires no extra inference-time computations, parameters, memory footprint, or energy consumption.This addresses deployment settings where computational resources and real-time inference constrain CNN enhancement.

2. Related work

Prior asymmetric-convolution methods primarily factorize square kernels for compression, acceleration, or architectural efficiency. ACNet instead uses asymmetric branches to enrich training representations and then fuses their learned knowledge into square kernels while remaining architecture-neutral.

2. Related work: Earlier methods factorize d × d convolutions into d × 1 and 1 × d layers to reduce parameters and computations.The equivalence holds directly only for rank-one kernels, whereas practical learned kernels can have higher intrinsic rank.
2. Related work: Architectures such as Inception-v3 and ENet use asymmetric convolutions as design elements for computational efficiency.The cited discussion notes that replacing convolutions this way is not always equivalent, particularly in low-level layers.
2. Related work: ACNet uses 1D asymmetric convolutions to enrich the feature space during training rather than factorizing or approximating square-kernel layers.The learned asymmetric kernels are subsequently fused into the square-kernel layers.
2. Related work: ACNet is architecture-neutral because it makes no assumptions about a specific architecture and is intended to provide universal benefits across models.It replaces square-kernel layers in off-the-shelf architectures without introducing hyper-parameters.
2. Related work: The fused ACNet model requires no extra parameters or computations during inference and can remain compatible with later compression techniques.The paper lists pruning, quantization, and feature-map compacting as examples.

3. Asymmetric Convolutional Network

The section formalizes convolution additivity for compatible kernels and uses it to train ACB branches before fusing them into an equivalent standard convolution at inference.

3.1. Formulation: A convolutional layer maps a C-channel feature map through D filters, with each output channel formed by summing convolutions over input channels.The notation defines F as a 3D kernel, M as the input feature map, and O as the output.
3.2. Exploiting the additivity of convolution: Convolution outputs can be added by element-wise adding compatible kernels on corresponding positions, provided the inputs and output resolutions align.The smaller kernel must be patchable onto the larger one; 3 × 1 and 1 × 3 kernels are compatible with 3 × 3.
3.2. Exploiting the additivity of convolution: The additivity condition is shared sliding windows: summing branch outputs equals applying the corresponding summed kernels to the same input window.Figure 2 illustrates this with 3 × 3, 1 × 3, and 3 × 1 layers.
3. Asymmetric Convolutional Network: An ACB replaces each 3 × 3 convolution, together with its following batch normalization when present, with parallel 3 × 3, 1 × 3, and 3 × 1 branches.Each branch is followed by batch normalization, and the branch outputs are combined.
3.3. ACB for free inference-time improvements: After training, ACNet converts every ACB into a standard convolution through BN fusion and branch fusion, preserving outputs without extra inference computation.BN parameters are folded into kernels and biases, after which asymmetric kernels are added to corresponding positions of the square kernel.
3.3. ACB for free inference-time improvements: The converted layer is equivalent only at inference time because random initialization and different gradient flows make the training dynamics nonequivalent.Thus, the fused network retains the original architecture and computational budget while reflecting ACB training.

4. Experiments

Experiments across CIFAR and ImageNet show that ACNet consistently improves diverse CNN architectures, while rotational-distortion and pruning studies examine robustness and the role of kernel skeletons.

Experimental setup: ACNet experiments compare off-the-shelf baselines with counterparts trained from scratch under identical configurations and complete convergence.The counterparts are converted back to the baseline structure before accuracy testing.
Performance improvements on CIFAR: CIFAR experiments cover Cifar-quick, VGG-16, ResNet-56, WRN-16-8, and DenseNet-40 on CIFAR-10 and CIFAR-100.Training uses common staircase learning-rate schedules or the original WRN-16-8 configuration, with padding, random cropping, and left-right flipping.
Performance improvements on CIFAR: All evaluated CIFAR models gain accuracy by a clear margin, indicating that ACB benefits combine with multiple architectures.The reported comparisons are summarized in Tables 1 and 2.
Performance improvements on ImageNet: 1.52%, 0.78% and 1.18% are the single-crop Top-1 accuracy increases for AlexNet, ResNet-18 and DenseNet-121 on ImageNet, respectively.The ImageNet study uses representative plain-style, residual, and densely connected architectures.
Ablation studies: Removing horizontal kernels, vertical kernels, or branch batch normalization degrades performance, while the authors characterize rotational robustness as a possible contributor rather than the primary explanation.The experiments evaluate these design choices using AlexNet and ResNet-18 on ImageNet.
Ablation studies: Horizontal kernels improve robustness on 180°-rotated and up-down-flipped images, with ResNet-18 reaching 1.27% higher accuracy than the base model on 180°-flipped inputs.The horizontal-kernel model is 0.34% higher on original inputs and 0.75% higher than the vertical-kernel counterpart on 180°-rotated inputs.
ACB enhances the skeletons of square kernels: Pruning experiments show that removing corner weights harms accuracy less than pruning skeleton weights, supporting greater importance for the central crisscross positions.In fused ACNet, pruning skeletons destroys the model at 13% global sparsity, equivalent to removing 23.4% of skeleton weights.
ACB enhances the skeletons of square kernels: ACNet further strengthens kernel skeletons: skeleton positions exceed A values of 0.666, while the central point reaches 1.000 and corners fall below 0.400 in fused ResNet-56.Adding asymmetric kernels to borders instead produces lower accuracy than regular ACNet, and the authors attribute effectiveness partly to skeleton enhancement.

5. Conclusion

ACNet uses ACBs to improve diverse CNN architectures while preserving the original architecture and inference cost. Its effectiveness is associated with robustness to rotational distortions and stronger square-kernel skeletons.

ACB sums outputs from square, horizontal, and vertical convolutional branches.
ACNet replaces square-kernel layers during training and converts back to the original architecture afterward.
ACNet improves various plain-style, residual, and densely connected models on CIFAR and ImageNet.
ACNet strengthens robustness to rotational distortions and the skeletons of square kernels without extra inference-time computations or tunable hyper-parameters.