Source-linked AI summary
Active Convolution: Learning the Shape of Convolution for Image Classification
Yunho Jeon, Junmo Kim
TL;DR
Research has focused more on CNN architectures than on the convolution unit itself. This paper introduces the learnable, deformable active convolution unit and reports improved image-classification performance after replacing conventional convolution units.
Problem
Research on CNNs has mostly focused on composing layers rather than addressing the convolution unit itself.
Method
The active convolution unit uses learnable position parameters so its receptive-field shape can vary and be learned through backpropagation.
Results
Replacing conventional convolution layers with ACUs improved equivalent networks, including a CIFAR-10 error rate of 7.33%, down 0.68% from the baseline.
Takeaways & Limitations
ACUs provide CNNs with greater convolutional flexibility and improved benchmark performance without manually tuning the convolution shape.
Takeaways & Limitations
The study changed only 3×3 convolutions in AlexNet because the effects of applying ACUs to larger convolutions had not been fully analyzed.
Abstract
from arXiv · showhide
In recent years, deep learning has achieved great success in many computer vision applications. Convolutional neural networks (CNNs) have lately emerged as a major approach to image classification. Most research on CNNs thus far has focused on developing architectures such as the Inception and residual networks. The convolution layer is the core of the CNN, but few studies have addressed the convolution unit itself. In this paper, we introduce a convolution unit called the active convolution unit (ACU). A new convolution has no fixed shape, because of which we can define any form of convolution. Its shape can be learned through backpropagation during training. Our proposed unit has a few advantages. First, the ACU is a generalization of convolution; it can define not only all conventional convolutions, but also convolutions with fractional pixel coordinates. We can freely change the shape of the convolution, which provides greater freedom to form CNN structures. Second, the shape of the convolution is learned while training and there is no need to tune it by hand. Third, the ACU can learn better than a conventional unit, where we obtained the improvement simply by changing the conventional convolution to an ACU. We tested our proposed method on plain and residual networks, and the results showed significant improvement using our method on various datasets and architectures in comparison with the baseline.
1. Introduction
CNN research has largely emphasized composing architectures and layers, while the convolution unit itself has received comparatively little attention. The paper introduces the active convolution unit (ACU), whose deformable shape can be learned during training to provide greater flexibility and representation power.
- Recent CNN research has mostly focused on composing layers rather than modifying the convolution unit itself.
- Prior work explored variations in activation and pooling units, including ReLU variants, spatial pyramid pooling, ROI pooling, and fractional pooling.
- Only a few studies considered convolution units themselves, including dilated convolution and permutohedral lattice convolution.
- The proposed ACU has no fixed receptive-field shape and can take more diverse forms during convolution.
- The ACU is deformable and learnable, with its shape learned during training.
- The paper presents the ACU as a convolution unit intended to provide greater flexibility and representation power for meaningful image-classification improvement.
2. Active Convolution Unit
The active convolution unit (ACU) replaces fixed convolution shapes with learnable synapse positions, allowing conventional, dilated, fractional, and freely structured receptive fields. Its positions use interpolation and backpropagation, with normalized gradients and training procedures used to stabilize movement.
- ACU concept: The ACU uses learnable position parameters, so its receptive-field shape can change during training through backpropagation.Synapse positions are differentiable and jointly learned with weights and bias.
- ACU capacity: The ACU generalizes conventional convolution, including fixed conventional and dilated forms, while also supporting fractional positions and unrestricted synapse layouts.A conventional 3 × 3 convolution is recovered by assigning the corresponding integer synapse positions.
- Interpolation: Bilinear interpolation connects synapses to inter-neuron positions, converting the discrete input space into a continuous one for convolution.Real-valued position parameters address nonlattice points using nearby integer samples.
- Parameterization: The ACU adds 2×K position parameters to D×C×K convolution weights, and shares the position set across output units.Here K, C, and D denote synapses, input channels, and output channels, respectively.
- Optimization: Position derivatives are valid between lattice points but may be nondifferentiable at lattice points under four-neighbor interpolation.The paper reports that an appropriate position learning rate avoids practical problems from this issue.
- Optimization: Normalized position gradients help control synapse movement and make the ACU easier to train despite layer-dependent gradient fluctuations.The paper notes that small movement can make the ACU ineffective, whereas large movement can diversify synapses excessively.
3. ACU with a Plain Network
On a plain network, replacing conventional 3 × 3 convolutions with ACUs improved CIFAR performance while adding few parameters. Learned positions largely preserved conventional receptive fields in lower layers but expanded in higher layers, and synapse movement continuously deformed outputs during training.
- Experimental setup: Only 96 parameters were added when six conventional convolution layers were replaced with ACUs, with all other parameters held constant.Each ACU used nine synapses, and position parameters were shared across kernels in the same layer.
- Experimental results: 0.68% lower CIFAR-10 error and 0.74% lower CIFAR-100 error were obtained with ACUs than with the plain-network baseline.CIFAR-10 error decreased from 8.01% to 7.33%.
- Experimental results: Both training loss and test error were lower for the ACU than for the baseline on CIFAR-10.The training curve compares solid training-loss lines and dotted test-error lines.
- Learned Position: Lower-layer ACUs settled near conventional 3 × 3 convolutions, whereas higher-layer ACUs tended to expand their receptive fields.The last ACU resembled two dilated convolutions, while the first layer was almost identical to conventional convolution.
- Learned Position: Synapse positions in the first layer tended to remain near their initial shape, while positions in the last layer grew monotonically during training.The position trajectories track horizontal position αk for nine synapses across iterations.
- Learned Position: Continuous synapse movement spatially deformed ACU outputs at every iteration, providing an augmentation effect without explicit data augmentation.The same inputs and weights therefore did not always produce the same outputs.
4. ACU with the Residual Network
ACUs improved both basic and bottleneck residual networks on CIFAR while learning increasingly wider receptive-field shapes in higher layers. Different residual blocks with the same dimensionality also learned different ACU shapes.
- 4.1. Basic Residual Network: The basic residual network used five residual blocks in a 32-layer architecture with projection shortcuts for dimension increases and identity shortcuts otherwise.The residual structure comparison distinguishes basic residual and bottleneck blocks.
- 4.1. Basic Residual Network: 0.47% lower CIFAR-10 error was achieved by replacing residual-block convolutions with ACUs in the basic residual network.The ACU residual network reached 7.54% error versus 8.01% for the baseline, and CIFAR-100 improved by 0.68%.
- 4.2. Bottleneck Residual Network: 0.52% lower CIFAR-10 error was achieved when one 3 × 3 convolution in the bottleneck residual network was replaced with an ACU.The 47-layer bottleneck baseline reached 7.64% error, while the ACU version reached 7.12%; CIFAR-100 gained 0.46%.
- 4.2. Bottleneck Residual Network: In the bottleneck network, higher-layer ACU positions became wider, while ACUs in different residual blocks with the same dimensions learned different receptive-field shapes.Fifteen convolutions were changed to ACUs in this analysis.
5. Experiment on Place365
On Place365, the ACU was evaluated in AlexNet and a residual network by replacing selected 3 × 3 convolutions. It improved classification performance and learned receptive-field shapes that varied across network depth.
- The ACU was tested on Place365-Standard, a benchmark with more than 1.8 million training images across 365 scene categories.
- AlexNet: AlexNet was trained with its 3 × 3 convolutions replaced by ACUs, while its 11 × 11 and 5 × 5 convolutions were retained.
- AlexNet: 0.79% improvement was obtained for AlexNet by changing the 3 × 3 convolution layers to ACUs.The ACU and baseline had nearly identical accuracies during the 50k-iteration warm-up, after which ACU test error began to decrease.
- Residual network: The residual network gained 0.49% in top-5 accuracy after eight convolutions were replaced with ACUs, adding 128 parameters.
- Learned shapes: Learned ACU shapes expanded receptive-field coverage in higher layers, with greater coverage on Place365 than in the CIFAR-10/100 experiment.In AlexNet, the first ACU layer learned a shape resembling combinations of two receptive fields, while the final shape enlarged its receptive field.
6. Conclusion
The paper concludes that ACUs provide learnable position parameters and greater freedom than conventional convolution units. Experiments showed improved performance, while the authors identify multiple position sets per layer as a direction for expanding representational power.
- ACUs provide more freedom to conventional convolution by learning position parameters through backpropagation.
- Simply replacing convolution layers with ACUs boosted the performance of equivalent networks.
- The paper shared one set of position parameters, while multiple sets per layer could expand representational power.
- The conclusion is supported by reported classification accuracy evaluated using the average score over standard 10 crops.