Source-linked AI summary
LDConv: Linear deformable convolution for improving convolutional neural networks
Xin Zhang, Yingze Song, Tingting Song, Degang Yang, Yichen Ye, Jie Zhou, Liming Zhang
TL;DR
Standard and deformable convolutions limit sampled-shape flexibility or incur quadratic parameter growth. LDConv introduces arbitrary-sized, deformable kernels with linearly growing parameters, and experiments on COCO2017, VOC 7+12, and VisDrone-DET2021 demonstrate its advantages.
Problem
Standard convolutions use fixed sampled shapes and square kernels, while Deformable Conv retains quadratic parameter growth and limited flexibility in kernel size and initial shape.
Method
LDConv generates initial coordinates for arbitrary-sized kernels and uses offsets to adjust irregular sampled positions for changing targets.
Results
LDConv provides arbitrary sampled shapes and sizes with linear parameter growth, and object-detection experiments across COCO2017, VOC 7+12, and VisDrone-DET2021 demonstrate its advantages.
Takeaways & Limitations
LDConv is presented as a plug-and-play convolutional operation that can replace standard convolutions and offer flexible network overhead-performance trade-offs.
Takeaways & Limitations
On VisDrone-DET2021, LDConv lacks the flexible attentional weights that RFAConv uses to distinguish complex background information in UAV images.
Abstract
from arXiv · showhide
Neural networks based on convolutional operations have achieved remarkable results in the field of deep learning, but there are two inherent flaws in standard convolutional operations. On the one hand, the convolution operation is confined to a local window, so it cannot capture information from other locations, and its sampled shapes is fixed. On the other hand, the size of the convolutional kernel are fixed to k $\times$ k, which is a fixed square shape, and the number of parameters tends to grow squarely with size. Although Deformable Convolution (Deformable Conv) address the problem of fixed sampling of standard convolutions, the number of parameters also tends to grow in a squared manner. In response to the above questions, the Linear Deformable Convolution (LDConv) is explored in this work, which gives the convolution kernel an arbitrary number of parameters and arbitrary sampled shapes to provide richer options for the trade-off between network overhead and performance. In LDConv, a novel coordinate generation algorithm is defined to generate different initial sampled positions for convolutional kernels of arbitrary size. To adapt to changing targets, offsets are introduced to adjust the shape of the samples at each position. LDConv corrects the growth trend of the number of parameters for standard convolution and Deformable Conv to a linear growth. Moreover, it completes the process of efficient feature extraction by irregular convolutional operations and brings more exploration options for convolutional sampled shapes. Object detection experiments on representative datasets COCO2017, VOC 7+12, and VisDrone-DET2021 fully demonstrate the advantages of LDConv. LDConv is a plug-and-play convolutional operation that can replace the convolutional operation to improve network performance. The code for the relevant tasks can be found at https://github.com/CV-ZhangXin/LDConv.
1. Introduction
Standard and deformable convolutions constrain sampled shapes or parameter counts, motivating LDConv as a flexible replacement with arbitrary kernel sizes and sampled shapes.
- Standard convolutions use fixed square kernels and sampled locations that cannot dynamically adapt to changing objects.
- Deformable Conv adjusts sampled shapes with offsets, but its parameter count still grows quadratically and omits arbitrary parameter counts and initial shapes.
- LDConv supports convolution kernels with arbitrary parameter counts and sampled shapes, enabling linear parameter growth and flexible hardware-performance trade-offs.
- LDConv can replace standard convolutions, while corresponding LDConv sizes are used in FasterBlock and GSBottleneck to improve both modules.
- A coordinate-generation algorithm creates initial sampled positions for convolutional kernels of arbitrary sizes.
- Offsets adjust irregular-kernel sampling positions for target variation, and three methods extract features from the resulting kernels.
2. Related works
Related work develops convolutional operators with broader receptive fields, variable kernels, or improved spatial modeling, but does not uniformly support arbitrary sampled shapes and parameter counts.
- Prior work proposed novel convolutional operations to improve convolutional performance from multiple perspectives.
- Involution addresses limited spatial modeling and inefficiency from sharing convolution kernels across locations and channels.
- CKConv treats kernels as continuous functions for arbitrarily large memory ranges, while FlexConv learns variable kernel sizes with fixed parameter counts.
- Increasing two-dimensional kernel size expands receptive fields but causes trainable parameters to scale quadratically, making large kernels costly and difficult to train.
- The authors position LDConv as combining arbitrary parameter counts and sampled shapes for efficient feature extraction, addressing limitations in prior flexible-convolution methods.
3. Methods
LDConv introduces arbitrary-size, irregular convolution kernels by generating initial sampled coordinates and adapting them with learned offsets. This design supports linear parameter growth while enabling irregular feature extraction and flexible sampled shapes.
- Initial sampled coordinates: LDConv generates initial sampled coordinates for convolution kernels with arbitrary numbers of parameters, including non-square sizes.The coordinate-generation algorithm constructs sampled grids for arbitrary kernel sizes, including a regular base grid and optional additional coordinates.
- Motivation: Standard convolution uses fixed regular grids, while Deformable Conv adjusts sampling with offsets but still restricts kernels to regular parameter counts.These constraints limit arbitrary kernel sizes and preserve squared parameter growth as kernel size increases.
- Offset-based resampling: At each feature-map position, learned offsets are added to the original coordinates to produce new sampling coordinates adapted to the target.For a kernel with N parameters, the offsets have dimensions (B, 2N, H, W).
- Flexibility and efficiency: LDConv changes Deformable Conv’s squared parameter-growth trend to linear growth, allowing finer control over computation and hardware overhead.It also supports diverse initial sampled shapes, including multiple designs for kernels with five parameters.
- Irregular feature extraction: LDConv extracts irregular-kernel features using alternative implementations based on Conv3d, 1 × 1 Conv2d, or column Conv2d.The alternatives reshape or stack resampled features differently before reducing them to the output feature representation.
- Scope of the operation: Unlike methods specialized for regular or task-specific shapes, LDConv supports arbitrary parameter counts and varied sampled shapes for adaptable convolution.Its sampled shapes can be designed initially and further modified through offsets.
4. Experiments
Experiments across COCO2017, VOC 7+12, and VisDrone-DET2021 evaluate LDConv with different kernel sizes, architectures, and competing convolutional operations. The results examine accuracy, overhead, generalization, and the effect of initial sampled shapes.
- Experimental setup: LDConv with different sizes was evaluated by replacing convolutional operations in YOLOv5, YOLOv7-tiny, and YOLOv8 detectors across three object-detection datasets.The experiments use COCO2017, VOC 7+12, and VisDrone-DET2021, with reported accuracy and efficiency measures.
- COCO2017: On COCO2017, increasing LDConv size generally improves detection accuracy while also increasing model parameters and computational overhead.For LDConv size 5, YOLOv5n reportedly improves AP50, AP75, and AP by three percentage points while reducing parameters and computational overhead versus standard convolution.
- VOC 7+12: On VOC 7+12, larger LDConv sizes likewise raise detection accuracy alongside incremental increases in parameter count and computational demand.The experiments use YOLOv7-tiny and report both mAP50 and mAP.
- VisDrone-DET2021: On VisDrone-DET2021, LDConv sizes provide a tunable efficiency–accuracy trade-off: size 3 reduces parameters and overhead but lowers detection performance relative to baseline.The study reports mAP50 and mAP and states that larger sizes can be adjusted to explore performance changes.
- Comparison experiments: At nine parameters, LDConv and Deformable Conv perform identically on COCO2017 because regular LDConv is equivalent to Deformable Conv at that size.LDConv additionally supports irregular sizes such as five or eleven parameters, which Deformable Conv does not implement.
- Comparison experiments: LDConv outperforms DSConv in the reported comparisons, while RFAConv performs better on VisDrone-DET2021 and CKConv requires substantial computation for stronger settings.The paper attributes DSConv’s weaker results to its tubular-target design and notes LDConv’s lack of flexible attention weights for complex UAV backgrounds.
- Initial sampled shapes: Different initial sampled shapes produce different detection accuracies on VisDrone-DET2021, making suitable initial-shape selection important for particular networks and datasets.On COCO2017, accuracy fluctuations across initial shapes are described as small, whereas VisDrone experiments with LDConv size 5 show clearer differences.
5. Analysis and discussion
LDConv performance varies with the initial sampled shape, indicating that offset adjustment is limited and that initialization matters. Experiments identify promising shapes and show that size 5 provides good results with modest training-speed impact.
- Different initial sampled shapes produce different network behavior, showing that LDConv initialization affects performance.The experiments evaluate LDConv with size 5 in YOLOv5n.
- Offset adjustment is limited, so initial sampled coordinates remain consequential after training.The paper measures sampled-point changes using Average Offset.
- Red and blue initial samples are more suitable for VisDrone-DET2021 than the other tested samples.These shapes corresponded to better detection accuracy in the reported experiment.
- LDConv with size 5 gave good network results across COCO2017, VOC 7+12, and VisDrone-DET2021.The authors report that training speed at size 5 was not much different from the original model, while larger sizes increased training time.
6. Conclusion
LDConv addresses limitations of fixed convolutional shapes and square parameter growth by supporting arbitrary sampled shapes and sizes. Its flexibility also enables exploration of task-specific kernels and different initial shapes.
- LDConv allows convolution kernels to use arbitrary sampled shapes and sizes, providing more diverse kernel choices than standard and deformable convolution.The conclusion frames this flexibility as a response to varied object shapes.
- Different initial sampled shapes can improve FasterBlock and GSBottleneck.The paper reports exploring multiple coordinate shapes for LDConv of size 5.
- Future work will explore suitable LDConv sizes and sampled shapes for specific computer-vision tasks.The proposed flexibility supports this planned task-specific exploration.