Source-linked AI summary
Adaptive Sparse Convolutional Networks with Global Context Enhancement for Faster Object Detection on Drone Images
Bowei Du, Yecheng Huang, Jiaxin Chen, Di Huang
TL;DR
Efficient drone-image detection must reduce computation and latency on resource-constrained UAV hardware while retaining accuracy. CEASC optimizes detection heads with global context enhancement and adaptive masking; on VisDrone and UAVDT, it accelerates multiple detectors with competitive accuracies.
Problem
Drone-image detection requires low latency, but high-resolution imagery is computationally demanding and UAV hardware is resource-constrained.
Method
CEASC combines context-enhanced sparse convolution with context-enhanced group normalization and adaptive multi-layer masking to optimize detection-head efficiency across foreground scales.
Results
On VisDrone and UAVDT, CEASC accelerates various base detectors with competitive accuracies; with GFL V1, it reduces GFLOPs by 71.4%, increases FPS by 60%, and improves mAP by 0.3%.
Takeaways & Limitations
CEASC is a plug-and-play detection-head optimization approach that can be extended to existing state-of-the-art detectors for drone-imagery acceleration.
Takeaways & Limitations
Existing high-accuracy aerial-image methods may require multiple inferences per image, limiting their application on resource-constrained UAV platforms.
Abstract
from arXiv · showhide
Object detection on drone images with low-latency is an important but challenging task on the resource-constrained unmanned aerial vehicle (UAV) platform. This paper investigates optimizing the detection head based on the sparse convolution, which proves effective in balancing the accuracy and efficiency. Nevertheless, it suffers from inadequate integration of contextual information of tiny objects as well as clumsy control of the mask ratio in the presence of foreground with varying scales. To address the issues above, we propose a novel global context-enhanced adaptive sparse convolutional network (CEASC). It first develops a context-enhanced group normalization (CE-GN) layer, by replacing the statistics based on sparsely sampled features with the global contextual ones, and then designs an adaptive multi-layer masking strategy to generate optimal mask ratios at distinct scales for compact foreground coverage, promoting both the accuracy and efficiency. Extensive experimental results on two major benchmarks, i.e. VisDrone and UAVDT, demonstrate that CEASC remarkably reduces the GFLOPs and accelerates the inference procedure when plugging into the typical state-of-the-art detection frameworks (e.g. RetinaNet and GFL V1) with competitive performance. Code is available at https://github.com/Cuogeihong/CEASC.
1. Introduction
Drone-image detection requires low-latency models because UAV hardware is resource-constrained, while tiny objects, heavy detection heads, and varying foreground scales complicate efficient inference. CEASC addresses this trade-off with context-enhanced sparse convolution and adaptive masking, reducing computation while maintaining competitive accuracy.
- UAV hardware is resource-constrained, creating demand for lightweight detectors with fast inference and low latency.
- RetinaNet’s detection head accounts for 82.3% of overall GFLOPs when using a ResNet18 backbone with 512 input channels.
- Sparse convolutions reduce computation through learnable masks but depend strongly on selecting meaningful foreground regions.
- Fixed activation ratios can increase computation or reduce accuracy because aerial foreground coverage varies substantially across scales and viewpoints.
- CEASC combines context-enhanced sparse convolution with adaptive multi-layer masking to balance detection accuracy and efficiency.
- CEASC reduces computational costs across RetinaNet and GFL V1 while maintaining competitive accuracies on drone benchmarks.
2. Related Work
Aerial-image detectors commonly target tiny objects through coarse-to-fine processing or sparse detection heads, but these approaches face efficiency or context-coverage challenges. CEASC instead optimizes detection heads across FPN layers using global context and adaptive masking.
- Object Detection on Aerial Images: Coarse-to-fine aerial detectors first locate large instances or dense sub-regions, then apply fine detection to find small objects.
- CEASC: CEASC optimizes detection heads at FPN layers by combining focal information, global context, and adaptive mask ratios.
- Object Detection on Aerial Images: Coarse-to-fine methods can achieve high accuracy but require multiple inferences per image, limiting use on resource-constrained UAV platforms.
- Efficient Detection: Lightweight detection research includes neural architecture search, pruning, knowledge distillation, and lightweight model design.
- Efficient Detection: Lightweight backbones use techniques such as depth-wise separable convolutions and group convolutions, while lightweight heads include efficient or re-parameterized designs.
- Sparse Convolution: Sparse detection heads reduce computation with pixel-wise masks, including FPN feature gating and cascade sparse queries for high-resolution tiny-object detection.
3. Method
CEASC optimizes sparse detection heads by combining global-context enhancement with adaptive masking across FPN levels. Its components target missing contextual information and unstable foreground coverage while balancing accuracy and efficiency.
- 3. Method: CEASC is designed as a plug-and-play detection-head optimization that can be integrated with multiple base detectors.The paper evaluates integration with GFL V1, RetinaNet, Faster-RCNN, and FSAF; Table 1 reports AP/AR and GFLOPs/FPS on VisDrone.
- Motivation: Dense detection heads process entire feature maps, wasting computation on background despite tiny drone objects occupying limited foreground regions.Sparse convolution instead operates on masked foreground areas to accelerate inference.
- 3.1.1 Sparse Convolution: Sparse convolution converts learned soft features into binary masks, so only locations with mask value 1 undergo inference-time convolution.The mask is generated with Gumbel-Softmax during training and thresholding during inference.
- 3.1.2 Context Enhancement: CESC injects global contextual features into sparse convolution layers by normalizing sparse outputs with the global feature mean and standard deviation.CE-GN replaces statistics from activated sparse elements, while residual addition further preserves context.
- 3.2. Adaptive Multi-layer Masking: AMM estimates distinct foreground activation ratios at FPN levels and trains masks to match those ratios, avoiding the fixed-ratio trade-off between computation and foreground coverage.The target ratio is computed from positive foreground pixels divided by all feature-map pixels, and the masking loss penalizes deviations.
4. Experiments
Experiments evaluate CEASC across drone benchmarks, detector architectures, ablations, normalization choices, global-context encoders, and masking strategies. The results show substantial efficiency gains while preserving competitive detection accuracy.
- Evaluation on Different Detectors: CEASC reduces GFLOPs by at least 60% and increases FPS by 20%–60% across four base detectors, with slight mAP fluctuations.The evaluated detectors are GFL V1, RetinaNet, Faster-RCNN, and FSAF.
- On Detailed Designs in CESC: CE-GN improves accuracy over no normalization by 2.6% and exceeds GN, BN, and IN by 0.7%, 2,6%, and 0.8%, respectively.Feature visualizations also report higher correlation with dense-convolution features when CE-GN replaces GN.
- On Detailed Designs in CESC: Point-wise convolution achieves the highest detection accuracy among compared context encoders, the lowest GFLOPs among convolution-based approaches, and the highest FPS overall.The comparison includes plain 3 × 3 convolution, GhostModule, CBAM, and Criss-Cross Attention.
- On Adaptive Multi-Layer Masking: AMM outperforms fixed mask ratios by adapting to scale, while optimal fixed ratios differ between VisDrone and UAVDT.The reported mAP-optimal fixed ratios are 0.9 on VisDrone and 0.95 on UAVDT; layer-wise estimation also outperforms global estimation in mAP and FPS.
- Comparison to SOTA: Against the baseline GFL V1, CEASC cuts GFLOPs by 71.4% and raises inference FPS by 60% with a 0.3% mAP improvement on VisDrone.On UAVDT, it reduces GFLOPs by 76.3%, increases inference speed by 38.9%, and gains 0.2% mAP.
5. Conclusion
The paper proposes CEASC, a plug-and-play detection-head optimization approach for drone imagery. Its CESC and AMM modules compensate for lost global context and adapt mask ratios, accelerating detectors with competitive accuracy.
- Conclusion: CEASC combines CESC with CE-GN and AMM to enhance context, stabilize foreground representations, and adapt mask ratios across feature-pyramid levels.The approach is designed as a plug-and-play optimization for existing detectors.
- Conclusion: Experiments on VisDrone and UAVDT show that CEASC accelerates various base detectors while maintaining competitive accuracies.The conclusion summarizes the method's cross-benchmark efficiency and accuracy outcome.
A. More Implementation Details
The implementation adapts detector-specific settings for fair comparisons and evaluates CEASC with multiple detector architectures and lightweight baselines.
- Detector Settings: RetinaNet and FSAF use the GFL V1 setting, while Faster-RCNN uses a modified RPN head with four Convolution-GN-ReLU layers and 256 channels.The Faster-RCNN modification follows QueryDet and is intended to balance accuracy and efficiency.
- Comparison Settings: For MobileNet V2 and ShuffleNet V2, selected feature-map layers feed the FPN; QueryDet is reimplemented with unified 1,333×800 inputs and without P2 computation.These settings support the comparison reported in Table 8.
B. Additional Ablation Studies
Additional ablations examine residual structures and contextual-feature choices. The reported results favor the proposed residual design for capturing global context.
- On Residual Structures: The ablation compares no residual, a raw-input skip connection F := F + X, and the proposed global-context skip connection F := F + G.These variants test how residual information is incorporated into CEASC.
- On Residual Structures: The proposed residual approach reaches the best performance, highlighting its advantage in capturing global context.The result is reported in the supplementary ablation table.
B.2. On Acceleration Strategies
The ablation compares FPN-layer selection and depth-wise separable convolutions as acceleration strategies. The complete approach outperforms the compared alternatives in both accuracy and efficiency.
- Acceleration strategies: The study evaluates using only P3+P4 FPN layers and replacing standard convolutions with DWS convolutions.P3 and P4 are selected because P5–P7 are unlikely to activate for sparse convolutions.
- Evaluation: The results are summarized using accuracy and efficiency measures, including mAP, GFLOPs, and FPS.
- Results: CEASC outperforms the compared counterparts in both accuracy and efficiency.The reported advantage is attributed to context-enhanced and adaptively masked sparse convolutions.
B.3. On Context Clues
The context-clue ablation shows that interpolation is less favorable than the compared approach because it reduces accuracy while increasing computation.
- Context clues: Interpolation from focal areas causes a drop in accuracy.
- Context clues: Interpolation consumes more computations than the compared context-clue alternative.
- Evaluation: The comparison evaluates context clues using mAP, GFLOPs, and FPS on VisDrone.
B.4. On Training Epochs
The training-epoch study tests CEASC under multiple epoch settings and reports consistent performance gains. More epochs yield higher accuracy, while 15 epochs is identified as a good trade-off.
- B.4. On Training Epochs: CEASC consistently boosts performance by a large margin across 12, 15, and 24 training epochs.
- B.4. On Training Epochs: Using more training epochs produces higher accuracy.
- B.4. On Training Epochs: The study identifies 15 training epochs as a good trade-off.
B.5. More Visualized Results
Additional visualizations examine CE-GN feature correlations and AMM-generated masks across FPN layers. The masks cover foreground areas, while CE-GN features more closely resemble dense-convolution features.
- B.5. More Visualized Results: AMM-generated masks cover foreground areas across visualized FPN layers.Highlighted regions indicate areas activated for computation.
- B.5. More Visualized Results: The visualizations connect foreground coverage with sparse convolutions concentrating computation on activated regions.The reported effect is improved efficiency without sacrificing much precision.
- B.5. More Visualized Results: CE-GN-normalized features have higher correlation with dense-convolution features than GN-normalized features.This indicates that CE-GN enhances focal features using global context.