Source-linked AI summary
UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery
Huaxiang Zhang, Kai Liu, Zhongxue Gan, Guo-Niu Zhu
TL;DR
UAV object detection is challenged by manually tuned components and end-to-end detectors designed mainly for natural images. UAV-DETR addresses this with spatial-frequency multi-scale fusion, frequency-focused downsampling, and semantic alignment, improving accuracy across UAV datasets, including VisDrone.
Problem
Existing UAV-OD methods rely on manually designed components requiring extensive tuning, while end-to-end models are mainly designed for natural images and are less effective for UAV imagery.
Method
UAV-DETR is an efficient end-to-end detector using multi-scale spatial-frequency fusion, frequency-focused downsampling, and semantic alignment across fusion paths.
Results
On VisDrone, UAV-DETR-R18 improves AP by 3.1% and AP50 by 4.2% over baseline RT-DETR-R18, while UAVVaste results maintain a competitive advantage.
Takeaways & Limitations
UAV-DETR achieves higher accuracy than existing approaches with similar computational costs while maintaining real-time inference speeds on VisDrone and UAVVaste.
Abstract
from arXiv · showhide
Unmanned aerial vehicle object detection (UAV-OD) has been widely used in various scenarios. However, most existing UAV-OD algorithms rely on manually designed components, which require extensive tuning. End-to-end models that do not depend on such manually designed components are mainly designed for natural images, which are less effective for UAV imagery. To address such challenges, this paper proposes an efficient detection transformer (DETR) framework tailored for UAV imagery, i.e., UAV-DETR. The framework includes a multi-scale feature fusion with frequency enhancement module, which captures both spatial and frequency information at different scales. In addition, a frequency-focused down-sampling module is presented to retain critical spatial details during down-sampling. A semantic alignment and calibration module is developed to align and fuse features from different fusion paths. Experimental results demonstrate the effectiveness and generalization of our approach across various UAV imagery datasets. On the VisDrone dataset, our method improves AP by 3.1\% and $\text{AP}_{50}$ by 4.2\% over the baseline. Similar enhancements are observed on the UAVVaste dataset. The project page: https://github.com/ValiantDiligent/UAV-DETR
I. INTRODUCTION
UAV imagery creates distinctive detection challenges, while existing end-to-end detectors are mainly designed for natural images. UAV-DETR addresses this gap with multi-scale spatial-frequency processing, frequency-focused downsampling, and semantic alignment for small and occluded objects.
- UAV-OD algorithms often rely on manually designed components such as NMS and anchor boxes, requiring extensive task-specific tuning.
- Existing DETR models face high computational cost and poor real-time performance, while natural-image design limits their suitability for UAV imagery.
- UAV imagery contains small and occluded objects, making detailed feature extraction and surrounding contextual information valuable for detection.
- UAV-DETR combines spatial and frequency information across multiple scales to preserve high-frequency components and critical spatial details.
- The framework includes MSFF-FE, frequency-focused downsampling, and semantic alignment to enhance small-object detection, preserve dual-domain information, and align fusion paths.
II. RELATED WORK
Related UAV-OD methods balance accuracy, efficiency, and deployment practicality, but commonly depend on costly pipelines, NMS, or spatial-domain features. UAV-DETR combines real-time end-to-end detection with multi-scale fusion across spatial and frequency domains and learned feature alignment.
- Two-stage coarse-to-fine UAV-OD methods can achieve high accuracy but introduce computational overhead unsuitable for resource-limited environments.
- Most UAV-OD methods optimize model lightweighting or processing pipelines, while frequency-domain information remains underutilized.
- YOLO-based detectors balance performance and real-time capability but typically require NMS, adding inference cost and potentially unstable hyperparameters.
- RT-DETR removes NMS and uses attention-based intra-scale interaction, CNN-based cross-scale fusion, and uncertainty-minimal query selection.
- Feature fusion across levels can suffer semantic gaps and spatial misalignment, especially for small and densely distributed objects.
- UAV-DETR performs multi-scale fusion in spatial and frequency domains and uses learned offsets to align features across fusion paths.
III. METHODOLOGY
UAV-DETR extends an RT-DETR-based architecture with modules for multi-scale spatial-frequency fusion, frequency-focused processing, and semantic feature alignment. Its fusion design preserves high-frequency and multi-scale information while controlling computational cost through partial processing and residual connections.
- UAV-DETR builds on RT-DETR with MSFF-FE, frequency-focused downsampling, SAC, and Inner-SIoU replacing GIoU.
- MSFF-FE preserves small-object details by combining spatial and frequency information across multiple scales.
- The Focus module compresses spatial information into channels, while cross-stage partial processing divides features so only one branch receives multi-scale and frequency enhancement.
- Three convolution kernel sizes capture multi-scale information from the spatial feature branch.
- The Frequency-Focused module uses learned parameters to balance spatial and frequency components, dynamically adjusts frequency-domain filtering, and retains spatial features through residual connections.
- Large kernels capture long-range dependencies, small kernels optimize channel-wise information, and residual connections support training before final branch fusion.
B. Frequency-Focused Downsampling
The frequency-focused downsampling module processes pooled features through parallel branches that preserve key features and enhance important components before recombination. Their outputs are concatenated and projected to the desired channel size.
- FD first applies average pooling, divides the pooled feature map along channels, and processes the two parts in parallel.
- One branch uses stride-2 convolution to reduce spatial dimensions while preserving key features.
- The other branch applies frequency enhancement on one path and max pooling followed by 1×1 convolution on another.
- The parallel outputs are concatenated and passed through a 1×1 convolution to reduce channels to the desired size.
- The two processed branches are concatenated to form the module’s final output.
C. Semantic Alignment and Calibration
The semantic alignment and calibration module fuses features from different fusion processes by balancing spatial and frequency information, aligning feature coordinates, and weighting the aligned outputs.
- SAC fuses features obtained from different fusion processes and is designed to align them semantically.
- The module unifies channels, upsamples x2, and generates a frequency-enhanced feature before fusion with the original feature.
- A learned gating function adaptively balances spatial and frequency-domain information during feature fusion.
- Learned 2D offsets adjust sampling grids through grid-based sampling to align x1 and xfused spatially.
- Element-wise weighted summation combines the aligned features using learned attention weights α and β.
D. LOSS Function
The paper replaces GIoU with Inner-SIoU to improve small-object box regression by combining expanded-box overlap with SIoU-based geometric penalties.
- Inner-SIoU replaces GIoU because GIoU is less effective for small objects, especially at low IoU values.
- Inner-SIoU combines Inner-IoU and SCYLLA-IoU to enhance small-object detection and geometric alignment.
- The auxiliary bounding box is scaled by 1.25 to improve sensitivity and speed convergence.
- Inner-IoU computes overlap using expanded predicted and ground-truth boxes whose width and height are scaled by 1.25.
- The Inner-SIoU loss combines standard IoU loss with LSIoU, which includes angle, distance, and shape penalties.
A. Experimental Setup
Experiments use VisDrone and UAVVaste to evaluate detection performance and cross-dataset generalization, with three UAV-DETR backbone variants trained under a common setup.
- Experiments evaluate UAV-DETR on VisDrone and UAVVaste object-detection datasets.
- VisDrone-2019-DET contains 6,471 training images, 548 validation images, and 3,190 test images across ten annotated object categories.
- UAVVaste contains 772 images and 3716 hand-labeled waste annotations from urban and natural environments.
- The study uses ResNet18, ResNet50, and EfficientFormerV2 backbones for three UAV-DETR variants.
- Models are trained for 400 epochs with batch size 4, AdamW optimization, and early stopping with patience 20.
B. Comparative Experiments
Comparative experiments show that UAV-DETR improves accuracy over RT-DETR and remains competitive across computational budgets and datasets, including the smaller UAVVaste dataset.
- 3.1% improvement in AP and 4.2% increase in AP50 are achieved by UAV-DETR-R18 over baseline RT-DETR-R18 on VisDrone.
- 3.1% increase in AP and 4.1% rise in AP50 are achieved by UAV-DETR-R50 over its baseline on VisDrone.
- UAV-DETR-R18 outperforms all methods below 100 GFLOPs and achieves the best accuracy in that computational class.
- UAV-DETR also outperforms similarly costly object detectors in accuracy, including comparisons with PP-YOLOE-P2-Alpha-l.
- On UAVVaste, UAV-DETR maintains a competitive advantage despite the dataset containing less data than VisDrone.
C. Ablation Studies
The ablation study shows that successive additions to RT-DETR-R18 improve detection accuracy, with the full UAV-DETR configuration achieving the strongest reported AP and AP50 while largely retaining real-time performance.
- Experimental setup: The study evaluates configurations using Inner-SIoU, MSFF-FE, FD, and SAC, with Table III designated as the ablation comparison.The modules are defined as Inner-SIoU, Multi-Scale Feature Fusion with Frequency Enhancement, Frequency-Focused Downsampling, and Semantic Alignment and Calibration.
- Component ablation: Adding MSFF-FE raises AP from the baseline’s 26.7 to 28.4, supporting multi-scale feature fusion with frequency enhancement.The baseline RT-DETR-R18 has an AP of 26.7 and AP50 of 44.6.
- Runtime: UAV-DETR largely maintains RT-DETR’s real-time performance under the reported PyTorch, 32-bit floating-point evaluation.The FPS comparison is reported in Table V.
D. Visualization
Visualization results indicate that UAV-DETR focuses more effectively on small objects and their surrounding context than the baseline, aiding localization of occluded objects while still showing occasional noise misfocus.
- Small-object localization: UAV-DETR produces higher heat values around small objects than the baseline, indicating improved localization of these objects.The visualization uses backpropagation through bounding-box predictions on VisDrone images.
- Contextual information: The model also attends to surrounding information, supporting its use of contextual features when small-object details are insufficient.The paper links this contextual attention to improved detection of small objects.
- Feature representation: UAV-DETR combines spatial and frequency information to retain high-frequency features and preserve details important for small-object detection.MSFF-FE and FD are introduced to address losses of edge and texture details during feature fusion and downsampling.
- Feature alignment: SAC aligns semantic and spatial information across feature maps after frequency-domain operations can create misalignment.The module aligns features from different fusion paths.
- Occlusion and failure cases: The yellow-box examples show better localization of occluded objects, while red-box regions reveal occasional attention to irrelevant noise.The noise-focused regions are identified as a remaining challenge for future work.