UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery

Huaxiang Zhang; Kai Liu; Zhongxue Gan; Guo-Niu Zhu

Source-linked AI summary

UAV-DETR: Efficient End-to-End Object Detection for Unmanned Aerial Vehicle Imagery

Huaxiang Zhang, Kai Liu, Zhongxue Gan, Guo-Niu Zhu

arXiv:2501.01855v3Jan 3, 2025cs.CV

TL;DR

UAV object detection is challenged by manually tuned components and end-to-end detectors designed mainly for natural images. UAV-DETR addresses this with spatial-frequency multi-scale fusion, frequency-focused downsampling, and semantic alignment, improving accuracy across UAV datasets, including VisDrone.

Problem
Existing UAV-OD methods rely on manually designed components requiring extensive tuning, while end-to-end models are mainly designed for natural images and are less effective for UAV imagery.
Source: §I · p. 1
Method
UAV-DETR is an efficient end-to-end detector using multi-scale spatial-frequency fusion, frequency-focused downsampling, and semantic alignment across fusion paths.
Source: §I · pp. 1–2
Results
On VisDrone, UAV-DETR-R18 improves AP by 3.1% and AP50 by 4.2% over baseline RT-DETR-R18, while UAVVaste results maintain a competitive advantage.
Source: B. Comparative Experiments · pp. 4–5
Takeaways & Limitations
UAV-DETR achieves higher accuracy than existing approaches with similar computational costs while maintaining real-time inference speeds on VisDrone and UAVVaste.
Source: §V · p. 6

Abstract

from arXiv · show

Unmanned aerial vehicle object detection (UAV-OD) has been widely used in various scenarios. However, most existing UAV-OD algorithms rely on manually designed components, which require extensive tuning. End-to-end models that do not depend on such manually designed components are mainly designed for natural images, which are less effective for UAV imagery. To address such challenges, this paper proposes an efficient detection transformer (DETR) framework tailored for UAV imagery, i.e., UAV-DETR. The framework includes a multi-scale feature fusion with frequency enhancement module, which captures both spatial and frequency information at different scales. In addition, a frequency-focused down-sampling module is presented to retain critical spatial details during down-sampling. A semantic alignment and calibration module is developed to align and fuse features from different fusion paths. Experimental results demonstrate the effectiveness and generalization of our approach across various UAV imagery datasets. On the VisDrone dataset, our method improves AP by 3.1\% and $\text{AP}_{50}$ by 4.2\% over the baseline. Similar enhancements are observed on the UAVVaste dataset. The project page: https://github.com/ValiantDiligent/UAV-DETR

I. INTRODUCTION

UAV imagery creates distinctive detection challenges, while existing end-to-end detectors are mainly designed for natural images. UAV-DETR addresses this gap with multi-scale spatial-frequency processing, frequency-focused downsampling, and semantic alignment for small and occluded objects.

UAV-OD algorithms often rely on manually designed components such as NMS and anchor boxes, requiring extensive task-specific tuning.Source: §I · p. 1
Existing DETR models face high computational cost and poor real-time performance, while natural-image design limits their suitability for UAV imagery.Source: §I · p. 1
UAV imagery contains small and occluded objects, making detailed feature extraction and surrounding contextual information valuable for detection.Source: §I · p. 1
UAV-DETR combines spatial and frequency information across multiple scales to preserve high-frequency components and critical spatial details.Source: §I · p. 1
The framework includes MSFF-FE, frequency-focused downsampling, and semantic alignment to enhance small-object detection, preserve dual-domain information, and align fusion paths.Source: §I · pp. 1–2

II. RELATED WORK

Related UAV-OD methods balance accuracy, efficiency, and deployment practicality, but commonly depend on costly pipelines, NMS, or spatial-domain features. UAV-DETR combines real-time end-to-end detection with multi-scale fusion across spatial and frequency domains and learned feature alignment.

Two-stage coarse-to-fine UAV-OD methods can achieve high accuracy but introduce computational overhead unsuitable for resource-limited environments.Source: A. Object Detection in UAV Imagery · p. 2
Most UAV-OD methods optimize model lightweighting or processing pipelines, while frequency-domain information remains underutilized.Source: A. Object Detection in UAV Imagery · p. 2
YOLO-based detectors balance performance and real-time capability but typically require NMS, adding inference cost and potentially unstable hyperparameters.Source: B. Real-Time End-to-End Object Detection · p. 2
RT-DETR removes NMS and uses attention-based intra-scale interaction, CNN-based cross-scale fusion, and uncertainty-minimal query selection.Source: B. Real-Time End-to-End Object Detection · p. 2
Feature fusion across levels can suffer semantic gaps and spatial misalignment, especially for small and densely distributed objects.Source: C. Feature Fusion · p. 2
UAV-DETR performs multi-scale fusion in spatial and frequency domains and uses learned offsets to align features across fusion paths.Source: C. Feature Fusion · p. 2

III. METHODOLOGY

UAV-DETR extends an RT-DETR-based architecture with modules for multi-scale spatial-frequency fusion, frequency-focused processing, and semantic feature alignment. Its fusion design preserves high-frequency and multi-scale information while controlling computational cost through partial processing and residual connections.

UAV-DETR builds on RT-DETR with MSFF-FE, frequency-focused downsampling, SAC, and Inner-SIoU replacing GIoU.Source: §III · p. 2
MSFF-FE preserves small-object details by combining spatial and frequency information across multiple scales.Source: A. Multi-Scale Feature Fusion with… · p. 2
The Focus module compresses spatial information into channels, while cross-stage partial processing divides features so only one branch receives multi-scale and frequency enhancement.Source: A. Multi-Scale Feature Fusion with… · p. 2
Three convolution kernel sizes capture multi-scale information from the spatial feature branch.Source: A. Multi-Scale Feature Fusion with… · p. 3
The Frequency-Focused module uses learned parameters to balance spatial and frequency components, dynamically adjusts frequency-domain filtering, and retains spatial features through residual connections.Source: A. Multi-Scale Feature Fusion with… · p. 3
Large kernels capture long-range dependencies, small kernels optimize channel-wise information, and residual connections support training before final branch fusion.Source: A. Multi-Scale Feature Fusion with… · p. 3

B. Frequency-Focused Downsampling

The frequency-focused downsampling module processes pooled features through parallel branches that preserve key features and enhance important components before recombination. Their outputs are concatenated and projected to the desired channel size.

FD first applies average pooling, divides the pooled feature map along channels, and processes the two parts in parallel.Source: B. Frequency-Focused Downsampling · p. 3
One branch uses stride-2 convolution to reduce spatial dimensions while preserving key features.Source: B. Frequency-Focused Downsampling · p. 3
The other branch applies frequency enhancement on one path and max pooling followed by 1×1 convolution on another.Source: B. Frequency-Focused Downsampling · p. 3
The parallel outputs are concatenated and passed through a 1×1 convolution to reduce channels to the desired size.Source: B. Frequency-Focused Downsampling · p. 3
The two processed branches are concatenated to form the module’s final output.Source: B. Frequency-Focused Downsampling · p. 3

C. Semantic Alignment and Calibration

The semantic alignment and calibration module fuses features from different fusion processes by balancing spatial and frequency information, aligning feature coordinates, and weighting the aligned outputs.

SAC fuses features obtained from different fusion processes and is designed to align them semantically.Source: C. Semantic Alignment and Calibration · p. 3
The module unifies channels, upsamples x2, and generates a frequency-enhanced feature before fusion with the original feature.Source: C. Semantic Alignment and Calibration · pp. 3–4
A learned gating function adaptively balances spatial and frequency-domain information during feature fusion.Source: C. Semantic Alignment and Calibration · p. 4
Learned 2D offsets adjust sampling grids through grid-based sampling to align x1 and xfused spatially.Source: C. Semantic Alignment and Calibration · p. 4
Element-wise weighted summation combines the aligned features using learned attention weights α and β.Source: C. Semantic Alignment and Calibration · p. 4

D. LOSS Function

The paper replaces GIoU with Inner-SIoU to improve small-object box regression by combining expanded-box overlap with SIoU-based geometric penalties.

Inner-SIoU replaces GIoU because GIoU is less effective for small objects, especially at low IoU values.Source: D. LOSS Function · p. 4
Inner-SIoU combines Inner-IoU and SCYLLA-IoU to enhance small-object detection and geometric alignment.Source: D. LOSS Function · p. 4
The auxiliary bounding box is scaled by 1.25 to improve sensitivity and speed convergence.Source: D. LOSS Function · p. 4
Inner-IoU computes overlap using expanded predicted and ground-truth boxes whose width and height are scaled by 1.25.Source: D. LOSS Function · p. 4
The Inner-SIoU loss combines standard IoU loss with LSIoU, which includes angle, distance, and shape penalties.Source: D. LOSS Function · p. 4

A. Experimental Setup

Experiments use VisDrone and UAVVaste to evaluate detection performance and cross-dataset generalization, with three UAV-DETR backbone variants trained under a common setup.

Experiments evaluate UAV-DETR on VisDrone and UAVVaste object-detection datasets.Source: A. Experimental Setup · p. 4
VisDrone-2019-DET contains 6,471 training images, 548 validation images, and 3,190 test images across ten annotated object categories.Source: A. Experimental Setup · p. 4
UAVVaste contains 772 images and 3716 hand-labeled waste annotations from urban and natural environments.Source: A. Experimental Setup · p. 4
The study uses ResNet18, ResNet50, and EfficientFormerV2 backbones for three UAV-DETR variants.Source: A. Experimental Setup · p. 4
Models are trained for 400 epochs with batch size 4, AdamW optimization, and early stopping with patience 20.Source: A. Experimental Setup · p. 4

B. Comparative Experiments

Comparative experiments show that UAV-DETR improves accuracy over RT-DETR and remains competitive across computational budgets and datasets, including the smaller UAVVaste dataset.

3.1% improvement in AP and 4.2% increase in AP50 are achieved by UAV-DETR-R18 over baseline RT-DETR-R18 on VisDrone.Source: B. Comparative Experiments · p. 4
3.1% increase in AP and 4.1% rise in AP50 are achieved by UAV-DETR-R50 over its baseline on VisDrone.Source: B. Comparative Experiments · p. 4
UAV-DETR-R18 outperforms all methods below 100 GFLOPs and achieves the best accuracy in that computational class.Source: B. Comparative Experiments · p. 4
UAV-DETR also outperforms similarly costly object detectors in accuracy, including comparisons with PP-YOLOE-P2-Alpha-l.Source: B. Comparative Experiments · p. 5
On UAVVaste, UAV-DETR maintains a competitive advantage despite the dataset containing less data than VisDrone.Source: B. Comparative Experiments · p. 5, Table II · p. 5

C. Ablation Studies

The ablation study shows that successive additions to RT-DETR-R18 improve detection accuracy, with the full UAV-DETR configuration achieving the strongest reported AP and AP50 while largely retaining real-time performance.

Experimental setup

The study evaluates configurations using Inner-SIoU, MSFF-FE, FD, and SAC, with Table III designated as the ablation comparison.The modules are defined as Inner-SIoU, Multi-Scale Feature Fusion with Frequency Enhancement, Frequency-Focused Downsampling, and Semantic Alignment and Calibration.Source: C. Ablation Studies · pp. 5–6, Table III · p. 5

Component ablation

Adding MSFF-FE raises AP from the baseline’s 26.7 to 28.4, supporting multi-scale feature fusion with frequency enhancement.The baseline RT-DETR-R18 has an AP of 26.7 and AP50 of 44.6.Source: C. Ablation Studies · p. 6

Runtime

UAV-DETR largely maintains RT-DETR’s real-time performance under the reported PyTorch, 32-bit floating-point evaluation.The FPS comparison is reported in Table V.Source: Table V · p. 5, C. Ablation Studies · p. 6

D. Visualization

Visualization results indicate that UAV-DETR focuses more effectively on small objects and their surrounding context than the baseline, aiding localization of occluded objects while still showing occasional noise misfocus.

Small-object localization

UAV-DETR produces higher heat values around small objects than the baseline, indicating improved localization of these objects.The visualization uses backpropagation through bounding-box predictions on VisDrone images.Source: D. Visualization · p. 6

Contextual information

The model also attends to surrounding information, supporting its use of contextual features when small-object details are insufficient.The paper links this contextual attention to improved detection of small objects.Source: D. Visualization · p. 6, E. Discussion · p. 6

Feature representation

UAV-DETR combines spatial and frequency information to retain high-frequency features and preserve details important for small-object detection.MSFF-FE and FD are introduced to address losses of edge and texture details during feature fusion and downsampling.Source: E. Discussion · p. 6, §V · p. 6

Feature alignment

SAC aligns semantic and spatial information across feature maps after frequency-domain operations can create misalignment.The module aligns features from different fusion paths.Source: E. Discussion · p. 6, §V · p. 6

Occlusion and failure cases

The yellow-box examples show better localization of occluded objects, while red-box regions reveal occasional attention to irrelevant noise.The noise-focused regions are identified as a remaining challenge for future work.Source: E. Discussion · p. 6