Source-linked AI summary
The Unmanned Aerial Vehicle Benchmark: Object Detection and Tracking
Dawei Du, Yuankai Qi, Hongyang Yu, Yifan Yang, Kaiwen Duan, Guorong Li, Weigang Zhang, Qingming Huang, Qi Tian
TL;DR
Existing UAV datasets are limited in task and scenario coverage despite UAVs' broad surveillance advantages and distinctive challenges. The paper constructs UAVDT, a densely annotated benchmark covering detection and tracking across complex UAV scenes, then evaluates recent methods. Results show that current methods perform relatively worse on UAVDT, while the benchmark exposes challenges such as small objects and camera motion and highlights real-time constraints.
Problem
Existing UAV datasets are limited in task and scenario coverage, motivating a benchmark for unconstrained UAV scenes.
Method
The paper constructs UAVDT from 10 hours of UAV video, annotating about 80,000 frames with bounding boxes and attributes for DET, SOT, and MOT, then evaluates recent algorithms.
Results
Current state-of-the-art methods perform relatively worse on UAVDT, whose scenes include challenges such as high density, small objects, and camera motion.
Takeaways & Limitations
UAVDT provides a unified platform for evaluating detection and tracking methods in real UAV scenarios and exposes research needs around small objects and real-time performance.
Takeaways & Limitations
Embedded UAV platforms impose harsh computational-resource requirements, so future work should consider real-time constraints alongside accuracy.
Abstract
from arXiv · showhide
With the advantage of high mobility, Unmanned Aerial Vehicles (UAVs) are used to fuel numerous important applications in computer vision, delivering more efficiency and convenience than surveillance cameras with fixed camera angle, scale and view. However, very limited UAV datasets are proposed, and they focus only on a specific task such as visual tracking or object detection in relatively constrained scenarios. Consequently, it is of great importance to develop an unconstrained UAV benchmark to boost related researches. In this paper, we construct a new UAV benchmark focusing on complex scenarios with new level challenges. Selected from 10 hours raw videos, about 80,000 representative frames are fully annotated with bounding boxes as well as up to 14 kinds of attributes (e.g., weather condition, flying altitude, camera view, vehicle category, and occlusion) for three fundamental computer vision tasks: object detection, single object tracking, and multiple object tracking. Then, a detailed quantitative study is performed using most recent state-of-the-art algorithms for each task. Experimental results show that the current state-of-the-art methods perform relative worse on our dataset, due to the new challenges appeared in UAV based real scenes, e.g., high density, small object, and camera motion. To our knowledge, our work is the first time to explore such issues in unconstrained scenes comprehensively.
1 Introduction
UAVs offer mobile, wide-view surveillance but introduce high density, small objects, camera motion, and real-time constraints. Existing datasets are limited in task or scenario coverage, motivating UAVDT as a comprehensive unconstrained benchmark for detection and tracking.
- Existing datasets commonly use fixed or car-mounted cameras with limited surveillance viewing angles, while limited UAV datasets target specific tasks or constrained scenes.
- UAVs provide easy deployment, high mobility, a large view scope, and uniform scale compared with traditional surveillance cameras.
- UAV imagery creates high-density scenes, small objects, and rapid object motion or rotation from high-speed flight and camera movement.
- Real-time algorithms must maintain comparable accuracy on embedded UAV platforms with constrained computing resources.
- UAVDT provides about 80,000 representative frames from 10 hours of raw video for object detection, single-object tracking, and multiple-object tracking.
2 UAVDT Benchmark
UAVDT is a multi-task benchmark built from urban UAV videos and detailed annotations spanning shooting conditions, object properties, and tracking challenges. Compared with related datasets, it emphasizes denser objects and more varied real-world scenarios.
- Dataset construction: UAVDT contains 100 video sequences selected from over 10 hours of urban UAV footage recorded at 30 fps and 1080 × 540 resolution.
- Dataset construction: About 80,000 frames covering over 2,700 vehicles and 0.84 million bounding boxes were annotated by more than 10 domain experts with double-checking.
- Attribute annotation: MOT annotations define weather condition, flying altitude, and camera view, while DET annotations additionally cover vehicle category, occlusion, and out-of-view.
- Attribute annotation: Flying altitude changes object scale and clarity: high-altitude videos can contain more than 100 vehicles while tiny objects occupy about 0.005% of a frame.
- Attribute annotation: SOT sequences include eight attributes, and 74% of videos contain at least four visual challenges while 27% of frames belong to long-term tracking videos.
- Evaluation design: The benchmark uses 30 training and 70 testing sequences from different locations to reduce overfitting to particular scenarios.
- Comparison with existing datasets: UAVDT has object density 10.524, exceeding UAV123 at 1.00, Campus at 0.02, DETRAC at 8.64, and KITTI at 5.35.
- Comparison with existing datasets: Unlike related datasets focused on specified scenes, UAVDT spans different weather conditions, flying altitudes, and camera views.
3 Evaluation and Analysis
The benchmark evaluates detection, multiple-object tracking, and single-object tracking methods under complex UAV conditions. Across tasks, current methods face substantial accuracy, robustness, and runtime challenges, especially with small, dense, occluded, or fast-moving objects.
- Evaluation Protocol: Representative state-of-the-art algorithms are trained on the training set and evaluated on the testing set, revealing failures in complex scenarios.The benchmark evaluates methods for each task using available implementations or author-provided code.
- Object Detection: 34.35% AP is achieved by R-FCN on this dataset versus 70.06% AP on KITTI-D hard set.The authors associate the gap with many small objects and cluttered backgrounds caused by UAV shooting perspectives and altitude.
- Object Detection: Detection performance drops sharply for large vehicles, occlusion, and out-of-view conditions, while low-altitude and side-view sequences with more object detail perform better.The authors attribute difficulty partly to scarce truck and bus examples and interference affecting small objects.
- Object Detection: Region-based detectors obtain relatively good accuracy but run below 5fps, whereas region-free methods operate at almost real-time speed.This exposes a practical accuracy–runtime trade-off under constrained computing resources.
- Multiple Object Tracking: MDP with Faster-RCNN achieves 43.0 MOTA and 61.5 IDF1, while SORT with Faster-RCNN reaches 59.8 ± 10.3 MOTA on MOT16 but performs much worse here.High object density increases false positives, false negatives, identity switches, and fragmentations in the UAV dataset.
- Multiple Object Tracking: MOT accuracy varies with detection input and attributes: Faster-RCNN helps when details are clear, while R-FCN provides more stable inputs in fog and high-altitude scenes.SSD and RON produce more accurate candidates and more balanced tracking performance across attributes.
- Single Object Tracking: All evaluated SOT trackers perform poorly relative to established benchmarks; MDNet obtains 46.4 success and 72.5 precision versus OTB100’s 69.4 and 92.8.KCF also scores 29.0 success here versus 33.1 on UAV-123, indicating additional difficulty in this UAV setting.
- Single Object Tracking: MDNet performs best on 7 of 8 SOT attributes, but the most accurate trackers are far from real time, with MDNet running at 0.28fps.CPU-real-time trackers such as Staple-CA and KCF achieve success scores of 39.5 and 29.0, respectively.
4 Discussion
The benchmark exposes weaknesses in current detection and tracking methods under realistic UAV conditions, highlighting challenges from real-time constraints, mutable scenes, motion, and small objects.
- Current algorithms generally perform poorly on the benchmark compared with their plausible performance on other datasets.
- Realtime issues: Real-time deployment remains difficult because deep learning methods require harsh computational resources on embedded UAV platforms.The paper discusses pruning, compression, and low-bit representations as approaches to improve efficiency while maintaining comparable accuracy.
- Scene priors: Scene priors may improve robustness because different methods perform best in different scenarios.The discussion points to sequence-specific object-background classifiers as one way to handle mutable scenes.
- Motion clues: Motion clues remain important because appearance information is not always reliable amid fierce object and background motion.The paper notes that LSTM-based approaches pursue this direction but have not yet met expectations.
5 Conclusion
The paper introduces a challenging UAV benchmark spanning detection and tracking in complex real-world scenarios. It combines large-scale annotated video data, circumstance attributes, and extensive evaluation to support unified assessment of real-scene methods.
- The benchmark contains 100 UAV-captured videos totaling 80k frames from complex scenarios for DET, MOT, and SOT.
- All frames have manually labeled bounding boxes and three circumstance attributes: weather condition, flying altitude, and camera view.The SOT dataset adds eight attributes, including background clutter, camera rotation, and small object.
- An extensive evaluation of recent state-of-the-art methods establishes a unified platform for assessing detection and tracking methods in real scenarios.