PaperlayerView exact revision on arXiv

Source-linked AI summary

Detection and Tracking Meet Drones Challenge

Pengfei Zhu, Longyin Wen, Dawei Du, Xiao Bian, Heng Fan, Qinghua Hu, Haibin Ling

arXiv:2001.06303v3cs.CV

TL;DR

Drone-captured imagery poses unresolved detection and tracking challenges, while existing benchmarks provide limited scale and scenario coverage. The paper introduces and evaluates VisDrone, a large-scale benchmark with four tracks collected across diverse Chinese urban and suburban settings. Across the challenges, methods improve through architectures such as multi-scale and cascade models, but video temporal modeling and real-world deployment remain open problems.

  • Problem

    Drone imagery introduces viewpoint, scale, motion-blur, and occlusion challenges, while existing datasets remain limited for comprehensive algorithm evaluation.

  • Method

    The paper constructs VisDrone, organizes three workshops, reviews submitted methods, and evaluates image detection, video detection, single-object tracking, and multi-object tracking.

  • Results

    VisDrone challenge methods improve detection and tracking performance using approaches including deformable convolution, global context, multi-scale features, ensembles, and cascades.

  • Takeaways & Limitations

    VisDrone provides a broad evaluation platform for advancing object detection and tracking research on drone-captured imagery.

  • Takeaways & Limitations

    Video object detection remains limited because most submissions rely on image detectors and temporal consistency modeling is still an open question.

Abstract

from arXiv · show

Drones, or general UAVs, equipped with cameras have been fast deployed with a wide range of applications, including agriculture, aerial photography, and surveillance. Consequently, automatic understanding of visual data collected from drones becomes highly demanding, bringing computer vision and drones more and more closely. To promote and track the developments of object detection and tracking algorithms, we have organized three challenge workshops in conjunction with ECCV 2018, ICCV 2019 and ECCV 2020, attracting more than 100 teams around the world. We provide a large-scale drone captured dataset, VisDrone, which includes four tracks, i.e., (1) image object detection, (2) video object detection, (3) single object tracking, and (4) multi-object tracking. In this paper, we first present a thorough review of object detection and tracking datasets and benchmarks, and discuss the challenges of collecting large-scale drone-based object detection and tracking datasets with fully manual annotations. After that, we describe our VisDrone dataset, which is captured over various urban/suburban areas of 14 different cities across China from North to South. Being the largest such dataset ever published, VisDrone enables extensive evaluation and investigation of visual analysis algorithms for the drone platform. We provide a detailed analysis of the current state of the field of large-scale object detection and tracking on drones, and conclude the challenge as well as propose future directions. We expect the benchmark largely boost the research and development in video analysis on drone platforms. All the datasets and experimental results can be downloaded from https://github.com/VisDrone/VisDrone-Dataset.

1 INTRODUCTION

Drone-captured visual data creates demanding conditions for object detection and tracking, while existing benchmarks remain limited in scale and scenario coverage. VisDrone addresses this gap with a large-scale benchmark spanning four detection and tracking tracks.

  • Challenges: Drone imagery introduces viewpoint, scale, and motion-blur variations that challenge conventional vision algorithms.Drone cameras monitor objects from arbitrary viewpoints and altitudes, while movement causes considerable video motion blur.
  • Motivation: Existing drone datasets remain limited in size and covered scenarios, restricting thorough evaluation of detection and tracking algorithms.The paper identifies a need for a more general and comprehensive benchmark for drone-platform video analysis.
  • Challenge program: More than 100 research teams participated in three challenge workshops held with ECCV 2018, ICCV 2019, and ECCV 2020.The challenges focused on object detection and tracking across four tracks.
  • Benchmark scope: VisDrone provides image detection, video detection, single-object tracking, and multi-object tracking tracks for drone-captured data.The tracks cover detection in individual images and videos, online target-state estimation, and trajectory recovery.
  • Dataset: The dataset contains 263 video clips, 179,264 frames, 10,209 static images, and more than 2.5 million annotated bounding boxes.Data spans urban and rural regions, 14 Chinese cities, varied object types, and sparse or crowded scenes.

2 RELATED WORK

Prior work includes broad detection and tracking benchmarks, specialized drone datasets, and challenges targeting UAV algorithm development. VisDrone is positioned as a broader dataset collected across varied urban scenes and drone-specific conditions.

  • Benchmarks and surveys: Existing surveys organize object detection, single-object tracking, and multi-object tracking methods, while workshops promote UAV algorithm development and comparison.The cited surveys cover detection frameworks and tracking methods; UAV workshops address hardware, software, efficiency, and repeatable evaluation.
  • Existing datasets: Established benchmarks cover image detection, video detection, and visual tracking, including PASCAL VOC, MS COCO, ILSVRC, TrackingNet, and LaSOT.These datasets target generic object categories, video objects, or long-term tracking across diverse contexts.
  • Drone datasets: Drone-captured datasets have been proposed for detection, tracking, and semantic segmentation, but many focus on limited scenarios or tasks.Examples include aerial car counting, action detection, single-object tracking, and campus-based video sequences.
  • VisDrone: VisDrone differs by collecting data in varied urban scenes with large scale and viewpoint variations and heavy occlusions.The dataset is described as systematically collected to advance drone-based detection and tracking research.

3 VISDRONE OVERVIEW

VisDrone is a large-scale, multi-task drone benchmark assembled from diverse platforms, locations, conditions, and evaluation subsets. Its organization supports training, validation, challenge competition, and public testing.

  • Dataset collection: VisDrone contains 263 video clips with 179,264 frames and 10,209 static images collected from diverse drone platforms and 14 Chinese cities.The data covers varied weather, lighting, and daily-life scenarios.
  • Evaluation organization: The benchmark uses separate training, validation, test-challenge, and test-dev subsets for development, competition, and public evaluation.For each task, subsets are captured at different locations while sharing similar scenarios and attributes.
  • Challenge procedure: Participants submit object locations or trajectories for selected tasks using the provided training data, with additional training data allowed after declaration.The paper describes account registration and task-specific submission procedures.

4 DET TRACK

The DET track evaluates multi-category localization in challenging drone images using category- and IoU-based metrics. Results show that multi-scale, attention, dilated-convolution, ensemble, and cascade strategies improve detection, but small objects remain difficult.

  • Task and data: The DET track requires bounding boxes and confidence scores for all instances of predefined categories in each image.Its dataset contains 10,209 images divided among training, validation, test-challenge, and test-dev subsets.
  • Annotation: Ten people and vehicle categories are annotated with occlusion and truncation attributes for detailed algorithm analysis.Instances with truncation ratios above 50% are excluded from evaluation.
  • Metrics: AP averages performance across ten IoU thresholds from 0.50 to 0.95, while AP50 and AP75 use single IoU thresholds.AR1, AR10, AR100, and AR500 measure maximum recall under specified detection limits.
  • Methods and results: The strongest detectors address small objects using multi-scale features, attention, dilated convolution, ensemble mechanisms, or cascade architectures.These components extract salient semantic features with larger receptive fields or combine complementary detectors and localization branches.
  • Methods and results: 34.57 mAP is achieved by DroneEye2020 in VisDrone-DET2020, compared with 16.09% mAP for the baseline Cascade R-CNN.The result is attributed to recursive feature pyramids and switchable strous convolution within a Cascade R-CNN-based system.
  • Discussion: The best AP remains below 35%, and person and bicycle detection remains below 25% mAP, showing persistent difficulty with small-scale objects.The discussion identifies robust real-world detection as an ongoing need.

5 VID TRACK

The VID track evaluates video object detection under degraded appearances and temporal challenges, using trajectory-rich drone videos. Results show continued gains from stronger detectors, while effective long-range temporal modeling remains unresolved.

  • Track definition: VID requires bounding boxes and confidences for predefined object categories in every video frame, adding temporal consistency beyond image detection.The track uses video clips rather than individual images and evaluates the resulting detections with DET metrics.
  • Dataset: The VID data contain 96 clips across training, validation, challenge-test, and dev-test subsets, with severe category imbalance.Training includes 56 clips and 24,198 frames; car trajectories exceed bus trajectories by more than 50×.
  • Methods: Most submissions extend image detectors with temporal coherence, data association, or feature aggregation to address motion blur and defocus.Representative approaches include optical-flow propagation, box linkage, end-to-end detection and tracking, and multi-frame feature fusion.
  • Methods: Feature aggregation methods combine information across frames, but approaches using only a few consecutive frames are less effective for long-term object dynamics.Recurrent methods are introduced as a way to capture longer-range temporal context, although RNN vanishing gradients remain a concern.
  • Results: 21.57% AP was the best VisDrone-VDT2018 challenge result, achieved by CFE-SSDv2 through feature enhancement over SSD.EODST achieved the second-best result at 16.54% AP by associating detections with ECO tracking.
  • Results: 29.22% AP was achieved by DBAI-Det in VisDrone-VID2019, while all top five detectors surpassed the prior challenge leader.Most methods remained computationally expensive, running below 10 fps on a workstation with one GTX 1080Ti GPU.
  • Discussion: VID remains difficult because motion blur, pose variation, defocus, occlusion, viewpoint changes, and fast motion undermine temporal association.All submitted methods achieved below 15% mAP on the person and bicycle categories, leaving temporal-information use an open question.

6 SOT TRACK

The SOT track evaluates online localization of a target initialized in the first frame, using annotated sequences, attributes, and success and precision metrics. Results show deep trackers and large-scale training became increasingly important, while long-term tracking and robustness to abrupt motion, low resolution, and occlusion remain difficult.

  • Track definition: SOT algorithms locate a target’s bounding box in subsequent frames from its initial first-frame bounding box.
  • Annotations and evaluation: The dataset annotates target boxes in every frame and 12 attributes, including camera motion, occlusion, scale variation, and viewpoint change.
  • Annotations and evaluation: Success score, computed as the area under the success curve, is the primary metric, while precision measures location accuracy within a distance threshold.
  • Results and analysis: 76.5 success score and 92.3 precision score were achieved by LTNMI on the test-challenge 2018 set, the best reported VisDrone-SOT2020 result.LTNMI combines ATOM, SiamRPN++, Siam RCNN, and DiMP.
  • Results and analysis: 55.5 success score and 91.9 precision score from SMILEv2 in VisDrone-SOT2020 surpassed ED-ATOM’s 48.9 success score and 81.9 precision score in VisDrone-SOT2019.The paper attributes the challenge between these editions to 25 newly collected long-term tracking sequences.
  • Discussion: CNN-based trackers with online updates increasingly dominated, but long-term tracking and robustness to abrupt motion, low resolution, and occlusion remained challenging.The discussion also identifies large-scale training data as crucial for CNN-based trackers.

7 MOT TRACK

The MOT track evaluates methods that recover object trajectories from drone videos, using protocols that distinguish settings with and without input detections. Results indicate that detection quality, motion modeling, and appearance modeling are central factors in performance.

  • Evaluation Protocol: The benchmark evaluates MOT without input detections using mAP averaged across IoU thresholds, while input-detection settings use CLEAR-MOT metrics.The CLEAR-MOT set includes MOTA, MOTP, IDF1, FAF, MT, ML, FP, FN, IDS, and FM.
  • MOT Track: MOT recovers object trajectories by associating detections across sequential video frames.Tracking-by-detection methods commonly use appearance or motion information with optimization algorithms such as Hungarian or max-flow min-cut.
  • Results and Analysis: MOT performance highly relies on input object detection quality, motion models, and appearance models.The reported challenge results identify these three aspects as the main performance factors.
  • Results and Analysis: 25% higher AP was achieved by DBAI-Tracker in VisDrone-MOT2019 than by Ctrack in VisDrone-VDT2018, partly attributed to Cascade R-CNN detections.Cascade R-CNN provided more accurate input detections than Faster R-CNN and RetinaNet in the reported comparison.
  • Results and Analysis: Motion modeling improved tracking, with Ctrack reaching 16.12% AP by aggregating prediction events and stitching tracks using temporal constraints.Three of the top four VisDrone-MOT2019 trackers also integrated temporal modules such as optical flow.
  • Results and Analysis: VisDrone-MOT2020 AP scores further improved over VisDrone-MOT2019, while re-identification models contributed substantially to tracking performance.SOMOT benefited from combining global and local appearance, whereas Deepsort and YOLO-TRAC achieved 42.1% AP without re-identification models.
  • Discussion: The discussion identifies two-stage detection-and-association pipelines as suboptimal and motivates unified detection-and-tracking frameworks.The paper states that unified models can reduce computational complexity and allow the tasks to share information.

8 CONCLUSION AND FUTURE RESEARCH

VisDrone provides a large-scale foundation for evaluating drone-based detection and tracking while reviewing current methods and identifying unresolved deployment challenges. The paper highlights limits in evaluation, efficiency, appearance-only modeling, and motion exploitation.

  • Benchmark and research impact: VisDrone gathers and organizes a vast collection of annotated object instances to advance drone-based detection and tracking research.The benchmark is introduced as a large-scale resource built with over 6,000 worker hours.
  • Relations between detection and tracking: Detection and tracking are mutually beneficial: detections generate proposals for tracking, while tracking exploits temporal information to improve detection.This relationship motivates joint detection-and-tracking frameworks.
  • Performance evaluation: Evaluation protocols should weight detection and trajectory errors according to the application scenario.Surveillance emphasizes identity switches, trajectory fragmentations, and false positives, whereas autonomous driving places greater concern on false negatives.
  • Effectiveness and efficiency: Limited computation on drone edge devices makes efficiency as important as accuracy for real-world deployment.The 2018–2020 challenges emphasized accuracy, motivating future evaluation of running efficiency.
  • Unsolved problems: Current methods remain prone to failure under viewpoint change, motion blur, abrupt motion, and small-object conditions.Appearance information alone is insufficient for blurred or occluded objects and similarly appearing targets.
  • Unsolved problems: Motion-aware methods help in some scenarios, but reliable motion models still require substantial improvement for drone applications.The paper identifies fully exploiting motion information as a promising future direction.
Loading 2001.06303v3…