PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for UAV-based Object Detection and Tracking: A Survey

Xin Wu, Wei Li, Danfeng Hong, Ran Tao, Qian Du

arXiv:2110.12638v1cs.CV

TL;DR

UAV object detection and tracking require methods suited to flexible but challenging remote-sensing data. This survey synthesizes deep learning approaches across static image detection, video object detection, and multi-object tracking, and reviews datasets and evaluations. It concludes that static detection is dominated by adapted classical architectures, whereas UAV-specific video methods remain limited.

  • Problem

    UAV data introduce image degradation, small and diverse object scales, complex backgrounds, direction variation, and real-time constraints that challenge detection and tracking.

  • Method

    The survey reviews DL-based UAV object detection and tracking for small UAVs across SOD, VID, and MOT, covering methods, datasets, evaluations, and future directions.

  • Results

    Most UAV static detection methods use YOLO, Faster R-CNN, or SSD, while relatively few methods are designed specifically for UAV VID and MOT.

  • Takeaways & Limitations

    Combining detection with tracking can stabilize detections, provide object labels and identities, and reduce false and missed detections through trajectory analysis.

Abstract

from arXiv · show

Owing to effective and flexible data acquisition, unmanned aerial vehicle (UAV) has recently become a hotspot across the fields of computer vision (CV) and remote sensing (RS). Inspired by recent success of deep learning (DL), many advanced object detection and tracking approaches have been widely applied to various UAV-related tasks, such as environmental monitoring, precision agriculture, traffic management. This paper provides a comprehensive survey on the research progress and prospects of DL-based UAV object detection and tracking methods. More specifically, we first outline the challenges, statistics of existing methods, and provide solutions from the perspectives of DL-based models in three research topics: object detection from the image, object detection from the video, and object tracking from the video. Open datasets related to UAV-dominated object detection and tracking are exhausted, and four benchmark datasets are employed for performance evaluation using some state-of-the-art methods. Finally, prospects and considerations for the future work are discussed and summarized. It is expected that this survey can facilitate those researchers who come from remote sensing field with an overview of DL-based UAV object detection and tracking methods, along with some thoughts on their further developments.

I. INTRODUCTION

UAV remote sensing has expanded object detection and tracking by providing flexible, low-altitude, high-resolution data acquisition. This survey reviews deep learning methods for small UAVs, their applications, and related research developments.

  • Satellites and manned aircraft face cost and pilot-safety constraints that limit their application scope.
  • UAV remote sensing combines UAVs, remote sensing, GPS positioning, and IMU attitude determination to obtain low-altitude, high-resolution imagery.
  • UAVs provide flexible video collection with reduced geographic, weather, and time limitations, supporting land-cover, agriculture, traffic, city, and disaster monitoring.
  • Since 2015, deep neural networks have become mainstream for UAV object detection and tracking, replacing traditional feature-based approaches for large data volumes.
  • The survey focuses on UAVs weighing fewer than 30 kilograms and reviews recent DL-based detection and tracking methods, challenges, and future directions.
  • Small light UAVs are the predominant study and application platform, with battery power, multirotor designs, sub-30-kilogram weight, and usually less than one-hour flight time.

B. Challenges

UAV detection and tracking must handle degraded imagery, uneven object distributions, small targets, and strict real-time requirements. The survey organizes these challenges and distinguishes static, video, and multi-object tracking tasks.

  • UAV video tasks face image degradation, uneven object intensity, small object size, perspective specificity, complex backgrounds, scale diversity, direction diversity, and real-time constraints.
  • Environmental changes, high-speed flight, and camera rotation produce fuzzy or noisy images, motivating noise reduction and camera-distortion correction.
  • Wide-angle imaging and flexible camera movement create dense overlapping regions prone to repeated detections and sparse regions prone to missed detections.
  • Most UAV ground objects occupy fewer than 32 × 32 pixels, making their features less distinct and increasing false and missed detections.
  • Real-time processing must quickly and accurately locate moving ground objects in drone video.
  • SOD detects objects in images, VID uses adjacent-frame context to modify current detections, and MOT predicts trajectories to estimate moving states.

III. OBJECT DETECTION FROM UAV-BORNE IMAGES

UAV image detection methods address acquisition variability, scale diversity, and direction diversity through preprocessing, augmentation, multi-scale representations, and specialized convolutions. The surveyed development includes successive static-detection architectures and rotation-aware strategies.

  • Postprocessing commonly increases training samples and diversity in object size, direction, and illumination, but its effectiveness varies across studies.
  • UAV imaging produces overlapping, covered, displaced, and variably sized ground objects because of flight-altitude and payload limitations.
  • The static-detection literature developed from data-augmentation and rotated-region methods toward architectures including shallow CNNs, ORNs, PTN, FR-O, ETNs, RFN, and direction-aware networks.
  • Scale diversity is addressed mainly with multi-scale feature maps or dilated and deformable convolution kernels.
  • Dilated convolution expands the receptive field without losing resolution or coverage.

C. Object Detection on Small Objects

Small and densely arranged UAV objects provide limited feature information, so detection methods emphasize scale-aware architectures, enhanced representations, contextual pooling, and object-relation modeling. Direction-sensitive approaches further address rotated targets, while real-time deployment constrains model design.

  • C. Object Detection on Small Objects: UAV altitude creates scale diversity, small object sizes, and dense arrangements that reduce extractable feature information.
  • C. Object Detection on Small Objects: Scale-aware small-object methods include RRNet, HRDNet, Cascade networks, UAV-YOLO, MPFPN, DAGN, GANet, FS-SSD, and ResNeXt-d.
  • C. Object Detection on Small Objects: Perceptual GANs generate super-resolved small-object representations by exploiting structural correlations between large and small objects.
  • C. Object Detection on Small Objects: Context-aware ROI pooling addresses distortion of small objects after pooling, while CapsNets model relationships among overlapping objects.
  • D. Object Detection on Direction Diversity: Rotation-invariant detection uses angle-based augmentation, oriented proposals, directional filters, or multi-angle RPN anchors, with augmentation increasing training cost and overfitting risk.
  • E. Object Detection on Detection Speed: Real-time UAV detection favors ARM, mobile, and embedded platforms or trimmed architectures such as YOLO and tiny-YOLO.

F. Object Detection on Others

Video object detection (VID) addresses temporal challenges in UAV footage by combining frame-level detection with temporal and contextual information. Deep learning methods have evolved from multi-stage pipelines toward integrated architectures and temporal-context networks.

  • VID faces UAV-specific challenges including camera change and motion blur.
  • VID first detects objects independently in each frame, then compensates missed detections using temporal and contextual information.
  • Early VID methods used multi-stage pipelines that separately modified frame detections with temporal context.
  • Later methods integrated feature enhancement, decision fusion, tracking, and advanced detectors to improve accuracy, robustness, or real-time performance.
  • Optical-flow-based VID estimates motion between consecutive frames to incorporate temporal information.

RNN, LSTM, etc

The survey describes temporal modeling for UAV video detection through optical flow, recurrent memory, and tracking-based approaches. These methods use motion, long-term sequence information, or object association to improve video detection.

  • Optical-flow methods obtain motion information from neighboring frames and can support lightweight, end-to-end video detection.One mobile optical-flow network achieved MAP 60.2 and 25 frames per second on a Huawei Mate 8 cellphone, but multiple-frame flow computation prevented back-propagation revision during training.
  • Memory-network approaches use RNNs, LSTMs, or GRUs to learn temporal information from video sequences.
  • Association LSTM and ConvLSTM methods model object associations or fuse multi-frame features while targeting accuracy and timeliness.
  • Spatio-temporal memory networks model and align long-term object appearance and motion dynamics end to end.
  • Tracking-based networks use tracking or joint detection-and-tracking frameworks to exploit the similarity between VID and object tracking.

A. Tracking-by-Detection

Tracking-by-Detection (TBD) remains the mainstream MOT strategy: detect objects in each frame and associate them with prior detections. Its effectiveness depends strongly on detection quality and is challenged by UAV motion and rapid object movement.

  • A. Tracking-by-Detection: TBD detects all objects in the current frame and associates them with the previous frame for tracking.
  • A. Tracking-by-Detection: TBD can track newly appearing objects throughout a video, but detection accuracy decisively affects tracking results.
  • A. Tracking-by-Detection: Common TBD methods include CMOT, MDP, SORT, DSORT, GOG, CEM, SMOT, and IOUT.
  • A. Tracking-by-Detection: In common TBD systems, deep learning performs object detection while traditional data-association methods link detections across frames.
  • A. Tracking-by-Detection: UAV flight motion, jitter, and ambient light produce noise, false alarms, and missed detections, while rapid movement can defeat frame-to-frame matching.

B. Single Object Tracking Assisted Multiple Object Tracking

SOT-assisted MOT methods predict object trajectories to address weaknesses in frame-by-frame association, while memory networks use historical trajectories for matching. The survey also reviews UAV datasets spanning detection, tracking, vehicles, people, and actions.

  • B. Single Object Tracking Assisted Multiple Object Tracking: SOT-assisted MOT uses single-object trackers to predict trajectories and address Tracking-by-Detection failures.
  • B. Single Object Tracking Assisted Multiple Object Tracking: Combining SOT with Kalman filtering, IOUT, or detector candidates can improve association and detection results, but occlusion may cause tracking drift.
  • B. Single Object Tracking Assisted Multiple Object Tracking: Offline-trained Siamese-RPN trackers can exceed 80 frames per second, whereas online SOT updates consume substantial CPU resources.
  • B. Single Object Tracking Assisted Multiple Object Tracking: Memory-network MOT methods use historical trajectories and appearance, action, or interaction cues to learn matching similarities.
  • The survey reviews open UAV datasets covering video detection, single-object tracking, multiple-object tracking, vehicle counting, and human action detection.

VII. EXPERIMENT RESULTS AND ANALYSIS

The survey evaluates representative detection and tracking methods using four benchmark UAV datasets: VisDrone, UAVDT, Okutama-Action, and Stanford UAV.

  • Four benchmark datasets—VisDrone, UAVDT, Okutama-Action, and Stanford UAV—are used to illustrate representative detection and tracking performance.Annotated image examples from these datasets are shown in Fig. 9.

A. Evaluation of Object Detection from UAV-borne Images

UAV-image detection is evaluated mainly with overlap-based AP and AR metrics across benchmark datasets, where leading methods differ by dataset and scene complexity.

  • Evaluation metrics: AP and AR evaluate UAV-image detection through overlap between predicted and ground-truth bounding boxes at specified IoU thresholds.Common measures include AP IoU=0.50:0.05:0.95, AP IoU=0.50, AP IoU=0.75, and AR values with object-count limits.
  • Evaluation protocol: The compared detection studies use AP = AP IoU=0.50:0.05:0.95 as the principal evaluation criterion across different UAV datasets.The survey compares methods according to AP within each specific dataset.
  • VisDrone dataset: 52.77% AP on the VisDrone-DET validation set is achieved by NDFT, the best result among the compared methods.NDFT transfers domain-robust features learned through UAVDT to VisDrone.
  • UAVDT dataset: UAVDT detection is complicated by varied scenes, overlapping small objects, and weather conditions.These factors increase difficulty for single, multiple, and overlapping small-object detection.
  • UAVDT dataset: 56.92% AP on the UAVDT testing set is achieved by D2Det, the best result among the compared methods.D2Det uses dense local regression; NDFT and ResNet101-FPN follow with 52.03% and 49.05% AP scores.

B. Evaluation of Object Detection from UAV-borne Video

Video detection and tracking are evaluated with detection and identity-based metrics across UAV benchmarks, with different methods leading on different datasets and evaluation settings.

  • Video detection metrics: Video object detection uses the same AP and AR indicators as UAV-image detection, including multiple IoU thresholds and maximum-recall limits.The survey reports results mainly on VisDrone and also considers Okutama-Action and UAVDT.
  • VisDrone-VID: 27.27% AP on the VisDrone-VID validation set is achieved by TDFA, reported as the best among the compared video-detection methods.TDFA uses a two-stream refined FlowNet pipeline and is described as robust to small-scale objects.
  • Okutama-Action: 87.18% AP 50 on the Okutama-Action test dataset is achieved by TDFA, while STDnet methods obtain 34.60% AP for objects under 16 × 16 pixels.These results are compared with five baseline works.
  • Object tracking metrics: Tracking evaluation includes MOTA, MOTP, IDF1, FAF, MT, ML, FP, FN, IDS, and FM, with some studies also reporting AP.IDF1 uses true-positive, false-positive, and false-negative identity counts.
  • Tracking results: TNT leads the VisDrone-MOT challenge on several tracking metrics, while PAS reaches 50.80% AP on the VisDrone-MOT testing set.IPGAT performs best on UAVDT and Stanford Drone testing datasets in terms of IDF1, MT, ML, and FN.
  • Tracking results: Baseline tracking results remain far from practical-application requirements, although they provide directions and a theoretical basis for future research.For UAVDT, SORT is superior on most metrics under the reported evaluation setting.

D. Estimation of Computation Cost

The survey estimates computation cost for UAV detection and tracking methods using reported speed, GPU, and backbone information, while comparing UAV-specific and classic networks.

  • Cost indicators: Computation cost is assessed through speed, GPU, and backbone-model indexes reported for reviewed UAV methods.The survey relates computation cost to these factors and lists them in Tables XIII–XV.
  • Cost estimation: The estimated computation cost depends on NVIDIA GPU computing power and the backbone network used by each method.The survey uses GPU capability as the basis for estimating cost from the corresponding backbone.
  • Comparison scope: The reported comparison tables cover UAV-exclusive and classic detection and tracking networks across image detection, video detection, and tracking settings.The cited table captions identify comparisons for UAVDT, VisDrone-VID, VisDrone-MOT, Stanford Drone, and related testing sets.

VIII. DISCUSSION AND CONCLUSION

The survey organizes UAV deep-learning research across SOD, VID, and MOT, while highlighting dataset, computational, and application-level constraints. It finds that video detection and tracking remain comparatively underdeveloped, with limited UAV-specific methods and imperfect performance.

  • The survey analyzes UAV deep-learning methods across three topics: SOD, VID, and MOT.
  • UAV data: Public UAV datasets are mainly visible-data collections, with VisDrone reaching 3840 × 2160 resolution and Vehicle Dataset providing visible–thermal infrared data.Vehicle Dataset also includes oriented bounding-box annotations rather than only horizontal boxes.
  • DL Method: Most UAV static detectors use YOLO, Faster R-CNN, or SSD, with YOLO- and SSD-based methods advantageous in speed.Examples include UAV-YOLO, ComNet, SlimYOLOv3, DAGN, Dshnet, NDFT, D2det, and FS SSD.
  • DL Method: 65.2% is the highest reported AP for VisDrone-VID, while 50.80% is the highest reported performance for VisDrone-MOT.The survey attributes remaining difficulty to interactions among ground objects and complex tracking scenes.
  • Computer platforms: Reviewed UAV systems mainly run on NVIDIA GPUs, with processing speeds ranging from roughly 0.2fps to 50fps across image sizes.Raspberry Pi-based systems achieve 8.5fps, 4.5fps, or 15fps in cited detection and tracking systems but lack generality.
  • Combining detection with tracking can stabilize detections, provide object identities and fine labels, and reduce false and missed detections through trajectory analysis.
Loading 2110.12638v1…