PaperlayerView exact revision on arXiv

Source-linked AI summary

AU-AIR: A Multi-modal Unmanned Aerial Vehicle Dataset for Low Altitude Traffic Surveillance

Ilker Bozcan, Erdal Kayacan

arXiv:2001.11737v2cs.CVcs.RO

TL;DR

Object-detection datasets for UAVs often omit flight-related sensor data, despite the distinct challenges posed by aerial imagery. AU-AIR addresses this gap by collecting annotated traffic-surveillance video with multimodal UAV data and evaluating mobile detectors. The dataset supplies a foundation for computer-vision and robotics research, while experiments show that natural-image-trained mobile networks struggle on aerial images.

  • Problem

    Existing UAV datasets commonly discard flight-related sensor data, and natural-image object detectors face difficulty because aerial images differ in layouts and object sizes.

  • Method

    The paper constructs AU-AIR from real-world outdoor traffic-surveillance videos with frame-level object annotations, flight data, and mobile object-detector baselines.

  • Results

    Mobile networks trained on natural images have trouble detecting objects in aerial images, while YOLOv3-Tiny achieves higher AP and mAP than MobileNetv2-SSDLite with similar TX2 inference times.

  • Takeaways & Limitations

    AU-AIR's visual, annotation, and flight data support research spanning computer vision, robotics, and data fusion.

Abstract

from arXiv · show

Unmanned aerial vehicles (UAVs) with mounted cameras have the advantage of capturing aerial (bird-view) images. The availability of aerial visual data and the recent advances in object detection algorithms led the computer vision community to focus on object detection tasks on aerial images. As a result of this, several aerial datasets have been introduced, including visual data with object annotations. UAVs are used solely as flying-cameras in these datasets, discarding different data types regarding the flight (e.g., time, location, internal sensors). In this work, we propose a multi-purpose aerial dataset (AU-AIR) that has multi-modal sensor data (i.e., visual, time, location, altitude, IMU, velocity) collected in real-world outdoor environments. The AU-AIR dataset includes meta-data for extracted frames (i.e., bounding box annotations for traffic-related object category) from recorded RGB videos. Moreover, we emphasize the differences between natural and aerial images in the context of object detection task. For this end, we train and test mobile object detectors (including YOLOv3-Tiny and MobileNetv2-SSDLite) on the AU-AIR dataset, which are applicable for real-time object detection using on-board computers with UAVs. Since our dataset has diversity in recorded data types, it contributes to filling the gap between computer vision and robotics. The dataset is available at https://bozcani.github.io/auairdataset.

I. INTRODUCTION

Existing aerial datasets largely treat UAVs as flying cameras, while natural-image object detectors struggle with aerial layouts and object sizes. AU-AIR addresses this gap with real-world multimodal UAV data, frame-level annotations, and mobile-detector baselines.

  • Natural-image datasets and aerial images differ in object layouts and sizes, causing object detectors trained on natural imagery to struggle with aerial imagery.
  • UAV datasets commonly capture visual data while discarding flight-related information such as time, location, and internal sensor measurements.
  • AU-AIR provides 8 traffic-surveillance video streams totaling over 2 hours, with aerial video, time, GPS coordinates, altitude, IMU data, and velocity.
  • Frames contain bounding-box annotations for traffic-related object categories and corresponding flight data across camera angles from 45 degrees to 90 degrees.
  • The dataset contains 32,823 labeled frames and 132,034 annotated instances across eight traffic-related object categories, with mobile detectors evaluated for real-time onboard use.

A. Related Work

Prior UAV datasets generally provide visual data and object annotations, whereas multimodal drone datasets often target navigation without object annotations. AU-AIR combines flight sensors with annotated visual data and mobile-detector baselines for object detection.

  • Existing UAV object-detection datasets commonly annotate people and vehicles but primarily include visual data and bounding boxes, with UAVDT offering only coarse flight-altitude labels.
  • Some multimodal drone datasets include visual, GPS, and IMU data but lack visual object annotations because they focus on UAV navigation.
  • AU-AIR is presented as the first multimodal UAV dataset for object detection, combining visual data, object annotations, and flight data.
  • The dataset contributes a real-time-oriented baseline by training and testing mobile object detectors while emphasizing differences between aerial and natural-image detection.

II. OBJECT DETECTION IN NATURAL IMAGES VS AERIAL IMAGES

Aerial images pose detection challenges distinct from natural images: objects are smaller, less conventionally occluded, shorter in appearance, and variable with altitude. Bird-view geometry also reduces visible height information.

  • Deep-learning object detectors are commonly trained on natural-image datasets captured with handheld cameras, whose side-view imagery differs substantially from aerial imagery.
  • Objects occupy fewer pixels in aerial images than in natural images, making them harder to detect and potentially requiring higher-resolution network inputs.
  • Natural images often contain foreground-object occlusion, whereas bird-view aerial images are less likely to show occlusion by other foreground instances.
  • Aerial perspective makes objects appear short and squat, diminishing information about object height; side-view bounding-box heights can provide clues instead.
  • Changes in UAV altitude alter object size and appearance, so recordings across altitudes vary the severity of aerial-detection challenges.

III. AU-AIR – THE MULTI-MODAL UAV DATASET

AU-AIR is a low-altitude, traffic-surveillance dataset combining video frames, object annotations, and corresponding UAV sensor data.

  • AU-AIR combines video clips, object bounding-box annotations, and sensor data for corresponding frames in a traffic-surveillance scenario captured below 30 meters.

A. UAV Platform

The AU-AIR UAV platform records aerial traffic-surveillance video alongside manually annotated object instances across varied altitudes, camera angles, and sampling conditions.

  • A. UAV Platform: The Parrot Bebop 2 records 1920 × 1080 video at 30 fps, while sensor data are logged every 20 milliseconds.The platform uses an on-board camera for video capture.
  • A. UAV Platform: Eight traffic-surveillance video clips total approximately two hours and contain 32,823 extracted frames recorded under varied lighting conditions.Recordings were made at roads in Aarhus, Denmark, including sunny, partly sunny, and cloudy conditions.
  • A. UAV Platform: Flight altitudes range from 10 to 30 meters, while camera angles vary from 45 to 90 degrees to represent aerial-surveillance challenges.A 90-degree angle is perpendicular to the Earth.
  • A. UAV Platform: Five frames are extracted per second from the 30-fps videos to reduce redundant frames while retaining 1920×1080 resolution.Both raw videos and extracted frames have the same resolution.
  • A. UAV Platform: Each frame is annotated with bounding boxes and category indices for eight traffic-related object types.The categories are person, car, bus, van, truck, bike, motorbike, and trailer.
  • A. UAV Platform: Three AMT workers independently annotate each frame, after which overlapping same-label boxes are combined using a 0.75 threshold or manually refined.Boxes and labels are manually fine-tuned when the merging condition is not satisfied.
  • A. UAV Platform: The dataset records frames mainly at 10, 20, and 30 meters with camera angles ranging from 45 to 90 degrees.These altitude and angle combinations expand the variety of captured aerial views.

D. Sensor Data

AU-AIR associates each extracted frame with temporal, geographic, flight-state, and motion attributes, with velocities and rotations expressed in the UAV body frame.

  • D. Sensor Data: Each extracted frame includes sensor data attributes recorded during video capture.The supplied passages identify date, time, GPS location, altitude, IMU rotations, and axis velocities.
  • D. Sensor Data: The dataset records the current timestamp, latitude, longitude, and altitude for each frame.Latitude and longitude come from GPS, while altitude comes from an altimeter.
  • D. Sensor Data: IMU-derived roll, pitch, and yaw angles describe rotation around the UAV’s x, y, and z axes.These correspond respectively to φ, θ, and ψ.
  • D. Sensor Data: Vx, Vy, and Vz represent speed along the UAV body-frame x, y, and z axes.The velocities and rotation angles are calculated according to the UAV body frame.

IV. EVALUATION AND ANALYSIS

The evaluation prioritizes real-time applicability on UAV onboard computers over state-of-the-art accuracy, using mobile detectors selected for their accuracy–inference-time trade-off.

  • IV. EVALUATION AND ANALYSIS: The study trains and evaluates YOLOv3-Tiny and MobileNetv2-SSDLite with emphasis on real-time performance rather than state-of-the-art accuracy.The selected detectors are intended for applicability on onboard computers.

A. Baseline networks

The baseline networks use original-paper default settings, COCO-trained backbones, and output layers adapted to AU-AIR’s eight object classes.

  • A. Baseline networks: YOLOv3-Tiny and MobileNetv2-SSDLite are benchmarked with default parameters and COCO-trained backbones.Examples of default parameters include learning rate and input size.
  • A. Baseline networks: The detectors are adapted to AU-AIR by changing their last layers for eight object classes.The dataset split uses 60% training, 10% validation, and 30% testing samples.

B. Comparison Metrics

The study evaluates mobile object detectors on aerial imagery using mAP and qualitative comparisons, showing that AU-AIR training improves detection over COCO-only training and that YOLOv3-Tiny offers higher accuracy with similar inference speed.

  • Comparison Metrics: mAP measures detection performance by averaging category-level AP over 11 recall values at an IoU threshold of 0.5.The evaluation uses discretized recall values from 0 to 1.
  • Comparison Metrics: COCO-only networks have poor results, whereas AU-AIR training with COCO pre-training performs better on the same aerial inputs.The qualitative comparison also includes complete bird-view images and perspective-affected objects.
  • Comparison Metrics: Motorbike and bicycle categories have significantly lower AP values than other categories, potentially reflecting class imbalance and small object sizes.Bus has the highest AP, while human AP is relatively higher despite similarly small instances; van and trailer are absent from COCO.
  • Comparison Metrics: AU-AIR-trained baselines detect objects across aerial images captured at different altitudes and view angles.Qualitative examples are provided in Fig. 8.
  • Comparison Metrics: YOLOv3-Tiny has higher AP and mAP than MobileNetv2-SSDLite, while inference times are similar at 17.5 FPS and 17 FPS on TX2, respectively.The comparison concerns the two mobile detector baselines.

V. DISCUSSION

The discussion identifies dataset imbalance, annotation quality, platform diversity, and environmental scope as limitations, while highlighting sensor-linked frames as a basis for future multimodal research.

  • Limitations and Future Work: AU-AIR has imbalanced object-category counts, motivating additional data collection and synthetic augmentation for underrepresented categories.Proposed transformations include brightness changes, translation, and rotation.
  • Limitations and Future Work: Manual AMT annotation may leave unlabelled instances or loose bounding boxes despite multiple annotators and manual checks.The authors consider a three-step workflow involving drawing, verification, and completeness checking.
  • Limitations and Future Work: Each AU-AIR frame includes corresponding sensor data, but the current work provides baselines only for object annotations and visual data.Future directions include multimodal object detection, UAV navigation and control, and additional visual sensors.
  • Limitations and Future Work: The dataset was collected using one ready-to-fly Parrot Bebop 2 quadrotor, with broader platform and camera diversity left for future collection.Future samples may use different UAV types, camera resolutions, and frame rates.
  • Limitations and Future Work: Traffic surveillance is the primary environmental context, and future work aims to increase the number of contexts for greater dataset diversity.The stated scope boundary is the dominance of traffic-surveillance environments.

VI. CONCLUSIONS

The paper presents AU-AIR as a multimodal UAV dataset combining visual, object-annotation, and flight data to connect computer vision with robotics. It also establishes mobile-detector baselines and finds that models trained on natural images struggle with aerial imagery.

  • AU-AIR combines visual data, object annotations, and flight data in a multimodal UAV dataset collected outdoors.
  • The dataset is intended to fill the gap between computer vision and robotics and support research focused on data fusion.
  • Mobile networks trained on natural images have trouble detecting objects in aerial images.
Loading 2001.11737v2…