PaperlayerView exact revision on arXiv

Source-linked AI summary

UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles

Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, Zhiheng Li

arXiv:2104.00946v4cs.CV

TL;DR

UAV human-behavior research lacks benchmarks with sufficient scale, modality, task coverage, and environmental diversity. This paper introduces UAV-Human and a fisheye action-recognition method guided by flat RGB videos; the authors report efficacy on the dataset.

  • Problem

    Existing UAV behavior benchmarks have limited data, modalities, task categories, subject diversity, and environmental diversity, despite the need for comprehensive evaluation resources.

  • Method

    The paper constructs a multi-modal UAV benchmark and proposes Guided Transformer I3D, which learns unbounded fisheye transformations guided by flat RGB images.

  • Results

    The benchmark spans four behavior-understanding tasks, and experiments show the proposed fisheye action-recognition method is efficacious on UAV-Human.

  • Takeaways & Limitations

    UAV-Human provides a large, challenging, and comprehensive resource for UAV-based human action, pose, and behavior understanding.

  • Takeaways & Limitations

    The dataset is released for non-commercial research and academic use with consent from captured subjects.

Abstract

from arXiv · show

Human behavior understanding with unmanned aerial vehicles (UAVs) is of great significance for a wide range of applications, which simultaneously brings an urgent demand of large, challenging, and comprehensive benchmarks for the development and evaluation of UAV-based models. However, existing benchmarks have limitations in terms of the amount of captured data, types of data modalities, categories of provided tasks, and diversities of subjects and environments. Here we propose a new benchmark - UAVHuman - for human behavior understanding with UAVs, which contains 67,428 multi-modal video sequences and 119 subjects for action recognition, 22,476 frames for pose estimation, 41,290 frames and 1,144 identities for person re-identification, and 22,263 frames for attribute recognition. Our dataset was collected by a flying UAV in multiple urban and rural districts in both daytime and nighttime over three months, hence covering extensive diversities w.r.t subjects, backgrounds, illuminations, weathers, occlusions, camera motions, and UAV flying attitudes. Such a comprehensive and challenging benchmark shall be able to promote the research of UAV-based human behavior understanding, including action recognition, pose estimation, re-identification, and attribute recognition. Furthermore, we propose a fisheye-based action recognition method that mitigates the distortions in fisheye videos via learning unbounded transformations guided by flat RGB videos. Experiments show the efficacy of our method on the UAV-Human dataset. The project page: https://github.com/SUTDCV/UAV-Human

1. Introduction

UAV-based human behavior understanding faces distinctive viewpoint, motion, resolution, and dataset-coverage challenges. UAV-Human addresses these gaps with a large multi-modal benchmark and a fisheye action-recognition method guided by flat RGB videos.

  • Motivation: UAV footage varies in viewpoints, motion blur, subject resolution, and camera attitudes, requiring methods tailored to UAV scenarios.These variations arise from UAV motion and continuously changing flight heights and attitudes.
  • Motivation: Existing UAV datasets are limited in sample scale, subject diversity, real-world conditions, UAV viewpoints, modalities, and task coverage.Most existing datasets provide annotations for only one or two tasks.
  • UAV-Human: UAV-Human combines action recognition, pose estimation, person re-identification, and attribute recognition in a multi-modal benchmark collected across varied environments and flight conditions.The collection spans daytime and nighttime, rural districts and cities, multiple months, subjects, sensors, attitudes, and speeds.
  • UAV-Human: UAV-Human contains 22,476×3 action-recognition video sequences from 119 subjects and 155 activities, plus annotations for pose, re-identification, and attributes.The benchmark includes 22,476 pose frames, 41,290 re-identification frames covering 1,144 identities, and 22,263 attribute frames.
  • Method: The proposed Guided Transformer I3D learns unbounded transformations for fisheye videos using flat RGB guidance to mitigate distortion.The method targets action recognition with wide-angle fisheye cameras used on UAVs.

2. Related Work

Prior UAV behavior datasets are narrow in scale, scenes, tasks, and modalities, while UAV-Human is designed as a broader benchmark. Its UAV collection also introduces viewpoint and motion variation absent from many ground-camera datasets.

  • UAV-based datasets: Prior UAV behavior benchmarks are limited in dataset size, scene diversity, task categories, and captured data modalities.These limitations motivate a comprehensive benchmark for multi-modal behavior analysis.
  • UAV-based datasets: Okutama-Action contains 12 action classes performed by 9 subjects in a baseball-field setting, limiting scenario diversity.It is described as relatively small and collected with RGB videos.
  • UAV-based datasets: UAV-Gesture provides 119 RGB samples for 13 UAV-control gestures performed by 10 subjects against relatively monotonous backgrounds.The dataset targets UAV control gesture and pose analysis.
  • UAV-based datasets: PRAI-1581 provides only a single RGB modality and annotations for one person re-identification task.It is a UAV-based person-ReID dataset.
  • UAV-based datasets: AVI contains 2K RGB images covering five action classes and 25 subjects, with annotations for violent-action recognition and pose estimation.The subjects have a narrow age range of 18 to 25.
  • Comparison with ground-camera datasets: Unlike ground-camera multi-modal datasets, UAV-Human uses varied speeds, heights, attitudes, and trajectories that produce unique viewpoints, resolution changes, camera movements, and motion blur.Many ground-camera datasets were collected in relatively simple, static, and monotonous scenes.

3. UAV-Human Dataset

UAV-Human is a large-scale, multi-modal benchmark spanning four human behavior understanding tasks, with diverse UAV viewpoints, environments, modalities, and challenging capture conditions.

  • Dataset scope: UAV-Human contains 67,428 annotated action-recognition video sequences, 22,476 pose-estimation frames, 41,290 person re-identification frames, and 22,263 attribute-recognition frames.
  • Data modalities: Six data modalities include fisheye, night-vision, RGB, infrared, depth, and skeleton data captured using multiple UAV-mounted sensors.
  • Capture diversity: Samples span 45 rural and urban sites, two seasons, daytime and nighttime, and adverse rainy and windy conditions.
  • Capture diversity: Varying UAV attitudes, positions, heights from 2 to 8 meters, and flexible views produce camera shakes, motion blur, resolution changes, and viewpoint variation.
  • Dataset scope: The benchmark provides annotations for action recognition, pose estimation, person re-identification, and human attribute recognition.
  • Task annotations: Action recognition uses 155 balanced activity classes from 119 subjects, while pose estimation labels 17 body joints across 22,476 images.
  • Task annotations: Person re-identification includes 41,290 images from 1,144 identities with varied viewpoints, poses, and illumination.

4. Guided Transformer Network for Fisheye Video Action Recognition

GT-I3D addresses fisheye distortion in UAV action-recognition videos by learning unbounded feature transformations guided by a fixed RGB stream. The model trains with paired fisheye and flat RGB videos but uses only the fisheye stream during testing.

  • Fisheye cameras provide wide views for UAVs, but their distortions make action recognition challenging.
  • GT-I3D combines a fisheye stream with a pre-trained RGB stream built on an I3D baseline.The RGB stream guides training and is fixed during GT-I3D optimization.
  • Guided Transformer I3D: GT-Modules warp distorted fisheye feature maps using 3D localization networks, unbounded transformation parameters, grid generation, and sampling.The modules are inserted before I3D maxpooling layers and preserve feature-map shape across depth, channels, height, and width.
  • Training: RGB guidance constrains transformed fisheye features with a Kullback–Leibler divergence loss, while classification loss updates the fisheye stream.The objective aggregates guidance losses across inserted GT-Modules and compares predicted action scores with class labels.
  • Training: Training uses paired RGB and fisheye videos from the same action instance, with uniformly sampled frames supplied to the two streams.The implementation uses 64-frame inputs, SGD, batch size 16, a 4e-3 initial learning rate, 30K iterations, and two Nvidia RTX2080Ti GPUs.
  • Testing: During testing, the fisheye stream alone performs action recognition, so flat RGB videos are not required at deployment.This enables use when the UAV is equipped only with a fisheye camera.

5. Experiments

Experiments evaluate UAV-Human across modalities and tasks, revealing challenges from UAV viewpoint variation, motion blur, scale changes, occlusions, and overhead capture. The proposed fisheye method improves fisheye action recognition, while skeleton methods outperform video-based methods and pose estimation remains relatively weak.

  • Action Recognition: 28.72% and 26.56% are the highest CSv1 accuracies for night-vision and IR inputs, respectively.Night-vision and IR can provide clearer information in the dataset’s dark environments.
  • Action Recognition: 23.24% is achieved by the full fisheye model on CSv1, competitive with the RGB model’s 23.86%.The method rectifies distorted fisheye videos and improves the performance of original fisheye methods.
  • Action Recognition: 21.68% and 21.49% are obtained by guidance-loss-only and transformer-only variants on CSv1, while the full model achieves the highest accuracy among fisheye methods.The guidance loss constrains feature maps from fisheye and RGB streams using KL divergence.
  • Action Recognition: Skeleton-based methods outperform video-based methods because skeletal representations are more robust to changing backgrounds, scales, locations, and viewpoints.Skeleton normalization through scaling and translation can further support robustness in the UAV setting.
  • Pose Estimation: 56.9% and 56.5% are reported by the two evaluated pose-estimation methods, indicating relatively weak performance on UAV-Human.Multiple UAV attitudes, varying scales and views, diverse postures, and complex occlusions increase the pose-estimation challenge.
  • Attribute Recognition: Clothing colors and styles achieve the lowest attribute-recognition accuracies, reflecting varied dressing types and viewpoint changes.The dataset was collected over a relatively long period with multiple UAV attitudes.

6. Conclusion

UAV-Human is presented as a large, challenging, comprehensive benchmark for UAV-based human behavior understanding. The paper also proposes GT-I3D for distorted fisheye action recognition, with experiments showing its efficacy.

  • Conclusion: UAV-Human is described as the largest, most challenging, and most comprehensive UAV-based dataset for human action, pose, and behavior understanding.The benchmark is intended to encourage exploration and deployment of data-intensive learning models.
  • Conclusion: GT-I3D is proposed for action recognition in distorted fisheye videos, and experimental results show the method’s efficacy.The method addresses fisheye distortion through the proposed action-recognition approach.
Loading 2104.00946v4…