Source-linked AI summary
Okutama-Action: An Aerial View Video Dataset for Concurrent Human Action Detection
Mohammadamin Barekatain, Miquel Martí, Hsueh-Fu Shih, Samuel Murray, Kotaro Nakayama, Yutaka Matsuo, Helmut Prendinger
TL;DR
Existing human action-detection datasets lack aerial sequences and several real-world airborne challenges. Okutama-Action addresses this gap with a fully annotated UAV video dataset, which is more challenging than existing datasets due to abrupt camera motion, action transitions, concurrent actions, and multi-labeled actors.
Problem
Existing datasets lack representative aerial video and real-world airborne challenges such as dynamic action transitions, scale changes, and abrupt camera motion.
Method
The paper constructs Okutama-Action from diverse UAV sequences, annotating 12 outdoor action classes and concurrent, multi-labeled human actions.
Results
Okutama-Action contains 43 fully annotated sequences and is more challenging than existing datasets for concurrent human action detection.
Takeaways & Limitations
Okutama-Action provides a dataset for training and evaluating spatiotemporal detection of concurrent human actions from mobile aerial platforms.
Takeaways & Limitations
The action-detection experiment is limited to single-label annotations because SSD cannot handle multiple labels.
Abstract
from arXiv · showhide
Despite significant progress in the development of human action detection datasets and algorithms, no current dataset is representative of real-world aerial view scenarios. We present Okutama-Action, a new video dataset for aerial view concurrent human action detection. It consists of 43 minute-long fully-annotated sequences with 12 action classes. Okutama-Action features many challenges missing in current datasets, including dynamic transition of actions, significant changes in scale and aspect ratio, abrupt camera movement, as well as multi-labeled actors. As a result, our dataset is more challenging than existing ones, and will help push the field forward to enable real-world applications.
1. Introduction
Okutama-Action addresses the lack of aerial-view action datasets representative of UAV applications. It introduces a diverse, challenging dataset designed around real-world airborne conditions and concurrent human actions.
- UAV-based surveillance and search-and-rescue require recognizing human actions and tracking actors from aerial viewpoints.
- Existing action datasets are unsuitable for UAV use because their viewpoints, object scales, and outdoor-action coverage differ from aerial scenarios.
- Okutama-Action contains UAV-captured sequences from different altitudes and angles, with long videos, 12 outdoor action classes, and frame-level labels for all humans.
- The dataset includes actors with multiple simultaneous labels and visually similar actions, making it more challenging for action detection models.
- The paper trains and evaluates an action detection model on Okutama-Action and compares its performance with results on other datasets.
2. Related work
Reviews of existing datasets identify a shortage of mobile aerial video and several airborne challenges. Okutama-Action is designed to cover these gaps through concurrent, transitioning, multi-labeled actions and richer video properties.
- Current datasets lack aerial sequences from mobile platforms and omit challenges such as action transitions, scale changes, aspect-ratio changes, and abrupt camera motion.
- Existing datasets: UCF Sports and J-HMDB provide spatial annotations but commonly rely on low-resolution, trimmed videos containing a single action.
- Existing datasets: UCF-101 includes camera motion and viewpoint changes, but its actions are mainly indoor or sports activities, with same-action concurrency and short, low-resolution videos.
- Existing datasets: LIRIS-HARL and DALY contain concurrent action categories but lack dynamic action transitions and representative aerial actions.
- Existing datasets: UT-Interaction includes continuous concurrent actions and transitions, but has few videos, limited interaction types, a static camera, and no partial occlusion.
- Okutama-Action: Okutama-Action provides 43 fully annotated sequences with up to 9 transitioning actors, up to 10 actions or actors, 3840x2160 resolution, and approximately one-minute sequences.
3. Okutama-Action development
Okutama-Action was constructed from UAV footage to represent diverse outdoor actions, viewpoints, actors, and transitions. Its annotations and dataset statistics support training and evaluation of concurrent aerial action detection.
- Data collection: All videos were captured by DJI Phantom 4 UAVs at a baseball field in Okutama, Japan.
- Action selection: The dataset selects 12 everyday outdoor actions and groups them into human-to-human interaction, human-to-object interaction, and non-interaction.
- UAV configuration: UAV capture settings varied across 10–45 meters altitude and 45- or 90-degree camera angles to provide diverse viewpoints.
- Data collection: Twenty-two scenarios used up to 9 actors, varied execution styles and objects, crowded and empty frames, random actions, and dynamic action transitions.
- Data collection: Sequences were recorded with two UAVs in varied viewpoints and lighting conditions at 4K resolution and 30 FPS, with camera metadata included.
- Dataset annotations: Annotations were manually created at 10 FPS, interpolated to 30 FPS, and supplied in both multi-label and single-label forms because current detectors cannot handle multiple labels.
- Dataset summary and split: 43 sequences contain 77365 frames at 30 FPS in 4K resolution, split into 33 train-validation and 10 test sequences by scenario.
- Comparison with other datasets: Okutama-Action is presented as an aerial dataset representative of airborne scenarios and as the first spatiotemporal action-detection dataset with multi-labeled actors.
4. Experimental results
The experiments adapt SSD for pedestrian and action detection on Okutama-Action, evaluating detection performance under aerial-view conditions and model input choices.
- Pedestrian detection: SSD is a unified single-network object detector that predicts refined default-box coordinates and object-class likelihoods in one forward pass.It uses features from different network stages across scales and aspect ratios without a separate region-proposal step.
- Pedestrian detection: 72.3% mAP@0.5 was achieved for pedestrian detection on the Okutama-Action test set.With one class, mAP@0.5 is the Average Precision for the Pedestrian class.
- Pedestrian detection: Pedestrian detection performs poorly when pedestrians are too small, which occurs at UAV altitudes higher than approximately 30 meters.This identifies a practical altitude-related limitation of the detector on the aerial dataset.
- Action detection: The action detector uses a two-stream RGB and optical-flow model, but evaluation is limited to its first detection step because SSD cannot handle multiple labels.The full pipeline would merge appearance and motion scores before constructing temporally consistent action tubes.
- Action detection: Increasing input size improves action-detection performance for most classes, while 45-degree camera views perform better by giving actors more image pixels.The reported comparison covers 512x512 and 960x540 models; temporal actions such as Running are often confused with Walking under frame-level classification.
5. Conclusions
Okutama-Action is a high-resolution aerial dataset for concurrent human action detection, designed around challenges that make detection more difficult than existing datasets.
- Okutama-Action contains 43 minute-long sequences covering 12 action classes for concurrent human action detection.
- The dataset combines abrupt camera motion, dynamic action transitions, concurrent actions, and multi-labeled actors.
- An action detection model evaluated on Okutama-Action demonstrates the dataset’s difficulty.
- Future work includes deep models for multi-labeled outputs and evaluating multiple-object tracking on the dataset.