PaperlayerView exact revision on arXiv

Source-linked AI summary

MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions

Yixuan Li, Lei Chen, Runyu He, Zhenzhi Wang, Gangshan Wu, Limin Wang

arXiv:2105.07404v2cs.CV

TL;DR

Spatio-temporal action detection lacks realistic benchmarks for fine-grained actions performed concurrently by multiple people in untrimmed video. The paper introduces MultiSports, a high-quality sports dataset built around dense annotations and well-defined action boundaries, and shows that it is substantially more challenging than earlier benchmarks while exposing key detection challenges.

  • Problem

    Existing action-detection benchmarks provide limited coverage of multi-person, fine-grained, spatially and temporally localized actions in realistic video.

  • Method

    The paper constructs MultiSports using three design criteria—multi-person motion-dependent scenes, well-defined boundaries, and fine-grained complex actions—and annotates four sports with dense spatial and temporal labels.

  • Results

    MultiSports is substantially more challenging than J-HMDB and UCF101-24, with lower frame mAP and video mAP, while error analysis identifies fine-grained recognition, temporal localization, occlusion, and long-range context as key challenges.

  • Takeaways & Limitations

    MultiSports provides a realistic benchmark for studying spatio-temporal action detection in complex multi-person sports scenes.

Abstract

from arXiv · show

Spatio-temporal action detection is an important and challenging problem in video understanding. The existing action detection benchmarks are limited in aspects of small numbers of instances in a trimmed video or low-level atomic actions. This paper aims to present a new multi-person dataset of spatio-temporal localized sports actions, coined as MultiSports. We first analyze the important ingredients of constructing a realistic and challenging dataset for spatio-temporal action detection by proposing three criteria: (1) multi-person scenes and motion dependent identification, (2) with well-defined boundaries, (3) relatively fine-grained classes of high complexity. Based on these guide-lines, we build the dataset of MultiSports v1.0 by selecting 4 sports classes, collecting 3200 video clips, and annotating 37701 action instances with 902k bounding boxes. Our datasets are characterized with important properties of high diversity, dense annotation, and high quality. Our Multi-Sports, with its realistic setting and detailed annotations, exposes the intrinsic challenges of spatio-temporal action detection. To benchmark this, we adapt several baseline methods to our dataset and give an in-depth analysis on the action detection results in our dataset. We hope our MultiSports can serve as a standard benchmark for spatio-temporal action detection in the future. Our dataset website is at https://deeperaction.github.io/multisports/.

1. Introduction

MultiSports addresses the difficulty of detecting fine-grained actions performed concurrently by multiple people in untrimmed video. It introduces a densely annotated, high-quality benchmark designed around realistic action boundaries, motion dependence, and complex sports actions.

  • 1. Introduction: MultiSports targets spatio-temporal detection of concurrent human actions that existing temporal detectors cannot spatially distinguish in multi-person scenes.The benchmark is motivated by real-world video analysis settings where multiple people perform different actions simultaneously.
  • 1. Introduction: The benchmark is designed around multi-person motion-dependent recognition, semantically and temporally well-defined boundaries, and fine-grained actions requiring pose and motion information.These criteria aim to represent realistic challenges rather than single-person, coarse, or temporally repetitive actions.
  • 1. Introduction: The dataset contains 66 action classes from basketball, volleyball, football, and aerobic gymnastics, whose boundaries and categories are supported by sports expertise or official documentation.Recognizing some actions requires long-term context; for example, football kicks may need up to 5 seconds of context to distinguish pass types.
  • 1. Introduction: MultiSports uses exhaustive 25 fps frame-wise bounding boxes and action labels, with professionals annotating temporal and category labels and crowd workers completing boxes using tracking assistance.Quality control and high-resolution competition videos support consistent annotations across countries and performance levels.
  • 1. Introduction: Benchmarking shows MultiSports is substantially more challenging than J-HMDB and UCF101-24, with lower frame mAP and video mAP, and exposes fine-grained recognition, temporal localization, occlusion, and long-range-context challenges.The authors combine baseline evaluation with detailed error analysis to characterize these difficulties.
  • 1. Introduction: The paper contributes a realistic multi-person benchmark and systematic studies intended to reveal key spatio-temporal action-detection challenges.The dataset is presented as a potential standard benchmark for future research.

2. Related Work

Related datasets differ in annotation density, spatial and temporal localization, action granularity, and the number of concurrent actors. MultiSports is positioned as a sports-focused benchmark combining fine-grained actions, multiple people, and frame-by-frame spatio-temporal annotations.

  • 2. Related Work: Traditional action-recognition datasets use trimmed clips centered on single actions, while newer fine-grained datasets still generally contain only one concurrent action.These settings differ from MultiSports’ focus on multiple concurrent sports actions.
  • 2. Related Work: Temporal action-detection datasets annotate when actions occur but lack spatial annotations needed to identify concurrent actions performed by multiple people.This distinguishes temporal localization benchmarks from spatio-temporal detection datasets.
  • 2. Related Work: Earlier spatio-temporal benchmarks typically use short videos with one person and coarse-grained categories, whereas MultiSports uses multiple concurrent actions, fine-grained categories, fast movement, deformation, and dense instances.The comparison identifies the setting that MultiSports is designed to evaluate.
  • 2. Related Work: Frame-level and clip-level detectors have been developed for earlier spatio-temporal benchmarks, with frame detections commonly linked into final action tubes.This work adapts such detector families to the MultiSports setting.

3. The MultiSports Dataset

MultiSports is a challenging benchmark for spatio-temporal action detection, designed around multi-person scenes, fine-grained actions, well-defined boundaries, and dense spatial-temporal annotations. It contains diverse sports data with quality-controlled labeling and challenging distributions of action frequency and duration.

  • Dataset Design: MultiSports provides a benchmark centered on multi-person scenes, fine-grained categories, well-defined temporal boundaries, and spatial-temporal localization.Its design targets concurrent actions, subtle motion cues, long-tailed categories, duration variation, fast movement, deformation, and occlusion.
  • Dataset Construction: Annotations combine athlete-defined temporal and category labels with crowd-sourced frame-by-frame bounding-box refinement using FCOT tracking.A handbook specifies action boundaries, while quality control includes domain-expert checks and manual correction of bounding boxes.
  • Dataset Construction: The dataset contains 66 action categories from four sports, with 800 clips per sport selected from 247 competitions.The sports are basketball, volleyball, football, and aerobic gymnastics.
  • Dataset Statistics: MultiSports differs from prior datasets through longer clips, 66 categories, 37,701 instances, 11.8 instances per clip, and the largest number of bounding boxes among compared datasets.The reported clip duration is 20.9 seconds, compared with 1.2 or 6.9 seconds for J-HMDB and UCF101-24.
  • Dataset Statistics: Action frequencies are long-tailed, ranging from 3 to 3,477 instances per category, while instance durations vary substantially and complicate temporal localization.The train, validation, and test sets are split at the clip level in a 3:1:2 ratio for each sport.
  • Quality Control: High-resolution competition videos and domain-informed quality control support detailed annotations, while tracking quality varies across sports and is hardest for aerobic gymnastics.Bounding boxes are checked manually, including at 5 fps during the second annotation stage.

4. Experiments and Analysis

Experiments show that MultiSports is substantially harder than prior benchmarks, especially for fine-grained, fast, and temporally precise actions. Error analysis attributes performance losses to classification, temporal-boundary, and missed-detection errors.

  • 4.2. Spatio-temporal Action Detection Results: 25.22% frame-mAP, 12.88% video-mAP@0.2, and 0.62% video-mAP@0.5 are obtained by MOC on MultiSports, demonstrating its difficulty relative to UCF101-24 and JHMDB.The benchmark evaluates frame-mAP and video-mAP across representative action detectors and datasets.
  • 4.2. Spatio-temporal Action Detection Results: 11.02% versus 4.54% is the SlowFast–SlowOnly frame-mAP gap on MultiSports and AVA, respectively, indicating greater value from fine temporal analysis for sports actions.Aerobic turn improves by +30 AP and aerobic horizontal support by +54 AP, while several short, intense actions also improve.
  • 4.3. Error Analysis: SlowFast Det. achieves higher video-mAP than MOC despite relatively low recall because its Faster R-CNN person detector produces fewer false positives.The comparison is reported in the error analysis of MultiSports detection results.
  • 4.3. Error Analysis: EC and EN are common false-positive errors, while EC&T, EC&T&L, and ET show that inaccurate temporal boundaries remain more prevalent than spatial localization errors.EN also reflects insufficient negative samples for people without sports actions in MOC training.
  • 4.4. Ablation Study: MultiSports benefits more from longer temporal context for frame-mAP than UCF101-24, but fixed tubelet lengths and inaccurate movement prediction limit video-level gains.The analysis motivates longer context, more accurate movement estimation, and flexible temporal detection.
  • 4.4. Ablation Study: 11% performance loss occurs when SlowFast Det. is evaluated on untrimmed rather than trimmed MultiSports videos, compared with 2% on AVA.The larger drop, especially at video-mAP@0.5, indicates the importance of temporal localization for high-quality action tubes.

5. Conclusion

The conclusion presents MultiSports as a dense, diverse benchmark for fine-grained spatio-temporal action detection in multi-person sports scenes. It emphasizes accurate boundaries, high-quality data, and the resulting research challenges.

  • 5. Conclusion: MultiSports combines fine-grained action classes, accurate boundaries, multiple-person settings, high-quality videos, dense annotations, and diverse competitions, countries, and genders.The dataset is positioned for action detection and potential sports-analysis applications.

Appendix A: More Dataset Details

The appendix describes a leakage-controlled split designed to preserve class coverage despite severe imbalance. Training, validation, and testing data are separated by match.

  • Appendix A: More Dataset Details: Instances are manually split into training, validation, and testing sets, with clips from the same match restricted to one split.The procedure is intended to provide enough instances per class while avoiding leakage between training and validation/testing.

A.2 Comparison with other type of Dataset

The comparison frames MultiSports as a multi-person dataset with complex sports actions rather than atomic daily activities. Its action categories involve greater deformation and displacement than those in MEVA.

  • A.2 Comparison with other type of Dataset: MultiSports differs from MEVA through fine-grained, complicated sports categories rather than atomic daily activities with limited deformation and displacement.MEVA contains 37 activities and 17,055 instances, whereas MultiSports contains 66 activities and 37,701 instances.

Appendix B: More Ablation Study

Adding temporal noise to ground-truth annotations substantially worsens performance, supporting the value of MultiSports’ well-defined temporal boundaries.

  • Performance is much worse when temporal noise is added to training annotations, indicating that well-defined boundaries matter for detection.The experiment randomly shortens and offsets each instance while keeping other settings unchanged.
  • The boundary ablation suggests that high-quality temporal annotations help detectors localize action boundaries more accurately.
  • The study tests temporal-boundary quality by sampling shortened, offset annotations from original ground-truth instances.

Appendix C: Method Details

The appendix specifies baseline detector architectures, linking procedures, training settings, and the dataset split used for evaluation.

  • Baseline methods: ROAD detects and classifies action boxes independently per frame, then links them into action tubes with an online algorithm.The MultiSports implementation uses MOC’s Python linking code instead of ROAD’s original MATLAB code.
  • Baseline methods: YOWO fuses spatial key-frame and spatiotemporal features before predicting frame detections, which are linked into tubes by dynamic programming.Because YOWO’s original linking algorithm is trimmed, the appendix uses the same linking algorithm as MOC on MultiSports.
  • Baseline methods: MOC processes K frames to produce tubelet results, while SlowFast Det. extends key-frame person proposals into 3D regions for action classification.The appendix uses DLA34 for MOC and a Faster R-CNN person detector for SlowFast Det.
  • Data split: The train and validation splits are separated by match so clips from the same match do not appear in both purposes.The split is designed to provide enough instances for each class despite severe imbalance.
  • Ablation setup: Temporal-boundary noise is evaluated as an ablation, with Table 7 documenting its effect on detector performance.

D.1 Error Tree

The error tree decomposes detection failures into missed, unmatched, repeated, classification, temporal-localization, and spatial-localization errors, including their combinations.

  • Error categories: Repeated detections occur when a lower-confidence detection duplicates a ground truth already matched by another detection.
  • Error categories: Classification errors match a ground-truth tubelet spatially and temporally but assign the wrong action class.
  • Localization errors: Temporal-localization errors have adequate spatial overlap but insufficient temporal IoU, while spatial-localization errors have the opposite pattern.
  • Combined errors: Combined categories capture simultaneous classification, temporal-localization, and spatial-localization failures, including ET&L and EC&T&L.
  • Error-tree overview: Figure 8 organizes the ten detection-error categories into a tree structure for systematic analysis.
  • Error categories: Missed detections are unmatched ground-truth tubelets, whereas not-matched errors are detections with no intersection with any ground-truth tubelet.

D.2 More Visualization of Error Analysis

The appendix provides additional visualizations of MOC with K=11 as a supplement to Figure 7.

  • Additional visualizations of MOC(K=11) supplement the examples presented in Figure 7.
  • The visualizations are intended to provide further examples for the error analysis.
  • Figure 9 contains the supplementary visualization set.

D.3 Confusion Matrix

The confusion analysis shows that action recognition errors arise when similar motions require contextual, interaction-aware, or temporally extended reasoning. Aerobic gymnastics performs best when categories depend primarily on individual actors.

  • Aerobic gymnastics performs best because its categories relate only to individual actors.
  • Actions with similar motions are confused when distinguishing spatio-temporal context is necessary.
  • Basketball drive versus dribble requires modeling person-person interaction and spatial localization.
  • Football through ball versus pass requires long-term temporal modeling and reasoning.
  • Defend versus protect in volleyball depends on whether the ball was blocked back or spiked by an opponent several frames earlier.
Loading 2105.07404v2…