Source-linked AI summary
FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding
Dian Shao, Yue Zhao, Bo Dai, Dahua Lin
TL;DR
Fine-grained action analysis requires temporal decomposition and subtle semantic discrimination beyond the coarse categories emphasized by existing benchmarks. The paper introduces FineGym, a high-quality gymnastics dataset with hierarchical annotations and empirical studies showing unresolved modeling challenges. Its scope is currently fine-grained human action recognition on gymnastics, with additional annotation types planned for later versions.
Problem
Fine-grained action understanding remains limited because existing benchmarks and methods focus largely on coarse categories, while sport analytics requires detailed temporal and semantic analysis.
Method
The paper constructs FineGym from professional gymnastics videos, using three semantic annotation levels and two temporal annotation levels with multi-stage collection and quality control.
Results
Empirical studies find that representative methods perform far from satisfactorily on fine-grained recognition and identify sparse sampling, motion, temporal dynamics, and pre-training as important modeling choices.
Takeaways & Limitations
FineGym provides a challenging benchmark and reveals key challenges that may guide future research in fine-grained action understanding.
Takeaways & Limitations
The current FineGym version mainly provides annotations for fine-grained human action recognition on gymnastics, while more annotation types are planned for succeeding versions.
Abstract
from arXiv · showhide
On public benchmarks, current action recognition techniques have achieved great success. However, when used in real-world applications, e.g. sport analysis, which requires the capability of parsing an activity into phases and differentiating between subtly different actions, their performances remain far from being satisfactory. To take action recognition to a new level, we develop FineGym, a new dataset built on top of gymnastic videos. Compared to existing action recognition datasets, FineGym is distinguished in richness, quality, and diversity. In particular, it provides temporal annotations at both action and sub-action levels with a three-level semantic hierarchy. For example, a "balance beam" event will be annotated as a sequence of elementary sub-actions derived from five sets: "leap-jump-hop", "beam-turns", "flight-salto", "flight-handspring", and "dismount", where the sub-action in each set will be further annotated with finely defined class labels. This new level of granularity presents significant challenges for action recognition, e.g. how to parse the temporal structures from a coherent action, and how to distinguish between subtly different action classes. We systematically investigate representative methods on this dataset and obtain a number of interesting findings. We hope this dataset could advance research towards action understanding.
1. Introduction
FineGym addresses the limits of coarse-grained action recognition by providing a dataset for temporally decomposing actions and semantically distinguishing fine-grained sub-classes. Empirical studies show that current methods remain inadequate in this setting and highlight several modeling challenges.
- Existing benchmarks and techniques achieve high accuracy mainly on coarse-grained categories, where background context can help distinguish actions.
- Fine-grained action understanding requires both temporal decomposition into smaller elements and semantic differentiation between sub-classes.
- FineGym is a large-scale, high-quality dataset with three semantic levels—event, set, and element—and manually annotated action and sub-action temporal structures.
- Tests of representative methods find fine-grained recognition performance far from satisfactory and revisit sampling, modalities, temporal dynamics, and pre-training.
- The studies report that sparse frames are insufficient, motion is significantly important, temporal dynamics require correct modeling, and pre-training on
- FineGym’s dataset and analyses reveal key challenges in fine-grained action understanding and may point to future research directions.
2. Related Work
Prior action-recognition research has been driven by increasingly challenging benchmarks and models, with annotations expanding beyond category labels to temporal and spatial-temporal information.
- Action-recognition datasets progressed from early benchmarks such as KTH and Weizmann to larger, more challenging datasets including UCF101, Kinetics, and ActivityNet.
- Existing datasets provide annotations ranging from category labels and temporal locations to spatial-temporal bounding boxes.
- Representative recognition pipelines combine per-frame 2D CNN semantics with 1D temporal aggregation, including sparse segment sampling in TSN.
3. The FineGym Dataset
FineGym is a gymnastics benchmark built from professional competition videos, with hierarchical semantic and temporal annotations, quality control, and challenging fine-grained instances. Its design supports both recognition studies and analysis of action-centric, subtle, and temporally complex behavior.
- FineGym targets fine-grained human action recognition in gymnastics, while additional annotation types are reserved for succeeding versions.
- The dataset organizes actions into events, sets, and elements, and annotates event locations plus sub-action locations within each event instance.
- FineGym construction covers data preparation, annotation collection, and quality control for a complex fine-grained dataset.
- Official high-resolution competition records are selected because ordinary web-collected or self-recorded videos may not match detailed gymnastics labels.
- Element labels are assigned through manually built decision trees with attribute-based queries, whose root-to-leaf paths can provide attributes and difficulty scores.
- FineGym v1.0 includes 10 event categories and 530 defined element categories, with 354 element categories having at least one instance and a heavy-tailed distribution.
- Over 95% of videos are 720P or 1080P, while annotations are designed to remain consistent and clean across semantic and temporal granularities.
- Fine-grained instances include diverse viewpoints, poses, and subtle category differences, making FineGym action-centric and challenging for recognition.
4. Empirical Studies
The study evaluates representative recognition and localization methods across FineGym’s granular action settings, exposing failures in temporal modeling, subtle visual distinctions, and motion-intensive cases.
- Experimental setup: Representative methods are evaluated for element recognition across all events, within events, and within sets, plus temporal localization at action and sub-action levels.The recognition study includes 2D-CNN, 3D-CNN, and skeleton-based pipelines; SSN is used for localization.
- Element-level action recognition: Long-tailed instance distributions cause all methods to overfit frequent elements, especially in Gym288.
- Element-level action recognition: Temporal modeling is important: TRN and TSM outperform TSN by large margins, while motion features help in most settings except motion-intensive Vault recognition.
- Analysis: 12 frames are needed for TSN on Gym99 to reach saturation, indicating that every frame counts for FineGym’s fine-grained recognition.The 12-frame setting accounts for 30% of all frames.
- Analysis: Kinetics pre-training is not always helpful on FineGym, potentially because coarse- and fine-grained actions have different temporal patterns.
- Analysis: Current methods struggle with intense motion, subtle spatial semantics, complex temporal dynamics, and reasoning such as counting salto repetitions.Skeleton-based methods face additional difficulty because gymnast poses are hard to estimate reliably.
5. Potential Applications and Discussion
FineGym’s annotations support applications beyond recognition and localization, while also motivating research on efficient modeling of highly informative, subtly different actions and complex temporal dynamics.
- Potential applications: FineGym can support gymnastic auto-scoring by combining official difficulty scores with visually estimated quality scores.
- Potential applications: Its standardized, diverse instances and consistent sub-action context may support action generation focused on the actions themselves.
- Potential applications: Decision-tree-derived attributes enable multi-attribute prediction, while manually built decision trees can support model interpretation and reasoning.
- Discussion: Open research questions include balancing accuracy with efficiency and modeling complex temporal dynamics efficiently, effectively, and robustly.
6. Conclusion
FineGym is a high-quality, action-centric gymnastics dataset with consistent semantic and temporal annotations, and empirical studies that reveal challenges for fine-grained action understanding.
- Conclusion: FineGym combines high-quality, action-centric data with diverse action instances and annotations spanning multiple semantic and temporal granularities.
- Conclusion: Empirical investigations of representative methods produce findings that expose new challenges and may guide future action-understanding research.