Source-linked AI summary
BABEL: Bodies, Action and Behavior with English Labels
Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, Michael J. Black
TL;DR
Human-motion semantics require data combining language labels, precise 3D motion, and coverage of actions as they occur naturally. BABEL constructs and annotates AMASS mocap at sequence and frame levels, then benchmarks 3D action recognition, finding realistic learning challenges in its diverse, long-tailed action distribution.
Problem
Existing datasets provide either many action labels without ground-truth 3D motion or precise mocap with sparse labels and few actions, limiting semantic coverage of human movement.
Method
BABEL collects sequence-level and precisely aligned frame-level language labels for AMASS mocap, including simultaneous actions and transitions, and benchmarks a 2s-AGCN model for 3D action recognition.
Results
28,055 sequence labels and 63,353 frame labels cover 13,220 AMASS sequences at sequence level and 10,892 sequences with dense frame annotations.
Takeaways & Limitations
BABEL provides a benchmark for modeling realistic action distributions and supports tasks including action recognition, temporal localization, motion synthesis, pose estimation, and few-shot learning.
Takeaways & Limitations
Annotator disagreement varies across action categories, action counts, and segments, with generally larger variance for longer sequences; the dataset also reports potential interface, pay-structure, and label-processing biases.
Abstract
from arXiv · showhide
Understanding the semantics of human movement -- the what, how and why of the movement -- is an important problem that requires datasets of human actions with semantic labels. Existing datasets take one of two approaches. Large-scale video datasets contain many action labels but do not contain ground-truth 3D human motion. Alternatively, motion-capture (mocap) datasets have precise body motions but are limited to a small number of actions. To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences. BABEL consists of action labels for about 43 hours of mocap sequences from AMASS. Action labels are at two levels of abstraction -- sequence labels describe the overall action in the sequence, and frame labels describe all actions in every frame of the sequence. Each frame label is precisely aligned with the duration of the corresponding action in the mocap sequence, and multiple actions can overlap. There are over 28k sequence labels, and 63k frame labels in BABEL, which belong to over 250 unique action categories. Labels from BABEL can be leveraged for tasks like action recognition, temporal action localization, motion synthesis, etc. To demonstrate the value of BABEL as a benchmark, we evaluate the performance of models on 3D action recognition. We demonstrate that BABEL poses interesting learning challenges that are applicable to real-world scenarios, and can serve as a useful benchmark of progress in 3D action recognition. The dataset, baseline method, and evaluation code is made available, and supported for academic research purposes at https://babel.is.tue.mpg.de/.
1. Introduction
BABEL addresses the lack of large-scale, densely labeled 3D human-motion data by annotating AMASS mocap with sequence- and frame-level language labels that capture concurrent and transitional actions. It provides a broad benchmark for 3D action recognition and related semantic-motion tasks.
- Motivation: Existing datasets offer either many action labels without ground-truth 3D motion or precise mocap with sparse labels and few actions.BABEL is designed to address this gap with diverse, densely annotated labels for all actions occurring in mocap sequences.
- Dataset design: BABEL labels actions at sequence and frame levels, precisely aligning frame labels with movement spans while allowing simultaneous actions and transitions.Every frame is assigned at least one action, and all actions in each frame are labeled.
- Dataset scale: 43.5 hours of AMASS mocap yield 15,472 unique language labels organized into 260 action categories across eight broad semantic groups.Categories include simple actions, complex activities, and body-part interactions.
- Dataset scale: 28,055 sequence labels and 63,353 frame labels cover 13,220 sequences at sequence level and 10,892 sequences with dense frame annotations.Dense frame annotations span more than 37 hours, while sequence annotations span more than 43 hours; sequences average 6.06 segments and 4.02 unique action categories.
- Benchmark: BABEL benchmarks 3D action recognition under a long-tailed class distribution that presents learning challenges closer to real-world applications.The dataset is also positioned for temporal localization, motion synthesis, pose estimation, and few-shot learning.
2. Related Work
Related datasets differ in motion fidelity, label granularity, action diversity, and whether sequences are continuous or cropped. BABEL combines ground-truth mocap with dense per-frame labels for continuous natural movement and overall sequence labels.
- Mocap datasets: CMU, Human3.6M, MoVi, KIT, and LaFan1 characterize mocap sequences using small fixed sets of actions, tags, keywords, or selected activities.Their reported action inventories range from 12 everyday actions in Human3.6M to 2605 sequences in CMU with six semantic categories.
- BABEL comparison: BABEL uniquely supplies dense per-frame labels for 37.5 hours of continuous ground-truth movement and sequence-level labels for 43.5 hours of AMASS mocap.The comparison table distinguishes motion accuracy, action count, duration, frame alignment, and continuity.
- Language labels: The KIT Motion-Language Dataset describes motion sequences with natural-language sentences, whereas BABEL aligns action labels precisely with the sequence.BABEL focuses on action labels tied to movement spans rather than only sequence-level descriptions.
- Frame-level labels: CMU MMAC provides precise frame labels for 17 cooking actions, while other mocap labeling methods partially automate temporal segmentation using known fixed classes.BABEL instead identifies and precisely labels all actions occurring in each frame.
- Tracked 3D datasets: NTU RGB+D 60 and 120 provide large-scale 3D action-recognition data from tracked RGBD sequences, but their samples are short cropped segments tied to specific actions.BABEL sequences are continuous and represent simultaneous, sequential, and transitional natural movement.
3. Dataset
BABEL combines crowdsourced sequence- and frame-level action labels for AMASS mocap, using dense temporal annotation and semantic processing to represent varied, concurrent, and sequential movement.
- BABEL action labels: BABEL collects sequence labels for overall actions and frame labels aligned to fine-grained action spans, including simultaneous actions and transitions.Frame annotation requires every frame to contain at least one action label and all actions in a frame to be labeled.
- Data collection: BABEL elicits free-form labels to capture action diversity, while accepting label variance that later requires semantic normalization.Annotators can also select from an autocomplete list of basic actions, but most often enter their own labels.
- BABEL action labels: Annotators first name all actions and then mark each action’s temporal span through an interactive segment-based interface.The interface supports changing segment duration and start/end times by click-and-drag operations.
- BABEL action labels: 66,018 action segments cover 10,892 sequences, combining frame labels for multi-action sequences with sequence labels for single-action sequences.Frame labels were collected for 6,663 sequences where annotators agreed that multiple actions were present.
- Label processing: 15,472 raw action labels are clustered in a 300D Word2Vec space with K-means, then manually verified and adjusted into 260 action categories.Raw labels are preprocessed through lowercasing, whitespace removal, and lemmatization; some labels may belong to multiple categories.
- Label processing: The processed categories support modeling movement at different semantic levels, from body-part movements to high-level activities, using either categories or embedded raw labels.This supports tasks such as action recognition and action synthesis at different levels of semantic abstraction.
4. Analysis
BABEL exposes simultaneous actions and temporal transitions in natural movement, then analyzes their structure and models transition probabilities to generate plausible action sequences. Annotator disagreement remains a documented source of bias, especially for longer sequences.
- 4.1. Simultaneous actions: 49,952 simultaneous-action instances span 2,907 unique action-category pairs, capturing overlaps longer than 0.1 seconds.The analysis distinguishes hierarchical, complementary, and superimposed relationships among simultaneous actions.
- 4.2. Temporally adjacent actions: BABEL represents transitions as directed action-segment links, ignoring overlaps under 0.1 seconds and collapsing intervening transition segments.The transition count determines edge thickness in the action graph.
- 4.2. Temporally adjacent actions: Adjacent-action analysis identifies frequent transitions and semantically meaningful chains such as sit → stand up → walk.Walk has the most diverse adjacent-action set, while some pairs are bidirectional and others are more directional.
- 4.2. Temporally adjacent actions: An order-3 Markov chain P(ai|ai−1, ai−2, ai−3) produces plausible action sequences through random walks.The model estimates transition probabilities from temporally adjacent actions in BABEL.
- 4.3. Bias: Annotators vary in action categories, action counts, and segments for the same sequence, with greater variance generally appearing in longer sequences.The paper discusses interface, pay-structure, and label-processing sources of bias in supplementary material.
5. Experiments
The experiments benchmark 3D action recognition on BABEL using 2s-AGCN after converting mocap into normalized 25-joint skeleton chunks. Results show that long-tailed classes challenge recognition, while class-balanced focal loss improves class-balanced performance.
- 5. Experiments: BABEL action recognition predicts action categories from mocap segments using normalized 25-joint skeleton representations.AMASS poses are converted to Cartesian joint coordinates and mapped to one or more segment-level action categories.
- 5. Experiments: 2s-AGCN applies spatial and temporal graph convolutions, with separate joint and bone streams; experiments use only the joint stream.The architecture adapts graph topology by layer and sample, while the two-stream prediction averages stream scores in the full model.
- 5. Experiments: 5 sec. motion chunks form individual recognition samples after segments are divided into contiguous, non-overlapping chunks and short final chunks are repeated and truncated.The chunking procedure follows the 2s-AGCN preprocessing approach.
- 5. Experiments: BABEL evaluates 260 long-tailed action categories and uses BABEL-60 and BABEL-120 subsets to examine recognition under different class counts.The subsets are drawn from 10,892 dense-subset sequences.
- 5. Experiments: Top-1-norm is much lower than Top-1, indicating inefficient learning from BABEL’s long-tailed class distribution.Top-1-norm averages Top-1 across categories, while their difference reflects class-specific bias and class imbalance.
- 5. Experiments: Class-balanced focal loss significantly improves Top-1-norm performance across all BABEL subsets.The training comparison includes standard cross-entropy and focal-loss settings, with focal loss weighting inaccurate predictions more heavily.
- 5. Experiments: BABEL serves as an additional recognition benchmark for models learning both frequent and infrequent actions under realistic class imbalance.Its diverse, long-tailed distribution contrasts with the controlled motion and action distribution of NTU RGB+D.
6. Conclusion
BABEL provides dense action labels for mocap sequences, including simultaneous actions and transitions, and establishes a challenging benchmark for realistic 3D action recognition.
- BABEL labels all actions in mocap sequences, including simultaneous actions and transitions between actions.
- The dataset includes dense annotations for high-quality mocap data rather than only sparse action labels.
- Label diversity and a long-tailed sample distribution make action recognition on BABEL challenging.
- BABEL serves as an additional action-recognition benchmark for evaluating models on realistic data distributions.
- The authors position BABEL as a resource to accelerate progress in understanding human movement semantically.