Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset

Motivation

• Existing datasets often contain body-only motion, omitting facial expressions and hand gestures needed for expressive and realistic movement.Specialized actions are also difficult to capture in indoor scenes.Source: §1 · p. 1
• Markerless video capture and automatic pose scripting provide a scalable alternative for collecting massive, semantically labeled motion data.The paper notes that text-driven motion generation tolerates some capture error because semantic correspondence matters more than vertex-level accuracy.Source: §1 · p. 2

Dataset and pipeline

• Motion-X contains 15.6M frames and 81.1K sequences with precise 3D whole-body annotations, pose descriptions, and semantic labels.The dataset combines Internet videos, professional and outdoor actions, game and animation motions, and eight existing action datasets.Source: §1 · p. 2
• The paper proposes an automatic motion and text annotation pipeline for efficient, high-quality whole-body text-motion data at scale.Its contributions include sequence-level and frame-level text descriptions alongside whole-body motion.Source: §1 · p. 2

Evaluation

• Experiments demonstrate annotation accuracy and benefits for 3D whole-body motion generation and mesh recovery.The reported benefits concern expressive, diverse, and natural motion generation.Source: §1 · p. 2

Motion-X

• Motion-X provides indoor and outdoor whole-body motion and text annotations, including sequence-level semantics and frame-level pose descriptions.Table 1 compares these properties with existing text-motion datasets.Source: Table 1 · p. 3, §2 · p. 3

Existing datasets

• Existing sequential motion-and-text benchmarks primarily support action recognition, human-object interaction, and motion generation.The related-work discussion reviews datasets developed for these three tasks.Source: §2 · pp. 2–3
• AMASS unified 15 optical marker-based motion-capture datasets through a common SMPL framework and parameterization.BABEL and HumanML3D expanded language annotations with sequence, subsequence, or multiple clip descriptions.Source: §2 · p. 3

Remaining gaps

• Existing text-motion datasets lack facial expressions and hand gestures, sufficient quantity, diverse motions and scenes, fine-grained descriptions, and long sequences.Motion-X is introduced to bridge these coverage and labeling gaps.Source: §2 · p. 3

Dataset overview

• 15.6M 3D whole-body SMPL-X annotations and 81.1K sequence-level semantic descriptions are provided alongside frame-level whole-body pose descriptions.All annotations are manually checked for quality.Source: §3.1 · p. 3
• Figure 2 summarizes body, hand, and face motion diversity for each sub-dataset, highlighting hand movements and facial expressions beyond body-only motion data.Diversity is measured using averaged temporal standard deviations of body, hand, and face keypoints.Source: Figure 2 · p. 3

Annotation pipeline

• The overall collection and annotation process comprises six steps, from LLM-designed motion prompts and video collection through preprocessing, motion capture, captioning, and manual inspection.The pipeline includes both motion and text annotation stages.Source: §3.2 · p. 4

Dataset overview

• Motion-X integrates 37K sequences from existing datasets with 32.5K monocular online videos covering general and domain-specific actions.The existing sources include multi-view, human-scene-interaction, action-recognition, and body-only motion-capture datasets.Source: §3.2 · p. 4

Annotation pipeline

• The whole-body motion-capture pipeline estimates 2D and 3D keypoints, optimizes local pose and global translation, and supports single- and multi-view videos.Multi-view-specific processing is represented by dashed lines in Figure 5.Source: Figure 5 · p. 5

Pipeline overview

• The annotation pipeline targets high-quality whole-body motion capture from massive videos using three specialized techniques.These are hierarchical whole-body keypoint estimation, score-guided adaptive temporal smoothing, and learning-based 3D human model fitting.Source: §4.1 · p. 5

Keypoint estimation

• Hierarchical keypoint estimation addresses hand and face detection errors caused by small regions, occlusion, and interaction.The method uses a customized hierarchical approach for 2D whole-body keypoint annotation.Source: §4.1 · p. 5

Temporal smoothing

• Confidence-adaptive smoothing enlarges the filter window for low-confidence keypoints, reducing outliers while preserving motion details.Unlike fixed-window smoothing, the method adapts its window size using keypoint confidence scores.Source: §4.1 · p. 5

3D fitting

• Progressive learning-based fitting registers SMPL-X poses using image clues and motion priors, while physical constraints reduce interpenetration and foot skating.The fitting objective combines joint alignment with smoothness, penetration, and physical losses.Source: §4.1 · p. 6

Global optimization

• Global motion optimization jointly refines trajectories and camera poses to align estimated motion with video evidence.Its losses include 2D keypoint distance, trajectory consistency, trajectory regularization, and camera smoothness.Source: §4.1 · p. 6

Text annotation

• Whole-body descriptions combine facial emotion labels, body posecodes, and fine-grained hand-gesture descriptions into natural-language frame annotations.The resulting dataset contains 81.1K clips with 15.6M SMPL-X poses and corresponding pose and semantic text labels.Source: §4.2 · p. 7

Evaluation scope

• Experiments assess the annotation pipeline, text-driven motion generation, and whole-body human mesh recovery.The evaluation includes 2D keypoints, 3D SMPL-X fitting, generation benchmarks, cross-dataset comparisons, text-label ablations, and mesh recovery.Source: §5 · p. 7, Table 3 · p. 7, Table 4 · p. 8

Annotation accuracy

• Over 15% average precision improvement is reported for 2D keypoint annotation against four state-of-the-art methods on COCO-WholeBody.All methods use the same 256×192 input image size for comparison.Source: §5.1 · p. 7
• Over 30% error reduction is reported for progressive 3D SMPL-X fitting against four open-sourced human mesh recovery methods.Evaluation uses MPVPE, PA-MPVPE, and PA-MPJPE on EHF.Source: §5.1 · p. 7

Cross-dataset comparison

• 13.174 versus 9.837 diversity is reported for Motion-X and HumanML3D ground-truth motions, respectively.The comparison evaluates MLD trained separately on the two datasets and supports broader motion types in Motion-X.Source: §5.2 · p. 8

Text-driven generation

• Models trained on Motion-X generate more accurate, semantically corresponding, expressive, diverse, and natural motions than the comparison shown against HumanML3D.The Motion-X-trained model also performs well after finetuning on the HumanML3D subset, including better performance than intra-data training on its test set.Source: Figure 8 · p. 8, §5.2 · p. 9
• Whole-body pose descriptions produce a 38% FID reduction when added to semantic labels in the text-label ablation.Adding face and body pose texts yields consistent improvements, while combining whole-body descriptions gives the largest reported reduction.Source: §5.2 · p. 9

• Motion-X contains 144.2 hours of whole-body motions, 81.1K annotated motions, and 15.6M frame-level pose descriptions.Source: §6 · p. 9
• The dataset addresses existing mocap limitations by covering expressive whole-body motions rather than primarily indoor body-only actions.Source: §6 · p. 9
• Experiments demonstrate annotation accuracy and benefits for expressive, diverse, natural motion generation and 3D whole-body human mesh recovery.Source: §6 · p. 9
• The markerless annotation pipeline is less accurate than multi-view marker-based capture, and existing evaluation metrics can disagree with visual results.Source: §6 · p. 9
• Motion-X provides large diversity in hand and face joints and many long motions exceeding 240 frames.Source: §A.1 · p. 11

• The unified framework gathers 81.1K sequences and supplies whole-body motion, semantic labels, and pose descriptions where source datasets lacked them.Source: §A.2 · p. 11
• AMASS body motions receive facial expressions from BAUM, semantic labels from HumanML3D, and pose descriptions from the whole-body captioning module.Source: §A.2 · p. 11
• IDEA400 expands NTU120 categories to 400 actions with self-contact, human-object contact, and expressive whole-body motions.Source: §A.2 · pp. 11–12
• AIST, HAA500, HuMMan, GRAB, and EgoBody are reannotated or augmented to provide whole-body motion and complementary semantic or pose text.Source: §A.2 · p. 12
• Motion-X also collects 33K online monocular videos spanning real-life scenes, professional motions, and diverse appearances.Source: §A.2 · p. 12
• Lower-body augmentation selects compatible AMASS motion, while facial augmentation inserts interpolated BAUM expressions and emotion labels.Source: §A.3 · pp. 12–13

2D Keypoints

• The hierarchical Transformer keypoint estimator performs better than Openpose and MediaPipe, especially for hand keypoint localization in challenging poses.Source: §B.1 · p. 14

SMPL-X Parameters

• The learning-based fitting method aligns SMPL-X parameters better than Hand4Whole and OSX, particularly on difficult poses.Source: §B.2 · p. 14
• The fitting method is more robust than SMPLify-X for physically plausible poses under hard poses, low resolution, heavy occlusion, and depth ambiguity.Source: §B.2 · p. 14

Motion Sequences

• Six motion styles for the same dance-ballet semantic label illustrate one-to-many text-to-motion diversity.Source: §B.3 · p. 14
• Additional visualizations show variation in facial expressions, hand poses, and body motions across motion scenes.Source: §B.3 · p. 14

Motion Representation

• SMPL-X represents each pose state with body rotations, hand rotations, jaw pose, facial expression, and global translation.Source: §C.1 · p. 14
• The pose state includes θb for body rotations, θh for hand rotations, θf for jaw pose, ψ for facial expression, and r for global translation.Source: §C.1 · p. 14

Evaluation Metrics

• Evaluation uses FID, multimodality, diversity, R-precision, and multimodal distance with pretrained motion and text feature extractors.Source: §C.1 · p. 14

Computational Costs

• Annotating 1M frames takes about 72 hours using 8 NVIDIA A100 GPUs.Source: §C.1 · p. 15

• HumanML3D is limited in expressing rich hand and face motions because it contains body and minimal hand movements.Source: §C.2 · p. 15
• The annotation quality of Motion-X is compared with Hand4Whole and OSX, as well as the fitting method SMPLify-X.Source: Figure 12 · p. 16, Figure 13 · p. 17
• Motion-X and HumanML3D are compared using MLD models trained on each dataset, evaluating diversity in generated and ground-truth motions.Diversity is measured with average temporal standard deviation of SMPL-X parameters and joint positions.Source: §C.2 · p. 16, Table 9 · p. 17
• Motion-X produces higher temporal deviation, especially in hand and face parameters, indicating more substantial movements over time.Source: §C.2 · p. 17
• Text-conditioned motion generation aims to synthesize long-term, diverse, natural human motions from descriptions or other conditions.Source: D Related Work · pp. 17–18

• The markerless annotation pipeline is less accurate than multi-view marker-based motion capture, despite outperforming existing markerless methods.The authors report that a 30 mm PA-MPVPE error is acceptable for text-driven motion generation.Source: §E.1 · p. 18
• Improved annotation quality remains future work because higher-quality motion annotations would benefit both generation and human reconstruction.Source: §E.1 · p. 18
• Existing evaluation metrics are not always consistent with visual results, and SMPL-X parameters may not be optimal for expressive whole-body motion.Source: §E.1 · p. 18
• Despite inevitable errors, scalable automatic annotation can support large-scale learning from noisy labels for related tasks.Source: §E.2 · p. 18
• Motion-X data is distributed under CC BY-NC-SA, while users must consult original licenses for incorporated sub-datasets.Source: F License · pp. 18–19