PaperlayerView exact revision on arXiv

Source-linked AI summary

Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset

Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, Lei Zhang

arXiv:2307.00818v2cs.CV

TL;DR

Existing text-motion datasets are limited by body-only coverage, restricted scenes, and coarse or incomplete labels. Motion-X introduces a scalable whole-body motion-and-text annotation pipeline and a large dataset, with experiments showing benefits for expressive motion generation and whole-body mesh recovery. The paper also reports that the pipeline’s motion quality remains below multi-view marker-based capture and that evaluation metrics can disagree with visual results.

  • Problem

    Existing motion datasets commonly lack facial expressions, hand gestures, fine-grained pose descriptions, sufficient scale, and diverse scenes.

  • Method

    The paper develops an automatic annotation pipeline for motion and text, then applies it to videos and existing datasets to construct whole-body annotations and labels.

  • Results

    Motion-X contains 15.6M 3D whole-body SMPL-X annotations across 81.1K sequences, and experiments show benefits for expressive motion generation and whole-body mesh recovery.

  • Takeaways & Limitations

    Motion-X provides large-scale whole-body motion with sequence-level semantic labels and frame-level pose descriptions for downstream motion generation and mesh recovery research.

  • Takeaways & Limitations

    The markerless annotation pipeline has lower motion quality than multi-view marker-based capture, and existing evaluation metrics are not always consistent with visual results.

Abstract

from arXiv · show

In this paper, we present Motion-X, a large-scale 3D expressive whole-body motion dataset. Existing motion datasets predominantly contain body-only poses, lacking facial expressions, hand gestures, and fine-grained pose descriptions. Moreover, they are primarily collected from limited laboratory scenes with textual descriptions manually labeled, which greatly limits their scalability. To overcome these limitations, we develop a whole-body motion and text annotation pipeline, which can automatically annotate motion from either single- or multi-view videos and provide comprehensive semantic labels for each video and fine-grained whole-body pose descriptions for each frame. This pipeline is of high precision, cost-effective, and scalable for further research. Based on it, we construct Motion-X, which comprises 15.6M precise 3D whole-body pose annotations (i.e., SMPL-X) covering 81.1K motion sequences from massive scenes. Besides, Motion-X provides 15.6M frame-level whole-body pose descriptions and 81.1K sequence-level semantic labels. Comprehensive experiments demonstrate the accuracy of the annotation pipeline and the significant benefit of Motion-X in enhancing expressive, diverse, and natural motion generation, as well as 3D whole-body human mesh recovery.

1 Introduction

Motion-X addresses limitations in existing motion datasets by combining expressive whole-body coverage with scalable motion and text annotation. The resulting dataset supports diverse motion generation and whole-body mesh recovery.

  • Motivation: Existing datasets often contain body-only motion, omitting facial expressions and hand gestures needed for expressive and realistic movement.Specialized actions are also difficult to capture in indoor scenes.
  • Motivation: Markerless video capture and automatic pose scripting provide a scalable alternative for collecting massive, semantically labeled motion data.The paper notes that text-driven motion generation tolerates some capture error because semantic correspondence matters more than vertex-level accuracy.
  • Dataset and pipeline: Motion-X contains 15.6M frames and 81.1K sequences with precise 3D whole-body annotations, pose descriptions, and semantic labels.The dataset combines Internet videos, professional and outdoor actions, game and animation motions, and eight existing action datasets.
  • Dataset and pipeline: The paper proposes an automatic motion and text annotation pipeline for efficient, high-quality whole-body text-motion data at scale.Its contributions include sequence-level and frame-level text descriptions alongside whole-body motion.
  • Evaluation: Experiments demonstrate annotation accuracy and benefits for 3D whole-body motion generation and mesh recovery.The reported benefits concern expressive, diverse, and natural motion generation.

2 Preliminary and Related Work

Prior text-motion datasets support several human-motion tasks but remain limited in scale, diversity, expressive coverage, and description granularity. Motion-X is positioned to address these gaps with whole-body motion and sequence- and frame-level text labels.

  • Motion-X: Motion-X provides indoor and outdoor whole-body motion and text annotations, including sequence-level semantics and frame-level pose descriptions.Table 1 compares these properties with existing text-motion datasets.
  • Existing datasets: Existing sequential motion-and-text benchmarks primarily support action recognition, human-object interaction, and motion generation.The related-work discussion reviews datasets developed for these three tasks.
  • Existing datasets: AMASS unified 15 optical marker-based motion-capture datasets through a common SMPL framework and parameterization.BABEL and HumanML3D expanded language annotations with sequence, subsequence, or multiple clip descriptions.
  • Remaining gaps: Existing text-motion datasets lack facial expressions and hand gestures, sufficient quantity, diverse motions and scenes, fine-grained descriptions, and long sequences.Motion-X is introduced to bridge these coverage and labeling gaps.

3 Motion-X Dataset

Motion-X combines data from eight existing datasets and online videos through a unified pipeline, producing whole-body annotations and text labels across diverse motions and scenes.

  • Dataset overview: 15.6M 3D whole-body SMPL-X annotations and 81.1K sequence-level semantic descriptions are provided alongside frame-level whole-body pose descriptions.All annotations are manually checked for quality.
  • Dataset overview: Figure 2 summarizes body, hand, and face motion diversity for each sub-dataset, highlighting hand movements and facial expressions beyond body-only motion data.Diversity is measured using averaged temporal standard deviations of body, hand, and face keypoints.
  • Annotation pipeline: The overall collection and annotation process comprises six steps, from LLM-designed motion prompts and video collection through preprocessing, motion capture, captioning, and manual inspection.The pipeline includes both motion and text annotation stages.
  • Dataset overview: Motion-X integrates 37K sequences from existing datasets with 32.5K monocular online videos covering general and domain-specific actions.The existing sources include multi-view, human-scene-interaction, action-recognition, and body-only motion-capture datasets.
  • Annotation pipeline: The whole-body motion-capture pipeline estimates 2D and 3D keypoints, optimizes local pose and global translation, and supports single- and multi-view videos.Multi-view-specific processing is represented by dashed lines in Figure 5.

4 Automatic Annotation Pipeline

The pipeline automatically captures expressive whole-body motion and generates semantic descriptions from videos. It combines hierarchical keypoint estimation, adaptive smoothing, progressive SMPL-X fitting, global optimization, and automated text annotation.

  • Pipeline overview: The annotation pipeline targets high-quality whole-body motion capture from massive videos using three specialized techniques.These are hierarchical whole-body keypoint estimation, score-guided adaptive temporal smoothing, and learning-based 3D human model fitting.
  • Keypoint estimation: Hierarchical keypoint estimation addresses hand and face detection errors caused by small regions, occlusion, and interaction.The method uses a customized hierarchical approach for 2D whole-body keypoint annotation.
  • Temporal smoothing: Confidence-adaptive smoothing enlarges the filter window for low-confidence keypoints, reducing outliers while preserving motion details.Unlike fixed-window smoothing, the method adapts its window size using keypoint confidence scores.
  • 3D fitting: Progressive learning-based fitting registers SMPL-X poses using image clues and motion priors, while physical constraints reduce interpenetration and foot skating.The fitting objective combines joint alignment with smoothness, penetration, and physical losses.
  • Global optimization: Global motion optimization jointly refines trajectories and camera poses to align estimated motion with video evidence.Its losses include 2D keypoint distance, trajectory consistency, trajectory regularization, and camera smoothness.
  • Text annotation: Whole-body descriptions combine facial emotion labels, body posecodes, and fine-grained hand-gesture descriptions into natural-language frame annotations.The resulting dataset contains 81.1K clips with 15.6M SMPL-X poses and corresponding pose and semantic text labels.

5 Experiment

Experiments evaluate annotation accuracy, motion generation, and mesh recovery using Motion-X. Results show stronger annotation performance, broader motion diversity, improved text-conditioned generation, and benefits for whole-body mesh recovery.

  • Evaluation scope: Experiments assess the annotation pipeline, text-driven motion generation, and whole-body human mesh recovery.The evaluation includes 2D keypoints, 3D SMPL-X fitting, generation benchmarks, cross-dataset comparisons, text-label ablations, and mesh recovery.
  • Annotation accuracy: Over 15% average precision improvement is reported for 2D keypoint annotation against four state-of-the-art methods on COCO-WholeBody.All methods use the same 256×192 input image size for comparison.
  • Annotation accuracy: Over 30% error reduction is reported for progressive 3D SMPL-X fitting against four open-sourced human mesh recovery methods.Evaluation uses MPVPE, PA-MPVPE, and PA-MPJPE on EHF.
  • Cross-dataset comparison: 13.174 versus 9.837 diversity is reported for Motion-X and HumanML3D ground-truth motions, respectively.The comparison evaluates MLD trained separately on the two datasets and supports broader motion types in Motion-X.
  • Text-driven generation: Models trained on Motion-X generate more accurate, semantically corresponding, expressive, diverse, and natural motions than the comparison shown against HumanML3D.The Motion-X-trained model also performs well after finetuning on the HumanML3D subset, including better performance than intra-data training on its test set.
  • Text-driven generation: Whole-body pose descriptions produce a 38% FID reduction when added to semantic labels in the text-label ablation.Adding face and body pose texts yields consistent improvements, while combining whole-body descriptions gives the largest reported reduction.

6 Conclusion

Motion-X is a large-scale 3D expressive whole-body motion dataset built with a systematic annotation pipeline. It improves coverage of expressive, diverse, and long motions while retaining documented limitations in annotation quality and evaluation metrics.

  • Motion-X contains 144.2 hours of whole-body motions, 81.1K annotated motions, and 15.6M frame-level pose descriptions.
  • The dataset addresses existing mocap limitations by covering expressive whole-body motions rather than primarily indoor body-only actions.
  • Experiments demonstrate annotation accuracy and benefits for expressive, diverse, natural motion generation and 3D whole-body human mesh recovery.
  • The markerless annotation pipeline is less accurate than multi-view marker-based capture, and existing evaluation metrics can disagree with visual results.
  • Motion-X provides large diversity in hand and face joints and many long motions exceeding 240 frames.

A.2 Processing of Each Sub-dataset

Motion-X unifies heterogeneous datasets and online videos through whole-body annotation, semantic labeling, pose captioning, and targeted motion augmentation. The resulting processing fills missing facial, hand, lower-body, and textual information across sources.

  • The unified framework gathers 81.1K sequences and supplies whole-body motion, semantic labels, and pose descriptions where source datasets lacked them.
  • AMASS body motions receive facial expressions from BAUM, semantic labels from HumanML3D, and pose descriptions from the whole-body captioning module.
  • IDEA400 expands NTU120 categories to 400 actions with self-contact, human-object contact, and expressive whole-body motions.
  • AIST, HAA500, HuMMan, GRAB, and EgoBody are reannotated or augmented to provide whole-body motion and complementary semantic or pose text.
  • Motion-X also collects 33K online monocular videos spanning real-life scenes, professional motions, and diverse appearances.
  • Lower-body augmentation selects compatible AMASS motion, while facial augmentation inserts interpolated BAUM expressions and emotion labels.

B More Annotation Visual Results

Visual evaluations support the proposed annotation pipeline and show that Motion-X captures expressive and diverse whole-body motion. Comparisons emphasize keypoint localization, SMPL-X fitting, and one-to-many motion styles.

  • 2D Keypoints: The hierarchical Transformer keypoint estimator performs better than Openpose and MediaPipe, especially for hand keypoint localization in challenging poses.
  • SMPL-X Parameters: The learning-based fitting method aligns SMPL-X parameters better than Hand4Whole and OSX, particularly on difficult poses.
  • SMPL-X Parameters: The fitting method is more robust than SMPLify-X for physically plausible poses under hard poses, low resolution, heavy occlusion, and depth ambiguity.
  • Motion Sequences: Six motion styles for the same dance-ballet semantic label illustrate one-to-many text-to-motion diversity.
  • Motion Sequences: Additional visualizations show variation in facial expressions, hand poses, and body motions across motion scenes.

C.1 Experiment Setup

Motion-X represents expressive whole-body motion with SMPL-X and evaluates text-motion generation using established alignment and diversity metrics. Its annotation pipeline combines hierarchical keypoint estimation, temporal smoothing, and learning-based fitting.

  • Motion Representation: SMPL-X represents each pose state with body rotations, hand rotations, jaw pose, facial expression, and global translation.
  • Motion Representation: The pose state includes θb for body rotations, θh for hand rotations, θf for jaw pose, ψ for facial expression, and r for global translation.
  • Evaluation Metrics: Evaluation uses FID, multimodality, diversity, R-precision, and multimodal distance with pretrained motion and text feature extractors.
  • Computational Costs: Annotating 1M frames takes about 72 hours using 8 NVIDIA A100 GPUs.

C.2 More Ablation Study

Motion-X addresses limited expressive coverage in prior datasets by evaluating whole-body motion, including hand and face movements, against established annotation and generation settings.

  • HumanML3D is limited in expressing rich hand and face motions because it contains body and minimal hand movements.
  • The annotation quality of Motion-X is compared with Hand4Whole and OSX, as well as the fitting method SMPLify-X.
  • Motion-X and HumanML3D are compared using MLD models trained on each dataset, evaluating diversity in generated and ground-truth motions.Diversity is measured with average temporal standard deviation of SMPL-X parameters and joint positions.
  • Motion-X produces higher temporal deviation, especially in hand and face parameters, indicating more substantial movements over time.
  • Text-conditioned motion generation aims to synthesize long-term, diverse, natural human motions from descriptions or other conditions.

E.1 Limitation

The paper identifies annotation quality and evaluation representation as limitations, while arguing that scalable noisy annotations remain useful for downstream motion and reconstruction tasks.

  • The markerless annotation pipeline is less accurate than multi-view marker-based motion capture, despite outperforming existing markerless methods.The authors report that a 30 mm PA-MPVPE error is acceptable for text-driven motion generation.
  • Improved annotation quality remains future work because higher-quality motion annotations would benefit both generation and human reconstruction.
  • Existing evaluation metrics are not always consistent with visual results, and SMPL-X parameters may not be optimal for expressive whole-body motion.
  • Despite inevitable errors, scalable automatic annotation can support large-scale learning from noisy labels for related tasks.
  • Motion-X data is distributed under CC BY-NC-SA, while users must consult original licenses for incorporated sub-datasets.
Loading 2307.00818v2…