Source-linked AI summary
AMASS: Archive of Motion Capture as Surface Shapes
Naureen Mahmood, Nima Ghorbani, Nikolaus F. Troje, Gerard Pons-Moll, Michael J. Black
TL;DR
Existing human-motion datasets are limited in size and variety, and their differing body parameterizations make them difficult to unify for learning models of motion. The paper introduces MoSh++ and AMASS, which represent marker-based captures as realistic SMPL-based meshes with body, soft-tissue, and hand motion. AMASS unifies 15 datasets into a publicly available collection containing 42 hours of motion, 346 subjects, and 11,451 motions.
Problem
Existing mocap datasets are limited in size and complexity, while differing markersets and body formulations make them difficult to combine for deep learning.
Method
MoSh++ fits a combined SMPL-H and DMPL body model to mocap markers to recover body shape, pose, soft-tissue deformations, and hand pose in a unified representation.
Results
AMASS unifies 15 mocap datasets into a consistent format containing 42 hours of data, 346 subjects, and 11,451 motions.
Takeaways & Limitations
The unified representation preserves body shape and surface information while supporting rigged meshes, animation, visualization, and large-scale training-data generation.
Takeaways & Limitations
MoSh++ is not currently real-time, and the evaluation lacks ground-truth data for hand-articulation accuracy.
Abstract
from arXiv · showhide
Large datasets are the cornerstone of recent advances in computer vision using deep learning. In contrast, existing human motion capture (mocap) datasets are small and the motions limited, hampering progress on learning models of human motion. While there are many different datasets available, they each use a different parameterization of the body, making it difficult to integrate them into a single meta dataset. To address this, we introduce AMASS, a large and varied database of human motion that unifies 15 different optical marker-based mocap datasets by representing them within a common framework and parameterization. We achieve this using a new method, MoSh++, that converts mocap data into realistic 3D human meshes represented by a rigged body model; here we use SMPL [doi:10.1145/2816795.2818013], which is widely used and provides a standard skeletal representation as well as a fully rigged surface mesh. The method works for arbitrary marker sets, while recovering soft-tissue dynamics and realistic hand motion. We evaluate MoSh++ and tune its hyperparameters using a new dataset of 4D body scans that are jointly recorded with marker-based mocap. The consistent representation of AMASS makes it readily useful for animation, visualization, and generating training data for deep learning. Our dataset is significantly richer than previous human motion collections, having more than 40 hours of motion data, spanning over 300 subjects, more than 11,000 motions, and will be publicly available to the research community.
1. Introduction
The paper develops MoSh++ to recover realistic body shape, pose, soft-tissue dynamics, and hand motion from mocap markers, enabling AMASS, a large unified human-motion database. AMASS combines varied datasets in a consistent SMPL-based representation while preserving body shape and surface information.
- MoSh++: MoSh++ fits the SMPL/DMPL surface model to observed markers, recovering body shape, pose, and soft-tissue dynamics while providing a rigged skeleton for animation.Unlike conventional methods that filter surface motion as noise, the approach retains body shape and surface dynamics.
- Motivation: Existing mocap datasets are too limited in size and complexity for deep learning and difficult to combine because they use varying markersets and procedures.MoSh++ addresses both the data-scale problem and the challenge of producing a coherent formulation across datasets.
- AMASS: AMASS combines 15 motion-capture datasets with marker sets ranging from 37 to 91 markers into a single SMPL-based format.The representation is independent of the marker set while retaining original marker-data richness, including 3D body shape.
- AMASS: 42 hours, 346 subjects, and 11,451 motions make AMASS a large unified database of human motion.The dataset is intended to support applications requiring large amounts of training data.
- AMASS: Each AMASS frame contains SMPL shape, DMPL soft-tissue, and full SMPL pose parameters, including hand articulations.Users can ignore shape and soft-tissue parameters or normalize bodies to a common shape when their application requires only pose.
2. Related Work
Prior mocap reconstruction and dataset-merging methods use incompatible body representations or discard surface detail, limiting unified motion collections. This work instead fits markers directly to a full 3D body surface that supports skeleton derivation, marker generation, rendering, and synthetic training data.
- Motion Capture Datasets: Different skeletal models and pre-specified marker sets make existing marker-based recordings difficult to unify.Existing skeleton-fitting methods also lose rich surface information during reconstruction.
- Surface Models from Markers: Earlier surface-reconstruction methods are limited by fixed poses, scan requirements, incompatible body models, or incomplete motion capture.MoSh removes the need for individual dense scans but does not capture real soft-tissue dynamics or hands, while its BlendSCAPE formulation is incompatible with standard graphics packages.
- Motion Capture Datasets: Previous dataset aggregation normalizes subject size and supports pose modeling from joint locations but discards body-shape information.The section contrasts this skeleton-centered representation with retaining the full body surface.
- AMASS Approach: The proposed marker-first approach recovers the full 3D body surface without losing generality because any desired skeleton or marker set can be derived from it.This directly addresses the representation mismatch between existing mocap corpora.
- Applications: A body model also enables texturing and rendering virtual bodies in different scenes, including synthetic training-data generation for computer vision.These capabilities extend the use of reconstructed motion beyond skeleton-based pose and motion modeling.
3. Technical Approach
MoSh++ extends MoSh into a more distributable and expressive fitting pipeline by combining SMPL-based body modeling with hand and soft-tissue estimation, marker-occlusion handling, and SSM-tuned optimization. The resulting model uses a compact parameterization while retaining a standard rigged mesh and improving fitting accuracy over MoSh.
- 3.1. The Body Model: MoSh++ replaces BlendSCAPE with SMPL, providing a standard rigged mesh and skeleton for distribution and graphics use.The model is combined with SMPL-H and DMPL to represent hand pose and soft-tissue deformation.
- 3.1. The Body Model: SMPL-H and DMPL jointly model hand pose and soft-tissue deformations within the unified body representation.MoSh++ optimizes hand pose when hand markers are present and uses average MANO hand pose otherwise.
- 3.1. The Body Model: 16 shape and 8 dynamics components give MoSh++ better accuracy than MoSh using 100 shape components without overfitting mocap markers.The coefficient counts are selected using the SSM dataset.
- 3.2. Model Fitting: MoSh++ fits sparse markers in two stages: first estimating latent markers, shape, and poses, then optimizing per-frame pose with temporally regularized soft-tissue dynamics.Stage I uses 12 randomly chosen frames and excludes soft-tissue deformations; Stage II holds latent markers and shape constant over time.
- 3.2. Model Fitting: The fitting objective matches simulated to observed markers while regularizing shape, body pose, hand pose, marker locations, and soft-tissue coefficients.The latent-marker surface distance is prescribed as an average d = 9.5mm, and soft-tissue coefficients receive a Mahalanobis prior.
- 3.2. Model Fitting: A missing-marker factor raises the pose-prior weight from q = 1.0 with all markers visible to up to q = 3.5 when all markers are missing.This adaptive weighting is intended to produce more plausible poses under severe occlusion; the objective weights are tuned on SSM and adjusted for varying marker counts.
4. Evaluation
MoSh++ is evaluated against synchronized 4D body scans for shape, pose, and soft-tissue reconstruction, with hyperparameters selected on the SSM dataset. It improves reconstruction accuracy over MoSh, while hand-articulation accuracy lacks ground-truth evaluation.
- Evaluation setup: Hyperparameters are tuned by cross-validation on SSM, with line search used instead of exponentially growing grid search.The objective is to minimize reconstruction error on validation data while holding other parameters fixed during each one-parameter search.
- Evaluation setup: SSM synchronously records 67-marker mocap and 4D scans from three subjects performing 30 motions for ground-truth surface evaluation.MoSh++ is evaluated with both the full 67-marker set and a standard 46-marker subset using scan-to-model distances.
- Shape estimation: 7.4mm shape error versus 12.1mm for MoSh shows more accurate MoSh++ shape estimation on SSM with the standard 46-markerset.SSM selects 16 shape and 8 dynamic coefficients; adding more coefficients lowers marker error but overfits the markers and worsens ground-truth shape error.
- Pose and soft-tissue estimation: 8.1mm pose error versus 10.5mm for MoSh shows more accurate MoSh++ pose estimation without soft-tissue motion estimation.With dynamics terms enabled, MoSh++ also outperforms MoSh, achieving 7.3mm versus 10.24mm.
- Hand articulation: Hand-articulation accuracy is not quantitatively evaluated because the study lacks ground-truth hand data.Qualitative results show more realistic hand poses when hand capture is included.
5. AMASS Dataset
AMASS combines 15 mocap datasets by mapping heterogeneous marker data into common SMPL pose, shape, and soft-tissue parameters. The resulting public dataset contains 346 subjects, 11,451 motions, and 42 hours of recordings.
- Dataset construction: 15 mocap datasets are mapped into a common SMPL pose, shape, and soft-tissue representation using MoSh++.The source datasets use different marker counts and placements, and problematic mocap results are manually corrected or held out.
- Dataset scale: AMASS contains 346 subjects, 11,451 motions, and 42 hours of recordings.The dataset is described as the largest public dataset of human shape and pose and is made available to the research community.
6. Future Work and Conclusions
The paper identifies extensions for MoSh++ and concludes that it enables a unified AMASS dataset containing pose, shape, and soft-tissue motion.
- Future Work: MoSh++ can be extended to facial mocap markers using the FLAME model, and future work also targets missing markers and automatic marker labeling.The authors also note that AMASS could train models to denoise mocap data.
- Future Work: MoSh++ is not currently real-time, although the authors suggest parallel SMPL implementations could significantly improve runtime.This is identified as an implementation boundary rather than an accuracy limitation.
- Conclusions: 42 hours of mocap data are mapped into SMPL parameters in a unified AMASS format.Table 1 describes the constituent datasets and their conversion through MoSh++.
- Conclusions: AMASS includes SMPL pose with articulated hands, body shape, and soft-tissue motion, and the authors plan to incorporate more contributed mocap data.The conclusion presents MoSh++ as an extension that unifies marker-based recordings.
2. Optimization and Runtime
MoSh++ uses a multi-stage optimization pipeline whose runtime depends on motion characteristics, while the model-selection experiments determine suitable shape and dynamics component counts.
- Runtime: Stage I takes about 25 min/motion sequence, while Stage II takes about 0.5 sec/frame without dynamics and about 2 sec/frame with dynamics.These runtimes were measured on CPU for SSM sequences and depend heavily on frame rate and pose variation.
- Optimization: Varying the numbers of SMPL shape and DMPL dynamic components identifies an optimal configuration for reconstruction.Figure 9 evaluates mesh reconstruction errors while changing β and φ component counts.
3. Data Collection
The SSM dataset synchronizes optical motion capture with high-resolution 4D body scanning to provide jointly recorded markers and scans.
- Data Collection: The motion capture system and 4D scanning system are synchronized and calibrated together.This setup supports paired marker and scan recordings.
- Data Collection: SSM uses 24 OptiTrack Prime 17W cameras and fits each subject with 67 reflective markers.Subjects wore minimal clothing, with markers placed directly on skin wherever possible.
- Data Collection: The 4D scanner captures high-resolution 3D scans at 60 frames per second using stereo cameras, color cameras, speckle projectors, and LED panels.The system includes 22 stereo-camera pairs, 22 color cameras, and 34 speckle projectors.
4. Model Size
The model-size experiments select the numbers of shape and dynamic components that minimize validation error while producing natural-looking soft-tissue deformations.
- Model Size: 16 SMPL shape components and 8 DMPL dynamic components best minimize held-out validation error and produce natural-looking soft-tissue deformations.The experiments vary both component counts to choose the configuration used for shape and soft-tissue capture.
5. Diversity and Quality
AMASS captures broad variation in human motions and body shapes while reconstructing high-quality surface deformations from mocap markers.
- AMASS captures variations in human motions.
- The reconstructed body surface deformations are illustrated as high quality.
- The dataset spans a wide range of body shapes and motions.