Source-linked AI summary
We are More than Our Joints: Predicting how 3D Bodies Move
Yan Zhang, Michael J. Black, Siyu Tang
TL;DR
3D joint predictions leave body shape and pose underconstrained for realistic animation. MOJO instead predicts surface markers with a latent-frequency CVAE and recursively fits SMPL-X bodies; experiments report state-of-the-art results and realistic body animations, while inference speed and realism remain limitations.
Problem
Joint-based motion prediction leaves important body-pose degrees of freedom undefined and misses surface movement needed for realistic 3D animation.
Method
MOJO predicts mocap-style surface markers with a latent-frequency CVAE and recursively fits SMPL-X to predictions to maintain valid bodies over time.
Results
MOJO’s latent-DCT models outperform the state of the art, while recursive projection eliminates implausible body deformations and produces realistic 3D body movements.
Takeaways & Limitations
Surface markers and recursive projection enable diverse future motions represented as realistic, valid 3D bodies rather than unconstrained skeletons.
Takeaways & Limitations
Recursive projection slows inference, generated motion realism remains below ground truth, and performance on other or real marker settings is unexplored.
Abstract
from arXiv · showhide
A key step towards understanding human behavior is the prediction of 3D human motion. Successful solutions have many applications in human tracking, HCI, and graphics. Most previous work focuses on predicting a time series of future 3D joint locations given a sequence 3D joints from the past. This Euclidean formulation generally works better than predicting pose in terms of joint rotations. Body joint locations, however, do not fully constrain 3D human pose, leaving degrees of freedom undefined, making it hard to animate a realistic human from only the joints. Note that the 3D joints can be viewed as a sparse point cloud. Thus the problem of human motion prediction can be seen as point cloud prediction. With this observation, we instead predict a sparse set of locations on the body surface that correspond to motion capture markers. Given such markers, we fit a parametric body model to recover the 3D shape and pose of the person. These sparse surface markers also carry detailed information about human movement that is not present in the joints, increasing the naturalness of the predicted motions. Using the AMASS dataset, we train MOJO, which is a novel variational autoencoder that generates motions from latent frequencies. MOJO preserves the full temporal resolution of the input motion, and sampling from the latent frequencies explicitly introduces high-frequency components into the generated motion. We note that motion prediction methods accumulate errors over time, resulting in joints or markers that diverge from true human bodies. To address this, we fit SMPL-X to the predictions at each time step, projecting the solution back onto the space of valid bodies. These valid markers are then propagated in time. Experiments show that our method produces state-of-the-art results and realistic 3D body animations. The code for research purposes is at https://yz-cnsdqz.github.io/MOJO/MOJO.html
1 ETH Z¨urich, Switzerland 2Max Planck Institute for Intelligent Systems, T¨ubingen, Germany
MOJO represents motion with surface markers and predicts diverse future marker sequences with corresponding 3D bodies.
- MOJO predicts diverse future sequences of body-surface markers from past marker observations, alongside the 3D bodies they represent.
1. Introduction
Existing joint-based motion prediction leaves body shape, pose, and surface movement underconstrained, limiting realistic animation. MOJO addresses this with surface markers, latent-frequency generation, and recursive body projection.
- Joint locations leave important pose degrees of freedom undefined and cannot capture nuanced surface motion, limiting realistic body animation.
- MOJO represents motion with sparse mocap-style surface markers, which provide stronger constraints for recovering full 3D body shape and pose.
- Its CVAE preserves full temporal resolution and decomposes motion into DCT frequency bands to model fine-grained, high-frequency movement.
- Recursive projection fits SMPL-X to predicted markers after each step, returning error-accumulating predictions to the valid body manifold.
- Across AMASS, Human3.6M, and HumanEva-I, latent-DCT models outperform state of the art, while projection eliminates unrealistic body deformation.
2. Related Work
Related work covers deterministic and stochastic motion prediction, frequency-based analysis, and body representations, motivating MOJO’s latent-frequency and surface-marker design.
- Deterministic methods forecast a single future motion, while stochastic methods generate diverse plausible futures from one observed sequence.
- Prior work uses DCT and other frequency representations to process or regularize motion across temporal scales.
- Joint- and rotation-based body representations can suffer varying bone lengths, foot skating, or ambiguity in recovering complete shape and pose.
- MOJO combines latent DCT frequencies with surface markers to improve diversity and constrain realistic body recovery relative to skeleton-based representations.
3. Method
MOJO represents bodies as surface-marker sequences, generates motion through latent frequency bands, and recursively projects predictions onto valid SMPL-X bodies.
- 3.1. Preliminaries: SMPL-X provides MOJO’s parametric body mesh, with global translation, orientation, shape, body pose, and hand pose parameters.
- 3.2. Human Motion Representation: Surface markers are selected as SMPL-X vertices, coupling global translation with local pose and making body fitting easier than fitting from joints.
- 3.3. Motion Generator with Latent Frequencies: The CVAE preserves temporal resolution, transforms motion into DCT frequency bands, samples latent components, and reconstructs temporal motion with inverse DCT.
- 3.3. Motion Generator with Latent Frequencies: Sampling uses DLow transformations in the lowest L bands and standard-normal samples in higher bands, producing diverse sequences from frequency-specific latent variables.
- 3.4. Recursive Projection to the Valid Body Space: Recursive projection fits an SMPL-X mesh to each predicted marker set, using the previous fit for initialization and propagating valid body parameters through time.
4. Experiment
MOJO is evaluated on marker-based and skeleton-based motion prediction, with experiments isolating its marker layout, latent DCT space, and recursive projection components. The latent DCT space improves diversity while maintaining comparable prediction accuracy, although high-frequency modeling can reduce robustness to input changes.
- 4. Experiment: The experiments evaluate MOJO on AMASS, Human3.6M, and HumanEva-I, covering marker-based and skeleton-based stochastic motion prediction.AMASS uses CMU and MPI HDM05 for training and ACCAD and BMLhandball for testing; additional comparisons follow prior settings on Human3.6M and HumanEva-I.
- 4.2. Baselines: MOJO predicts 41 surface markers, while MOJO-DCT removes the latent DCT space and MOJOproj removes recursive projection.Suffixes can be combined to denote models missing both components.
- 4.3.1 Metrics: The evaluation measures diversity, ADE/FDE accuracy, robustness under input changes, and motion-frequency detail through FSE.Higher FSE indicates more motion detail, but it is interpreted jointly with prediction accuracy because high frequency can also indicate noise.
- 4.3.2 Results: The latent DCT space improves diversity by a large margin while keeping prediction accuracies comparable to the baseline.MMADE and MMFDE are slightly inferior, plausibly because high-frequency components make predictions sensitive to small input changes.
- 4.3.2 Results: Sampling only from N(0, I) across frequency bands performs much worse than sampling with DLow in low-frequency bands.The results support concentrating informative sampling in low-frequency bands rather than treating all bands equally.
4.4. Evaluation of Motion Realism
Motion realism is evaluated through body deformation, foot skating, perceptual quality, and visualized generated motions. Recursive projection preserves body shape by fitting SMPL-X at each step, while DCT modeling improves fine-grained movement and foot-skating behavior with dataset-dependent perceptual effects.
- 4.4. Evaluation of Motion Realism: Recursive projection preserves body shape by construction and keeps it close to the ground truth.Without projection, body-part shapes can deform; Table 3 measures head, upper-torso, and lower-torso deformation in millimeters.
- 4.4. Evaluation of Motion Realism: MOJO generates finer-grained body movements, and recursive projection converts predicted markers into valid 3D body meshes.The projection fits SMPL-X to predicted markers at each frame to maintain valid bodies over time.
- 4.4. Evaluation of Motion Realism: Figure 5 compares input and generated bodies, distinguishing MOJO-DCT from MOJO through solid and dashed image borders.Gray-green bodies denote inputs and red bodies denote generated motions.
- 4.4. Evaluation of Motion Realism: DCT produces fewer foot-skating artifacts, indicating that its high-frequency components better model foot movements.Perceptual results are dataset-dependent: MOJO is slightly worse than MOJO-DCT on ACCAD but better on BMLhandball.
4.5. Comparison between Body Representations
The comparison evaluates joint- and marker-based body representations, including recursive projection, using fitted-body joint metrics and bone deformation. More markers improve performance, while projection removes bone deformation and prevents false poses.
- The evaluation compares joint-based prediction, marker placements, and their combinations using fitted-body joints, diversity, prediction accuracy, and bone-deformation metrics.
- Recursive projection completely removes bone deformation and avoids false poses, although the unprojected variant is slightly better on other measures.The traditional joint-based pipeline can produce skeletons that cannot be fit to a body because of unrealistic bone lengths.
- Using 67 SSM2 markers significantly improves performance across the evaluated measures, showing that marker distribution and density matter.
- Joint locations are inferred from markers by commercial motion-capture systems, so adding joints to markers does not provide independent information or improve performance.
5. Conclusion
MOJO predicts diverse 3D body motions with surface markers, latent-frequency motion generation, and recursive SMPL-X projection. The method produces realistic bodies and eliminates implausible deformations relative to a joint-based pipeline, but inference is slower and realism remains below ground truth.
- MOJO predicts diverse plausible 3D body motions and uses surface markers, latent frequencies, and recursive projection to improve body validity and realism.
- Compared with a joint-based pipeline, recursive SMPL-X fitting keeps predicted bodies valid over time and eliminates implausible body deformations.
- The recursive projection scheme slows inference, and generated motion realism remains below ground truth.
**Appendix**
MOJO uses SMPL-X for its demonstration, but the recursive fitting procedure can also be implemented with other parametric 3D body models.
- MOJO is demonstrated with SMPL-X, but any other parametric 3D body model could be used by implementing recursive pose fitting to observed markers.The fitting is described as a straightforward optimization problem, and SMPL-X enables use of the AMASS dataset.
A. More Method Details
The appendix specifies the marker placements and motion-generator architectures used in the experiments. It distinguishes CMU and SSM2 marker settings and documents the MOJO, MOJO-DCT, and VAE+DCT variants.
- The experiments use CMU placement with 41 markers and SSM2 placement with 67 markers, illustrated from multiple body views.
- The appendix illustrates the CVAE architectures for MOJO, MOJO-DCT, and VAE+DCT, noting that recursive projection is added during testing.
- MOJO-DCT-proj encodes motion as a single feature vector, whereas the MOJO architecture uses frequency components.
B. More Experimental Details
The experiments examine implementation choices, evaluation measures, user-study setup, and how MOJO behaves under altered sequence lengths. They also document failure cases for an original DLow setting applied to body markers.
- AMASS sequence canonicalization: The AMASS preprocessing trims motions to 480 frames, downsamples them to 15 fps, and uses 15 input frames to predict 45 future frames.World coordinates are reset using the SMPL-X body mesh in the first frame.
- Evaluation measures: MMADE and MMFDE evaluate prediction accuracy across multiple plausible outputs, while the deformation score measures variation in pairwise marker distances within rigid body parts.The reported body parts are the head, upper torso, and lower torso.
- User study interface: The user study used a six-point Likert scale through AMT, with each video evaluated by three subjects.The study interface is shown in Figure S3.
- Performance of the original VAE setting in DLow: 81.10 diversity, ADE=2.79, FDE=4.71, MMADE=2.81, MMFDE=4.71, and FSE=0.0031 were obtained by the original DLow VAE setting, but its predicted markers were invalid.The setting produced higher diversity alongside considerably worse accuracy than the intended marker-based approach.
- Influence of the given frames: Shorter predicted sequences and longer input sequences consistently increased accuracy while decreasing diversity, making MOJO more confident and deterministic.Longer sequences can be generated by sliding-window recursion using the pretrained MOJO model.
C. More Analysis on the Latent DCT Space
The latent DCT analysis shows that frequency-band allocation controls diversity while preserving accuracy, and that most latent information concentrates in low frequencies. Marker experiments further show different preferred latent dimensions for MOJO variants.
- Latent dimensions and frequency bands: Applying DLow to more frequency bands consistently increases diversity while keeping motion-prediction accuracy stable.This result is reported for Human3.6M across latent-dimension and frequency-band settings.
- Visualization of the latent DCT space: Most information in the latent DCT space is concentrated in low-frequency bands, explaining why performance saturates when DLow is applied to additional bands.The conclusion holds with and without fully connected-layer bias.
- Visualization of the latent DCT space: Removing fully connected-layer bias has a trivial effect on performance, so the experiments retain those biases as trainable parameters.The comparison is made against the corresponding results in Table 2.