Source-linked AI summary
Learning Motion Priors for 4D Human Body Capture in 3D Scenes
Siwei Zhang, Yan Zhang, Federica Bogo, Marc Pollefeys, Siyu Tang
TL;DR
Monocular capture of people interacting with 3D scenes remains challenging because occlusions and partial views complicate realistic motion recovery. LEMO combines AMASS-learned motion priors, contact friction, and per-instance self-supervised infilling in a multi-stage pipeline. The authors report smooth, accurate, occlusion-robust motion with physically plausible interactions.
Problem
Monocular RGB-D capture of human-scene interactions remains challenging under occlusions and partial views, while existing recovered motions can exhibit jitters and skating.
Source: §1 · pp. 1–2Method
LEMO combines AMASS-learned motion-smoothness and contact-aware infilling priors with contact friction in a multi-stage optimization pipeline.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors report effective motion-prior use for monocular RGB-D capture and improvements across PROX and AMASS infilling evaluations.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recovering high-quality 3D human motion in complex scenes from monocular videos is important for many applications, ranging from AR/VR to robotics. However, capturing realistic human-scene interactions, while dealing with occlusions and partial views, is challenging; current approaches are still far from achieving compelling results. We address this problem by proposing LEMO: LEarning human MOtion priors for 4D human body capture. By leveraging the large-scale motion capture dataset AMASS, we introduce a novel motion smoothness prior, which strongly reduces the jitters exhibited by poses recovered over a sequence. Furthermore, to handle contacts and occlusions occurring frequently in body-scene interactions, we design a contact friction term and a contact-aware motion infiller obtained via per-instance self-supervised training. To prove the effectiveness of the proposed motion priors, we combine them into a novel pipeline for 4D human body capture in 3D scenes. With our pipeline, we demonstrate high-quality 4D human body capture, reconstructing smooth motions and physically plausible body-scene interactions. The code and data are available at https://sanweiliti.github.io/LEMO/LEMO.html.
1. Introduction
Everyday human-motion capture from monocular RGB-D video is attractive but remains less compelling than marker-based motion capture, with jitters, skating, occlusions, and partial views. LEMO combines learned motion priors, contact friction, and a multi-stage optimization pipeline to recover smoother, more plausible scene interactions.
Motivation
- The pipeline targets realistic motion capture in everyday scenes, where occlusions and partial views challenge reconstruction.The paper motivates this need by noting that expert knowledge and expensive setups make marker-based optical mocap impractical for everyday environments.Source: §1 · pp. 1–2
- Monocular RGB-D methods such as PROX offer accessible everyday capture, but recovered motions show severe skating and jitters and remain below commercial mocap quality.Multiview setups require controlled calibration and synchronization, while IMUs can suffer heading drift and interference; both may still produce jittery motion.Source: §1 · pp. 1–2
Contributions
- The motion-smoothness prior learns holistic full-body dynamics from AMASS in a latent space, using surface markers rather than joints to represent motion.The authors report improved reconstruction quality on PROX and motion naturalness on IMU-based 3DPW.Source: §1 · p. 2
- The AMASS-trained infiller predicts occluded motion and foot-contact states jointly, then adapts to each test sequence using visible body parts for self-supervised fine-tuning.Combined with contact friction, the authors report natural motion, proper foot-ground interaction, and improvements on AMASS and PROX.Source: §1 · p. 2
- LEMO combines a marker-based motion-smoothness prior, a per-instance self-supervised contact-aware infiller, and contact friction in a multi-stage capture pipeline.The authors report broad applicability of the priors and efficacy for monocular RGB-D capture in 3D scenes.Source: §1 · p. 2
2. Related Work
Prior work addresses temporal consistency, body reconstruction, and person-scene interaction through different representations and methods. LEMO combines learned motion priors with scene interaction modeling for realistic motion recovery from RGB(D) video.
Human motion recovery from RGB(D) sequences
- RGB(D) motion-recovery methods extend per-frame body reconstruction to sequences, but joint-based representations do not adequately model body shape or body-scene interactions.Other approaches use parametric body models with multi-view or monocular RGB(D) input, and some add temporal encoders or motion refinement.Source: §2 · pp. 2–3
Person-scene interaction
- Person-scene interaction methods range from scene constraints and physics-inspired terms to PROX’s detailed contact and collision modeling for SMPL-X bodies in 3D scenes.Other work uses physics engines or scene semantic labels, while Zhang et al. generate plausible human meshes in scenes without people.Source: §2 · p. 3
Human motion priors; Ours versus others
- LEMO learns smoothness in a latent space from AMASS and uses body markers with jointly predicted contact states, distinguishing its priors from joint-space smoothness and earlier infillers.The authors report that the approach produces smooth motions without degrading per-frame pose accuracy and consistently outperforms [28] for motion recovery and foot skating.Source: §2 · p. 3
3. Method
LEMO fits SMPL-X to RGB-D sequences in three stages: per-frame initialization, temporally smooth fitting with contact friction, and occlusion recovery using a motion infiller. The pipeline combines learned motion priors with scene constraints to recover plausible motion despite occlusions.
3.1. Overview
- The three-stage pipeline initializes SMPL-X per frame, adds a smoothness prior and contact friction for temporal fitting, then uses an infiller to recover plausible occluded motion and alleviate foot skating.Stage 1 provides initialization but does not ensure smoothness or handle body-scene occlusions.Source: §3.1 · p. 3
3.2. Per-frame Fitting
- Stage 1 fits SMPL-X independently to each RGB-D frame using image-joint, depth, body-prior, contact, and collision terms.The fitting objective follows PROX [19].Source: §3.2 · p. 3
3.3. Temporally Smooth Motion
- The Stage 2 smoothness prior trains an autoencoder on AMASS marker motion and penalizes changes in its latent sequence to regularize fitted motion over time.The marker representation and convolutional encoder are designed to capture long-range correlations and full-body dynamics.Source: §3.3 · p. 4
- Contact friction uses 194 foot and 113 gluteus vertices; below 0.01 m from the scene mesh, contacted-vertex normal velocity must be non-negative and tangential velocity small.The contact term based only on body-scene proximity cannot prevent skating artifacts.Source: §3.3 · pp. 4–5
3.4. Recovering Motion under Occlusion
- The motion infiller takes global motion as input and predicts local marker motion together with foot-ground contact states.During training, local motion is masked while global trajectory is assumed known; the network is trained to reconstruct local motion and contact labels.Source: Figure 3 · p. 5, §3.4 · p. 5
- At test time, the infiller is fine-tuned per sequence using visible markers, then its predictions are incorporated into Stage 3 fitting to recover occluded motion.The self-supervised tuning exploits visible body parts and is reported to improve prediction accuracy for visible and invisible parts.Source: §3.4 · p. 5
4. Experiments
The experiments evaluate motion smoothness and infilling priors on PROX, 3DPW, and AMASS. Results show improved motion naturalness and reconstruction accuracy, with contact prediction and per-instance fine-tuning benefiting infilling.
4.1. Datasets
- AMASS supplies training motions; PROX tests capture in indoor scenes, while 3DPW tests local motions because its moving camera limits global configuration accuracy.The experiments use 205 PROX clips and 300 3DPW clips for evaluation.Source: §4.1 · p. 6
4.2. Evaluation of Motion Smoothness Prior
- The smoothness prior is compared with DCT, velocity, and acceleration baselines on PROX and 3DPW, using the same fitting objective framework.The proposed objective adds the smoothness term to the data and pose-prior terms.Source: §4.2 · p. 6
4.2.1 Metrics
- Smoothness evaluation uses 2D joint error on PROX, MPJPE/PVE on 3DPW, acceleration-distribution PSKL, and PROX non-collision scores.Lower accuracy errors and PSKL are better; higher non-collision scores indicate less interpenetration.Source: §4.2.1 · pp. 6–7
4.2.2 Results
- On PROX and 3DPW, the smoothness prior achieves the lowest PSKL scores in both directions while preserving pose accuracy.The results also report comparable PROX non-collision scores and small MPJPE/PVE on 3DPW; latent sequences become smoother over time after fitting.Source: §4.2.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
4.3. Evaluation of Motion Infilling Prior
- For AMASS infilling evaluation, the study masks lower-body markers and contact labels, and also evaluates the prior on PROX.The AMASS test uses 130 sequences; PROX evaluation includes 2D joint accuracy, PSKL, and non-collision score.Source: §4.3 · pp. 7–8
4.3.1 Metrics
- Infilling accuracy is measured with full-body and masked-part MPJPE/MPMPE, PVE, and a foot-skating ratio based on heel-marker motion.Skating is counted when both heel markers exceed 10 cm/s while remaining below 10 cm in height.Source: §4.3.1 · p. 8
4.3.2 Results
- On AMASS, the proposed infiller outperforms its baseline across all metrics, including reconstruction errors for masked lower-body joints and markers.The evaluation compares full-body and masked-part accuracy across body markers, joints, and vertices.Source: §4.3.2 · p. 8, Table 3 · p. 8
- Predicted contacts alleviate foot skating, self-supervised fine-tuning improves whole-body and occluded-part fits, and infilling performs poorly on jittered input.The authors report that the infiller overfits noisy input during test-time fine-tuning when Stage 2 and Stage 3 are swapped.Source: §4.3.2 · p. 8
5. Conclusion
LEMO combines motion priors learned from high-quality motion-capture data with a multi-stage capture pipeline to reconstruct smooth, occlusion-robust motion and physically plausible human-scene interactions. The approach remains limited to intuitive physics terms, with broader physics-inspired motion modeling left as a research direction.
5. Conclusion
- The multi-stage pipeline combines learned motion priors with a physics-inspired contact-friction term for smooth, accurate, occlusion-robust motion and physically plausible interactions.The priors learn full-body dynamics and recover body parts occluded from the camera view.Source: §5 · p. 8
- The current pipeline incorporates intuitive physics terms such as contact, interpenetration, and friction, while broader physics-inspired motion modeling remains a research direction.The paper describes combining such modeling with data-driven motion priors as promising and challenging.Source: §5 · p. 8
A. Architecture Details
The motion-prior networks share five-block convolutional encoders and five-block deconvolutional decoders, but pooling is used only for motion infilling. The smoothness prior preserves full spatial-temporal resolution in its latent space.
A. Architecture Details
- Both motion-prior networks use five encoder blocks and five decoder blocks, with convolutional and deconvolutional layers, respectively.Each encoder block contains two conv3x3 layers with LeakyReLU activations; each decoder block contains two deconv3x3 layers with LeakyReLU activations.Source: A. Architecture Details · p. 12, Figure S1 · p. 12
- MaxPooling appears only in the motion infilling prior, not the motion smoothness prior.The smoothness prior avoids downsampling so its latent space retains the input motion's full spatial-temporal resolution and details.Source: A. Architecture Details · p. 12, Figure S1 · p. 12
- The encoder feature channels are [32] for smoothness and [32, 64, 128, 256, 256] for infilling.Source: A. Architecture Details · p. 12
B.1. Implementation Details
The priors are trained with ADAM at a 1e-4 learning rate, while the PROX optimization pipeline uses ADAM at 5e-3. Each 100-frame clip receives 900 optimization steps in both Stage 2 and Stage 3.
B.1. Implementation Details
- The proposed PROX pipeline uses ADAM at a 5e-3 learning rate, with 900 optimization steps per 100-frame clip in both Stage 2 and Stage 3.Source: §B.1 · p. 12
- Both motion priors are trained with ADAM at a 1e-4 learning rate, using 150 epochs for smoothness and 900 epochs for infilling.The respective batch sizes are 60 and 120.Source: §B.1 · p. 12
- Training and optimization use PyTorch 1.4.0 on a single TITAN RTX GPU.Source: §B.1 · p. 12
B.2. Marker Placement
The method represents the body with 81 surface markers: 67 body markers and 14 additional face and finger markers. The additional markers are used only by the motion smoothness prior.
B.2. Marker Placement
- The marker set contains 67 SMPL-X body-surface markers following the SSM2 setting and 14 additional markers on the face and fingers.Source: §B.2 · p. 12
- The 14 face and finger markers are used only for the motion smoothness prior, which also models hand motions and facial expressions.The infilling prior focuses mainly on lower-body motion infilling.Source: §B.2 · p. 12
- Surface markers are used because they better model degrees of freedom and incorporate body-shape information than body joints.Source: §B.2 · p. 12
B.3. Evaluation Details
The evaluation defines PSKL as a motion-distribution distance and assesses motion infilling through per-frame and temporal fitting. It also tests stage order, finding that placing motion infilling before temporal recovery degrades naturalness and pose accuracy.
B.3. Evaluation Details
- PSKL measures dataset distribution distance using power spectra of per-frame x, y, and z accelerations, averaged across feature dimensions.The feature count is F = 3M, where M is the number of body markers or joints; KL divergence is evaluated in both directions because it is asymmetric.Source: §B.3 · pp. 12–13
- The motion-prior marker layout follows the AMASS setting and adds 14 markers on the face and fingers.The figure shows front, back, and top views of the marker placement on SMPL-X.Source: Figure S2 · p. 13
- Motion-infilling evaluation first fits each frame, using the infilled-marker L1 loss and body- and hand-pose priors, to initialize temporal fitting.Temporal fitting adds the smoothness prior and a foot term; the proposed method uses predicted foot-ground contact states to penalize foot-vertex velocity.Source: §B.3 · p. 13
- The foot-contact heuristic penalizes foot-joint velocity when the joint is within 10 cm of the ground, using a 10 cm/s velocity threshold.This is the stated threshold-based foot term used in the evaluation details.Source: §B.3 · p. 13
- Swapping Stages 2 and 3 degrades motion naturalness (PSKL) and pose accuracy (2DJE).Table S1 reports the PROX ablation; PSKL is computed on markers and joints, in both PROX-to-AMASS and reverse directions.Source: §B.3 · p. 14, Table S1 · p. 14
C. Comparison with Regression-based Methods
On PROX, the paper compares its motion smoothness prior with regression-based denoising methods trained on AMASS with synthesized Gaussian input noise. The prior achieves higher joint accuracy and more realistic motions, and the authors note its compatibility with scene constraints in an optimization framework.
C. Comparison with Regression-based Methods
- The comparison trains regression-based denoisers on AMASS using Gaussian noise added to their input motions.These methods directly output smooth joint motions from noisy motion input.Source: C. Comparison with Regression-based… · p. 14
- On PROX, the proposed smoothness prior achieves higher joint accuracy and more realistic motion PSKL scores than regression-based denoising methods.The regressors are trained with synthesized noise, while the proposed prior is trained on clean motions and works on both IMU-captured 3DPW and Kinect-captured PROX.Source: C. Comparison with Regression-based… · p. 14
- The authors report that regression-based methods can generalize poorly across noise distributions and often produce inaccurate global reconstructions.They also note that it is unclear how to incorporate 3D scene constraints directly into the regressors, unlike the unified optimization framework used with their priors and interaction constraints.Source: C. Comparison with Regression-based… · p. 14