PaperlayerView exact revision on arXiv

Source-linked AI summary

Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Model

Yuming Du, Robin Kips, Albert Pumarola, Sebastian Starke, Ali Thabet, Artsiom Sanakoyeu

arXiv:2304.08577v1cs.CV

TL;DR

Standard HMDs provide only sparse head-and-hand tracking, leaving full-body and especially lower-body motion to be synthesized. The paper introduces AGRoL, a lightweight MLP-based conditional diffusion model with a motion-specific conditioning strategy. It reports accurate, smooth, state-of-the-art motion generation with real-time inference for online AR/VR tracking.

  • Problem

    Standard HMDs typically track only the head and hands, leaving accurate full-body motion—especially lower-body movement—to be reconstructed from sparse signals.

  • Method

    AGRoL is a lightweight MLP-based conditional diffusion model that uses block-wise timestep embedding for full-body motion synthesis from sparse IMU tracking.

  • Results

    AGRoL achieves state-of-the-art full-body motion synthesis while generating realistic, smooth motion at real-time inference speed.

  • Takeaways & Limitations

    The compact diffusion architecture is suitable for online AR/VR applications using only sparse tracking signals.

  • Takeaways & Limitations

    The authors identify occasional floor-penetration artifacts as a limitation shared by AGRoL and related approaches.

Abstract

from arXiv · show

With the recent surge in popularity of AR/VR applications, realistic and accurate control of 3D full-body avatars has become a highly demanded feature. A particular challenge is that only a sparse tracking signal is available from standalone HMDs (Head Mounted Devices), often limited to tracking the user's head and wrists. While this signal is resourceful for reconstructing the upper body motion, the lower body is not tracked and must be synthesized from the limited information provided by the upper body joints. In this paper, we present AGRoL, a novel conditional diffusion model specifically designed to track full bodies given sparse upper-body tracking signals. Our model is based on a simple multi-layer perceptron (MLP) architecture and a novel conditioning scheme for motion data. It can predict accurate and smooth full-body motion, particularly the challenging lower body movement. Unlike common diffusion architectures, our compact architecture can run in real-time, making it suitable for online body-tracking applications. We train and evaluate our model on AMASS motion capture dataset, and demonstrate that our approach outperforms state-of-the-art methods in generated motion accuracy and smoothness. We further justify our design choices through extensive experiments and ablation studies.

1. Introduction

AGRoL addresses full-body avatar tracking from the three sparse signals typically available in HMDs: the head and hands. It combines a lightweight MLP-based conditional diffusion model with block-wise timestep injection to generate smooth, real-time motion.

  • Motivation: Standard HMDs usually track only the head and hands, making high-fidelity full-body tracking—especially lower-body motion—an underconstrained problem.Adding lower-body IMUs increases cost and reduces user comfort.
  • Motivation: Diffusion models had not previously been used solely for motion reconstruction from sparse tracking information, motivating AGRoL.Existing conditional diffusion methods are not directly applicable because motion-joint representations differ from image representations.
  • Approach: AGRoL is a lightweight conditional diffusion model built on an MLP architecture for full-body motion synthesis from sparse IMU signals.The architecture uses common components including fully connected layers, SiLU activations, 1D convolutions, and layer normalization.
  • Approach: Block-wise timestep injection adds diffusion embeddings before intermediate network blocks, preserving denoising information that can otherwise be lost in MLP layers.The strategy is designed to reduce jittering artifacts and improve robustness when tracking signals are lost.
  • Results: AGRoL achieves state-of-the-art full-body motion synthesis while generating realistic, smooth motion at real-time inference speed.The model is presented as suitable for online AR/VR applications and does not require extra losses commonly used in other motion-prediction methods.

2. Related Work

Prior sparse-tracking methods either require more than the three signals typically available in HMDs or struggle with accurate full-body reconstruction. AGRoL addresses this setting with a custom diffusion model and a straightforward MLP architecture.

  • Motion Tracking from Sparse Tracking Inputs: Practical HMD tracking typically provides only three signals—the head and two wrists—whereas several prior methods require four or six IMUs.The additional inputs used by earlier systems limit direct applicability to standard HMD setups.
  • Motion Tracking from Sparse Tracking Inputs: Existing sparse-tracking approaches face difficulties predicting full-body pose, particularly in the lower body region.AGRoL is positioned as using only three IMU inputs while targeting this reconstruction challenge.
  • Diffusion Models for Motion Synthesis: Diffusion models support conditional generation, but recent motion-synthesis systems are complex and require multiple inference iterations.These characteristics hinder their use in real-time VR body tracking.
  • Human Motion Synthesis: Earlier human-motion synthesis approaches included sequence models, graph models, GANs, and control-based methods, often requiring full joint positions or environmental constraints.The related methods span future motion prediction and character-control formulations.

3. Method

AGRoL formulates full-body motion synthesis from sparse head-and-hand tracking as conditional diffusion over SMPL pose sequences. Its MLP backbone denoises motion while repeatedly injecting timestep information to preserve smoothness.

  • Problem Formulation: AGRoL predicts whole-body SMPL poses from sequences of sparse head-and-hand tracking features.The inputs include observed joint features, while outputs represent pelvis orientation and relative rotations of the remaining joints.
  • Diffusion Model: The model combines an MLP backbone with conditional diffusion to generate full-body motion from sparse tracking signals.The diffusion model conditions reverse denoising on the sparse tracking features and uses the MLP architecture as its prediction backbone.
  • Diffusion Model: Diffusion progressively corrupts motion with Gaussian noise and trains the reverse process to recover clean pose sequences.The model directly predicts clean poses rather than residual noise, using a mean-squared reconstruction objective.
  • Diffusion Model: At the terminal noising step, the motion sequence becomes Gaussian noise, which the reverse process progressively denoises.The forward process uses scheduled Gaussian transitions, with x1:N_T approaching an isotropic Gaussian distribution as T increases.
  • Diffusion Model: The architecture repeatedly injects timestep embeddings into every MLP block to prevent denoising information from being lost.Each timestep embedding is projected through a fully connected layer and SiLU activation before being added to the block input.

4. Experiments

AGRoL is evaluated on AMASS under two established data protocols using SMPL pose representations. On the reported subset benchmark, it achieves the best performance on several accuracy metrics and is especially strong for lower-body error and motion jitter.

  • Experimental Setup: The experiments use two AMASS training and testing protocols and represent poses with the SMPL human model.The protocols differ in their selected AMASS subsets, while the model predicts root global orientation and relative rotations of other joints.
  • Benchmark Results: AGRoL achieves the best MPJPE, MPJRE, and MPJVE performance in the reported AMASS subset comparison.The comparison also reports Lower PE and Jitter, where AGRoL particularly outperforms other models.
  • Experimental Setup: The reported evaluation metrics are MPJPE, MPJRE, MPJVE, and Jitter across the AMASS comparisons.These measure position, rotation, velocity, and motion smoothness-related performance, respectively, as specified in the benchmark table.
  • Benchmark Results: Table 2 distinguishes methods using pelvis information because pelvis location and rotation are unavailable to AGRoL during training and testing.Such methods are marked as not directly comparable under AGRoL’s input setting.
  • Implementation: Training uses a 12-block MLP with batch size 256, Adam-based optimization, and auto-regressive inference for longer sequences.The diffusion variant retains the MLP architecture while using AdamW instead of Adam.

4.2. Evaluation Metrics

AGRoL is evaluated with rotation-, velocity-, and position-related metrics on AMASS, including measures of accuracy and motion smoothness. The reported results show AGRoL surpassing previous methods across the metrics, with reduced jitter.

  • Evaluation Metrics: The evaluation groups nine metrics into rotation-, velocity-, and position-related categories.MPJRE measures rotation error; MPJVE and Jitter measure velocity error and smoothness; position metrics include MPJPE, Root PE, Hand PE, Upper PE, and Lower PE.
  • Evaluation Results: AGRoL surpasses all previous methods across the reported metrics, except for an insignificant 0.2 mm difference in Root PE.The evaluation uses AMASS and compares reconstruction accuracy and motion smoothness; qualitative results also compare AGRoL with AvatarPoser.
  • Evaluation Results: AGRoL improves generated-motion smoothness, reflected by reduced Jitter error compared with other methods.Jitter measures mean joint jerk in global space and serves as an indicator of motion smoothness.

4.4. Ablation Studies

The ablations examine the diffusion backbone, timestep-conditioning strategy, and inference sampling steps. Results favor the proposed MLP backbone, repetitive timestep injection, and five DDIM steps.

  • Architecture: The AGRoL diffusion model with the proposed MLP backbone outperforms versions using AvatarPoser or transformer backbones on most metrics.The compared backbones use the same diffusion-model hyperparameters, while AvatarPoser-Large adds transformer layers for comparable model size.
  • Diffusion Time Step Embedding: Without timestep embedding, position and rotation metrics remain reasonable, but velocity-related performance degrades.The ablation compares the no-embedding variant with Add, Concat, and RepIn strategies.
  • Diffusion Time Step Embedding: Repetitive timestep injection improves velocity-related metrics, alleviates jitter, and generates smoother motion than injecting the embedding only once.RepIn projects the timestep embedding separately for each MLP block and adds the resulting features to that block’s inputs.

4.5. Robustness to Tracking Loss

AGRoL is more robust to missing tracking inputs than previous methods under randomly masked input frames. With 10% of frames masked, prior methods degrade significantly while AGRoL shows less accuracy degradation.

  • Robustness to Tracking Loss: With 10% of input frames randomly masked, AGRoL shows less accuracy degradation than previous methods.The evaluation masks tracking frames during inference on AMASS to model temporal discontinuities caused by lost hand or controller tracking.

4.6. Inference Speed

AGRoL achieves real-time inference through its lightweight architecture and DDIM sampling. Its diffusion model generates 196 frames in 35 ms, while the predictive MLP requires 6 ms.

  • Inference Speed: 35 ms produces 196 AGRoL output frames using five DDIM sampling steps on a single NVIDIA V100 GPU.The reported generation uses the lightweight diffusion architecture combined with DDIM sampling.

5. Conclusion and Limitations

AGRoL is a lightweight MLP-based conditional diffusion model for full-body motion from sparse tracking signals, producing realistic, smooth motion at real-time speed for online AR/VR applications. The paper notes occasional floor penetration artifacts as a limitation.

  • AGRoL generates realistic and smooth human motion from sparse tracking signals while achieving real-time inference speed for online AR/VR applications.
  • Occasional floor penetration artifacts remain a limitation shared by AGRoL and related approaches.

A. Extra Ablation Experiments

The extra ablations examine geometric losses, sampling steps, sequence length, MLP depth, and diffusion prediction targets. Results favor the denoising objective alone, longer sequences up to N = 196, deeper networks up to 12 blocks, and direct clean-signal prediction.

  • Additional Losses: Additional geometric losses do not improve AGRoL, which achieves good results using only the denoising objective.
  • Sampling Steps: Using too few sampling steps worsens results, whereas more training-time sampling steps are beneficial.
  • Input/Output length: Longer input sequences reduce velocity error and jitter, but performance saturates after N = 196; AvatarPoser lacks consistent gains and degrades on some metrics.
  • Number of blocks in the MLP network: Performance improves as MLP blocks increase, with gains approaching a plateau beyond 12 blocks.
  • Predicting noise: Predicting the clean body motion at every diffusion step is substantially better than predicting residual noise across every metric.

B. Extra Datasets

On the challenging AIST++ dance dataset, AGRoL outperforms AvatarPoser and a predictive MLP across all reported metrics while producing smoother motion.

  • AGRoL achieves superior performance on all AIST++ metrics and produces smoother motion than AvatarPoser and the predictive MLP.It significantly reduces MPJVE, Jitter, and Lower PE compared with AvatarPoser.

C. Extra Qualitative Results

Qualitative comparisons show that AGRoL reconstructs full-body poses more accurately and smoothly than AvatarPoser, including on real VR inputs. Failures occur for underrepresented irregular poses and weak upper-to-lower-body correlations.

  • AGRoL reconstructs full-body poses more accurately, generates smoother motion, and alleviates jitter compared with AvatarPoser.
  • On real Quest HMD inputs, AGRoL shows reasonable generalization and generates smooth, precise motions.
  • AGRoL fails on irregular poses underrepresented in training and when lower-body motion is weakly correlated with the upper body.
Loading 2304.08577v1…