Source-linked AI summary
Character Controllers Using Motion VAEs
Hung Yu Ling, Fabio Zinno, George Cheng, Michiel van de Panne
TL;DR
The paper tackles how to generate purposeful, realistic human motion from example motion-capture clips. It learns an autoregressive conditional Motion VAE whose stochastic latent variables define an action space for reinforcement-learning controllers, and demonstrates stable motion generation and goal-directed control across tasks.
Problem
The paper asks how example motions can be generalized into new purposeful, high-quality, compact, efficient, and goal-directed human movements.
Method
The paper trains an autoregressive conditional MVAE on motion-capture data and uses its stochastic latent variables as the action space for reinforcement-learning control.
Results
The MVAE produces robust, high-quality, long-term motion predictions, while reinforcement learning learns control policies on top of the same model for desired movements.
Takeaways & Limitations
Separating task learning from motion-dynamics learning allows multiple control policies to be learned using the same motion model.
Takeaways & Limitations
Generated behavior depends on the distribution and connectivity of the example data, and the authors found walking-paced policies difficult to generate.
Abstract
from arXiv · showhide
A fundamental problem in computer animation is that of realizing purposeful and realistic human movement given a sufficiently-rich set of motion capture clips. We learn data-driven generative models of human movement using autoregressive conditional variational autoencoders, or Motion VAEs. The latent variables of the learned autoencoder define the action space for the movement and thereby govern its evolution over time. Planning or control algorithms can then use this action space to generate desired motions. In particular, we use deep reinforcement learning to learn controllers that achieve goal-directed movements. We demonstrate the effectiveness of the approach on multiple tasks. We further evaluate system-design choices and describe the current limitations of Motion VAEs.
1 INTRODUCTION
The paper addresses purposeful human-motion generation from example motions by learning an MVAE whose stochastic latent variables form an action space for reinforcement-learning control. It reports stable, high-quality motion generation and compact policies for goal-directed movement.
- Problem: The paper asks how example motions can be generalized into new purposeful motions for interactive character animation and control.The target applications include games, simulations, and virtual reality.
- Method: The MVAE predicts distributions of next poses from the current pose, with different latent-variable samples producing different feasible predictions.Because predictions are autoregressive, each predicted pose becomes the input for the following step.
- Control: The latent variables governing next-state predictions can serve as the action space for reinforcement learning, whose policy uses a reward function defining character goals.This separates learned motion dynamics from task control.
- Motion quality: The MVAE produces high-quality motion and supports stable generation, including fully-skinned characters without additional footskate cleanup.The authors also report that it can operate in a memoryless fashion conditioned only on the previous pose, which includes velocities.
- Results: Reinforcement learning can use the MVAE to produce compact control policies that generate desired goal-directed movement.This is presented as a principal contribution alongside the generative motion model.
2 RELATED WORK
The related work is organized around direct prediction versus model-then-control, and kinematic versus physics-based motion generation. The paper positions MVAE as a parametric model-then-control approach that combines stochastic motion generation with reinforcement-learning control.
- Taxonomy: Motion-generation methods can be classified as kinematic or physics-based, and as direct prediction or model-then-control.Kinematic methods predict motion without physics, whereas physics-based methods use simulation; model-then-control separates motion modeling from task control.
- Model-then-control: Model-then-control methods first learn a space of possible motions and then use planning or reinforcement learning to traverse it toward a goal.Prior approaches include motion graphs, continuous latent embeddings, and discrete actions defined through similarity neighborhoods.
- Direct prediction: Direct-prediction parametric models learn pose-by-pose behavior-specific motion models from task-specific examples.Autoregressive and recurrent models have been used for locomotion and other task-specific human motions, but long-term stability is a recognized challenge.
- MVAE positioning: The MVAE is a parametric model-then-control method whose generic motion model supports multiple future tasks through latent-space reinforcement-learning controllers.Its stochastic latent variable governs time evolution because it is sampled at run-time, unlike the cited follow-up VAE approach.
- MVAE positioning: The paper reports high-quality motion from a memory-free first-order autoregressive model trained with scheduled sampling, without contact annotations, post-processing, or time-convolutional structure.It describes this combination as distinct from prior parametric model-then-control approaches using reinforcement learning.
3 MOTION VAES
The MVAE learns a stochastic next-pose model from motion capture and generates motion autoregressively through latent-variable sampling. Its design addresses pose representation, decoder structure, posterior collapse, generalization, and long-term prediction stability.
- Pose Representation: A pose contains root linear and angular velocities, root-space joint positions and velocities, and six-dimensional joint orientations.Joint orientations use forward and upward vectors in character space rather than angle-based representations.
- Conditions and Predictions: The MVAE learns a distribution of possible next poses from motion-capture data and samples latent variables to generate future poses autoregressively.At run-time, the encoder is discarded and the decoder predicts one pose at a time from the current pose and a latent sample.
- Decoder: The decoder uses a mixture-of-experts architecture whose gating network blends predictions from six expert networks.The authors empirically associate this design with slightly better pose reconstruction and reduced visual artifacts.
- Avoiding Posterior Collapse: Conditioning on more consecutive poses improves reconstruction quality but reduces output diversity and can lead to posterior collapse.Passing the latent variable to every expert-network layer is reported to reduce the likelihood of collapse; one or two consecutive poses work well in these experiments.
- Balancing Motion Quality and Generalization: The quality-generalization balance is controlled through the relative weighting of reconstruction and KL-divergence losses.The paper frames this balance as a fundamental challenge for kinematic animation systems.
- Training Data: The motion database contains about 30,000 frames captured at 30 Hz across walking, running, turning, dynamic stopping, and resting motions.The data includes mirrored trajectories, and training does not use motion classification, foot-contact annotations, or gait-phase annotations.
- Stable Sequence Prediction: Scheduled sampling progressively exposes training to the run-time distribution, reducing instability caused by accumulating autoregressive reconstruction errors.Training uses supervised learning, scheduled sampling, and autoregressive prediction modes for 20, 20, and 140 epochs respectively.
4 MOTION SYNTHESIS
The MVAE supports random motion generation and sampling-based control by treating latent samples as actions that govern plausible motion evolution. Random walks can expose gaps in the motion database, while sampling-based control is limited on harder tasks.
- Random Walk: Random latent samples generate plausible motion from an initial character state, often continuing the source clip’s movement pattern.The synthesized motion can continue sprinting from a sprint-cycle pose or transition from stationary poses into locomotion.
- Random Walk: Random walks reveal whether the motion database contains transitions between different motions.Failure to transition after many samples indicates that additional transition motion capture data may be needed.
- Random Walk: The experiment used random walks to identify insufficient character-turning examples in the original motion capture database.
- Sampling-based Control: Sampling-based control performs N Monte Carlo roll-outs over horizon H, applies the first action from the best trajectory, and repeats this procedure.The method works modestly well for simple locomotion tasks such as Target.
- Sampling-based Control: Sampling-based control struggles to reach within two feet of the target and fails on more difficult joystick-control and path-following tasks.It offers faster artist design iteration at the cost of more runtime computation.
5 LEARNING CONTROL POLICIES
The system uses deep reinforcement learning to control an MVAE through latent actions, with task rewards shaping behavior from generated poses and environment feedback. PPO-based policies train efficiently across tasks, while an energy penalty produces visibly lower-effort motions.
- Learning Control Policies: Latent samples z serve as the action space for policies that guide characters toward varied goals, including targets, speeds, directions, and local-vision tasks.
- Learning Control Policies: The controller loop feeds the decoder-generated pose and task information to a policy, whose latent output becomes the decoder’s next input.The encoder is discarded because the next motion-capture pose is unavailable during control.
- Learning Control Policies: DRL policies map environment states to actions and are trained to maximize discounted reward.The policy uses a neural network, and the discount factor γ lies in [0, 1).
- Learning Control Policies: PPO is used to solve the DRL optimization problem, with a two-hidden-layer 256-unit ReLU policy whose Tanh output is scaled to [-4, +4].The scaling is chosen according to the latent distribution.
- Learning Control Policies: All described tasks train within one to six hours on the authors’ desktop machine using GPU-based data collection and training.
- Effort Penalty in Kinematics Animation: Including a joint-based energy penalty enables policies to find visibly lower-effort solutions than root-velocity regulation.The approach does not require a target velocity at runtime, supporting non-directly controllable characters or large crowds.
6 LOCOMOTION CONTROLLERS
The paper demonstrates MVAE-based reinforcement-learning controllers across target navigation, joystick control, path following, and maze exploration. These tasks show flexible goal representations, while timed targets, path-speed limits, and wall collisions expose important control constraints.
- Task suite: The controllers address Target, Joystick Control, Path Follower, and Maze Runner locomotion tasks.The tasks use target locations, desired direction and speed, predefined paths, or local vision sensing.
- Target: Target navigation rewards progress toward randomly placed targets and provides a one-time bonus when the pelvis comes within two feet.The arena is 120×80 feet, and a new target is selected after each successful reach.
- Target: Timed Target value functions peak at middle distances because nearby targets offer little progress reward while unreachable targets lose the target bonus.The remaining time determines whether the target can be reached, producing inverted parabolic cylinder-like surfaces.
- Joystick Control: Joystick Control simultaneously matches joystick heading and speed, with desired speed sampled from 0 to 24 feet per second.The reward combines angular and speed terms, and the direction and speed can be changed interactively at run-time.
- Path Follower: Path Follower receives four targets spaced 15 time steps apart, enabling adaptation to path variations without exposing the entire trajectory.On faster moving targets it cuts corners, and for 180-degree turns it improvises a small-radius loop.
- Maze Runner: Maze Runner uses local wall-distance vision and exploration bonuses, but hierarchical reinforcement learning is beneficial because a single policy often collides with walls.The hierarchical controller places a high-level target-location policy above a pretrained Target-task low-level controller.
7 DISCUSSION & EVALUATION
The evaluation finds that MVAEs support high-quality motion and responsive control, while revealing dependencies on decoder design, training data, and policy learning. The learned model also supports interpretable motion variation and task-specific behavior, but sampling-based control and constrained data limit performance.
- Patterns in the Gating Network: Expert activations reveal locomotion structure, with high-frequency oscillations aligned to foot strikes and low-frequency envelopes tracking overall character motion.Under Target control, activation patterns also correspond to a turn after reaching the target.
- MVAEs generate high-quality motions and responsive controls comparable to existing kinematic motion synthesis methods.
- Ablation on Decoder Architecture: The mixture-of-experts decoder produces higher motion quality with fewer visible artifacts than a non-mixture decoder.Without the mixture model, predicted poses can converge toward a mean pose and leave the character stuck while gliding.
- Using a Noisy Policy to Generate Motion Variations: Runtime action noise creates distinguishable motion variations while characters remain on task under the same Path Follower policy.The variations arise because noise accumulates over time in the characters’ trajectories.
- Limitations: Motion quality and controllability depend on the motion-data distribution, available motion connectivity, and choices in MVAE and policy design.The authors specifically report difficulty learning walking-pace policies and navigating tightly constrained environments.
8 CONCLUSIONS
The paper presents MVAEs as stochastic motion-dynamics models whose latent variables can serve as action spaces for reinforcement-learning controllers. Separating dynamics learning from task learning allows multiple control policies to use one motion model, while future work targets artist control, larger datasets, and richer environmental context.
- MVAEs produce robust, high-quality, long-term motion predictions with a simple memoryless first-order autoregressive model.
- Reinforcement learning uses the MVAE’s stochastic latent variable as an action space for learning control policies.
- Separating task learning from dynamics learning allows multiple control policies to be learned using the same motion model.
- Future work includes artist-facing controls, tools for improving motion-data connectivity, larger datasets, more environment context, and multi-agent settings.