Source-linked AI summary
Perpetual Humanoid Control for Real-time Simulated Avatars
Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu
TL;DR
Physics-based humanoid controllers struggle with noisy inputs, falls, and large-scale motion imitation without compromising physical realism. The paper introduces PHC with PMCP to scale imitation, add fail-state recovery, and control avatars perpetually without resets. PHC imitates 98.9% of AMASS without external forces and demonstrates robust video-driven avatar control, while highly dynamic motions remain challenging.
Problem
Physics-based humanoid control must handle noisy video or language poses, unexpected falls, and large motion datasets without relying on non-physical external forces.
Method
PHC uses PMCP to allocate new subnetworks for harder sequences and fail-state recovery, while AMP and relaxed ankle and toe deviations support natural, balanced behavior.
Results
98.9% of AMASS is imitated without external forces, with high success on unseen MoCap sequences and approximately 90% success under noisy video motion.
Takeaways & Limitations
The controller supports perpetual, reset-free simulated avatars driven by noisy video and demonstrates live webcam control at 30 fps.
Takeaways & Limitations
Highly dynamic motions such as high jumping and back flipping remain difficult, and progressive training is long.
Abstract
from arXiv · showhide
We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from video or generated from language) and unexpected falls. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces and learns to naturally recover from fail-state. Given reference motion, our controller can perpetually control simulated avatars without requiring resets. At its core, we propose the progressive multiplicative control policy (PMCP), which dynamically allocates new network capacity to learn harder and harder motion sequences. PMCP allows efficient scaling for learning from large-scale motion databases and adding new tasks, such as fail-state recovery, without catastrophic forgetting. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.
1. Introduction
Physics-based humanoid control must imitate large motion datasets while remaining stable under noisy inputs and unexpected failures. The paper introduces PHC and PMCP to scale imitation, recover from fail-states, and support real-time avatars without external forces or resets.
- Noisy video or language poses can cause simulated humanoids to fall, deviate from reference motion, and fail to recover.
- AMASS contains ten thousand clips and 40 hours of motion, but a single policy had not successfully imitated such a large dataset.
- RFC reaches up to 97% AMASS imitation but compromises physical realism by applying non-physical external forces.
- PHC uses PMCP to allocate capacity for progressively harder sequences and fail-state recovery without catastrophic forgetting.
- 98.9% of the AMASS dataset is imitated without external forces, with task-agnostic compatibility for off-the-shelf video pose estimators.
- The controller is demonstrated in a live 30 fps webcam-driven avatar setting and supports perpetual simulation without resets.
2. Related Works
Prior work establishes the benefits of physics-based motion imitation but remains constrained by training cost, dataset scale, physical realism, and fail-state handling. Progressive learning methods address forgetting through regularization, experts, or increased network capacity.
- Physics-based Motion Imitation: Physics-based characters support natural motion and human interactions, but reinforcement-learning training is time-consuming and costly.
- Physics-based Motion Imitation: Large-scale motion imitation remains challenging, with prior systems often limited to high-quality MoCap data or smaller datasets.
- Fail-state Recovery for Simulated Characters: Existing fail-state methods include floating-base simulation, resets to kinematic poses, and rerunning sampling, each with stated physical-realism or success limitations.
- Progressive Reinforcement Learning: Catastrophic forgetting arises when fine-tuning across diverse patterns, motivating weight regularization, multiple experts, and capacity expansion approaches.
3. Method
The method combines goal-conditioned reinforcement learning with an Adversarial Motion Prior, progressive primitives, and a composer to imitate motion, recover from failures, and scale across harder sequences.
- Goal-Conditioned Motion Imitation: The controller uses goal-conditioned RL, with policy inputs combining humanoid proprioception and a reference-motion goal state.The goal can represent reference poses or 3D keypoints, along with simulated body state and optional body shape information.
- Goal-Conditioned Motion Imitation: The reward combines motion-tracking, adversarial style, and energy terms to encourage accurate, natural, and stable behavior.The energy penalty regulates the policy and reduces high-frequency foot jitter, while the task reward switches between imitation and recovery objectives.
- Goal-Conditioned Motion Imitation: The action specifies PD targets directly rather than residual offsets, removing dependence on reference motion and external stabilizing forces.This action design is intended to improve robustness to noisy and ill-posed references.
- Goal-Conditioned Motion Imitation: Relaxed Early Termination permits small ankle and toe deviations from MoCap while retaining imitation and discriminator rewards for those joints.This allows the humanoid to remain balanced without encouraging nonhuman joint motion.
- Progressive Multiplicative Control Policy: PMCP freezes each learned primitive, adds a new primitive with lateral connections for harder sequences, and trains a composer over the frozen primitives.The progressive procedure addresses performance plateaus and catastrophic forgetting during large-scale motion learning.
- Progressive Multiplicative Control Policy: Fail-state recovery uses a final primitive that removes non-root reference information and reduces recovery to a point-goal objective toward the target root.The recovery task covers fallen, distant, and fallen-and-distant states, then resumes imitation through the composed controller.
4. Experiments
Experiments evaluate PHC on MoCap imitation, noisy video-derived motion, component ablations, real-time avatar control, and fail-state recovery. PHC achieves strong imitation and recovery while operating in real time without external stabilizing forces.
- Motion Imitation: PHC is evaluated against UHC with and without residual force control on AMASS and H36M motion sequences.The evaluation uses 11313 AMASS training sequences, 140 AMASS test sequences, and 140 H36M-Motion* sequences.
- Implementation: The composite policy runs at > 30 FPS after training on a single NVIDIA A100 GPU.The control policy runs at 30 Hz and the simulation at 60 Hz.
- Motion Imitation: PHC outperforms baselines on most MoCap metrics and maintains high success on unseen AMASS and H36M sequences.Compared with UHC without residual force, PHC also has lower acceleration error on unseen motion.
- Noisy Video Input: PHC achieves approximately 90% success when imitating noisy poses estimated from video.The study combines HybrIK joint rotations with MeTRAbs root estimates and finds that the keypoint-based controller is more robust than the rotation-based controller.
- Ablations: Ablations show that PMCP improves robustness and imitation performance, while adding fail-state recovery without compromising motion imitation.Relaxed early termination improves balance use, and direct MCP benefits from larger capacity but performs worse than the progressive pipeline.
- Real-time Simulated Avatars: In live demonstrations, the keypoint-based controller remains stable while imitating webcam pose streams and motion generated by a language model.The avatar demo includes posing and jumping motions and runs at 30 fps.
- Fail-state Recovery: Both keypoint-based and rotation-based controllers recover from fall states with success rates above 90%, including when fallen and 3 meters from the reference.Recovery is evaluated over 1000 randomized trials using standing-still reference motion.
5. Discussions
PHC achieves high-quality motion imitation and fail-state recovery, while remaining imperfect on highly dynamic motions and costly to train. The authors also identify tighter integration with downstream estimators and generators as future work.
- The controller does not achieve 100% training-set success, with high jumping and back flipping remaining challenging.The authors hypothesize these motions require planning and intent not conveyed by the single-frame pose target.
- Progressive training increases training time, and the disjoint video-estimation process may be insufficient for better downstream tasks.The paper calls for tighter integration with pose estimation and language-based motion generation.
- PHC imitates human motion from MoCap and noisy input while recovering from fail-states and controlling real-time avatars without resets.
- Future work includes imitating 100% of training motions, adding terrain and scene awareness, and integrating more tightly with pose estimation and motion generation.
- The project page presents qualitative results for MoCap, noisy video motion, real-time single- and multi-person avatars, language-based avatars, and fail-state recovery.
B.1. Training Details
The humanoid supports varied body shapes and simulation representations, while training uses filtered AMASS motions and large-scale parallel simulation. Once trained, PHC runs in real time with a compact model.
- Humanoid Construction: The humanoid uses the SMPL kinematic structure and supports randomly selected gender and body-shape variations in capsule-based and mesh-based forms.The body geometry is scaled to produce an average weight of 70 kg, with configurable inter-humanoid collision for multi-person use.
- Training Process: Training samples motions from the current training set and normalizes them to the simulated body shape using forward kinematics.
- Training Process: 1536 humanoids are simulated in parallel, requiring around 7 days to collect approximately 10 billion samples.
- Data Preparation: 11313 filtered AMASS sequences are used for training and 140 sequences for testing after removing human-object interaction motions and other invalid sequences.
- Runtime: ∼32FPS with simulation and rendering and ∼50FPS without rendering demonstrate real-time execution.
- Model Size: The final four-primitive model is 28.8 MB, comparable to UHC at 30.4 MB.
B.2. Real-time Use Cases
PHC connects causal, real-time off-the-shelf pose estimators and a language-based motion generator to physics-based avatar imitation. The system supports multi-person interaction and chaining generated motion clips.
- Real-time Physics-based Virtual Avatars from Video: Yolov8 detects people, while MeTRAbs and HybrIK provide 3D pose inputs for PHC.MeTRAbs supplies keypoints to the keypoint-based controller; HybrIK supplies inputs to the rotation-based controller.
- Real-time Physics-based Virtual Avatars from Video: Both pose-estimation methods are causal, use no temporal information, and run at approximately 30FPS.
- Real-time Physics-based Virtual Avatars from Video: HybrIK adds learned inverse kinematics to recover joint angles from estimated 3D keypoints, while MeTRAbs uses a keypoint-only formulation.
- Real-time Physics-based Virtual Avatars from Video: The video pipeline supports real-time multi-person physics-based interaction with optional inter-humanoid collision.
- Virtual Avatars from Language: MDM generates 3D keypoints that connect to PHC's keypoint-based imitator for language-based motion control.
- Virtual Avatars from Language: PHC can chain fixed-length generated motion clips without requiring additional blending between clips.
B.3. Progressive Neural Network (PNN) Details
The progressive neural network grows capacity by training new primitives on motions earlier primitives fail, preserving prior primitives while adding harder sequences. Weight sharing offers a faster alternative to lateral connections, and training reveals persistent failures alongside forgetting.
- Progressive Neural Network: PNN begins with one primitive trained on the full dataset, then forms a hard-motion subset from sequences that primitive fails to imitate.
- Progressive Neural Network: The first primitive is frozen before a new primitive is created to learn the remaining hard motions.
- Progressive Neural Network: Lateral connections pass earlier primitives' activations into subsequent layers through learnable connection matrices.
- Progressive Neural Network: Weight sharing and warm starts perform equally effectively to lateral connections under the same hard-negative mining procedure.Each newly learned primitive adds sequences that PHC can imitate.
- Progressive Neural Network: Weight sharing significantly decreases training time compared with the alternative architecture.
- Training Behavior: Across 529 failed sequences, 30 consistently fail while other sequences can be learned and later forgotten during training.The figure uses white pixels for successful imitation and black pixels for unsuccessful imitation at each epoch.
C. Supplementary Results
The supplementary results identify forgetting as a central difficulty when learning large motion datasets and show that adding fail-state recovery can substantially reduce imitation performance.
- Around 30% of failed sequences overlap across evaluations, indicating that the policy forgets previously learned motions during training.Backflips, cartwheeling, and acrobatics account for much of this overlap and remain consistently difficult to learn jointly.
- Fail-state recovery reduces H36M-Test-Video performance to Succ: 42.5%, Eg-mpjpe: 87.3, and Empjpe: 55.9, versus P(1) at Succ: 59.4%, Eg-mpjpe: 60.2, and Empjpe: 34.4.The comparison motivates PMCP by showing that learning recovery can cause severe forgetting of imitation skills.
C.2. Additional Ablations
Additional ablations examine architectural choices, humanoid geometry, and primitive count, finding that multiplicative control and more primitives improve performance while preserving practical inference speed.
- PNN Lateral Connection vs. Weight Sharing: Weight sharing and lateral connections produce similar motion-imitation performance and inference speed in the evaluated PNN setup.The authors attribute this to task similarity, which reduces the need for lateral connections to select prior experiences.
- MOE vs. MCP: MCP slightly outperforms top-1 MOE because MCP can activate all pretrained primitives simultaneously, whereas top-1 MOE activates only one.The comparison concerns pretrained and frozen primitives composed from the input.
- Capsule vs. Mesh Humanoid: Mesh-based humanoids achieve similar performance to capsule-based humanoids while reducing simulation speed from 32 FPS to 30 FPS.The slowdown is attributed to the greater computational cost of simulating meshes.
- Number of primitives: Performance increases as the number of primitives grows, with especially rapid improvement from two to three primitives.The fail-state recovery primitive does not improve motion imitation, and inference speed changes little because the networks are relatively small.
- Number of primitives: Highly dynamic motions such as high-jumping, backflipping, and cartwheeling are increasingly difficult to learn together because their required movement setups can contradict one another.The supplementary discussion notes that these sequences can be overfit individually but are significantly harder to learn jointly.
D. Extended Limitation and Discussions
The extended discussion reports remaining limitations in motion coverage, real-time input quality, camera and tracking assumptions, and the naturalness of recovery transitions.
- PHC reaches a 98.9% AMASS training-set success rate but still struggles with dynamic motions such as backflipping.Real-time avatar performance is noticeably worse than offline performance.
- Noisy and discontinuous monocular pose estimates can provide incorrect movement signals, while PHC relies on stable velocity estimation for movement cues.Depth ambiguity, jittery keypoints, and difficult real-time velocity estimation are identified as sources of degradation.
- PHC assumes 30 FPS motion input, but few pose estimators operate at least at 30 FPS and their frame rates can fluctuate with computer load.
- Multi-person tracking and identity switches can cause humanoids to exchange places, producing a jarring experience.
- Recovery remains imperfect: gait artifacts and sudden jolts or snaps can appear during recovery and the transition back to motion imitation.The discussion calls for better rewards or additional trajectory observations to improve naturalness.
- The controller demonstrates perpetual video-based and language-based avatar imitation, including motion inbetweening between multiple generated clips.The authors position PMCP as a framework for progressive reinforcement learning and multi-task learning.