Source-linked AI summary
MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting
Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, Xue Bin Peng
TL;DR
Physics-based character controllers have largely specialized in narrow tasks and control modalities, motivating a more reusable approach. MaskedMimic trains one controller to inpaint full motions from partial multimodal descriptions, and demonstrates diverse tasks through partial constraints without task-specific training or reward engineering. The paper also identifies motion-quality and long-horizon planning limitations.
Problem
Prior physics-based controllers specialize in narrow tasks and modalities, while versatile control across dynamic instructions and diverse scenes remains challenging.
Method
MaskedMimic trains a single physics-based controller on randomly masked motion sequences to inpaint full-body motion from partial keyframes, joint targets, text, objects, or combinations thereof.
Results
A single unified control architecture performs joystick steering, reaching, VR tracking, full-body tracking, path-following, object interaction, and text-to-motion without task-specific training or reward engineering.
Takeaways & Limitations
Partial constraints provide an intuitive interface for composing diverse character-animation tasks within one control architecture.
Takeaways & Limitations
Generated motions can exhibit jittering, difficult training motions may not be reproduced, and irregular-terrain behavior lacks human-like long-horizon foot-placement planning.
Abstract
from arXiv · showhide
Crafting a single, versatile physics-based controller that can breathe life into interactive characters across a wide spectrum of scenarios represents an exciting frontier in character animation. An ideal controller should support diverse control modalities, such as sparse target keyframes, text instructions, and scene information. While previous works have proposed physically simulated, scene-aware control models, these systems have predominantly focused on developing controllers that each specializes in a narrow set of tasks and control modalities. This work presents MaskedMimic, a novel approach that formulates physics-based character control as a general motion inpainting problem. Our key insight is to train a single unified model to synthesize motions from partial (masked) motion descriptions, such as masked keyframes, objects, text descriptions, or any combination thereof. This is achieved by leveraging motion tracking data and designing a scalable training method that can effectively utilize diverse motion descriptions to produce coherent animations. Through this process, our approach learns a physics-based controller that provides an intuitive control interface without requiring tedious reward engineering for all behaviors of interest. The resulting controller supports a wide range of control modalities and enables seamless transitions between disparate tasks. By unifying character control through motion inpainting, MaskedMimic creates versatile virtual characters. These characters can dynamically adapt to complex scenes and compose diverse motions on demand, enabling more interactive and immersive experiences.
1 INTRODUCTION
MaskedMimic addresses the challenge of controlling physics-based characters across diverse tasks by training one controller from partially masked, multimodal motion descriptions. It replaces task-specific reward engineering with an inpainting interface supporting keyframes, joint targets, text, objects, and combinations thereof.
- Motivation: Prior physics-based controllers typically specialize in narrow tasks and modalities, limiting unified control across complex scenarios.The motivating scenario combines uneven-terrain locomotion, text-guided animation, and object interaction.
- Approach: The proposed framework trains a single controller on randomly masked motion sequences to predict the original full-motion sequence.This formulation uses existing motion-capture information, including kinematic trajectories, text descriptions, and scene information.
- Control interface: Goal-engineering provides intuitive partial constraints that replace complex reward functions and support novel combinations of objectives.The authors describe this interface as analogous to prompt engineering and report positive transfer between tasks.
- Approach: MaskedMimic generates full-body motion from partial descriptions including target keyframes, joint positions or rotations, text instructions, object interactions, or combinations thereof.The framework is designed as a unified physics-based character-control system based on motion inpainting.
- Scope: A single model reproduces tasks spanning full-body tracking, VR tracking, scene interaction, terrain traversal, and text control.These task families are listed as being supported within one model through goal-engineering techniques.
2 RELATED WORK
Related work spans task-specific physics controllers, kinematic and physics-based motion inpainting, human-object interaction, text-to-motion, and latent generative models. MaskedMimic distinguishes itself by combining physical grounding with flexible multimodal sparsity in one controller.
- Physics-based character animation: Early physics-based animation controllers were manually designed per task and required lengthy engineering that did not scale to broad behavior repertoires.More recent systems learned scene-aware behaviors, but the cited progression remains organized around specialized tasks such as locomotion and object interaction.
- Human-object interaction: Physics-based formulations address unrealistic penetration and floating artifacts in kinematic human-object interaction by enforcing physically plausible dynamics.MaskedMimic builds on physics-based interaction work while introducing a controller that also handles other modalities.
- Text to motion: MaskedMimic directly conditions one physics-based controller on text and kinematic constraints without a separate text-to-motion model.The resulting interface provides text-based stylization of simulated motions.
- Latent generative models: Latent generative controllers reduce task-specific data curation but often require additional hierarchical controllers because their latent representations are not intuitively grounded.MaskedMimic instead formulates control as inpainting over partial multimodal constraints extracted from the data.
- Motion inpainting: Kinematic motion inpainting infers full-body motion from sparse sensors or keyframes, whereas prior physically animated tracking systems often assume fixed sparsity patterns.The related work includes autoregressive, variational, and diffusion-based kinematic methods, along with fixed-joint physical tracking.
- Motion inpainting: MaskedMimic supports flexible sparsity in which joints, keyframes, objects, and text can be observed or unobserved in any combination.This enables unconditional generation, inbetweening, VR control across irregular terrain, and object interaction.
3 PRELIMINARIES
The framework first trains a goal-conditioned motion-tracking policy with reinforcement learning, then distills it into a multimodal student using online behavioral cloning. The two-stage design separates versatile physical skill learning from flexible control through partial goals.
- Reinforcement learning: In the first stage, reinforcement learning trains a motion-tracking controller on a large dataset of motion clips.The agent observes states and future goals, samples actions from a policy, transitions through environment dynamics, and receives rewards.
- Reinforcement learning: The reinforcement-learning objective maximizes discounted cumulative reward over trajectories generated by the policy and environment dynamics.The discount factor γ ∈ [0, 1) determines the policy’s effective horizon.
- Behavioral cloning: In the second stage, behavioral cloning distills the reinforcement-learned teacher into a student policy directed by multimodal inputs.The distillation uses DAgger, collecting trajectories under the student and relabeling them with teacher actions.
- Behavioral cloning: DAgger uses the state-and-goal distribution induced by the student policy to mitigate drift during online behavioral cloning.The student’s own visited states and goals determine the distribution used for active distillation.
4 SYSTEM OVERVIEW
MaskedMimic uses a two-stage design: a fully-constrained tracker is trained first, then distilled into a partially-constrained controller that generates motions from masked goals.
- Stage 1: Fully-Constrained Controller: The framework first trains a fully-constrained controller πFC to imitate full-body kinematic target motions in scene-aware environments.
- Stage 2: Partially-Constrained Controller: The second stage distills πFC into a partially-constrained controller πPC that operates from goals specifying only some elements of a desired motion.
- Stage 2: Partially-Constrained Controller: Randomly masked target observations train the controller to infer complete motions from partial motion descriptions.
5 FULLY-CONSTRAINED CONTROLLER
The fully-constrained controller learns physics-based motion tracking from full-body targets and scene observations, using reinforcement learning and a motion-tracking reward across varied environments.
- Controller Training: πFC is trained with reinforcement learning to imitate a large library of reference motions across irregular environments and object-interaction settings.
- Observations: The controller observes the current humanoid state, future target poses, and surrounding terrain or object heightmaps.
- Model Architecture: A transformer-based policy predicts actions from sequential character, terrain, and target-motion information, while a critic estimates the value function.
- Reward Function: Its motion-tracking reward combines global joint positions and rotations, root height, joint velocities, joint angular velocities, and an energy penalty.
- Training Playground: The training environment separates flat terrain, irregular terrain, and an object playground to support imitation, robust locomotion, and object interaction.
6 VERSATILE PARTIALLY-CONSTRAINED CONTROLLER
The partially-constrained controller treats character control as motion inpainting: it learns from masked goals to generate diverse full-body motions satisfying keyframe, text, and object constraints.
- Motion Inpainting: πPC learns to model the fully-constrained controller’s actions while observing only partial constraints, enabling task control without task-specific training.
- Goal Modalities: Supported goals include arbitrary joint targets over future times, text commands, and object-based interaction constraints.
- Goal Composition: Multiple goals can be combined, such as following a root trajectory while receiving a text instruction to walk with raised hands.
- Modeling Diversity: A conditional VAE addresses ambiguity in partial goals by sampling multiple realistic and physically plausible motions.
- Runtime Inference: The learned prior generates motions from partial constraints at runtime, while the full-observation encoder is used only during training.
- Masking: Temporally structured masking improves generalization by keeping selected joints visible across consecutive frames rather than resampling masks independently each step.
7 EXPERIMENTAL SETUP
The evaluation uses large-scale simulation and multimodal motion datasets to test a single unfine-tuned controller across tracking, sparse control, terrain variation, and user-defined tasks.
- Training Configuration: Training uses 16,384 parallel Isaac Gym environments across four A100 GPUs, with controllers operating at 30 Hz and simulation at 120 Hz.
- Datasets: The dataset aggregation combines AMASS for keyframe conditioning, HumanML3D for text commands, and SAMP for object interactions.
- Evaluation Protocol: The benchmark reports both success-rate and error-rate metrics across common tasks from prior systems.
- Evaluation Protocol: The unified model is evaluated without task-specific fine-tuning, using user-specified goals or goals extracted from kinematic recordings.
- Evaluation Tasks: Experiments cover full-body tracking, VR-style sparse joint tracking, irregular terrains, path following, steering, reaching, and object interaction.
- Generalization: Goal-engineered finite-state machines test generalization to new tasks without additional task-specific training, including sitting on held-out objects.
8 RESULTS
MaskedMimic demonstrates versatile physics-based control across full-body tracking, sparse VR tracking, irregular terrain, object interaction, and text commands. It generalizes to unseen motions and novel task combinations, although long-horizon text instructions remain challenging.
- Motion Tracking: MaskedMimic tracks unseen full-body motions and reconstructs complete behaviors from sparse VR or head-only signals.It reproduces fighting, dancing, cartwheels across irregular terrain, and head-path-following runs.
- Motion Tracking: 39.5 train and 45.8 test MPOJPE on VR tracking, achieved without additional task-specific training.MaskedMimic is applied directly to sparse tracking and outperforms the compared models by a significant margin.
- Motion Tracking: 62.5% lower tracking failure rate on unseen motions than PHC+ for the fully-constrained tracker.The controller also supports irregular terrains and object interactions.
- Motion Tracking: A single monolithic network generalizes better than expert-based alternatives while avoiding expert selection and blending.The experiments attribute the improvement to the unified architecture’s ability to capture diverse motions.
- Scene and Object Interaction: MaskedMimic transitions between navigation and object-interaction controls, including approaching unseen chairs or sofas and sitting or lying on rough terrain.These object–terrain combinations were not observed during training.
- Text Control: Text control works for simple atomic behaviors but struggles with commands requiring long-term reasoning.The authors speculate that the relatively short observation history hampers long-term reasoning capabilities.
9 LIMITATIONS AND FUTURE WORK
MaskedMimic’s main limitations concern motion quality, manual goal-engineering, and capabilities beyond static-scene interaction. Future work targets improved motion realism and terrain behavior, automated goal specification, and dynamic multi-agent interactions.
- Motion Quality: Some generated motions exhibit unnatural jittering, and challenging motions such as backflips and breakdancing remain difficult to reproduce.The authors suggest discriminative-reward fine-tuning as one possible mitigation for unrealistic behaviors.
- Motion Quality: On irregular terrain, the character tends to imitate standard walking instead of planning foot placements suited to rugged regions.The authors hypothesize that naive root-to-floor-distance normalization contributes to this limitation.
- Goal-Engineering: Designing goals for large groups of characters in complex scenes such as crowds could be challenging and labour-intensive.The authors identify automated goal-engineering, including possible large-language-model assistance, as future work.
- New Capabilities: The model’s current scope does not include dynamic-scene interactions such as manipulating objects, using tools, or complex multi-agent interactions.These behaviors are identified as directions for expanding MaskedMimic’s capabilities.
10 DISCUSSION
MaskedMimic formulates physics-based character control as motion inpainting from partial information and uses one unified architecture across diverse tasks. The discussion emphasizes flexible constraint satisfaction, broad task coverage, and intuitive control without task-specific training or reward engineering.
- 10 DISCUSSION: MaskedMimic is trained on randomly masked multimodal motion clips, including target joint positions or rotations, text descriptions, and objects.The model learns to fill missing information coherently and generate plausible motions satisfying partial constraints.
- 10 DISCUSSION: A single unified control architecture performs joystick steering, reaching, VR-tracking, full-body tracking, path-following, object interaction, and text-to-motion synthesis.The reported tasks span multiple control modalities and do not require task-specific training or reward engineering.
- 10 DISCUSSION: Learning from partial specifications gives the model a general understanding of producing realistic, physically-plausible movements directed toward varied goals.The authors connect this capability to simpler, more intuitive, and more flexible control of simulated characters.
- 10 DISCUSSION: The controlled character is a 3D physically simulated humanoid based on the neutral SMPL body shape, with 69 degrees of freedom.Its observed state includes root and joint motion information plus hand and foot positions.
- 10 DISCUSSION: The resulting state representation has 358 dimensions, while the action space contains 69 target joint rotations for PD controllers.Joint rotations are encoded using tangent and normal vectors for the state and a 3D exponential map for actions.
B MASKEDMIMIC– PRIOR CONSTRUCTION
The prior construction represents constraints, objects, text, history, and terrain as modality-specific encoded inputs to a transformer-based controller. Its latent distribution is produced from transformer features and used by a decoder to output PD-control actions.
- Inputs: Any joint can receive position or rotation constraints at any timestep, represented relative to the humanoid’s local frame and corresponding body part.A target pose with no constraints is masked from entering the transformer.
- Inputs: Target objects are represented by bounding-box coordinates, direction, and category type, while text is embedded into a 512-dimensional XCLIP vector.Object features are expressed relative to the character’s local coordinate frame.
- Inputs: Historical input stores the previous 40 poses, with one pose sampled from each group of eight and a time value appended to each pose.Unavailable history is masked at episode start, while mid-sequence initialization uses historic kinematic poses.
- Inputs: The terrain heightmap uses a 16x16 square grid with 10cm spacing, is flattened, and is always provided to its encoder.The grid is rotated with the character before encoding.
- Architecture: Each input type uses a shared encoder, and the resulting tokens are combined with the current-pose encoding in a transformer.Inputs are normalized with a running-mean standard-deviation normalizer before encoding.
- Architecture: The transformer output is mapped through MLP heads to a latent distribution, while the decoder combines the current pose, terrain, and sampled latent to produce PD targets.The transformer uses latent dimension 512, four layers, and four self-attention heads; the decoder outputs the action.
- Latent Sampling: A fixed noise sample is maintained throughout each episode and resampled when the episode terminates.This implements the reparameterization-based latent sampling used during training and inference.
B.1 Training hyperparameters
Training uses simulated rollouts with A2C for the fully constrained controller and supervised behavior cloning for MaskedMimic. The configuration scales KL regularization, parallelizes simulation across many Isaac Gym environments, and runs the policy at 30 Hz.
- B.1 Training hyperparameters: Each episode uses 32-step rollouts, after which MaskedMimic is trained with A2C using γ=0.99.Episodes begin from states sampled by the prioritized sampling mechanism and use a unique fixed noise sample until reset.
- B.1 Training hyperparameters: The reward configuration uses w_gr=0.3, w_gt=0.5, w_jv=0.1, w_jav=0.1, w_rh=0.2, and w_enrg=0.0005.The corresponding reward scales are c_gr=2, c_gt=100, c_jv=0.5, c_jav=0.1, and c_rh=100.
- B.1 Training hyperparameters: The KL coefficient increases from 0.0001 to 0.01 across 6000 epochs, beginning at epoch 3000.Training uses 16,384 Isaac Gym environments across four A100 GPUs for 14 days, with decisions at 30 Hz and a 64D latent space.
- B.1 Training hyperparameters: MaskedMimic uses a mini-batch size of 8,192 and one epoch with a supervised behavior-cloning objective.Unlike the fully constrained controller, it does not use gradient accumulation for this supervised objective.
B.2 Masking
MaskedMimic trains on varied masked motion sequences so the controller must reconstruct motion across partial observations and gaps. The masking scheme combines near-term joint constraints, long-term conditions, repeated patterns, and hidden modalities.
- Masking strategy: The model uses K=11 future poses: 10 for near-term constraints and one for a random long-term pose.For the first 10 poses, sparsity usually persists from the preceding pattern, with occasional random joint and constraint sampling.
- Masking strategy: With probability 1%, the scheme masks a randomly sized time gap of 1–9 poses while retaining at least one observable future pose.This exposes the model to inbetweening while preserving a future reference.
- Long-term conditioning: Long-term target poses, text, or object constraints multiply the sampled time gap by 4, allowing motion generation without near-term constraints.The longer gaps are intended for long-term conditioning signals.
- Modality masking: Across episodes, objects are hidden with 20% probability, text with 80% probability, and a long-term target pose is supplied with 20% probability.These masking choices vary which conditioning modalities remain available.
- Implementation: The masking implementation tracks time-gap state, observation-type masks, and position or rotation constraints for conditionable bodies.Its returned mask records whether each supported joint is constrained by translation or rotation.