Source-linked AI summary
OmniControl: Control Any Joint at Any Time for Human Motion Generation
Yiming Xie, Varun Jampani, Lei Zhong, Deqing Sun, Huaizu Jiang
TL;DR
Text-based human motion generation has struggled to incorporate sparse, global spatial constraints over joints beyond the pelvis. OmniControl uses diffusion-based spatial and realism guidance to control any joint at any time with one model, improving pelvis-control realism and accuracy while supporting other joints. The authors report realistic, coherent motions that conform to spatial constraints, while noting persistent foot-skating cases.
Problem
Text-based motion methods struggle to incorporate flexible global spatial constraints over arbitrary joints and sparse keyframes.
Method
OmniControl is a diffusion-based model combining analytic spatial guidance with realism guidance to control any joint at any time using one model.
Results
OmniControl outperforms state-of-the-art methods on pelvis control in motion realism and control accuracy and achieves promising control over other joints.
Takeaways & Limitations
A single model can control multiple joints together while generating motion that conforms to spatial constraints and remains realistic and coherent.
Takeaways & Limitations
The approach still produces many foot-skating cases, and its realism guidance does not perfectly amend whole-body motion.
Abstract
from arXiv · showhide
We present a novel approach named OmniControl for incorporating flexible spatial control signals into a text-conditioned human motion generation model based on the diffusion process. Unlike previous methods that can only control the pelvis trajectory, OmniControl can incorporate flexible spatial control signals over different joints at different times with only one model. Specifically, we propose analytic spatial guidance that ensures the generated motion can tightly conform to the input control signals. At the same time, realism guidance is introduced to refine all the joints to generate more coherent motion. Both the spatial and realism guidance are essential and they are highly complementary for balancing control accuracy and motion realism. By combining them, OmniControl generates motions that are realistic, coherent, and consistent with the spatial constraints. Experiments on HumanML3D and KIT-ML datasets show that OmniControl not only achieves significant improvement over state-of-the-art methods on pelvis control but also shows promising results when incorporating the constraints over other joints.
1 INTRODUCTION
OmniControl targets flexible spatial control over any joint at any time in text-conditioned human motion generation. It combines spatial and realism guidance to improve control accuracy while preserving realistic, coherent motion.
- Existing methods struggle to integrate global spatial constraints for joints beyond the pelvis and to handle sparse control signals.Relative pose representations define other joints with respect to the pelvis, limiting flexible global control.
- OmniControl introduces a diffusion-based model that incorporates flexible spatial control signals over any joint at any given time.It retains relative pose representations while converting generated motion to global coordinates for direct comparison with controls.
- The method combines spatial guidance for control accuracy with realism guidance for coherent whole-body motion.Spatial guidance enforces the controlled joints’ constraints, while realism guidance refines the remaining joints.
- OmniControl outperforms state-of-the-art text-based motion generation methods on pelvis control in both motion realism and control accuracy.The result is reported on the HumanML3D and KIT-ML datasets.
- A single model can control multiple joints together, supporting applications that connect generated human motion with surrounding objects and scenes.Examples include controlling both wrists or controlling the head and a wrist together.
2 RELATED WORK
Text-based human motion synthesis methods have difficulty incorporating flexible spatial constraints. OmniControl addresses this challenge in sequence-level diffusion-based motion generation.
- Auto-regressive methods recursively generate motion frame by frame and are primarily tailored for real-time scenarios.
- Sequence-level methods generate entire fixed-length motion sequences and can integrate with generative models such as VAEs and diffusion models.They support prompts from external sources including text.
- Prior text-based methods struggle to make generated motion follow global spatial controls for arbitrary joints and sparse constraints.Inpainting-based approaches are limited by relative pose representations centered on the pelvis.
- OmniControl is proposed as a sequence-level method for incorporating flexible spatial constraints into text-based human motion synthesis.
3 OMNICONTROL
OmniControl extends diffusion-based text-conditioned motion generation to flexible spatial constraints on arbitrary joints and frames. It combines analytic spatial guidance with realism guidance to balance constraint adherence and coherent whole-body motion.
- Overview: OmniControl represents spatial constraints as joint xyz positions across frames, with unspecified joints set to zero, enabling control of any joint at any keyframe.The model still uses relative pose representations for input and output while converting generated motion to global coordinates for guidance.
- Background: The diffusion model denoises a noisy motion sequence conditioned on text, directly predicting the clean motion x0 at each denoising step.The reverse process starts from Gaussian noise and uses the predicted clean motion to compute the denoising transition mean.
- Background: Relative pose representations make existing inpainting methods struggle with sparse pelvis constraints and controls on non-pelvis joints.Global positions must be converted into relative locations using preceding-frame or pelvis information, which can produce unreasonable velocities or leg lengths.
- Spatial guidance: Analytic spatial guidance computes an L2 distance between generated global joint locations and constraints, then uses its gradient to perturb the predicted mean.Global coordinates are obtained by converting local joint positions through accumulated pelvis translations, rotations, and relative joint positions.
- Realism guidance: Realism guidance learns feature corrections at Transformer layers, densely and implicitly perturbing whole-body motion to complement spatial guidance.A trainable Transformer copy receives the text prompt and spatial control signal, adding learned residual corrections to the original model.
- Realism guidance: Spatial guidance can enforce constraints but may leave other joints unchanged, producing incoherent motion and foot sliding.Because the perturbed position is only a small part of the motion, the diffusion model may ignore needed changes elsewhere in the body.
4 EXPERIMENTS
Experiments on HumanML3D and KIT-ML evaluate OmniControl against prior methods, ablate its guidance components, and examine inference-time and signal-density trade-offs. OmniControl improves pelvis control and supports multiple-joint control with one model.
- Comparison to Other Methods: OmniControl consistently outperforms existing methods on HumanML3D pelvis control across realism and control-accuracy metrics.It reduces FID by 54.1% versus PriorMDM and Average Error by 79.2% versus GMD, while achieving the lowest foot skating ratio.
- Comparison to Other Methods: OmniControl controls pelvis, feet, head, and wrists using a single model instead of separate joint-specific models.The evaluated joints include pelvis, left foot, right foot, head, left wrist, and right wrist.
- Ablation Studies: Spatial guidance decreases Average Error by 90% and improves Trajectory, Location, and Average Error control metrics.Without spatial guidance, generated motion cannot tightly follow the spatial constraints.
- Ablation Studies: Spatial guidance using gradients with respect to predicted µt produces 83.8% lower Average Error than gradients with respect to noised motion xt.The predicted-µt design takes 121 seconds, compared with 99 seconds for the compared perturbation scheme.
- Ablation Studies: Removing realism guidance decreases FID by 50% relative to the full model and produces unreal and incoherent motions.The realism component refines joints that spatial guidance alone may leave unchanged.
- Deeper Dive into OmniControl: The selected spatial-guidance settings balance inference time and Average Error with Ts = 10, Ke << Kl, and Kl = 500.Smaller Ts increases Average Error, while larger Ts or Ke increases inference time without necessarily improving performance.
- Deeper Dive into OmniControl: As spatial-signal density increases, MDM and PriorMDM show rising FID and foot-skating ratio, whereas OmniControl remains stable.Under all density levels, GMD performs worse than OmniControl; dense inpainting controls yield Average Error of 0 for MDM and PriorMDM.
- Deeper Dive into OmniControl: Controlling multiple joints together supports applications that connect isolated human motion with surrounding objects and scenes.The capability uses one OmniControl model.
5 CONCLUSION
The paper concludes that OmniControl controls any human joint at any time in text-based motion generation. Its complementary spatial and realism guidance supports realistic motion that conforms to spatial signals, validated by experiments and ablations on HumanML3D and KIT-ML.
- Conclusion: OmniControl combines complementary spatial and realism guidance to generate realistic motion while conforming to input spatial control signals.The conclusion presents this combination as the basis of the method’s effectiveness.
- Conclusion: Experiments and ablation studies on HumanML3D and KIT-ML validate OmniControl’s effectiveness.
- Conclusion: The acknowledgement credits Google Cloud Program Credits and a Google gift fund for support.
A.1 PSEUDO CODE
The supplied passage identifies training-time hardware context for the baseline methods and OmniControl.
- A.1 PSEUDO CODE: Training-time comparisons use GPU-specific baselines, with OmniControl trained on one NVIDIA RTX A5000 GPU.GMD uses one NVIDIA RTX 3090, while MDM and PriorMDM use one NVIDIA GeForce RTX 2080 Ti.
A.2 MORE IMPLEMENTATION DETAILS
Implementation details specify OmniControl’s training setup, model initialization, diffusion conditioning, and sparse-control evaluation procedure.
- A.2 MORE IMPLEMENTATION DETAILS: OmniControl is trained in PyTorch on one NVIDIA A5000 GPU with batch size 64, AdamW, learning rate 1e −5, and 250,000 iterations.Training takes 29 hours.
- A.2 MORE IMPLEMENTATION DETAILS: The motion diffusion and realism-guidance models resume Tevet et al. pretrained weights and are fine-tuned together.Text prompts are encoded with CLIP and generation uses a classifier-free process.
- A.2 MORE IMPLEMENTATION DETAILS: The implementation controls V frames, where V denotes the number of frames selected for spatial control density.The variance term is clipped as ˆΣt = min(Σt, 0.01).
- A.2 MORE IMPLEMENTATION DETAILS: For sparse-control evaluation, MDM and PriorMDM receive ground-truth velocity inserted at specified times.The adjustment addresses their inability to naturally handle sparse control signals under relative pelvis representations.
A.3 INFERENCE TIME
The paper reports inference times for OmniControl’s submodules, full pipeline, and baseline methods. Measurements use an NVIDIA A5000 GPU, with GMD timing taken from its paper.
- Measurement setting: Inference time is measured on an NVIDIA A5000 GPU.
- Baseline reporting: GMD’s inference time comes from the timing reported in its original paper.
- Inference-time comparison: Table 5 compares inference time across OmniControl’s submodules, full pipeline, and baseline methods.The table distinguishes the motion-generation model used within submodules from the MDM baseline in overall methods.
A.4 MORE DISCUSSIONS ABOUT CONTROLLING MULTIPLE JOINTS
OmniControl evaluates multiple-joint control by randomly sampling one of 57 possible joint combinations for each motion sequence. Its performance is lower than single-joint control, but the comparison uses different ground truths.
- Evaluation design: There are 57 possible combinations for six joint types, making exhaustive evaluation impractical.
- Evaluation design: The study randomly samples one possible joint combination per motion sequence for evaluation.
- Results: Multiple-joint control performs worse than single-joint control, although the comparison is not apple-to-apple because the ground truths differ.
A.5 LIMITATIONS AND FUTURE PLAN
The paper identifies foot skating, slow diffusion inference, and conflicts between joint constraints as limitations. Future work targets better realism, physical plausibility, and computational speed.
- Motion quality: Foot skating remains common because realism guidance does not perfectly amend whole-body motion from spatial control signals.The authors suggest more effective realism designs and physical constraints to reduce skating.
- Computational cost: Diffusion inference requires roughly 1,000 forward passes to produce one result.The authors identify faster computational strategies as a future-work direction.
- Conflicting constraints: Conflicting spatial control signals for two joints can produce unnatural motions.The paper leaves improving this multi-joint conflict case for future work.
A.6 WHY DIDN’T WE USE GLOBAL POSE REPRESENTATION FOR HUMAN MOTION GENERATION
The appendix explains that global pose representations are difficult to train and reduce text-to-motion performance, despite bone-length constraints. It also discusses how control density affects realism and error metrics.
- Global representation: A global joint-position representation caused MDM to fail to converge and produce unreasonable human poses on HumanML3D.The representation uses D = 66 for 22 joints on HumanML3D and D = 63 for 21 joints on KIT-ML.
- Global representation: Adding bone-length loss to global coordinates still caused a significant performance drop for text-based human motion generation.The paper concludes that global coordinates are not optimal for this task.
- Dataset caveat: KIT-ML foot-skating evaluation is omitted because noisy ground-truth foot heights and existing skating make the ratio inaccurate.
- Control density: Higher control density lowers FID and foot skating ratio for OmniControl and GMD, but raises both metrics for MDM and PriorMDM.The authors attribute the difference to whether methods adjust unconstrained joints to preserve realistic whole-body motion.
- Evaluation interpretation: Dense-control Avg. error is zero for MDM and PriorMDM because their inpainting process leaves the input control joints unchanged.
- Metrics: Foot skating ratio measures frames where either contacting foot skids more than 2.5 cm while remaining below 5 cm height.FID measures motion naturalness, while R-Precision and Diversity measure text relevance and motion variability.