Source-linked AI summary
Kimodo: Scaling Controllable Human Motion Generation
Davis Rempe, Mathis Petrovich, Ye Yuan, Haotian Zhang, Xue Bin Peng, Yifeng Jiang, Tingwu Wang, Umar Iqbal, David Minor, Michael de Ruyter, Jiefeng Li, Chen Tessler, Edy Lim, Eugene Jeong, Sam Wu, Ehsan Hassani, Michael Huang, Jin-Bey Yu, Chaeyeon Chung, Lina Song, Olivier Dionne, Jan Kautz, Simon Yuen, Sanja Fidler
TL;DR
Human motion applications need scalable, high-quality data and intuitive control, while existing datasets and acquisition approaches remain limited. Kimodo addresses this with a large-scale optical-mocap-trained diffusion model whose text and kinematic-constraint control supports high-quality motion authoring; its design reduces artifacts and constraint errors, but generation remains offline and takes several seconds.
Problem
Human motion applications need high-quality, diverse data and accessible control, while common benchmarks remain small and generative models must improve motion quality, directability, and coverage.
Method
Kimodo is a kinematic motion diffusion model using explicit motion diffusion, a two-stage root-and-body architecture, and text conditioning plus diverse kinematic constraints.
Results
Kimodo generates high-quality human motions controlled through text and varied kinematic constraints, while ablations show that one-stage prediction increases foot skating and constraint errors.
Takeaways & Limitations
Kimodo provides a foundation for scalable human-motion authoring and can be applied to robot motion after retargeting the optical-mocap training data.
Takeaways & Limitations
Kimodo is designed for offline authoring and can take several seconds to generate motion, limiting direct use for runtime control in robotics and digital-twin simulation.
Abstract
from arXiv · showhide
High-quality human motion data is becoming increasingly important for applications in robotics, simulation, and entertainment. Recent generative models offer a potential data source, enabling human motion synthesis through intuitive inputs like text prompts or kinematic constraints on poses. However, the small scale of public mocap datasets has limited the motion quality, control accuracy, and generalization of these models. In this work, we introduce Kimodo, an expressive and controllable kinematic motion diffusion model trained on 700 hours of optical motion capture data. Our model generates high-quality motions while being easily controlled through text and a comprehensive suite of kinematic constraints including full-body keyframes, sparse joint positions/rotations, 2D waypoints, and dense 2D paths. This is enabled through a carefully designed motion representation and two-stage denoiser architecture that decomposes root and body prediction to minimize motion artifacts while allowing for flexible constraint conditioning. Experiments on the large-scale mocap dataset justify key design decisions and analyze how the scaling of dataset size and model size affect performance.
1. Introduction
Human motion data is increasingly needed across robotics, simulation, and interactive media, but existing acquisition methods are costly, slow, or difficult to scale. Kimodo addresses this need with a large-scale motion diffusion model designed for high-quality, controllable, and diverse motion authoring.
- Motivation: Robotics, digital twins, and games increasingly require large amounts of plausible human motion data.Applications include realistic humanoid behavior, dynamic simulated environments, and interactive digital humans.
- Motivation: Hand animation, studio mocap, teleoperation, and monocular video reconstruction each impose substantial cost, time, instrumentation, or quality challenges.These limitations make conventional motion acquisition difficult to apply broadly.
- Design goals: An ideal motion-generation model should match optical mocap quality, offer intuitive directable controls, and produce diverse motions for many applications.The stated goals combine quality, accessibility, and breadth of generated behaviors.
- Kimodo: Kimodo trains a kinematic diffusion model on 700 hours of studio mocap to support text prompting and full-body, joint-level, waypoint, path, and foot-contact constraints.Its two-stage architecture separates root and body motion to improve specification following while reducing floating and foot-skating artifacts.
- Practical interface: The project includes an interactive authoring interface and released models for the SOMA body model and Unitree G1 robot.The interface demonstrates practical motion authoring across human and robot skeletons.
2. Key Results
Kimodo supports text-driven and constraint-driven motion authoring across human and robot embodiments, with diverse outputs and precise constraint accuracy. Scaling data, model capacity, and batch size improves different aspects of controllability and motion quality.
- Text control: Kimodo generates realistic SOMA motions across locomotion, daily activities, dancing, combat, and varied styles from text prompts.Prompts can describe multiple actions performed sequentially or simultaneously.
- Text control: Ten samples from one put-down prompt show plausible variations in hand use, height, and timing.The same prompt therefore produces behaviorally diverse motion outcomes.
- Multi-prompt generation: Multiple prompts can be chained with plausible transitions when a single prompt struggles to specify a longer action sequence.The authoring demo generates the segments sequentially and uses constraints between them for continuity.
- Kinematic control: Kimodo supports full-body keyframes, end-effector constraints, sparse 2D waypoints, and dense 2D paths that can be mixed for motion authoring.These controls support in-betweening, navigation-based path specification, and object-interaction planning.
- Kinematic control: 3.21 cm full-body keyframe joint error, 3.63 cm end-effector position error, 6.88 deg end-effector rotation error, and 3.63 cm root error are reported averages.The evaluation covers a diverse suite of constraint-conditioned generation cases.
- Scaling behavior: Increasing dataset size improves constraint following, while model size improves R-precision and FID; larger batch size further benefits text following.The reported scaling study varies data size, model size, and number of GPUs.
- Robot demonstrations: Kimodo can generate G1 robot demonstrations that are tracked by a physics-based humanoid policy and deployed on a real-world robot.Batched generation and constraints can enlarge and diversify demonstration datasets.
3. Large-Scale Motion Capture Dataset
Kimodo is trained and evaluated using Bones Rigplay, a large optical motion-capture dataset spanning many subjects, actions, styles, and text descriptions. The dataset is standardized across skeletons and augmented to support varied text and compositional motion training.
- Dataset scale and coverage: Bones Rigplay contains 700 hours of optical mocap from 170 human subjects with roughly equal male and female participation.Its motions cover locomotion, gestures, everyday activities, object interactions, combat, dancing, and athletics.
- Dataset scale and coverage: The dataset includes many styles, including tired, angry, happy, sad, scared, drunk, injured, stealthy, old, and childlike performances.Multiple subjects perform actions across the collection, broadening behavioral and stylistic coverage.
- Representation: Body-only motions are standardized to a uniform 27-joint skeleton and retargeted to SOMA, Unitree G1, and SMPL-X variants.The native skeleton is used for quantitative experiments, while retargeted versions support model variations.
- Text annotations: Each mocap sequence includes overview descriptions, fine-grained atomic-action descriptions, and augmented LLM paraphrases.These annotations provide multiple textual views of the same motion data.
- Training augmentation: Training combines full clips, action sub-clips, stitched clips, original descriptions, and LLM paraphrases sampled from a prespecified distribution.The augmentations target varied prompt detail and compositions of multiple actions.
4. Method: Kimodo
Kimodo uses an explicit motion diffusion model with a carefully designed representation and two-stage denoiser to support high-quality motion generation under text and kinematic constraints. Its mostly global features enable sparse world-space controls, while decomposed root and body prediction improves coordination and artifact avoidance.
- Core framework: Kimodo applies explicit motion diffusion directly to pose features, enabling text conditioning and intuitive kinematic control through direct pose-feature imputation.The denoiser iteratively predicts clean motion from noisy motion, conditioning signals, and the denoising step.
- Motion representation: The motion representation combines smoothed global root position, global heading, joint positions, velocities, rotations, and foot-contact features.The representation separates each pose into a global root component and a body component.
- Motion representation: A mostly global representation supports sparse root, joint-position, and world-space joint-rotation constraints without requiring temporal integration.The design also avoids heading-based canonicalization that can introduce discontinuities during abrupt heading changes.
- Motion representation: The smoothed root separates natural pelvis sway from a stable trajectory, allowing smooth 2D waypoint and path following while preserving natural pelvis movement.The trajectory is intended to provide a stable frame of reference and emulate paths drawn in animation tools.
- Two-stage denoiser: The two-stage denoiser predicts global root motion first, transforms it into a local representation, and then predicts body motion before concatenating both outputs.Both stages use 16-layer, 8-head transformers with latent size 1024; the full model has 282 M learnable parameters.
- Two-stage denoiser: Interleaved root and body corrections occur at every denoising step, supporting alignment while combining global-root constraints with invariant local conditioning.Ablations of the two-stage model and local root representation produce worse performance, according to the discussion.
- Generation: Direct constraint conditioning made gradient-based test-time guidance minimally beneficial, substantially slower, and generally unstable or difficult to tune.Users can adjust text and constraint conditioning weights to vary their influence on the output.
5. Related Work
Kimodo builds on text-conditioned motion generation and kinematic-control methods while targeting broader constraint coverage and large-scale training. Its explicit pose-space diffusion and direct imputation avoid several additional control mechanisms used by prior systems.
- Motion diffusion: Kimodo uses explicit pose-space diffusion because pose features support direct controllability through imputation.This differs from latent approaches that denoise in a learned latent space for efficiency.
- Kinematic control: Earlier control methods include ControlNet for dense and sparse positions, imputation with test-time guidance for sparse positions, and reinforcement-learning policies over diffusion models.These approaches provide context for Kimodo’s broader direct-imputation control interface.
- Kinematic control: Kimodo supports sparse and dense positional and rotational constraints through direct imputation without ControlNet fine-tuning, test-time guidance, optimization, or reinforcement learning.Its smoothed root representation also supports root constraints common in motion editing.
6. Quantitative Evaluation
The evaluation uses the large-scale Bones Rigplay dataset to test text-to-motion, text-plus-constraint generation, motion quality, and constraint accuracy. Ablations and scaling analyses show that architectural choices, more data, larger models, and larger batches affect different performance dimensions.
- Evaluation Setup: Evaluation uses novel-behavior test splits from Bones Rigplay, with about 5k held-out motions on the native 27-joint skeleton.Ten percent of motions are held out, with splits based on unique behaviors absent from training.
- Evaluation Setup: The test suite covers unconstrained text-to-motion with overview and fine-grained prompts, plus diverse text-and-kinematic-constraint cases.Constraints include keyframes, end-effector positions and rotations, 2D waypoints, and paths.
- Evaluation Setup: The evaluation reports Top-3 R-precision for text-following, TMR-based FID for motion quality, foot skate, and constraint distance errors.Constraint errors are separated into full-body positions, end-effector position/rotation, and 2D root position.
- Ablation Study: The one-stage denoiser preserves text-following but substantially increases foot skating and constraint errors relative to the two-stage design.The comparison uses baselines with similar parameter counts.
- Ablation Study: The smoothed root representation reduces foot skate while maintaining constraint accuracy, whereas direct pelvis projection can produce unnatural stealthy or elderly walking styles.The direct-pelvis baseline has lower 95th-percentile pelvis error but can optimize for the path at the expense of natural motion.
- Scaling Analysis: 10% of the training data significantly worsens foot skate and constraint accuracy, while increasing model size and batch size generally improves performance across metrics.Larger models particularly improve text-following and motion quality, whereas larger batches particularly improve text-following.
7. Conclusion
Kimodo is a large-scale kinematic motion diffusion model for high-quality human-motion authoring through text and diverse kinematic constraints. The authors position optical-mocap training as a foundation for scaling motion generation, while identifying runtime control, mixed data quality, and scene interaction as future challenges.
- Conclusion: Kimodo generates high-quality human motions from text and supports a variety of kinematic constraints for intuitive authoring.The model can be retargeted for direct robot-motion generation.
- Future Challenges: Kimodo is offline and takes several seconds to generate motion, whereas robotics and digital-twin simulation require runtime control that reacts to changing environments.The authors suggest learned latent-space diffusion and autoregressive reformulation as possible directions.
- Future Challenges: Combining clean mocap with noisy video-derived data and modeling scene and object interactions remain open challenges.The paper notes that gathering interaction data is especially difficult.
B. Contributors •
The paper credits contributors across project leadership, modeling, data collection, advising, and related research roles.
- Contributors: Davis Rempe is listed as project lead, with modeling, data, and advising contributors named separately.