PaperlayerView exact revision on arXiv

Source-linked AI summary

ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generation

Kaifeng Zhao, Mathis Petrovich, Haotian Zhang, Tingwu Wang, Siyu Tang, Davis Rempe

arXiv:2607.08741v1cs.GRcs.CVcs.LGcs.RO

TL;DR

Interactive motion generation needs both real-time responsiveness and control through complex text and long-horizon kinematic constraints. ARDY combines hybrid motion representation with autoregressive diffusion, achieving strong motion fidelity and control accuracy on HumanML3D while supporting interactive demonstrations.

  • Problem

    Existing offline methods offer expressive text and kinematic control but are too slow, while online methods limit controllability, text understanding, or long-horizon constraint execution.

  • Method

    ARDY uses a hybrid explicit-root and latent-body representation with a two-stage autoregressive diffusion model supporting online text and long-horizon kinematic constraints.

  • Results

    ARDY outperforms existing methods on HumanML3D in motion fidelity and control accuracy, while demonstrating diverse interactive text and kinematic control capabilities.

  • Takeaways & Limitations

    ARDY demonstrates that interactive human motion generation can combine real-time operation with online prompting and diverse flexible kinematic control.

  • Takeaways & Limitations

    Using all past motion frames as autoregressive history can be inefficient for extremely long-horizon tasks.

Abstract

from arXiv · show

Generating realistic 3D human motions in real-time within interactive applications is key for animation, simulation, and humanoid robotics. While recent offline motion generation approaches offer precise control via text and kinematic constraints, they lack the inference speed required for interactive settings. Conversely, existing online methods enable real-time synthesis but often sacrifice controllability or struggle with complex text semantics and long-horizon goals due to limited context windows. In this work, we introduce ARDY, a streaming generation framework that bridges this gap by enabling high-fidelity motion generation controllable via online text prompts and flexible kinematic constraints. ARDY employs a hybrid representation that combines explicit root features with a latent body embedding, balancing precise trajectory control with efficient generative learning. We propose a two-stage autoregressive transformer denoiser that features variable history context and supports conditioning on flexible, long-horizon kinematic constraints. By training on a large-scale motion capture dataset and being directly conditioned on text labels and kinematic constraints sampled from ground truth poses, ARDY natively learns controllable generation that supports online prompting and flexible long-horizon goals. Extensive evaluations on the HumanML3D benchmark and the large-scale, high-fidelity Bones Rigplay dataset demonstrate ARDY's high motion quality and constraint adherence, validating the efficacy of our key architectural decisions. Finally, we demonstrate the method's practical versatility through an interactive demo featuring dynamic text control, diverse keyframe pose constraints, path following, and interactive locomotion control via mouse and keyboard. Supplementary video results, code, and model releases can be found at https://research.nvidia.com/labs/sil/projects/ardy/.

1 Introduction

ARDY targets real-time human-motion generation while retaining the text controllability and flexible kinematic constraints of offline methods. It combines a hybrid pose representation with a two-stage autoregressive diffusion denoiser supporting variable history and long-horizon goals, and validates these choices through extensive evaluation and an interactive demo.

  • Motivation: Offline diffusion and masked-modeling methods offer expressive text and kinematic control, whereas online autoregressive generators are fast but often sacrifice one or both forms of controllability.Some online methods support text without kinematic control, while others support kinematic constraints without text input.
  • ARDY: ARDY combines complex text prompts and flexible kinematic goals with streaming generation for online interactivity.Its kinematic goals can extend beyond a single generation window, supporting long-horizon objectives.
  • Control: The model supports variable history context and long-horizon conditioning on full-body keyframes, root waypoints, root paths, and end-effector positions and rotations.These controls are identified as key contributions of the paper.
  • Architecture: Its interleaved two-stage autoregressive denoiser first predicts the clean explicit root, then predicts the clean latent body embedding conditioned on that root.The two stages operate within the denoising loop to maintain mutual influence between root and body motion.
  • Evaluation and demonstration: An interactive demo showcases dynamic text control, dense and sparse key-pose constraints, path following, and real-time mouse-and-keyboard locomotion control.The method is also evaluated on the large-scale, production-quality Bones Rigplay dataset.
  • Architecture: ARDY uses a hybrid latent-body and explicit-root representation for fast, controllable motion generation.The representation balances explicit root control with a latent body embedding.

2 Related Work

Prior work spans offline text- and constraint-conditioned motion generation and interactive autoregressive methods, but interactive approaches generally trade context or control for speed. ARDY combines real-time generation, online text prompting, variable history, and flexible long-horizon kinematic constraints through a two-stage diffusion model over hybrid motion representations.

  • Offline Human Motion Generation: Offline methods achieve high-quality text-conditioned motion and flexible “any-joint-any-time” kinematic control, but iterative denoising can be too slow for interactive applications.Diffusion models capture complex text–motion distributions and support flexible kinematic constraints, while their potentially long iterative denoising process limits interactivity.
  • Offline Human Motion Generation: Tokenized motion methods generate text-conditioned motion with masked transformers, while some support precise kinematic control through test-time optimization.These approaches use VQ-VAE motion tokenizers followed by masked transformers that iteratively predict masked poses.
  • ARDY: ARDY provides real-time generation with online text prompting, variable-length history contexts, flexible long-horizon kinematic constraints, and hybrid latent–explicit root representations.Its two-stage diffusion architecture denoises a hybrid combination of latent tokenized body and explicit root representations, yielding a capability combination unmatched by prior works.
  • Interactive Motion Generation: Autoregressive motion diffusion adapts offline generation to interactive settings primarily through shorter generation horizons and fewer denoising steps.Prior methods include next-pose denoising and support for kinematic constraints through inpainting or reinforcement learning.
  • Interactive Motion Generation: Latent interactive methods improve streaming generation but can be limited by short history context or lack of kinematic goal constraints.MotionStreamer uses variable history length and is more robust to complex prompts, but lacks kinematic goal constraints.

3 Method: ARDY

ARDY combines a hybrid motion representation with a motion tokenizer and a two-stage autoregressive diffusion model for real-time, text- and spatially conditioned generation. Its design preserves explicit root control while using compact latent body tokens and supports future kinematic goals through causal generation.

  • Architecture: ARDY comprises a motion tokenizer and an autoregressive two-stage motion diffusion model that denoises hybrid tokens containing latent body and explicit root motion.The tokenizer learns a compact body-motion representation before diffusion generation.
  • Hybrid Representation: The hybrid representation decouples motion by keeping root trajectories explicit and interpretable while encoding body motion in a compact latent space.This balances efficient generative learning with precise control through explicit feature overwriting.
  • Motion Tokenizer: The tokenizer uses a causal asymmetric conditional autoencoder to patch consecutive body frames into latent tokens, concatenate them with root features, and reconstruct body motion.The decoder converts global root motion to a local representation for conditional reconstruction, while causal attention preserves temporal ordering.
  • Autoregressive Generation: The autoregressive formulation conditions each short future-motion window on text, up to H previous tokens, and spatial goals extending up to F future tokens.Goals can specify joint positions or rotations at future timesteps, including waypoints, paths, pose keyframes, and end-effector constraints.
  • Diffusion Model: The interleaved two-stage diffusion model separates root and body denoising, overwrites root constraints for accurate trajectories, and supplies body goals and masks for detailed pose control.The decomposition is motivated by predicting body motion from clean root motion being easier than jointly generating root and body.
  • Training Objective: The reconstruction objective includes a foot-skating loss weighted at 0.01 to penalize velocities of foot joints predicted to contact the ground.This enforces stationary constraints during contact phases.

4 Interactive Motion Generation Demo

ARDY’s interactive demo enables real-time motion generation from streaming text prompts and mouse- or keyboard-driven spatial and locomotion constraints. Its autoregressive, latency-aware replanning supports long-horizon goals, diverse kinematic constraints, and efficient interactive control.

  • Interactive interface: The Viser interface supports real-time character control through streaming text prompts and interactive spatial constraints from mouse and keyboard inputs.The demo showcases ARDY’s test-time operation and qualitative results.
  • Autoregressive operation: ARDY generates motion autoregressively in windows of G frames, using previous predictions as history and truncating future constraints to a configurable context of up to 8 seconds.Constraints beyond the truncated future horizon enter conditioning only when generation advances within range.
  • Latency-aware replanning: Latency-aware replanning responds to new prompts, modified constraints, or buffer depletion while using buffered frames to maintain smooth transitions despite model inference latency.The non-blocking buffer is simultaneously played back and fed into the generation thread as history context.
  • Interactive performance: On an RTX 4090, average generation latency is 33 ms for the 4-step model and 63 ms for the 10-step model, with the latter slightly improving control accuracy.Both models use a generation window of G=40 frames.
  • Constraint-conditioned generation: The demo handles complex text and kinematic constraints spanning 10-second target goals, trajectory following, and full-body keyframes.These results demonstrate adaptation to dynamic text changes and robust constraint satisfaction.
  • Interactive locomotion control: Users control locomotion through mouse waypoints that define smoothed root paths or keyboard commands that modulate target root velocity in real time.Mouse trajectories are linearly interpolated and smoothed, while keyboard targets use the input and current velocity.

5 Analysis on Large-Scale Mocap Data

ARDY is analyzed on the large-scale Bones Rigplay dataset through constraint-conditioned evaluations, architectural ablations, and hyperparameter studies. Results support the hybrid representation, local-root decoding, two-stage denoising, and balanced generation settings for motion quality and constraint adherence.

  • Evaluation Setup: Bones Rigplay provides around 700 hours of diverse, studio-quality human motion with text descriptions for evaluating ARDY design variations.Its scale and quality enable a more robust testbed than smaller public datasets such as HumanML3D.
  • Evaluation Setup: Constraint-conditioned tests cover dense root trajectories, sparse waypoints, full-body keyframes, and end-effector position and orientation goals.Constraints are sampled from ground-truth test poses alongside text prompts, with slight translation and heading perturbations applied to some inputs.
  • Architectural Ablations: Removing global-to-local root conversion increases foot skating, indicating that local root representations preserve motion quality and physical plausibility.This comparison evaluates the tokenizer decoder’s root-conversion process against a decoder operating directly on the global root representation.
  • Architectural Ablations: The two-stage denoiser outperforms a one-stage baseline in text-conditioned fidelity and spatial constraint accuracy.The architecture separately predicts root trajectory and latent body motion tokens rather than predicting them jointly.
  • Hyperparameter Analysis: Extending the generation horizon improves FID and R-Precision, whereas a 4-frame horizon causes instability and degraded performance.An 8-frame horizon responds more rapidly to updated text prompts and learns constraint adherence faster than a 40-frame horizon.
  • Hyperparameter Analysis: Extremely few diffusion steps worsen generation quality and constraint adherence, while increasing steps yields slight gains in FID, R-Precision, and constraint accuracy.Few-step models nevertheless remain highly competitive, demonstrating efficient generation with the learned hybrid representation.
  • Tokenizer Analysis: A single-frame tokenizer patch learns faster initially but becomes unstable, while an 8-frame patch slightly improves FID and R-precision at the cost of skating and constraint accuracy.The results expose a trade-off between compression, motion quality, and constraint adherence.
  • Tokenizer Analysis: Tokenizer latent-space configurations perform similarly, while vanilla autoencoders can diverge at long horizons such as 40 frames.The default FSQ configuration uses 64 quantization levels and 128 dimensions, denoted FSQ 64-128.

6 Benchmark Evaluation

ARDY is evaluated on HumanML3D against offline and autoregressive state-of-the-art baselines for text- and constraint-conditioned motion generation. The evaluation measures text following, motion quality, and spatial constraint accuracy across arbitrary joint goals and long-horizon settings.

  • Evaluation Setup: ARDY is evaluated on HumanML3D using a 40-frame generation horizon, 10 diffusion steps, and a vanilla autoencoder tokenizer.HumanML3D contains around 30 hours of motion data with corresponding text descriptions; the HumanAct12 subset is excluded, while native SMPL joint rotations are preserved during retargeting.
  • Evaluation Metrics: The benchmark reports Top-3 R-precision for text following, FID and foot skating ratio for motion quality, and spatial control errors for constraint accuracy.FID measures similarity to the ground-truth motion distribution, while foot skating ratio measures the frequency of detected foot-skating frames.
  • Offline Comparison: Against MaskControl, ARDY is evaluated on arbitrary joint-position constraints at any frame, with comparisons both before and after test-time optimization.MaskControl is described as an offline state-of-the-art model specializing in accurate joint controls, and ARDY’s method yields significantly lower spatial control errors.
  • Autoregressive Comparison: Against DiP, ARDY surpasses the autoregressive baseline in both in-horizon and out-of-horizon constraint settings.DiP’s out-of-horizon joint error rises sharply, whereas ARDY maintains high accuracy for goals placed far into the future, supporting long-context constraint handling.

7 Discussion

ARDY enables interactive, controllable human motion generation with online text prompting and flexible kinematic goals, including long-horizon constraints. Experiments and demonstrations support its effectiveness, while limitations remain in extremely long-horizon history efficiency and diffusion-model dependence.

  • Contributions: ARDY supports interactive human motion generation with online text prompting and flexible kinematic goal constraints, including goals extending beyond one generation window.The framework is designed for interactive applications and supports long-horizon goals.
  • Demonstration: A real-time demonstration showcases interactive and instructable motion generation for future animation systems.The demonstration highlights ARDY’s practical potential in interactive animation settings.
  • Validation: Ablation studies on Bones Rigplay and experiments on HumanML3D validate ARDY’s architectural choices and show stronger motion fidelity and control accuracy than existing methods.The validation uses the large-scale, studio-quality Bones Rigplay dataset and the public HumanML3D benchmark.
  • Limitations: Using all past motion frames as autoregressive history can be inefficient for extremely long-horizon tasks.The paper identifies structured memory representations and update mechanisms as an important future direction.
Loading 2607.08741v1…