PaperlayerView exact revision on arXiv

Source-linked AI summary

TextOp: Real-time Interactive Text-Driven Humanoid Robot Motion Generation and Control

Weiji Xie, Jiakun Zheng, Jinrui Han, Jiyuan Shi, Weinan Zhang, Chenjia Bai, Xuelong Li

arXiv:2602.07439v1cs.ROcs.AI

TL;DR

TextOp addresses the difficulty of driving universal humanoid controllers with commands that remain flexible during execution. It combines streaming language-conditioned motion generation with low-level whole-body tracking, and real-robot experiments show continuous, responsive behavior across challenging motions. The paper also identifies missing environment perception and physical reasoning as a current boundary.

  • Problem

    Existing humanoid controllers use fixed trajectories or continuous teleoperation, leaving a gap between flexible language-based intent expression and real-time physical control.

  • Method

    TextOp combines an autoregressive text-conditioned motion diffusion model that generates short-horizon references with a low-level whole-body tracking policy for execution.

  • Results

    Real-robot experiments show continuous responses to language commands, smooth transitions across dancing and jumping, and reliable whole-body motion.

  • Takeaways & Limitations

    Streaming natural language can serve as a continuously revisable control signal for whole-body humanoid motion when paired with motion generation and tracking.

  • Takeaways & Limitations

    TextOp lacks explicit environment perception and interactive physical reasoning, so it cannot adapt motions to obstacles, objects, or dynamic surroundings.

Abstract

from arXiv · show

Recent advances in humanoid whole-body motion tracking have enabled the execution of diverse and highly coordinated motions on real hardware. However, existing controllers are commonly driven either by predefined motion trajectories, which offer limited flexibility when user intent changes, or by continuous human teleoperation, which requires constant human involvement and limits autonomy. This work addresses the problem of how to drive a universal humanoid controller in a real-time and interactive manner. We present TextOp, a real-time text-driven humanoid motion generation and control framework that supports streaming language commands and on-the-fly instruction modification during execution. TextOp adopts a two-level architecture in which a high-level autoregressive motion diffusion model continuously generates short-horizon kinematic trajectories conditioned on the current text input, while a low-level motion tracking policy executes these trajectories on a physical humanoid robot. By bridging interactive motion generation with robust whole-body control, TextOp unlocks free-form intent expression and enables smooth transitions across multiple challenging behaviors such as dancing and jumping, within a single continuous motion execution. Extensive real-robot experiments and offline evaluations demonstrate instant responsiveness, smooth whole-body motion, and precise control. The project page and the open-source code are available at https://text-op.github.io/

I. INTRODUCTION

TextOp addresses the gap between flexible human intent expression and physically executable humanoid control by using streaming language commands to drive a universal whole-body controller. Its two-level design combines interactive motion generation with robust tracking, enabling responsive, continuous motions on real hardware.

  • I. INTRODUCTION: Existing humanoid controllers rely mainly on fixed trajectories or continuous teleoperation, limiting runtime adaptability or long-term autonomy.Predefined trajectories are difficult to modify after execution begins, while teleoperation requires sustained human involvement.
  • I. INTRODUCTION: Interactive language-based motion generation remains insufficiently connected to real-time, physically executable humanoid control.Prior interactive methods primarily target kinematic animation and virtual characters rather than real-robot whole-body control.
  • I. INTRODUCTION: TextOp uses streaming natural-language commands and on-the-fly instruction modification instead of committing to a fixed motion plan.The system continuously interprets evolving text input and generates short-horizon motion references for immediate execution.
  • I. INTRODUCTION: An autoregressive text-conditioned motion diffusion model generates short-horizon kinematic trajectories, while a low-level tracking policy converts them into joint-level commands.The separation allows intent updates while maintaining stable whole-body control on a physical humanoid robot.
  • I. INTRODUCTION: Real-robot experiments demonstrate responsive, smooth, and precise control across continuous transitions such as dancing and jumping.The evaluations include extensive real-robot experiments and offline assessments.
  • I. INTRODUCTION: Robot-skeleton representations and generator-produced motion augmentation align motion generation with robot execution.These choices target robot kinematic structure and reduce the distribution gap between motion datasets and generated reference motions.

B. Interactive Motion Generation

TextOp extends interactive text-conditioned motion generation from virtual characters to physical humanoid control. It autoregressively generates short motion references from streaming language and history, then tracks them with a low-level policy for stable execution.

  • B. Interactive Motion Generation: Most text-driven motion generation systems operate offline, producing complete motion sequences rather than continuously updating them during execution.Interactive methods address this limitation by treating text and motion as time-varying streams.
  • B. Interactive Motion Generation: Prior interactive motion-generation methods primarily target virtual character animation and lack physical robot deployment.TextOp bridges this gap by integrating autoregressive generation with whole-body tracking for real humanoid control.
  • B. Interactive Motion Generation: At each time step, the high-level generator receives language and motion history to produce a future reference sequence of T_future = 8 frames.The generator conditions on T_history = 2 past reference-motion frames.
  • B. Interactive Motion Generation: The low-level tracking policy converts kinematic reference motion and robot state into executable control signals.Its role is to keep the executed motion close to the reference while maintaining stability and smoothness.

B. Data Preparation

TextOp prepares robot-aligned, language-paired motion data and trains complementary generation and tracking components. Its robot-skeleton representation encodes constrained kinematics, while self-rollouts and generated-motion augmentation reduce deployment mismatch.

  • B. Data Preparation: Training data combines retargeted AMASS motion capture with private dance and martial-arts data, filtered for reliable robot tracking.Sequences are mapped from SMPL to the 29-DoF robot skeleton and resampled to 50 Hz.
  • B. Data Preparation: Filtered motions are paired with temporally aligned BABEL language annotations and augmented by mirroring motion sequences together with their text descriptions.Frame-level annotations support language guidance and natural switching during generation.
  • B. Data Preparation: The robot-skeleton representation uses a DoF-based local incremental feature space that enforces robot kinematic constraints and is invariant to global pose.It is designed for constrained single-DoF robotic joints rather than human-style 3-DoF ball joints.
  • B. Data Preparation: The motion generator combines a Transformer-based VAE with a latent diffusion model trained autoregressively on robot motion features.The VAE learns the motion latent space before the LDM is trained to denoise motion latents.
  • B. Data Preparation: Self-rollouts replace segment histories with preceding predicted futures to reduce the distribution gap between training and deployment.This exposes the generator to its own sequential predictions during LDM training.
  • B. Data Preparation: The universal motion tracker is trained with goal-conditioned reinforcement learning in large-scale physical simulation and receives proprioception plus five future reference frames.The policy is optimized with PPO using tracking and regularization rewards.
  • B. Data Preparation: Generated-motion augmentation adds 5,368 clips totaling 31.48 hours, exposing the tracker to variability and noise from generator outputs.The augmentation uses 20-second text streams sampled from BABEL annotations.

E. Deployment

TextOp is evaluated through real-robot demonstrations, robustness tests, tracking-fidelity measurements, and latency analysis. The deployment experiments examine diverse continuous behaviors, command modification, perturbation recovery, and responsiveness.

  • TextOp evaluates real-world performance through qualitative demonstrations, quantitative robustness tests, and real-time measurements.
  • Complex Skill Demonstrations and Interactive Control: The robot performs diverse skills and seamlessly sequences varied commands in a single unbroken trial.Demonstrated behaviors include locomotion, dance, martial arts, and gestures.
  • Complex Skill Demonstrations and Interactive Control: Users can modify commands during execution, enabling smooth composition and transitions between skills.
  • Robustness Analysis: TextOp maintains high success rates and tracking accuracy across 30-second trials with random or fixed command streams and recovers from manual perturbations.The perturbation tests assess whether ongoing motions and text-driven commands are preserved after disturbances.
  • Real-Time Performance: 0.73 s is the average user interaction latency from typing a command to the robot’s physical response.The system measures latency across language input, motion generation, and low-level tracking stages.

C. Offline Evaluation

Offline evaluation tests TextOp’s motion generator on text-conditioned segments and transitions, then examines the tracking policy on diverse reference motions. The proposed representation achieves the strongest overall generation performance across most reported metrics.

  • Offline evaluations isolate the motion generator and tracking policy using models trained on public datasets.
  • 1) Motion Generation Evaluation: The motion generator is evaluated on BABEL text streams for complete command segments and transitions between consecutive segments.
  • 1) Motion Generation Evaluation: The proposed motion representation achieves state-of-the-art performance for individual text commands and high-quality transitions between consecutive segments.
  • 1) Motion Generation Evaluation: Its gains are attributed to a compact DoF-based representation that reduces modeling complexity and an incremental root-state representation that improves data efficiency.
  • 1) Motion Generation Evaluation: Transition smoothness is slightly below DART+Retarget because retargeting introduces smoothness in joint displacements.
  • 2) Motion Tracking Evaluation: Tracking evaluation focuses on robust execution of diverse references, especially generator-produced trajectories, and on the effect of generated-motion augmentation.

2) Motion Tracking Evaluation:

Motion-tracking experiments compare training configurations on generator-produced trajectories and unseen SnapMoGen data. Generated-motion training improves deployment-oriented tracking, while motion-capture training preserves generalization and combined training balances both settings.

  • The evaluation compares trackers trained on combined motion-capture and generated data, generated data alone, motion-capture data alone, and pretrained baselines.
  • Simulation evaluates generator-produced references and unseen SnapMoGen motion-capture data to measure deployment performance and generalization.
  • All TextOp variants achieve consistently high success rates and low tracking errors on generator-produced motions, with TextOp-G performing best overall.
  • TextOp-M+G improves upon TextOp-M across several tracking-fidelity and motion-quality metrics on generator-produced data.
  • On unseen SnapMoGen data, TextOp-M generalizes best, TextOp-G drops noticeably, and TextOp-M+G provides a competitive balance across both settings.
  • TextOp’s robot-skeleton features improve generated-motion quality, while generator-produced training motions align the tracker with its deployment distribution.
  • TextOp lacks explicit environment perception and interactive physical reasoning, limiting adaptation to obstacles, objects, and dynamic surroundings.

APPENDIX A ADDITIONAL IMPLEMENTATION DETAILS

The appendix details TextOp’s data preparation, robot-motion representation, and motion-generation architecture. It also describes continuity-focused training, self-rollout, and diffusion-based learning procedures.

  • Data Preprocessing and Dataset Construction: Motion data are retargeted, resampled, paired with language annotations, augmented, segmented, and filtered using tracking reliability.The preprocessing pipeline converts human motions to the G1 robot format and removes motions that fail tracking checks.
  • Robot Skeleton Motion Representation: The robot-skeleton representation uses local incremental motion features and an exactly invertible forward–inverse transformation.The representation includes root orientation, yaw increments, local translation, contact indicators, root height, joint positions, and joint increments.
  • Architecture: The generator uses a transformer-based VAE and latent diffusion model to reconstruct and predict future motion from historical motion and text.The LDM operates on noisy latent variables, diffusion timesteps, motion history, and text embeddings.
  • Training Details: Training organizes motion into overlapping history–future primitives to preserve temporal continuity between adjacent predictions.A self-rollout curriculum replaces dataset history with model-generated history with probability capped at 0.8.
  • Diffusion Training: The diffusion process gradually adds Gaussian noise to latent motion through a K-step Markov chain before training the model to predict the original latent.The noise schedule uses β_k, α_k = 1−β_k, and cumulative products of α_s.

C. Motion Tracking Policy

The motion-tracking policy is trained with an asymmetric actor–critic design in which the critic receives additional privileged information.

  • Motion Tracking Policy: The tracking policy uses asymmetric actor–critic training, with privileged information supplied to the critic network.The appendix refers to separate descriptions of observations, rewards, termination conditions, and domain randomization.

APPENDIX B ADDITIONAL EXPERIMENTAL DETAILS

The appendix supplements the paper with experimental setup information, including tracker observations, rewards, termination conditions, and domain-randomization settings.

  • Experimental Setup: The motion tracker’s observation, reward, termination, and domain-randomization configurations are documented in Tables VII–X.The observation table distinguishes base policy inputs from critic-only privileged inputs.
  • Experimental Setup: Base observations have dimensionality 431, increasing to 557 when privileged inputs are included.The table lists 14 key bodies used in the observation space.

A. Experimental Procedure

The appendix specifies real-robot deployment, command-stream construction, latency measurement, motion-generation evaluation, and the metrics used for tracking and motion quality.

  • Experimental Procedure: Real-robot trajectories combine motor-sensor joint angles with pelvis position and orientation from the G1’s built-in odometry.This procedure is used for quantitative evaluation of real-robot executions.
  • Experimental Procedure: Random-command trials use 30-second text streams assembled from sampled BABEL action descriptions with randomized word durations.Generated motions exceeding 2 meters of displacement from the initial position are discarded before execution.
  • Experimental Procedure: Real-time performance is assessed by measuring text encoding, motion generation, tracking-policy, and user-interaction latency.Module latency is measured over 100 or 1000 runs, while command switching is tested on the physical robot.
  • Motion Generation Evaluation: Motion-generation evaluation separates segment-level text–motion quality from transition-level smoothness and coherence.Transition clips contain 15 frames before and after each textual boundary.
  • Evaluation Metrics: Peak Jerk and Area Under the Jerk quantify instantaneous and cumulative deviation from smooth transition dynamics.Peak Jerk uses the maximum absolute jerk, whereas AUJ compares instantaneous jerk with the dataset level j_avg.
  • Evaluation Metrics: Success rate requires complete real-world execution without falling, hardware malfunction, or workspace-boundary violations.In simulation, success instead requires root-relative tracking error to remain below θ = 0.3 m throughout the trajectory.

C. Baseline Methods

This section describes baseline humanoid motion-tracking systems and the motion representations used for comparison. The baselines include hierarchical, unified-policy, and two-stage reinforcement-learning approaches.

  • Baseline Methods: TWIST2 is a scalable whole-body demonstration system whose low-level controller serves as a task-agnostic motion-tracking engine.The experiment uses its officially released low-level-controller checkpoint.
  • Baseline Methods: GMT trains a unified humanoid motion-tracking policy using adaptive sampling and a Motion Mixture-of-Experts architecture.The components balance easy and difficult motions during training and specialize across regions of the motion manifold.
  • Baseline Methods: Any2Track uses two-stage reinforcement learning with AnyTracker for general motion tracking and AnyAdapter for history-informed online dynamics adaptation.The experiment uses the officially released AnyTracker checkpoint because the second-stage adaptation module is not open-source.
  • Motion Representation: Table XII compares which motion-representation components are used by TextOp and the baseline methods.The comparison reports actual feature-vector components, with ✓ indicating use and × indicating non-use; skeleton-specific terms are defined in the table caption.

D. Additional Ablation Studies

This section reports additional ablations of Motion Generation hyperparameters and supplementary real-world robot results. The ablations examine trade-offs among distribution matching, semantic alignment, and motion smoothness.

  • Additional Ablation Studies: The ablations vary history and future lengths, primitive count, LDM size, guidance scale, denoising steps, and maximum rollout probability.The selected parameters reflect a trade-off among distribution matching, semantic alignment, and motion smoothness.
  • Additional Real-World Results: Additional real-world robot results are shown in Fig. 7.The figure provides supplementary demonstrations of continuous diverse skill execution.
  • Experimental Setup: Training uses an NVIDIA A100-PCIE GPU with 40 GB memory and an Intel Xeon Gold 6348 CPU.The training environment runs Ubuntu 22.04.3 LTS.
  • Experimental Setup: Deployment uses an NVIDIA GeForce RTX 4090 GPU and an Intel Core i7-13700 workstation.The deployment environment runs Ubuntu 20.04.6 LTS.
  • Additional Ablation Studies: Table XIII reports ablation results as mean ± standard deviation over three generator rollout seeds.The table evaluates key hyperparameters for Motion Generation.
Loading 2602.07439v1…