PaperlayerView exact revision on arXiv

Source-linked AI summary

MotionVLA: Vision-Language-Action Model for Humanoid Motion

Nonghai Zhang, Siyu Zhai, Yanjun Li, Zeyu Zhang, Zhihan Yin, Yandong Guo, Boxin Shi, Hao Tang

arXiv:2606.15142v1cs.CVcs.RO

TL;DR

Humanoid motion generation must represent low-frequency pose semantics and high-frequency physical dynamics without forcing them into one tokenization space. MotionVLA addresses this with a dual-stream frequency tokenizer and unified autoregressive model, achieving stronger diversity alignment on HumanML3D and multimodal consistency on MBench.

  • Problem

    Single-codebook motion tokenizers treat heterogeneous pose and velocity signals alike, although their frequency statistics differ substantially.

  • Method

    MotionVLA uses DSFT to independently tokenize Base pose-semantic and Phys physical-dynamic streams, then generates Phys tokens after Base tokens in one autoregressive sequence.

  • Results

    MotionVLA achieves the best Motion-Condition Consistency and Foot Sliding on MBench while attaining HumanML3D Diversity closest to real data, improving consistency from 0.53 to 0.55.

  • Takeaways & Limitations

    The results support frequency-aware dual-stream decoupling as a formulation for balancing multimodal condition alignment, motion diversity, and physical motion quality.

  • Takeaways & Limitations

    The study uses a lightweight 2B backbone and limited benchmarks, so its scaling behavior and cross-dataset generalization remain unsupported.

Abstract

from arXiv · show

Generating realistic humanoid motion from scene images and text involves both low-frequency pose semantics and high-frequency physical dynamics. However, many existing methods tokenize motion with a single shared codebook, forcing heterogeneous motion signals into the same quantization space. Our frequency-domain analysis of human motion data reveals a clear mismatch between single-codebook quantization and motion statistics: five DCT coefficients capture 93% of joint-position energy but only 37% of joint-velocity energy, which can bias quantization toward pose statistics and under-represent high-frequency velocity components. A second challenge lies in adapting a standard autoregressive model to effectively model high-frequency physical signals in motion sequences. Therefore, we propose DSFT, a dual-stream frequency tokenizer that separates motion into Base and physical streams and compresses them independently with DCT truncation and BPE. Furthermore, we present MotionVLA, a Qwen3.5-based model that arranges Base and physical tokens in a unified sequence, where Phys tokens are predicted after Base tokens. Experiments on HumanML3D and MBench show that, despite using a lightweight 2B backbone, MotionVLA reduces the Diversity gap to real data by over 50% on HumanML3D and improves Motion-Condition Consistency by 3.8% on MBench, supporting frequency-aware dual-stream decoupling as an effective formulation for autoregressive motion generation. Code: https://github.com/AIGeeksGroup/MotionVLA. Website: https://aigeeksgroup.github.io/MotionVLA.

1 Introduction

MotionVLA addresses the mismatch between unified motion tokenization and heterogeneous frequency statistics by introducing DSFT, which separately tokenizes semantic and physical streams. It then uses a unified autoregressive model to generate these streams under vision-language conditioning, improving motion diversity and motion-condition consistency.

  • Motivation: Five DCT coefficients capture 93% of joint-position energy, while unified tokenization under-represents high-frequency physical signals.The frequency mismatch biases quantization toward low-frequency pose structure and weakens velocity preservation.
  • Motivation: A standard autoregressive model prioritizes dominant low-frequency pose tokens, making fine-grained physical dynamics less reliably preserved over time.This imbalance can accumulate during generation and impair physically faithful motion sequences.
  • Method: DSFT separates joint-position semantics and joint-velocity dynamics into Base and Phys streams, compressing each independently in the frequency domain.The dual-stream tokenizer is designed to address heterogeneous motion statistics that a single codebook cannot represent well.
  • Method: MotionVLA models Base and Phys tokens in a unified autoregressive sequence conditioned on text and scene images.This combines decoupled semantic and physical token streams within a vision-language-to-motion framework.

2 The Proposed Method

MotionVLA combines DSFT, which independently tokenizes low-frequency pose semantics and high-frequency physical dynamics, with a Qwen3.5-based autoregressive model. It generates Base tokens before Phys tokens, using masked training and phase-aware decoding to preserve this semantic-to-physical structure.

  • Autoregressive Generation: MotionVLA arranges tokens as [MBOS, Base, MSEP, Phys, MEOS], so each Phys prediction can attend to the complete preceding Base context.This imposes a structured semantic-to-physical generation order within a unified autoregressive sequence.
  • DSFT: DSFT partitions motion into Base position-and-rotation dimensions and Phys velocity dimensions, yielding (Db, Dp) = (190, 73) on HumanML3D and (201, 75) on ViMoGen.The partition follows the observed separation between low-frequency pose semantics and high-frequency physical dynamics.
  • DSFT: 93% of Base-stream energy is covered by K = 5 DCT coefficients, compared with only 37% for the Phys stream.This motivates independent compression with Kb = 5 for Base and Kp = 25 for Phys.
  • DSFT: DSFT independently applies DCT truncation and BPE to produce complementary Base and Phys token streams, then reconstructs motion through inverse BPE and inverse DCT.The reconstructed streams are concatenated along the feature dimension to recover the complete motion sequence.
  • Autoregressive Generation: MotionVLA trains with masked next-token prediction and uses phase-aware inference masks that allow Base tokens before MSEP and Phys tokens afterward.The training mask restricts prediction to valid motion tokens and structural markers, while inference prevents stream mixing.

3 Experiments

Experiments evaluate MotionVLA across MBench and HumanML3D, covering scene-conditioned generation, text-to-motion generalization, tokenizer reconstruction, and ablations. Results show strong condition alignment, diversity, tokenizer compactness-fidelity trade-offs, and perceptual preference despite the lightweight 2B backbone.

  • MBench evaluation: MotionVLA achieves the best MBench results on Motion-Condition Consistency and Foot Sliding, while ranking second on Motion Generalizability and Jitter Degree with a 2B backbone.These results indicate improvements in multimodal condition alignment and suppression of local temporal artifacts, although not all physical metrics are dominated.
  • HumanML3D evaluation: On HumanML3D, MotionVLA achieves the Diversity score closest to real data and the highest MModality among generated methods while remaining competitive on R-Precision, FID, and MM-Dist.Because HumanML3D removes visual conditioning, these gains primarily reflect the motion representation rather than scene input.
  • Tokenizer reconstruction: DSFT produces a more compact token sequence and substantially lower reconstruction rFID than single-stream DCT+BPE, despite higher rRMSE and MPJPE.The comparison suggests that lower pointwise reconstruction error does not necessarily imply better tokenizer quality.
  • DSFT ablation: As Kp increases from 10 to 30, rRMSE and MPJPE decrease while rFID drops from 1.340 to 0.138; Kp=25 is selected as the default.The default balances compactness and fidelity while matching the main tokenizer configuration.
  • Perceptual evaluation: MotionVLA is preferred in 64.0% of comparisons, versus 14.0% for ViMoGen and 22.0% ties, indicating a perceptual advantage in overall motion quality.The perceptual comparison provides evidence beyond benchmark metrics for the quality of generated motion.

4 Discussions and Conclusions · A Related Work

The paper addresses humanoid motion generation through coordinated tokenizer design, autoregressive modeling, and evaluation, introducing DSFT to separate pose-related and dynamic signals into Base and Phys streams. It reports limitations from its lightweight 2B backbone, limited benchmarks, and fixed design choices that constrain broader conclusions and may not suit all motions or sequence lengths.

  • 4 Discussions and Conclusions: DSFT decomposes humanoid motion into separate Base and Phys streams in the frequency domain.The design responds to differing spectral characteristics between pose-related and dynamic signals.
  • 4 Discussions and Conclusions: Separate frequency budgets prevent pose-related and dynamic signals from being forced into one shared tokenization space.
  • 4 Discussions and Conclusions: The work combines tokenizer design, autoregressive modeling, and evaluation to address humanoid motion generation.
  • 4 Discussions and Conclusions: The study uses a lightweight 2B backbone and a limited set of benchmarks.These choices prevent broader conclusions about scaling behavior or cross-dataset generalization.
  • 4 Discussions and Conclusions: The framework fixes stream partitioning, truncation lengths, and Base-to-Phys generation order.These fixed choices may not be optimal for all motion types or sequence lengths.
  • 4 Discussions and Conclusions: Future work will extend evaluation to larger backbones and broader settings.

A.1 Human Motion Generation · A.2 Motion Tokenization and Representation · A.3 Vision-Language-Action Models

Prior work frames human motion generation around discrete motion representations and autoregressive language-modeling paradigms, while VLA research unifies vision, language, and action for embodied AI. Representative systems span text-driven motion synthesis, compressed tokenization, and high-frequency robot control.

  • A.1 Human Motion Generation: Text-driven human motion generation synthesizes realistic 3D human motion sequences from natural-language descriptions.
  • A.1 Human Motion Generation: Mainstream approaches combine VQ-VAE discretization with autoregressive Transformers for motion generation.
  • A.1 Human Motion Generation: T2M-GPT established a representative paradigm by combining VQ-VAE with GPT next-token prediction for text-driven motion generation.
  • A.2 Motion Tokenization and Representation: VQ-VAE introduces discrete bottlenecks that enable efficient compression and generation of continuous data.
  • A.2 Motion Tokenization and Representation: T2M-GPT showed that human motion can be effectively tokenized for text-driven generation, while MoMask uses RVQ for hierarchical representations.
  • A.3 Vision-Language-Action Models: VLA models unify visual perception, language understanding, and action generation in a single end-to-end framework for embodied AI.
  • A.3 Vision-Language-Action Models: OpenVLA uses 7B parameters and approximately 970,000 multi-robot trajectories to support generalization across robotic platforms and manipulation tasks.
  • A.3 Vision-Language-Action Models: π0 generates 50 Hz high-frequency dexterous manipulation with flow matching, while RT-2 and Octo address grounded control and generalist robot policies.

B Detailed datasets and metrics

MotionVLA is evaluated on MBench and HumanML3D using their official data splits and benchmark protocols. MBench assesses multimodal motion quality across eight dimensions, while HumanML3D reports standard text-to-motion metrics in text-only mode.

  • Datasets: ViMoGen-228K contains 228K motion sequences from optical capture, in-the-wild video annotation, and synthetic generation; MotionVLA trains on its official training split and evaluates on MBench.MBench contains 450 held-out prompts and follows the official evaluation protocol.
  • Evaluation Metrics: MBench reports eight dimensions covering condition matching, generalizability, temporal stability, expressiveness, contact realism, and self-intersection artifacts.These include Motion-Condition Consistency, Motion Generalizability, Jitter Degree, Dynamic Degree, Foot Floating, Foot Sliding, and Body Penetration.
  • Datasets: HumanML3D is evaluated under the standard text-to-motion setting using its official split, with MotionVLA operating in text-only mode because visual inputs are unavailable.Evaluation uses the official pretrained feature extractor.
  • Evaluation Metrics: HumanML3D reports FID, R-Precision (Top-1/2/3), MM-Dist, Diversity, and MModality under the standard evaluation pipeline.The metrics respectively capture distribution distance, retrieval accuracy, multimodal alignment, sample diversity, and variation under identical text conditions.

C Implementation Details

The implementation trains separate Base and Phys motion streams, then adapts a frozen Qwen3.5 backbone through embedding warm-up and LoRA fine-tuning. Training uses ViMoGen and HumanML3D motion data, while inference applies phase-aware decoding and inverse reconstruction.

  • Tokenizer Training: DS-FAST splits motion vectors into Base and Phys dimensions and trains the tokenizer on ViMoGen or HumanML3D sequences.ViMoGen uses 276 dimensions with Db=201 and Dp=75; HumanML3D uses 263 dimensions with Db=190 and Dp=73.
  • Phase 1 — Embedding Cold Start: 8,195 motion token embeddings are randomly initialized and trained for 500 steps while all Qwen3.5 transformer layers remain frozen.The warm-up uses learning rate 1×10−3 and the Adafactor optimizer, updating only embed_tokens and lm_head.
  • Phase 2 — LoRA Fine-Tuning: LoRA adapters are applied to all linear projections for 10 epochs on 8×H100 GPUs, while base Qwen weights remain frozen.embed_tokens and lm_head continue updating as full saved modules during this phase.
  • Training Data: ViMoGen combines 41,971 in-the-wild video samples with 170,942 optical MoCap samples, while HumanML3D uses 23,384/1,460/4,384 train/val/test sequences.ViMoGen includes image-plus-text and text-only inputs; HumanML3D uses text-only inputs.
  • Inference: Inference runs on one H100 GPU and constrains decoding to Base tokens before SEP and Phys tokens after SEP.BPE inverse mapping followed by IDCT reconstructs 276-dimensional ViMoGen or 263-dimensional HumanML3D motion sequences.

D DS-FAST Feature Partition Details

DS-FAST assigns motion dimensions to Base or Phys streams using a data-driven low-frequency energy ratio threshold of 0.6. It then uses explicit representation-specific slicing to preserve frequency-matched, physically homogeneous features without manual field-boundary annotation.

  • Partition criterion: An LFR threshold of 0.6 assigns each dimension to the Base or Phys stream according to its frequency profile.Table 10 provides the complete mapping for ViMoGen’s 276-dim SMPL+X representation and HumanML3D’s 263-dim representation.
  • ViMoGen mapping: For ViMoGen, DS-FAST explicitly slices non-contiguous velocity indices before DCT so each stream contains physically homogeneous features.The joints_vel block [192:258] lies between Base joints [126:192] and Base root_orient_6d [258:264].
  • HumanML3D mapping: In HumanML3D, the root joint position [4:7] moves to Phys, while non-root local positions [7:67] remain in Base because of their differing frequency profiles.The LFR boundary splits the 21-joint local_pos field rather than aligning with a semantic field boundary.
  • HumanML3D mapping: The first four joint-velocity elements [193:197] are assigned to Base because they represent the root joint and exhibit low LFR.These assignments follow the data-driven LFR criterion and require no manual field-boundary annotation.

E Human Preference Analysis · F Simulation and Real-Robot Demonstration

MotionVLA was evaluated in a blinded human preference study against ViMoGen using 500 comparisons across five domain experts. It received a clear majority preference, indicating higher perceived motion quality across front and side views.

  • E Human Preference Analysis: MotionVLA’s perceptual quality was assessed through anonymized side-by-side motion comparisons presented in a custom web-based interface.The study complemented quantitative benchmarks and evaluated motions from front and side camera perspectives.
  • E Human Preference Analysis: Five domain experts each evaluated 100 text-conditioned motion pairs, comparing MotionVLA with ViMoGen.Each pair contained one output from each model, and every motion was rendered from front and side views.
  • E Human Preference Analysis: Experts selected Good, Same, or Bad to indicate whether MotionVLA was better, comparable, or worse than the baseline.After de-anonymization, Good represented MotionVLA preference, Same represented no clear preference, and Bad represented baseline preference.
  • E Human Preference Analysis: 500 comparisons formed the aggregate preference analysis, covering 5 experts × 100 prompts.Preference rates were reported across all evaluations.
  • E Human Preference Analysis: 64.0% of evaluations preferred MotionVLA, compared with 14.0% favoring ViMoGen.The remaining evaluations corresponded to the no-preference category in the GSB study.
  • E Human Preference Analysis: MotionVLA demonstrated a substantial and consistent advantage in perceived motion quality across both front and side views.This conclusion follows from the aggregate GSB preference results against ViMoGen.

F.1 MuJoCo Simulation

The MuJoCo pipeline decodes motion tokens into per-frame motion vectors, converts them to SMPL-X parameters, and renders capsule-based skeletons. Rendering uses fixed offscreen settings and per-sequence foot-to-floor alignment for plausible ground contact.

  • Pipeline: Motion tokens are decoded by DS-FAST through inverse BPE and inverse DCT into 276-dimensional ViMoGen or 263-dimensional HumanML3D frame vectors.The vectors are then converted to SMPL-X parameters, including global orientation, 22-joint body pose, and root translation.
  • Pipeline: MuJoCo visualizes each frame as a capsule-based skeleton, with capsule geometries connecting adjacent joints and spheres marking joint centers.The qualitative motion visualizations use MuJoCo, a physics engine commonly used for locomotion and character-animation research.
  • Rendering Configuration: Frames are rendered offscreen with MuJoCo’s EGL backend at 1280×1024 resolution and composited at 20 fps from a fixed side-view camera.A per-sequence vertical offset aligns the lowest foot position with the floor plane to ensure plausible ground contact.

F.2 Real-Robot Deployment … End-to-End Generation Example #2

MotionVLA is deployed on a Unitree G1 EDU humanoid, where text prompts are converted into executable joint-angle trajectories in real time. End-to-end examples show Base–Phys token generation, DS-FAST reconstruction, and visualized motion sequences.

  • F.2 Real-Robot Deployment: MotionVLA generates motion tokens from text, decodes them with DS-FAST into joint-angle trajectories, retargets them to the G1 configuration, and executes them in real time.This real-robot deployment verifies execution on physical hardware.
  • F.2 Real-Robot Deployment: Figure 7 presents three deployment examples, each pairing a text prompt with three exocentric frames captured at successive time steps.
  • End-to-End Generation Example #1: The first end-to-end example prompts motion for removing a shirt, putting it on the head, bending down, and picking up something from the ground.
  • End-to-End Generation Example #1: Its generated sequence uses <mot_bos>, <mot_sep>, and <mot_eos> to structure Base and Phys streams, which DS-FAST reconstructs through inverse BPE and inverse DCT.The reconstructed motion is visualized at uniform time intervals.
  • End-to-End Generation Example #2: The second end-to-end example uses the prompt “Generate motion for: The man walks towards the camera.”
  • End-to-End Generation Example #2: Its token sequence separates Base and Phys streams with <mot_sep>, bounded by <mot_bos> and <mot_eos>, and DS-FAST reconstructs motion via inverse BPE and inverse DCT.The reconstructed sequence is visualized at uniform time intervals.
  • End-to-End Generation Example #2: The second example likewise displays Base tokens, then <mot_sep>, then Phys tokens, before terminating with <mot_eos>.
Loading 2606.15142v1…