PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-UniMo: Towards Unified Motion-Language Understanding and Generation in the Open World

Guocun Wang, Kenkun Liu, Guorui Song, Jing Lin, Zhe Huang, Luyuan Zhang, Dake Zhong, Choo Sin Wai, Xiaoguang Han, Haoqian Wang

arXiv:2609.14615v1cs.CVcs.AI

TL;DR

Open-world motion-language models face text-dominated representations, weak motion-language alignment, and cumulative errors in autoregressive motion generation. Open-UniMo combines a shared motion-language token space, motion-consistent CoT, and SFT-to-GRPO training, alongside a benchmark for both directions and their consistency. Its analysis finds that coupling M2T with the learnable T2M path yields stronger cross-modal representations, while M2T understanding is not primarily limited by vocabulary size.

  • Problem

    Unified motion-language models face text-dominated representations, weak cross-modal interaction, and cumulative prediction errors on long motion sequences.

    Source: §I · p. 2

  • Method

    Open-UniMo extends Qwen’s vocabulary with motion tokens, uses motion-consistent CoT in bidirectional SFT, and applies GRPO post-training, with Open-MoBench evaluating T2M, M2T, and bidirectional consistency.

    Source: §I · p. 2, F. Reinforcement Learning with GRPO · p. 5

  • Results

    M2T understanding is not primarily limited by motion-token vocabulary size, while unified training coupling M2T to the learnable T2M path yields stronger cross-modal representations.

    Source: §I · p. 2, §VI · p. 13

  • Takeaways & Limitations

    The analysis supports treating generation and understanding as coupled paths in autoregressive motion-language modeling.

    Source: §VI · p. 13

Abstract

from arXiv · show

Unified motion generation and understanding is crucial for embodied AI systems that can both synthesize and interpret human actions in open-world environments. Existing motion-language models often treat motion as an auxiliary modality of a language model, leading to text-dominated representations and limited cross-modal interaction. Moreover, the next-token prediction paradigm is not naturally suited to long motion sequences, where autoregressive generation may accumulate prediction errors. To address these challenges, we propose Open-UniMo, a unified Large Motion-Language Model (LMLM) trained on million-scale open-world motion-language data. Open-UniMo promotes modality parity by extending Qwen's vocabulary of about 150K text tokens with 64K motion tokens, enabling motion and language to share a unified token space. We further introduce motion-consistent Chain-of-Thought reasoning as an intermediate representation to bridge language semantics and motion dynamics. Open-UniMo is trained with a two-stage pipeline, where supervised fine-tuning establishes CoT-guided bidirectional motion-language mapping and Group Relative Policy Optimization (GRPO) improves semantic alignment while mitigating cumulative errors in autoregressive motion-token generation. To support comprehensive evaluation, we propose Open-MoBench, a VLM-guided benchmark for assessing text-to-motion (T2M) generation, motion-to-text (M2T) understanding, and bidirectional consistency. Extensive experiments show that Open-UniMo achieves state-of-the-art performance on both conventional metrics and Open-MoBench. Furthermore, ablation studies reveal that M2T understanding is not primarily limited by motion-token vocabulary size; instead, coupling M2T with the learnable T2M generation path yields stronger cross-modal representations, demonstrating that generation can facilitate understanding in AR-based motion-language modeling.

I. INTRODUCTION

Open-UniMo unifies open-world motion generation and understanding using a shared motion-language token space and CoT-guided, two-stage training. The introduction also presents Open-MoBench and reports that coupling M2T with the learnable T2M path supports stronger cross-modal representations.

I. INTRODUCTION

  • Motion generation synthesizes movements from textual instructions, while motion understanding interprets motion sequences as descriptions; the tasks have largely been studied separately.The introduction characterizes them as inherently inverse tasks and notes prior unified efforts within a single LLM backbone.Source: §I · pp. 1–2
  • Existing unified models lag task-specific methods, especially on open-world data, amid weak alignment, autoregressive error accumulation, modality imbalance, and limited evaluation.The introduction also notes one-to-many text-motion correspondence and conventional metrics’ limited coverage of semantic, temporal, physical, and robustness properties.Source: §I · p. 2
  • Motion-consistent CoT bridges language semantics and motion dynamics, while GRPO post-training refines format correctness and cross-modal alignment beyond token-level supervision.The CoT traces are grounded in motion rendered as video and paired with captions for VLM annotation.Source: §I · p. 2
  • Open-MoBench is a VLM-guided framework that evaluates T2M and M2T across diagnostic dimensions and tests bidirectional consistency.Its cyclic protocol checks whether transformations preserve motion and language semantics.Source: §I · p. 2
  • Open-UniMo scales unified modeling to million-scale motion-language data, adds 64K motion tokens to Qwen’s vocabulary, and uses CoT-guided SFT followed by GRPO.The shared token space treats motion as a first-class modality; the pipeline is reported to improve semantic alignment and mitigate cumulative autoregressive errors.Source: §I · p. 2
  • Unified training couples M2T with the learnable T2M path, and the paper reports stronger cross-modal representations than M2T-only training as motion-token capacity grows.The reported analysis concludes that M2T understanding is not primarily limited by motion-token vocabulary size.Source: §I · p. 2

A. Human Motion Generation

The related-work discussion traces progress in motion generation and unified modeling, then identifies open-world and evaluation gaps. It situates motion-consistent CoT and Open-MoBench as responses to those gaps.

A. Human Motion Generation

  • Motion generation progressed from deterministic text-to-pose mappings to probabilistic methods, diffusion, masked prediction, and autoregressive discrete-token generation, with recent work targeting open-world settings.The cited discussion notes continuing challenges in long-horizon temporal coherence and fine-grained semantic alignment for autoregressive methods.Source: A. Human Motion Generation · p. 3

B. Unified Motion Generation and Understanding

  • Unified motion-language models combine generation and understanding, but open-world evaluation and the interaction between high-capacity motion tokens and autoregressive generation remain insufficiently explored.Prior approaches include multi-granularity training, modality-specific modeling, CoT and GRPO post-training, and broader multimodal task frameworks.Source: B. Unified Motion Generation and… · p. 3

C. Chain-of-Thought for Generative Models

  • Open-UniMo scales motion-consistent CoT—reasoning traces grounded in rendered motion videos—to million-scale data and uses it as a shared intermediate representation for T2M and M2T.The discussion distinguishes this grounding from approaches relying only on caption rewriting.Source: C. Chain-of-Thought for Generative… · p. 3

D. Evaluation of Motion-Language Models

  • Existing motion benchmarks largely assess generation or use manual evaluation, while Open-MoBench jointly evaluates T2M, M2T, and bidirectional consistency with a scalable VLM-guided protocol.The paper reports validating the protocol’s reliability through human preference testing.Source: D. Evaluation of Motion-Language Models · p. 3

III. METHOD

The method uses a two-stage SFT and GRPO pipeline for bidirectional motion-language mappings, a 272-dimensional motion representation, and FSQ tokenization. Its quantizer converts encoder latents into discrete codes for language-model training.

A. Overview

  • Open-UniMo uses CoT-guided SFT for bidirectional mappings, followed by GRPO with task-specific rewards to improve semantic alignment and mitigate cumulative motion-token prediction errors.The model represents motion and language with modality parity.Source: A. Overview · p. 4

B. Motion Representation

  • Each motion frame uses a refined 272-dimensional representation that avoids inverse-kinematics processing and supports lossless conversion to SMPL.It includes root linear and angular velocities, local joint positions and velocities, and local joint rotations in root coordinates.Source: B. Motion Representation · p. 4

C. Motion Tokenization

  • Finite Scalar Quantization discretizes normalized encoder latents by rounding each dimension into levels, producing discrete codes; the tokenizer is frozen during later LMLM training.The code is trained with a reconstruction loss, and wavelet transforms are applied before encoding and inverse transforms after decoding to reduce quantization artifacts.Source: C. Motion Tokenization · p. 4

D. Unified LMLM with Modality Parity

Open-UniMo gives motion and language a shared token space, then uses motion-grounded Chain-of-Thought reasoning and task-specific GRPO rewards to support bidirectional mapping. Its GRPO training targets semantic alignment and cumulative motion-token errors.

D. Unified LMLM with Modality Parity

  • Open-UniMo extends Qwen’s approximately 150K text-token vocabulary with 64K motion tokens, treating motion as a first-class modality in a unified token space.The shared space supports both text-to-motion and motion-to-text mapping.Source: D. Unified LMLM with Modality Parity · p. 4

E. Supervised Fine-Tuning with Chain-of-Thought

  • Motion-consistent CoT traces are generated from Blender-rendered motion videos paired with captions, grounding the intermediate reasoning in motion dynamics.SFT uses paired motion, caption, and CoT samples for bidirectional mapping; T2M and M2T outputs follow distinct tagged formats.Source: E. Supervised Fine-Tuning with… · p. 5, Figure 3 · p. 5

F. Reinforcement Learning with GRPO

  • GRPO samples candidate outputs and normalizes rewards within each group to compute relative advantages, avoiding an explicit critic and reducing policy-gradient variance.The objective also clips policy updates and regularizes the current policy toward a reference policy.Source: F. Reinforcement Learning with GRPO · p. 5
  • The asymmetric reward scheme combines format validity with motion-fidelity and semantic-alignment rewards for T2M, while M2T uses caption-level similarity.The T2M motion reward compares generated and reference motions; its semantic reward compares generated motion with the input caption. M2T uses CLIP text-encoder caption similarity.Source: F. Reinforcement Learning with GRPO · pp. 5–6
  • Open-UniMo’s T2M outputs better capture semantic details and natural dynamics than the representative approaches shown in Fig. 4 [8], [11], [18].This is a qualitative comparison on Open-MoBench.Source: Figure 4 · p. 6
  • GRPO optimizes these task-specific rewards to improve semantic alignment and generation quality while mitigating cumulative errors in autoregressive motion-token prediction.The M2T caption reward is scaled by 2 to harmonize its magnitude with the T2M reward.Source: F. Reinforcement Learning with GRPO · p. 6

A. Overview

Open-MoBench addresses the limited diagnostic reach of conventional motion-language metrics by evaluating text-to-motion, motion-to-text, and bidirectional consistency. Its T2M evaluation uses open-world captions and scores physical plausibility, semantic alignment, and temporal coherence.

A. Overview

  • Conventional metrics such as FID, R-Precision, BLEU, CIDEr, and BertScore provide limited diagnosis of alignment, plausibility, temporal coherence, and cross-modal consistency.One text description can correspond to plausible motions differing in amplitude, rhythm, or style, limiting what unidirectional metrics reveal.Source: A. Overview · p. 6
  • Open-MoBench evaluates T2M generation, M2T understanding, and bidirectional consistency with a VLM judge across multiple diagnostic dimensions.The benchmark also compares VLM-induced model rankings with rankings from a human preference study.Source: A. Overview · p. 6

B. Text-to-Motion Evaluation

  • The T2M evaluation samples 1,000 captions from two open-world video datasets, summarizes them into motion-centric descriptions, and assesses rendered SMPL motions on a 0–5 scale.The VLM judge scores physical plausibility, semantic alignment, and temporal coherence.Source: B. Text-to-Motion Evaluation · pp. 6–7
  • On the M2T qualitative comparison, Open-UniMo produces descriptions more coherent, contextually relevant, and semantically faithful than the open-source models shown in Fig. 5 [11], [15], [38].This figure caption reports an M2T comparison, distinct from the T2M evaluation setup.Source: Figure 5 · p. 7

C. Motion-to-Text Evaluation

Open-MoBench evaluates motion-to-text descriptions against rendered motion and ground-truth captions, then tests cross-modal semantic preservation through bidirectional cycles.

C. Motion-to-Text Evaluation

  • M2T evaluation renders 1,000 MotionMillion motions for VLM scoring of caption detail, semantic alignment, and temporal order, with pose colors clarifying left–right orientation.Samples use 272-dimensional representations and are converted to HumanML3D’s 263-dimensional format when required by baselines.Source: C. Motion-to-Text Evaluation · p. 7

D. Bidirectional Consistency Evaluation

  • Because language–motion mappings are one-to-many, the bidirectional protocol checks semantic preservation across cyclic T2M/M2T transformations rather than isolated task scores.A VLM assesses the input, intermediate, and reconstructed modalities on a 0–5 scale.Source: D. Bidirectional Consistency Evaluation · pp. 7–8

V. EXPERIMENTS AND RESULTS

Experiments use HumanML3D and the million-scale MotionMillion corpus, combining conventional metrics with Open-MoBench diagnostics. Open-UniMo is trained with SFT followed by GRPO.

A. Dataset

  • MotionMillion contains over 2,000 hours of recordings and more than 2 million text–motion pairs, alongside HumanML3D’s 14,616 clips and 44,970 descriptions.MotionMillion captions are generated with GPT-4o and paraphrased 20 times by Llama 3.1-8B; CoT annotations are constructed for both datasets.Source: A. Dataset · p. 8

B. Evaluation Metrics

  • Evaluation combines standard T2M distribution and retrieval metrics and M2T language metrics with Open-MoBench diagnostics for physical plausibility, semantics, temporal coherence, and bidirectional preservation.Open-MoBench uses Gemini-3-Flash for its VLM-guided assessments.Source: B. Evaluation Metrics · p. 8

C. Implementation Details

  • Open-UniMo uses a two-stage SFT and GRPO training schedule, with SFT run for 250k iterations and GRPO applied for 5,000 steps.SFT takes about 18 days at global batch size 96; GRPO uses a global batch size of 60 and generates six outputs per iteration.Source: C. Implementation Details · pp. 8–9

D. Main Results

Open-UniMo performs competitively on HumanML3D and maintains strong performance on open-world MotionMillion. Its reported T2M and M2T results improve with CoT-guided training and GRPO.

a) Scaling from HumanML3D to MotionMillion

  • Open-UniMo is state-of-the-art or competitive across major HumanML3D metrics and maintains strong performance after scaling to MotionMillion.The paper describes MotionMillion as a more challenging setting with richer motion dynamics, more diverse descriptions, and a larger motion-token space.Source: D. Main Results · p. 9

b) Text-to-Motion Results

  • On MotionMillion, Open-UniMo achieves the best R-Precision and MM-Dist; SFT+GRPO reaches 0.860/0.950/0.973 R-Precision and lowest MM-Dist of 21.947.Without CoT, GRPO improves over SFT in retrieval alignment and motion-text distance; the CoT-guided variant further improves semantic alignment, though its FID is higher.Source: D. Main Results · p. 9

c) Motion-to-Text Results

  • On the Open-MoBench M2T subset, CoT-guided SFT raises BLEU-1 from 52.80 to 61.98 and CIDEr from 10.51 to 25.62; SFT+GRPO reaches 64.44 BLEU-1 and 27.74 CIDEr.SFT-to-SFT+GRPO gains also hold on the full MotionMillion validation set.Source: D. Main Results · pp. 9–10
  • The weaker symmetric-reward variant supports using caption-level semantic rewards for M2T rather than simply applying T2M-style motion-space alignment rewards.The paper identifies caption-level semantic rewards as more suitable for motion understanding.Source: D. Main Results · p. 10

E. Ablation Study

The ablations show complementary roles for CoT and GRPO, while joint T2M and M2T training improves understanding beyond what motion-token vocabulary size alone explains. Open-MoBench evaluations report advantages across T2M, M2T, and bidirectional consistency, with an important asymmetry in the M2T→T2M cycle.

E. Ablation Study

  • GRPO consistently improves the SFT baseline without CoT, while the full CoT-plus-GRPO model achieves the best reported R-Precision, MM-Dist, BLEU-1, BLEU-4, CIDEr, and BertScore.CoT alone does not consistently improve all metrics; CoT and GRPO are complementary, with CoT-based variants showing stronger retrieval alignment and motion-text consistency despite lower FID for some models without CoT.Source: E. Ablation Study · pp. 10–11

G. Evaluation on Open-MoBench

F. The Synergy Effect of Unified Modeling

  • On MotionMillion, joint training raises M2T BLEU-1 from 34.32 to 53.85 without CoT; the full model reaches 57.56 BLEU-1 and 14.90 CIDEr.M2T-only performance remains much lower, and changing its vocabulary from 512 to 65,536 produces no consistent performance trend.Source: F. The Synergy Effect of Unified… · p. 11

G. Evaluation on Open-MoBench

  • 1.742 versus 1.587: the final model improves T2M→M2T consistency, but its best M2T score does not yield the highest M2T→T2M consistency.Under-specified intermediate captions may omit details needed to reconstruct the original motion.Source: G. Evaluation on Open-MoBench · pp. 11–12

H. Human Preference Testing

The human-preference study compares human and VLM-induced model rankings using pairwise preferences across sampled Open-MoBench outputs. The rankings agree strongly, supporting VLM-as-Judge as a scalable approximation, while M2T→T2M consistency remains a notable exception for the full CoT+GRPO model.

H. Human Preference Testing

  • The study samples 100 examples per task and evaluation dimension, with five master’s-degree volunteers comparing anonymized outputs under a shared criterion.Human preferences are converted to pairwise win ratios; close comparisons within 0.05 are ties.Source: H. Human Preference Testing · p. 12
  • VLM-as-Judge rankings show perfect correlations on all M2T dimensions and T2M→M2T consistency, and high correlations on other main T2M dimensions and M2T→T2M consistency.Both human and VLM comparisons rank the full CoT+GRPO model below non-CoT variants on M2T→T2M consistency.Source: H. Human Preference Testing · p. 12

VI. CONCLUSION

  • The conclusion reports state-of-the-art performance on conventional metrics and Open-MoBench, and finds that coupling M2T with the learnable T2M path yields stronger cross-modal representations.It concludes that generation can facilitate understanding in AR-based motion-language models.Source: §VI · pp. 12–13

I. ADDITIONAL DETAILS OF OPEN-MOBENCH

Open-MoBench renders motion videos and adds visual cues to support VLM-based action assessment, while its prompts structure both model training and benchmark judging. Its supplementary materials also describe broader linguistic coverage and qualitative tools for examining bidirectional consistency.

I. ADDITIONAL DETAILS OF OPEN-MOBENCH

  • YOLOv11-Pose processes Blender-rendered frames, with green marking left limbs and red marking right limbs to make directional and body-part cues explicit.Source: §I · p. 15, Figure 7 · p. 15
  • Open-MoBench prompts contain broader open-world action descriptions and richer linguistic diversity than HumanML3D captions.Source: §I · p. 15, Figure 9 · p. 15
  • Qualitative comparisons examine whether semantics persist through T2M→M2T and M2T→T2M cycles; the feature visualization compares Open-UniMo motion clusters with ground-truth clusters.The passage reports stronger overlap with ground truth for Open-UniMo in the M2T→T2M visualization.Source: §I · p. 15, Figure 8 · p. 15, Figure 10 · p. 16
  • The human-preference interface presents anonymized model outputs for comparison under the same criterion.Source: §I · p. 15, Figure 11 · p. 16

II. PROMPT TEMPLATES

  • Training and judging prompts specify structured reasoning, task-specific output formats, and scoring criteria for T2M, M2T, and bidirectional consistency.For M2T→T2M consistency, the reconstructed motion video is compared with the input motion’s ground-truth caption as a textual proxy.Source: §II · p. 15, Figure 12 · p. 17, Figure 13 · p. 18
Loading 2609.14615v1…