PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoMotionGPT: Geometry-Aligned Motion Understanding with Large Language Models

Zhankai Ye, Bofan Li, Yukai Jin, Shuoqiu Li, Wei Wang, Yanfu Zhang, Shangqian Gao, Xin Liu

arXiv:2601.07632v4cs.CVcs.AI

TL;DR

Existing motion-token pipelines link quantization and semantic embeddings through token IDs, leaving their geometric relationship insufficiently aligned for nuanced motion reasoning. GeoMotionGPT establishes a shared orthogonal basis with differentiable decoder-only quantization, sparse projection, and staged regularization. It improves aggregated Average by 22.4% on HumanML3D and 14.4% on KIT-ML over the strongest baselines, with ablations supporting its main designs.

  • Problem

    Existing pipelines decouple motion quantization from semantic embedding learning and link them solely through token IDs, disregarding motion-code geometry in the embedding space.

  • Method

    GeoMotionGPT aligns motion codes and LLM embeddings through shared orthogonality, using decoder-only Gumbel-Softmax quantization, sparse projection, and orthogonal regularization.

  • Results

    22.4% and 14.4% improvements in aggregated Average over the strongest baselines are reported on HumanML3D and KIT-ML, respectively.

  • Takeaways & Limitations

    The findings support preserving geometric structure in both motion-token and embedding spaces for effective motion-language alignment.

  • Takeaways & Limitations

    The evaluation focuses on motion understanding and does not assess whether the tokenization and regularization generalize to motion synthesis settings.

Abstract

from arXiv · show

Discrete motion tokenization has recently enabled Large Language Models (LLMs) to serve as versatile backbones for motion understanding and motion-language reasoning. However, existing pipelines typically decouple motion quantization from semantic embedding learning, linking them solely via token IDs. This approach fails to effectively align the intrinsic geometry of the motion space with the embedding space, thereby hindering the LLM's capacity for nuanced motion reasoning. We argue that alignment is most effective when both modalities share a unified geometric basis. Therefore, instead of forcing the LLM to reconstruct the complex geometry among motion tokens from scratch, we present a novel framework that explicitly enforces orthogonality on both the motion codebook and the LLM embedding space, ensuring that their relational structures naturally mirror each other. Specifically, we employ a decoder-only quantizer with Gumbel-Softmax for differentiable training and balanced codebook usage. To bridge the modalities, we use a sparse projection that maps motion codes into the LLM embedding space while preserving orthogonality. Finally, a two-stage orthonormal regularization schedule enforces soft constraints during tokenizer training and LLM fine-tuning to maintain geometric alignment without hindering semantic adaptation. Extensive experiments show that our framework improves the aggregated Average by 22.4% over the strongest baseline on HumanML3D and by 14.4% on KIT-ML, while ablations confirm the effectiveness of the tokenizer, projection, and regularization designs.

1 Introduction

GeoMotionGPT addresses the geometric mismatch created when motion quantization and LLM embedding learning are connected only through token IDs. It aligns both spaces with a shared orthogonal basis using differentiable quantization, sparse projection, and staged regularization, yielding gains on two benchmarks.

  • Motivation: Existing pipelines connect motion codes and LLM embeddings through token IDs while disregarding relational geometry between motion codes.This decoupled protocol can hinder nuanced motion reasoning by failing to align motion-space geometry with embedding-space geometry.
  • Geometric alignment: GeoMotionGPT uses a shared orthogonal geometric basis for the motion codebook and LLM embedding space.The framework explicitly enforces orthogonality so both modalities’ relational structures naturally mirror each other.
  • Architecture: The framework combines a decoder-only quantizer with Gumbel-Softmax, a sparse projection mechanism, and a two-stage regularization schedule.These designs support differentiable quantization, direct codebook regularization, and geometric alignment during tokenizer training and LLM fine-tuning.
  • Experiments: 22.4% improvement in aggregated Average over the strongest baseline is achieved on HumanML3D, with ablations validating the core components.The reported experiments cover tokenizer design, sparse projection, and orthogonal regularization.
  • Experiments: 14.4% improvement in aggregated Average over the strongest baseline is achieved on KIT-ML.The results are reported as consistent gains across both benchmarks.

2 Related Work

Prior motion-language research connects motion and text through shared embedding objectives or token-like sequences modeled with language objectives. Related representation-learning work uses orthogonality and other structured codebook mechanisms to improve compatibility, stability, and utilization.

  • Motion Understanding Using LLMs: One motion-understanding line aligns motion and text in a shared embedding space using contrastive, retrieval, or CLIP-style objectives.These methods target semantic matching across modalities.
  • Motion Understanding Using LLMs: Another line represents motion as discrete or token-like sequences and applies language-modeling objectives to text–motion problems.This includes reciprocal tokenized modeling and GPT-style motion-language models.
  • Orthogonality in Representation Learning: Orthogonality is used as a geometric bias for stable, well-conditioned, and distribution-compatible representations.Related work connects it with singular-value control, rotation-like transformations, isometry-motivated learning, and geometric embedding losses.
  • Orthogonality in Representation Learning: Structured codebook research targets simpler training, greater capacity under limited token budgets, high utilization, and structured reuse.These goals motivate practical alternatives for codebook design and training.

3 Our Approach

GeoMotionGPT formulates motion-language alignment as geometric unification: orthogonality is imposed on motion codes and transferred into the LLM embedding space through a structure-preserving sparse projection. A differentiable DVQ tokenizer, balanced code usage, and two-stage regularization jointly establish and maintain this alignment during training.

  • 3.1 Geometric Unification as Alignment: GeoMotionGPT treats alignment as a geometric unification problem rather than a connection based only on discrete token IDs.The framework models the motion manifold, LLM embedding space, and discrete vocabulary as spaces whose relational structure should be aligned.
  • 3.1 Geometric Unification as Alignment: Orthogonal regularization makes normalized motion-code correlations approach the identity matrix, encouraging pairwise orthogonality, linear independence, and distinctness.The codebook is row-normalized before its Gram matrix is used to define the orthogonal loss.
  • 3.2 Structure-Preserving Sparse Projection: A fixed sparse projection maps D-dimensional motion codes into the higher-dimensional LLM space while preserving their inner-product structure.Each source dimension is assigned to a unique randomly selected target dimension, with the remaining target dimensions set to zero; this yields an isometric embedding under the stated conditions.
  • 3.3 Decoder-Only Vector Quantization (DVQ): The decoder-only quantizer replaces nondifferentiable nearest-neighbor assignment with differentiable Gumbel-Softmax quantization, enabling direct codebook regularization.A straight-through estimator supports gradients for the hard one-hot output, which the decoder maps back to reconstructed motion.
  • 3.3 Decoder-Only Vector Quantization (DVQ): Entropy-based utilization regularization drives motion-token frequencies toward a uniform distribution to reduce token collapse and use the codebook capacity.The DVQ objective combines reconstruction, orthogonality, and utilization terms with separate weighting coefficients.
  • 3.4 Two-Stage Orthonormal Regularization: Two-stage orthonormal regularization first establishes codebook geometry, then adapts projected motion tokens during LLM instruction tuning while preserving their orthogonal anchor.The original text embeddings remain frozen, while projected motion-token embeddings and LLM weights are optimized under a soft orthogonal constraint.

4 Experiment

The experiments evaluate GeoMotionGPT across language backbones and standard motion benchmarks, using codebook-utilization and aggregated captioning metrics. The HumanML3D comparison reports a 22.4% improvement over the strongest baseline.

  • Experimental Setup: Experiments use GPT-2, Qwen 3-0.6B, and LLaMA 3.2–1B, with full fine-tuning for GPT-2 and LoRA adaptation for the other backbones.
  • Experimental Setup: The evaluation covers HumanML3D and KIT-ML under the MotionGPT3 protocol for fair comparison.
  • Evaluation Metrics: Codebook utilization is measured by the percentage of non-zero-use codes and the standard deviation of usage counts.
  • Evaluation Metrics: The aggregated average combines retrieval-style alignment with text-generation quality, while individual component metrics remain separately reported.
  • Results: 22.4% improvement in aggregated average over the strongest baseline is reported for HumanML3D under GPT-2.

4.3 Codebook Distribution Analysis

GeoMotionGPT shows more balanced codebook usage and strong motion-understanding performance across HumanML3D and KIT-ML. Ablations attribute gains to DVQ, sparse projection, orthogonal regularization, and an appropriate orthogonal-loss strength.

  • Codebook Distribution: GeoMotionGPT uses the codebook less skewedly than VQ-VAE, with reduced dominance by frequent codes and more stable usage across codes.Both methods remain heavy-tailed, but GeoMotionGPT distributes usage more broadly.
  • Benchmark Performance: 22.4% improvement in aggregated Average over MotionGPT3 on HumanML3D, driven mainly by stronger caption quality.CIDEr, BLEU@4, and BERTScore improve by 107.9%, 33.3%, and 39.2%, respectively, while retrieval recalls remain competitive.
  • Benchmark Performance: 14.4% improvement over MotionGPT on KIT-ML, where GeoMotionGPT achieves the highest Average of 41.71.It leads on R@2/R@3, MMDist, and CIDEr, although MotionGPT remains better on BLEU@4 and BERTScore.
  • Tokenizer Ablation: 4.7% improvement in Average when replacing PQ-VAE with DVQ, despite slight decreases in R@1 and R@2.CIDEr improves from 44.06 to 50.69 and MMDist decreases from 2.92 to 2.80.
  • Regularization Ablation: 15.8% improvement in Average when orthogonal regularization replaces pairwise cosine-distance regularization.MMDist improves from 3.07 to 2.68 and CIDEr from 43.71 to 59.71.
  • Initialization Ablation: 8.3% reduction in Average when sparse projection is replaced by stochastic initialization under the same orthogonal-loss ratio of 10^-2.Without sparse projection, training converges more slowly and remains at higher loss.
  • Regularization Strength: 13.1% improvement in Average at the best orthogonal-loss ratio of 10^-2, while the stronger 10^-1 ratio raises recalls but degrades captioning quality.Figure 3 also shows faster, lower-loss optimization with orthogonal regularization.

4.6 Case Studies

Case studies show that GeoMotionGPT produces more diverse, temporally coherent token sequences and captions that better capture fine-grained motion semantics. These qualitative patterns support more balanced codebook utilization and stronger motion-language alignment.

  • Codebook Usage: The richer discrete representation provides more informative inputs for downstream motion-language modeling.This observation aligns with the gains reported during subsequent LLM fine-tuning.
  • Codebook Usage: GeoMotionGPT produces more diverse yet temporally coherent token transitions than VQ-VAE on identical ground-truth motions.VQ-VAE often generates long runs of identical IDs, whereas DVQ yields richer transitions.
  • Text Quality: GeoMotionGPT generates more faithful, fine-grained captions that capture directional cues and specific motion semantics.Examples include leftward jumps, stumbling direction, and hand actions, whereas VQ-VAE often produces generic or partially incorrect descriptions.

5 Conclusion

GeoMotionGPT aligns discrete motion tokenization with LLM embedding spaces through geometry-aware design. Results on both benchmarks and component ablations support preserving geometric structure across token and embedding spaces.

  • Conclusion: GeoMotionGPT combines decoder-only DVQ tokenization, sparse projection-based initialization, and orthogonal regularization to align motion tokens with LLM embeddings.The framework improves Average by 22.4% on HumanML3D and 14.4% on KIT-ML over the strongest baselines.
  • Conclusion: Ablations support the contribution of each component, with DVQ, sparse projection, and moderate-strength orthogonal regularization providing the strongest overall trade-off.The findings support preserving geometric structure in both token and embedding spaces for motion-language alignment.

Limitations

The paper identifies two limitations: limited comparison across geometry-aware objectives and evaluation restricted to motion understanding rather than motion generation.

  • The ablations remain centered on the framework’s components and do not compare broader geometry-aware objectives.Unexamined alternatives include whitening-, spectral-, and entropy-based constraints.
  • Evaluation covers motion understanding tasks such as captioning and motion–text alignment, but excludes motion generation.The authors leave unconditional, text-conditioned, and controllable generation for future investigation.

A.1 Additional Implementation Details

The appendix details DVQ’s optimization, quantization behavior, backbone settings, and temporal tokenization. It emphasizes differentiable training, scheduled discretization, deterministic inference, and a fully discrete reconstruction bottleneck.

  • Additional Implementation Details: The implementation evaluates GPT-2 with full fine-tuning and Qwen 3-0.6B or LLaMA 3.2–1B with LoRA adaptation.The appendix also specifies AdamW-based downstream optimization and separate DVQ and LLM training schedules.
  • Additional Implementation Details: Temporal downsampling makes each motion token represent a fixed temporal window and reduces the effective sequence length.The design targets temporally coherent patterns rather than frame-level noise.
  • Additional Implementation Details: Training uses stochastic Gumbel-Softmax sampling, whereas inference deterministically selects the encoder-logit arg max.This separates differentiable exploration during optimization from stable, reproducible token sequences at inference.
  • Additional Implementation Details: DVQ uses a decoder-only structure whose decoder consumes only code embeddings, forcing reconstruction signals through the discrete bottleneck.Unlike encoder–decoder VQ-VAE designs, it has no continuous latent bypass.
  • Additional Implementation Details: Table 6 reports the backbone and adaptation configurations used to compare GeoMotionGPT variants on HumanML3D.The table organizes performance by LLM backbone and training strategy.
  • Additional Implementation Details: Gumbel-Softmax provides differentiable codebook sampling, while temperature scheduling moves assignments from smooth exploration toward near-discrete tokenization.The temperature starts high for gradient propagation and is gradually annealed toward sharper assignments.

A.2 Evaluation on Different LLM Backbones

On HumanML3D, GPT-2 with full-parameter fine-tuning performs best among the evaluated GeoMotionGPT backbone configurations. Its advantage spans captioning and motion–text alignment metrics.

  • Evaluation on Different LLM Backbones: 17.4% higher aggregated Average than Qwen3-0.6B with LoRA is achieved by GPT-2 with full fine-tuning.
  • Evaluation on Different LLM Backbones: 9.0% higher aggregated Average than LLaMA3.2-1B with LoRA is achieved by GPT-2 with full fine-tuning.
  • Evaluation on Different LLM Backbones: GPT-2 performs better in both captioning and alignment, including retrieval accuracy and motion-text alignment.The reported advantage includes higher caption consensus/semantic quality and stronger retrieval performance.
Loading 2601.07632v4…