PaperlayerView exact revision on arXiv

Source-linked AI summary

UniMo: Unifying Human and Animal Motion Generation

Zeyu Zhang, Zhiyuan Zhang, Siheng Wang, Yiran Wang, Danning Li, Ian Reid, Richard Hartley

arXiv:2609.12342v1cs.CVcs.GR

TL;DR

Animal motion generation faces diverse skeletal topologies and limited high-quality motion-language data. UniMo uses point-cloud representations and dynamic sampling, alongside the UniML3D dataset, and reports strong results across human and animal benchmarks.

  • Problem

    Animal motion generation is hindered by diverse skeletal topologies and a shortage of large, high-quality motion-language datasets compared with human motion.

    Source: §1 · p. 2

  • Method

    UniMo converts parametric skeletons into unparametric point clouds for unified human-animal generation and uses dynamic sampling to allocate more points to active joints.

    Source: §1 · p. 2

  • Results

    UniMo achieves state-of-the-art results across UniML3D, AnimalML3D, HumanML3D, and KIT-ML, with strong performance in text-motion alignment and motion quality across human and animal categories.

    Source: §1 · p. 2, §5.3 · p. 7

  • Takeaways & Limitations

    Experiments on UniML3D and standard benchmarks demonstrate the superiority of UniMo over existing approaches and the potential of point-based modeling for generalizable, high-quality motion generation.

    Source: §7 · p. 8

    UniML3D remains limited to curated motion sequences and excludes motion data extracted from real-world videos, constraining scalability and diversity for naturalistic, in-the-wild animal behaviors.

    Source: §6 · p. 8

Abstract

from arXiv · show

The conditional generation of 3D motion has emerged as a key research topic due to its wide applicability across robotics, AR/VR, gaming, and content creation. However, extending recent advances in text-driven human motion generation to the animal domain remains challenging due to two core limitations. First, animals exhibit highly diverse skeletal topologies, unlike the standard human structure, making unified modeling across species difficult and leading to inefficient per-species models. Second, existing animal motion datasets suffer from limited scale and annotation quality, constraining model performance. To address these challenges, we propose UniMo, a unified point cloud-based motion generation framework that bypasses topological discrepancies by converting parametric skeletons into unparametric representations, further enhanced by dynamic sampling that allocates more points to active joints. Additionally, we present UniML3D, a large-scale motion-language dataset spanning both human and animal categories, containing 145,907 motion sequences and 433,388 captions-over 102x larger than existing animal datasets. Our method achieves state-of-the-art results on UniML3D and three public benchmarks including HumanML3D, KIT-ML, and AnimalML3D, demonstrating the feasibility and effectiveness of unified human-animal motion generation. Website: https://steve-zeyu-zhang.github.io/UniMo.

1 Introduction

The paper targets the topology and data-quality challenges that complicate animal motion generation. It introduces UniMo and UniML3D, then reports strong results across human and animal benchmarks.

1 Introduction

  • Highly varied animal morphologies make unified skeletal modeling difficult, leading existing methods to train separate, inefficient models for each species.Animal motion-language datasets also lack the scale and annotation quality of human datasets.Source: §1 · p. 2
  • UniMo converts parametric skeletons into unparametric point clouds for unified human-animal modeling and allocates more points to active joints through dynamic sampling.The sampling strategy is based on motion dynamics.Source: §1 · p. 2
  • UniML3D contains 145,907 motion sequences and 433,388 captions across human and animal categories, over 102× larger than AnimalML3D.The dataset is presented as a response to limited scale and annotation quality in existing animal datasets.Source: §1 · p. 2
  • UniMo achieves state-of-the-art results on UniML3D, HumanML3D, KIT-ML, and AnimalML3D, surpassing prior approaches in text-motion alignment and motion quality across human and animal categories.The experiments cover both the full UniML3D dataset and its animal-only subset.Source: §1 · p. 2

2 Related Work

Prior animal-motion work includes approaches that depend on human-to-animal conversion, category-specific models, or reference motions. Human-motion research spans diffusion, VQ-VAE, autoregressive, and related modeling approaches.

2.0.1 Animal motion generation.

  • Animal-motion methods include human-motion conversion, category-specific generation [41], reference-based transfer and editing [33] [8], and token flattening that loses spatio-temporal structure [15].The section notes that the cited cross-category methods rely on reference motions and cannot generate arbitrary motions from text conditioning.Source: §2.0.1 · p. 2

2.0.2 Human motion generation.

  • Human-motion methods include diffusion and VQ-VAE models, latent-space generation, long-sequence modeling, retrieval augmentation, masked autoregression, bidirectional modeling, and spatial-temporal joint modeling [40].The section also describes memory optimization and essential-frame selection for extended sequences [38, 37].Source: §2.0.2 · pp. 2–3

3 Datasets: UniML3D

UniML3D combines curated animal motions and expert-reviewed captions with human motion-language data. Its construction uses quantitative motion metrics and human evaluation to select and annotate animal sequences.

3 Datasets: UniML3D

  • UniML3D contains 127,380 animal motion sequences across 31 categories, combined with HumanML3D and KIT-ML data for 145,907 sequences and 433,388 captions overall.The animal portion has 382,140 corresponding captions.Source: §3 · p. 3
  • Animal sequences are selected from Zoo-300K using R-Precision, FID, MM-Dist, and human expert evaluation, then recaptioned with Qwen2.5-VL-72B-Instruct [3].Experts select the top three of ten generated captions per motion, with at least two experts reviewing each selection.Source: §3 · p. 3
  • UniML3D is reported to have better FID and R-Precision than previous animal motion-language datasets, indicating more realistic motion and more precise text annotations.The passage presents this comparison as evidence about dataset motion realism and annotation precision.Source: §3 · p. 3

4 Methodology

UniMo encodes motion as a dynamically sampled point-cloud sequence, models its temporal structure with quantized tokens, and reconstructs articulated motion from generated point clouds.

4.1 Overview

  • The framework combines a Skeleton Encoder-Decoder, Point Cloud Tokenizer, and Mask Transformer to model motion in a unified point-based representation.The tokenizer compresses point clouds into quantized latent vectors, and the Mask Transformer predicts masked tokens from sequence context.Source: §4.1 · p. 3
  • Dynamic sampling converts joint positions and orientations into point clouds, allocating samples according to joint motion magnitude.The algorithm transforms local points to global coordinates to form a point-cloud sequence.Source: §4.1 · p. 3
  • The Skeleton Decoder reconstructs full motion from predicted point clouds, predicting keypoints, estimating rotations with shortest-arc quaternion inverse kinematics, and recovering positions with forward kinematics.The reconstructed sequence is guided by a template skeleton.Source: §4.1 · p. 3

4.2 Skeleton Encoder and Decoder

UniMo represents articulated motion as a geometry-aware point cloud, with dynamic sampling emphasizing moving joints. Its decoder maps predicted keypoints back to skeletal motion through inverse and forward kinematics, trained with reconstruction and rotation-validity objectives.

4.2.1 Skeleton Encoder.

  • The encoder represents the articulated skeleton as a dense point cloud by sampling points along bones and applying quaternion-based transformations.The representation is intended to preserve bone orientation and hierarchical motion structure.Source: §4.2.1 · pp. 3–4
  • Motion-aware sampling combines bone length and joint motion magnitude to allocate more points to active joints while keeping the total point count fixed.Points are sampled along bone segments and transformed into global coordinates to form the motion point cloud.Source: §4.2.1 · p. 4

4.2.2 Skeleton Decoder.

  • The decoder estimates joint rotations by aligning rest-bone directions with decoded keypoints, adding a roll rotation, then reconstructs joint positions through forward kinematics.The direction alignment uses a shortest-arc quaternion; forward kinematics follows the skeletal hierarchy.Source: §4.2.2 · pp. 4–5

4.2.3 Learning Objective.

  • The learning objective trains the encoder-decoder to reconstruct keypoints and joint positions while regularizing predicted quaternions to unit norm.The skeleton loss combines keypoint, position, and quaternion terms.Source: §4.2.3 · p. 5

4.3 Point Cloud Tokenizer

The point cloud tokenizer compresses each frame into a latent vector, discretizes the temporal sequence with a learned codebook, and decodes the tokens to reconstruct the point cloud.

4.3.1 Tokenizer Encoder.

  • The VQ encoder maps each frame's spatial points to a latent vector, then selects nearest codebook embeddings along time to create discrete tokens.The latent vector preserves the sequence's temporal dimension, and the selected embeddings form a quantized latent sequence.Source: §4.3.1 · p. 5

4.3.2 Tokenizer Decoder.

  • The tokenizer decoder reconstructs the original point cloud sequence from the quantized latent tokens.The objective combines point-cloud reconstruction, commitment to quantized embeddings, and codebook-stabilization losses.Source: §4.3.2 · p. 5
  • The tokenizer's training objective balances reconstruction error, encoder commitment to quantized embeddings, and codebook stabilization.The reconstruction error can be measured using Chamfer Distance.Source: §4.3.2 · p. 5

4.4 Mask Transformer

The Mask Transformer models temporal dependencies among quantized motion tokens. It uses positional encoding and masked-token prediction to learn contextualized sequence representations.

4.4 Mask Transformer

  • The Mask Transformer models temporal dependencies in quantized latent sequences to support motion generation.Its contextualized token representations encode dynamics across tokens.Source: §4.4 · p. 6
  • Sinusoidal positional encoding adds token-order information before the sequence is passed through a multi-layer Transformer encoder.The encoder produces contextualized representations for each token.Source: §4.4 · p. 6
  • During training, random token positions are masked and the model learns to reconstruct their original codebook indices from context.The masked-position objective supports sequence modeling over discrete motion tokens and captures long-range dependencies.Source: §4.4 · p. 6

5 Experiments

Experiments evaluate UniMo on UniML3D and public human and animal motion benchmarks using standard text-to-motion metrics. The reported comparisons, ablations, and efficiency results support its effectiveness across datasets and identify a strong point-cloud and model configuration.

5.1 Public Benchmarks and Evaluation Metrics

  • The evaluation uses HumanML3D and KIT-ML for human motion and AnimalML3D for animal motion, alongside UniML3D.Source: §5.1.1 · p. 6

5.1.1 Benchmarks.

  • FID and R-Precision assess realism and accuracy, MM-Dist measures motion-text alignment, and diversity and MModality assess motion variation.MModality measures diversity among motions generated from the same text description.Source: §5.1.2 · p. 6, KIT-ML [22] · p. 7

5.3 Comparative Study

  • UniMo achieves state-of-the-art performance across UniML3D, AnimalML3D, HumanML3D, and KIT-ML, with best or second-best scores across most metrics on the human benchmarks.On UniML3D it consistently outperforms baselines across the reported metrics on both the full dataset and animal-only subset; on AnimalML3D it matches or surpasses prior methods.Source: §5.3 · p. 7

5.4 Ablation Study

  • Reducing Mask Transformer depth from 6 to 3 layers lowers R Precision@3 from 0.848 to 0.823 and raises FID from 0.040 to 0.056.A codebook size of 256 instead of 512 lowers MModality from 2.800 to 2.645; reducing latent dimension from 384 to 256 raises FID to 0.062 and lowers R Precision@3 to 0.819.Source: §5.4.2 · p. 8

5.5 Efficiency

  • UniMo requires 63 hours of training on one A100 GPU and has autoregressive inference latency of 0.08 seconds per frame.The cited comparisons are 81 hours and 0.11 seconds per frame for Motion Avatar, and 113 hours and 0.16 seconds per frame for OmniMotionGPT.Source: §5.5 · p. 8

6 Limitations and Future Work

UniML3D is limited to curated motion sequences and excludes motion extracted from real-world videos. The authors identify this as a constraint on scalability and diversity, particularly for naturalistic, in-the-wild animal behaviors.

  • UniML3D excludes motion extracted from real-world videos, constraining its scalability and diversity, particularly for naturalistic, in-the-wild animal behaviors.Source: §6 · p. 8
  • Figure 4 compares UniMo's efficiency with prior methods.Source: Figure 4 · p. 8
  • The authors propose using video sources and motion reconstruction to build larger datasets with automatically extracted 3D motion.They suggest this direction could enhance realism, variety, and generalization capacity.Source: §6 · p. 8

7 Conclusion

The conclusion presents UniMo as a point-cloud framework for human and animal motion with diverse skeletal topologies, alongside the UniML3D dataset. Experiments on UniML3D and standard benchmarks are reported to demonstrate superiority over existing methods.

  • UniMo converts parametric skeletons to unparametric point clouds and uses dynamic sampling for motion synthesis across species.Source: §7 · p. 8
  • The work constructs UniML3D, described as the largest unified motion-language dataset to date, to advance data availability for human and animal motion generation.Source: §7 · p. 8
  • Experiments on UniML3D and standard benchmarks demonstrate the method's superiority over existing approaches.Source: §7 · p. 8

A User Study and Quantative Evaluation

A 50-participant survey assessed the quality, diversity, and text alignment of generated motions. Participants rated motion quality, diversity, and alignment highly, and most preferred UniMo in pairwise comparisons.

Related papers

This paper cites

Loading 2609.12342v1…