PaperlayerView exact revision on arXiv

Source-linked AI summary

UniMate: One Unified Model to Animate Diverse Skeletons

Linzhan Mou, Jiahui Lei, Zhiyang Dou, Chenyue Cai, Chaoyue Song, Adam Finkelstein, Szymon Rusinkiewicz

arXiv:2609.05415v1cs.CVcs.GRcs.LG

TL;DR

Existing learned animators struggle with arbitrary skeletal topologies and often require templates, reference motion, or per-skeleton adaptation. UniMate combines topology-aware diffusion modeling with the UniML3D dataset to generate text-conditioned motion for diverse rigs, achieving strong quality, generalization, and efficiency while supporting zero-shot downstream tasks.

  • Problem

    Existing learned animators are topology-constrained by category-specific templates or require per-skeleton fine-tuning and reference motions at inference.

  • Method

    UniMate uses a topology-aware diffusion transformer with graph-aware attention, Spec-RoPE, and global topological conditioning, trained on the canonically unified UniML3D dataset.

  • Results

    UniMate achieves state-of-the-art quality, generalization, and efficiency for topology-agnostic motion generation and mesh animation, while supporting zero-shot transfer, in-betweening, expansion, and editing.

  • Takeaways & Limitations

    A single pretrained model can animate heterogeneous rigged assets from text without test-time optimization or per-skeleton specialization.

  • Takeaways & Limitations

    UniMate can produce contact artifacts and is less reliable on rare skeletal topologies and out-of-distribution motions.

Abstract

from arXiv · show

Recent advances in automatic rigging now deliver animation-ready 3D assets at scale, yet generating the motion to drive them remains a bottleneck. Existing learned animators are topology-constrained: they rely on category-specific templates or require per-skeleton fine-tuning and reference motions at inference. We present UniMate, a unified foundation model that synthesizes articulated motion for arbitrary skeletons from a rigged 3D asset and a text prompt, with no test-time optimization or per-skeleton retraining. UniMate introduces a topology-aware diffusion transformer, which integrates skeletal topology into attention via three mechanisms: (1) a graph-aware attention bias from pairwise joint relations and geodesic distances; (2) a spectral rotary position embedding generalizing RoPE to arbitrary kinematic trees via the graph Laplacian; and (3) a global topological conditioner attention-pooled from the rest-pose skeleton. We also curate UniML3D, 13,006 motion sequences spanning bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects with unified canonicalization and text pairing. Trained on this dataset, UniMate outperforms state-of-the-art baselines in quality, generalization, and efficiency, and supports zero-shot cross-topology transfer, in-betweening, expansion, and text-guided editing. Our project page is available at https://linzhanmou.com/unimate/.

1 Introduction

UniMate addresses topology constraints in learned animation by combining a topology-aware diffusion transformer with a large, canonically unified motion dataset for diverse skeletons.

  • Existing animators rely on fixed category-specific skeleton templates, while topology-agnostic alternatives still require per-skeleton fine-tuning or reference motions at inference.
  • UniMate synthesizes articulated motion from a rigged 3D asset and text prompt without test-time fitting or per-skeleton specialization.
  • TADiT jointly reasons over rest-pose kinematics and motion manifolds using graph-aware attention bias, Spec-RoPE, and a global topological conditioner.The mechanisms encode pairwise joint relations, spectral structure, and global skeleton context.
  • UniML3D contains 13,006 animation sequences across diverse skeleton categories and 3,584 text prompts, unified through filtering and canonicalization.The dataset spans bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid-object motion.
  • Experiments report state-of-the-art quality, generalization, and efficiency, with zero-shot support for motion transfer, in-betweening, expansion, and text-guided editing.

2 Related Work

Prior animation methods are constrained by fixed skeletons, costly optimization, or source-motion requirements, whereas UniMate provides text-conditioned generation across heterogeneous rigs.

  • Mesh-based animation methods often require costly per-asset optimization or produce kinematically unconstrained vertex-wise deformations.
  • Most learned motion generators assume fixed skeleton templates, limiting their applicability across heterogeneous characters and objects.
  • Example-based methods support topology flexibility by transferring or stitching existing motions, but still depend on exemplar source motion.
  • AnyTop jointly models heterogeneous animal skeletons but uses a small corpus, requires target motion at inference, and lacks text conditioning.
  • UniMate covers humans, animals, and articulated objects with text conditioning and end-to-end rigged-mesh animation without reference motion or per-skeleton training.

3 Method

UniMate represents heterogeneous skeletons and motions in a shared space, then uses topology-aware diffusion attention to generate text-conditioned animation across diverse rigs. Its design combines canonicalized skeletal descriptors, graph structure, spectral coordinates, and efficient factorized spatiotemporal processing.

  • Skeleton and Motion Representation: UniMate builds a shared representation from rest-pose skeletal descriptors and per-frame motion features for heterogeneous articulated assets.The representation includes ordered kinematic trees, rest-pose geometry, local topology, spectral descriptors, and 12-dimensional joint motion features.
  • Skeleton and Motion Representation: Topology-diameter normalization removes skeleton scale differences while preserving relative kinematic structure.Geodesic distance along the kinematic tree defines the topology diameter used for normalization.
  • Topology-Aware Diffusion Transformer: TADiT jointly processes rest-pose kinematics and motion manifolds with graph-aware attention bias, Spec-RoPE, and a global topological conditioner.The model conditions generation on a rest-pose skeleton and text prompt using conditional flow matching.
  • Capabilities: The unified model supports prompt-diverse motion on one skeleton, diverse samples for one prompt, cross-topology animation, and drift-free long-horizon generation.The long-horizon variant is trained at 180 frames, while the visual examples demonstrate prompt-aligned motion across characters and articulated objects.
  • Topology-Aware Diffusion Transformer: Factorized attention separates joint interactions within each frame from temporal interactions across frames, reducing cost for heterogeneous skeletons.The joint branch receives kinematic structure through graph-aware bias and Spec-RoPE, while the temporal branch uses frame-index RoPE.
  • Topology-Aware Diffusion Transformer: Spectral coordinates replace arbitrary joint serialization with graph-intrinsic positions whose frequencies capture global-to-local skeletal structure.Low-frequency Laplacian modes encode coarse organization, while higher-frequency modes encode finer relationships.

4 UniML3D Dataset

UniML3D is built from noisy, heterogeneous 4D motion sources through rigorous filtering and canonicalization. The resulting dataset spans diverse skeleton types and provides standardized motion representations for cross-topology learning.

  • Data preprocessing: Raw motion sources require preprocessing because they contain disconnected skeletons, broken roots, non-functional joints, implausible motions, and inconsistent coordinate frames.These issues make direct cross-topology motion learning unsuitable without rigorous preprocessing.
  • Dataset composition: 13,006 motion sequences and 2,140,232 frames span bipedal, quadrupedal, avian, marine, insectoid, serpentine, and articulated rigid objects.The dataset is curated from Truebones, Mixamo, and Objaverse-XL.
  • Skeleton-based filtering: Skeleton-based filtering retains connected primary trees, corrects root alignment, removes phantom joints, discards static clips, and filters implausible motion.The filtering pipeline uses skinning weight, forward kinematics, activity, root velocity, and joint angular jitter criteria.
  • Training pipeline: Balanced sampling and augmentation are applied on the fly during training after filtering, annotation, and canonicalization.The pipeline is summarized in Fig. 10.
  • Motion canonicalization: Motion canonicalization places clips in a shared coordinate frame, aligns initial facing direction, expresses rotations relative to rest pose, and applies global and local normalization.The canonical frame uses y-axis up, the root at the origin, and facing aligned with positive z.

5 Experiments

Experiments evaluate UniMate across heterogeneous-rig generation, topology-aware comparisons, mesh animation, applications, and ablations. The results show prompt-faithful motion, stronger unseen-skeleton metrics, broad zero-shot functionality, and benefits from topology-aware components.

  • Qualitative results: UniMate produces prompt-faithful, temporally coherent motion across humanoids, quadrupeds, avians, insects, and articulated rigid objects.The model also demonstrates controllable prompt variation, generative diversity, and drift-free long-horizon generation.
  • Text-conditioned mesh animation: UniMate outperforms prior mesh-animation methods on most VBench metrics and achieves the highest scores across all user-study criteria.AnimateAnyMesh is smoother mainly because its outputs are near-static, while V2M4 relies on separately generated driving videos and is substantially slower.
  • More applications: The same pretrained model supports zero-shot motion transfer, in-betweening, expansion, and editing without fine-tuning or auxiliary networks.These applications differ only in which motion tokens remain fixed during sampling.
  • Contact behavior: Generated contact-rich motions can exhibit foot sliding, drift, hovering, or ground penetration.Fig. 16 illustrates hind-paw contact drift rather than planted contacts.
  • Ablation: Removing graph-aware attention bias, Spec-RoPE, or global topological conditioning degrades quality, with the global conditioner causing the largest quality drop.Without the global conditioner, diversity increases but motions become less stable and often jittery.

6 Limitations and Future Work

UniMate has limitations in contact handling and in rare or out-of-distribution topology and motion regimes. The paper attributes these issues to the absence of a unified contact model and scarce long-tailed animation data.

  • Contact and foot sliding: Without a unified contact model, UniMate may produce foot sliding, drift, hovering, or ground penetration in contact-rich motions.The paper suggests morphology-aware contact objectives, constraint-guided sampling, or IK-based foot locking where contacts are well defined.
  • Rare topologies and motions: Rare skeletal topologies and out-of-distribution motions can yield static, jittery, or semantically inaccurate results.The paper attributes this reduced reliability to scarce, long-tailed 4D animation data favoring humanoids and common locomotion.

7 Conclusion

UniMate synthesizes articulated motion for arbitrary-topology skeletons from a rigged asset and text without test-time optimization or per-skeleton specialization. Its topology-aware transformer and UniML3D dataset support strong quality, generalization, efficiency, and zero-shot animation applications.

  • Conclusion: UniMate synthesizes articulated motion for arbitrary-topology skeletons from a rigged 3D asset and text prompt without test-time optimization or per-skeleton specialization.The model couples motion and skeletal structure through topology-aware attention mechanisms and is trained with UniML3D.
  • Conclusion: The same pretrained model supports zero-shot motion transfer, in-betweening, expansion, and editing.The paper positions UniMate and UniML3D as a foundation for scalable, controllable animation of arbitrary rigged assets.

A.1 Data Sources and Statistics

UniML3D combines motion from multiple sources into a canonical skeleton–motion–caption dataset spanning diverse morphologies and rig complexities. Its preprocessing standardizes joint semantics, filters clips, canonicalizes motion, and normalizes features for unified training.

  • Data sources: UniML3D contains 13,006 motion sequences assembled from Truebones, Mixamo, and Objaverse-XL sources.The dataset includes 2,140,232 frames and combines animal, human, and articulated-object motion.
  • Dataset statistics: Bipedal characters comprise 84.1% of the dataset, while serpentine and marine categories account for 0.3% and 0.8%, respectively.This long-tailed morphology distribution motivates square-root-balanced sampling.
  • Dataset statistics: Retained skeletons have widely varying complexity, with a median of 38 joints and a mean of 40.4 joints.The joint-count distribution is bimodal and spans substantially different rig structures.
  • Curation pipeline: The curation pipeline filters rigs and clips, standardizes joint names to anatomical vocabulary, and produces canonical skeleton–motion–caption samples.Skeleton-level processing selects and cleans the main tree, while clip-level processing removes static or physically implausible motion.
  • Canonicalization: Motion canonicalization serializes skeletons breadth-first, scales them by topology diameter, aligns facing direction, expresses rotations relative to rest pose, and normalizes features.Root and non-root features use separate dataset-level statistics.
  • Canonicalization: The preprocessing uses synchronized multi-view renderings and body-derived facing cues rather than assuming camera labels reveal canonical orientation.Four static cameras are sampled chronologically at relative azimuths of 0°, 90°, 180°, and 270°.

B.5 Training

UniMate is trained with conditional flow matching and uses topology and text conditioning during inference to generate and recover rig-driven motion efficiently. The sampling pipeline converts predicted motion features into mesh animation through forward kinematics and skinning.

  • Training: UniMate trains with conditional flow matching using geodesic and smoothness regularizers weighted by 0.5 and 0.1.Optimization uses AdamW with a 1×10^-4 learning rate and gradient clipping at norm 1.0.
  • Training: Training runs for 100k optimization steps on 8 NVIDIA H100 GPUs with a global batch size of 256.The configuration uses bfloat16 mixed precision and approximately one day of wall-clock time.
  • Inference: At inference, UniMate canonicalizes each asset, computes topology descriptors, encodes text with FLAN-T5, and reuses these conditions across samples.The descriptors include rest-pose geometry, graph relations, depths, Laplacian eigenvectors, and joint-name embeddings.
  • Inference: A 60-frame clip takes roughly 1.2 seconds to sample end-to-end on one NVIDIA H100 GPU.The timing includes text encoding and forward-kinematic recovery.
  • Motion recovery: Predicted joint rotations and root trajectories are recovered through forward kinematics and applied to meshes using linear blend skinning or dual-quaternion blending.Separate output heads decode root and non-root feature conventions before motion recovery.

C Theoretical Analysis of Spec-RoPE

Spec-RoPE derives rotary coordinates from the Laplacian spectrum of a kinematic tree, providing structural position information without relying on arbitrary joint ordering. Its analysis establishes relative-coordinate behavior, permutation equivariance up to spectral ambiguities, and a reduction to standard RoPE on regular graphs, while identifying limitations for degenerate eigenspaces and truncated learned encodings.

  • Spectral construction: Spec-RoPE constructs joint coordinates from the non-trivial eigenvectors of the kinematic graph Laplacian.The constant eigenvector is discarded and the first m non-trivial eigenvectors are retained.
  • Translation invariance: Under linear angle maps, rotary query–key inner products depend on spectral coordinates through their difference.This gives Spec-RoPE a relative structural-information property analogous to standard RoPE.
  • Permutation equivariance: Spec-RoPE is equivariant to joint permutations up to the sign and basis ambiguities inherent in Laplacian eigendecompositions.SignNet removes eigenvector sign ambiguity, but degenerate eigenspaces retain orthogonal basis freedom.
  • Limitations: Repeated or near-repeated eigenvalues can make spectral axes unstable, and truncation can cut through an ambiguous eigenspace.Graph biases, rest-pose geometry, depth, and joint-name embeddings provide complementary structural cues.
  • Connection to standard RoPE: On path graphs, the leading non-trivial eigenvector is monotone in token index, so Spec-RoPE recovers standard 1D RoPE up to a monotone reparameterization.Cartesian products of paths similarly recover multi-axis RoPE up to per-axis reparameterization.
  • Structural interpretation: The spectral embedding admits an effective-resistance interpretation that motivates attention depending on relative graph geometry.This interpretation is a soft structural prior rather than a literal theorem for the deployed architecture.
  • Limitations: The deployed encoding uses truncated learned spectral features, SignNet, and explicit graph biases, so the analytical small-variance picture does not transfer verbatim.The paper presents the distance-based interpretation as qualitative guidance for the full architecture.

D.2 Held-Out Evaluation Set

The held-out evaluation tests UniMate on seven unseen Truebones skeleton types spanning six morphology categories, alongside irregular automatically rigged assets and mesh-animation comparisons. Qualitative results show prompt-faithful motion and identify complementary failure modes and component-specific effects in ablations.

  • Held-out evaluation set: The held-out set contains 58 evaluation sequences from seven unseen Truebones skeleton types spanning six morphology categories.The skeletons are Raptor, Leapord, Raindeer, Parrot2, Jaws, Crab, and KingCobra.
  • Held-out evaluation set: Additional tests use AI-generated and automatically rigged meshes with non-canonical proportions and irregular topologies absent from the training sources.These assets probe generalization beyond artist-authored rigs.
  • Evaluation scope: The supplementary experiments extend comparisons across held-out morphologies, mesh-animation baselines, architecture and dataset-curation ablations, and foot-sliding analysis.These experiments complement the main-paper evaluation.
  • Qualitative comparisons: UniMate performs prompted actions on unseen avian and crustacean rigs, whereas AnyTop remains close to idle in-place motion across those cases.The comparison extends beyond previously shown quadruped and biped locomotion examples.
  • Qualitative comparisons: AnimateAnyMesh’s smoothness is associated with near-static outputs, while V2M4 deviates from prompts and accumulates mesh distortion over time.UniMate produces prompt-faithful, kinematically coherent motion in the illustrated cases.
  • Qualitative ablations: Ablations associate graph-aware attention with sharper joint coordination, Spec-RoPE with transferring actions to unseen skeletons, and global conditioning with motion stability.Removing the global conditioner raises diversity but visibly degrades stability and produces the largest FID increase.

E.4 Dataset-Curation Ablation

The ablation study shows that dataset curation materially affects motion quality, diversity, and physical plausibility. Rest-pose rotation rebasing is especially important for unified kinematic representation, while augmentations broaden coverage.

  • Removing rest-pose rotation rebasing degrades FID from 0.757 to 1.179, the largest motion-quality loss among curation stages.Expressing rotations relative to the rest pose provides a unified kinematic basis across heterogeneous skeletons.
  • Disabling on-the-fly augmentations reduces diversity from 9.200 to 8.265, indicating that augmentation primarily broadens topological coverage.
  • Removing implausibility filtering allows erratic airborne dynamics to appear at generation time, while removing normalization causes drift and hovering.The full curation pipeline instead produces prompted motion with plausible, grounded dynamics.
  • Implausibility filtering discards 8.7% of candidate motion clips as a diagnostic measure of curation selectivity.

E.5 Foot-Sliding Evaluation and Foot-Locking Post-Processing

The evaluation characterizes foot-sliding artifacts on held-out legged skeletons and shows that zero-shot foot locking substantially reduces them. The post-process is evaluated only where ground contact is well defined.

  • The evaluation covers 20 held-out legged skeletons—10 bipedal and 10 quadrupedal—excluding serpentine, marine, in-flight, and articulated-object rigs.This scope reflects that ground contact is not uniformly defined across morphologies.
  • Skate measures how often planted limbs drift, whereas Slide measures the magnitude of their height-weighted horizontal drift.
  • Foot locking detects contact segments, anchors each contact joint to its mean horizontal position, and applies damped least-squares inverse kinematics over the limb chain.The correction is applied zero-shot without retraining and blended at segment boundaries.
  • Foot locking reduces the skating ratio from 0.107 to 0.023 and sliding distance from 0.542 to 0.191.Slide is reported in units of 10^-2L, where L is rest-pose root height.
  • Residual sliding comes from near-ground frames outside detected segments, blended boundaries, and brief touchdowns removed by the minimum-length filter.

F Limitations and Future Work

UniMate’s future work is shaped by limitations in data coverage, conditioning modalities, and fine-grained spatial control. The authors propose richer data and additional control channels to address these boundaries.

  • UniML3D remains much smaller than visual foundation-model corpora and is long-tailed, with humanoids dominant and several morphologies sparsely represented.
  • Rare motions and underrepresented species are harder to cover than well-represented humanoid locomotion despite balancing and augmentation.
  • The current conditioning interface accepts only a rigged skeleton and natural-language prompt, excluding video, exemplar motion, shape-only, and partial skeletal inputs.
  • Text cannot precisely impose spatial directives such as foot targets, hand trajectories, or environmental contact constraints.
  • Sampling-time guidance and in-context partial trajectory tokens are proposed as control mechanisms that could add fine-grained spatial constraints without retraining the backbone.
Loading 2609.05415v1…