Source-linked AI summary
RigMo: Unifying Rig and Motion Learning for Generative Animation
Hao Zhang, Jiahao Luo, Bohui Wan, Yizhou Zhao, Zongrui Li, Michael Vasilkovsky, Chaoyang Wang, Jian Wang, Narendra Ahuja, Bing Zhou
TL;DR
Existing animation pipelines typically separate rigging and motion and depend on human-provided or ground-truth rig information, limiting scalable, interpretable modeling. RigMo jointly learns explicit rigs and motion from raw mesh sequences using complementary latent spaces, then applies Motion-DiT in the motion latent space. Across multiple datasets, it reports smooth, interpretable, physically consistent rigs with improved accuracy and generalization over existing methods.
Problem
Animation research commonly models structure and motion separately, while auto-rigging and motion-generation pipelines rely on human annotations or ground-truth rigs.
Method
RigMo factorizes mesh deformations into rig and motion latents that decode into Gaussian bones, skinning weights, and time-varying transformations, with Motion-DiT generating motion in the learned latent space.
Results
RigMo discovers smooth, interpretable, physically consistent rigs and outperforms existing auto-rigging and deformation methods in accuracy and generalization across multiple datasets.
Takeaways & Limitations
The unified representation provides explicit, controllable rigs and motions for scalable general-object animation and downstream motion generation.
Abstract
from arXiv · showhide
Despite significant progress in 4D generation, rig and motion, the core structural and dynamic components of animation are typically modeled as separate problems. Existing pipelines rely on ground-truth skeletons and skinning weights for motion generation and treat auto-rigging as an independent process, undermining scalability and interpretability. We present RigMo, a unified generative framework that jointly learns rig and motion directly from raw mesh sequences, without any human-provided rig annotations. RigMo encodes per-vertex deformations into two compact latent spaces: a rig latent that decodes into explicit Gaussian bones and skinning weights, and a motion latent that produces time-varying SE(3) transformations. Together, these outputs define an animatable mesh with explicit structure and coherent motion, enabling feed-forward rig and motion inference for deformable objects. Beyond unified rig-motion discovery, we introduce a Motion-DiT model operating in RigMo's latent space and demonstrate that these structure-aware latents can naturally support downstream motion generation tasks. Experiments on DeformingThings4D, Objaverse-XL, and TrueBones demonstrate that RigMo learns smooth, interpretable, and physically plausible rigs, while achieving superior reconstruction and category-level generalization compared to existing auto-rigging and deformation baselines. RigMo establishes a new paradigm for unified, structure-aware, and scalable dynamic 3D modeling.
1. Introduction
RigMo unifies rig discovery and motion modeling from raw deforming mesh sequences, addressing the annotation dependence and structural separation of prior pipelines. Its explicit, structure-aware representation supports scalable inference and downstream motion generation.
- Limitations of prior work: Existing pipelines separate rigging from motion, often relying on artist-designed skeletons, skinning weights, or ground-truth rigs.These assumptions limit scalability, consistency, and applicability to arbitrary geometries.
- Unified rig-motion learning: RigMo jointly learns rig structure and motion from raw mesh sequences without human rig annotations.It encodes dynamic sequences into complementary spatial and temporal latent spaces.
- Unified rig-motion learning: The decoder produces explicit Gaussian bones, skinning weights, and per-frame SE(3) transformations for fully animatable representations.The Gaussian-bone abstraction depends on bone count rather than mesh resolution and is resolution-insensitive.
- Downstream generation: Motion-DiT generates temporal trajectories in RigMo’s structure-aware motion latent space rather than raw vertex coordinates.This provides a path toward controllable motion synthesis and downstream motion-generation tasks.
- Experimental outcome: Across DeformingThings4D, Objaverse-XL, and TrueBones, RigMo recovers physically plausible skeletons, coherent skinning structures, and realistic motions.The reported results establish a unified, structure-aware, and controllable framework for 4D generative animation.
2. Related Works
Prior work includes latent 4D generation, auto-rigging, and inverse skinning, but these approaches commonly sacrifice controllability, generality, or scalability. RigMo instead learns shared, interpretable articulation representations that transfer across objects and motions.
- 4D Generative Models: 4D generative models commonly embed geometry and motion into latent spaces but often produce animations without explicit rigs or motion parameters.The lack of explicit structure limits practical controllability.
- Auto-Rigging: Template-based auto-rigging generalizes poorly beyond humanoid shapes, while template-free methods may produce dense skeletons or rely on handcrafted geometric cues.These methods differ in how much they constrain joint topology and geometric reasoning.
- Optimization-Based Inverse Skinning: Inverse-skinning methods recover rig parameters from articulated deformations, but classical LBS restricts realism and optimization-based solutions are difficult to scale.Recovered rigs can also remain tightly bound to individual sequences and fail to generalize across subjects or motion styles.
- RigMo-VAE: RigMo’s dual-path encoder separates static geometry from dynamic motion before decoding Gaussian-bone rig components and motion parameters.The framework learns a compact representation of spatial structure and temporal dynamics.
- Comparison with prior work: RigMo predicts rig structure and motion in one feed-forward pass, producing shared articulation representations that transfer across diverse objects and motions.This contrasts with per-sequence optimization and supports generalization across unseen categories and motion styles.
3. Method
RigMo jointly learns explicit rig structure and motion from raw mesh sequences through complementary latent spaces, then uses these representations for controllable motion generation.
- RigMo architecture: RigMo uses separate rigging and motion branches to encode spatial articulation and temporal deformation, enabling feed-forward inference of rig structure and motion.The RigMo VAE learns the rigging representation and motion parameters, while Motion DiT operates in the learned motion-latent space.
- Rigging branch: The rigging branch processes canonical geometry with topology-aware attention and bone tokens to predict Gaussian bone parameters and bone–vertex correspondences.Farthest Point Sampling selects bone tokens and coordinates, followed by cross-attention and per-bone Gaussian parameter prediction.
- Motion branch: Motion is represented by per-frame vertex displacements and decoded into local and global SE(3) transformations for deforming the mesh.The motion branch extracts temporal deformation patterns, while local and root decoders produce bone-level and global transformations.
- Gaussian skinning: Gaussian bones define soft spatial influences, whose Mahalanobis-based skinning weights and learned transformations deform vertices through Linear Blend Skinning.Each bone is an anisotropic Gaussian ellipsoid, and hierarchical transformations combine root and local bone motion before skinning.
- Gaussian skinning: Geodesic-aware refinement suppresses weights across topologically distant regions, producing cleaner skinning with vertices typically influenced by only 2–3 bones.The refinement addresses spatially close but disconnected regions such as arm–torso contacts.
- Training and motion generation: RigMo trains without rig annotations using vertex-motion reconstruction and KL regularization, while Motion DiT generates motion latents conditioned on static rigging features.The diffusion transformer predicts generated frames while retaining observed frames as conditioning signals.
4. Experiments
Experiments evaluate RigMo’s unified rig–motion representation across datasets, baselines, generalization settings, reconstruction, efficiency, and qualitative behavior. Results indicate robust rig transfer, interpretable structures, and resolution-agnostic deformation handling.
- Experimental focus: RigMo-VAE is the paper’s primary experimental focus, while Motion DiT demonstrates its learned latent space for downstream motion generation.The authors note that further Motion DiT results appear in the supplementary material.
- Datasets: The corpus combines approximately 20,000 sequences from DeformingThings4D, TrueBones, and Objaverse-XL to cover diverse deformable and articulated motions.The datasets contribute 1,972, 1,287, and 17,024 sequences, respectively.
- Rigging evaluation: RigMo learns robust rig structures from motion without ground-truth rig supervision, whereas UniRig’s plausible-looking rigs can collapse during unseen animations.The comparison reports severe deformation artifacts for UniRig under actual animation and stable deformations for RigMo.
- Rigging evaluation: Rigging evaluation uses 100 random train–test splits of DeformingThings4D to measure cross-motion rig transferability under annotation-free conditions.Training sequences optimize baseline rigs, while testing sequences evaluate transfer with rig parameters fixed.
- Reconstruction evaluation: RigMo achieves the best geometric accuracy and fastest inference among reconstruction baselines, using 48/128 tokens rather than AnimateAnyMesh’s 512-token representation.The comparison emphasizes reduced representation size, faster inference, and lower memory use.
- Qualitative analysis: Qualitative results show semantically meaningful Gaussian bone–vertex correspondences and conditional Motion DiT animations that respect learned rig constraints.The demonstrations cover diverse object categories and novel motion patterns.
- Diagnostics: Gaussian bones and motion transformations transfer across mesh resolutions because they are defined continuously in 3D space rather than tied to vertex indices.The predicted parameters can be applied back to the original mesh resolution, preserving deformation quality when tessellation changes.
- Diagnostics: Removing geodesic-aware refinement substantially reduces reconstruction quality, while increasing bone tokens from 48 to 128 improves CD-L1 by 0.018%.The refinement addresses kinematic topology, and 48 tokens offer a more favorable efficiency trade-off despite 128 tokens achieving the best quantitative reconstruction.
5. Conclusion
RigMo jointly learns rig structure and motion from raw mesh sequences without rig annotations, then uses structure-aware latents for motion generation. Experiments report smooth, interpretable, physically consistent rigs with improved accuracy and generalization.
- RigMo jointly learns rig structure and motion dynamics directly from raw mesh sequences without any rig annotations.
- Its two latent spaces decode into Gaussian bones, skinning weights, and bone transformations, producing fully animatable representations by construction.
- Motion-DiT operates on RigMo’s structure-aware latents to support controllable motion generation and downstream applications.
- Experiments report smooth, interpretable, physically consistent rigs and improved accuracy and generalization over existing auto-rigging and deformation methods.
Overview of Supplementary Materials
The supplementary materials extend the main manuscript with additional comparisons, implementation and evaluation details, ablations, and video demonstrations. These additions provide a more complete view of RigMo’s effectiveness.
- The supplement adds comparisons with prior work and side-by-side evaluations of the 48-token and 128-token RigMo-VAE configurations.
- It provides detailed Motion DiT settings, training configuration, dataset preparation, complete quantitative evaluations, and analysis of observed trends.
- Video demonstrations include 1-frame →9-frame prediction and side-by-side reconstruction comparisons.
- Together, the supplementary materials offer a more complete view of RigMo’s effectiveness and support the main-manuscript claims.
Comparison with prior work.
Prior methods separately model dynamic geometry or rig structure, often requiring many latent tokens, ground-truth rigs, or human annotations. RigMo unifies rigging and motion without manual labels to provide interpretable, generalizable representations for deformable objects.
- Existing dynamic-generation and rigging methods do not jointly model rigging and motion in a unified, self-supervised manner for general deformable objects.
- Dynamic 3D models often lack explicit kinematic structure and use hundreds of latent tokens, whereas RigMo uses 48 tokens with an interpretable rig and motion parameters.
- Human- and category-centric motion models assume ground-truth rigs, limiting their applicability when skeletons are unavailable or inconsistent across datasets.
- Static-geometry rigging methods depend on annotated datasets affected by inconsistency, noise, artist variation, and limited category coverage.
- RigMo unifies rigging and motion without manual labels, yielding interpretable rigs, consistent deformations, temporally stable trajectories, and strong generalization.
MotionDiT Evaluation
MotionDiT is evaluated with controlled sparse-conditioning experiments that combine rig-aware structural cues with observed-frame masks. The study compares conditioning variants using short- and long-horizon prediction settings on the DT4D test dataset.
- Conditioning: MotionDiT receives rig-aware cues from RigMo's rig branch, including latent rig features, skinning weights, and Gaussian bone centers.These signals are injected into the first cross-attention layer to provide articulation structure and expected deformation ranges.
- Conditioning: Observed frames are encoded as a frame-mask sequence and injected into a second cross-attention layer for controlled interpolation and long-horizon prediction.The mask specifies which frames are visible to the model.
- Sparse-conditioning settings: The evaluation tests 1-frame →1-frame prediction for local temporal smoothness and short-term motion fidelity.The model predicts the next frame from one observed frame at time t.
- Sparse-conditioning settings: The evaluation also tests 1-frame →9-frame prediction for temporally coherent and physically plausible long-horizon motion.One observed frame conditions prediction of the following nine frames.
- Ablation variants: The ablation compares no frame condition, latent-rig-only conditioning, and full MotionDiT conditioning with decoded rig attributes and frame-mask guidance.Table 5 reports Chamfer Distance (L1 / L2) on training and validation sets for the DT4D test dataset.