PaperlayerView exact revision on arXiv

Source-linked AI summary

Skin Tokens: A Learned Compact Representation for Unified Autoregressive Rigging

Jia-peng Zhang, Cheng-Feng Pu, Meng-Hao Guo, Yan-Pei Cao, Shi-Min Hu

arXiv:2602.04805v1cs.GRcs.AI

TL;DR

Automated rigging remains limited by inefficient, decoupled skinning-weight prediction. TokenRig introduces compact SkinTokens and unified autoregressive rig generation, producing rigs that approach professional artist workflows in fidelity.

  • Problem

    Existing skinning methods treat sparse skinning matrices as high-dimensional regression targets, making prediction inefficient and largely separate from skeleton generation.

  • Method

    SkinTokens discretizes skinning weights with an FSQ-CVAE, while TokenRig autoregressively generates skeletal parameters and SkinTokens in one unified sequence.

  • Results

    TokenRig produces automated skeletal rigs and skinning predictions that approach the fidelity of professional artist workflows.

  • Takeaways & Limitations

    The framework provides a unified generative approach to automated rigging and skinning for diverse 3D models.

  • Takeaways & Limitations

    The reinforcement-learning stage improves geometric validity, but physics-based rewards remain a future direction for ensuring dynamically plausible deformations.

Abstract

from arXiv · show

The rapid proliferation of generative 3D models has created a critical bottleneck in animation pipelines: rigging. Existing automated methods are fundamentally limited by their approach to skinning, treating it as an ill-posed, high-dimensional regression task that is inefficient to optimize and is typically decoupled from skeleton generation. We posit this is a representation problem and introduce SkinTokens: a learned, compact, and discrete representation for skinning weights. By leveraging an FSQ-CVAE to capture the intrinsic sparsity of skinning, we reframe the task from continuous regression to a more tractable token sequence prediction problem. This representation enables TokenRig, a unified autoregressive framework that models the entire rig as a single sequence of skeletal parameters and SkinTokens, learning the complicated dependencies between skeletons and skin deformations. The unified model is then amenable to a reinforcement learning stage, where tailored geometric and semantic rewards improve generalization to complex, out-of-distribution assets. Quantitatively, the SkinTokens representation leads to a 98%-133% percents improvement in skinning accuracy over state-of-the-art methods, while the full TokenRig framework, refined with RL, enhances bone prediction by 17%-22%. Our work presents a unified, generative approach to rigging that yields higher fidelity and robustness, offering a scalable solution to a long-standing challenge in 3D content creation.

1 introduction

The introduction identifies skinning prediction as a high-dimensional, largely unsolved bottleneck worsened by its decoupling from skeleton generation. It proposes SkinTokens and TokenRig as unified discrete and autoregressive solutions, with reinforcement learning for rig refinement.

  • Problem: Skinning remains a significant, largely unsolved challenge because prior methods regress the entire high-dimensional N×J skinning matrix from geometric features.Direct regression treats skinning as a separate downstream problem.
  • Problem: Decoupled skeleton and skinning models prevent mutual reinforcement, producing skeletons without deformation knowledge and skinning for potentially suboptimal skeletal structures.The performance ceiling is further constrained by scarce datasets containing comprehensive skeleton and skinning annotations.
  • SkinTokens: SkinTokens is a learned, compact, discrete skinning representation that reframes continuous regression as discrete token prediction using an FSQ-CVAE conditioned on local mesh geometry.This representation is intended to make unified modeling possible.
  • TokenRig: TokenRig is a unified, end-to-end autoregressive framework that generates a coherent sequence interleaving skeletal parameters with corresponding skin tokens.The formulation models cross-modal dependencies between skeletal placement and surface skins, which decoupled approaches ignore.
  • Reinforcement learning: The proposed rig-refinement framework uses reinforcement learning with novel reward functions to improve the generalization and robustness of generated rigs.Its generative formulation makes TokenRig suitable for reinforcement-learning refinement.

2 Related Works

Prior work progresses from geometry-driven skeleton and skinning heuristics to learned, template-free, and autoregressive methods. Yet skinning remains commonly formulated as unstable, expensive high-dimensional regression, motivating SkinTokens as an alternative representation.

  • Skeleton Generation: Early skeletonization methods use geometric or topological heuristics, including signed distance fields, medial representations, voxel cores, and erosion thickness.These approaches infer skeletal structures without data-driven priors.
  • Skeleton Generation: Deep-learning methods shift skeleton generation toward data-driven priors, with template-based approaches trading high fidelity for poor generalization beyond fixed character classes.Template-free methods such as RigNet and MoRig instead employ graph neural networks to address arbitrary characters.
  • Skeleton Generation: Recent methods discretize skeletal hierarchies into tokens and use Transformers to model skeleton generation autoregressively, inspired by sequential editing workflows and LLMs.This reformulation reflects professional 3D software workflows and aims to capture global structural dependencies.
  • Skinning: Skinning methods range from geometry-based segmentation and heat diffusion to learned graph models and alternative sparse representations, but they generally regress high-dimensional weight matrices.Dense regression is unstable and computationally expensive for disconnected or non-watertight meshes, while geodesic descriptors reduce robustness on complex topologies.

3 Method · 3.1 Overview

The method reframes automatic rigging as unified generative sequence modeling through three stages, beginning with SkinTokens, a discrete representation for skinning weights learned via an FSQ-CVAE.

  • 3.1 Overview: The method reframes automatic rigging as a unified, generative sequence modeling task.The overview refers to Figure 2 for the overall method.
  • 3.1 Overview: The approach is organized into three key stages.
  • 3.1 Overview: SkinTokens provide a novel discrete representation for skinning weights.
  • 3.1 Overview: SkinTokens are learned using an FSQ-CVAE.The passage cites Mentzer et al. 2023 and Sohn et al. 2015 in connection with the FSQ-CVAE.
  • 3.1 Overview: The representation transforms skinning-weight regression from a high-dimensional, sparse matrix problem into token prediction.
  • 3.1 Overview: The resulting token prediction task is described as tractable compared with the original regression problem.The original problem is characterized as intractable because it involves a high-dimensional, sparse matrix.

3.2 SkinTokens: A Learned Discrete Representation for Skinning

SkinTokens reframe skinning-weight prediction as learning a compact, discrete representation from sparse weight matrices conditioned on mesh geometry. An FSQ-CVAE encodes unordered mesh and skinning-weight point sets, quantizes latent weight features into tokens, and reconstructs bounded per-vertex weights using sparsity-aware supervision.

  • Representation learning: The FSQ-CVAE reconstructs per-bone skinning weights conditioned on full mesh geometry, using mesh features and latent weight features as inputs.The discrete tokens are concatenated with shape features and decoded into per-vertex weights with a final sigmoid activation bounded in [0, 1].
  • Representation learning: SkinTokens are learned by quantizing continuous skinning latent features into a discrete representation suitable for sequence modeling.VecSet-style encoders separately process unordered mesh and skinning-weight point sets before producing the quantized tokens.
  • Training strategy: Nested dropout and hybrid point sampling improve compositional robustness and concentrate decoder capacity on regions with non-zero ground-truth weights.The decoder receives uniformly sampled surface points together with dense samples from important deformation regions, while the shape encoder uses uniform samples to match inference.
  • Sparsity-aware training: Dice loss strengthens supervision for non-zero skinning weights, counteracting the extreme class imbalance that weakens standard BCE training.The implementation combines BCE, Dice, and a small MSE term; Dice uses ε = 10^-4 for numerical stability.

3.3 TokenRig: Unified Autoregressive Modeling

TokenRig represents the complete rig as one discrete sequence combining skeleton tokens with SkinTokens, enabling unified autoregressive generation. Because skinning follows the fully generated skeleton, self-attention models long-range dependencies beyond local features.

  • Unified representation: TokenRig formulates rigging as unified sequence generation by representing skeleton and skinning as a single coherent sequence of discrete tokens.This moves beyond decoupled, multi-stage pipelines.
  • Skeletal tokenization: Skeletal tokenization serializes joint coordinates as uniformly quantized integer tokens and orders bones using predefined templates or chain-part structure.Each coordinate token contains d_xi, d_yi, and d_zi.
  • Sequential SkinTokens: Following the complete skeletal sequence, each bone’s skinning influence is appended as a sequence of T_D discrete SkinTokens from the pretrained FSQ-CVAE.The sequence uses boundary markers and bone-chain type tokens as structural identifiers.
  • Global conditioning: Conditioning SkinToken generation on the fully generated skeleton lets self-attention access all joint positions and bone types, capturing long-range dependencies in skinning.This holistic conditioning improves on methods that predict skinning from local features alone.

3.4 Generalization via Reinforcement Learning Refinement

TokenRig adds a GRPO-based post-training refinement stage to address out-of-distribution rigging failures by directly optimizing explicit geometric and semantic rewards. Four rewards target joint coverage, bone containment, skinning coverage and sparsity, and deformation smoothness, with group-relative advantages stabilizing policy updates.

  • Refinement Method: GRPO refines TokenRig against explicit, non-differentiable geometric and semantic rewards instead of costly annotated preference data.The method optimizes professional rigging criteria directly and uses group-relative baselines rather than a separate critic network.
  • Reward Design: Volumetric Joint Coverage encourages joints throughout significant mesh parts, preventing missing bones in extremities.The mesh is voxelized at resolution r^3 with r=196, and voxel-joint distances are aggregated using an exponential kernel with α=0.05.
  • Reward Design: The four-reward suite penalizes missing joints, protruding bones, unbound or overly dense skinning, and distorted animated deformations.These rewards address common failures including bone protrusion, unconnected vertices, excessive skinning density, and spiky deformation artifacts.
  • Reward Design: Bone-Mesh Containment penalizes sampled bone points outside occupied voxels, keeping the generated skeleton geometrically consistent with the mesh.The reward checks uniformly sampled points along each generated bone for containment within the voxelized mesh.
  • Reward Design: Deformation Smoothness evaluates Linear Blend Skinning under 5 randomly sampled poses and penalizes edge-length distortion to preserve local surface geometry.The motion reward targets spiky or distorted animation artifacts, using ε=10^-6 for numerical stability.
  • Policy Optimization: Group-relative advantages normalize rewards across sampled outputs, stabilizing training and enabling TokenRig to self-correct generation logic based on geometric validity.For each input mesh, the policy samples a group of outputs, verifies structural consistency, decodes them, and computes normalized advantages.

4 Experiment · 4.1 Implementation and Experimental Setup

The experiments use a diverse, normalized composite dataset and robustness-oriented augmentations to model varied topologies, articulation styles, and noisy inputs. SkinTokens and TokenRig are trained with compact architectures, specified token budgets, hybrid optimization, and a reward-weighted post-training stage.

  • 4.1 Dataset Configuration: The composite dataset combines Articulation2.0 (70%), VRoid Hub (20%), and ModelsResource (10%) to cover diverse topologies and articulation styles.All geometry is normalized to the canonical unit cube [−1, 1]3.
  • 4.1.1 Dataset Configuration: SkinTokens latent vectors form anatomical clusters by bone category, indicating learned body-part concepts invariant to specific mesh geometries.The visualization samples 300 instances from the VRoid dataset before quantization.
  • 4.1.2 Robustness-Oriented Data Augmentation: The augmentation pipeline simulates topological imperfections and irregular structures found in in-the-wild assets.It combines structural and geometric perturbations to improve robustness.
  • 4.1.2 Robustness-Oriented Data Augmentation: With p=0.5, training deletes up to 50% of joints or subtrees and reconnects up to 30% of joints to new parents during VAE training.Reconnected joints merge their skinning weights to simulate topology edits.
  • 4.1.3 Model Architecture and Training: SkinTokens uses a 110M-parameter 3DShape2Vecset backbone, an asymmetric 2-layer encoder and 10-layer decoder, and a 64,000-entry FSQ vocabulary.Training uses at most T_D=32 skin tokens and T_W=384 shape tokens as auxiliary conditions.
  • 4.1.3 Model Architecture and Training: TokenRig adopts Qwen3-0.6B with GQA and RoPE, while training uses Muon for attention layers and AdamW for remaining parameters.FSQ-CVAE training runs 400k iterations at batch size 320, and TokenRig runs 300k iterations at batch size 160.
  • 4.1.3 Model Architecture and Training: Post-training uses 800 steps on high-complexity AI-generated meshes with learning rate 10^-6, group size G=24, clip ratio ε=0.2, and KL penalty β=0.1.Reward weights emphasize volumetric joint coverage with w_vj=5, while w_vk=w_sc=w_mo=1.

4.2 Analysis of the SkinTokens Representation

SkinTokens preserve skinning quality despite aggressive compression, concentrating relevant information in a small number of discrete tokens. The learned latent space also organizes skinning patterns into anatomically meaningful clusters across varied meshes.

  • Evaluation metric: IoU is used alongside MAE because near-zero predictions can achieve low MAE while missing sparse, critical influence regions.Weights are categorized as active when they exceed ε=10^-2.
  • Qualitative comparison: TokenRig produces clean, locally coherent influence maps that closely match ground truth, especially in fine-grained regions such as fingers.Baseline predictions exhibit bleeding artifacts in which weights spill onto unconnected mesh parts.
  • Compact reconstruction: IoU scores remain robust as the token count T_D decreases, showing that relevant skinning information is concentrated in the top few tokens.This demonstrates high reconstruction fidelity with a highly compact representation.
  • Compression performance: 183.74× compression ratio is achieved with the selected C=[8, 8, 8, 5, 5, 5] configuration while retaining 86.2% codebook utilization.The configuration targets a codebook size of 64,000, reducing the skinning matrix to a discrete SkinToken sequence.
  • Latent structure: The continuous latent space forms distinct, well-separated clusters for anatomical parts such as Head, Hips, and LeftLeg across 300 VRoid instances.Clusters are colored by bone identifiers mapped to the Mixamo template despite substantial geometric variation across source meshes.

4.3 Comparison

TokenRig outperforms established rigging baselines in skeletal generation and skinning prediction across the evaluated datasets and metrics. Its advantages include improved skeletal topology, artifact-free local skinning weights, and lower deformation distortion.

  • Skeletal Generation: TokenRig achieves the lowest J2J and B2B Chamfer Distance errors across both evaluated datasets, indicating accurate joint placement and topological alignment.Skeletal structure is evaluated with J2J, J2B, and B2B Chamfer Distance metrics on ModelsResource and Articulation 2.0.
  • Skeletal Generation: Baseline comparisons reveal that RigNet often misses terminal skeleton chains, whereas UniRig over-segments meshes with excessive joints and irregular topology.The reported weaknesses are attributed to RigNet’s MST-based connectivity inference and UniRig’s tendency to over-segment meshes.
  • Skinning Prediction: 0.0163 vs. 0.0573 on ModelsResource is TokenRig’s L1 Error compared to RigNet, demonstrating substantially lower skinning reconstruction error.Skinning evaluation uses a weight threshold of ε=10^-2 and includes Precision, Recall, Motion Loss, L1 Error, and L1 Variance.
  • Skinning Prediction: TokenRig produces cleaner, artifact-free weight maps because its FSQ-CVAE decoder enforces strict locality, unlike baselines that exhibit bleeding across disconnected mesh components.The improvements are especially visible in complex articulations, where baseline weights may be over-smoothed or bleed across regions.
  • Skinning Prediction: TokenRig’s superior Motion Loss scores indicate that its skinning weights reduce distortion during actual deformations, not merely static reconstruction error.Motion Loss measures deformation fidelity under linear blend skinning.

4.4 Ablation Study

The ablations validate Dice Loss, GRPO post-training, and robustness-oriented data augmentation as critical components of the framework. Dice Loss improves sparse skinning optimization, GRPO supports out-of-distribution generalization, and augmentation modules improve skeletal prediction robustness.

  • Reinforcement Learning: GRPO fine-tuning maintains or improves performance on standard benchmarks while extending generalization to out-of-distribution assets.Its qualitative gains are shown on complex in-the-wild meshes that differ substantially from the training data.
  • Reinforcement Learning: GRPO-trained models synthesize coherent skinning for auxiliary structures such as wings, capes, and tails that the base model sometimes ignores or handles ambiguously.The improvements are guided by volumetric coverage and bone-mesh containment rewards.
  • Data Augmentation: Removing any single augmentation module consistently degrades skeletal prediction accuracy across all datasets.The evaluated modules include non-uniform scaling, sub-tree dropping, and random joint deletion, using T_D = 4.
  • Data Augmentation: Random joint deletion is particularly important because simulating topological imperfections improves robustness to diverse and noisy production geometry.Removing joint deletion strategies increases prediction error and confirms the value of modeling imperfect topology during training.
  • Loss Function Design: Removing Dice supervision significantly degrades IoU performance across all datasets under the sparse skinning-weight setting.The ablation compares a baseline using codebook size C = [8, 8, 8, 5, 6] = 15,360 and T_D = 32 skin tokens against training with only BCE and MSE.

5 Conclusion

TokenRig is presented as an automated skeletal rigging and skinning framework approaching professional artist workflows, grounded in the view that automatic skinning is fundamentally a representation problem. Future work includes user guidance and physics-based rewards to improve flexibility and dynamic plausibility.

  • Contributions: TokenRig provides automated skeletal rigging and skinning weight prediction approaching the fidelity of professional artist workflows.The conclusion identifies representation as the longstanding bottleneck in automatic skinning.
  • Future Work: User-specified topological templates or interactive guidance could transform TokenRig into a flexible artist-directed co-pilot.This is proposed as a direction for extending the autoregressive model.
  • Future Work: Physics-based rewards could further ensure the dynamic plausibility of generated deformations during animation.The conclusion notes that the current reinforcement-learning stage improves geometric validity.
Loading 2602.04805v1…