Source-linked AI summary
SK-Adapter: Skeleton-Based Structural Control for Native 3D Generation
Anbang Wang, Yuzhuo Ao, Shangzhe Wu, Chi-Keung Tang
TL;DR
Native 3D generators lack precise control over structural articulations, which text and image prompts do not fully specify. SK-Adapter encodes 3D skeletons as topology-aware tokens injected into a frozen backbone, achieving structural control and quality preservation while supporting local editing. The paper also introduces a 24k-pair dataset and reports performance gains over baselines, with fidelity bounded by the base model.
Problem
Native 3D generators provide fidelity and speed but lack precise structural control, while text and image prompts cannot fully prescribe 3D articulations.
Method
SK-Adapter encodes joint coordinates and topology as sparse skeletal tokens and injects them through cross-attention into a frozen 3D generation backbone.
Results
SK-Adapter achieves state-of-the-art structural fidelity and semantic consistency, including a ReRigging Score of 0.2228 versus 0.2818 for SKDream and 0.2740 for SpaceControl.
Takeaways & Limitations
The framework enables precise native 3D skeletal control and region-specific local editing while preserving the foundation model’s generative priors and quality.
Takeaways & Limitations
Generation quality and fidelity remain bounded by the pretrained 3D base model, including occasional fine-detail, facial-structure, and complex attribute failures.
Abstract
from arXiv · showhide
Native 3D generative models have achieved remarkable fidelity and speed, yet they suffer from a critical limitation: inability to prescribe precise structural articulations, where precise structural control within the native 3D space remains underexplored. This paper proposes SK-Adapter, a simple yet efficient and effective framework that unlocks precise skeletal manipulation for native 3D generation. Moving beyond text or image prompts, which can be ambiguous for precise structure, we treat the 3D skeleton as a first-class control signal. SK-Adapter is a lightweight structural adapter network that encodes joint coordinates and topology into learnable tokens, which are injected into the frozen 3D generation backbone via cross-attention. This design allows the model to not only effectively ``attend'' to specific 3D structural constraints but also preserve its original generative priors. To bridge the data gap, we contribute the Objaverse-TMS dataset, a large-scale dataset of 24k text-mesh-skeleton pairs. Extensive experiments confirm that our method achieves robust structural control while preserving the geometry and texture quality of the foundation model, significantly outperforming existing baselines. Furthermore, we extend this capability to local 3D editing, enabling region-specific editing of existing assets with skeletal guidance, which is unattainable by previous methods. Project page: https://sk-adapter.github.io/
1 Introduction
SK-Adapter addresses the difficulty of prescribing precise 3D articulations by encoding skeletal structure directly in the native 3D generation space. It uses lightweight adapter modules and extends skeletal control to local 3D editing while preserving the foundation model’s generative capabilities.
- Text and image prompts provide semantics or view-specific cues but cannot precisely prescribe full-asset 3D articulations or atypical topologies.
- 2D skeletal conditioning introduces depth ambiguity, self-occlusion errors, and potential texture or geometric degradation through multi-stage reconstruction.
- SK-Adapter treats skeletons as sparse spatial tokens containing geometric coordinates and topological constraints, injected through skeletal cross-attention layers.
- The framework freezes the pretrained backbone and trains lightweight skeleton-encoder and cross-attention modules as a parameter-efficient adaptation strategy.
- SK-Adapter extends structural control to tuning-free, region-specific 3D editing using skeleton prompts for local addition and replacement.
- It constructs Objaverse-TMS with 24k text-mesh-skeleton triplets and reports improved structural alignment and generation quality over existing baselines.
2 Related Work
Native 3D generative models use latent-space architectures to produce high-fidelity assets efficiently, replacing slower optimization and multi-view reconstruction workflows. Subsequent methods target greater structural granularity, applicability, and editability.
- Recent 3D generation has shifted from per-asset optimization and multi-view reconstruction toward native 3D generative models such as Trellis.
- These models commonly combine a VAE with a Diffusion Transformer to denoise directly in structured 3D latent or volumetric spaces.
- Native 3D models unify high fidelity and consistency while avoiding inconsistent multi-view synthesis and inefficient optimization.
3 Method
The method formulates skeleton-guided generation as mapping text and a structured 3D skeleton to an asset, then injects topology-aware skeletal features into a frozen 3D transformer. A related masked sampling procedure supports local editing.
- 3.1 Problem Formulation: The target is a 3D asset consistent with both a textual prompt and structural guidance from a 3D skeleton.
- 3.1 Problem Formulation: A skeleton comprises 3D joint coordinates and a topology graph representing hierarchical or kinematic connectivity.
- 3.1 Problem Formulation: The learning objective preserves skeletal structure while maintaining the quality and diversity of large-scale 3D generative priors.
- 3.2 SK-Adapter: SK-Adapter encodes joint-based positional tokens and injects them into Trellis sparse structure-transformer blocks for native 3D generation.
- 3.2 SK-Adapter: GRPE represents the skeleton as a graph and incorporates topological distances and relations into attention through structural biases.
- 3.2 SK-Adapter: The resulting skeletal embedding integrates geometric coordinates with hierarchical topology, while cross-attention lets voxel features query skeletal information.
- 3.2 SK-Adapter: A zero-initialized residual projection begins as a null signal and gradually modulates voxel latents without destabilizing the pretrained distribution.
- 3.2 SK-Adapter: Training uses latent flow matching while freezing the transformer and optimizing the GRPE encoder, cross-attention, and projection layers.
4 Experiments
Experiments evaluate SK-Adapter using the Objaverse-TMS dataset, TMS-eval benchmark, alignment and fidelity metrics, baseline comparisons, editing demonstrations, and ablations. Results show stronger structural control and visual quality, faster generation, and skeleton-guided local editing.
- Dataset: Objaverse-TMS comprises 24K text-mesh-skeleton triplets assembled from aligned modalities after filtering incomplete rigging.The dataset covers articulated assets including humanoids, animals, and other objects.
- Evaluation Setup: TMS-eval contains 140 balanced test instances spanning humanoids, animals, and other objects.The benchmark includes 54 humanoids, 63 animals, and 23 objects.
- Quantitative Results: SK-Adapter achieves a ReRigging Score of 0.2228, versus 0.2818 for SKDream and 0.2740 for SpaceControl, while also obtaining the highest overall CLIP Score.Lower ReRigging Scores indicate better structural adherence.
- Quantitative Results: SK-Adapter records an overall PickScore of 21.01 and KD-DINO of 0.7778, compared with KD-DINO values of 1.3809 for SKDream and 1.7821 for SpaceControl.These results assess human-aligned perceptual quality and distributional visual fidelity.
- Runtime: SK-Adapter and SpaceControl generate assets in less than 15 seconds, whereas SKDream requires around 40 seconds per sample.The comparison uses the same hardware conditions.
- Editing: Skeleton-guided editing supports local addition and re-posing while preserving coherence between the edited mesh and the original asset.Editing uses an identified bounding box and an edited skeleton.
- Ablation Study: Removing skeletal cross-attention increases the ReRigging Score from 0.2355 to 0.5049, while the full model retains the highest PickScore across categories.Removing topology encoding also reduces articulation accuracy, demonstrating the importance of both architectural components.
5 Conclusion
SK-Adapter adds topology-aware skeletal control to native 3D generation while preserving the foundation model’s generative priors. The paper builds on TRELLIS’s rectified-flow, structured-latent pipeline and trains only lightweight adapter components.
- Conclusion: SK-Adapter encodes 3D skeletons as sparse, topology-aware spatial tokens and injects them into a frozen 3D flow transformer.The framework targets faithful structural alignment without discarding the backbone’s pretrained 3D knowledge.
- Conclusion: The conclusion positions skeleton control as a bridge between semantic synthesis and explicit topological articulation for more interpretable and controllable 3D generation.The stated application scope includes production-ready, articulable 3D assets.
- Conclusion: TRELLIS transforms noise into data through a deterministic rectified-flow path and represents 3D assets using local structured latents on a grid.Its pipeline separately generates coarse spatial structure and local feature vectors.
- Conclusion: GRPE uses topological-distance and edge-relation codebooks to encode pairwise structural relationships in skeleton attention.The skeleton is represented as a rooted tree with joint coordinates and connectivity.
B.1 Topological Distance Codebook ED
The topological distance codebook represents how far apart joints are in the skeleton tree, clips long paths, and supplies learned embeddings to attention components.
- Topological Distance Codebook: Topological distance D_ij is the shortest-path length between joints i and j in the skeleton tree.The distance counts edges on the unique tree path connecting the joints.
- Topological Distance Codebook: Distances are clipped at d_max to keep the embedding table compact and reduce overfitting to rare long-range joint pairs.This clipping maps longer paths to a bounded set of distance indices.
- Topological Distance Codebook: d_max = 5 yields d_max + 1 = 6 discrete distance levels in the experiments.The levels are enumerated in Table 6.
- Topological Distance Codebook: Three independent embedding matrices are learned for the distance codebook, with rows selected by the clipped distance index and injected into attention.The embeddings enter the corresponding attention components described in the main text.
B.2 Edge Relation Codebook ER
The edge relation codebook captures the semantic and directed roles connecting skeleton joints, complementing distance with hierarchical relationship information.
- Edge Relation Codebook: The edge relation codebook defines six mutually exclusive relation types for ordered joint pairs.The relation types are summarized in Table 7.
- Edge Relation Codebook: Diagonal entries distinguish leaf joints as End-Effector (5) from other joints as Self (0).The dedicated end-effector embedding reflects the role of fingertips, toes, and head as inverse-kinematics targets.
- Edge Relation Codebook: Off-diagonal relations are asymmetric for Parent and Child, while Sibling is symmetric and marks branching joints sharing a parent.These relations encode directed kinematic structure and branching points.
- Edge Relation Codebook: Three embedding matrices are learned for the relation codebook and incorporated into the attention mechanism.The relation embeddings complement the topology-aware attention design.
B.3 Summary
The topology codebook combines compact distance and relation embeddings to provide attention with structural inductive biases for skeleton-guided generation.
- Summary: The topology codebook contains six learnable embedding matrices and 36F parameters in total.There are three matrices for distances and three for relations.
- Summary: For F = 1024, the codebook adds only approximately 37K parameters relative to the backbone.The paper characterizes this as negligible overhead.
- Summary: Distance embeddings encode proximity along the kinematic chain, while relation embeddings encode semantic structural roles.Together they provide rich structural inductive biases to attention.
C SK-Adapter Configuration
SK-Adapter adds a lightweight adapter to the Trellis backbone through cross-attention while retaining a frozen foundation model. Its trainable parameters are broken down in Table 8, with the architecture compared against Trellis’s Structure Flow Transformer.
- Parameter Configuration: The topology codebook contains 73,728 parameters, and the total adapter overhead is ∼151M trainable parameters.The codebook comprises 6 tables × 2 layers.
- Architecture: SK-Adapter connects to the backbone through an additional cross-attention layer in each DiT block.This distinguishes the adapter architecture from Trellis’s Structure Flow Transformer.
D More Visualizations
The visualizations present additional qualitative generations and examples of skeleton-guided editing. Together, they illustrate both asset synthesis and editing use cases.
- Generated Assets: Additional qualitative results show assets generated with SK-Adapter.
- Editing: Editing examples illustrate skeleton-guided modifications of existing 3D assets.
F Limitation
SK-Adapter’s quality remains constrained by the pretrained 3D base model, especially for fine details, semantic specificity, and highly complex skeletons. The authors propose stronger foundation models, text encoders, and larger-scale training as future remedies.
- Base-Model Constraints: Generation quality and fidelity remain bounded by the capabilities of the pretrained 3D base model.
- Generation Quality: Fine geometric and textural details can remain rough, while faces may exhibit distorted or structurally incorrect features.Complex or highly specific prompt attributes may also fail to transfer perfectly into final geometry and texture.
- Structural Complexity: When conditioning skeletons become too complex, ambiguous guidance can produce messy, poorly resolved local geometries.The paper highlights intersecting finger topology in a Spider-Man hand as an example.
- Future Work: Future work targets higher-resolution 3D foundation models, more powerful text encoders, and larger-scale SK-Adapter training.