Source-linked AI summary
Stroke3D: Lifting 2D strokes into rigged 3D model via latent diffusion models
Ruisi Zhao, Haoren Zheng, Zongxin Yang, Hehe Fan, Yi Yang
TL;DR
Existing 3D generation methods often lack animatable geometry, while skeleton creation offers limited structural control and paired training data are scarce. Stroke3D generates rigged meshes from 2D strokes and text using latent graph-based skeleton generation followed by TextuRig-augmented, SKA-DPO-refined mesh synthesis. The method reports plausible skeletons, high-quality aligned meshes, and strong benchmark performance.
Problem
Existing methods struggle to generate animatable geometry and provide fine-grained skeleton control, while high-quality paired skeleton-textured-mesh data are scarce.
Method
Stroke3D uses Sk-VAE and Sk-DiT for stroke- and text-conditioned skeleton generation, then synthesizes meshes using TextuRig-augmented SKDream and SKA-DPO.
Results
Stroke3D produces plausible skeletons and high-fidelity rigged meshes aligned with user inputs, with strong results on skeleton and mesh benchmarks.
Takeaways & Limitations
The framework enables a skeleton-first workflow for creating ready-to-animate 3D assets directly from intuitive 2D strokes and text prompts.
Takeaways & Limitations
Performance deteriorates for rare or underrepresented concepts, whose generated skeletons can be unstable or caption-sensitive.
Abstract
from arXiv · showhide
Rigged 3D assets are fundamental to 3D deformation and animation. However, existing 3D generation methods face challenges in generating animatable geometry, while rigging techniques lack fine-grained structural control over skeleton creation. To address these limitations, we introduce Stroke3D, a novel framework that directly generates rigged meshes from user inputs: 2D drawn strokes and a descriptive text prompt. Our approach pioneers a two-stage pipeline that separates the generation into: 1) Controllable Skeleton Generation, we employ the Skeletal Graph VAE (Sk-VAE) to encode the skeleton's graph structure into a latent space, where the Skeletal Graph DiT (Sk-DiT) generates a skeletal embedding. The generation process is conditioned on both the text for semantics and the 2D strokes for explicit structural control, with the VAE's decoder reconstructing the final high-quality 3D skeleton; and 2) Enhanced Mesh Synthesis via TextuRig and SKA-DPO, where we then synthesize a textured mesh conditioned on the generated skeleton. For this stage, we first enhance an existing skeleton-to-mesh model by augmenting its training data with TextuRig: a dataset of textured and rigged meshes with captions, curated from Objaverse-XL. Additionally, we employ a preference optimization strategy, SKA-DPO, guided by a skeleton-mesh alignment score, to further improve geometric fidelity. Together, our framework enables a more intuitive workflow for creating ready to animate 3D content. To the best of our knowledge, our work is the first to generate rigged 3D meshes conditioned on user-drawn 2D strokes. Extensive experiments demonstrate that Stroke3D produces plausible skeletons and high-quality meshes.
1 INTRODUCTION
Stroke3D addresses limitations in animatable 3D generation and skeleton control by generating rigged meshes from 2D strokes and text through a skeleton-first pipeline. It combines latent graph-based skeleton generation with augmented mesh synthesis and reports strong skeleton and mesh results.
- Existing 3D methods often produce static geometry without skeletal hierarchies, while skeleton-conditioned mesh generation is limited by scarce paired skeleton-textured-mesh data.
- Stroke3D generates rigged 3D meshes from user-drawn 2D strokes and text prompts through a skeleton-first workflow.The framework is designed for non-professional users and separates skeleton generation from mesh synthesis.
- Sk-VAE encodes skeleton graphs into latent space, while Sk-DiT generates embeddings conditioned on strokes and text before reconstructing a 3D skeleton.Stroke features provide structural conditioning, and text supplies semantic guidance through cross-attention.
- TextuRig augments SKDream training with textured, rigged models and captions, while SKA-DPO further refines mesh geometric fidelity and skeleton alignment.TextuRig is curated from Objaverse-XL with texture verification and detailed recaptioning.
- Stroke3D achieves the lowest Chamfer Distance on most skeleton metrics and improves MeanInst. SKA score by nearly 10 points over the SKDream baseline.These evaluations use the MagicArticulate and SKDream benchmarks for skeleton and mesh generation, respectively.
- Stroke3D achieves plausible skeletons and high-fidelity rigged meshes that align with user inputs, with automatic skinning tools enabling fully rigged assets.
2 RELATED WORK
Related work spans diffusion-based 3D generation, mesh-to-skeleton rigging, and reinforcement learning for diffusion models. Stroke3D combines these directions in a latent graph-diffusion pipeline with skeleton-mesh alignment optimization.
- 3D Generation: Diffusion-based 3D generation evolved from SDS-optimized NeRFs toward multistage reconstruction and transformer-based models conditioned on images or point clouds.
- Skeleton Generation: Traditional rigging methods predict skeletons from meshes, while newer approaches investigate direct skeleton generation and related controllable formulations.
- RL in Diffusion: Reinforcement-learning tools have been applied to diffusion models to improve generation quality, using signals such as human preference, multiview consistency, or VLM evaluation.
- RL in Diffusion: Stroke3D applies SKA-DPO to refine skeleton-to-mesh generation with a skeleton-mesh alignment reward signal.
3 METHOD
Stroke3D prepares skeleton and mesh data, then generates 3D skeletons from graph structure, text, and user-drawn 2D strokes before synthesizing aligned meshes. Its mesh stage combines TextuRig data with preference-guided optimization, while evaluation compares skeleton-generation quality using Chamfer Distance.
- Data Preparation: Stroke3D prepares skeleton data with descriptive captions, curates the textured and rigged TextuRig dataset, and records user-drawn skeleton topology through a canvas tool.TextuRig is built from Objaverse-XL assets filtered by UniRig, with texture verification and model re-captioning.
- Skeletal Graph Variational Autoencoder: The Sk-VAE represents a 3D skeleton as a graph and learns latent structural embeddings by aggregating information between neighboring joints.The graph contains 3D joint coordinates and edges defining skeleton topology; the encoder uses GCN and TransformerConv components.
- Skeletal Graph Diffusion Transformer: Sk-DiT generates latent skeleton embeddings with graph-aware self-attention and text cross-attention, while conditioning denoising on 2D joint coordinates, edge topology, and text.TransformerConv restricts attention to connected graph nodes, and classifier-free guidance is used for textual conditioning.
- Enhanced Mesh Synthesis via TextuRig and SKA-DPO: The mesh synthesis stage augments skeleton-to-mesh training with TextuRig because existing rigging data often lacks corresponding texture information.TextuRig contains textured, rigged models with captions selected through a targeted curation pipeline.
- Enhanced Mesh Synthesis via TextuRig and SKA-DPO: SKA-DPO ranks multi-view mesh candidates by skeleton-mesh alignment and fine-tunes the model to favor higher-scoring geometries.The preference dataset pairs winning and losing candidates, and the objective encourages more accurate noise prediction for winning samples than losing samples relative to a reference model.
- Evaluation: Skeleton quality is evaluated with CD-J2J, CD-J2B, and CD-B2B across three categories, with lower and second-lowest scores highlighted in the table.The method also uses Figure 4 for qualitative comparison of skeleton generation.
4 EXPERIMENT
Stroke3D is evaluated on skeleton and skeleton-to-mesh generation using established benchmarks and alignment metrics. Results show improved structural accuracy, mesh alignment, convergence, and robustness to sparse sketches.
- Evaluation setup: Stroke3D is evaluated on MagicArticulate for skeleton generation and SKDream for skeleton-to-mesh generation, with CD-based skeleton metrics and SKA mesh-alignment scores.The mesh evaluation follows SKDream’s protocol, using 108 test samples and four inference views.
- Skeleton evaluation: Stroke3D achieves the lowest overall skeleton error across categories and substantially lower Chamfer Distance than RigNet and SKDream.The evaluation uses Joint-to-Joint, Joint-to-Bone, and Bone-to-Bone Chamfer Distance metrics.
- Mesh evaluation: MeanInst. and MeanClass reach 87.84 and 84.21 after applying SKA-DPO, with smoother meshes that more faithfully conform to conditioning skeletons.Qualitative comparisons report fewer geometric artifacts than vanilla SKDream.
- Ablation studies: A structural Jxy condition accelerates convergence in small-data ablations, and removing it makes large-scale training struggle to converge.Models trained with Jxy achieve lower loss more quickly in both one-sample and five-sample experiments.
- Ablation studies: A preference-score margin of 0.10 provides the best trade-off across evaluation scores, while the model remains stable when fewer than five joints are dropped.The sparsity study indicates resilience to minor occlusions or incomplete sketches.
5 CONCLUSION
Stroke3D separates controllable skeleton generation from mesh synthesis to create ready-to-animate assets from 2D strokes and text prompts. Experiments report improved skeleton quality, mesh alignment, and fidelity to user intent.
- Conclusion: Stroke3D directly generates ready-to-animate 3D assets from user-drawn 2D strokes and text prompts through a two-stage pipeline.The stages are controllable skeleton generation followed by enhanced mesh synthesis.
- Conclusion: Sk-VAE and Sk-DiT produce high-quality 3D skeletons with precise structural accuracy from the user inputs.The skeleton-generation stage is designed to provide structural and semantic control.
- Conclusion: TextuRig and SKA-DPO produce detailed geometry and textures that remain well aligned with the generated skeleton.The conclusion attributes the enhanced mesh synthesis to the curated dataset and preference optimization.
- Conclusion: Experimental results report that Stroke3D outperforms existing methods and supports an intuitive workflow for animation applications.The conclusion frames the method as a streamlined workflow for creating complex rigged assets.
APPENDICES
The appendix documents dataset selection, canonical orientation, alignment heuristics, and an ablation framework for skeleton generation. It combines joint names, geometry, principal directions, and VLM assistance to handle varied models.
- Data preparation: The dataset uses Articulation-XL 2.0 and filters models to character, anthropomorphic, animal, and plant categories with 0–30 skeleton nodes.The selected range is described as covering the most common articulated assets.
- Data preparation: Canonical alignment sets facing, up, and right directions to +Z, +Y, and +X, mapping XY, XZ, and YZ planes to front, top, and side views.This standardization addresses inconsistent orientations in the source data.
- Alignment methods: Human-like models are aligned from joint-name keywords when available, while incomplete names trigger symmetry-based geometric alignment.The structural method uses nearly symmetric bone pairs, with the longest pair treated as the legs.
- Alignment methods: Principal-direction alignment uses PCA for human-like models and average bone directions for plants and animals, with axis correction to preserve a right-handed frame.The method adapts its vertical-axis prior to the model category.
- Ablation studies: The training-data ablation compares CD-J2J, CD-J2B, and CD-B2B across five categories to assess skeleton-generation data choices.The appendix identifies the lowest and second-lowest scores for comparison.
- Alignment methods: VLM assistance aligns difficult animal cases by identifying front, side, and top views from three orthographic renders, although it is slower.The method is intended for complex poses where heuristics fail.
- Alignment methods: Preprocessing selects a root joint, splits bones into segments, downweights extremely long bones, and simplifies dense branching by averaging representative segments.These steps reduce alignment bias from tails and highly branched structures such as hands.
A.5 EFFECTIVENESS OF ALIGNMENT METHODS
Alignment reliability depends on the available structural cues and the model category. Joint names are strongest, while animal diversity makes principal-direction heuristics less reliable and motivates VLM assistance.
- Alignment effectiveness: Joint-name alignment is the most robust method and nearly guarantees 100% success when sufficient joint names are available.This method is used when semantic joint annotations provide enough directional information.
- Alignment effectiveness: Structural alignment recognizes most legs but can confuse front and back, causing some models to face backward.Its main failure mode is directional ambiguity rather than inability to identify articulated structure.
- Alignment effectiveness: Principal-direction alignment works well for most human-like and plant models but performs poorly on animals with diverse body orientations.Penguins and crocodiles illustrate why animals lack a shared main axis.
- Alignment effectiveness: The VLM-assisted method aligns most animals, but texture-free three-view renders still leave a small fraction incorrectly judged.In practice, the first three heuristics handle character, anthropomorphic, and plant models, while VLM assistance handles animals.
B.1 TRAINING DATA AND TAGGING STRATEGY
The training strategy combines descriptive and viewpoint tags with rotational alignment, while dataset size has the strongest influence on skeleton-generation performance.
- Tagging and alignment: Descriptive and viewpoint tags were stochastically added to prompts, while rotational alignment was used to improve semantic and geometric consistency.Validation prompts included all available tags consistently.
- Training settings: Four settings compare reduced-data, raw, no-tag, and full training configurations with tagging and rotational alignment.The full setting combines the complete dataset with both tagging and alignment.
- Qualitative analysis: Figure 10 evaluates textual-caption effects, viewpoint variation, and generalization across viewpoints and orientations for common categories.
- Observed effects: Training data size has the most significant impact on performance, with smaller datasets causing clear degradation.Tags and rotational alignment also noticeably influence model performance.
B.2 GENERALIZATION AND LIMITATIONS
Stroke3D generalizes well for frequent categories and mild sketch variation, but performance is constrained by scarce data, caption sensitivity, and ambiguous stroke information.
- Data limitations: Rare or absent concepts can produce unstable or inconsistent skeletons because of limited training-data coverage.Carefully designed prompts can mitigate caption-sensitivity issues.
- Stroke ambiguity: Skeleton quality depends strongly on stroke information: visible joints provide stronger constraints, whereas overlapping joints in side views degrade generation.The limitation is especially pronounced when the model lacks strong priors.
- Generalization: Frequent categories such as humans and common animals support valid skeletons under novel poses, mild sketch noise, and inverted inputs.For example, a vertically flipped fox sketch can yield a logically consistent standing posture.
- Discussion: The primary bottleneck is dataset scale and coverage, motivating larger and more balanced data for rare categories.The authors identify expanded data as the main avenue for future improvement.
C TRAINING OBJECTIVE OF SK-VAE
Sk-VAE represents a 3D skeleton as a graph, encodes it into a latent distribution, samples a latent vector, and reconstructs joint coordinates using the original topology.
- Graph representation: A 3D skeleton is modeled as an undirected graph whose nodes contain 3D joint coordinates and whose edges define topology.
- VAE pipeline: The encoder maps the graph to a latent distribution, samples z, and the decoder reconstructs coordinates X′ conditioned on the original edge set E.
- Training objective: The training objective combines coordinate reconstruction loss with a weighted KL-divergence regularizer.The reconstruction term measures squared distance between original and decoded joint coordinates; the KL term regularizes the latent distribution toward a standard Gaussian.
D MORE RESULTS
Stroke3D maintains strong performance across fine-grained categories with diverse and complex articulated topologies.
- Fine-grained evaluation: Stroke3D outperforms all baselines across CD-J2J, CD-J2B, and CD-B2B on Mythical Creatures, Toys, and Weapons.
- Fine-grained evaluation: 0.029 CD-B2B on Toys surpasses MagicArticulate’s 0.038, demonstrating precise handling of rigid articulated and imaginary-object structures.
E ETHICS AND REPRODUCIBILITY STATEMENTS
Stroke3D aims to make rigged 3D asset creation more accessible while documenting dataset licensing, reproducibility plans, and ethical risks. The authors also identify limited pose variation as a performance constraint and future-work target.
- Stroke3D aims to democratize rigged 3D asset creation for non-professional users, making animation and virtual content creation more accessible.
- TextuRig derives from publicly available Objaverse-XL and UniRig-filtered assets, used consistently with their original licenses and terms of use.
- Stroke3D may be misused to create misleading, biased, or harmful content, and the authors acknowledge that training-data biases may be learned and perpetuated.
- The authors plan to release Stroke3D code, pretrained models, and the curated TextuRig dataset upon publication to support reproducibility.
- Performance is constrained by limited pose variation in the training data; proposed directions include more diverse skeletal poses and an end-to-end text-to-rigged-mesh network.