Source-linked AI summary
Step1X-3D: Towards High-Fidelity and Controllable Generation of Textured 3D Assets
Weiyu Li, Xuanyang Zhang, Zheng Sun, Di Qi, Hao Li, Wei Cheng, Weiwei Cai, Shihao Wu, Jiarui Liu, Zihao Wang, Xiao Chen, Feipeng Tian, Jianxiong Pan, Zeming Li, Gang Yu, Xiangyu Zhang, Daxin Jiang, Ping Tan
TL;DR
3D generation remains less mature than other generative modalities because of data scarcity, representation complexity, and ecosystem fragmentation. Step1X-3D addresses these issues with a curated 2M-asset dataset, a two-stage 3D-native geometry-and-texture framework, and open releases. The framework surpasses existing open-source approaches in asset quality while performing comparably to proprietary state-of-the-art solutions.
Problem
3D generation lags behind other modalities because data scarcity, representation complexity, and an underdeveloped ecosystem constrain progress toward production-ready systems.
Method
Step1X-3D combines rigorous curation of over 5M assets into 2M high-quality assets with a two-stage framework that separately generates TSDF geometry and diffusion-based textures.
Results
Step1X-3D surpasses existing open-source 3D generation approaches in asset quality while achieving performance comparable to proprietary state-of-the-art solutions.
Takeaways & Limitations
The open framework supports controllable 3D generation through image and semantic inputs, LoRA-based adaptation, and released models, code, and training data.
Takeaways & Limitations
The current implementation uses 256^3 TSDF grids and generates only albedo textures, excluding relighting and physically based rendering materials.
Abstract
from arXiv · showhide
While generative artificial intelligence has advanced significantly across text, image, audio, and video domains, 3D generation remains comparatively underdeveloped due to fundamental challenges such as data scarcity, algorithmic limitations, and ecosystem fragmentation. To this end, we present Step1X-3D, an open framework addressing these challenges through: (1) a rigorous data curation pipeline processing >5M assets to create a 2M high-quality dataset with standardized geometric and textural properties; (2) a two-stage 3D-native architecture combining a hybrid VAE-DiT geometry generator with an diffusion-based texture synthesis module; and (3) the full open-source release of models, training code, and adaptation modules. For geometry generation, the hybrid VAE-DiT component produces TSDF representations by employing perceiver-based latent encoding with sharp edge sampling for detail preservation. The diffusion-based texture synthesis module then ensures cross-view consistency through geometric conditioning and latent-space synchronization. Benchmark results demonstrate state-of-the-art performance that exceeds existing open-source methods, while also achieving competitive quality with proprietary solutions. Notably, the framework uniquely bridges the 2D and 3D generation paradigms by supporting direct transfer of 2D control techniques~(e.g., LoRA) to 3D synthesis. By simultaneously advancing data quality, algorithmic fidelity, and reproducibility, Step1X-3D aims to establish new standards for open research in controllable 3D asset generation.
1 Introduction
3D generation lags behind other generative modalities because of data, representation, and ecosystem challenges. Step1X-3D addresses these gaps with curated data, a two-stage native architecture, and open reproducibility.
- 3D generation remains less mature than text, image, audio, and video generation, with progress constrained by dataset, algorithmic, and ecosystem challenges.
- A multi-stage pipeline filters over 5M assets and produces a curated dataset of 2M high-quality assets with standardized geometry and texture properties.Nearly 800K assets derived from public data are planned for open release.
- The two-stage framework decouples geometry and texture synthesis, generating TSDF geometry before diffusion-based texture synthesis conditioned on geometric information.Texture generation uses multi-view conditioning, latent synchronization, and texture completion to preserve cross-view coherence.
- The geometry stage encodes point clouds into compact latent vectors, decodes TSDF representations with a perceiver-based architecture, and preserves details through sharp edge sampling and dual cross attention.
- The framework releases curated data, models, training code, and LoRA-based adaptation modules to improve reproducibility and controllability in open 3D research.
- Comparative experiments report higher asset quality than existing open-source approaches and performance comparable to proprietary state-of-the-art solutions.
2 Related work
Related work spans optimization-based, feed-forward, 2D-lifting, and native 3D generation paradigms. These approaches trade off flexibility, efficiency, representation fidelity, and cross-view consistency in different ways.
- 2.1 Optimization-based 3D Generation: Optimization-based methods iteratively align rendered 3D representations with 2D semantic features, supporting arbitrary text prompts but requiring prolonged optimization and offering limited image-conditioned support.
- 2.2 Feed-forward 3D Generation: Feed-forward methods learn direct mappings from 3D datasets to synthesized assets, avoiding iterative inference optimization and achieving order-of-magnitude acceleration.
- 2.2 Feed-forward 3D Generation: 2D-lifting-to-3D methods transfer knowledge from large-scale 2D datasets through single-stage reconstruction or two-stage multi-view synthesis.
- 2.3 3D Native Generation Paradigm: Native 3D frameworks explicitly model geometry and learn 3D features, but early methods were limited by small datasets and underdeveloped architectures, reducing fidelity and category coverage.
- 2.3 3D Native Generation Paradigm: Texture methods use depth-conditioned diffusion, UV inpainting, or multi-view generation, while independently generated views remain vulnerable to severe consistency issues.
3 Step1X-3D Geometry Generation
Step1X-3D’s geometry-generation pipeline combines curated training data with a latent-set VAE and 1D diffusion transformer. The framework supports diverse conditioning and transfers 2D control mechanisms to 3D synthesis.
- 3.1 Geometry Data Curation: The curation pipeline filters low-quality assets, converts meshes for watertight geometry supervision, and samples surface points with normals.Filtering targets poor textures, incorrect normals, transparent materials, and single-surface geometry; mesh-to-SDF conversion is enhanced with winding-number classification.
- 3.2 Step1X-3D Shape Generation: Sharp Edge Sampling augments uniformly sampled points with samples from geometrically salient regions before VAE encoding.The resulting point set includes corresponding normals, while geometry supervision uses separate volumetric, near-surface, and surface samples.
- 3.2 Step1X-3D Shape Generation: The geometry generator compresses point clouds into a 1D latent tensor and applies a Rectified Flow Transformer adapted from FLUX’s MMDiT architecture.Latent and condition tokens interact through cross-attention, while dual-stream and single-stream blocks support cross-modal feature learning.
- 3.2 Step1X-3D Shape Generation: The Shape VAE encodes point clouds with a latent vector set and decodes them into geometric functions using a scalable transformer-based perceiver architecture.Sharp Edge Sampling and Dual Cross Attention are incorporated to preserve geometric detail.
- 3.2 Step1X-3D Shape Generation: The decoder predicts TSDF values at 3D query points, which are sampled on a regular grid and reconstructed into surfaces with Marching Cubes.The truncated scale is set to 2/256, and Hierarchical Volume Decoding accelerates inference.
- 3.2 Step1X-3D Shape Generation: The architecture enables diverse conditioning and supports transferring 2D control techniques such as ControlNet, IP-Adapter, and LoRA to 3D mesh synthesis.The authors describe this transfer as enabled by the VAE-with-diffusion structure and provide LoRA-based adaptation support.
4 Step1X-3D Texture Generation
Step1X-3D generates textured assets through geometry post-processing, geometry-guided multi-view synthesis, texture-space synchronization, and UV-space completion. The pipeline targets topological integrity, geometric alignment, cross-view coherence, and seamless textures.
- Pipeline overview: The texture pipeline post-processes generated meshes, prepares assets, generates geometry-guided multi-view images, super-resolves them, and performs UV baking.Mesh processing addresses topological consistency and structural integrity before texture generation.
- Geometry preparation: Mesh post-processing verifies watertightness, fills holes in non-manifold geometry, and remeshes triangular faces into four sub-faces with Laplacian smoothing.
- Multi-view generation: Single-view-to-multi-view diffusion generates consistent views conditioned on the input image and target camera poses.The diffusion model is denoted DMV, with multi-view random noise as its starting input.
- Multi-view generation: The multi-view generator uses MV-Adapter with memory-efficient epipolar attention for 768×768 images and parallelized attention for consistency and conditional adherence.
- Geometry guidance: Normal and 3D position maps provide geometric guidance that improves texture detail synthesis and alignment with the mesh surface.
- Texture synchronization: Texture-space synchronization aligns view latents through UV unprojection, weighted texture fusion, and UV rasterization during denoising.The fusion weights use cosine similarities between ray directions and per-pixel normal maps.
- Texture completion: Multi-view images are upsampled to 2048 × 2048, inversely projected into texture space, and inpainted to repair occlusion-related discontinuities and holes.
5 Experiment
Step1X-3D is evaluated through visual demonstrations, controllability experiments, quantitative metrics, user studies, and comparisons with open-source and proprietary methods.
- The evaluation covers single-image geometry and texture generation, controllability, quantitative metrics, user studies, and comparisons with SOTA systems.Compared methods include Trellis, Hunyuan3D 2.0, TripoSG, Tripo-v2.5, Rodin-v1.5, and Meshy-4.
- 5.1 The Visual Quality Results of Step1X-3D Assets: Generated assets span cartoon, sketch, and photorealistic styles, varied geometric complexity, and single- or multi-object configurations.The geometry remains similar to input images while reconstructing plausible occluded spatial structures.
- 5.1 The Visual Quality Results of Step1X-3D Assets: Normal and position maps provide geometric guidance for plausible view completion, multi-view consistency, and geometry-texture alignment.The texture pipeline also targets rich textures and content and style matching with conditioning images.
- 5.2 Controllable 3D Generation: LoRA fine-tuning enables symmetry manipulation and hierarchical geometric-detail control across diverse 3D generation models.The control experiment uses approximately 30,000 annotated 3D models.
- 5.3 Comparison Results with SOTA Methods: 110 in-the-wild images form the benchmark, combining platform examples with Flux-generated images across 80 COCO object categories.The benchmark supports systematic comparison across different generation methods.
- 5.3 Comparison Results with SOTA Methods: Step1X-3D achieves the highest CLIP-Score and multiple second-highest geometric-semantic matching scores.Geometry uses Uni3D-I, OpenShapesc-I, and OpenShapepb-I; texture uses CLIP-Score.
- 5.3 Comparison Results with SOTA Methods: A 20-participant user study finds Step1X-3D comparable to current best-performing methods, while all methods remain below the theoretical upper bound.Participants assessed geometric plausibility, input similarity, texture clarity, and texture-geometry alignment on a 5-point scale.
6 Conclusion
The conclusion presents Step1X-3D as an open-source, high-fidelity framework for controllable 3D generation that targets the gap between proprietary and open research.
- Step1X-3D decouples geometry and texture synthesis, uses 2M curated assets and a hybrid VAE-DiT architecture, and supports 2D-to-3D control transfer.The planned release includes models, training code, and training data excluding self-collected assets.
7 Limitations
The authors identify TSDF resolution and material scope as current limitations of Step1X-3D.
- The current geometry pipeline converts meshes to TSDF at 256^3 resolution, while texture generation is limited to albedo.Future work targets higher geometric resolution, image relighting, and physically based rendering material textures.
8 Contributors
The section lists contributors associated with algorithm development and data contributions.
- Algorithm contributors are Weiyu Li, Xuanyang Zhang, Zheng Sun, Di Qi, Hao Li, Wei Cheng, Weiwei Cai, Zeming Li, Gang Yu, Xiangyu Zhang, and Daxin Jiang.
- Data contributors are Shihao Wu, Jiarui Liu, Zihao Wang, Xiao Chen, Feipeng Tian, and Jianxiong Pan.