Source-linked AI summary
Seed3D 1.0: From Images to High-Fidelity Simulation-Ready 3D Assets
Jiashi Feng, Xiu Li, Jing Lin, Jiahang Liu, Gaohong Liu, Weiqiang Lou, Su Ma, Guang Shi, Qinlong Wang, Jun Wang, Zhongcong Xu, Xuanyu Yi, Zihao Yu, Jianfeng Zhang, Yifan Zhu, Rui Chen, Jinxin Chi, Zixian Du, Li Han, Lixin Huang, Kaihua Jiang, Yuhan Li, Guan Luo, Shuguang Wang, Qianyi Wu, Fan Yang, Junyang Zhang, Xuanmeng Zhang
TL;DR
Seed3D 1.0 addresses the need for scalable, high-fidelity training environments that combine diverse content with physics accuracy. It generates simulation-ready 3D assets from single images using integrated geometry, multi-view, material, and texture components, then supports physics-engine integration and scene composition. The paper reports state-of-the-art benchmark performance, including superior results from its 1.5B geometry model over larger baselines, while noting suboptimal performance on in-the-wild images.
Problem
Embodied AI lacks scalable high-fidelity simulation environments that provide rich spatial-physical information and balance content diversity with rigorous physics.
Method
Seed3D 1.0 generates simulation-ready 3D assets from single images through integrated geometry, multi-view synthesis, material decomposition, texture completion, and scalable data infrastructure.
Results
Seed3D 1.0 achieves state-of-the-art geometry and texture benchmark performance, with its 1.5B geometry model outperforming larger baseline methods.
Takeaways & Limitations
The system provides scalable simulation-ready content for physics-based world simulators, robotic manipulation, and interactive reinforcement-learning environments.
Takeaways & Limitations
The multi-view approach can produce suboptimal results on in-the-wild images because its DiT architecture was not originally designed for multi-view generation.
Abstract
from arXiv · showhide
Developing embodied AI agents requires scalable training environments that balance content diversity with physics accuracy. World simulators provide such environments but face distinct limitations: video-based methods generate diverse content but lack real-time physics feedback for interactive learning, while physics-based engines provide accurate dynamics but face scalability limitations from costly manual asset creation. We present Seed3D 1.0, a foundation model that generates simulation-ready 3D assets from single images, addressing the scalability challenge while maintaining physics rigor. Unlike existing 3D generation models, our system produces assets with accurate geometry, well-aligned textures, and realistic physically-based materials. These assets can be directly integrated into physics engines with minimal configuration, enabling deployment in robotic manipulation and simulation training. Beyond individual objects, the system scales to complete scene generation through assembling objects into coherent environments. By enabling scalable simulation-ready content creation, Seed3D 1.0 provides a foundation for advancing physics-based world simulators. Seed3D 1.0 is now available on https://console.volcengine.com/ark/region:ark+cn-beijing/experience/vision?modelId=doubao-seed3d-1-0-250928&tab=Gen3D
1 Introduction
Seed3D 1.0 addresses the scarcity of high-fidelity embodied-AI training environments by generating simulation-ready 3D assets and composing them into scalable scenes. It combines detailed geometry, photorealistic textures, physically plausible materials, and physics-engine compatibility to bridge content diversity with simulation fidelity.
- 1 Introduction: Seed3D 1.0 targets the trade-off between diverse video-generated content and scalable physics-based simulation by generating simulation-ready 3D assets.Video-based approaches lack 3D consistency and intermediate feedback, whereas physics-based simulators are constrained by costly manual asset creation.
- 1 Introduction: Detailed geometry, photorealistic textures up to 4K resolution, and physically plausible PBR materials produce assets suitable for rendering and physical simulation.The model is designed to avoid geometric artifacts and texture misalignment while preserving consistent assets.
- 1 Introduction: Assets integrate into physics engines with minimal configuration, supporting simulation-based data generation and interactive reinforcement-learning environments.The assets create diverse robotic-manipulation scenarios while retaining environmental feedback for skill acquisition.
- 1 Introduction: A factorized scene-generation approach uses vision-language models for spatial planning and Seed3D generation for asset creation and placement across indoor and urban environments.This extends the system from individual objects to coherent scene composition.
- 1 Introduction: The geometry pipeline combines Seed3D-VAE for compact geometry encoding and TSDF decoding with Seed3D-DiT for high-fidelity 3D shape generation from images.Figure 2 identifies the two main components of the geometry-generation framework.
2 Model Design
Seed3D 1.0 combines latent geometry generation with multimodal texture synthesis to produce high-fidelity, simulation-ready 3D assets. Its pipeline includes image-conditioned shape generation, multiview material estimation, and UV texture completion.
- 2.1 Geometry Generation: The geometry pipeline targets watertight, manifold shapes that preserve structural details for reliable physics simulation.
- 2.1 Geometry Generation: Seed3D-VAE encodes 3D geometry into compact latent representations, while Seed3D-DiT synthesizes diverse shapes conditioned on reference images.The VAE uses a point-based encoder and continuous TSDF decoder; the DiT operates in the learned latent space.
- 2.1 Geometry Generation: Seed3D-DiT combines DINOv2 and RADIO image features with double-stream and single-stream transformer blocks for cross-modal shape generation.Flow matching uses length-aware timestep shifts for longer latent sequences and deterministic inference sampling.
- 2.2.1 Seed3D-MV: Seed3D-MV generates consistent multiview RGB images from reference images and geometry, using in-context conditioning over geometry, image, and text tokens.The model uses specialized positional encoding and shifted timestep sampling to accommodate multiview sequences.
- 2.2.2 Seed3D-PBR: Seed3D-PBR decomposes multiview RGB outputs into consistent albedo, metallic, and roughness maps with a parameter-efficient two-stream DiT design.The model jointly predicts albedo and metallic-roughness maps while maintaining cross-view consistency.
- 2.2.3 Seed3D-UV: Seed3D-UV uses coordinate-conditioned diffusion to complete UV textures missing regions caused by limited view coverage and self-occlusion.
3 Data
Seed3D 1.0 builds an automated data pipeline and distributed infrastructure for converting heterogeneous raw 3D assets into consistent training datasets. Processing includes standardization, deduplication, orientation canonization, quality filtering, rendering, remeshing, and scalable asset management.
- 3.1 Data Preprocessing: The preprocessing pipeline transforms heterogeneous raw 3D collections into high-quality, diverse, and consistent training-ready datasets.Its stages address data complexity, heterogeneity, duplication, orientation, quality, rendering, and mesh validity.
- 3.1 Data Preprocessing: Assets are standardized into unified GLB mesh representations, while rendered features support geometric deduplication and canonical orientation prediction.Deduplication uses four canonical viewpoints with RGB images and normal maps; orientation transformations are then applied automatically.
- 3.1 Data Preprocessing: Quality filtering combines aesthetic scoring with VLM assessment of usability, category, and data type before retaining acceptable assets.
- 3.1 Data Preprocessing: Multi-view rendering uses Blender Cycles with varied lighting, viewpoints, and materials to generate RGB images, normal maps, and camera coordinate maps.The pipeline samples viewpoints and stochastic illumination for geometry generation and PBR estimation.
- 3.1 Data Preprocessing: CUDA-based remeshing converts arbitrary meshes into watertight representations while preserving external surface details for valid SDF extraction.
- 3.2 Data Engineering Infrastructure: The infrastructure combines metadata indexing, scalable object storage, web-based curation, and Ray Data distributed processing with heterogeneous CPU/GPU scheduling.Rendering is CPU-intensive, whereas mesh remeshing requires GPU acceleration.
4 Model Training
Seed3D training uses progressive stages for geometry and two-stage training for texture models. The schedules move from broad foundational learning toward higher-resolution or curated high-quality data to improve output quality while preserving generalization.
- 3 Model Training: Seed3D-DiT training progresses from pre-training through continued training to supervised fine-tuning.This strategy progressively improves model capacity and output quality.
- 3 Model Training: Pre-training learns foundational shape generation and image–geometry alignment from scratch using low-resolution representations with 256 latent tokens.
- 3 Model Training: Continued training increases latent sequence length to 4096 tokens to capture finer geometric details and surface structures.Enhanced augmentation is used to maintain generalization at higher resolutions.
- 3 Model Training: Supervised fine-tuning on a curated high-quality subset further improves geometric accuracy and surface detail using reduced learning rates.
- 3 Model Training: Texture models train first on the full dataset and then fine-tune on curated high-quality data to improve quality while maintaining generalization across textures and materials.
5 Training Infrastructure
Seed3D 1.0 uses hardware-aware optimization, memory-efficient parallelism, activation checkpointing, and fault-tolerance mechanisms for stable, high-throughput large-scale training.
- The training infrastructure combines hardware-aware optimization, memory-efficient parallelism, and fault tolerance for stable, high-throughput large-scale training.
- Fused CUDA kernels, FlashAttention, and Apex optimizers reduce memory-access overhead and improve arithmetic intensity for performance-critical operations.
- Hybrid Sharded Data Parallelism combines within-node data parallelism with cross-node Fully Sharded Data Parallelism to reduce communication overhead while sharding model states.
- Multi-Level Activation Checkpointing selectively checkpoints or offloads activations with asynchronous prefetching, reducing memory use with minimal performance impact.
- The stability framework uses proactive machine health checks and reactive recovery mechanisms to handle hardware failures and communication disruptions.
6 Inference
Seed3D 1.0 converts a single image into a simulation-ready textured asset through sequential geometry, multi-view, material, texture, and integration stages.
- The inference pipeline sequentially performs geometry generation, multi-view synthesis, PBR material estimation, UV texture completion, and final asset integration.
- Seed3D-DiT predicts 3D shape in latent space, and the VAE decoder reconstructs the mesh using dual marching cubes with accelerated hierarchical extraction.
- Seed3D-MV generates multi-view-consistent RGB images that are back-projected onto the mesh and baked into UV space, producing partial textures.
- Seed3D-PBR decomposes multi-view images into albedo and metallic-roughness components, which are baked into UV space as physically based material properties.
- Seed3D-UV inpaints incomplete albedo and MR UV maps using coordinate conditioning to maintain geometric consistency.
- Final integration combines completed albedo, metallic, and roughness maps with watertight manifold geometry and exports assets in OBJ or GLB formats.
7 Model Performance
Seed3D 1.0 achieves strong geometry, multi-view, material, texture, and user-evaluation results against established baselines, with UV completion improving incomplete textures.
- 7.1.1 Geometry Generation: Seed3D 1.0 achieves the highest scores across all geometry metrics and outperforms the larger 3B Hunyuan3D-2.1 model.
- 7.1.1 Geometry Generation: Seed3D 1.0 preserves fine geometric details, structural accuracy, and overall shape fidelity better than baseline methods in qualitative comparisons.
- 7.1.2 Texture Generation: Seed3D-MV and Seed3D-PBR achieve state-of-the-art or best performance across their multi-view and material-generation metrics.
- 7.1.2 Texture Generation: Ground-truth multi-view inputs substantially improve Seed3D 1.0∗ performance by removing errors from multi-view generation.
- 7.1.2 Texture Generation: Seed3D 1.0 improves fine-detail preservation, text clarity, material quality, and realistic PBR surface properties compared with baseline methods.
- 7.1.3 User Study and Ablation: In a study of 14 evaluators and 43 images, Seed3D 1.0 receives consistently higher ratings across six quality dimensions, especially geometry and material quality.
8 Application
Seed3D 1.0 supports robotic simulation and coherent scene generation by integrating generated assets into Isaac Sim and arranging individually generated objects using predicted layouts.
- 8.1 Simulation-ready Generation: Seed3D 1.0 assets integrate into NVIDIA Isaac Sim for robotic manipulation experiments involving grasping and multi-object interactions.
- 8.1 Simulation-ready Generation: Physics simulation provides real-time feedback on contact forces, object dynamics, and manipulation outcomes while preserving fine geometric details for contact simulation.
- 8.1 Simulation-ready Generation: These environments support scalable manipulation-data generation, interactive learning from physics feedback, and multi-view, multi-modal evaluation data for VLA models.
- 8.2 Scene-level Generation: For scene generation, a VLM infers object identities and spatial relationships, producing layout maps with object scales, positions, and orientations.
- 8.2 Scene-level Generation: The system generates each object separately and assembles the objects according to the predicted layout to create coherent indoor and urban scenes.
9 Conclusion
Seed3D 1.0 generates simulation-ready 3D assets from single images using integrated geometry, view synthesis, material, and texture components. The system reports strong benchmark and user-study results, while supporting direct physics-engine deployment and robotic manipulation simulation.
- Seed3D 1.0 generates simulation-ready 3D assets from single images with detailed geometry, photorealistic textures, and physically based materials.Its assets maintain watertight, manifold geometry for direct physics-engine deployment.
- The model combines Seed3D-DiT, Seed3D-MV, Seed3D-PBR, and Seed3D-UV for geometry, multi-view synthesis, material decomposition, and texture completion.
- Experimental evaluations report state-of-the-art geometry and texture benchmark performance, including superior results from the 1.5B-parameter geometry model against larger baselines.User studies also assess visual clarity, geometric accuracy, and material realism.
- Generated assets integrate into environments such as Isaac Sim and support scalable robotic manipulation training-data generation and evaluation.
A Contributions and Acknowledgments
The acknowledgments state that all Seed3D contributors are listed alphabetically by last name.
- All Seed3D contributors are listed in alphabetical order by their last names.
- The contributor list uses last names as the ordering key.
- The alphabetical ordering applies to all contributors listed for Seed3D.
A.1 Core Contributors
The core-contributor list names the researchers credited for Seed3D. The names are presented across two consecutive groups.
- The first contributor group includes Jiashi Feng, Xiu Li, Jing Lin, Jiahang Liu, and Jiahang Liu's listed colleagues through Jianfeng Zhang, Yifan Zhu.
- The second contributor group includes Rui Chen, Jinxin Chi, Zixian Du, Li Han, and the remaining listed contributors through Xuanmeng Zhang.