Source-linked AI summary
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images
Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, Sanja Fidler
TL;DR
Existing 3D generative methods often lack detailed geometry, flexible topology, textures, or straightforward compatibility with standard graphics software. GET3D directly generates explicit textured meshes from 2D image supervision, achieving strong results across diverse categories and improving over prior methods, while remaining limited to synthetic-data evaluation and category-specific training.
Problem
Prior 3D generative models often lack geometric detail, flexible topology, textures, or direct usability in standard graphics software, despite the need for scalable 3D content creation.
Method
GET3D combines differentiable surface extraction, differentiable rendering, and 2D GAN techniques to directly generate explicit textured meshes from 2D images.
Results
GET3D generates high-quality textured meshes with arbitrary topology across categories including cars, chairs, animals, motorbikes, human characters, and buildings, with significant improvements over prior methods.
Takeaways & Limitations
The explicit mesh output can be imported into current graphics engines and supports extensions such as text-guided 3D synthesis and decomposed material or view-dependent lighting generation.
Takeaways & Limitations
GET3D relies on 2D silhouettes and known camera distributions, so it was evaluated only on synthetic data; it is also trained separately per category.
Abstract
from arXiv · showhide
As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in terms of the quantity, quality, and diversity of 3D content is becoming evident. In our work, we aim to train performant 3D generative models that synthesize textured meshes which can be directly consumed by 3D rendering engines, thus immediately usable in downstream applications. Prior works on 3D generative modeling either lack geometric details, are limited in the mesh topology they can produce, typically do not support textures, or utilize neural renderers in the synthesis process, which makes their use in common 3D software non-trivial. In this work, we introduce GET3D, a Generative model that directly generates Explicit Textured 3D meshes with complex topology, rich geometric details, and high-fidelity textures. We bridge recent success in the differentiable surface modeling, differentiable rendering as well as 2D Generative Adversarial Networks to train our model from 2D image collections. GET3D is able to generate high-quality 3D textured meshes, ranging from cars, chairs, animals, motorbikes and human characters to buildings, achieving significant improvements over previous methods.
1 Introduction
GET3D addresses the need for scalable 3D content creation by directly generating detailed, textured meshes with arbitrary topology from 2D image supervision. It targets practical use in standard graphics software and demonstrates strong results across diverse shape categories.
- Motivation: Manual 3D asset creation is time-consuming, technically demanding, and difficult to scale for large collections of diverse models.This limits efficient population of games, movies, and other virtual environments.
- Requirements: Practically useful 3D generators should produce detailed geometry, arbitrary topology, textured meshes, and learn from widely available 2D images.These requirements align the output with standard graphics software while avoiding dependence on explicit 3D supervision.
- Prior Limitations: Earlier methods typically satisfy only subsets of these requirements: point clouds lack textures, voxels lack detail and texture, neural fields disregard texture, and template-based meshes restrict topology.Many approaches also require explicit 3D supervision or post-processing to obtain meshes.
- Prior Limitations: Neural-rendering approaches synthesize multi-view-consistent images but do not guarantee meaningful 3D shapes, and extracting textures from their fields is non-trivial.This makes direct use of their outputs in common 3D workflows difficult.
- GET3D: GET3D directly outputs explicit textured 3D meshes with detailed geometry and arbitrary topology, using differentiable surface extraction and rendering for 2D-image training.Its efficient renderer supports training at resolutions as high as 1024 × 1024.
- Results: GET3D demonstrates state-of-the-art unconditional 3D shape generation across complex categories including chairs, motorbikes, cars, human characters, and buildings.The evaluation uses data from ShapeNet, Turbosquid, and Renderpeople.
2 Related Work
Related work explores multiple representations for 3D generation and 3D-aware image synthesis, but these approaches commonly trade off resolution, topology, textures, rendering efficiency, or direct mesh usability. GET3D instead outputs textured meshes ready for standard graphics engines.
- 3D Generative Models: Voxel generators face high memory and computational costs at high resolution, motivating alternatives such as point clouds, implicit fields, and octrees.These representations form major directions in prior 3D content generation research.
- 3D Generative Models: Template-deformation methods generate textured meshes but cannot represent complex topology or shapes with varying genus, whereas PolyGen and SurfGen omit textures.GET3D is positioned to combine arbitrary topology with textured output.
- 3D-Aware Generative Image Synthesis: Neural volume-rendering methods can be slow, produce limited-resolution images, or reduce multi-view consistency when efficiency is improved through low-resolution rendering and upsampling.EG3D partially mitigates the consistency issue with a dual discriminator.
- 3D-Aware Generative Image Synthesis: GET3D directly outputs textured 3D meshes that can be readily used in standard graphics engines, unlike methods based on neural rendering.This avoids the non-trivial extraction of textured surfaces from neural-rendering representations.
3 Method
GET3D separates geometry and texture generation, extracts explicit meshes with differentiable surface modeling, and trains them through differentiable rendering and 2D adversarial supervision.
- Generative model: GET3D uses separate geometry and texture branches, with two latent codes controlling shape generation and surface appearance.The geometry branch outputs a surface mesh, while the texture branch produces a field queried at surface points for colors.
- Geometry generator: DMTet represents geometry as an SDF on a deformable tetrahedral grid and enables differentiable extraction of explicit meshes with arbitrary topology and genus.Predicted SDF values and vertex deformations determine the extracted surface through differentiable marching tetrahedra.
- Geometry generator: Marching tetrahedra extracts faces where neighboring SDF signs differ, with linearly interpolated surface vertices allowing gradients to reach SDF values and deformations.This differentiable construction supports backpropagation from extracted mesh geometry into the generator.
- Texture generator: The texture field maps surface-point coordinates, conditioned on both latent codes, to RGB colors and avoids requiring a topology-specific texture map.Its inputs combine the geometry and texture latent representations so texture can depend on the generated shape.
- Differentiable rendering and training: A differentiable rasterizer renders generated meshes into silhouettes and images, whose surface coordinates are used to query textures before 2D adversarial training.Separate discriminators operate on RGB images and silhouettes, and differentiability backpropagates image-based gradients to the 3D generators.
- Differentiable rendering and training: A cross-entropy regularizer on neighboring SDF values is added to remove internal floating faces that are invisible from any view.The regularization strength is controlled by a hyperparameter.
4 Experiments
GET3D is evaluated on textured-mesh quality, qualitative appearance, latent-space editing, materials, text guidance, and design choices across several datasets and categories. Results show detailed, diverse outputs, strong comparisons with prior methods, and sensitivity to volume subdivision for thin structures and image resolution for FID and shape quality.
- Experimental setup: GET3D is evaluated on ShapeNet and Turbosquid using geometry and texture metrics, with comparisons against supervised 3D generators and 3D-aware image-generation methods.Geometry uses Coverage and Minimum Matching Distance from Chamfer Distance and Light Field Distance; texture uses FID under original-rendering and extracted-mesh protocols.
- Qualitative results: GET3D generates diverse, high-quality geometry and topology, very thin structures, and complex textures across cars, animals, houses, and other categories.The explicit textured-mesh output can be exported into Blender for rendering.
- Latent-space exploration: Interpolating geometry and texture latent codes produces smooth, meaningful transitions, while local latent perturbations generate novel and diverse shapes for editing.The interpolation moves from left to right across both latent codes, and the local perturbation experiment explores nearby latent directions.
- Ablations: Volume subdivision significantly improves performance for classes with thin structures such as motorbikes, but provides no gains on chairs and cars.The authors hypothesize that the initial tetrahedral resolution already captures detailed geometry for chairs and cars.
- Ablations: Higher training image resolution improves FID and shape quality because the network can access details absent from low-resolution images.The result supports the importance of higher-resolution training images for this model.
- Materials and relighting: Unsupervised material generation discovers plausible material differences and supports realistic relighting with complex specular effects under different lighting conditions.The material representation uses base color, metallic, and roughness properties and can be rendered with real-world HDR panoramas.
- Downstream applications: GET3D supports text-guided 3D synthesis by fine-tuning a pretrained model with CLIP, producing textured meshes and comparing against Text2Mesh.Text2Mesh requires an input mesh, so generated meshes from GET3D are supplied as its input geometry.
5 Conclusion
GET3D synthesizes high-quality 3D textured meshes with arbitrary topology and is trained using only 2D images. The authors report improvements over prior methods, while noting limitations involving supervision, camera knowledge, synthetic evaluation, and category-specific training.
- GET3D synthesizes high-quality 3D textured meshes with arbitrary topology using only 2D images as supervision.
- The model achieves significant improvements over previous state-of-the-art methods on generating 3D shapes across multiple categories.
- GET3D relies on 2D silhouettes and known camera distributions during training, so it was evaluated only on synthetic data.
- GET3D is trained per category, and future multi-category training could better represent inter-category diversity.
- The model can generate textured meshes readily imported into current graphics engines, but training-data biases require caution for sensitive applications.
6 Disclosure of Funding
The supplied passages describe GET3D’s architecture and an improved shared-backbone generator, alongside implementation details for geometry, texture, rendering, and training. The improved design is reported to improve geometry–texture disentanglement and unconditional-generation metrics.
- Geometry Generator: The geometry generator predicts SDF values and vertex offsets from a learned feature volume, positional encodings, and conditional fully connected layers.
- Geometry Generator: Volume subdivision divides each tetrahedron into eight smaller tetrahedra to support higher-resolution modeling of thin structures.
- Texture Generator: The texture generator uses a StyleGAN2-inspired tri-plane representation conditioned on concatenated geometry and texture latent codes.
- Improved Generator: The improved generator shares one backbone for geometry and texture, improving information flow and enabling better disentanglement; qualitative and quantitative gains are reported.
- Training: Two StyleGAN discriminators separately process RGB outputs and 2D silhouettes, while camera conditioning supports canonicalized orientations and geometric evaluation.
A.6 Training Procedure and Hyperparameters
GET3D follows StyleGAN2 training practices while training the generator and 2D discriminators from scratch without progressive training or pretrained initialization.
- GET3D uses StyleGAN2-style minibatch standard deviation, exponential moving averages, non-saturating logistic loss, and R1 regularization.
- The model and its 2D discriminators are trained from scratch without progressive training or pretrained checkpoints.
B.1 Datasets
The evaluation uses ShapeNet, TurboSquid, and RenderPeople datasets spanning synthetic objects, diverse textured shapes, and photorealistic humans. Additional analyses examine geometry, retrieval, discriminator stability, and image synthesis against baselines.
- Datasets: ShapeNet provides over 51k shapes across 55 categories, with Car, Chair, and Motorcycle used for evaluation.
- Datasets: The Animal category contains 442 textured shapes, while the House category contains 563 shapes used for qualitative evaluation.
- Datasets: RenderPeople contributes 500 photorealistic human models for qualitative analysis of generated virtual-environment characters.
- Evaluation Metrics: FID-3D renders baseline neural fields through extracted meshes so the metric reflects both geometry and texture; for GET3D, FID-3D and FID-Ori are identical.
- Geometry Comparison: GET3D generates complex geometry, different topology, varying genus, greater detail, and greater diversity than the compared baselines.
- Shape Retrieval: Generated samples are novel relative to retrieved training-set shapes, according to the shape-retrieval analysis.
- Training Stability: Two discriminators greatly reduce training instability and help obtain good results compared with a single discriminator.
- Synthesized Images: GET3D produces 2D image results comparable to EG3D and significantly better than PiGAN and GRAF while directly outputting graphics-engine-compatible meshes.
C.2 Additional Qualitative Results of GET3D
GET3D produces diverse, high-quality textured shapes and supports finer geometric detail. Its two-discriminator design avoids the instability observed with a single discriminator for RGB images and silhouettes.
- Qualitative results: GET3D generates high-quality shapes with diverse textures across chairs, cars, animals, motorbikes, humans, and houses.Additional visualizations include 360° turntable animations for more than 400 shapes.
- Camera conditioning: GET3D with camera conditioning achieves 10.25 compared with 11.63 without camera conditioning.The values are reported for the ShapeNet Car ablation.
- Shape novelty: Generated shapes significantly differ from their closest training-set shapes in both geometry and texture while maintaining quality and diversity.Closest-shape retrieval uses Chamfer distance to identify training-set neighbors.
- Geometric detail: Volume subdivision enables finer motorbike details, including handles and steel wires, that are otherwise difficult to represent.The comparison is performed with and without volume subdivision on ShapeNet motorbikes.
- Discriminator ablation: A single discriminator for RGB images and silhouettes causes significant training instability and can lead to divergence.The authors attribute this to the discriminator overfitting geometry or texture and producing poor gradients for the other branch.
C.3.2 Ablation on Using Camera Condition for Discriminator
Camera-pose conditioning guides GET3D toward canonical orientations, while removing it only slightly degrades the reported FID performance.
- Camera conditioning: Camera-pose conditioning helps GET3D learn shapes in the canonical orientation.This simplifies evaluation with metrics that assume canonicalized input shapes.
- Qualitative robustness: Additional qualitative results trained with predicted silhouettes or noisy cameras are reported as visually similar to the original GET3D results.The predicted-silhouette examples use the Mask-Random setting, while the noisy-camera results are rendered in Blender.
C.4 Robustness to Noisy Cameras
GET3D remains visually capable of generating reasonable textured meshes when trained with noisy cameras, imperfect silhouettes, or GANverse3D-derived data, although FID can decline.
- Noisy cameras: Adding camera noise harms FID, while visual quality shows only little degradation.The authors hypothesize that the FID drop reflects a mismatch between noisy training cameras and the testing camera distribution.
- Predicted silhouettes: GET3D can generate high-quality shapes when trained with imperfect masks, despite observed drops in FID scores.The imperfect masks are predicted with Detectron2 and PointRend to mimic real-world segmentation.
- GANverse3D data: Training on GANverse3D-derived data still yields reasonable 3D textured meshes with variation in geometry and texture.The data combines generated multi-view car images, predicted silhouettes, and camera initialization from structure-from-motion.
- Comparison with EG3D: GET3D achieves comparable 2D-image FID to EG3D but significantly outperforms EG3D on 3D shape synthesis.The comparison uses the Human Body dataset rendered from Renderpeople.
D Material Generation for View-dependent Lighting Effects
GET3D can extend texture generation to material properties for view-dependent lighting. Material generation improves FID under real-world lighting, while simulated lighting performs slightly worse but remains reasonable.
- Material representation: GET3D predicts Disney BRDF properties—base color, roughness, and metallic parameters—instead of only RGB texture values.The texture branch outputs a five-channel reflectance property on surface points.
- Differentiable rendering: The rendering pipeline adds a shading step after rasterization and uses spherical Gaussians to analytically compute outgoing radiance without ray sampling.The rendering equation combines incoming radiance, the BRDF, surface normals, and sampled light directions.
- Lighting data: Real-world HDR panoramas provide diverse outdoor lighting during training, with 724 panoramas collected from three sources.Environment maps are converted into representations using 32 spherical-Gaussian lobes.
- Evaluation: The evaluation measures realism of 2D renderings under real-world lighting using FID.This metric reflects the intended graphics workflow for generated 3D assets.
- Results: 18.53 FID with real-world light improves over the 20.78 texture-only baseline, while 22.83 FID with ambient and directional light is slightly worse.These values are reported in the material-generation ablation.
E Text-Guided 3D Synthesis
The text-guided 3D synthesis method adapts a dual-Generator design with directional CLIP loss, optimizing a trainable generator against a frozen one across rendered camera views. Qualitative results cover multiple object categories, materials, lighting conditions, and geometric-detail comparisons.
- Method: The method randomly samples N = 16 camera views and renders paired images from frozen and trainable generators for each optimization iteration.The paired renderings support computation of the directional CLIP loss.
- Method: Directional CLIP loss aligns the translation of rendered-image embeddings with the translation from a class label to a query text.The implementation uses two pre-trained CLIP models, ViT-32/B and ViT-B/16, for different detail levels.
- Qualitative results: Material properties are visualized for seven generated cars and relit under four different lighting conditions.This evaluates material generation together with relighting behavior.
- Qualitative results: Generated outputs are qualitatively evaluated across Turbosquid animals and houses, ShapeNet motorbikes, cars, and chairs, and Renderpeople.Additional comparisons examine generated geometry and 2D images against baseline methods.
- Geometric detail: With volume subdivision, the model generates finer motorbike geometry, including handles and steel wires.The comparison is performed on ShapeNet motorbikes with and without volume subdivision.