Source-linked AI summary
TextMesh: Generation of Realistic 3D Meshes From Text Prompts
Christina Tsalicoglou, Fabian Manhardt, Alessio Tonioni, Michael Niemeyer, Federico Tombari
TL;DR
Diffusion-based text-to-3D methods have produced impressive results but often generate impractical NeRFs with oversaturated, cartoonish appearance. TextMesh uses an SDF-based geometry representation and multi-view texture refinement to produce more realistic 3D meshes suitable for standard graphics pipelines.
Problem
Existing diffusion-supervised text-to-3D methods tend to produce NeRFs instead of standard meshes and oversaturated models that are less realistic.
Method
TextMesh models geometry with an SDF for easier mesh extraction and refines mesh textures using color- and depth-conditioned, multi-view diffusion supervision.
Results
TextMesh produces more photorealistic textures, with survey participants preferring the refined textures over the initial ones.
Takeaways & Limitations
The resulting 3D meshes are intended for direct use in standard computer graphics pipelines and applications such as AR and VR.
Abstract
from arXiv · showhide
The ability to generate highly realistic 2D images from mere text prompts has recently made huge progress in terms of speed and quality, thanks to the advent of image diffusion models. Naturally, the question arises if this can be also achieved in the generation of 3D content from such text prompts. To this end, a new line of methods recently emerged trying to harness diffusion models, trained on 2D images, for supervision of 3D model generation using view dependent prompts. While achieving impressive results, these methods, however, have two major drawbacks. First, rather than commonly used 3D meshes, they instead generate neural radiance fields (NeRFs), making them impractical for most real applications. Second, these approaches tend to produce over-saturated models, giving the output a cartoonish looking effect. Therefore, in this work we propose a novel method for generation of highly realistic-looking 3D meshes. To this end, we extend NeRF to employ an SDF backbone, leading to improved 3D mesh extraction. In addition, we propose a novel way to finetune the mesh texture, removing the effect of high saturation and improving the details of the output 3D mesh.
1. Introduction
TextMesh addresses the difficulty of generating realistic, usable 3D content from text by targeting standard meshes rather than NeRFs and reducing oversaturation through SDF-based geometry and texture refinement.
- Motivation: Text-to-3D generation is harder than text-to-image generation because its output space is larger, requires 3D consistency, and lacks abundant text–3D training pairs.
- Limitations of prior work: DreamFusion uses diffusion supervision with view-dependent prompts but produces oversaturated, cartoonish colors and NeRF representations that are impractical for standard graphics pipelines.
- TextMesh: TextMesh modifies DreamFusion to represent radiance with an SDF, enabling surface extraction as the volume’s 0-level set.
- TextMesh: The method retextures mesh renderings using color- and depth-conditioned diffusion, jointly processing multiple views to improve realism, detail, and consistency.
- Contributions: The resulting meshes are presented as geometrically high quality, naturally textured, and ready for existing graphics pipelines and AR/VR applications.
2. Related work
Prior text-to-3D work ranges from CLIP-supervised mesh deformation to diffusion-supervised neural fields. TextMesh responds by combining an SDF representation for mesh extraction with texture refinement for greater photorealism.
- 3D reconstruction: Traditional 3D reconstruction commonly uses depth or voxel representations, while neural fields provide flexible view synthesis but are less directly aligned with standard mesh-based workflows.
- Photorealistic image generation: Text-to-image diffusion models provide high-fidelity, flexible synthesis because they can condition generation on text and seed images.
- 3D generation from text: Earlier text-to-3D methods use CLIP objectives to deform meshes or spheres, whereas newer methods use diffusion models and SDS to optimize NeRFs from text.
- Limitations of prior work: DreamFusion’s volumetric output complicates downstream graphics use, while high guidance weights tend to produce oversaturated rather than photorealistic textures.
- TextMesh: TextMesh replaces the radiance-field representation with an SDF and adds texture refinement to improve mesh extraction and photorealism.
3. Method
TextMesh first learns an SDF-based neural field from text using SDS, extracts a mesh, and then refines its texture with jointly diffused multi-view RGB-D renderings. A final optimization stage uses photometric supervision and a small SDS term to improve details, reduce saturation, and smooth view transitions.
- 3.3. Photorealistic Texturing Using Multi-View Consistent Diffusion: Jointly processing the four tiled views produces multi-view-consistent diffusion outputs that become pseudo-ground-truth views for texture optimization.Processing views independently can introduce inconsistencies, whereas joint processing enforces consistency during diffusion.
- 3.1. Initial Scene Representation: An SDF replaces a density-based NeRF representation because its zero-level set supports direct mesh extraction with Marching Cubes.The method retains volume rendering by transforming signed distance values into density values.
- 3.2. Text-to-3D via Score-based Distillation: The method trains an initial neural distance field with volume rendering and SDS, using noisy rendered images, diffusion noise prediction, and text conditioning to optimize the field.The SDS gradient is computed at 64×64 resolution with Imagen, and the full elevation range is sampled to avoid bottom bleeding artifacts.
- 3.3. Photorealistic Texturing Using Multi-View Consistent Diffusion: The extracted mesh has low-frequency, oversaturated texture, so the method freezes geometry and refines color using depth-conditioned Stable Diffusion on four canonical RGB-D views.The four views are front, back, and both sides, and are tiled into a 2 × 2 grid before diffusion.
- 3.3. Photorealistic Texturing Using Multi-View Consistent Diffusion: A second texture optimization combines photometric loss with a small SDS component to correct minor view misalignments and complete smooth transitions across the mesh.The texture remains anchored to the original through pseudo-ground-truth renderings, while a small guidance weight limits changes that could produce saturated colors.
4. Evaluation
TextMesh is evaluated against prior text-to-3D methods using quantitative metrics, qualitative comparisons, ablations, user preferences, and mesh-quality analyses. The results indicate improved photorealism, smoother geometry, and the importance of texture finetuning and multi-view processing, while standard metrics do not fully capture 3D consistency.
- Experimental Setup: The evaluation uses CLIP R-Precision and FIDCLIP on 35 prompts, with images rendered from 60 azimuthal angles at 30-degree elevation.R-Precision measures text-image correlation, whereas FIDCLIP evaluates feature-space distribution similarity using CLIP ViT-B-32 features.
- Comparison with state-of-the-art: TextMesh performs on par with DreamFusion, with somewhat worse R-Precision but better FIDCLIP, while CLIP-Mesh obtains the best R-Precision despite qualitatively worse results.The authors attribute CLIP-Mesh’s quantitative advantage to directly optimizing the CLIP metric during training.
- Comparison with state-of-the-art: Qualitative comparisons show more photorealistic TextMesh outputs than CLIP-Mesh, Magic3D, and DreamFusion, whose results tend to appear cartoonish and oversaturated.The comparison includes both RGB renderings and 3D shape or volume views.
- Ablation Study: The multi-view component is essential for realistic results, as SDS-only optimization performs worst overall.The default setting uses depth conditioning and four tiled mesh views processed jointly by Stable Diffusion.
- User Study: A user study with 30 participants found preference for re-textured meshes, particularly for texture details and color.Participants compared meshes before and after photorealistic texturing.
- Mesh Quality: The SDF-based approach produces smoother mesh predictions and, with full elevation-range camera sampling, avoids spurious surfaces found in DreamFusion outputs.The authors identify high-quality mesh extraction as important for downstream applications such as AR.
- Mesh Quality: Multiple-view inspection indicates that the optimized meshes remain 3D consistent and retain realistic texture from arbitrary viewpoints.Figure 6 provides a qualitative evaluation of consistency across views.
5. Conclusion
TextMesh represents geometry as a distance field, extracts a mesh, and refines its texture for a more detailed and natural appearance. Its photometric, depth-conditioned refinement produces more photorealistic textures, preferred by more survey participants than the initial texture.
- TextMesh optimizes geometry as a distance field, extracts the mesh, and refines its original texture for a more detailed and natural appearance.
- Texture refinement is supervised primarily with photometric loss on enhanced 2D mesh renderings from a depth-conditioned image-to-image diffusion model.
- Score Distillation Sampling is used only to smooth transitions within the multi-view texture supervision.
- The refined textures are more photorealistic and were preferred by a larger portion of survey participants than the initial, unrefined texture.