Source-linked AI summary
Text2Tex: Text-driven Texture Synthesis via Diffusion Models
Dave Zhenyu Chen, Yawar Siddiqui, Hsin-Ying Lee, Sergey Tulyakov, Matthias Nießner
TL;DR
Text2Tex addresses the challenge of automatically creating high-fidelity, consistent textures for 3D meshes from text. It progressively inpaints partial textures across viewpoints using depth-aware diffusion, generation masks, and automatic view selection, outperforming existing text-driven and GAN-based approaches in reported experiments.
Problem
Automatic 3D content generation remains limited by the laborious human effort required to design textures, while text-based methods struggle to produce high-fidelity and consistent textures for target meshes.
Method
Text2Tex progressively generates and refines high-resolution partial textures across multiple viewpoints with a pre-trained depth-aware diffusion model, using generation masks and automatic viewpoint selection.
Results
Text2Tex performs favorably against language-based texture synthesis methods on FID, KID, and user study evaluations, and outperforms category-specific GAN-based methods on ShapeNet cars.
Takeaways & Limitations
The method demonstrates effective high-quality 3D texture synthesis from language cues across large-scale 3D content generation experiments.
Takeaways & Limitations
The diffusion backbone tends to produce shading effects in textures, and removing them through prompt engineering requires additional human effort that may not scale to massive generation targets.
Abstract
from arXiv · showhide
We present Text2Tex, a novel method for generating high-quality textures for 3D meshes from the given text prompts. Our method incorporates inpainting into a pre-trained depth-aware image diffusion model to progressively synthesize high resolution partial textures from multiple viewpoints. To avoid accumulating inconsistent and stretched artifacts across views, we dynamically segment the rendered view into a generation mask, which represents the generation status of each visible texel. This partitioned view representation guides the depth-aware inpainting model to generate and update partial textures for the corresponding regions. Furthermore, we propose an automatic view sequence generation scheme to determine the next best view for updating the partial texture. Extensive experiments demonstrate that our method significantly outperforms the existing text-driven approaches and GAN-based methods.
1. Introduction
Text2Tex addresses the difficult problem of automatically producing high-fidelity, consistent 3D textures from language cues. It combines depth-aware diffusion, progressive multi-view synthesis, dynamic view partitioning, and automatic viewpoint selection, outperforming competing approaches in reported evaluations.
- Automating texture design remains challenging because fully automatic 3D content generation still requires laborious human effort.
- Text-guided 3D texturing must satisfy both language fidelity and high, consistent quality across target meshes.
- Text2Tex uses a pre-trained depth-aware text-to-image diffusion model to progressively synthesize and backproject partial textures across multiple viewpoints.
- Its view partitioning computes normal-to-view similarity maps and creates generation masks that tell diffusion regions to generate, update, keep, or ignore.
- The method performs favorably against language-based methods on FID, KID, and user studies, and outperforms category-specific GANs on ShapeNet cars.
- Text2Tex automatically determines the viewpoint order for generating and updating texture space.
- Extensive experiments on many 3D objects demonstrate effectiveness for large-scale 3D content generation.
2. Related work
Prior 3D generation research uses varied representations and increasingly leverages 2D vision-language and diffusion priors. Figure 2 illustrates Text2Tex’s generate-then-refine pipeline for progressively building a mesh texture from multiple views.
- 3D Generation from 3D and 2D data: 3D generative models use representations including voxels, point clouds, meshes, and signed distance functions.
- 3D Generation from 3D and 2D data: Because 3D data is scarce and difficult to collect and annotate, 3D-trained generative models produce limited-quality and less-diverse structure and texture.
- 3D Generation from 3D and 2D data: Differentiable-rendering methods learn texture generation from 2D images but are typically trained for specific shape categories.
- Text-Guided Generation: Text-guided image generation progressed from CLIP-based GAN and VQ-GAN approaches toward diffusion models with superior visual quality and training stability.
- Text-to-3D from 2D data: Figure 2 depicts progressive texture generation: render a viewpoint, generate missing appearance with depth-to-image diffusion, backproject it, and repeat before refinement.
3. Method
Text2Tex progressively synthesizes and refines mesh textures by combining depth-aware diffusion inpainting, view-dependent generation masks, and back-projection across automatically selected viewpoints. The method separates new, update, and keep regions to reduce inconsistent or stretched artifacts during multi-view texture generation.
- Progressive Texture Generation: Text2Tex uses a pretrained depth-aware diffusion model to progressively synthesize and update 3D textures from multiple rendered viewpoints.The generate-then-refine scheme operates across viewpoints and back-projects generated 2D appearances into texture space.
- Depth-Aware Image Inpainting: A generation mask conditions inpainting by blending denoised estimates with noised latent codes in selected image regions.The final denoised latent is decoded into the output image after mask-guided sampling.
- Progressive Texture Generation: Viewpoints are represented by azimuth, elevation, and distance, then used to render depth maps and progressively back-project generated appearances into UV texture space.The initial viewpoint generates visible missing texture, and subsequent viewpoints continue the process.
- Dynamic View Partitioning: The dynamic partition assigns visible regions to new, update, keep, or ignore objectives based on texel visibility and view-dependent similarity.New regions use pure-noise denoising, while update regions refine existing texture and keep regions remain frozen.
- Texture Refinement with Automatic Viewpoint Selection: Texture refinement selects the next viewpoint with the largest normalized update-region area and applies mild denoising before back-projecting the revised appearance.The strategy uses densely defined refinement viewpoints to reduce seams and stretches that remain visible from novel viewpoints.
4. Results
Text2Tex is evaluated on curated Objaverse and ShapeNet car data against text-driven and GAN-based baselines, with quantitative, user-study, qualitative, and ablation analyses. Results favor Text2Tex and show benefits from inpainting, texture updating, and viewpoint refinement, while shading remains a limitation.
- Experiment Setup: 410 curated Objaverse meshes across 225 categories and ShapeNet cars provide the evaluation settings.The Objaverse subset excludes thin, unrecognizable, overly simple, miscategorized, over-triangulated, and scanned objects.
- Quantitative Results: 19% improvement in FID and 26% improvement in KID on Objaverse over previous state-of-the-art text-driven methods.The authors report that these gains indicate more realistic textures across varied object geometries and categories.
- Quantitative Results: 21% improvement in FID and 12% improvement in KID over Texturify on ShapeNet cars.Texturify is identified as the previous state-of-the-art category-specific GAN-based texture synthesis method.
- User Study: Users preferred Text2Tex over CLIPMesh, Text2Mesh, and Latent-Paint at rates of 83.92%, 76.47%, and 64.18%, respectively.The study collected 604 responses from 41 users who judged realism and fidelity to text prompts.
- Qualitative Results: Qualitatively, Text2Tex produces more consistent, cleaner, sharper, and more detailed textures than the compared text-driven and GAN-based methods.Compared baselines exhibit sketchy, repetitive, or blurry textures, whereas Text2Tex better follows the input geometries.
- Ablation Studies and Limitations: Depth-aware inpainting improves consistency, texture updating reduces stretched and blurry artifacts, and more refinement viewpoints reduce seams and remaining flaws.Automatic viewpoint selection further improves refinement, while shading effects from the diffusion backbone remain an acknowledged limitation requiring prompt engineering or future fine-tuning.
5. Conclusion
Text2Tex synthesizes high-quality 3D textures from text by progressively generating and updating partial textures across viewpoints. Its generation masks and automatic viewpoint sequencing support consistent, detailed textures across varied geometries without extra manual effort.
- 5. Conclusion: Text2Tex synthesizes high-quality textures for 3D meshes from text prompts using depth-aware diffusion inpainting across multiple viewpoints.The method progressively generates high-resolution partial textures and backprojects them into texture space.
- 5. Conclusion: Dynamic generation masks guide which texels to generate, update, preserve, or ignore, reducing inconsistent and stretched artifacts across viewpoints.The masks are derived from the rendered view and guide corresponding partial-texture updates.
- 5. Conclusion: Automatic viewpoint sequence generation selects the next best refinement view, and experiments show consistent, highly detailed textures without extra manual effort.The conclusion positions this result across various object geometries.
Supplementary Material
The supplementary material documents the Objaverse subset, user-study details, and additional results and analyses.
- Supplementary Material: The supplement provides Objaverse subset categories, user-study details, and additional results and analyses.These materials are organized in Sections A, B, and C, respectively.
A. Objaverse Subset
The Objaverse evaluation subset is a manually curated collection of 410 high-quality textured objects spanning 225 categories.
- A. Objaverse Subset: The subset samples 3 random meshes from each category before manual curation.Filtering removes thin or unrecognizable meshes, overly simple textures, incorrect category assignments, over-triangulated objects, and scanned objects.
- A. Objaverse Subset: 410 high-quality textured objects across 225 categories remain after curation.The curation is intended to improve input quality and reduce processing time.
- A. Objaverse Subset: The supplementary material lists the categories included in the Objaverse subset.The category list includes objects such as Bible, CD player, Lego, airplane, armchair, and many others.
B. User Study Details
The user study compares Text2Tex with baseline methods through randomized pairs of textured objects rendered from multiple viewpoints, asking users which sample best represents each prompt.
- The study randomly selects 5 pairs of textured objects from each baseline and Text2Tex.
- Each object is rendered from 8 preset viewpoints to improve sample visualization.
- Users choose the sample from each pair that best represents the text prompt.
C. Additional Qualitative Results
Additional results show Text2Tex producing detailed, prompt-responsive textures while preserving object structure across varied colors, complex styles, and unrealistic prompts.
- Additional comparisons: Additional qualitative comparisons on Objaverse objects further showcase Text2Tex results.
- Additional comparisons: Compared with CLIPMesh and Text2Mesh, Text2Tex produces more detailed and realistic textures while preserving input geometry structure.Latent-Paint remains competitive visually but can blend structural details incorrectly, such as marshmallows with a cappuccino plate.
- Stylizing the same objects: Text2Tex generates highly detailed backpack textures that remain faithful to prompted colors.
- Stylizing the same objects: Text2Tex reflects complicated styles such as “baroque” and “cyberpunk” in texture space.The results indicate potential for stylizing high-quality 3D textures beyond simple attributes.
- Creative texture synthesis: With unrealistic prompts, Text2Tex wraps creative appearances onto Porsche geometry while retaining recognizable object properties.For the “hippo” prompt, hippopotamus eyes align with the Porsche lamps because of their semantic similarity.