Source-linked AI summary
TEXTure: Text-Guided Texturing of 3D Shapes
Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, Daniel Cohen-Or
TL;DR
TEXTure addresses the difficulty of producing seamless, consistent textures on 3D shapes from stochastic image diffusion. It iteratively paints meshes from multiple viewpoints using depth-conditioned diffusion and a trimap-guided sampling process. The method supports generation, transfer, and editing, with reported speed and quality advantages over previous approaches, while fixed viewpoints and depth deviations remain limitations.
Problem
Generating seamless, globally consistent textures on 3D shapes remains challenging because stochastic image diffusion can produce inconsistencies across viewpoints.
Method
TEXTure iteratively paints a mesh from multiple viewpoints using a pretrained depth-conditioned diffusion model and trimap-guided diffusion for keep, refine, and generate regions.
Results
TEXTure is effective for texture generation, transfer, and editing, with reported significant speedups and higher-quality generated textures than previous approaches.
Takeaways & Limitations
The method transfers semantic textures from colored meshes or few unaligned images without explicit mapping or reconstruction and supports text- and scribble-based editing.
Takeaways & Limitations
Fixed viewpoints may not fully cover adversarial geometries, while depth-guided diffusion can deviate from geometry and create conflicting mesh projections.
Abstract
from arXiv · showhide
In this paper, we present TEXTure, a novel method for text-guided generation, editing, and transfer of textures for 3D shapes. Leveraging a pretrained depth-to-image diffusion model, TEXTure applies an iterative scheme that paints a 3D model from different viewpoints. Yet, while depth-to-image models can create plausible textures from a single viewpoint, the stochastic nature of the generation process can cause many inconsistencies when texturing an entire 3D object. To tackle these problems, we dynamically define a trimap partitioning of the rendered image into three progression states, and present a novel elaborated diffusion sampling process that uses this trimap representation to generate seamless textures from different views. We then show that one can transfer the generated texture maps to new 3D geometries without requiring explicit surface-to-surface mapping, as well as extract semantic textures from a set of images without requiring any explicit reconstruction. Finally, we show that TEXTure can be used to not only generate new textures but also edit and refine existing textures using either a text prompt or user-provided scribbles. We demonstrate that our TEXTuring method excels at generating, transferring, and editing textures through extensive evaluation, and further close the gap between 2D image generation and 3D texturing.
1. Introduction
TEXTure directly applies depth-conditioned diffusion while iteratively painting meshes from multiple viewpoints. Its trimap-based sampling addresses inconsistent textures, and the method supports texture generation, transfer, and editing.
- Method: TEXTure directly applies full denoising with a depth-conditioned diffusion model to texture a given 3D mesh.This differs from approaches that indirectly use Stable Diffusion through score distillation.
- Problem: Naïvely painting views independently causes inconsistent textures and noticeable seams because diffusion generation is stochastic.The paper illustrates this failure mode in Figure 2(A).
- Method: A dynamic trimap partitions each rendered view into “keep”, “refine”, and “generate” regions before diffusion sampling.The modified process freezes “keep” regions, improves global consistency in “generate” regions, and repaints “refine” regions using existing texture.
- Transfer: TEXTure transfers textures from colored meshes or a few unaligned images without surface-to-surface mapping or intermediate reconstruction.It learns semantic texture tokens using extensions of Textual Inversion and DreamBooth for depth-conditioned models.
- Editing: TEXTure supports text-only refinement and user scribble edits that modify existing texture maps and fuse edits into the 3D shape.Text prompts alter the entire texture map, while scribbles define regions to refine and leave the remainder fixed.
- Results: Evaluation reports effectiveness across texture generation, transfer, and editing, with faster runtime and higher-quality generated textures than previous approaches.The paper characterizes the speedup and quality improvements as significant.
2. Related Work
Related work spans text-to-image diffusion, 2D and 3D texture transfer, and diffusion-based 3D generation. Prior 3D texturing methods use optimization or score distillation, but are described as slower and less defined than TEXTure.
- Text-to-Image Diffusion Models: Text-to-image diffusion models generate detailed images from text and can condition denoising on depth maps or inpainting masks.Stable Diffusion uses a frozen CLIP text encoder and supports additional input modalities.
- Text-to-Image Diffusion Models: Textual Inversion learns pseudo-tokens from exemplar images, while DreamBooth further fine-tunes a pretrained diffusion model for user-specific concepts.These methods address challenges in editing images and injecting concepts from few exemplars.
- Texture and Content Transfer: 3D texture transfer methods address both color and geometry using statistical sampling, metric learning, internal learning, or geometric-feature relationships.The cited methods include transfer from source shapes or colored meshes to target shapes.
- 3D Shape and Texture Generation: Text2Mesh, Tango, and CLIP-Mesh use CLIP-space similarity optimization to generate 3D shapes and textures.Their parameterizations include UV textures, per-vertex colors, and small geometric displacements.
- 3D Shape and Texture Generation: Latent-Paint and related methods use score distillation to texture or refine 3D representations, but the paper reports slower convergence and less-defined textures than TEXTure.These approaches apply pretrained image diffusion models as optimization guidance.
3. Method
TEXTure incrementally paints meshes from multiple viewpoints using depth-conditioned diffusion and a dynamic keep–refine–generate trimap. It extends this process to texture transfer and prompt- or scribble-guided editing.
- Text-Guided Texture Synthesis: TEXTure incrementally paints visible mesh regions from fixed viewpoints, rendering depth and current colorization before projecting each generated image back to the texture atlas.The process uses a pretrained depth-to-image diffusion model and UV-mapped atlas representation.
- Text-Guided Texture Synthesis: The trimap labels first-seen areas as “generate,” previously painted areas needing a better angle as “refine,” and stable areas as “keep.”A meta-texture map tracks previously viewed regions and their coloring angles.
- Text-Guided Texture Synthesis: Masked diffusion injects noised existing renders into “keep” regions, while an interleaved depth-to-image and inpainting process improves global consistency in “generate” regions.The inpainting model completes generate regions, whereas depth conditioning guides the depth-to-image stages.
- Text-Guided Texture Synthesis: A checkerboard mask constrains early “refine” sampling so newly generated textures locally align with previous completions.The mask is applied for the first 25 sampling steps, with granularity controlled by mask resolution and constrained-step count.
- Texture Transfer: Texture transfer learns a texture token across spectrally augmented geometries and viewpoint tokens, then applies the fine-tuned diffusion model to a new target shape.Low-frequency deformations regularized by the mesh Laplacian spectrum help disentangle texture from source geometry.
- Texture-Editing: Texture editing treats an entire map or selected scribbled areas as “refine” regions while keeping the remaining texture fixed.The same TEXTuring process supports both text-only refinement and user-provided scribble edits.
4. Experiments
Experiments evaluate TEXTure for text-guided generation, texture transfer, and editing, showing detailed, consistent outputs, faster generation, and favorable user judgments.
- Qualitative Results: TEXTure generates highly detailed textures aligned with object geometry across local and global views, including individuals and challenging shapes such as the Klein bottle.Qualitative results include consistent turtle and elephant textures, fine facial details for Einstein, and successful texturing of non-orientable geometry.
- Qualitative Comparisons: Compared with competing methods, TEXTure produces finer details and avoids the globally inconsistent or blurry textures observed in CLIP-guided and score-distillation approaches.The comparisons report sporadic speaker placement in competing methods and missing sharp details for Latent-Paint.
- Qualitative Comparisons: Approximately 5 minutes are required to generate one texture with TEXTure, versus 19 through 45 minutes for alternative methods.The reported runtime comparison accompanies higher-quality visual results.
- User Study: In a 30-respondent user study, TEXTure significantly outperforms Text2Mesh and Latent-Paint on overall quality and text fidelity while reducing runtime by 6.4× and 9.2×, respectively.Ratings are averaged across 10 prompts and corresponding meshes on a 1-to-5 scale.
- Ablation Study: The ablation shows that distinguishing “keep” and “generate” regions and adding “refine” regions improves local and global consistency, robustness, and sharpness.The complete scheme addresses inconsistent patches visible in the naïve painting scheme.
- Texture Transfer and Editing: Learned texture tokens transfer semantic textures from meshes or a few images to new geometries without explicit surface mapping or reconstruction.Image-based transfer can produce semantically similar textures on different shapes, while editing preserves input semantics and adds prompt-matching details or blends scribbled regions.
5. Discussion, Limitations and Conclusions
TEXTure uses iterative depth-to-image diffusion and trimap-guided painting for fast, high-quality 3D texture generation, transfer, and editing, while retaining limitations in global consistency and viewpoint coverage.
- Discussion and Conclusions: TEXTure replaces computationally demanding score distillation with modified image-to-image diffusion applied from a small set of viewpoints.The paper reports a fast process capable of generating high-quality textures in mere minutes.
- Discussion and Conclusions: The method addresses seamless 3D texturing through iterative painting with a pretrained depth-to-image diffusion model and trimap partitioning.The trimap formulation separates “keep”, “refine”, and “generate” regions during painting.
- Limitations: Global inconsistencies may remain because information is occluded across views, including different-looking eyes generated from different viewpoints.The limitation is demonstrated in Figure 10.
- Limitations: Eight fixed viewpoints may not fully cover adversarial geometries, and depth-guided outputs can deviate from input geometry, causing conflicting projections that later iterations cannot fix.The paper suggests dynamically selecting viewpoints to maximize mesh coverage as a possible remedy.
- Discussion and Conclusions: The authors position TEXTure as a practical step toward faster high-quality texture creation for 3D artists, game developers, and modelers.They also present trimap partitioning as a framework for future refinement.