Source-linked AI summary
TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models
Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, Kangxue Yin
TL;DR
Texture synthesis for 3D assets has received less attention, while SDS-based use of image diffusion relies on lengthy optimization and high guidance weights. TexFusion samples textures consistently across rendered views, producing diverse, globally coherent results and state-of-the-art text-driven texture synthesis performance.
Problem
3D textures have been studied less than geometry, and SDS-based methods require high guidance weights and lengthy optimization.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
Text conditioning provides detailed control, and the method is not limited to single categories or dependent on explicit 3D textures as training data.
Source: §1 · p. 2TexFusion is methodologically distinct from concurrent TEXTure-based methods, whose proposed improvements could be combined with it in future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We present TexFusion (Texture Diffusion), a new method to synthesize textures for given 3D geometries, using large-scale text-guided image diffusion models. In contrast to recent works that leverage 2D text-to-image diffusion models to distill 3D objects using a slow and fragile optimization process, TexFusion introduces a new 3D-consistent generation technique specifically designed for texture synthesis that employs regular diffusion model sampling on different 2D rendered views. Specifically, we leverage latent diffusion models, apply the diffusion model's denoiser on a set of 2D renders of the 3D object, and aggregate the different denoising predictions on a shared latent texture map. Final output RGB textures are produced by optimizing an intermediate neural color field on the decodings of 2D renders of the latent texture. We thoroughly validate TexFusion and show that we can efficiently generate diverse, high quality and globally coherent textures. We achieve state-of-the-art text-guided texture synthesis performance using only image diffusion models, while avoiding the pitfalls of previous distillation-based methods. The text-conditioning offers detailed control and we also do not rely on any ground truth 3D textures for training. This makes our method versatile and applicable to a broad range of geometry and texture types. We hope that TexFusion will advance AI-based texturing of 3D assets for applications in virtual reality, game design, simulation, and more.
1. Introduction
Textures are important to vivid 3D assets but have received less attention than geometry, while text-conditioned image diffusion offers detailed and diverse image priors. TexFusion uses those priors for mesh texture synthesis through multiview latent diffusion and reports coherent, high-quality results.
1. Introduction
- Textures remain less studied than geometry despite their role in assigning colors and materials that make 3D asset renderings vivid.Text-conditioned image diffusion models offer detailed and diverse image priors for texture synthesis.Source: §1 · p. 1
- SDS-based methods require high classifier-free guidance that can cause oversaturation and low diversity, as well as lengthy optimization for each sample.The paper frames these as common limitations of prior methods using image diffusion to synthesize 3D objects.Source: §1 · p. 2
- TexFusion samples latent diffusion trajectories across object views, aggregates each denoising step on a shared latent texture map, then optimizes a neural color field for RGB output.The method uses Stable Diffusion with depth conditioning as its diffusion backbone.Source: §1 · p. 2
- TexFusion generates high-quality, globally coherent, prompt-aligned textures across varied tasks and achieves state-of-the-art text-driven texture synthesis performance.The authors report diverse outputs without restricting generation to single categories or requiring explicit 3D textures as training data.Source: §1 · p. 2
2. Related Work
Prior texture work spans pattern tiling, 3D-prior learning, and diffusion-based 3D generation, but these approaches leave gaps in global coherence, text conditioning, or texture-specific synthesis. TexFusion is distinguished by interleaving multiview denoising and texture aggregation.
Classic Computer Graphics Techniques
- Classic texture synthesis tiled exemplar patterns, sometimes with direction fields, establishing texture representations while exposing the challenge of global coherence.The paper notes that learning-based priors are needed to move beyond simple patterns toward plausible shape-specific details.Source: §2 · p. 2
Texture Synthesis with 3D Priors
Diffusion Models
- Diffusion-based 3D generation has targeted point clouds, meshes, and neural fields, but not specifically 3D texture synthesis.These approaches focus on geometry generation rather than texture synthesis.Source: §2 · p. 2
Distilling 3D Objects from 2D Image Diffusion Models
Concurrent Work
- TexFusion interleaves denoising across camera views with texture-map aggregation, unlike TEXTure [62], which completes each view’s denoising before projecting it.The authors report that this design reduces view inconsistencies and improves quality; MultiDiffusion [3] similarly aggregates denoising predictions but addresses image synthesis rather than 3D texturing.Source: §2 · p. 3
3. Background
The background describes diffusion as iterative denoising, then introduces latent diffusion and the paper’s rendering and texture representation. These pieces establish how image diffusion can operate on rendered views of a textured mesh.
3. Background
- Diffusion models iteratively denoise data; their denoisers learn to predict perturbation noise, and generation starts from Gaussian noise.The paper describes constructing noisy inputs from data and sampled noise under a noise schedule.Source: §3 · p. 3
- Classifier-free guidance combines conditional and unconditional predictions for stronger conditioning, and TexFusion uses DDIM for iterative sampling.Conditioning can be randomly dropped during training to learn both predictions.Source: §3 · p. 3
Latent Diffusion Models (LDMs) and Stable Diffusion
- Latent diffusion models encode images into lower-spatial-dimension latents, perform diffusion there, and decode latents back into images; Stable Diffusion 2.0 also supports depth conditioning.The paper uses Stable Diffusion with depth conditioning to control object shape and configuration in synthesized images.Source: Latent Diffusion Models (LDMs) and… · p. 4
- Textures are represented as multichannel images in a mesh’s UV parameterization, while a renderer maps them to camera images and an inverse projection maps image values back to UV space.The described rendering setup directly projects textures into screen space without modeling lighting or shading.Source: Latent Diffusion Models (LDMs) and… · p. 4
4. Texture Sampling with 2D Diffusion Models
TexFusion addresses inconsistent texture generation across 2D views by synchronizing diffusion through a shared latent texture map. Its pipeline aggregates view predictions, converts them into RGB textures, and uses camera and resolution choices to preserve semantic consistency and detail.
4. Texture Sampling with 2D Diffusion Models
- Independent view sampling lacks cross-view coherence, while sequential inpainting can leave early-view errors that later views cannot reconcile with the geometry.The desired sampling distribution for each camera view is conditioned on the others.Source: §4 · p. 4, Figure 3 · p. 4
4.1. Sequential Interlaced Multiview Sampler
- SIMS interlaces synchronization across views with denoising, producing 3D-consistent images while avoiding the degradation caused by averaging latent predictions.It communicates earlier views’ denoising direction to later views; in the single-view case, it is equivalent to DDIM sampling.Source: §4.1 · pp. 4–5
4.2. The TexFusion Algorithm
- TexFusion applies SIMS to latent renders from SD2-depth, then uses a post-process to produce color-space textures.Its camera set can be tailored to improve resolution and ameliorate occlusion.Source: §4.2 · p. 5
4.2.1 (Inverse) Rendering of Latent Textures
- Latent textures are rendered by nearest-pixel lookup without shading; interpolation and mipmapping are avoided because they alter early-step noise statistics.Depth maps condition the diffusion model, and Gaussian noise is added to background pixels to match the current diffusion-step distribution.Source: §4.2.1 · p. 5
4.2.2 Aggregating Latent Textures.
- Aggregation selects texture values from the camera with the best view quality, overwriting lower-quality regions when better views become available.View quality is measured from screen-space derivatives; the resolution choice also helps prevent block-like aliasing in high-quality views.Source: §4.2.2 · p. 6
4.2.3 From Latent to RGB Textures
- A neural color field maps mesh-surface coordinates to RGB and is optimized against decoded multiview images to smooth latent-decoding inconsistencies.The field uses a hash encoding and shallow MLP, with L2 and VGG perceptual losses and no view-dependent parameterization.Source: §4.2.3 · p. 6
4.2.4 Geometry Processing, Camera, and Multiresolution Refinement
- Small camera views can cause semantic content drift, so TexFusion first generates a low-resolution map with object-covering cameras, then refines it with more numerous, narrower views.The first-round latent map is upsampled and partially noised to initialize the second round, whose views feed the final color-field distillation.Source: §4.2.4 · pp. 6–7
5. Experiments
TexFusion was evaluated against TEXTure across 86 mesh-text pairs, with qualitative, FID, and user-study comparisons; additional variants explore detail gains and their robustness costs.
5.1. Qualitative Comparisons
- TexFusion shows no visible seam lines or stitching artifacts, while TEXTure often has severe top- and back-side artifacts and weaker global coherence.The authors say frequent view communication in SIMS ameliorates the global-consistency problem.Source: §5.1 · p. 7
- TexFusion produces smoothly illuminated textures and text prompts control varying appearances on the same mesh; TEXTure more often bakes in specular highlights and shadows.The authors hypothesize that interlacing aggregations in SIMS remove view-dependent effects during sampling.Source: §5.1 · p. 7
- TexFusion samples one texture in approximately 3 minutes on a single GPU.The authors report it is slightly faster than TEXTure at 5 minutes and an order of magnitude faster than SDS methods reporting upwards of 40 minutes.Source: §5.1 · pp. 7–8
5.2. Quantitative Comparisons
- FID scores indicate TexFusion textures are closer in appearance to the SD2-depth-generated comparison set than TEXTure’s.The authors note that absolute FID values are likely high because each image set is relatively small.Source: Table 1 · p. 8, §5.2 · p. 8
- Users overall preferred TexFusion for prompt alignment, natural color, and fewer artifacts, although they judged TEXTure as more detailed.Source: §5.2 · pp. 8–9
5.3. Improving Texture Details
- Non-stochastic DDIM and ControlNet can add material or geometric detail, but reduce robustness; the DDIM setting may artifact on clean textures, while ControlNet may capture low-poly face boundaries.ControlNet’s “guess mode” yields high-contrast textures with strong-lighting appearance, whereas “normal mode” yields realistic textures with smooth lighting.Source: §5.3 · p. 9
6. Conclusion and Limitations
TexFusion uses multiview latent-diffusion sampling with interleaved texture aggregation to synthesize 3D textures. The authors report diverse, high-quality, globally coherent outputs, while noting limits in sharpness and generation speed.
6. Conclusion and Limitations
- TexFusion runs a 2D denoiser on different object views and aggregates predictions in a latent texture map after each denoising iteration.Source: §6 · p. 9
- The authors report efficient generation of diverse, high-quality, globally coherent textures with detailed text control, but sharpness is not yet ideal and generation is not real-time.Source: §6 · p. 9
- Concurrent text-to-texture improvements based on TEXTure include refinement and per-prompt finetuning, and the authors say these could be combined with TexFusion in future work.Source: §6 · p. 9
7. Algorithm Details
The appendix specifies SIMS sampling choices, including the DDIM schedule, guidance, camera coverage, and object alignment.
Adapting DDIM schedule
- DDIM uses 50 uniformly spaced steps truncated to (300, 1000), with final-step sequential aggregation at 300 and x0 predictions as outputs.The schedule and final-step outputs are specified for the sampler.Source: §7 · p. 14
- The temperature τ scales DDIM noise; τ < 1 lowers posterior variance without changing its expectation, with τ=0.5 for coarse sampling and τ=0 for refinement.Both stages use η=1 as the most robust configuration reported.Source: §7 · p. 14
Classifier-free guidance
- Classifier-free guidance controls alignment to depth and text; the joint setting is w_joint=5, and human heads and bodies receive an additional text-only term for stylization.The formula differs from SD2/w-depth, which applies classifier-free guidance only to text while including depth in both terms.Source: §7 · p. 14
Additional geometry processing
- Objects with a meaningful front are aligned toward +x, with +y as up.Source: §7 · p. 14
Camera configuration
- Most objects use nine cameras; the first sampling round jitters them into 18 views, while human characters use three sets of eight cameras to cover occluded surfaces.The human-camera sets differ in their y-position offsets.Source: §7 · pp. 14–15
8. Additional Results
The additional results report runtime comparisons and qualitative baseline findings, alongside supplementary multiview examples.
8.1. Qualitative Results
- The supplementary video provides multiview renderings of the main-paper examples for further inspection.Source: §8.1 · p. 15
8.2. Additional Baselines
- Stable DreamFusion failed to converge for most prompts; Latent-Painter [48] often showed oversaturation and UV-boundary artifacts, which TexFusion avoids.The Stable DreamFusion visuals were selected from prompts where it produced reasonable geometry; TexFusion decodes multiview latent images and is agnostic to UV parameterization.Source: §8.2 · p. 15, Figure 11 · p. 15
8.3. Runtime Comparison
- 17.7x w.r.t stable-dreamfusion and 10x w.r.t. Latent Painter; TexFusion’s 9-camera run takes 2.2 min versus TEXTure’s 2.9 min.The 24-camera TexFusion configuration takes 6.2 min.Source: §8.3 · pp. 16, 18, Table 2 · p. 18
9. Experiment details
The experiment details describe the user-study protocol and the sources of the mesh dataset.
9.1. User study details
- Each survey participant compared a TEXTure result with a TexFusion result for the same prompt, shown side by side in randomized left-right order, and answered four questions.Each prompt was shown to three participants, and responses were aggregated.Source: §9.1 · p. 18
9.2. Dataset description
- The dataset contains 35 meshes from varied sources, including ShapeNetCore.v1, a Text2Mesh human model, Stanford 3D scans, and licensed commercial assets.The passage also identifies free Turbosquid models and freely shared scans among the sources.Source: §9.2 · p. 18, Table 3 · p. 19, Table 4 · p. 20
Related papers
This paper cites
- Learning Transferable Visual Models From Natural Language Supervision2021
- Score-Based Generative Modeling through Stochastic Differential Equations2020
- Classifier-Free Diffusion Guidance2022
- Diffusion Models Beat GANs on Image Synthesis2021
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics2015
- Hierarchical Text-Conditional Image Generation with CLIP Latents2022