Source-linked AI summary
TEXGen: a Generative Diffusion Model for Mesh Textures
Xin Yu, Ze Yuan, Yuan-Chen Guo, Ying-Tian Liu, JianHui Liu, Yangguang Li, Yan-Pei Cao, Ding Liang, Xiaojuan Qi
TL;DR
Learning directly in UV texture space offers an alternative to mesh-texturing methods that rely on test-time optimization with pretrained 2D diffusion models. TEXGen trains a large diffusion model with a hybrid 2D-3D architecture to generate high-resolution textures feed-forward, and the paper reports state-of-the-art results alongside several zero-shot applications.
Problem
Existing learning-based mesh-texturing methods are restricted to specific categories, while test-time optimization methods can be slow, require parameter tuning, inherit 2D-prior limitations, and produce poor 3D consistency.
Source: §1 · p. 2Method
TEXGen trains a diffusion model directly on UV texture maps, using a hybrid network that combines UV-space convolutions with 3D point-cloud attention.
Source: §1 · p. 2, §4.2 · p. 3Results
The authors report state-of-the-art results and present TEXGen as the first feed-forward model for generating textures for general objects, with high-quality detail, condition preservation, and global coherence.
Source: §1 · p. 2, §5.1 · p. 6, §6 · p. 10Takeaways & Limitations
The trained model supports zero-shot text-guided texture synthesis, inpainting, and sparse-view texture completion.
Source: §6 · p. 10Further acceleration through model compression or diffusion acceleration methods such as consistency distillation is left to future work.
Source: §5.1 · p. 8
Abstract
from arXiv · showhide
While high-quality texture maps are essential for realistic 3D asset rendering, few studies have explored learning directly in the texture space, especially on large-scale datasets. In this work, we depart from the conventional approach of relying on pre-trained 2D diffusion models for test-time optimization of 3D textures. Instead, we focus on the fundamental problem of learning in the UV texture space itself. For the first time, we train a large diffusion model capable of directly generating high-resolution texture maps in a feed-forward manner. To facilitate efficient learning in high-resolution UV spaces, we propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. Leveraging this architectural design, we train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images. Once trained, our model naturally supports various extended applications, including text-guided texture inpainting, sparse-view texture completion, and text-driven texture synthesis. Project page is at http://cvmi-lab.github.io/TEXGen/.
1 INTRODUCTION
TEXGen targets scalable, generalizable mesh texturing by learning directly in UV texture space rather than relying on test-time optimization with pretrained 2D diffusion models. Its hybrid 2D-3D architecture supports feed-forward, high-resolution texture generation and several training-free applications.
1 INTRODUCTION
- Prior learning-based methods are category-restricted, while test-time optimization methods require per-object tuning and can have poor 3D consistency.The latter also inherit limitations of 2D priors.Source: §1 · p. 2
- The model uses UV maps for scalable, high-resolution detail and direct supervision from ground-truth textures, enabling diffusion training without relying solely on rendering loss.The authors contrast this with earlier approaches that did not achieve end-to-end training or feed-forward inference on general object datasets.Source: §1 · p. 2
- The hybrid architecture combines UV-space convolutions for local detail with 3D sparse convolutions and attention for global dependencies disrupted by UV parameterization.Sparse 3D features aim to maintain continuity with manageable computation instead of dense voxel or point features.Source: §1 · p. 2
- The trained diffusion model generates high-resolution maps, including 1024×1024 textures, from text prompts and single-view images, and supports synthesis, inpainting, and sparse-view completion.The contribution summary describes these extensions as training-free.Source: §1 · p. 2
- TEXGen is presented as the first end-to-end, feed-forward model for generating high-resolution texture maps for general objects, without additional stages or test-time optimization.The authors also report state-of-the-art results and training-free applications.Source: §1 · p. 2
2 RELATED WORK
Prior mesh-texturing work includes test-time optimization with 2D diffusion priors, learned texture generators, and feed-forward 3D generation models. The section identifies remaining constraints in generalization, optimization, 3D awareness, and texture detail.
2 RELATED WORK
- Test-time methods use pretrained 2D diffusion priors or geometry-conditioned generation, but often require per-instance optimization and lack 3D awareness or cross-view consistency.Score-distillation approaches can also produce artifacts such as Janus views and unnatural color.Source: §2 · p. 2
- Implicit texture fields can miss high-frequency detail, while GAN-based mesh methods are susceptible to mode collapse.Texturify and Mesh2Tex predict textures for mesh faces using convolutional designs and GAN training.Source: §2 · p. 3
- Direct UV-map generators have generally been category-specific; Paint3D handles generalized objects but still uses test-time optimization and a two-stage pipeline.The authors note that Paint3D’s trained diffusion model removes light effects and fills holes, with cumulative quality losses sometimes degenerating final details.Source: §2 · p. 3
- Feed-forward 3D reconstruction models avoid per-instance optimization but can yield over-smoothed unseen regions and coarse-grained feature interactions that are difficult to adapt to surface textures.The cited models also lack varied outcomes by design.Source: §2 · p. 3
3 OVERVIEW
The overview frames TEXGen as conditional texture generation for a 3D mesh, using UV maps and a hybrid network to train a diffusion model. At inference, it denoises from noise and supports several training-free extensions.
3 OVERVIEW
- Given a 3D mesh, the objective is to generate high-quality surface textures conditioned on user-provided images or text prompts.Source: §3 · p. 3
- UV texture maps are used as mesh texture representation; they are compact and suitable for diffusion training, and their characteristics motivate development of a new network architecture.Source: §3 · p. 3
- The model trains a hybrid 2D-3D network and diffusion process to generate high-resolution texture maps for a mesh from a single-view image and text description.Source: §3 · p. 3
- At inference, the model iteratively denoises from an image of noise and supports training-free text-guided synthesis, inpainting, and sparse-view completion.Source: §3 · p. 3
4 METHOD
TEXGen uses UV texture maps for efficient, high-resolution detail learning, while incorporating 3D point features to address the global consistency lost through UV fragmentation. Its conditional diffusion model combines a hybrid 2D–3D network with diffusion and rendering supervision, and supports several texture-generation tasks at inference.
4.1 Representation for Texture Synthesis
- UV maps support grid-based texture learning and direct supervision, but fragmentation can disrupt global 3D consistency across mesh islands.TEXGen combines UV-space detail learning with 3D points to preserve global consistency.Source: §4.1 · p. 3
4.2 Model Construction
- The denoising network conditions on text and image inputs, using projected image pixels as a partial texture and global embeddings to modulate features.The network predicts velocity, and image conditioning can be omitted at inference.Source: §4.2 · pp. 3–4
- The hybrid block applies UV convolutions for local surface detail, then remaps features to 3D points to learn neighborhood relationships and global structure.The 3D branch uses sparse features for scalability rather than extracting high-resolution details.Source: §4.2 · p. 4
4.3 Diffusion Learning
- Training adds noise to real texture maps and uses v-prediction for diffusion loss, with an auxiliary LPIPS loss on rendered views.The predicted texture is rendered from random viewpoints and compared with corresponding ground-truth images; the final loss weights are 1 for diffusion and 0.5 for rendering.Source: §4.3 · pp. 5–6
4.4 Texture Generation
- At inference, the model iteratively denoises Gaussian noise with conditioning such as a single-view image or text prompt, using 30-step DDIM sampling.Text-only generation can use a ControlNet-generated image; the model also supports partial-texture inpainting and completion from sparse views.Source: §4.4 · p. 6
5 EXPERIMENTS
TEXGen is evaluated on 120,000 training pairs and 400 evaluation pairs, with comparisons, applications, and design ablations. It produces detailed, coherent textures, performs favorably in reported comparisons, supports text-driven and partial-texture applications, and benefits from its hybrid block and tuned guidance weight.
5 EXPERIMENTS
- On 120,000 training pairs and 400 evaluation pairs from processed Objaverse meshes, TEXGen generates textures conditioned on a single-view image and text prompt.Reported results show rich local detail, preservation of condition information, and global coherence.Source: §5 · p. 6, §5.1 · p. 6
5.1 Main Results and Comparisons
- Compared with TEXTure, Text2Tex, and Paint3D, TEXGen produces rich detail and global coherence while avoiding the Janus problem seen in those methods.The comparisons also report overly smooth textures, artifacts, and weaker preservation of the guided image for the other methods.Source: §5.1 · pp. 6, 8
- TEXGen significantly outperforms other methods on FID and KID and is reported to achieve state-of-the-art texture quality with faster inference.The comparison evaluates multi-view renderings; evaluations take under 10 seconds without test-time optimization.Source: Table 1 · p. 8, §5.1 · p. 8
5.2 Applications
- Text-conditioned scene texturing uses Depth-ControlNet-generated views, and human preference and MLLM scores favor TEXGen over other methods.The scene objects can be customized with individual text prompts.Source: §5.2 · p. 9
- TEXGen fills randomly masked texture regions seamlessly and completes unseen regions from sparse multi-view images with continuity and coherence.The sparse-view examples begin with unprojected textures that retain many unseen regions.Source: §5.2 · p. 9, Figure 8 · p. 9, Figure 9 · p. 9
5.3 Model Analysis
- In a parameter-matched ablation, the hybrid-block Model A performs best on FID and KID; UV-only Model B shows style inconsistencies, while point-only Model C lacks high-frequency detail.The ablation trains on 10,000 house models and tests on 100; qualitative results associate the UV-only variant with inconsistencies and the point-only variant with blur.Source: §5.3 · p. 10, Table 3 · p. 10, Figure 10 · p. 10
- A classifier-free guidance weight of 2.0 provides the reported optimal balance for TEXGen, differing from the 7.5 weight typical of image diffusion models.The ablation evaluates different guidance weights using FID and KID.Source: §5.3 · p. 10, Table 4 · p. 10
6 CONCLUSION
TEXGen is a feed-forward diffusion model for high-resolution textures on general 3D objects, using a hybrid 2D–3D block to balance local detail and global interactions. It supports several zero-shot texture applications.
6 CONCLUSION
- TEXGen generates detailed, coherent high-resolution textures for general 3D objects in a feed-forward manner.The conclusion presents it as the first feed-forward model capable of generating textures for general objects and says it sets a new benchmark.Source: §6 · p. 10
- Its hybrid 2D–3D block manages local detail fidelity alongside global 3D-aware interactions.Source: §6 · p. 10
- The model supports zero-shot text-guided inpainting, sparse-view texture completion, and text-to-texture synthesis.Source: §6 · p. 10
A.1 Implementation Details
The implementation uses a curated Objaverse dataset, a staged hybrid network, and large-scale diffusion training on 32 A100 GPUs.
A.1 Implementation Details
- From over 800K Objaverse meshes, processing yielded 120,400 textured meshes, split into 120,000 training and 400 evaluation examples.Meshes were filtered, re-unfolded into a single UV atlas with xAtlas, and diffuse colors were baked onto the reparameterized meshes; Gemini supplied captions.Source: §A.1 · p. 12
- The five-stage network uses UV blocks first, then hybrid blocks, with sparse convolution replacing point attention in stage two.The remaining three stages use hybrid blocks with 2, 4, and 6 attention layers; channel counts are 32, 256, 1024, 1024, and 2048.Source: §A.1 · p. 12
- Training ran for 400K iterations on 32 A100 GPUs with batch size 64 and AdamW.The learning rate followed a cosine schedule from 2e-4 to zero; β1 = 0.9, β2 = 0.999, and weight decay was 0.05.Source: §A.1 · p. 12
A.2 More Qualitative Results
Qualitative results indicate that TEXGen avoids duplicated facial features and handles real-scan meshes with irregular geometry and UV layouts.
A.2 More Qualitative Results
- Training on 3D data and generating the full UV map in one forward pass effectively prevents the Janus problem.Here, the Janus problem is unintended duplication of features such as eyes and noses on both sides of a 3D asset.Source: §A.2 · p. 12
- TEXGen handles real-scan models with non-smooth surfaces and fragmented, irregular UV maps.The text reports high-quality texture maps that maintain the original objects’ details and authenticity.Source: §A.2 · p. 12
- A supplementary video demonstrates mesh renderings produced by the model.Source: §A.2 · p. 12
A.3 Limitations and Discussions
The training condition images are pose- and shape-aligned, limiting direct use with arbitrary images for texture transfer. The discussion also identifies PBR material-map generation as future work.
A.3 Limitations and Discussions
- Training condition images are pose- and shape-aligned, which may not suit texture transfer from arbitrary images.The authors suggest dense image information through mechanisms such as cross-attention as a possible direction, while identifying a suitable dataset as the primary challenge.Source: §A.3 · p. 12
- Extending TEXGen to generate PBR material maps is future work that requires collecting and processing suitable training data.Source: §A.3 · p. 12
- The model is reported to avoid the Janus problem on 3D avatars.Source: Figure 11 · p. 13
- The method is reported to handle real-scan models with non-smooth surfaces and fragmented UV maps.Source: Figure 12 · p. 14