PaperlayerView exact revision on arXiv

Source-linked AI summary

GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation

Jiawei Lu, Yingpeng Zhang, Zengjun Zhao, He Wang, Kun Zhou, Tianjia Shao

arXiv:2409.18401v1cs.CVcs.AI

TL;DR

Text-to-texture generation must adapt 2D diffusion models to 3D surfaces while maintaining texture quality and cross-view consistency. The paper combines geometry-aware local attention, latent merging, and texture completion, and reports that it surpasses baseline methods in detail, color richness, and consistency.

  • Problem

    Text-guided texture synthesis must bridge the gap between 2D images and 3D surfaces, while high-quality text-labeled texture data are scarce.

    Source: §1 · p. 1

  • Method

    The framework combines pretrained-model local attention guided by 3D patch proximity with latent merging and texture dilation.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method significantly surpasses baseline methods in detail, color richness, and multi-view consistency while preserving pretrained text-to-image models' generative potential.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Without additional training or fine-tuning, the framework is compatible with a wide range of existing Stable Diffusion models and auxiliary models such as IP-Adapter and LoRAs.

    Source: §4.4 · p. 7

    Mesh-texture alignment remains imperfect, largely because currently available ControlNets have limited control capabilities.

    Source: Limitation · p. 7

Abstract

from arXiv · show

Large-scale text-guided image diffusion models have shown astonishing results in text-to-image (T2I) generation. However, applying these models to synthesize textures for 3D geometries remains challenging due to the domain gap between 2D images and textures on a 3D surface. Early works that used a projecting-and-inpainting approach managed to preserve generation diversity but often resulted in noticeable artifacts and style inconsistencies. While recent methods have attempted to address these inconsistencies, they often introduce other issues, such as blurring, over-saturation, or over-smoothing. To overcome these challenges, we propose a novel text-to-texture synthesis framework that leverages pretrained diffusion models. We first introduce a local attention reweighing mechanism in the self-attention layers to guide the model in concentrating on spatial-correlated patches across different views, thereby enhancing local details while preserving cross-view consistency. Additionally, we propose a novel latent space merge pipeline, which further ensures consistency across different viewpoints without sacrificing too much diversity. Our method significantly outperforms existing state-of-the-art techniques regarding texture consistency and visual quality, while delivering results much faster than distillation-based methods. Importantly, our framework does not require additional training or fine-tuning, making it highly adaptable to a wide range of models available on public platforms.

1 Introduction

Text-guided texture synthesis must bridge 2D image diffusion and 3D surfaces while balancing texture quality and cross-view consistency. The paper introduces a pretrained-diffusion framework combining geometry-aware local attention with latent merging and texture completion.

1 Introduction

  • Texture synthesis lacks high-quality text-labeled data, and the gap between 2D images and 3D surfaces makes consistent, high-quality textures challenging.Even geometric guidance such as depth maps in ControlNet does not resolve the challenge.Source: §1 · p. 1
  • Existing methods either optimize 3D structure with Score Distillation Sampling, often slowly and with reduced diversity and quality, or generate views through sequential inpainting or multi-view diffusion.The cited optimization approaches include Poole et al. 2022 and Lin et al. 2023; the view-generation approaches include Chen et al. 2023b and Liu et al. 2023c.Source: §1 · pp. 1–2
  • The framework uses 3D patch locations to weight cross-view self-attention, adds a latent-space merge pipeline, and fills uncolored UV pixels by discretizing UV space into sub-UV islands.Nearby patches receive higher attention weights than farther patches.Source: §1 · p. 2
  • The approach significantly surpasses baseline methods in detail, color richness, and multi-view consistency while preserving the generative potential of pretrained text-to-image models.The evaluations cover a variety of 3D objects.Source: §1 · p. 2

2 Related Works

Related work moves from text-conditioned image diffusion and its spatial-control methods to pretrained 2D diffusion approaches for 3D generation and mesh-guided texture synthesis.

2.1 Text-to-Image Diffusion Models

  • Text-to-image diffusion transforms random noise into visuals through Markov chains; ControlNet and T2I-Adapter support spatially conditioned generation using inputs such as depth or normal maps.Stable Diffusion applies the diffusion process in latent space to reduce computational cost.Source: §2.1 · p. 2

2.2 Text-driven 3D Generation

  • Text-driven 3D generation faces scarce publicly available labeled 3D data, so methods use pretrained 2D diffusion for score distillation or fine-tuned multi-view and novel-view generation.The cited approaches include VSD for enriching detail and diversity, and Zero-1-to-3, MVDream, and SyncDreamer for view consistency.Source: §2.2 · p. 2

2.3 Mesh-guided Texture Synthesis

  • Mesh-guided texture synthesis has moved from category-limited GANs to pretrained diffusion methods using sequential generation and inpainting, multi-view diffusion, or score distillation.The cited GAN works include Oechsle et al. 2019 and Siddiqui et al. 2022.Source: §2.3 · p. 2

3 Method

The method adapts pretrained image-space diffusion to mesh-conditioned texture synthesis, using geometry-aware local attention and latent-space merging to balance detail, consistency, and diversity. It then reconstructs a texture map and completes pixels left invalid by self-occlusion.

3.1 Preliminary

3.2 Local Attention

  • Geometry-aware local attention reweights self-attention using distances between rendered 3D patch positions, emphasizing corresponding surface regions across views.The method replaces the original similarity with the geometry-derived weight matrix only during early diffusion steps, avoiding late-stage blurring and shape distortion.Source: §3.2 · p. 4

3.3 Consistent Texture Synthesis

  • View-dependent latent merging combines partial textures across views, beginning with similar weights and shifting toward view-specific influence as diffusion progresses.This schedule promotes style consistency early while preserving diversity and high-frequency details later; merging is omitted in the final five denoising steps to prevent artifacts from reprojection of low-resolution latents.Source: Latent merge pipeline · p. 5
  • A fast texture-completion method approximates surface-space color propagation to fill invalid map pixels left by self-occlusion.The method avoids geodesic-distance completion because its computational cost is prohibitively high.Source: Final Texture Synthesis · pp. 5–6

4 Experiments

The experiments report strong human and Pick Score preferences, and ablations show how local attention and latent merging contribute to consistency and detail. The framework also supports Stable Diffusion extensions such as IP-Adapter and LoRAs.

4.1 Implementation details

  • Tests used 35 meshes and 63 mesh-prompt pairs, with eight fixed viewpoints; synthesis finished within one minute on an NVIDIA A800.The diffusion process took around 50 seconds, followed by around 2 seconds for final texture synthesis.Source: Dataset · p. 6, §4.1 · p. 6

4.2 Comparisons

  • Comparisons included four text-to-texture methods and Meshy-3, with FID, KID, Pick Score, and a user study evaluating rendered results.FID and KID used 12 rendered viewpoints; the user study assessed consistency, diversity, and overall quality.Source: §4.2 · p. 6, Quantitative comparisons · pp. 6–7
  • The method achieves the highest Pick Score and is preferred by most evaluators for consistency, diversity, and overall quality.The study collected 30 valid responses; the authors report the quantitative results in Table 1.Source: Table 1 · p. 6, Quantitative comparisons · p. 7

4.3 Ablation Studies

  • Geometry-aware local attention yields vivid color, fine-grained detail, and cross-view consistency, unlike global attention's detail and variance loss.The ablation compares attention strategies in generated views; Figure 5 also shows generated images and rendered textures.Source: Effectiveness of local attention · p. 7, Figure 5 · p. 7
  • The full pipeline with latent merging has the best consistency in final renderings, while versions without merging show severe inconsistencies.Figure 5's last column depicts the rendered result with synthesized texture.Source: Effectiveness of latent merge pipeline · p. 7, Figure 5 · p. 7

4.4 More Applications

  • The framework works with existing Stable Diffusion models without additional training and can incorporate IP-Adapter for image guidance or LoRAs for artistic styles.The paper points to supplementary results for these extensions.Source: §4.4 · p. 7

5 Discussions

The discussion identifies limitations in filling fully occluded regions and in mesh-texture alignment with current ControlNets.

5 Discussions

  • Texture dilation may over-smooth fully occluded regions that should contain complex textures.The paper also notes that pretrained 2D diffusion models can retain bias that produces unwanted anatomical features, despite mitigation from local attention and perspective prompts.Source: Failure Cases · p. 7
  • Mesh-texture alignment remains imperfect, which the authors attribute largely to the limited control capabilities of currently available ControlNets.They identify baked-in lighting as another common field limitation and leave it for future work.Source: Limitation · p. 7

6 Conclusions

The conclusion presents a text-prompted pipeline for generating consistent, high-quality textures for 3D meshes using pretrained Stable Diffusion models without further training or fine-tuning.

6 Conclusions

  • The proposed pipeline generates consistent, high-quality textures for 3D meshes from textual prompts using pretrained Stable Diffusion without further training or fine-tuning.The authors describe it as versatile across geometry and texture types and adaptable to models on model-sharing platforms.Source: §6 · p. 7

A Surface space color dilation

Surface-space color dilation fills UV pixels invisible from all views by selecting and blending nearby textured pixels using surface and connectivity cues.

A Surface space color dilation

  • The dilation iteratively fills invalid UV pixels by weighted averaging candidate colors selected using 3D distance, vertex-normal similarity, and sub-UV connectivity.Unlike naive UV-space flood fill, surface-space propagation avoids transferring colors between pixels that are not adjacent in 3D.Source: A Surface space color dilation · p. 10

B Implementation details

The implementation uses ComfyUI with nvdiffrast and fixes ControlNet and surface-space dilation parameters across experiments.

B Implementation details

  • ControlNet strength is 1.0; dilation uses grid size s = 64, distance threshold dth = 0.02, angle threshold ath = 90°, n = 30 nearest neighbors, and 10 iterations.The implementation uses ComfyUI and nvdiffrast for rendering and inverse rendering.Source: B Implementation details · p. 10

C More Results

The additional results report local attention's balance of multi-view consistency and detail, compare the method with Meshy-3, and show outputs across varied meshes and styles. The section also illustrates texture dilation across UV islands.

C More Results

  • Local attention achieves better visual quality and considerable diversity than Meshy-3 while maintaining surface consistency, whereas Meshy-3 can show ghosting and over-saturation.The comparison places our results on the left and Meshy-3 on the right in each group.Source: C More Results · p. 10, Figure 9 · p. 11
  • In the attention ablation without latent merging, local attention achieves the best multi-view consistency while preserving rich details close to unconstrained diffusion.The figure compares neighbor-only attention, attention to all other views, and local attention.Source: C More Results · p. 10, Figure 7 · p. 11
  • Additional comparisons show results across different methods, Objaverse meshes, and meshes from industrial games.The figures also include results across varied meshes and styles.Source: C More Results · p. 10, Figure 10 · p. 12, Figure 11 · p. 13
  • Surface-space dilation lets colors propagate between distant UV islands by selecting candidate textured pixels using 3D distance, normal similarity, and island connectivity.The algorithm iteratively processes pixels invisible from all views and computes their colors from weighted candidates.Source: Figure 6 · p. 11

Related papers

Loading 2409.18401v1…