PaperlayerView exact revision on arXiv

Source-linked AI summary

NaTex: Seamless Texture Generation as Latent Color Diffusion

Zeqiang Lai, Yunfei Zhao, Zibo Zhao, Xin Yang, Xin Huang, Jingwei Huang, Xiangyu Yue, Chunchao Guo

arXiv:2511.16317v1cs.CV

TL;DR

Multi-view texturing faces difficulties with occlusion, fine-grained alignment, and cross-view consistency. NaTex predicts texture color directly in 3D through a geometry-aware color VAE and multi-control DiT; the authors report improved texture results and generalization to downstream tasks. Its VAE reconstruction quality remains a boundary for higher-resolution textures.

  • Problem

    MVD texturing struggles with occluded regions, precise texture-to-geometry alignment, and consistency across views, content, color, and illumination.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    NaTex treats texture as a dense color point cloud and uses a geometry-aware color VAE with a multi-control DiT to generate color directly in 3D.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    NaTex is reported to outperform prior methods in texture coherence and alignment, with consistent advantages across four assessment metrics and near-perfect boundary alignment in visual comparisons.

    Source: §4.1 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    NaTex generalizes to material generation, texture refinement, and part segmentation and texturing, including applications without task-specific training.

    Source: §1 · p. 3, §5 · p. 8

    The VAE's reconstruction quality could be improved to support higher-resolution textures.

    Source: C. Limitations and Future Works · p. 12

Abstract

from arXiv · show

We present NaTex, a native texture generation framework that predicts texture color directly in 3D space. In contrast to previous approaches that rely on baking 2D multi-view images synthesized by geometry-conditioned Multi-View Diffusion models (MVDs), NaTex avoids several inherent limitations of the MVD pipeline. These include difficulties in handling occluded regions that require inpainting, achieving precise mesh-texture alignment along boundaries, and maintaining cross-view consistency and coherence in both content and color intensity. NaTex features a novel paradigm that addresses the aforementioned issues by viewing texture as a dense color point cloud. Driven by this idea, we propose latent color diffusion, which comprises a geometry-awared color point cloud VAE and a multi-control diffusion transformer (DiT), entirely trained from scratch using 3D data, for texture reconstruction and generation. To enable precise alignment, we introduce native geometry control that conditions the DiT on direct 3D spatial information via positional embeddings and geometry latents. We co-design the VAE-DiT architecture, where the geometry latents are extracted via a dedicated geometry branch tightly coupled with the color VAE, providing fine-grained surface guidance that maintains strong correspondence with the texture. With these designs, NaTex demonstrates strong performance, significantly outperforming previous methods in texture coherence and alignment. Moreover, NaTex also exhibits strong generalization capabilities, either training-free or with simple tuning, for various downstream applications, e.g., material generation, texture refinement, and part segmentation and texturing.

1. Introduction

The introduction motivates native 3D texturing as an alternative to MVD pipelines, whose projection-and-baking workflow faces occlusion, alignment, and cross-view consistency challenges. NaTex predicts color in 3D and combines geometry-aware encoding with flexible controls for texture generation and downstream tasks.

1. Introduction

  • MVD texturing generates geometry-aligned views and backprojects them into 3D textures, leveraging pretrained image models for quality and diversity [12] [14] [58] [61].The approach is also used in commercial products [20].Source: §1 · p. 2
  • MVD texturing struggles with occluded regions, fine-detail alignment, and consistency across content, color, and illumination, producing artifacts during projection and baking.Figure 2 illustrates these challenges in comparison with NaTex.Source: §1 · p. 2, Figure 2 · p. 2
  • NaTex models texture as a dense color point cloud and predicts RGB color directly for 3D coordinates using latent diffusion, avoiding intermediate UV maps or Gaussian splatting.Its color point-cloud VAE achieves over 80× compression for efficient DiT generation.Source: §1 · pp. 2–3
  • NaTex co-designs its VAE and DiT: a geometry branch supplies surface cues, while positional embeddings and geometry tokens guide color generation.The multi-control DiT supports geometry and image controls, as well as initial-texture control for material generation and refinement.Source: §1 · p. 3
  • NaTex-2B significantly outperforms prior approaches in texture occlusion, coherence, and alignment, with state-of-the-art performance particularly in coherence and alignment.The framework is also presented as extensible to material modeling and part-level semantic segmentation.Source: §1 · p. 3

2. Related Works

Prior 3D texturing work adapts 2D generative models or uses native 3D representations, but projection-based methods still face consistency and fine-detail alignment challenges. NaTex instead generates texture directly in 3D using latent diffusion.

2.1. 3D Texturing via 2D Priors

2.2. Native 3D Texture Generation

  • Earlier native methods use GAN, feed-forward, or diffusion generators across face, point, UV, texture-field, and Gaussian-splatting representations [35] [42] [49] [53] [54] [55].Other approaches include octree compression, UV-map VAEs, SLAT, and triplane refinement.Source: §2.2 · p. 3
  • NaTex generates textures directly in 3D from input geometry within a latent-diffusion framework, using a geometry-aware color VAE and a multi-control color DiT.Figure 3 presents the architecture and shows NaTex-generated assets.Source: §2.2 · p. 4, Figure 3 · p. 4

3. Method

NaTex represents texture as color in 3D space and generates it with a geometry-aware color VAE and a multi-control color DiT. Its method combines native geometry guidance with controls for images and initial textures, supporting several downstream 3D tasks.

3. Method

  • NaTex synthesizes textures directly in 3D, addressing view- and UV-space limitations including occlusion handling, geometry alignment, and UV-quality dependence.The framework uses a geometry-aware color VAE and a multi-control color DiT.Source: §3 · p. 4, §3.1 · p. 4

3.1. Color Representation and VAE

  • Textures are represented as a 3D color field, and a geometry-aware VAE encodes point clouds containing color, position, and normal information.A tightly integrated geometry branch encodes coordinates and normals, with its latent queries guiding the texture encoder; the shared latent space also accommodates material and semantic signals.Source: §3.1 · pp. 4–5

3.2. Multi-Control Color DiT

  • The multi-control color DiT combines image and native geometry conditions, using positional embeddings for coarse structure and geometry latents for fine-grained surface guidance.The isomorphic geometry and texture latent sets are concatenated channel-wise for pointwise guidance; an encoded initial texture can also provide color control.Source: §3.2 · p. 5

3.3. Applications

  • Color control supports material generation by encoding roughness and metallic as channels and conditioning generation on an albedo latent set.The approach reuses the color VAE framework for these material signals.Source: §3.3 · p. 6
  • With color control, NaTex acts as a texture refiner that inpaints occluded regions and corrects texture in five steps.The passage describes this process as fast and efficient for downstream refinement tasks.Source: §3.3 · p. 6
  • The framework also supports training-free part segmentation from 2D image segmentation and 3D part texturing that handles occluded regions.Part segmentation uses the input image's segmentation to generate an aligned texture map; part texturing predicts color directly on 3D part surfaces.Source: §3.3 · p. 6

4. Experiments

Across comparisons, NaTex outperforms other methods on texture generation and produces cleaner, better-aligned refinements. Geometry-aware conditioning and longer token lengths improve alignment, while one-step generation is possible without distillation.

4.1. Comparison

  • Reconstruction quality improves as latent size increases, despite training with at most 6144 tokens.PSNR is measured on the color point cloud; PSNR*, SSIM*, and LPIPS* are measured on rendered 2D images.Source: §4.1 · pp. 6–7
  • NaTex consistently outperforms compared methods across c-FID, LPIPS, CMMD, and CLIP-I, with near-perfect geometry-boundary alignment.The albedo comparison includes Paint3D, TexGen, Hunyuan3D-2, RomanTex, and MaterialMVP; Figure 7 uses albedo-only renders.Source: §4.1 · p. 7, Figure 7 · p. 7, Table 2 · p. 8
  • Against OpenCV interpolation, NaTex produces cleaner, better-aligned textures in occluded regions.Figure 8 contrasts conventional inpainting with NaTex's neural inpainting on a window occlusion.Source: §4.1 · p. 8, Figure 8 · p. 8

4.2. Evaluation

  • Removing RoPE or replacing the tightly coupled shape embedding with a standalone shape VAE weakens image-texture alignment; without tight embedding, color sometimes diffuses.The examples include awning stripes, traffic-light colors, and color diffusion on a chair back.Source: §4.2 · p. 8, Figure 9 · p. 8
  • Generation quality and alignment improve as token length increases, and the model supports one-step generation without distillation.The authors attribute one-step generation capability to strong conditioning.Source: §4.2 · p. 8, Figure 10 · p. 8

5. Conclusion

NaTex formulates texture generation directly in 3D using a latent-diffusion paradigm, addressing challenges of occlusion, alignment, and consistency. The framework also supports downstream tasks without task-specific training.

5. Conclusion

  • NaTex formulates 3D texture generation as latent diffusion without a multi-stage pipeline using 2D lifting priors.The conclusion presents texture generation directly in 3D space as the framework's central formulation.Source: §5 · p. 8
  • The VAE and DiT designs mitigate challenges in occlusion texturing, fine-grained texture-geometry alignment, and texture consistency.Source: §5 · p. 8
  • NaTex demonstrates versatility across downstream tasks, including applications that require no task-specific training.Source: §5 · p. 8

A. Implementation Details

Implementation uses a 300M-parameter color VAE and a 1.9B-parameter color DiT trained with flow matching. Training uses rendered views and sampled color point clouds from meshes.

A. Implementation Details

  • The color VAE has 300M parameters and the color DiT has 1.9B parameters; both are trained using a flow-matching objective.Source: A. Implementation Details · p. 9
  • The VAE is trained with at most 6144 tokens; DiT training uses batch size 256, 500 warm-up steps, and a learning rate decaying from 1 × 10−4 to 1 × 10−5.Source: A. Implementation Details · p. 9
  • After pretraining converges, illumination-invariant loss is added with weight 5; classifier-free guidance drops conditioning embeddings with 10% probability, and default inference uses 5 steps at guidance scale 2.Source: A. Implementation Details · p. 9
  • Meshes are sampled into uniform color point clouds, while 24 views are rendered with varied elevations and point, area, or HDRI lighting.Source: A. Implementation Details · p. 9

B. More Details on Applications

NaTex adapts its texture framework to materials, segmentation, part texturing, and refinement through different control signals. The applications include zero-shot uses and fine-tuned variants, with reported results for each.

B. More Details on Applications

C. Limitations and Future Works

NaTex’s limitations point to improvements in reconstruction quality, data curation, part segmentation, and handling adjacent closed surfaces. The authors also identify opportunities to incorporate more 2D priors into texture refinement.

C. Limitations and Future Works

Loading 2511.16317v1…