Source-linked AI summary
Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models
Xianfang Zeng, Xin Chen, Zhongqi Qi, Wen Liu, Zibo Zhao, Zhibin Wang, Bin Fu, Yong Liu, Gang Yu
TL;DR
Paint3D addresses the challenge of generating high-quality 3D textures without embedded illumination, which can cause problems during relighting. It combines multi-view 2D diffusion with shape-aware UV refinement and reports state-of-the-art texturing performance, including improved FID and KID. The method remains limited by multi-face failures and does not generate material maps or edit geometry.
Problem
3D texture generation must support diverse objects and conditions while eliminating illumination coupled to textures produced by pretrained models.
Method
Paint3D progressively generates coarse textures from multi-view 2D diffusion and refines them in UV space using position-aware UV Inpainting and UVHD diffusion models.
Results
Paint3D achieves state-of-the-art texturing performance and improves FID by 29.93% and KID by 39.42% over prior methods.
Takeaways & Limitations
The resulting 2K UV textures are semantically consistent, lighting-less, and suitable for graphics editing and synthesis tasks.
Takeaways & Limitations
Paint3D can fail on multi-face cases and does not generate material maps or edit 3D geometry.
Abstract
from arXiv · showhide
This paper presents Paint3D, a novel coarse-to-fine generative framework that is capable of producing high-resolution, lighting-less, and diverse 2K UV texture maps for untextured 3D meshes conditioned on text or image inputs. The key challenge addressed is generating high-quality textures without embedded illumination information, which allows the textures to be re-lighted or re-edited within modern graphics pipelines. To achieve this, our method first leverages a pre-trained depth-aware 2D diffusion model to generate view-conditional images and perform multi-view texture fusion, producing an initial coarse texture map. However, as 2D models cannot fully represent 3D shapes and disable lighting effects, the coarse texture map exhibits incomplete areas and illumination artifacts. To resolve this, we train separate UV Inpainting and UVHD diffusion models specialized for the shape-aware refinement of incomplete areas and the removal of illumination artifacts. Through this coarse-to-fine process, Paint3D can produce high-quality 2K UV textures that maintain semantic consistency while being lighting-less, significantly advancing the state-of-the-art in texturing 3D objects.
1. Introduction
Paint3D addresses the need for diverse 3D textures that are compatible with relighting by generating high-resolution, lighting-less UV maps from text or image conditions. Its coarse-to-fine framework combines multi-view 2D diffusion with UV-space refinement to remove incomplete regions and illumination effects.
- The coarse stage samples multi-view images from a depth-aware 2D diffusion model and back-projects them onto the mesh to form an initial texture map.
- Pre-illuminated textures can introduce inappropriate shadows during relighting and lighting errors in common graphics workflows.
- Separate UV Inpainting and UVHD diffusion models refine incomplete regions and remove lighting influences in the texture map.
- Paint3D produces semantically consistent textures without intrinsic illumination effects and achieves state-of-the-art texturing performance.
- Paint3D generates diverse, high-resolution 2K UV textures for untextured 3D meshes from textual or visual conditions.
2. Related Work
The related work spans traditional texture synthesis, iterative 2D diffusion methods, optimization-based 3D generation, and models trained directly on 3D data. Paint3D is presented within this progression as a coarse-to-fine framework using multi-view diffusion and UV-space refinement.
- Traditional methods place exemplar patterns or use global optimization, while learning-based approaches target plausible textures for complex 3D shapes.
- Paint3D’s framework combines multi-view image diffusion, mesh back-projection, and UV-space refinement as a coarse-to-fine texturing pipeline.
- Iterative 2D diffusion methods progressively paint textures from multiple viewpoints but can lack global information and produce view inconsistencies.
- Optimization-based methods use vision-language guidance or score distillation for text-to-3D generation, whereas Paint3D learns on the whole texture map while preserving 3D geometry.
- Generative texturing methods trained on 3D data include implicit texture fields, but discrete supervision can hinder high-quality texture synthesis.
3. Method
Paint3D progressively generates coarse texture maps from depth-aware multi-view diffusion and back-projection, then refines them in UV space. Separate position-aware diffusion components fill holes, remove illumination, and enhance texture definition to produce complete, lighting-less, high-resolution maps.
- Coarse Texture Generation: Paint3D first samples depth-conditioned multi-view images and back-projects them onto the mesh to create an initial coarse texture map.The process alternates rendering, sampling, and back-projection across viewpoints, progressively covering the mesh.
- Coarse Texture Generation: The coarse map is progressively fused view by view, preserving previously textured regions while updating uncolored areas until the mesh is covered.For non-initial viewpoints, rendered partial RGB images and masks guide depth-aware inpainting before back-projection.
- Texture Refinement in UV Space: UV-space refinement addresses coarse-map holes and lighting shadows introduced by 2D diffusion and self-occlusion during rendering.The refinement process uses UV-space diffusion to improve completeness and visual quality.
- Texture Refinement in UV Space: A position-map encoder injects 3D adjacency information into diffusion, allowing texture fragments in UV space to be refined with shape awareness.The position map stores 3D point coordinates for nonbackground UV elements and is fused through a dedicated encoder.
- Texture Refinement in UV Space: Paint3D performs UV inpainting followed by UVHD enhancement, producing complete, lighting-less, high-resolution, and diverse UV texture maps.UV inpainting fills texture holes, while UVHD enhances visual aesthetics; the models are trained with UV texture-space supervision and lighting-less priors.
4. Experiments
Paint3D is evaluated on text-to-texture and image-to-texture tasks using quantitative, qualitative, user-study, and ablation analyses. It outperforms baselines on reported quality and fidelity measures, while ablations show that both coarse generation and refinement components contribute to performance.
- 4.2–4.3. Comparisons: Qualitatively, Paint3D produces illumination-free textures with finer details than text-conditioned baselines and represents image-conditioned texture details more accurately than TEXTure.The text comparison includes Latent-Paint, TEXTure, and Text2Tex; the image comparison highlights high-frequency details such as armor lines.
- 4.2. Comparisons on Text-to-Texture: Paint3D outperforms text-to-texture baselines in overall quality and fidelity to text prompts in a 30-user study over 60 meshes.Users rated both criteria on a 1-to-5 scale using 360-degree presentations.
- 4.4. Ablation Studies: Removing either the coarse stage or refinement stage produces inferior results, with missing coarse initialization causing semantic problems and missing refinement leaving textures pre-illuminated.The full model integrates coarse generation with UV-space refinement, as summarized in Table 3.
- 4.4. Ablation Studies: Six viewpoints give the best result, while sampling symmetric viewpoint pairs during one diffusion process further improves the texture.Increasing viewpoint count helps up to the best setting but does not make performance monotonically better.
5. Disscusion
Paint3D generates semantically consistent, high-quality 2K UV textures that are lighting-less through a coarse-to-fine framework. Its limitations include multi-face failures, difficulty generating material maps, and inability to edit geometry.
- Paint3D produces high-quality, lighting-less, diverse 2K texture maps that maintain semantic consistency.
- The method combines multi-view texture fusion with separate UV Inpainting and UVHD models for incomplete regions and illumination artifacts.
- Multi-face inconsistencies in the coarse stage can cause failures because the pretrained 2D diffusion model was not explicitly trained on multi-view datasets.
- Paint3D remains unable to generate material maps or edit the geometry of 3D assets.
Appendix
The appendix contains additional qualitative results, experiments, and discussion of Paint3D’s failure cases.
- The appendix provides more qualitative results, additional experiments, and discussion of failure cases.
A. Qualitative Results
Paint3D produces lighting-less textures that respond appropriately to different lighting directions and supports text- and image-conditioned qualitative generation.
- Paint3D’s lighting-less texture maps produce appropriate shadows when textured meshes are illuminated from different directions.
- Text-to-texture results include samples generated from test-set prompts under various random seeds.
- Additional text-to-texture results show each textured mesh from three viewpoints.
- Image-to-texture samples use wild-collected input images and show each textured mesh from two viewpoints.
B. Additional Experiments
Additional experiments study position maps in the UV Inpaint and UVHD modules and compare Paint3D with category-specific texture-generation approaches.
- The appendix evaluates the effectiveness of position maps in the UV Inpaint and UVHD modules.
- The appendix also provides further comparisons with category-specific texture-generation approaches.
B.1. Evaluation of Position Map
The position map is essential to both UV Inpainting and UVHD refinement. Removing it significantly reduces performance and causes semantic confusion or distortion.
- B.1. Evaluation of Position Map: Removing the position map significantly decreases performance in both UV Inpainting and UVHD, demonstrating its crucial role in UV-space refinement.Table 5 evaluates both ablations, while the accompanying experiments describe a significant performance decrease without position-map guidance.
- B.1. Evaluation of Position Map: Without a position map, UV Inpainting produces semantically confused textures in previously uncolored regions.The purple area marks the uncolored region in the visualization.
- B.1. Evaluation of Position Map: Without a position map, UVHD produces distorted or semantically incoherent enhanced textures.The visualization shows distortion in one case and loss of semantic coherence in another.
B.2. Comparisons with Category-Specific Model
On ShapeNet chairs and tables, Paint3D is compared with the category-specific state-of-the-art Point-UV under text and image conditions. Paint3D achieves comparable results in both settings.
- B.2. Comparisons with Category-Specific Model: Paint3D achieves comparable texture-generation results to Point-UV on ShapeNet chairs and tables under both text and image conditions.Point-UV is selected as the category-specific state-of-the-art baseline with both textual and visual conditional inputs.
C. Discussion on failure case
Paint3D can fail during coarse texture generation because inconsistent multi-view images may create duplicate faces. The authors attribute this issue to the pretrained 2D diffusion model's lack of explicit multi-view training.
- C. Discussion on failure case: Paint3D’s coarse stage can generate duplicate faces across front and back views, producing failure cases such as repeated mouse or lion faces.The failure arises from inconsistent multi-view texture images sampled by the pretrained 2D diffusion model.
- C. Discussion on failure case: The authors suggest fine-tuning or retraining the 2D diffusion model on large-scale multi-view datasets to improve texture consistency.This proposed remedy targets the coarse-stage multi-view inconsistency.