PaperlayerView exact revision on arXiv

Source-linked AI summary

Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering

Kim Youwang, Tae-Hyun Oh, Gerard Pons-Moll

arXiv:2312.11360v2cs.CVcs.AIcs.GR

TL;DR

Generated assets are impractical for graphics engines because implicit geometry and textures require conversion to explicit meshes and compatible maps; manual unwrapping yields heterogeneous mappings that impede transfer and editing. Paint-it optimizes PBR texture maps for an untextured mesh from text using DC-PBR, a convolution-based neural re-parameterization. Paint-it produces high-quality PBR textures for general objects, humans, and animals, with maps compatible with graphics-engine relighting and material control.

  • Problem

    Generated assets are impractical for graphics engines because implicit geometry and textures require conversion to explicit meshes and compatible maps; manual unwrapping yields heterogeneous mappings that impede transfer and editing.

    Source: §1 · p. 2

  • Method

    Paint-it optimizes PBR texture maps for an untextured mesh from text using DC-PBR, a convolution-based neural re-parameterization.

    Source: §1 · p. 2

  • Results

    Paint-it produces high-quality PBR textures for general objects, humans, and animals, with maps compatible with graphics-engine relighting and material control.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The synthesized maps can be used in graphics engines for relighting and test-time material control, and different maps can create diverse appearances for one mesh.

    Source: §1 · p. 2

    Optimization takes approximately 15–30 minutes per mesh.

    Source: §6 · p. 8

Abstract

from arXiv · show

We present Paint-it, a text-driven high-fidelity texture map synthesis method for 3D meshes via neural re-parameterized texture optimization. Paint-it synthesizes texture maps from a text description by synthesis-through-optimization, exploiting the Score-Distillation Sampling (SDS). We observe that directly applying SDS yields undesirable texture quality due to its noisy gradients. We reveal the importance of texture parameterization when using SDS. Specifically, we propose Deep Convolutional Physically-Based Rendering (DC-PBR) parameterization, which re-parameterizes the physically-based rendering (PBR) texture maps with randomly initialized convolution-based neural kernels, instead of a standard pixel-based parameterization. We show that DC-PBR inherently schedules the optimization curriculum according to texture frequency and naturally filters out the noisy signals from SDS. In experiments, Paint-it obtains remarkable quality PBR texture maps within 15 min., given only a text description. We demonstrate the generalizability and practicality of Paint-it by synthesizing high-quality texture maps for large-scale mesh datasets and showing test-time applications such as relighting and material control using a popular graphics engine. Project page: https://kim-youwang.github.io/paint-it

1. Introduction

Handcrafted 3D texturing is laborious, while existing text-driven texture approaches can produce plausible colors without representing material properties or complex surface reflections. Paint-it optimizes PBR texture maps with DC-PBR and produces high-quality results compatible with graphics-engine applications.

1. Introduction

  • Hand-designing realistic, diverse 3D assets demands cumbersome, time-consuming labor.The paper motivates text-driven generation as a way to reduce this burden.Source: §1 · p. 1
  • Prior text-driven texture methods back-project generated image colors onto meshes, which may yield implausible textures without material properties or complex surface reflections.Texture diversity matters because the same mesh can be reused to create different appearances.Source: §1 · p. 2
  • Paint-it synthesizes high-fidelity texture maps from a text description on an untextured mesh through optimization, addressing practical texture-generation challenges.The paper identifies pixel-based parameterization as difficult for synthesizing PBR texture maps.Source: §1 · p. 2
  • DC-PBR re-parameterizes disentangled PBR maps with convolutional neural kernels, scheduling learning by texture frequency and filtering high-frequency noisy SDS signals.The maps include diffuse, roughness and metalness, and normal components.Source: §1 · p. 2
  • Paint-it produces high-quality textures for general meshes and its PBR maps integrate with graphics engines for relighting and material control.Different PBR maps can also simulate diverse appearances for the same mesh.Source: §1 · p. 2, Figure 2 · p. 2

2. Related Work

Related work spans text-driven 3D asset generation and methods that synthesize texture maps for meshes. Paint-it focuses on spatially structured PBR texture maps that remain compatible with practical graphics workflows.

Text-driven 3D Asset Generation

  • Text-driven asset generation commonly optimizes each instance using rendered views and text-similarity signals from CLIP or text-conditional diffusion models.The passage attributes this approach to the absence of large-scale paired text–3D asset datasets.Source: §2 · p. 3
  • Mesh-based methods use explicit geometry and offer faster rasterization and graphics-engine compatibility, while volume-based approaches require mesh extraction and texture unwrapping.Hybrid methods may still remesh and unwrap after volume optimization, introducing seams and losing editability.Source: §2 · p. 3

Text-driven Texture Map Synthesis

  • Paint-it’s mesh-based texture-map focus supports photorealistic renderings with fast, stable optimization and practical texture transfer or mesh animation.The paper contrasts its smooth, photorealistic assets with the non-smooth textures attributed to per-point material estimation.Source: §2 · p. 3
  • Image back-projection and latent-feature texture methods are limited to RGB colors, while Fantasia3D estimates per-point PBR materials rather than spatially structured maps.Paint-it instead optimizes neural re-parameterized PBR material maps without remeshing or texture unwrapping for each mesh.Source: §2 · p. 3

3. Paint-it: Text-Driven PBR Texture Synthesis via Neural Re-parameterized Optimization

Paint-it uses Score-Distillation Sampling (SDS) to synthesize PBR texture maps for a mesh from a text description. It optimizes a convolutional neural re-parameterization of the maps, rendering the textured mesh into multiple views to compute diffusion-guided updates.

3.1. Preliminary: Score-Distillation Sampling

  • SDS updates a differentiable 3D representation using a text-conditioned diffusion model’s noise estimate to make rendered images conform to the prompt.The gradient estimate can provide text-conditioned optimization without paired text–3D data.Source: §3.1 · p. 3

3.2. Goal of Paint-it

  • Paint-it aims to produce text-conforming diffuse, roughness-and-metalness, and detail surface-normal maps for a given untextured mesh.Source: §3.2 · pp. 3–4

3.3. DC-PBR: Deep Convolutional PBR Texture Map Re-parameterization

  • DC-PBR replaces direct pixel optimization with a neural U-Net parameterization of PBR texture maps, using convolutional kernels and a fixed random input.The U-Net output contains the diffuse, roughness-and-metalness, and normal maps.Source: §3.3 · p. 4

3.4. Text-driven DC-PBR Optimization

  • At each iteration, Paint-it renders the mesh with its predicted PBR maps, then uses text-guided SDS gradients to update the neural parameters.The method uses differentiable mesh rendering and rasterizes the textured mesh to obtain rendered images.Source: §3.4 · pp. 4–5

4. Analysis: Effect of the Deep Convolutional Re-parameterization for PBR Texture Maps

The analysis shows that deep convolutional re-parameterization schedules texture learning from coarse to fine, unlike pixel optimization, and remains robust to noisy SDS gradients. The authors attribute this behavior to the convolutional architecture’s texture priors.

4.1. Analysis of Fitting Behavior

  • The fitting analysis isolates parameterization from diffusion randomness, comparing pixel-value fitting with a fixed-noise convolutional U-Net and tracking five Fourier frequency bands.The U-Net is randomly initialized and has skip connections.Source: §4.1 · p. 5
  • Pixel optimization fits all frequency bands simultaneously, whereas neural re-parameterization fits low frequencies faster and defers high frequencies.The authors hypothesize this schedule helps prioritize texture content early; pixel optimization jointly fits noise and texture signals.Source: §4.1 · p. 5

4.2. Analysis of Optimization with the SDS Loss

  • The SDS baseline uses total-variation regularization on the diffuse map, while sampled noise, timestep, and camera views yield incoherent gradients.The regularizer compensates for the pixel parameterization’s lack of inductive bias, strengthening the baseline for comparison.Source: §4.2 · pp. 5–6
  • Under SDS, neural re-parameterization learns low-frequency structure before later details, while pixel optimization fits high-frequency noise early.This produces perceptually coarse-to-fine texture synthesis.Source: §4.2 · p. 6, Figure 4 · p. 6
  • The authors postulate that diverse convolution kernels and repeated spatial compositions induce texture priors, including smoothness, behind the frequency scheduling.They connect these architectural properties to filtering high-frequency noise gradients from SDS.Source: §4.2 · p. 6

5. Experiments

Paint-it outperforms recent competing methods on FID and user-study realism ratings, and qualitative comparisons show vivid, realistic, consistent textures. Its experiments span diverse meshes and materials, while ablations distinguish the contributions of PBR representation and neural re-parameterization.

5.1. Qualitative Results

  • Qualitative results span Objaverse objects, RenderPeople humans, and SMAL animals, with varied materials, skin and cloth distinctions, and appearance consistency.Examples include metallic crowns, rough mushrooms, and patterns on animal appearances.Source: §5.1 · p. 6, Figure 5 · p. 7

5.2. Comparison with Competing Methods

  • On identical meshes and prompts, Paint-it produces more vivid, realistic, consistent textures; Fantasia3D also misses the mug’s metallic surface.Text2Tex and TEXTure show multi-view inconsistencies and baked lighting, while Fantasia3D’s PBR materials are less well disentangled.Source: §5.2 · pp. 6, 8, Figure 6 · p. 7, Figure 7 · p. 8
  • Paint-it outperforms recent methods on FID and user-study scores, and alone exceeds a realism rating of four among 30 users.The evaluation uses rendered multi-view meshes from an Objaverse subset.Source: Table 1 · p. 8, §5.2 · p. 8

5.3. Ablation studies

  • Diffuse-only optimization is decent but less realistic without reflection or stereoscopic effects, while pixel optimization with regularization still yields jittered textures.The ablation contrasts these settings with the full PBR representation and neural re-parameterization.Source: §5.3 · p. 8, Figure 8 · p. 8

6. Discussion, Limitation, and Conclusion

Paint-it synthesizes high-fidelity PBR texture maps for meshes across large-scale datasets, including objects, humans, and animals. Its stated limitation is optimization time of approximately 15–30 minutes per mesh.

6. Discussion, Limitation, and Conclusion

  • Paint-it synthesizes high-fidelity PBR texture maps for large-scale mesh datasets, including general objects, humans, and animals.The paper presents this as evidence of the method's performance and potential.Source: §6 · p. 8
  • Optimization takes approximately 15–30 minutes per mesh, which the authors identify as Paint-it's current limitation.They suggest an efficient loss using Consistency models as a possible way to accelerate it.Source: §6 · p. 8
  • The authors identify curating a PBR texture-map dataset and training a feed-forward generative model as a promising future direction.This proposal is based on their synthesized texture maps for large-scale mesh datasets.Source: §6 · p. 8

— Supplementary Material —

The supplementary material provides additional details and results, alongside a brief video introduction and qualitative results. It describes DC-PBR's fixed-noise U-Net reparameterization and its network settings.

— Supplementary Material —

A.1. DC-PBR: Network Design

  • DC-PBR replaces pixel-based PBR texture optimization with a randomly initialized U-Net that maps fixed random noise to the texture map.The input is z ∼ N(0, I) with shape H×W×3; the output has shape H×W×8.Source: §A.1 · p. 12
  • DC-PBR uses a Deep Image Prior-style encoder-decoder with skip connections and five downsampling and upsampling levels.Source: §A.1 · p. 12
  • The reported synthesis settings are a learning rate of 5 · 10^-4 and 1,000 total iterations.Source: §A.1 · p. 12

A.2. Details of SDS Loss

Paint-it computes SDS guidance from multiple rendered views, with extra face-focused views for humans and animals. The supplement explains that view-dependent gradients can be incoherent and relates this noisy guidance to DC-PBR's frequency-scheduled filtering.

A.2. Details of SDS Loss

  • For most meshes, SDS uses four randomly sampled views; human and animal cases use eight views, including four face-focused renders.Face-focused views center the head and use a specified elevation range.Source: §A.2 · p. 12
  • Paint-it uses directional text-prompt engineering to mitigate the Janus problem.Source: §A.2 · p. 12
  • Each iteration shares one sampled noise value and timestep across rendered views, while the timestep range narrows from [0.2, 0.98] to [0.3, 0.5].Source: §A.2 · p. 12
  • Random noise and timestep sampling, along with views exposing different mesh parts, can produce incoherent SDS update directions.Source: §A.2 · p. 12
  • With the same prompt, noise, timestep, and nearly identical adjacent views, the five-view toy example still produced a high-rank gradient matrix.Its singular-value ratios were [2.1868, 1.7937, 1.7838, 1.6689, 1.0000], which the authors interpret as incoherent multi-view guidance.Source: §A.2 · pp. 12–13, Figure S1 · p. 12
  • DC-PBR's frequency-scheduled optimization filters high-frequency noisy SDS signals.Source: §A.2 · p. 13

A.3. Details of Frequency-based Analysis

The frequency analysis divides the diffuse-map FFT into fixed radial bands and tracks their energy through optimization. The supplement also describes a user study in which participants rated rendered results from five methods for realism.

A.3. Details of Frequency-based Analysis

  • The analysis divides the diffuse-map FFT into five fixed, non-overlapping radial frequency bands and sums response magnitudes within each band to track energy.Figure S2 depicts the FFT and the bands, with low frequencies at the center and high frequencies toward the outside.Source: §A.3 · p. 13, Figure S2 · p. 13
  • The plotted band values are average energies, and the DC component occupies less than 1% of the lowest-frequency band.Consequently, the pixel brightness hardly affects that band's energy.Source: §A.3 · p. 13
  • DC-PBR learns low frequencies first, then mid frequencies, while filtering the highest-frequency noise during optimization.Source: §A.3 · p. 13

A.4. Details of User Study

  • Thirty users rated rendered results from five texture methods on a 1–5 realism scale across ten untextured Objaverse meshes.The method order was randomly shuffled for each question; Figure S3 shows the study interface.Source: §A.4 · p. 13, Figure S3 · p. 13

A.5. Details & Discussion about Optimization Time

Paint-it’s optimization takes about 15 minutes for general object meshes and about 30 minutes for more complex humans or animals. DC-PBR performs better than pixel optimization under noisy SDS, and its convolutional parameterization supports smoother, more detailed textures than the tested alternatives.

A.5. Details & Discussion about Optimization Time

  • Without skip connections, U-Net fails to fit the highest frequency band and cannot increase mid-to-high frequencies during text-driven synthesis.With skip connections, DC-PBR can synthesize fine-grained texture details.Source: Figure S5 · p. 14, Figure S6 · p. 14
  • DC-PBR finds a better solution than vanilla pixel optimization under noisy SDS, with no significant optimization-time difference.General object meshes take about 15 minutes; more complex humans or animals take about 30 minutes.Source: §A.5 · pp. 13–14

B.1. More Baseline for PBR Representation

  • DC-PBR yields smoother, more vivid texture results and a better FID score than pixel optimization and feature-grid-plus-MLP baselines.The authors postulate that local hash-grid gradients lack non-local texture smoothness, whereas spatial CNN kernels promote it.Source: §B.1 · p. 14

B.2. Effects of DC-PBR Design Choice

The design-choice study finds that U-Net skip connections matter for synthesizing fine-grained, high-frequency texture details. The ablation’s FID evaluation uses Objaverse meshes and reports that full DC-PBR enhances generated-texture realism.

B.2. Effects of DC-PBR Design Choice

  • The design study compares U-Net with and without skip connections to investigate their effect on DC-PBR optimization with SDS loss.The study uses a randomly initialized U-Net with skip connections as its DC-PBR parameterization.Source: §B.2 · p. 14
  • Without skip connections, U-Net fails to fit high frequencies and produces blurry textures, while U-Net with skip connections enables fine-grained details.The authors conclude that skip connections are responsible for synthesizing high-frequency texture details.Source: §B.2 · p. 15

C.1. More Quantitative Comparison

  • Full DC-PBR enhances the realism of generated textures in an FID ablation using 410 Objaverse meshes for real samples and 50 meshes for generated samples.Table S1 reports the ablation FID scores.Source: §C.1 · p. 15, Table S1 · p. 15

C.2. More Qualitative Results

Paint-it is evaluated on untextured meshes from Objaverse and RenderPeople using manually written prompts or captions generated by multimodal language models. Comparisons report realistic, vivid textures, and the method also supports animating textured rigged meshes while preserving their UV coordinates.

C.2. More Qualitative Results

  • Objaverse and RenderPeople meshes are textured using manually written prompts or captions generated by multimodal language models.The resulting textured meshes are shown in Figures S8 and S9.Source: §C.2 · p. 15

C.3. More Comparison Results

  • Paint-it comparisons on the same meshes and prompts report more realistic and vivid textures than competing methods.Fantasia3D’s per-point PBR materials lack texture smoothness and yield jittering; its SDF re-meshing can also degrade geometry and produce floating artifacts.Source: §C.3 · p. 15

C.4. Paint-it for Animated Meshes

  • Paint-it can texture rigged meshes while preserving their original UV coordinates, enabling animation with motion sequences.The paper demonstrates this workflow using canonical-pose SMPL-X meshes and animated avatars.Source: §C.4 · p. 15

Related papers

Loading 2312.11360v2…