PaperlayerView exact revision on arXiv

Source-linked AI summary

Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects

Raphael Bensadoun, Yanir Kleiman, Idan Azuri, Omri Harosh, Andrea Vedaldi, Natalia Neverova, Oran Gafni

arXiv:2407.02430v1cs.CVcs.AIcs.GRcs.LG

TL;DR

Text-conditioned texture generation still faces challenges in combining global consistency, quality, and speed. Meta 3D TextureGen conditions a text-to-image model on 3D features in 2D and completes a UV texture map. The authors report state-of-the-art results and an optional enhancement network that reaches 4k resolution.

  • Problem

    Text-to-image-based texture methods have not yet combined global consistency, quality, and speed, which the paper identifies as important for real-world applications.

    Source: §1 · p. 2

  • Method

    Meta 3D TextureGen conditions a text-to-image model on 2D renders of 3D features, then completes a UV texture map from weighted backprojections and shape features.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    The authors report state-of-the-art results with view and shape consistency and text fidelity, and an optional enhancement network that reaches 4k pixel resolution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method brings texture generation closer to an applicable tool for artists and general users by combining consistency, prompt adherence, speed, and high resolution.

    Source: §8 · p. 10

    PBR material maps, including tangent normals, metallic, and roughness, are outside the method and left for future work.

    Source: §6 · p. 10

Abstract

from arXiv · show

The recent availability and adaptability of text-to-image models has sparked a new era in many related domains that benefit from the learned text priors as well as high-quality and fast generation capabilities, one of which is texture generation for 3D objects. Although recent texture generation methods achieve impressive results by using text-to-image networks, the combination of global consistency, quality, and speed, which is crucial for advancing texture generation to real-world applications, remains elusive. To that end, we introduce Meta 3D TextureGen: a new feedforward method comprised of two sequential networks aimed at generating high-quality and globally consistent textures for arbitrary geometries of any complexity degree in less than 20 seconds. Our method achieves state-of-the-art results in quality and speed by conditioning a text-to-image model on 3D semantics in 2D space and fusing them into a complete and high-resolution UV texture map, as demonstrated by extensive qualitative and quantitative evaluations. In addition, we introduce a texture enhancement network that is capable of up-scaling any texture by an arbitrary ratio, producing 4k pixel resolution textures.

1. Introduction

Texture creation for 3D assets is difficult and time-consuming, while text-conditioned texture methods still face consistency, shape-alignment, and speed gaps. Meta 3D TextureGen conditions jointly generated views on 2D renders of 3D features, completes a UV texture, and offers optional 4k enhancement.

1. Introduction

  • Texture creation is difficult and time-consuming for 3D artists, especially for complex shapes, and requires professional skills.Textures let artists control appearance independently of shape and apply to assets made by artists or generated automatically.Source: §1 · p. 2
  • Text-to-image-derived texture methods still face gaps in global consistency and text faithfulness, including seams, incoherence, and the Janus effect.The authors link these issues to frontal-view bias and limited 3D understanding in text-to-image models.Source: §1 · p. 2
  • Prior methods struggle to preserve fine shape detail and align textures when translating 3D shape information into 2D space.The cited approaches condition in UV space on vertex or normal maps, or in image space on depth maps, and have lower quality on highly detailed objects.Source: §1 · p. 2
  • Iterative texture-generation methods can require several to thousands of forward passes and take minutes, limiting quick design iteration and some practical uses.The paper identifies user-generated-content applications and designers’ quick creative iterations as examples.Source: §1 · p. 2
  • The method conditions a fine-tuned text-to-image model on 2D renders of 3D features and generates texture views jointly to improve view and shape consistency and text fidelity.The authors say this effectively eliminates global consistency issues such as the Janus problem.Source: §1 · p. 2
  • An additional network can enhance texture quality and increase resolution by an arbitrary ratio, reaching 4k pixel resolution.The UV-space image-to-image network also fills missing information, removes residual artifacts, and improves effective resolution.Source: §1 · p. 2

2. Related work

The related work spans text-to-image and multi-view generation, followed by texture-generation methods ranging from optimization and direct UV generation to iterative inpainting. Figure 2 outlines Meta 3D TextureGen’s two-stage image-to-UV approach.

2.1. Image generation

  • Text-to-image generators include transformer and pixel- or latent-diffusion approaches; this method starts from a pretrained latent-diffusion model and extends it to the task.Figure 2 shows the method’s geometry-aware text-to-image stage generating multiple texture views from position and normal renders.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2. Multi-view generation

  • Multi-view generation methods produce views for 3D shape reconstruction, but their geometry is not predetermined as it is in texture generation.Their concurrent geometry generation permits geometry changes to improve multi-view consistency.Source: §2.2 · p. 3

2.3. Texture generation

  • Texture methods include optimization, GANs, direct UV diffusion, and iterative inpainting; cited limitations include restricted object domains, slow inference, and global inconsistencies.Point-UV Diffusion requires separate models by object category; iterative methods can take minutes, while SyncMVD retains global-consistency issues and FlashTex uses an SDS optimization stage.Source: §2.3 · pp. 3–4

3. Preliminaries and data processing

The method uses rendered shape features and UV-baked channels as inputs to its stages. Shape renders encode position and normal information, while UV preparation creates a single non-overlapping texture layout.

3. Preliminaries and data processing

  • The method uses rendered shape features and UV-space baked texture maps as representations for its first and second stages, respectively.The rendered channels include position and normal passes used for conditioning.Source: §3 · p. 4, §3.1 · p. 4

3.1. Shape renders

  • Position renders are global rather than view-dependent, while normal renders contain high-frequency details.Figure 3 contrasts these properties with depth renders.Source: Figure 3 · p. 4

3.2. UV maps

  • UV preparation rearranges islands into a single square texture without overlaps, generates a map when needed, and filters out unsuitable layouts.The channels are baked into UV space; the combined pass is the training target, while position and normal passes condition the network.Source: §3.2 · p. 4
  • The qualitative comparison reports higher-quality textures than previous work while preserving local consistency and text-prompt adherence.Figure 4 compares methods on local consistency, quality, and text alignment.Source: Figure 4 · p. 5

4. Method

Meta 3D TextureGen generates a complete, text-matched UV texture in two stages: jointly generate consistent views conditioned on 3D features, then backproject and inpaint them. An optional patch-based network increases texture resolution at flexible ratios.

4.1. Stage I: Generation in image space

  • The first stage generates four views at fixed 90° intervals and a fixed 20° elevation, covering the object from 360°.It uses matching position and normal render grids alongside the text prompt.Source: §4.1 · p. 5

4.1.1 Geometry-aware 2D conditioning.

  • Position renders provide view-independent point correspondence, while normal renders convey orientation and fine geometric details that depth can miss.The method advocates position and normal renders rather than depth-map conditioning.Source: §4.1.1 · pp. 5–6

4.1.2 Multi-view image generation from text

  • A fine-tuned latent-diffusion U-Net generates a grid of four mesh views from text, position and normal grids, and 2D Gaussian noise.The network is conditioned on the diffusion step as well, though that input is omitted from the displayed formulation.Source: §4.1.2 · p. 6

4.2. Stage II: Generation in UV space

  • The second stage takes the blended partial texture and UV-space position and normal maps to inpaint occluded areas and improve texture quality.Its output is the final texture in UV space.Source: §4.2 · p. 6

4.2.1 Backprojection and incidence-based weighted blending

  • Low-incidence views are less reliable, so the method weights backprojected views by the cosine similarity of viewing direction and pixel normal.It blends four views with α = 6; naïvely averaging can introduce artifacts, especially in fine patterns or writing.Source: §4.2.1 · pp. 6–7

4.2.2 UV-space inpainting network

  • Occlusions and pixel-level mapping holes leave the backprojected texture sparse; a UV-space latent-diffusion U-Net uses a mask to inpaint missing regions.The network is conditioned on the partial map and UV position and normal maps.Source: §4.2.2 · p. 7

4. Method

  • For artist-made assets, textures were preferable to all baselines and had better visual fidelity; the two-stage method produces a consistent, text-matched UV map.The first stage generates views conditioned on 3D shape features; the second uses weighted backprojection and inpainting.Source: §4 · p. 5, Table 1 · p. 7

4.2.3 Texture enhancement network

  • The optional patch-based enhancement network flexibly upscales textures, including to 4k, while 2D MultiDiffusion aggregation targets patch seams and pattern or color mismatches.Patch-based prediction addresses GPU memory limits; a tiled-VAE approach supports high-resolution encoding and decoding.Source: §4.2.3 · p. 7

5. Experiments

The method is reported to outperform prior work in user studies and metric comparisons, with experiments examining dataset setup, evaluation, qualitative quality, and component ablations.

5. Experiments

  • User studies and numerical metric comparisons place the method at state-of-the-art among five named baselines.The comparison includes TEXTure, Text2tex, SyncMVD, Paint3D, and Meshy 3.0.Source: §5 · p. 7

5.1.2 Evaluation data

  • Evaluation used 54 CC-licensed Sketchfab objects without a No-AI tag, two Stanford scan objects, and four creative prompts per object.A separate prompt describing each original texture supported FID and KID evaluation; comparisons used 1024 × 1024 textures without enhancement.Source: §5.1.2 · p. 8

5.2.1 User study

  • Overall, users preferred the method over every baseline for both overall quality and artifact reduction.The study included 33 participants and 754 responses; pairwise 360° videos were randomized, and votes were tallied by max-voting.Source: §5.2.1 · p. 8

5.2.2 Metrics

  • FID and KID compared generated and ground-truth textures rendered from 32 evenly spaced viewpoints, while runtime measured complete texture generation for a prompt and mesh.Baseline runtimes were measured on a different GPU setup, so the authors expected the method to be fastest on the same GPU but did not establish that comparison under identical hardware.Source: §5.2.2 · p. 9

5.3. Qualitative comparisons

  • Qualitative comparisons examined text fidelity, global and local consistency, and UV-map usability against prior methods.The authors report that baseline examples show Janus effects and text-alignment difficulties, while their UV maps are cleaner and closer to artist-created maps.Source: Figure 6 · p. 8, §5.3 · p. 9

5.4. Ablation study

  • Without stage I, UV-only conditioning struggled to capture 3D semantics, producing text-alignment problems and seams at UV-fragment boundaries.The issue was especially apparent for non-global prompts.Source: §5.4 · p. 10
  • Without stage II, four generated views often left parts of objects unpainted, and the backprojected texture was lower quality than the full method.The UV-space stage inpaints occluded areas, refines existing texture, and mitigates backprojection artifacts.Source: §5.4 · p. 10
  • Simple averaging of generated views produced blurry areas and lost fine details compared with weighted-incidence blending.Source: §5.4 · p. 10

6. Limitations

The method does not generate PBR material maps, is not real-time, and relies on 3D training data that is comparatively limited in scale.

6. Limitations

  • PBR maps such as tangent normals, metallic, and roughness are outside the method's scope and left for future work.Source: §6 · p. 10
  • Although currently the fastest texture-generation method, it is not real-time or fast enough for every application, and reliance on 3D datasets limits training scale relative to image and video datasets.Source: §6 · p. 10

7. Ethical considerations

The paper notes that many generative-method use cases are outside its scope and calls for examining data, models, uses, safety, risk, bias, and societal impact before real-world implementation.

7. Ethical considerations

  • Many generative-method use cases remain unexamined, so real-world implementation warrants scrutiny of data, models, uses, safety, risk, bias, and societal impact.For texture generation specifically, predefined shapes provide some risk mitigation by constraining users to an existing structure.Source: §7 · p. 10

8. Conclusions

Meta 3D TextureGen aims to bring text-based 3D texturing closer to an applicable tool for artists and general users through consistency, prompt control, speed, and high resolution.

8. Conclusions

  • The method combines global consistency, text-prompt adherence, speed, and 4k resolution for generating diverse textures for gaming, animation, and VR/MR assets.Source: §8 · p. 10

A.1. Training details

The models share a base text-to-image architecture and are fine-tuned with stage-specific losses, training schedules, and inference solvers.

A.1. Training details

  • All models use the same 1024 × 1024 base text-to-image model, with conditioning encoded by its image encoder and concatenated channel-wise.Additional input channels are added as zero-weighted channels to the first convolution layer.Source: §A.1 · p. 14
  • Stage I uses L2 loss, while Stage II and texture enhancement use L1 loss; all models use v-prediction with zero-terminal-SNR rescaling.The authors report that zero-terminal-SNR rescaling benefits training on renderings and UV maps with large background areas.Source: §A.1 · p. 14
  • Training uses learning rate 1e-5 and batch size 256 on 32 H100 GPUs; Stage I and II train for 15k steps each, and enhancement for 28k.Inference uses DDPM with 60 steps for Stages I and II, and DDIM with 50 steps for enhancement.Source: §A.1 · p. 14

A.2. Texture enhancement model training pipeline

The texture enhancement pipeline uses a diffusion-based, patch-based upsampler with tailored degradation and blending choices to produce high-resolution texture maps.

A.2. Texture enhancement model training pipeline

  • The patch-based upsampler uses Multi-Diffusion blending to support arbitrary upscaling ratios without seams or noticeable artifacts between patches.A tiled-VAE approach addresses memory issues when encoding and decoding large images and latent maps.Source: §A.2 · p. 14
  • Unlike Real-ESRGAN’s degradation pipeline, the method omits Unsharp Masking and additive Gaussian noise to avoid associated texture artifacts.The authors describe Real-ESRGAN degradations as producing over-smoothed textures, excessively sharpened edges, high-contrast patterns, and ringing.Source: §A.2 · p. 14
  • The enhancement network is a diffusion model trained on high-quality texture maps, with patch-based processing and tiled-VAE handling for large inputs.Its design draws inspiration from Real-ESRGAN; the passage describes the resulting upsampler as tailored to very high-resolution texture maps.Source: §A.2 · p. 14

B.1. Evaluation dataset

The evaluation materials describe the asset dataset, a user study, real-world environment applications, and visual examples of generated and enhanced textures.

B.1. Evaluation dataset

  • Evaluation meshes come from Sketchfab under CC Attribution, respecting artists’ NoAI requests; bold prompts are reserved for FID and KID calculation.The dataset meshes, artist credits, and prompts are listed in Tables 4–9.Source: §B.1 · p. 14, Table 4 · p. 20, Table 5 · p. 21

B.2. Applications

  • The method is demonstrated building realistic and stylized VR environments, extending evaluation beyond isolated assets on white backgrounds.The authors note that single-asset evaluation captures details and artifacts but not how multiple assets blend in environments.Source: §B.2 · p. 14, Figure 10 · p. 15

B.3. User study

  • The pairwise user study compared the method with five baselines using randomized ordering and included 33 participants with varied 3D experience.Participants comprised 10 3D artists, 18 people with some proficiency, and 5 with no prior background.Source: §B.3 · p. 14, Table 3 · p. 15

C. Visualization details

  • Visualization examples include diverse generations from different seeds for a shared prompt and generated textures in VR environments, excluding the skybox.Figures 11–13 show generated and enhanced UV textures or 3D renders, with closer views illustrating fine detail.Source: Figure 9 · p. 15, Figure 10 · p. 15, Figure 11 · p. 16

Related papers

Loading 2407.02430v1…