PaperlayerView exact revision on arXiv

Source-linked AI summary

TaoTex: Boosting Texture Detail Fidelity for Native 3D Material Generation

Xiuchao Wu, Shuichang Lai, Jiangjing Lyu, Chengfei Lyu

arXiv:2609.34934v1cs.CV

TL;DR

Detailed texture reconstruction remains difficult for 3D generation, particularly for complex patterns and smooth surfaces. TaoTex combines high-frequency training assets, detail-oriented features and losses, and viewpoint embeddings; the paper reports improved texture detail fidelity for single- and multi-view inputs.

  • Problem

    Existing 3D generation methods struggle to reconstruct complex textures, and native 3D methods often fail on planar or smooth surfaces with sparse geometric cues.

    Source: §1 · pp. 1–2

  • Method

    TaoTex combines high-frequency asset construction, multi-level feature fusion, a latent-to-pixel loss transition, and learnable viewpoint embeddings for native 3D material generation.

    Source: §1 · p. 2

  • Results

    TaoTex significantly outperforms existing approaches in preserving texture details for both single- and multi-view inputs.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The co-designed data, architecture, and training strategies support high-fidelity native 3D material reconstruction from single- or multi-view inputs.

    Source: §5 · p. 9

    Voxel resolution constrains reconstruction of minute text or patterns, while higher resolutions demand prohibitive GPU memory.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent 3D generation models can produce accurate geometries while still struggling to reconstruct detailed textures. We propose a diffusion-based native 3D material generation model TaoTex, which faithfully recovers intricate textures through tailored strategies and improvements. First, we develop a data construction agent to create high-frequency textured 3D assets to bridge the data gap in public datasets. Training with these data significantly enhances the ability of TaoTex to recover challenging details such as text and patterns. Second, we design a multi-level feature fusion (MLFF) module to adaptively integrate local and global features of the conditional input, providing more complete texture cues for the diffusion model and thereby enhancing reconstruction fidelity. To alleviate VAE reconstruction errors, we adopt a latent-to-pixel space loss transition, further improving the pixel-level details and generation quality. Finally, we scale TaoTex to multi-view inputs by incorporating learnable viewpoint embeddings, achieving accurate and consistent material reconstruction across views. Extensive experiments demonstrate that our method significantly outperforms existing approaches in preserving texture details in both single- and multi-view settings.

1 INTRODUCTION

Existing 3D generation methods struggle with detailed textures, especially on smooth surfaces and in multi-view settings. TaoTex addresses these gaps through high-frequency training assets, detail-preserving features and losses, and a multi-view strategy.

1 INTRODUCTION

  • Existing diffusion-based Image-to-3D methods struggle to faithfully reconstruct complex textures on 3D surfaces from input images, limiting applicability to simple-texture objects rather than texture-rich categories.View-space methods can produce ghosting in overlapping regions, require UV-space inpainting for self-occluded areas, and cause projection artifacts through texture-geometry misalignment; 3D-space generation inherently ensures global consistency and avoids self-occlusion.Source: §1 · pp. 1–2
  • Planar and smooth surfaces lack distinctive geometric cues, while public datasets, conditional features, and VAE reconstruction accuracy each constrain detail fidelity.Existing methods also often lack multi-view support or perform poorly with multiple views.Source: §1 · p. 2
  • A data construction agent creates high-frequency textured 3D assets, including text and logo patterns, to train TaoTex on challenging details.The agent is based on the Qwen LLM and creates diverse assets with varied material properties.Source: §1 · p. 2
  • MLFF adaptively combines semantic and local image information, while a latent-to-pixel loss transition mitigates VAE reconstruction errors and improves texture detail fidelity.These strategies target detail-poor conditional features and inaccuracies introduced by the 3D VAE.Source: §1 · p. 2
  • Learnable viewpoint embeddings, symmetric geometry data, and a low-SNR schedule support single- and multi-view inputs with globally consistent, seamless texture reconstruction.The paper reports that this generation framework outperforms existing leading methods.Source: §1 · p. 2

2 RELATED WORK

Related work covers view-space texture synthesis and native 3D texture generation. The former has improved cross-view consistency, while the latter ensures global consistency but still struggles with sharp, high-frequency details.

2.1 VIEW-SPACE TEXTURE GENERATION

  • View-space methods progressed from sequential view synthesis to multi-view diffusion and specialized projection, attention, and material approaches.Sequential view-by-view pipelines can accumulate cross-view inconsistencies; later approaches improve consistency, but high-frequency regions, self-occlusion, and depth boundaries remain challenging.Source: §2.1 · pp. 2–3

2.2 NATIVE 3D TEXTURE GENERATION

  • Native 3D methods use representations including point clouds, octrees, UVs, Gaussian attributes, and structured latent spaces, with trade-offs in resolution, efficiency, or graphics compatibility.These methods provide global consistency and seamless coverage, but existing frameworks remain limited in sharp, high-frequency appearance detail.Source: §2.2 · p. 3

3 METHOD

TaoTex generates native 3D PBR materials from geometry and one or more images. Its method combines high-frequency asset construction, detail-preserving training, and viewpoint-aware multi-view reconstruction.

3 METHOD

  • TaoTex generates albedo, roughness, and metallic materials from input geometry and one or more images using an o-voxel representation.Source: §3 · p. 3

3.1 DATA CONSTRUCTION AGENT

  • The data agent generates meshes with structured UV layouts, synthesizes and checks detailed UV-space textures, then binds verified textures and PBR parameters to create high-frequency assets.For efficiency, the authors reuse category-specific meshes and generate diverse textures over their UV layouts because the task demands more texture diversity than geometric diversity.Source: §3.1 · pp. 3–4

3.2 DETAIL PRESERVATION SCHEME

  • MLFF aggregates shallow and deep DINOv2 features to address detail loss in semantic features, while late-stage pixel-space supervision counters latent compression errors.The pixel-space stage renders albedo, roughness, and metallic maps across viewpoints; input views favor photometric fidelity, while two novel views emphasize perceptual realism.Source: §3.2 · pp. 4–5

3.3 MULTI-VIEW RECONSTRUCTION

  • For multi-view inputs, learnable viewpoint embeddings distinguish view tokens; symmetric HFT assets and low-SNR timestep sampling support learning viewpoint cues and cross-view transitions.The model supports up to four canonical views; embeddings are jointly optimized with the denoiser and fusion module and remain fixed at inference.Source: §3.3 · p. 5

3.4 IMPLEMENTATION DETAILS

  • Training uses Objaverse-XL, TexVerse, and HFT assets rendered from 16 viewpoints, with multi-layer DINOv2 features fused by channel attention and MLPs.Training proceeds in two 100K-step phases, first with latent-space loss and then pixel-space loss, while alternating single- and multi-view inputs.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

TaoTex outperforms competing methods in single- and multi-view texture reconstruction. Ablations support the contributions of high-frequency training data, multi-level feature fusion, pixel-space loss, and viewpoint embeddings.

4.1 COMPARISONS

  • TaoTex substantially surpasses competing methods across single-view metrics and leads all multi-view baselines; Figures 6 and 7 show stronger detail and consistency.Source: §4.1 · p. 7, Table 2 · p. 7, Figure 6 · p. 7
  • In single-view comparisons, TRELLIS.2 lacks high-frequency detail, UniTEX shows fragmentation, Ink3D has geometry-appearance misalignment, and MaterialMVP struggles with fine details.Source: §4.1 · p. 7
  • With four-view conditioning, TaoTex places text and patterns on the correct surfaces with sharp, consistent textures, unlike blurry or misplaced baseline results.Source: §4.1 · p. 7, Figure 7 · p. 8
  • Commercial comparisons favor TaoTex for fine-grained fidelity, material disentanglement, and stability, while other models often show misalignment or ghosting on symmetric objects.Source: §4.1 · pp. 7–8, Figure 8 · p. 9

4.2 ABLATION STUDY

  • Training with HFT data improves recovery of high-frequency details, including clear text across curved surfaces.Source: §4.2 · p. 8
  • MLFF better preserves fine details than semantic features alone; without it, patterns and text become fragmented, blocky, or distorted.Source: §4.2 · p. 8
  • Pixel-space loss alleviates material SC-VAE reconstruction errors and refines fine details, especially tiny text distorted by latent-only optimization.Source: §4.2 · p. 8
  • Without viewpoint embeddings, repetitive patterns on nearly symmetric objects can be misplaced across surfaces, producing distorted and fragmented textures.Source: §4.2 · pp. 8–9

5 CONCLUSION

TaoTex combines co-designed data, architecture, and training strategies for high-fidelity native 3D material reconstruction from single- or multi-view inputs, but minute details remain constrained by voxel resolution.

5 CONCLUSION

  • TaoTex targets high-fidelity native 3D material reconstruction from single- or multi-view inputs through co-designed data, architecture, and training strategies.Source: §5 · p. 9
  • The approach combines high-frequency asset creation, MLFF with pixel-space loss, and viewpoint embeddings for accurate multi-view reconstruction.Source: §5 · p. 9
  • Voxel resolution constrains reconstruction of minute text or patterns, while higher resolution demands prohibitive GPU memory.Source: §5 · p. 9

Related papers

Loading 2609.34934v1…