PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Native 3D Texture Generation Necessarily Require 3D Assets for Training?

Jiangshan Wang, Zeqiang Lai, Jiayi Guo, Xin Yang, Xin Huang, Jiarui Chen, Ziheng Ouyang, Chunchao Guo, Xiangyu Yue

arXiv:2609.34621v1cs.CV

TL;DR

High-quality 3D assets are scarce and costly to acquire, motivating the question of whether native 3D texture generation can be trained from images instead. Tex-Zero constructs 3D training samples from 2D images and trains its VAE and DiT without real 3D assets. It generates high-fidelity textures with fine-grained details and outperforms representative baselines trained on real textured 3D data.

  • Problem

    High-quality 3D assets for training native texture generation are scarce and costly to acquire, while high-quality 2D images are abundant.

    Source: §1 · pp. 1–2

  • Method

    Tex-Zero transforms 2D images into geometrically diverse 3D samples and trains its VAE and DiT on image-derived data, encoding conditioning views and target textures in a unified latent space.

    Source: §1 · pp. 2–3

  • Results

    Tex-Zero generates high-quality textures with fine-grained details and outperforms several representative baselines despite training without textured 3D assets.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The results suggest a new paradigm for constructing training data for 3D texture generation from abundant 2D images.

    Source: §1 · p. 3, §5 · p. 11

Abstract

from arXiv · show

Native 3D texture generation synthesizes colors directly in 3D space for a given geometry, conditioned on multi-view reference images. It is generally believed that training such models requires large-scale, high-quality real 3D asset data, whose acquisition remains a long-standing and challenging problem. In this work, we propose Tex-Zero, demonstrating that a high-fidelity native 3D texture generation framework can be trained without 3D assets. Our key observation is that only high-quality and fine-grained color information is essential for 3D texture training, while the required geometric information is less critical and can be manually constructed rather than obtained from real 3D assets. This finding makes it possible to transform abundant, high-quality 2D images into effective training samples for 3D texture generation. Specifically, we convert high-quality 2D images into 3D training samples by representing each image as a plane in 3D space and applying patch-wise random rotations and aggregation to construct complex geometric structures. Using these constructed image data, we train the Tex-Zero VAE, which can reconstruct real 3D assets with high quality despite never observing them during training. Building upon the Tex-Zero VAE, we train the Tex-Zero DiT also exclusively on the constructed image data, where the conditioning 2D multi-view images are transformed into planes in 3D space and also encoded by the Tex-Zero VAE, thereby reducing the representation gap and improving generation quality. Extensive experiments show that Tex-Zero generates high-fidelity 3D textures with fine-grained details solely using images as training data, offering a promising perspective on the data paradigm for scaling 3D texture generation.

1 Introduction

Tex-Zero trains native 3D texture generation models using only 2D images, addressing the scarcity and acquisition cost of high-quality 3D assets. Its image-derived samples preserve detailed color information while using constructed geometry, and support high-quality reconstruction and texture generation on real 3D assets.

1 Introduction

  • Native 3D texture generation predicts colors directly in 3D space, avoiding projection and fusion errors associated with conventional 2D-view texturing.It aims to preserve geometric alignment and global consistency from multiview references.Source: §1 · p. 1
  • The approach targets a data bottleneck: high-quality 3D assets are scarce and costly to acquire, while high-quality 2D images are abundant.Both image pixels and 3D surface points assign colors to spatial locations.Source: §1 · pp. 1–2
  • Tex-Zero converts 2D images into 3D training samples by rotating image patches independently and aggregating them to create complex local geometry and occlusion patterns.Patch transformations retain each patch’s detailed texture while aggregation reduces floaters.Source: §1 · p. 2
  • The authors identify fine-grained color information as essential for training, while the geometry can be manually constructed without semantically meaningful 3D assets.The resulting image-derived samples train both the 3D VAE and DiT.Source: §1 · p. 2
  • The image-trained Tex-Zero VAE faithfully reconstructs real 3D assets and preserves fine-grained details when reconstructing 2D images represented as 3D planes.These results provide a shared latent space for 2D images and 3D textures; existing 3D-trained VAEs often produce blurry, low-quality 2D reconstructions.Source: §1 · p. 2
  • Experiments show Tex-Zero transfers knowledge learned on constructed data to real 3D assets at inference, generating high-fidelity 3D textures and outperforming representative baselines trained on real textured 3D data.Conditioning multi-view images and target 3D textures are represented in a unified latent space.Source: §1 · p. 3

2 Related Works

Prior work includes view-based texturing, native geometry-aligned texture generation, and synthetic-data approaches to 3D learning. The supplied related-work discussion identifies native 3D texture training with synthetic data as an unexplored area.

2 Related Works

  • View-based texturing projects and fuses multiview images onto geometry, while native methods generate textures in geometry-aligned representations but rely heavily on high-quality 3D training assets.Native representations include UV maps, octree-aligned 3D Gaussians, continuous texture functions, native surface colors, and structured geometry–appearance latents.Source: §2 · p. 3
  • Synthetic-data studies show that carefully designed data can generalize to real-world inputs without fully matching their semantic distribution.Examples include procedural textured shapes, synthetic scenes, and generated multiview examples for 3D reconstruction.Source: §2 · p. 3
  • Using synthetic data to train native 3D texture generation models remains largely unexplored in the cited related-work discussion.Source: §2 · p. 3

3 Methods

Tex-Zero trains a native 3D texture framework solely on image-derived samples and transfers it to real assets at inference. Its pipeline constructs 3D training data from 2D images, trains a VAE for textures and image conditions, then uses a DiT conditioned on image and geometry features.

3.1 Preliminary

  • The task predicts colors at geometry-defined 3D locations from geometry and reference images, with conventional training using textured assets and rendered multiview conditions.Geometry includes point positions, normals, and sparse-voxel coordinates; texture is represented by normalized RGB colors.Source: §3.1 · p. 4

3.2 From 2D Image to 3D Texture Supervision

  • Each image becomes a colored 3D plane, whose non-overlapping patches are independently rotated while their point colors remain unchanged.Pixel coordinates determine 3D positions, preserving the original image’s spatial and color information.Source: §3.2 · p. 4
  • Aggregating rotated patches in a bounded region creates overlap and occlusions; voxelized geometry-color samples train the VAE, while rendered canonical views complete DiT training tuples.If points share a voxel, only one is retained; the full tuple contains geometry, colors, and conditioning images with foreground masks.Source: §3.2 · pp. 4–5

3.3 Tex-Zero VAE

  • VAE training begins with randomly rotated single planes before switching to independently rotated patches; the warm-up supports stable optimization, without which training fails to converge.The full pipeline then exposes the VAE to varied surface orientations, discontinuities, and visibility patterns.Source: §3.3 · p. 5
  • The Tex-Zero VAE transfers from image-only training to real 3D texture reconstruction and also reconstructs plane-represented 2D images with fine detail.Encoding both image conditions and target textures with the VAE provides a unified latent space that bridges the modalities.Source: §3.3 · pp. 5–6, Figure 3 · p. 6

3.4 Tex-Zero DiT

  • The Tex-Zero DiT generates surface colors from noisy texture latents, encoded multiview conditions, and geometry features, then the VAE decoder maps generated latents to target-surface RGB colors.Training uses only constructed image-derived samples and varies the subset of conditioning views; inference uses real geometry and multiview images.Source: §3.4 · pp. 6–7
  • The DiT encodes six canonical-view image planes with the frozen VAE and combines their tokens with geometry features; joint rotary embeddings distinguish spatial positions and view identity.Training randomly varies the number and combination of views, including occluded regions that must be predicted from visible context.Source: §3.4 · pp. 6–7

4 Experiments

Tex-Zero is evaluated as an image-trained system for reconstructing textures and generating native 3D textures. Experiments also test whether combining 2D and 3D data helps, and which preprocessing and conditioning choices matter.

4.1 Experimental setup

  • The VAE and DiT are trained on large-scale public image datasets; evaluation compares texture quality on rendered views and point clouds against representative methods and 3D-trained models.The image datasets comprise approximately 11.1 million images; rendered-view metrics are PSNR, SSIM, and LPIPS, with PSNR-PC used for point clouds.Source: §4.1 · p. 7

4.2 Reconstruction

  • The image-trained VAE achieves competitive reconstruction on real 3D assets and consistently outperforms 3D-trained counterparts and other baselines on 2D image reconstruction.Figure 4 shows fine-grained details in reconstructed real assets.Source: Figure 4 · p. 7, Table 1 · p. 8, §4.2 · p. 8

4.3 Generation

  • Tex-Zero consistently outperforms NaTex and TRELLIS.2 across all reported metrics in six-view and front-view evaluations, with finer details and more faithful reference-appearance transfer.The comparison is reported in Table 2 and Figure 5.Source: Table 2 · p. 8, §4.3 · p. 8

4.4 2D & 3D Joint Training

  • Combining 2D images with textured 3D assets improves VAE reconstruction over 3D-only training, while mixed-data VAE training also improves DiT generation.The reported benefit applies when textured 3D training data are available.Source: §4.4 · p. 8, Table 3 · p. 9

4.5 Ablation Study

  • Preprocessing ablations show that fixed planes fail to transfer to real assets, global rotation enables meaningful 3D reconstruction, and patch-wise rotation plus aggregation further improves reconstruction.With 16 aggregated patches, the VAE achieves the best results across all metrics.Source: §4.5 · pp. 9–10, Table 4 · p. 10
  • The complete pipeline, using a shared image-trained VAE to encode conditioning images, achieves the best quantitative generation results and improves detail fidelity over alternative conditioning strategies.Image-trained DINO or an independently trained VAE loses fine-grained details; a 3D-trained VAE reduces the representation gap but produces blurry textures.Source: Table 5 · p. 10, §4.5 · p. 10, Figure 7 · p. 10

5 Conclusion

Tex-Zero trains its VAE and DiT without textured 3D assets by converting 2D images into geometrically diverse 3D samples. Experiments show high-quality texture generation with fine-grained details and results above representative baselines.

5 Conclusion

  • Tex-Zero is a high-fidelity native 3D texture generation framework trained without textured 3D assets.Both its VAE and DiT are trained entirely on large-scale image data converted into geometrically diverse 3D samples.Source: §5 · p. 11
  • The Tex-Zero VAE reconstructs both real 3D textures and 2D images with high performance, while the DiT uses a unified latent space for multi-view images and target textures.The DiT is built upon the Tex-Zero VAE and trained solely from images.Source: §5 · p. 11
  • Experiments show that Tex-Zero generates high-quality textures with fine-grained details and outperforms several representative baselines.The authors suggest image-based training-data construction as a new paradigm for 3D texture generation.Source: §5 · p. 11

A More Explanations on Data Construction

The paper argues that textured 3D assets serve separable appearance and geometry roles, neither of which requires the model to generate geometry itself.

A More Explanations on Data Construction

  • A textured asset supplies appearance through color and organizes it in 3D through geometry; the VAE represents colors over geometry-defined sparse neighborhoods, while the DiT learns how geometry organizes and partially reveals appearance across views.Neither module is trained to generate geometry itself.Source: A More Explanations on Data Construction · p. 14
  • Realistic object geometry is not intrinsically required for training because high-quality images can provide appearance supervision and constructed geometries can provide directional, spatial, and visibility interactions.Patch-wise transformations and spatial aggregation instantiate these task-relevant interactions without semantically meaningful 3D assets.Source: A More Explanations on Data Construction · p. 14

A.1 Converting Image Patches into 3D Training Samples

Each source image is represented as colored points on a plane in 3D, then its patches are transformed while preserving their colors and internal appearance. Voxelization turns the resulting points into sparse 3D training support.

A.1 Converting Image Patches into 3D Training Samples

  • Each image pixel becomes a colored point on the plane z = 0, with its pixel coordinates, surface normal, and RGB value defining its 3D representation.The points initially share the unit normal.Source: §A.1 · p. 14
  • All points within a patch receive the same rotation and translation, changing the patch’s position and orientation without changing its colors.The transformation also preserves internal spatial structure, edges, and fine-grained texture patterns.Source: §A.1 · p. 14
  • After voxelization, the transformed points form sparse 3D support with a defined color at every occupied location, making them usable as 3D VAE training data.Source: §A.1 · p. 14

A.2 Why the Data Can Train the 3D VAE

The 3D VAE reconstructs surface colors on a supplied geometric support rather than predicting geometry. Image-derived supports are transformed to provide the directional and spatial diversity needed to train it without semantically meaningful object shapes.

A.2 Why the Data Can Train the 3D VAE

  • The 3D VAE encodes and reconstructs surface colors on a given support; geometry supplies occupied voxels and neighborhoods rather than being predicted.Source: §A.2 · p. 14
  • Training only on a fixed plane leaves 3D convolution directions without supervision, while randomly rotating planes changes which voxel offsets are activated.Source: §A.2 · p. 15
  • Global rotation changes a plane’s orientation but leaves each sample coplanar; independently rotating patches removes this restriction and allows multiple local surface orientations.Source: §A.2 · p. 15
  • Spatial aggregation brings differently oriented patches within local receptive fields, allowing 3D features to depend jointly on surfaces rather than isolated planes.The construction combines global rotation for directional supervision, patch-wise rotation for varied orientations, and aggregation for interacting neighborhoods.Source: §A.2 · p. 15

A.3 Why the Data Can Train the DiT

The constructed data expose the DiT to geometry-organized multi-view observations and completion of unseen surface regions. These supervision tasks mirror the inference problem without requiring semantically realistic training shapes.

A.3 Why the Data Can Train the DiT

  • The DiT learns to map multi-view observations with known viewpoints and given geometry to a complete surface texture.Source: §A.3 · pp. 15–16
  • Patch arrangements and viewpoints vary projected regions, depth ordering, and occlusions, giving the DiT examples of how geometry organizes and relates multi-view appearance.Source: §A.3 · p. 16
  • Because each patch’s complete appearance is preserved while only geometry-dependent portions are visible in conditioning views, the data train geometry-conditioned completion from visible context.When hidden content is ambiguous, the target is its conditional distribution rather than deterministic recovery.Source: §A.3 · pp. 16–17
  • The constructed data jointly supervise multi-view integration and completion of unseen surface appearance, the two components identified as the real-asset inference problem.Source: §A.3 · p. 17

A.4 Conclusion.

  • Semantically realistic object shapes are unnecessary because the construction preserves within-patch visible–hidden appearance relationships while supplying 3D supports and observation mechanisms required for real texturing.Source: §A.4 · p. 17, §A.3 · p. 17

B Implementation Details

The appendix covers data representation, VAE architecture, DiT conditioning, and optimization; unless otherwise specified, network settings refer to the f16c16 VAE and its corresponding DiT.

B Implementation Details

B.1 Data Representation and Preprocessing

The appendix describes how image patches are converted into sparse 3D color samples, how the VAE and DiT encode and train on them, and how textures are sampled at inference. It also specifies the evaluation setups for VAE reconstruction and multi-view image-to-texture generation.

B.1 Data Representation and Preprocessing

  • Images become 3D samples through independently rotated patches placed in overlapping arrangements, then voxelized without averaging colors; overlap does not guarantee physically connected or intersection-free surfaces.Samples with fewer than 100,000 retained points are rejected.Source: §B.1 · p. 18

B.2 Tex-Zero VAE

  • The VAE reconstructs RGB colors at geometry-defined occupied voxels, using pointwise color error and image-space perceptual loss on reassembled images.The image-space loss compares prediction and reference at the same rendered coordinates to isolate appearance reconstruction from voxelization differences.Source: §B.2 · pp. 18–19

B.3 Tex-Zero DiT

  • The frozen VAE separately encodes target colors and normals, while six rendered conditioning views supply color latents whose positional coordinates encode image layout and viewing direction.The DiT predicts texture-latent velocity from noisy target latents, image conditions, and normal features.Source: §B.3 · pp. 20–21

B.4 Optimization and Sampling

  • Both models use AdamW with warm-up followed by cosine decay, while variable occupied-voxel counts are handled with point-cloud-size-bucketed data loading.Source: §B.4 · p. 21
  • At inference, image and normal conditions stay fixed as the texture latent evolves from Gaussian noise, with classifier-free guidance and 49 Euler updates before decoding.The sampler uses 50 uniformly spaced time points over the noise-to-data interval.Source: §B.4 · p. 22

C Experiment Settings

The experiments assess VAE reconstruction against several pretrained and 3D-trained baselines, then evaluate multi-view image-to-3D texture transfer on an internal test set. The DiT evaluation focuses on transferring appearance information onto target geometry.

C Experiment Settings

  • VAE reconstruction is evaluated on 200 internal real 3D assets with fine-grained appearance and on ImageNet validation images, against FLUX VAE, TRELLIS.2, NaTex, and a VAE trained on about 1M 3D assets.The 3D-trained comparison uses the same VAE architecture and otherwise unchanged training settings.Source: C Experiment Settings · p. 22
  • Multi-view texture transfer is evaluated on 160 internal shape-and-view samples by comparing generated-asset renderings with corresponding ground-truth images.The setup compares NaTex and TRELLIS, whose released checkpoint supports only one input view, while focusing on appearance transfer to the target geometry.Source: C Experiment Settings · p. 22
Loading 2609.34621v1…