Source-linked AI summary
Texturify: Generating Textures on 3D Shape Surfaces
Yawar Siddiqui, Justus Thies, Fangchang Ma, Qi Shan, Matthias Nießner, Angela Dai
TL;DR
High-quality textured 3D data are limited, making texture generation from 3D shape inputs challenging. Texturify learns surface textures from separate collections of 3D shapes and real images, without 3D color supervision or image-shape correspondence. It reports an average 22% FID improvement over the state of the art on car and chair collections.
Problem
High-quality textured 3D shapes are limited, making data-driven texture generation from 3D inputs challenging.
Source: §1 · p. 2Method
Results
22% average FID improvement over the state of the art is reported for texturing ShapeNet chairs and cars.
Source: §1 · p. 2Takeaways & Limitations
Texturify generates high-quality, coherent textures on given mesh surfaces and is presented as usable in standard computer graphics pipelines.
Source: §5 · p. 14Textures can bake in lighting effects because the method does not factor out lighting, and texture resolution is limited by the number of faces at the highest 4-RoSy level.
Source: §4 · pp. 12, 14
Abstract
from arXiv · showhide
Texture cues on 3D objects are key to compelling visual representations, with the possibility to create high visual fidelity with inherent spatial consistency across different views. Since the availability of textured 3D shapes remains very limited, learning a 3D-supervised data-driven method that predicts a texture based on the 3D input is very challenging. We thus propose Texturify, a GAN-based method that leverages a 3D shape dataset of an object class and learns to reproduce the distribution of appearances observed in real images by generating high-quality textures. In particular, our method does not require any 3D color supervision or correspondence between shape geometry and images to learn the texturing of 3D objects. Texturify operates directly on the surface of the 3D objects by introducing face convolutional operators on a hierarchical 4-RoSy parametrization to generate plausible object-specific textures. Employing differentiable rendering and adversarial losses that critique individual views and consistency across views, we effectively learn the high-quality surface texturing distribution from real-world images. Experiments on car and chair shape collections show that our approach outperforms state of the art by an average of 22% in FID score.
1 Introduction
Textured 3D content remains difficult to generate because high-quality textured 3D data are scarce, while existing shape datasets often have simplistic or uniform textures. Texturify learns textures from separate 2D images and 3D shapes, generating them directly on mesh surfaces without 3D color supervision or image-shape correspondence.
1 Introduction
- Textured 3D content remains less explored partly because high-quality textured 3D data are scarce and shape datasets often contain simplistic or uniform textures.Existing volume-based texture approaches can learn inefficiently and produce blurry results because textures are defined on surfaces.Source: §1 · p. 2
- 22% average FID improvement over the state of the art is reported for texturing ShapeNet chairs and cars.The comparison uses real-world imagery for training.Source: §1 · p. 2
- Texturify trains from separate collections of 2D images and 3D shapes without requiring 3D texture supervision or correspondence between images and geometry.The method also does not require semantic part information.Source: §1 · p. 2
- A GAN conditioned on shape geometry and a latent texture code generates textures directly on mesh faces using face convolutions on a 4-way rotationally symmetric quad mesh.The method uses 3D structural neighborhood relations rather than common 2D UV parameterization.Source: §1 · p. 2
2 Related Works
Prior texture methods variously depend on aligned image-mesh data, textured scans, retrieval assets, or implicit and volumetric representations. Texturify instead generates textures for existing untextured shapes through mesh-surface operations.
2 Related Works
- Optimization methods rely on aligned 2D/3D data, and cited completion approaches use partial textured scans or 3D supervision; Texturify instead uses uncoupled shape and image collections.PhotoShape also requires detailed image and object segmentations and cannot produce materials absent from its synthetic asset dataset [41].Source: §2 · p. 3
- LTG [54] uses viewpoint-specific image and silhouette sets with UV atlases, which can introduce seams and struggle with non-convex shapes; Texturify has no explicit viewpoint-partition assumption and operates on mesh faces.LTG uses silhouettes rather than surface features for training, whereas Texturify is geometry-aware.Source: §2 · p. 4
- 3D-aware image-synthesis methods generate views rather than textures for given meshes, and coupled pseudogeometry and color can be difficult to separate for downstream use.Texturify conditions on untextured geometry.Source: §2 · p. 4
- Texturify builds on TextureConv's 4-RoSy surface parameterization, using mesh-face convolutions to generate high-fidelity textures on high-resolution meshes.MeshCNN targets segmentation on smaller or decimated meshes, while TangentConv applies planar convolutions to projected local geometry.Source: §2 · p. 4
3 Method
Texturify learns to generate textures on untextured meshes using 2D images and same-class shape collections, without shape-image correspondence. It processes mesh surfaces hierarchically and uses rendered views to train for realistic, cross-view-consistent textures.
3 Method
- Texturify learns texture generation from 2D image collections and untextured meshes of the same class, without requiring shape-image correspondence or 3D texture supervision.A mesh and latent texture code are inputs to a generator that outputs surface textures.Source: §3 · pp. 4–5, §3.3 · p. 7
3.1 Parameterization
- A hierarchical 4-RoSy quad-mesh parameterization supports seam-free, minimally distorted face processing while preserving geodesic neighborhoods.The representation is precomputed at multiple resolutions for hierarchical feature processing.Source: §3.1 · p. 6, Figure 3 · p. 6
3.2 Surface Operators
- Face convolutions combine each face’s features with its neighbors, while pooling and unpooling aggregate features between mesh-resolution levels.Pooling selects finer-layer faces nearest to each coarser face by Chamfer distance; unpooling maps fine faces to the coarse layer by minimal Chamfer distance.Source: §3.2 · pp. 6–7
3.3 Surface Texture GAN Framework
- A surface U-Net encoder-decoder uses coarse features for global structure and finer features for local detail, then renders textures from multiple views for adversarial training.A StyleGAN2-inspired decoder is conditioned on encoder surface features; image and patch-consistency discriminators critique rendered views and encourage consistency across views.Source: §3.3 · pp. 7–8
3.4 Implementation Details
- Training uses six mesh resolutions, renders each mesh at 512 × 512 from four random viewpoints, and supplies 16 cropped patches to the consistency discriminator.The 16 patches are four 64 × 64 crops from each rendered viewpoint.Source: §3.4 · pp. 8–9
4 Experiments
Experiments on ShapeNet chairs and cars, using real Photoshape and CompCars images, compare Texturify with alternative texture-generation approaches and examine key design choices. Results also characterize its latent texture space and identify resolution and lighting limitations.
Comparison against state of the art
- Texturify produces realistic, detailed chair textures, while 3D-space methods blur textures and UV-based methods show seam artifacts, especially on non-convex chairs.The car examples show diverse textures across sedans, sportscars, SUVs, and hatchbacks.Source: Figure 5 · p. 9, Figure 6 · p. 10, §4 · p. 11
Surface features
- Surface features improve generation over using none; normals combined with curvature provide the most informative descriptor, while positional features and some geometric features have drawbacks.The comparison covers face-centroid position, Laplacian, curvature, discrete fundamental forms, and normals.Source: §4 · p. 11, Figure 7 · p. 12
Patch consistency and viewpoints
- A patch-consistency discriminator improves KID and FID and yields more globally coherent textures; using more training views also reduces cross-view artifacts, with diminishing benefit beyond four.The study compares one, two, and four views per object during an optimization step.Source: §4 · pp. 11–12
Mesh and rendering resolution
- Higher mesh resolution improves fine-scale detail, and higher rendering resolution during training enables more detailed textures.The six-level mesh uses 24,576 faces at its highest level, versus 6,144 for the five-level comparison; still higher resolutions were prohibitive in memory.Source: §4 · pp. 11–12
Learned texture latent space behavior
- For a fixed shape, traversing the learned latent space yields smooth interpolations with valid textures; the same code also gives similar texture styles across different shapes.The paper identifies style transfer as one possible application of this cross-shape consistency.Source: §4 · p. 12, Figure 9 · p. 13, Figure 10 · p. 14
Limitations
- The learned textures can bake in lighting effects, and output resolution is limited by the face count at the highest 4-RoSy level.The authors suggest learned implicit functions or subdivision at those faces could capture higher resolutions.Source: §4 · pp. 12, 14
5 Conclusion
Texturify generates coherent textures directly on mesh surfaces from separate collections of 3D geometry and 2D images, without requiring 2D–3D correspondence or 3D color supervision. The authors report that its surface parameterization and face convolutions outperform state-of-the-art methods quantitatively and qualitatively.
5 Conclusion
- Texturify combines 4-RoSy parameterization, face convolutions, and geometric features to outperform state-of-the-art methods quantitatively and qualitatively.Source: §5 · p. 14
- The method synthesizes high-quality, coherent textures directly on a given mesh surface.Source: §5 · p. 14
- Texturify learns from separate 3D shape and 2D image collections without 2D–3D correspondence or explicit 3D color supervision.Source: §5 · p. 14
A Network Architecture
The architecture encodes surface geometry on a hierarchical mesh and decodes it into textures using face convolutions and style-conditioned synthesis. The authors report that FaceConvs outperform TextureConvs, especially for cars.
A Network Architecture
- Zero-padding fixes neighborhood size at singularity vertices, which comprise 0.89% of chair vertices and 1.95% of car vertices.Source: A Network Architecture · p. 19
- The encoder processes face normals and curvature at the finest hierarchy level, then pools features across levels for a U-Net-style decoder.Source: Figure 11 · p. 19
- FaceResNet blocks replace 2D convolutions with FaceConv so feature extraction operates on the mesh surface.Source: Figure 11 · p. 19
- The decoder upsamples learned noise from a 24-face coarse mesh level, combines it with encoder features and style codes, and progressively generates RGB textures.Source: Figure 12 · p. 20
- FaceConvs produce significantly better geometry-aware texture synthesis than TextureConvs, especially on cars.Source: A Network Architecture · p. 19, Table 3 · p. 20
B Baseline Methods
The baseline comparisons adapt texture-generation architectures to different representations, including 3D grids, UV atlases, and triplanes. Their training and rendering setups vary by method.
B Baseline Methods
- TextureFields training was unstable with only real-world images, so the baseline mixed real images and ShapeNet textured renders with probability p = 0.5.Source: B Baseline Methods · p. 20
- The reported comparison covers state-of-the-art texture-generation approaches on ShapeNet chairs and cars trained on real-world 2D images.Source: Table 4 · p. 21
- The SPSG-inspired baseline replaces surface operations with 3D convolutions and trilinear resampling, predicting an RGB color grid rendered using the shape’s TSDF.Source: B Baseline Methods · p. 21
- The UV baseline uses six-view normal and curvature atlases, 2D ResNet features, and a conditioned StyleGAN decoder to predict a single texture atlas.Source: B Baseline Methods · pp. 21–22
- The EG3D-inspired baseline generates a triplane and TSDF feature grid, then decodes sampled features into mesh-face colors for differentiable rendering.Source: B Baseline Methods · p. 22, Figure 14 · p. 22
C Discussion & Outlook
Texturify produces semantically consistent textures for broad text prompts where the compared Text2Mesh approach can place smaller images of objects on the surface. The discussion also identifies baked-in lighting effects as a limitation.
C Discussion & Outlook
- Text-based synthesis optimizes the latent code of a pretrained Texturify generator to maximize CLIP scores between the query and rendered views.Source: C Discussion & Outlook · p. 22
- For broad prompts such as “brown chair” or “blue sedan car,” Texturify produces semantically consistent textures, unlike Text2Mesh’s smaller object imagery.Source: C Discussion & Outlook · pp. 22, 24
- The paper identifies disentangling lighting from diffuse texture and combining per-face colors with local texture MLPs as future directions.Source: C Discussion & Outlook · p. 24
- Because the method does not model illumination, generated textures can bake in training-image lighting effects such as shadows or specular highlights.Source: Figure 18 · p. 24, C Discussion & Outlook · p. 24