PaperlayerView exact revision on arXiv

Source-linked AI summary

Mesh2Tex: Generating Mesh Textures from Image Queries

Alexey Bokhovkin, Shubham Tulsiani, Angela Dai

arXiv:2304.05868v1cs.CV

TL;DR

Realistic texturing remains challenging when object geometry reconstructed from images is inexact. Mesh2Tex learns a surface-based texture manifold from uncorrelated meshes and photorealistic images, then optimizes it to match a single RGB query. The paper reports improved performance over prior work in both unconditional and image-conditioned texture generation.

  • Problem

    Geometry generation and reconstruction have advanced, but generating realistic textures for downstream 3D rendering remains challenging, particularly when reconstructed geometry is inexact.

    Source: §1 · p. 2

  • Method

    Mesh2Tex learns a hybrid mesh-neural-field texture manifold from uncorrelated object meshes and photorealistic images, then optimizes within it to match a single RGB query.

    Source: §1 · p. 2

  • Results

    Mesh2Tex outperforms the state of the art in both unconditional and image-conditioned texture generation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method produces realistic textures on a 3D shape that perceptually match image queries without requiring exact geometry matches.

    Source: §5 · p. 8

    Mesh2Tex does not explicitly model semantics, which may lead to texture distortions in semantically meaningful areas.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

Remarkable advances have been achieved recently in learning neural representations that characterize object geometry, while generating textured objects suitable for downstream applications and 3D rendering remains at an early stage. In particular, reconstructing textured geometry from images of real objects is a significant challenge -- reconstructed geometry is often inexact, making realistic texturing a significant challenge. We present Mesh2Tex, which learns a realistic object texture manifold from uncorrelated collections of 3D object geometry and photorealistic RGB images, by leveraging a hybrid mesh-neural-field texture representation. Our texture representation enables compact encoding of high-resolution textures as a neural field in the barycentric coordinate system of the mesh faces. The learned texture manifold enables effective navigation to generate an object texture for a given 3D object geometry that matches to an input RGB image, which maintains robustness even under challenging real-world scenarios where the mesh geometry approximates an inexact match to the underlying geometry in the RGB image. Mesh2Tex can effectively generate realistic object textures for an object mesh to match real images observations towards digitization of real environments, significantly improving over previous state of the art.

1. Introduction

Mesh2Tex addresses the gap between geometry-focused 3D generation and the need for realistic texturing. It learns a surface-tied texture manifold from uncorrelated meshes and images, then optimizes textures to match single-image queries, outperforming prior work in both generation settings.

1. Introduction

  • Geometry-focused 3D generation and single-view reconstruction do not by themselves provide the high-quality texturing needed for realistic environments.The introduction frames realistic texture generation as complementary to geometry generation and reconstruction.Source: §1 · p. 2
  • Mesh2Tex outperforms the state of the art in both unconditional and image-conditioned texture generation.The approach targets high-fidelity texture generation for known object geometry and image-based synthesis from a single RGB observation.Source: §1 · p. 2
  • A hybrid mesh-neural-field representation ties textures to mesh-face barycentric coordinates and supports high-resolution surface texturing.The learned manifold is trained from uncorrelated collections of object meshes and photorealistic images using adversarial differentiable rendering.Source: §1 · p. 2
  • Mesh2Tex optimizes within its learned texture manifold to match a single real-world RGB image, using patch-based perceptual optimization guided by dense correspondences.The method is designed to remain robust across differing views and geometry mismatches.Source: §1 · p. 2

2. Related Work

Prior work includes optimization from aligned views and learned texture generation across several representations. Mesh2Tex instead combines surface-based texture modeling with image-query transfer designed for pose and geometry misalignment.

2. Related Work

  • Optimization-based texturing can produce realistic results, but the cited approaches require aligned 2D/3D data and cannot texture unobserved regions.The section contrasts global RGB-to-geometry mapping with adversarial differentiable-rendering optimization [15].Source: §2 · p. 2
  • Learned texturing spans material retrieval, voxel colors, UV textures, per-face mesh colors, and implicit texture fields.Mesh2Tex models texture on the mesh surface while using a shared implicit field per face.Source: §2 · p. 2
  • Image-guided 3D work includes text-based texture optimization, while Mesh2Tex targets image-based transfer across shapes despite pose and geometry misalignment.The paper motivates image queries as practical because images are often easy to capture or find and visually specific.Source: §2 · p. 3

3. Method

Mesh2Tex learns a texture manifold on a hybrid mesh-field representation and uses it to generate textures or transfer texture from a single RGB query. Its method combines face-wise features and a shared neural field with pose estimation, correspondence-guided style losses, and optional surface-feature refinement.

3.1. Overview

  • The overview learns a texture manifold conditioned on a shape mesh, then traverses it to match an image query using pose estimates, normalized object coordinates, and global and local patch style losses.The approach is designed for arbitrary query viewpoints and inexact geometric matches.Source: §3.1 · p. 3

3.2. Mesh-Field Texture Generation

  • A mesh encoder-decoder predicts per-face features from geometric face attributes and a latent texture code, while a shared neural field maps face coordinates and features to colors at arbitrary resolution.The field is sampled on mesh faces; its output is supervised through adversarial differentiable rendering against photorealistic 2D images.Source: §3.2 · pp. 3–4
  • Face convolutions provide coarse features, which the neural field refines into surface colors; the generator is trained adversarially through differentiable rendering.A rendered feature-map proxy loss also supervises the representation alongside the implicit-field rendering loss.Source: Figure 3 · p. 4, §3.2 · p. 4

3.3. Texture Transfer from a Single Image

  • For a single-image query, the method estimates coarse azimuth and elevation and predicts dense normalized object coordinates to guide texture transfer.The pose network classifies angles into 12 azimuth and 5 elevation bins; the correspondence network predicts object coordinates from the image.Source: §3.3 · p. 4
  • Texture transfer optimizes latent texture codes with global and local style losses, matching rendered images and NOC-corresponding patches across multiple scales and VGG layers.Corresponding patches are selected by nearest NOC values, accommodating geometry mismatch or inconsistent pose prediction.Source: §3.3 · pp. 4–5
  • After latent-code optimization, unfreezing the last two face-convolution layers and optimizing surface features with the patch loss can capture more local image detail.At test time, optimization uses 100 latent-code iterations followed by 300 refined-weight iterations, taking about 400 seconds.Source: §3.3 · p. 5

4. Results

Mesh2Tex evaluates unconditional texture generation and single-image texture transfer on synthetic and real data, including settings with unknown alignment or differing geometry. Its mesh-field representation and NOC-guided optimization produce detailed textures and remain effective under several of these mismatches.

4.1. Unconditional Texture Generation

  • Mesh2Tex’s hybrid mesh-field representation generates richer textures with finer-scale detail than state-of-the-art baselines in unconditional generation.The comparison covers ShapeNet meshes; Figure 5 highlights fine detail, particularly on complex car textures.Source: Figure 5 · p. 5, §4.1 · p. 6, Table 1 · p. 6

4.2.1 Synthetic Image Query Experiments

  • For aligned single-image queries on exact geometry, optimization through the texture manifold produces more detailed textures that better match query-image texture changes than baselines.Tables 2 and 3 evaluate distribution quality and perceptual match, respectively.Source: §4.2.1 · p. 6, Table 2 · p. 7, Table 3 · p. 8
  • With exact geometry but unknown image alignment, NOC guidance makes Mesh2Tex more robust than baselines, producing more plausible textures despite alignment-related performance degradation.GET3D is not evaluated with LPIPS in this setting because it can change geometry.Source: §4.2.1 · p. 6, Table 5 · p. 8
  • For synthetic queries transferred to different geometry, all methods lose performance, but Mesh2Tex yields more coherent, detailed textures on novel shapes than baselines.The approach uses NOC-guided style optimization on its mesh-field representation.Source: §4.2.1 · p. 6, Table 6 · p. 8, Table 7 · p. 8

4.2.2 Real Image Query Experiments

  • With real images and closely matching but inexact geometry and alignment, NOC-guided optimization produces query-consistent textures while baselines struggle with incomplete, cluttered objects.Real-image view-dependent effects such as reflections and specular highlights make evaluation challenging.Source: §4.2.2 · p. 6, Table 2 · p. 7, Table 3 · p. 8
  • For real-image queries with close geometry but unknown alignment, optimization through the texture manifold generates more perceptually representative textures.The evaluation includes qualitative results in Figure 7 and quantitative measures in Tables 4 and 5.Source: §4.2.2 · p. 7, Figure 7 · p. 7, Table 4 · p. 8
  • For real-image transfer to arbitrary same-class shapes, the mesh-field representation retains detail and NOC-guided patch loss improves robustness during optimization.The evaluated queries include ScanNet and CompCars images, with qualitative results in Figure 8.Source: §4.2.2 · p. 7, Figure 8 · p. 7, Table 6 · p. 8

4.3. Ablations

  • Ablations associate finer complex-texture detail with the mesh-field representation, while patch-style loss, NOC guidance, and surface-feature optimization improve texture quality in different ways.Patch-style loss captures local detail and robustness to query mismatch; NOC guidance establishes coarse correspondence, and surface-feature optimization slightly improves local detail. Mesh2Tex does not explicitly model semantics, which can distort semantically meaningful areas.Source: §4.3 · pp. 7–8

5. Conclusion

Mesh2Tex learns a hybrid mesh-field texture manifold for realistic texturing from real-world imagery. It supports single-image texture transfer without requiring an exact geometric match, with potential use of easily captured images as guidance for holistic shape texturing.

5. Conclusion

  • Mesh2Tex learns a hybrid mesh-field texture manifold to generate realistic, high-quality textures on shape geometry from real-world imagery.Its NOC-guided local and global style loss supports texture optimization on shapes that do not require exact geometric matches to RGB image queries.Source: §5 · p. 8
  • Test-time optimization transfers texture from a single image query, while easy-to-capture images can guide holistic shape texturing for content creation.Source: §5 · pp. 8–9

A. Additional Generation Results

Additional qualitative results compare unconditional texturing on ShapeNet meshes. Mesh2Tex produces more realistic textures than the compared state-of-the-art approaches, while GET3D still exhibits artifacts even with dense per-object training views.

A. Additional Generation Results

B. Additional Results for Texture Transfer from a Single RGB Image

Additional texture-transfer results cover synthetic and real-world image queries. The examples show transfer across query alignment conditions and arbitrary same-class geometry, as well as real-world cases with differing geometry, pose, and view-dependent effects.

B. Additional Results for Texture Transfer from a Single RGB Image

C. Qualitative Ablation Visualizations

The qualitative visualizations compare unconditional ShapeNet texturing and image-guided texture transfer, including aligned and unaligned queries and transfer to arbitrary same-class geometry. The ablation results highlight texture-mapping artifacts and lost detail when optimization components are removed.

C. Qualitative Ablation Visualizations

  • Without patch loss or NOC guidance, unaligned-query texture transfer produces messy textures with stripe artifacts and disordered mapping; optimizing only latent codes loses details.Source: C. Qualitative Ablation Visualizations · p. 11
  • Unconditional ShapeNet results show Mesh2Tex textures as more realistic and detailed than the compared state-of-the-art results.Source: Figure 9 · p. 11
  • Figures 10–12 visualize query-based texture optimization with aligned and unaligned ShapeNet images, including transfer to arbitrary same-class geometry.Source: Figure 10 · p. 12, Figure 11 · p. 13, Figure 12 · p. 14

D. Network Architecture

Mesh2Tex combines a surface encoder, generator, and local neural field to produce mesh textures. Image-query transfer also uses pose and NOC predictors; the supplied examples cover aligned, unaligned, and arbitrary same-class geometry.

D. Network Architecture

  • The generator produces coarse per-face RGB values and features for the local neural field, while the discriminator follows Texturify.The surface encoder, generator, and neural field architectures are shown in Figs. 17 and 18.Source: D. Network Architecture · p. 14
  • For each surface point, Ψ derives its feature by barycentrically averaging vertex features computed from incident face features.The barycentric weights are the areas of the three triangles formed by the point and face vertices.Source: D. Network Architecture · p. 14
  • The real-world transfer examples compare aligned images with close ShapeNet geometry, unaligned images with similar geometry, and unaligned images with arbitrary same-class geometry.The arbitrary-geometry examples include ScanNet and CompCars query images.Source: Figure 13 · p. 15, Figure 14 · p. 16, Figure 15 · p. 17
  • A fixed, learnable auxiliary latent vector of size 512 is concatenated with surface and generator features to produce the surface-point color.The authors report that this auxiliary latent enhanced high-resolution texture consistency.Source: D. Network Architecture · p. 17
  • Texture transfer uses a ResNet18-based pose predictor to estimate azimuth and elevation angles, alongside an NOC predictor that maps an RGB image and foreground mask to a three-channel NOC image.The pose predictor’s final 512-dimensional features are refined before angle estimation; the NOC predictor uses an EfficientNet-b4 backbone.Source: D. Network Architecture · p. 17

E. Implementation Details

Mesh2Tex training uses differentiable rendering, multiple rendered views, and separate optimizer settings for PhotoShape and CompCars. Query-image optimization refines latent codes and generator weights, while predictor networks are pretrained on synthetic renders and fine-tuned on real-world data.

E. Implementation Details

  • Training uses batch size 2 and renders 8 views per shape, with distinct encoder, generator, Ψ, and discriminator learning rates for PhotoShape and CompCars.PhotoShape rates are 1e-4, 12e-4, 1e-4, and 14e-4; CompCars rates are 1e-4, 15e-4, 5e-4, and 1e-4, respectively.Source: E. Implementation Details · p. 17
  • Query-image optimization updates latent codes for 100 iterations, then refines the two final generator synthesis blocks for 300 more iterations at learning rate 1e-2.The style loss uses VGG19 features from the 2nd, 4th, 8th, 12th, and 16th convolutional layers, with 64-pixel patches and foreground masks.Source: E. Implementation Details · pp. 17–18
  • The qualitative ablation visualizes texture transfer from unaligned synthetic ShapeNet queries using the query-image pose.The caption specifies the comparison setup but does not report an ablation outcome.Source: Figure 16 · p. 18
  • Pose and NOC predictors are pretrained on synthetic ShapeNet renders and fine-tuned on real-world ScanNet and CompCars data, except CompCars NOC prediction lacks NOC data.Pose prediction uses batch size 128; NOC prediction uses batch size 64, both on 512x512 images.Source: E. Implementation Details · p. 18
  • The architecture overview depicts a hierarchical surface encoder processing normals and curvatures, then a generator combining geometric features, latent texture code, and learned noise into per-face features.The neural field fuses encoder and generator features by barycentric coordinates with an auxiliary latent vector to produce surface color.Source: Figure 17 · p. 19, Figure 18 · p. 19

Related papers

Loading 2304.05868v1…