PaperlayerView exact revision on arXiv

Source-linked AI summary

Texture Fields: Learning Texture Representations in Function Space

Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, Andreas Geiger

arXiv:1905.07259v1cs.CV

TL;DR

Texture reconstruction of 3D objects remains limited by low-resolution or shape-dependent representations. Texture Fields regress a continuous 3D texture function with a neural network, supporting arbitrary shape representations and end-to-end learning. Experiments show high-frequency single-image reconstruction and probabilistic texture generation from 3D shapes.

  • Problem

    Texture reconstruction has received little attention, while existing representations limit resolution or rely on task-specific shape parameterizations.

  • Method

    Texture Fields regress a continuous function mapping 3D locations to texture values using a neural network, with conditional and latent-variable generative variants.

  • Results

    Experiments demonstrate high-frequency texture prediction from a single image, joint shape-texture reconstruction, and preliminary probabilistic texture synthesis from 3D shapes.

  • Takeaways & Limitations

    Texture Fields provide a continuous texture representation usable for conditional reconstruction and unconditional texturing of 3D objects.

  • Takeaways & Limitations

    Comparison with Im2Avatar is restricted because its training data cover only a subset of the authors' data, and quantitative comparison is unavailable for chairs and tables.

Abstract

from arXiv · show

In recent years, substantial progress has been achieved in learning-based reconstruction of 3D objects. At the same time, generative models were proposed that can generate highly realistic images. However, despite this success in these closely related tasks, texture reconstruction of 3D objects has received little attention from the research community and state-of-the-art methods are either limited to comparably low resolution or constrained experimental setups. A major reason for these limitations is that common representations of texture are inefficient or hard to interface for modern deep learning techniques. In this paper, we propose Texture Fields, a novel texture representation which is based on regressing a continuous 3D function parameterized with a neural network. Our approach circumvents limiting factors like shape discretization and parameterization, as the proposed texture representation is independent of the shape representation of the 3D object. We show that Texture Fields are able to represent high frequency texture and naturally blend with modern deep learning techniques. Experimentally, we find that Texture Fields compare favorably to state-of-the-art methods for conditional texture reconstruction of 3D objects and enable learning of probabilistic generative models for texturing unseen 3D models. We believe that Texture Fields will become an important building block for the next generation of generative 3D models.

1. Introduction

Texture reconstruction has lagged behind geometry reconstruction because existing texture representations limit resolution or depend on task-specific shape parameterizations. Texture Fields address these constraints with a neural continuous function that supports end-to-end learning and high-frequency texture prediction.

  • Texture reconstruction has received less attention than learning-based 3D geometry reconstruction.
  • Voxel representations scale cubically with resolution, restricting many methods to low resolutions such as 32^3 or 64^3 and limiting high-frequency detail.
  • An ideal texture representation should be independent of shape representations and able to represent high-frequency detail.
  • Texture Fields represent texture by learning a continuous 3D function parameterized by a deep neural network.The representation can be integrated into an end-to-end deep learning pipeline for 3D texture reconstruction.
  • Experiments show high-frequency texture prediction from a single image across multiple 3D object categories.The approach is also combined with a state-of-the-art 3D reconstruction method for joint shape and texture reconstruction, and extended to generative texture synthesis.

2. Related Work

Prior work represents reconstructed texture with voxels, point clouds, meshes, or image-based models, each imposing resolution, shape-quality, parameterization, or post-processing constraints. Texture Fields instead infer appearance continuously in 3D space and can operate with arbitrary shape representations.

  • Voxels: Voxel-based texture methods are constrained by computational and memory costs, limiting their reconstruction resolution.Texture Fields avoid discretization and are described as producing more detail than voxel-based approaches.
  • Point Clouds: Point-cloud approaches combine novel-view synthesis and depth estimation but typically require post-processing to infer missing texture information.
  • Point Clouds: Texture Fields infer appearance at any location in 3D space and can be combined with arbitrary shape representations.
  • Meshes: Mesh-based approaches commonly rely on category-specific templates and rigid texture parameterizations such as UV maps.Texture Fields represent texture for arbitrary meshes without requiring a category-specific template's UV texture map.
  • Generative Image Modeling: Image-based generative approaches exploit 3D knowledge to generate images, whereas Texture Fields directly predict texture in 3D space.
  • Continuous Representation Learning: Texture Fields extend continuous function representations from shape reconstruction and 2D color generation to 3D texture reconstruction and generation.

3. Method

Texture Fields represent object texture as a continuous neural function conditioned on 3D shape and, optionally, image appearance. The framework supports supervised conditional reconstruction and probabilistic unconditional generation using image-space training.

  • Texture Field representation: The model conditions texture predictions on a fixed-dimensional shape embedding to guide colors toward the object surface and incorporate geometric context.Shape information can include surface discontinuities that often align with image edges.
  • Conditional synthesis: In the conditional setting, an image encoder supplies a viewpoint-invariant latent code, allowing texture reconstruction without known camera extrinsics or exact shape correspondence.The input image and supervising rendered images need not be identical.
  • Texture Field representation: Texture Fields map 3D points, shape embeddings, and optional conditions to RGB colors through a neural network.The continuous function avoids fixed discretization and is trained as a regression model.
  • Conditional synthesis: Training renders depth maps and color images from arbitrary viewpoints, unprojects visible pixels into 3D points, and minimizes the ℓ1 difference between predicted and rendered colors.The shape encoder, image encoder, and Texture Field are trained jointly.
  • Unconditional synthesis: For unconditional synthesis, GAN and VAE variants use a Gaussian-sampled latent code to model multiple plausible textures given only a 3D shape.The GAN generator maps latent codes and shape embeddings to RGB images through the Texture Field.

4. Experimental Evaluation

The experiments evaluate Texture Fields for representation power, single-image texture reconstruction, real-image transfer, and unconditional texture generation. Across these settings, the method represents high-frequency detail, achieves strong global visual similarity, generalizes to real images, and supports latent texture modeling.

  • 4.1. Representation Power: Texture Fields represent high-frequency texture more effectively than a fixed 128^3 voxelization in single-object fitting experiments.The voxel representation is restricted by limited granularity, whereas Texture Fields avoid discretization.
  • 4.2. Single Image Texture Reconstruction: With ground-truth shapes, Texture Fields reconstruct and complete texture from a single image while transferring visible regions such as car windows and tires.The method outperforms projection and NVS qualitatively in ambiguous regions, where those baselines produce artifacts.
  • 4.2. Single Image Texture Reconstruction: For ground-truth-shape reconstruction, Texture Fields achieve the best FID and Feature-ℓ1 distance, while NVS achieves the best SSIM.FID and Feature-ℓ1 are described as global measures that better capture visual similarity and realism than the local SSIM score.
  • 4.2. Single Image Texture Reconstruction: In the full pipeline, Texture Fields combined with Occupancy Networks produce more consistent and realistic textured 3D models from a single view.The method and NVS tie for best SSIM, while Texture Fields achieve the best FID and Feature-ℓ1 distances.
  • 4.2. Single Image Texture Reconstruction: Texture Fields transfer texture from previously unseen real images to similar unseen CAD models and generalize reasonably well despite synthetic-only training.The experiment uses real car images and selected similar ShapeNet CAD models.
  • 4.3. Unconditional Model: Unconditional VAE and GAN models generate realistic textures, while VAE latent interpolations are smooth and support texture transfer between models.VAE samples are globally consistent but slightly blurry, whereas GAN samples are sharper but contain artifacts.

5. Conclusion

Texture Fields are presented as a continuous representation for 3D texture that predicts high-frequency appearance from a single view and also supports unconditional texturing from shape alone.

  • Texture Fields provide a continuous representation for texture of 3D shapes.
  • The method predicts high-frequency textures from a single object view.
  • Texture Fields also support unconditional texture generation when only the 3D object shape is given.

A. Implementation Details

The appendix describes the network architectures, the novel view synthesis baseline, and the image-based training pipeline used in the experiments.

  • The appendix provides additional information about the network architectures used in the experiments.
  • It explains the architecture of the novel view synthesis baseline.
  • It also gives more information about the pipeline for image-based training.

A.1. Architectures

The architectures encode shape and image information, inject shape and condition codes into a ResNet-based Texture Field, and use specialized VAE, GAN, and NVS components.

  • Texture Field: The Texture Field takes a 3D position, shape embedding, and condition or latent code, producing a three-dimensional color value for each queried point.The shape embedding supplies global shape information, while the condition or latent code represents image conditioning or latent texture information.
  • Shape Encoder: The shape encoder derives a feature vector from a point set using a PointNet-based network with ResNet blocks and max-pooling layers.The point cloud is sampled from the surface of the 3D model.
  • Image Encoder: The image encoder uses pretrained ResNet-18 to derive the image embedding z.
  • VAE Encoder: The VAE encoder maps an image and shape embedding s to the mean µ and log-standard deviation log σ of latent variables.
  • GAN Discriminator: The GAN discriminator combines an RGB image with its corresponding depth image and maps them to a single scalar.
  • NVS: The NVS baseline predicts an RGB image from a depth image with a U-Net architecture and injects the image encoding into each layer of the encoder.

A.3. Data preparation

Training data comprise rendered RGB and depth images from multiple random views of ShapeNet objects, with camera information retained for reprojection; conditional inputs use renderings from Choy et al.

  • The dataset uses ShapeNet cars, chairs, airplanes, and tables, rendered from 10 random upper-hemisphere views per object.The renderings include RGB images and corresponding depth images.
  • Camera intrinsics and extrinsics are stored so depth-image pixels can be reprojected to their 3D locations.
  • Conditional experiments use the renderings provided by Choy et al. as input images.

B. Further Results

This section provides additional experimental results through Figures 17–23. The figures extend the paper’s reported evaluations and visual examples.

  • Figures 17–23 present more results for the paper’s experiments.
  • The additional results are organized across seven consecutive figures.
  • The section directs readers to Figures 17–23 for expanded experimental visualizations.

C. Field Visualizations

The visualizations examine how Texture Fields respond to varying input shapes while preserving the same image condition. They show shape-dependent texture placement, successful transfer to arbitrary shapes, and several conditional and generative texturing outcomes.

  • Field Visualizations: Varying the input shape while keeping the image condition fixed tests what Texture Fields learn.
  • Field Visualizations: Texture Fields place predicted colors at shape locations, while unconstrained regions produce meaningless predictions.Interior regions commonly appear gray and exterior regions white.
  • Field Visualizations: Changing the input shape changes color locations, transferring image-conditioned texture onto arbitrary shapes.The authors conclude that Texture Fields implicitly decode shape embeddings and reconstruct texture at encoded shape locations.
  • Field Visualizations: Figure 17 shows texture predicted for untextured CAD models from a single view of the same objects.The predictions include details such as lights and number plates, although very high-frequency details can appear blurry.
  • Field Visualizations: Figure 18 shows texturing of untextured synthetic CAD models from a single real input image.
  • Field Visualizations: Figures 19–23 visualize the full reconstruction pipeline, GAN and VAE outputs, texture transfer, and latent-space interpolations.The full pipeline uses the same ONet shape reconstruction model for all approaches; VAE results are globally consistent but can be blurry, while GAN outputs show typical artifacts.
Loading 1905.07259v1…