PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Fanbo Xiang, Zexiang Xu, Miloš Hašan, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Hao Su

arXiv:2103.00762v1cs.CV

TL;DR

Volumetric neural renderers can produce photo-realistic images, but their entangled geometry and appearance are difficult to edit. NeuTex separates volumetric geometry from 2D texture appearance using mappings constrained by cycle consistency. It reports quality on par with or slightly below NeRF while enabling surface-aware appearance editing.

  • Problem

    Volumetric neural renderers entangle geometry and appearance in a representation that does not explicitly expose the surface, limiting easy appearance editing.

    Source: §1 · p. 1

  • Method

    NeuTex represents geometry as a volume and appearance as a 2D texture, using forward and inverse mappings with cycle consistency to discover a surface-aware texture space.

    Source: §1 · pp. 1–2

  • Results

    NeuTex’s rendering quality is on par with or slightly worse than NeRF, while significantly outperforming traditional mesh-based reconstruction and previous neural rendering methods.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The discovered texture space supports flexible surface appearance editing and can represent BRDF parameters as 2D maps for view synthesis and relighting.

    Source: §1 · p. 2, §6 · p. 9

    When viewpoints do not cover the object entirely, mask loss is necessary because density can otherwise darken renderings and fake shading effects.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Recent work has demonstrated that volumetric scene representations combined with differentiable volume rendering can enable photo-realistic rendering for challenging scenes that mesh reconstruction fails on. However, these methods entangle geometry and appearance in a "black-box" volume that cannot be edited. Instead, we present an approach that explicitly disentangles geometry--represented as a continuous 3D volume--from appearance--represented as a continuous 2D texture map. We achieve this by introducing a 3D-to-2D texture mapping (or surface parameterization) network into volumetric representations. We constrain this texture mapping network using an additional 2D-to-3D inverse mapping network and a novel cycle consistency loss to make 3D surface points map to 2D texture points that map back to the original 3D points. We demonstrate that this representation can be reconstructed using only multi-view image supervision and generates high-quality rendering results. More importantly, by separating geometry and texture, we allow users to edit appearance by simply editing 2D texture maps.

1. Introduction

Mesh reconstruction can struggle to produce realistic images, while volumetric neural renderers offer quality but entangle geometry and appearance in ways that hinder editing. NeuTex keeps volumetric geometry and separates appearance into a 2D texture, using cycle-consistent mappings to support surface-aware editing.

1. Introduction

  • Traditional meshes pair geometry with mapped textures, but mesh reconstruction can struggle to synthesize highly realistic images.Volume-based neural methods offer realistic rendering but entangle geometry and appearance, limiting easy editing.Source: §1 · p. 1
  • NeuTex represents geometry volumetrically and surface appearance with 2D texture maps, enabling reconstruction from multi-view images and conventional texture editing.Its technique also combines with Neural Reflectance Fields to reconstruct BRDF parameters as 2D maps for view synthesis and relighting.Source: §1 · pp. 1–2
  • A 3D-to-2D texture-mapping network assigns UV coordinates to shading points, and a texture network predicts their radiance in UV space.The inverse mapping and cycle-consistency loss constrain surface points and UVs to map back to one another; rendering-only supervision can otherwise produce a degenerate mapping.Source: §1 · p. 2
  • NeuTex is on par with or slightly worse than NeRF, yet significantly outperforms traditional mesh-based reconstruction and previous neural rendering methods.The paper identifies recovering a meaningful surface-aware texture parameterization for appearance editing as its first such result.Source: §1 · p. 2

2. Related Work

Prior work spans mesh reconstruction, neural volumetric rendering, and learned textures, but these approaches differ in their support for realistic synthesis and editable surface appearance. NeuTex discovers texture coordinates alongside volumetric reconstruction to provide a 2D appearance representation.

Neural rendering

  • NeuTex reports higher rendering quality than other neural rendering methods and a more uniform discovered surface than AtlasNet supervised by noisy COLMAP point clouds.The surface comparison uses rendering loss and cycle-mapping loss for NeuTex.Source: §2 · pp. 2–3

Mesh-based reconstruction and rendering

  • Mesh reconstruction remains challenging for photo-realistic synthesis, while view-dependent texture maps used to fix artifacts are not easily edited.NeuTex uses volumetric rendering while extracting surface appearance as view-independent textures.Source: §2 · p. 2

Neural rendering

  • Neural volumetric representations support photo-realistic rendering but do not explicitly reason about a scene’s 2D surface, leaving appearance difficult to modify after reconstruction.NeuTex instead exposes surface appearance as a 2D neural texture in an automatically discovered texture space.Source: §2 · pp. 2–3

Learning textures

  • Texture-learning methods often assume a known mesh template or UV mapping, whereas NeuTex jointly reconstructs volumetric geometry and discovers a texture space for arbitrary shapes.Other methods that regress colors from 3D points require a known mesh and do not provide an editable UV space.Source: §2 · p. 3
  • AtlasNet maps UV coordinates to 3D surfaces for generalized geometry, but its approach is not directly applicable to arbitrary shapes or realistic rendering.NeuTex learns forward and inverse mappings within differentiable volume rendering.Source: §2 · p. 3

3. Neural Texture Mapping

NeuTex separates volumetric geometry from appearance represented in a 2D texture space, while retaining differentiable volume rendering. An inverse mapping and cycle consistency regularize the surface-to-texture mapping during training.

3.1. Overview

  • NeuTex uses separate networks for volume density, 3D-to-2D texture mapping, and texture-space radiance, with an inverse mapping to regularize the discovered texture space.This decomposes geometry and appearance while retaining volumetric rendering.Source: §3.1 · p. 3, §3.2 · p. 4

3.2. Disentangled neural scene representation

  • The appearance function maps a 3D point to UV coordinates, then predicts radiance from those coordinates and the viewing direction.The texture space is usually a 2D unit sphere, with UV coordinates interpreted as points on that sphere.Source: §3.2 · p. 4, §3.3 · p. 4

3.3. Texture space and inverse texture mapping

  • An inverse mapping sends UV coordinates back into 3D, allowing a cycle correspondence that regularizes the texture mapping and supports surface reasoning.Pure rendering supervision can produce distorted mappings or map multiple points to the same UV coordinate.Source: §3.3 · p. 4

3.4. Training neural texture mapping

  • Checkerboard renderings show relatively uniform texture space with or without coarse-point-cloud initialization, unlike training without inverse mapping and cycle loss.The full network is trained end to end with rendering supervision and a cycle loss weighted by each shading point’s contribution to pixel color.Source: §3.4 · pp. 4–5, Figure 3 · p. 5
  • Mask loss is necessary when viewpoints do not cover the object entirely, but is often optional when view coverage is dense.The mask loss supervises the final ray transmittance using a ground-truth foreground-background mask.Source: §3.4 · p. 5

4. Implementation Details

NeuTex uses separate geometry and texture networks, with a staged training procedure that begins with noisy point-cloud supervision and then refines the surface without it.

4.1. Network details

  • The geometry, texture, and mapping networks are MLPs; positional encoding is applied to geometry and texture inputs but omitted from mapping networks to encourage smooth, uniform mapping.Viewing direction and UV coordinates are represented as unit vectors.Source: §4.1 · p. 5

4.2. Training details

  • Training begins with inverse-mapping initialization from a COLMAP point cloud using Chamfer loss, which facilitates training despite noise in the MVS reconstruction.The initialization is not required in most cases.Source: §4.2 · p. 5
  • Novel-view synthesis is evaluated with average PSNR and SSIM over four held-out views on five DTU scenes.Source: Table 1 · p. 6
  • After noisy point-cloud initialization, joint rendering and cycle-loss training produces a smoother inverse-mapped surface.The reported surface is smoother after training without point-cloud supervision.Source: §4.2 · pp. 5–6
  • The staged schedule uses Chamfer and rendering losses for 50,000 iterations, then the full loss for about 500,000 iterations, followed by texture-network fine-tuning with other networks frozen.The full process takes 2–3 days on a single RTX 2080Ti GPU.Source: §4.2 · p. 6

5. Results

NeuTex is evaluated on real scenes for view synthesis, texture editing, inverse mapping, and reflectance-field rendering. Its results combine realistic novel views with automatically discovered editable texture spaces.

5.1. Configuration

  • The evaluation uses five DTU scenes, two Neural Reflectance Fields scenes, and three captured scenes, with input image counts varying by source.The method requires clean, easily segmentable backgrounds; masks are used for the DTU and captured scenes.Source: §5.1 · p. 6

5.2. View synthesis results on DTU scenes

  • On five DTU scenes, NeuTex significantly outperforms COLMAP, DeepVoxels, and SRN, while its rendering quality is slightly worse than NeRF.The comparison covers qualitative and quantitative results; NeuTex’s rendered views are described as realistic and close to NeRF’s.Source: §5.2 · pp. 6–7

5.3. Texture mapping and appearance editing

  • NeuTex discovers a reasonably uniform texture space that meaningfully represents object surfaces, and texture edits produce consistent appearance changes across multiple views.The examples include full-map appearance changes and local additions such as logos, numbers, and patterns.Source: §5.3 · pp. 7–8, Figure 5 · p. 7
  • Without the inverse mapping network, rendering-only training yields degenerate UV mappings; cycle-loss training generally discovers a uniform space, and removing noisy point-cloud supervision lets the surface become smoother.The point-cloud initialization helps training but the inverse network can also function without it.Source: §5.3 · p. 8

5.4. Extension to reflectance fields

  • In reflectance fields, NeuTex maps BRDF parameters into texture space and produces realistic rendering with texture-space BRDF editing on two scenes.The geometry network predicts density and normal, while the texture network predicts reflectance parameters.Source: §5.4 · p. 8

6. Conclusion

NeuTex separates volumetric geometry from appearance represented in an automatically discovered texture space. Joint texture and inverse mapping networks with a surface-aware cycle loss support texture-space appearance editing.

6. Conclusion

  • NeuTex represents geometry as a 3D volume and appearance as a 2D texture in an automatically discovered texture space.Source: §6 · pp. 8–9
  • Joint 3D-to-2D texture mapping and 2D-to-3D inverse mapping, trained with a surface-aware cycle loss, discover a texture space that meaningfully explains the object surface.Source: §6 · p. 9
  • The representation enables flexible surface-appearance editing for neural volumetric rendering.Source: §6 · p. 9

Supplementary Material

The supplied supplementary material identifies the paper’s authors, affiliations, and arXiv record.

Supplementary Material

1. Cube map Clarification

The clarification explains how cubemaps depict spherical textures and why they are used instead of equirectangle maps.

1. Cube map Clarification

  • A cubemap records the projected color information across six faces of a unit box.An equirectangle map is an alternative projection of the same spherical function.Source: §1 · p. 11
  • Cubemaps are used because they involve less distortion than equirectangle maps, avoiding distortion near the top and bottom.The two map types show different projections of the same spherical texture.Source: §1 · p. 11
  • Cubemaps depict a spherical texture by projecting the sphere’s colors onto a cube centered at the origin.Opening up the cube produces the cubemap.Source: Figure 1 · p. 11, §1 · p. 11

2. Network Implementation Details

The appendix clarifies the network diagram and describes how point-cloud, cycle, rendering, and mask losses are used during initialization.

2.1. Network structure

  • Cubemaps and equirectangle maps represent the same spherical texture, with cubemaps having less distortion in the Y direction.Source: Figure 2 · p. 11

2.2. Training details in initialization

  • Initialization downsamples the COLMAP point cloud to 2,000–3,000 points and samples 2,500 UV coordinates uniformly on the unit sphere.The downsampled cloud is denoted Pgt and the UV coordinates P.Source: §2.2 · p. 11
  • For initialization, the COLMAP cloud is downsampled to 2,000–3,000 points as Pgt, while 2,500 UV coordinates are sampled uniformly on the unit sphere as P; network F−1 maps P to a 3D point cloud.Source: §2.2 · p. 11
  • An inverse loss applies a 2D-to-3D-to-2D cycle during initialization, rather than the paper’s 3D-to-2D-to-3D cycle loss.Source: §2.2 · p. 11

2.1. Network structure

  • Figure 3 identifies x as 3D coordinates, γ as positional encoding, u as texture-space points, d as viewing direction, and σ and c as predicted density and radiance.The figure also includes the UV mapping and texture function Ftex.Source: §2.1 · p. 11, Figure 3 · p. 12

2.2. Training details in initialization

  • Initialization also uses the same rendering and mask losses as the main training stage, with a=100 and b=c=1.Source: §2.2 · p. 12

3. Additional Results

The additional results report quantitative comparisons across five DTU scenes and show further visual comparisons and texture-editing results.

3.1. Full quantitative comparison

  • Across five DTU scenes, NeuTex is slightly worse than NeRF but significantly outperforms other traditional and neural rendering methods.The paper reports averaged quantitative results and provides detailed comparisons for individual scenes.Source: §3.1 · p. 12

3.2. Additional visual results

  • Figure 4 presents visual comparisons of methods on the remaining two DTU scenes.Source: §3.2 · p. 12
  • Figure 5 shows additional texture-editing results, and the attached video provides more view-synthesis and editing results.Source: §3.2 · p. 12
Loading 2103.00762v1…