PaperlayerView exact revision on arXiv

Source-linked AI summary

NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, Guofeng Zhang

arXiv:2207.11911v1cs.CVcs.GR

TL;DR

Existing neural rendering methods provide limited support for fine-grained editing of general objects, especially local non-rigid geometry and texture changes. NeuMesh addresses this gap by encoding disentangled geometry and texture codes on mesh vertices, with techniques for spatial distinguishability, training stability, and precise texture editing. Experiments and editing examples on real and synthetic data show high-fidelity rendering and flexible geometry and texture editing.

  • Problem

    Existing neural rendering methods offer limited editing functionality, often supporting only specific semantic categories or rigid transformations rather than fine-grained editing for general objects.

  • Method

    NeuMesh encodes a neural implicit field with separate geometry and texture codes on a mesh scaffold, supported by learnable sign indicators, distillation and fine-tuning, and spatial-aware optimization.

  • Results

    Experiments and editing examples on real and synthetic datasets demonstrate photo-realistic rendering quality and flexible geometry and texture editing.

  • Takeaways & Limitations

    NeuMesh supports mesh-guided geometry editing and designated texture swapping, filling, and painting within a neural implicit representation.

  • Takeaways & Limitations

    The method does not model fine-grained lighting effects and cannot represent objects that fail during mesh reconstruction, such as smoke or liquid.

Abstract

from arXiv · show

Very recently neural implicit rendering techniques have been rapidly evolved and shown great advantages in novel view synthesis and 3D scene reconstruction. However, existing neural rendering methods for editing purposes offer limited functionality, e.g., rigid transformation, or not applicable for fine-grained editing for general objects from daily lives. In this paper, we present a novel mesh-based representation by encoding the neural implicit field with disentangled geometry and texture codes on mesh vertices, which facilitates a set of editing functionalities, including mesh-guided geometry editing, designated texture editing with texture swapping, filling and painting operations. To this end, we develop several techniques including learnable sign indicators to magnify spatial distinguishability of mesh-based representation, distillation and fine-tuning mechanism to make a steady convergence, and the spatial-aware optimization strategy to realize precise texture editing. Extensive experiments and editing examples on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability. Code is available on the project webpage: https://zju3dv.github.io/neumesh/.

1 Introduction

NeuMesh addresses the limited support of neural implicit fields for fine-grained editing by encoding disentangled geometry and texture features on a mesh scaffold. This representation supports mesh-guided geometry deformation and several localized texture-editing operations while retaining photo-realistic rendering quality.

  • Motivation: Neural implicit fields support high-quality reconstruction and rendering, but existing editing methods are limited to specific semantic categories or rigid transformations.Their coordinate-based, voxel, or point-cloud encodings are poorly suited to non-rigid deformation and local texture editing.
  • NeuMesh: NeuMesh encodes local geometry and texture features as vertex-bounded codes on a mesh scaffold, allowing the representation to deform with the mesh.Implicit fields are decoded by interpolating the vertex codes during volume rendering.
  • NeuMesh: Separate geometry and appearance codes enable texture transfer across geometry by replacing appearance codes between objects.The representation supports texture swapping, filling, and painting in addition to non-rigid geometry deformation.
  • Technical techniques: Learnable sign indicators improve spatial distinguishability for queries near mesh surfaces, while distillation and fine-tuning promote steady training.A spatial-aware optimization strategy is used to improve texture-editing precision.
  • Results: Experiments on real and synthetic datasets demonstrate photo-realistic rendering quality and flexible geometry and texture editing.The authors report that the method is powerful for editing neural implicit fields across both modalities.

2 Related Works

NeuMesh places neural implicit fields on a mesh scaffold and directly stores appearance information on 3D vertices. This avoids reliance on flat UV maps or image warping for representing and editing complex objects.

  • Mesh-based representation and rendering: Existing mesh-rendering pipelines commonly require UV mapping, which limits representation of complex topology and delicate structure.MVS-based mesh proxies instead aggregate warped image features but are not feasible for high-level editing operations.
  • NeuMesh: NeuMesh encodes geometry and texture information directly on mesh vertices and retrieves interpolated local codes and signed distances for neural rendering.The retrieved quantities are passed to geometry and radiance decoders to produce SDF values and colors.

3 Method

NeuMesh represents a neural implicit field with locally bounded geometry and texture codes on a mesh scaffold, enabling mesh-guided geometry and designated texture editing. Its training combines sign indicators, distillation, fine-tuning, and regularization to support editable rendering.

  • Mesh-based representation: NeuMesh stores geometry codes, texture codes, and sign indicators on mesh vertices, then interpolates them for local implicit-field decoding.The mesh scaffold is reconstructed first, and each vertex stores learnable parameters used during volume rendering.
  • Mesh-based representation: The geometry and radiance decoders use interpolated local codes, signed-distance information, viewing direction, and SDF gradients to produce rendered pixels.Volume rendering uses accumulated transmittance and opacity derived from adjacent SDF values.
  • Mesh-based representation: Learnable sign indicators improve spatial distinguishability for queries across the mesh surface by helping compute interpolated signed distances.They are initialized similarly to vertex normals but adjusted during training.
  • Training: Distillation from a coordinate-based teacher and photometric fine-tuning transfer the model into a flexible mesh-based representation while improving appearance details.The losses supervise SDF values and colors from the teacher alongside image pixel colors.
  • Training: Regularization encourages learned sign indicators toward precomputed vertex normals and constrains spatial-gradient norms through an Eikonal loss.The total objective combines distillation, photometric, sign-indicator, and Eikonal losses with specified weights.
  • Geometry editing: Geometry editing requires deforming the corresponding mesh, which synchronously deforms the aligned implicit field and rendered object without fine-tuning.The method supports interactive vertex manipulation with mesh-deforming methods such as ARAP or software such as Blender.
  • Texture editing: Texture editing updates locally bounded texture codes and binding decoders, supporting texture swapping and designated regional editing pipelines.Texture swapping transfers interpolated source codes to target vertices after marking regions and aligning their geometry.

4 Experiments

Experiments evaluate NeuMesh on real DTU and synthetic NeRF 360° data for rendering, mesh reconstruction, geometry editing, texture editing, and model components.

  • Datasets: NeuMesh is evaluated on 15 DTU scenes and four representative NeRF 360°Synthetic scenes, including thin structures, complex shapes, and richly textured objects.DTU uses a 10% test split; NeRF-Synthetic scenes include Mic, Lego, Chair, and Hotdog.
  • Rendering and mesh quality: Rendering comparisons use NeuS and NeuTex on DTU and NeRF 360°Synthetic datasets, with PSNR, SSIM, and LPIPS as image-quality metrics.Mesh quality is measured separately with Chamfer distance.
  • Rendering and mesh quality: NeuMesh preserves clear appearance detail while maintaining mesh quality on par with NeuS, whereas NeuTex’s single UV atlas limits complex-shape representation.The comparison is illustrated in Fig. 5 and reported in the mesh-quality comparison.
  • Geometry editing: Mesh-guided deformation jointly deforms the mesh and rendered object, while naïve field warping applied to NeuS produces much jaggier object boundaries.The naïve solution warps query points using interpolated offsets from three nearest extracted-mesh vertices.
  • Texture editing: Texture editing demonstrates texture swapping, filling, and painting, including transferred can text, view-dependent reflections, and designated painted regions.NeuTex is compared only for texture painting because distorted UV mapping makes its other editing operations difficult.
  • Ablation studies: Ablations show that learnable sign indicators improve image quality, while both distillation and fine-tuning are indispensable for training the mesh-based representation.The sign-indicator improvement is more significant on DTU Scan 69 than on synthetic Lego; removing either training component causes blur or artifacts.

5 Conclusion

NeuMesh is presented as a mesh-based neural representation supporting high-fidelity volume rendering and flexible geometry and texture editing. The conclusion identifies lighting effects and mesh-scaffold dependence as limitations.

  • Conclusion: NeuMesh encodes learnable geometry and texture codes at mesh vertices to support high-fidelity rendering and flexible geometry and texture editing.Each vertex represents information for its neighboring local space.
  • Limitations: The method does not model fine-grained lighting effects such as shadowing and specular reflection under a particular lighting environment.The authors suggest material and lighting estimation as future work.
  • Limitations: Because NeuMesh relies on a mesh scaffold, it cannot represent objects that fail during reconstruction, such as smoke or liquid.

Supplementary Material

The supplementary material provides additional method details, geometry and texture editing descriptions, limitation discussions, and experiment results.

  • Supplementary Material: Supplementary sections cover the model architecture, geometry editing, texture editing, limitations, and additional experiment results.The architecture is in Section A, geometry editing in Section B.2, texture editing in Section B.3, limitations in Section C, and experiments in Section D.

A Model Architecture

NeuMesh extracts a mesh scaffold from NeuS’s SDF field and interpolates local geometry, texture, and signed-distance information from nearby vertices for neural rendering.

  • A Model Architecture: The architecture begins by extracting a triangle mesh from NeuS’s SDF field with marching cubes at voxel resolution 256 over spatial range [−1, 1].
  • A Model Architecture: For each query point, the model retrieves K = 8 nearest vertices and interpolates 32-dimensional geometry codes, 32-dimensional texture codes, and a learnable signed distance.
  • A Model Architecture: Rendering comparisons show better appearance details than NeuS and NeuTex, including the roof in DTU Scan 37 and metal grids in NeRF-Synthetic Mic.

B.1 Training Details

NeuMesh uses mesh deformation with signed-indicator compensation to preserve local consistency in the neural implicit field.

  • B.1 Training Details: Distillation and fine-tuning supervise NeuMesh codes and decoders with teacher-model outputs and images.Training uses NeuS as the teacher and additionally supervises queried color and SDF values.
  • B.1 Training Details: Mesh deformation requires preserving the interpolated signed distance in locally deformed or rotated regions.The representation is aligned with a mesh scaffold, so deformation must maintain local signed-distance consistency.
  • B.1 Training Details: Rotation compensation adjusts the learnable signed indicator using relative changes in vertex normals.The compensated indicator is defined as ˜h′(x) = ˜h(x) + δh_x.

B.3 Details of Texture Editing

Disentangled texture codes enable localized texture operations, including swapping, while comparisons show advantages over direct textured-mesh editing.

  • B.3 Details of Texture Editing: Texture editing updates or optimizes locally bounded texture codes and binding encoders for a selected region.The codes are stored on mesh vertices, making the editing region spatially localized.
  • B.3 Details of Texture Editing: Texture swapping transfers surface appearance by exchanging texture codes between corresponding source and target vertices.The operation requires finding vertex correspondences between the two surface regions.
  • B.3 Details of Texture Editing: Direct textured-mesh editing with template patterns cannot provide satisfactory results without estimating lighting and material properties.The comparison is demonstrated on a statue against NeuMesh editing.
  • B.3 Details of Texture Editing: Texture painting optimization changes only selected codes and takes about ∼1 hours with 8000 iterations.The supplied implementation passage also specifies a code-change threshold of 0.01.

Using neural implicit representation instead of traditional textured

Neural implicit representation offers photo-realistic volumetric rendering and fine-grained appearance editing, while textured-mesh quality remains bounded by reconstruction and texturing.

  • Using neural implicit representation instead of traditional textured: Neural implicit representation supports photo-realistic volumetric rendering and view-dependent effects on real-world and synthetic data.The passage gives shiny golden materials as an example of view-dependent effects.
  • Using neural implicit representation instead of traditional textured: Latent-space operations enable some fine-grained material editing and appearance variations on real-world scenes.These operations are presented as editing demands supported by the neural implicit representation.
  • Using neural implicit representation instead of traditional textured: Textured-mesh rendering quality is bounded by multi-view-stereo reconstruction and texturing.This limitation contrasts with the stated rendering properties of neural implicit representation.
  • Using neural implicit representation instead of traditional textured: Texture swapping is illustrated across different geometry in the supplied figure material.The figure and caption identify the example as texture swapping with different geometry.
  • Using neural implicit representation instead of traditional textured: For significant topology differences, the method recommends texture filling in UV spaces rather than direct texture swapping.Moderate geometry differences remain compatible with texture swapping.

D More Experiment Results

Additional experiments examine rendering quality, mesh resolution, representation design, texture editing, and hybrid operations across real and synthetic scenes.

  • D More Experiment Results: NeuMesh renders more details than competing methods, especially for complex shapes and textures.Examples include the roof in DTU Scan 37 and metal grids in NeRF-Synthetic Mic.
  • D More Experiment Results: 10K, 50K, and 100K mesh-vertex models show only slight metric-quality changes as vertex numbers decrease, while 10K still outperforms NeuS.The experiment uses DTU Scan 114 and demonstrates robustness to vertex sparsity.
  • D More Experiment Results: Learnable signed distance complements spatial distinguishability in the direction perpendicular to the surface.The comparison identifies this design as necessary for the single-layer mesh representation.
  • D More Experiment Results: A single-layer mesh representation achieves visual quality on par with uniform grid-based representations while retaining geometry and texture editing functionality.Thickening mesh vertices into uniform grids provides the comparison design but loses some fine-grained editing flexibility.
  • D More Experiment Results: Hybrid object-editing examples combine geometry and texture operations to integrate the representation into modern 3D modeling workflows.The examples are shown in Fig. P.
  • D More Experiment Results: Large-scale Tanks&Temple scenes demonstrate texture editing with a mesh scaffold and locally embedded features.The method can scale up when a scaffold mesh is available; unbounded backgrounds may use NeRF++ parameterization or segmentation masks.
Loading 2207.11911v1…