NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

Introduction

• Artists commonly use mesh-based workflows because polygon meshes allow precise shape and texture control, whereas neural implicit field editing remains limited.Mesh editing includes changing vertices and faces and texturing with UV maps.Source: §1 · pp. 1–2
• Prior neural implicit editing methods are limited to specific semantic categories or rigid transformations, restricting fine-grained local geometry and texture editing.The paper identifies coordinate-based MLPs, voxels, and scattered point clouds as poorly suited to such editing.Source: §1 · p. 2
• NeuMesh stores disentangled geometry and appearance codes on mesh vertices, enabling mesh-guided deformation, texture swapping, filling, and painting.Changing mesh geometry or local codes is reflected in the rendered output; appearance codes can be transferred across geometry.Source: §1 · pp. 2–3
• Learnable vertex sign indicators estimate signed distances without requiring watertight meshes, while distillation and fine-tuning guide stable optimization from a pretrained implicit field.The indicators improve spatial distinguishability across mesh faces; distillation transfers a baked MLP-based field into NeuMesh.Source: §1 · p. 3
• Spatial-aware texture optimization updates only codes selected by probing rays from painted pixels, targeting edits while preserving other regions.The approach is proposed to avoid view-specific overfitting and artifacts from fine-tuning on a single image.Source: §1 · p. 3
• Experiments and editing examples on real and synthetic datasets demonstrate photo-realistic rendering quality and flexible geometry and texture editing.The introduction reports the result across both real and synthetic data.Source: §1 · p. 4

Mesh-based representation and rendering

• Existing mesh-rendering pipelines rely on UV maps or warped source images, whereas NeuMesh encodes appearance directly on 3D vertices for objects difficult to unwrap.UV mapping limits applicability to complex topology and delicate structure, while MVS-based mesh pipelines require nearby source images and do not support high-level editing.Source: §2 · p. 4

Neural rendering

• Neural-rendering work inspired by NeRF includes scene decompositions into local representations—multiple tiny networks, point clouds [33], and voxels —that are not designed for easy polygon-mesh manipulation or geometry and texture editing.Source: §2 · p. 4

Neural scene editing

• Prior scene-editing methods are limited by rigid object transforms [12], out-of-distribution generalization, simple shapes [59], orthographic projection, or lack of fine-grained texture editing.NeuMesh uses an editable triangle-mesh scaffold with vertex-selectable regions to enable non-rigid geometry and fine-grained texture editing.Source: §2 · p. 4

3.1 Neural Mesh-based Implicit Field

• NeuMesh interpolates per-vertex geometry and texture codes and learnable signed distances from nearby mesh vertices, then decodes SDF and radiance for volume rendering.The signed distances complement spatial distinguishability; the sign indicators begin from vertex normals and adapt during training, with their effect increasing near the surface.Source: §3.1 · pp. 5–6, §3 · p. 5

3.2 Optimizing Mesh-based Implicit Field

• NeuMesh is trained by distilling SDF and color outputs from a coordinate-based teacher while fine-tuning against image colors, with sign-indicator and Eikonal regularization.The paper reports that this scheme transfers the MLP-based field into an editable mesh representation and produces better appearance details; sign indicators are regularized toward precomputed vertex normals.Source: §3.2 · pp. 6–7

3.3 Mesh-guided Geometry Editing

• Because the field is aligned to the mesh, deforming its vertices deforms the implicit field and rendered object without fine-tuning; transformed vertices receive normal-rotation compensation according to their sign indicators.Users can move selected vertices with mesh-deformation methods such as ARAP or modeling software.Source: §3.3 · pp. 7–8

3.4 Designatable texture editing

• Designatable texture editing updates texture codes and their decoder in selected mesh regions, enabling texture swapping, filling regions with repeated template textures, and painting from a single view.For painting, spatial-aware optimization updates only nearby affected texture codes, restricts rays to a dilated paint mask, and augments radiance-decoder inputs with random viewing directions.Source: §3.4 · pp. 8–9

4.1 Datasets

• DTU evaluation uses 15 scenes at 1600 × 1200 resolution, with 10% of images held out; NeRF 360° Synthetic evaluation uses four representative scenes.The four scenes cover thin structures (Mic), complex shapes (Lego), and rich textures (Chair and Hotdog).Source: §4.1 · pp. 9–10

4.2 Comparison of Rendering and Mesh Quality

• NeuMesh is comparable to or better than NeuS and NeuTex in rendering quality, while maintaining mesh quality on par with NeuS and preserving clear appearance detail.The mesh-quality comparison uses the DTU training split, whereas NeuS uses all images in its original paper; NeuTex struggles with complex shapes, while NeuS renders slightly blurred and smoothed details.Source: §4.2 · pp. 10–11, Table 1 · p. 10, Table 2 · p. 11

4.3 Experiment on Geometry Editing

• Mesh-guided deformation produces less jagged object boundaries than naïve field warping applied to NeuS.The mesh is deformed in Blender, and the rendered object deforms simultaneously; the baseline warps query points using interpolated offsets from the three nearest extracted-mesh vertices.Source: §4.3 · p. 11, Figure 6 · p. 11

4.4 Experiment on Texture Editing

• NeuMesh supports texture swapping, filling, and painting while retaining geometry or appearance properties relevant to each edit.Swapping transfers textures, including can text, without changing geometry; filling reuses source-model texture codes and decoder while retaining view-dependent effects. Painting directly on 2D images avoids NeuTex’s distorted UV-map placement difficulties, especially on complex shapes.Source: Figure 7 · p. 12, §4.4 · p. 13

4.5 Ablation Studies

• Ablations show that learnable sign indicators improve image quality, distillation and fine-tuning are both needed, and spatial-aware optimization avoids painting artifacts across views.Without distillation, fine-tuning produces noticeable artifacts; distillation alone yields blurrier images. Single-image painting overfits to its viewpoint and can disrupt appearance consistency, whereas spatial-aware optimization maintains other parts untouched.Source: §4.5 · pp. 13–14, Figure 8 · p. 13, Table 3 · p. 14

• Mesh-scaffold neural implicit fields use learnable vertex geometry and texture codes to support high-fidelity volume rendering and flexible geometry and texture editing.Each vertex's codes represent its neighboring local space.Source: §5 · p. 14
• The method does not model fine-grained lighting effects such as shadows or specular reflections under a lighting environment.The authors suggest material and lighting estimation as a possible future improvement.Source: §5 · p. 14
• Because it relies on a mesh scaffold, the method cannot represent objects that fail during reconstruction, such as smoke or liquid.Source: §5 · p. 14

Supplementary Material

• It covers model architecture, geometry and texture editing, limitations, and additional experiment results.These topics are organized in Sections A, B.2, B.3, C, and D, respectively.Source: Supplementary Material · p. 20

A Model Architecture

• NeuMesh extracts a 256-resolution marching-cubes mesh from NeuS’s SDF and interpolates 32-dimensional geometry and texture codes plus learnable signed distance from 8 nearest vertices.It applies positional encoding to signed distances, interpolated codes, and viewing directions using 8, 2, and 4 frequencies, respectively. Geometry and radiance use 3- and 4-layer MLP decoders with 256 hidden units, using SoftPlus and ReLU activations, respectively.Source: A Model Architecture · p. 20
• Rendering uses 64 coarse ray samples and progressive up-sampling to 64 fine samples, while mesh intersections precompute each ray’s near-far bounds.This yields 128 samples per ray and accelerates rendering and training.Source: A Model Architecture · p. 20
• NeuMesh renders better appearance details than NeuS and NeuTex, including the roof at DTU Scan 37 and metal grids at NeRF-Synthetic Mic.Source: A Model Architecture · p. 21

B.1 Training Details

• For each object, NeuMesh first trains a NeuS teacher, then optimizes codes and decoders using teacher outputs and images.Teacher supervision covers queried color and SDF values through the distillation loss.Source: §B.1 · p. 21
• Training uses 512 rays per batch on one Nvidia RTX3090-24G GPU, Adam at learning rate 0.0005, cosine annealing with 5000 warm-up steps, and takes about 16 hours per model.Source: §B.1 · p. 21
• For DTU's unbounded background, training supervises foreground masks with binary cross-entropy, following.Source: §B.1 · p. 21

B.2 Details of Geometry Editing

• Geometry editing deforms the corresponding mesh scaffold to deform the neural implicit field.Source: §B.2 · p. 21
• During local deformation or rotation, the interpolated signed distance should remain unchanged to preserve local consistency of the learnable signed distances.Source: §B.2 · p. 21
• The method compensates the learnable signed indicator by the relative change in vertex normal: h~′(x) = h~(x) + δh_x.The vertex-normal difference is averaged from nearby surface normals between the original and deformed meshes.Source: §B.2 · p. 22

B.3 Details of Texture Editing

• Texture editing updates or optimizes locally bounded vertex texture codes and, when needed, their binding encoders for the region of interest.Source: §B.3 · p. 22
• For swapping, 4–9 point correspondences guide Umeyama scaled alignment and ARAP deformation before target codes receive inverse-distance-weighted interpolations from four nearest source vertices.Source: §B.3 · p. 22
• For filling, a selected UV-map is tiled with a small template patch from a pretrained NeuMesh model, whose interpolated codes and radiance decoder are assigned to target vertices.The example template patch has approximately 10 vertices.Source: §B.3 · p. 22
• At edited-area boundaries, query colors from different texture decoders are fused using inverse-distance weighting to smooth the transition to the original appearance.Source: §B.3 · p. 22
• Painting traces rays from painted pixels to mesh faces, optimizes only hit-face vertex codes with Adam at learning rate 0.01, and takes about one hour for 8,000 iterations.The procedure is designed to preserve geometry and appearance elsewhere.Source: §B.3 · pp. 22–23
• Fig. K shows that direct textured-mesh editing with template patterns, without lighting and material estimation, gives unsatisfactory statue-editing results.The figure compares textured-mesh editing with NeuMesh editing.Source: §B.3 · p. 23

• Neural implicit representation supports photorealistic volumetric rendering, view-dependent effects, and fine-grained appearance editing, whereas textured-mesh quality is bounded by MVS and lacks such effects without BRDF and lighting estimates.These capabilities are described for real-world and synthetic data; latent-space operations enable material editing and appearance variations on real-world scenes.Source: Using neural implicit representation… · p. 23
• A learnable sign indicator provides spatially distinguishable interpolated signed distances for surface queries, avoiding unreliable physical distance signs on general meshes that may not be watertight.NeuMesh learns single-layer features on mesh surfaces, where bare code interpolation cannot distinguish inner from outer queries along the surface normal.Source: Using neural implicit representation… · p. 23
• Teacher NeuS distillation supplies an SDF field for extracting a mesh scaffold and alleviates local-minimum issues that can arise when training the locally embedded geometry and appearance features.The method uses distillation followed by fine-tuning rather than training from scratch.Source: Using neural implicit representation… · pp. 23–24
• Texture swapping works across moderate geometry differences; for significant topology differences, the paper recommends texture filling in UV space, independent of object geometry.The supplied figure caption identifies texture swapping with different geometry.Source: Figure L · p. 24, Using neural implicit representation… · p. 24
• Real-world deployment is limited by rendering at about 30 seconds per view and manual 3D-modeling software selection of ROI vertices; acceleration and semantic annotation are proposed alternatives.Suggested acceleration options include coefficient caching, multiresolution hash encoding, and MVS priors.Source: Using neural implicit representation… · p. 24
• Unlike scattered point-based features, mesh-vertex features support proxy deformation and UV texture editing, while embedded surface normals enable view-dependent effects in texture filling.Point-based methods make it difficult to identify which features to edit because many points contribute to pixel appearance or nearby ray samples.Source: Using neural implicit representation… · p. 24

Rendering quality comparison

• NeuMesh renders more detail than competing methods, especially for complex shapes and textures such as the roof in DTU Scan 37 and metal grids in NeRF-Synthetic Mic.The comparison highlights details in these examples.Source: D More Experiment Results · p. 24

Rendering quality with varying mesh vertex numbers

• With 10K vertices, rendering still outperforms NeuS despite slight metric-quality effects when reducing the mesh from 100K or 50K vertices.The evaluated configurations use 10K, 50K, and 100K vertices on DTU Scan 114.Source: D More Experiment Results · p. 25

Model design

• Learnable signed-distance encoding complements spatial distinguishability perpendicular to the surface and beats relative-position encoding; single-layer mesh features match uniform-grid visual quality while retaining editing flexibility.The uniform-grid alternatives are described as similar to NSVF or Plenoxel.Source: Figure M · p. 25, Table D · p. 25, D More Experiment Results · p. 26

Texture editing vs. radiance field warping

• Code-swapping texture edits look more natural than radiance-field warping, which produces visible artifacts when its warped texture field is spatially misaligned with the geometry.Code swapping keeps the edited texture field tightly fitted to the geometry.Source: D More Experiment Results · p. 26

Hybrid object editing

• Hybrid examples combine geometry and texture edits, demonstrating the method’s flexibility for integration into 3D modeling workflows.The examples combine multiple editing operations.Source: D More Experiment Results · p. 27

Large-scale scenes

• NeuMesh can scale to large scenes when a scaffold mesh is available, and the paper shows texture edits on Tanks&Temple scenes with foreground masks from NSVF.For complicated unbounded backgrounds, the paper suggests NeRF++ [70] parameterization or precomputed masks such as those used in NSVF and IDR.Source: D More Experiment Results · pp. 27–28

Influence of triangle quality

• NeuMesh delivers reasonable rendering quality with sparse or uneven triangulation; isotropic remeshing can help ensure a uniform vertex distribution.The mesh scaffold derives from the teacher NeuS SDF, and the paper cites isotropic remeshing by Botsch et al. as an example regularization algorithm.Source: D More Experiment Results · pp. 27–28