PaperlayerView exact revision on arXiv

Source-linked AI summary

NeRFifyMesh: Optimizing Neural Radiance Fields from Textured Meshes for Robotics Scene Building

Nillan Nimal, Mahboubeh Asadi, Sajad Saeedi

arXiv:2610.10387v1cs.RO

TL;DR

NeRFifyMesh addresses the costly, storage-intensive conversion of existing textured meshes to NeRFs using direct mesh-derived supervision rather than rendered multi-view images. It trains on a sampled point-based radiance field with color, opacity, and normals; the authors report competitive rendering comparisons and improved collision-simulation measures against Instant-NGP.

  • Problem

    Converting widely available textured meshes to NeRFs traditionally requires costly multi-view rendering, risks poor results when model regions are omitted, and creates storage demands for large datasets.

    Source: §I · p. 1

  • Method

    NeRFifyMesh samples mesh geometry and texture to create a discrete radiance field with color, binary opacity, point normals, and weighting terms for direct neural training.

    Source: §I · p. 1

  • Results

    NeRFifyMesh has the lowest LPIPS scores across all three datasets, competitive SSIM, and lower PSNR than Instant-NGP and NeRF; its simulations also show higher PSNR and lower trajectory errors than Instant-NGP for both objects.

    Source: B. Mesh Scene Rendering · p. 5, D. Simulation Experiments · p. 7

  • Takeaways & Limitations

    The representation supports unified scenes composed from multiple models and robotics simulations using geometry extracted from reusable mesh assets.

    Source: §V · p. 7, D. Simulation Experiments · p. 7

    High face counts complicate even mesh sampling, and inaccurate SDFs for non-watertight meshes can cause radiance-field errors, rendering artifacts, and noise.

    Source: E. Limitations · p. 7

Abstract

from arXiv · show

In robotics, scene representation plays a pivotal role in understanding and interacting with the environment. The advent of Neural Radiance Fields (NeRF) and its variants, as a novel representation, has opened a new frontier of research. In applications such as semantic mapping and simulation, roboticists aim to build scenes using multiple NeRF models, each representing an object. While extensive datasets of 3D mesh models already exist, there is an urgent need to develop tools to convert these assets to NeRF models for rapid algorithm development and testing. This paper presents a new pipeline for converting existing mesh models to NeRF representations by artificially generating a ground truth point-based radiance field through sampling mesh geometry and texture. This approach alleviates the need for camera-based sampling or rendering multi-view images of the original mesh to train the NeRF model. Extensive benchmarking demonstrates that our method yields comparable rendering quality to the baselines. Additionally, the application of this representation is shown by constructing unified NeRF scenes and performing collision simulations with extracted geometry.

I. INTRODUCTION

NeRFifyMesh addresses the need to convert widely available textured meshes into NeRFs without the costly, storage-intensive multi-view rendering used by traditional conversion pipelines. It derives training supervision directly from mesh geometry and texture, including surface normals, and demonstrates applications in scene composition and robotics simulation.

Motivation

  • Textured mesh models are widely available in databases including Amazon Berkeley Objects, Google Scanned Objects, ShapeNet, and Objaverse.Their prevalence in computer graphics and robotics makes mesh-to-NeRF conversion particularly relevant.Source: §I · p. 1
  • Traditional mesh-to-NeRF pipelines render multi-view images, which are costly to produce, may omit model regions, and require storage at dataset scale.The introduction notes that omitted areas can lead to poor results and cites Objaverse as an example of a large-scale dataset where storage is problematic.Source: §I · p. 1

Proposed pipeline

  • NeRFifyMesh samples mesh geometry and texture to generate a discrete radiance field with color, binary opacity, point normals, and significance weights for training.The pipeline removes the rendering optimization step and avoids needing high-quality ground-truth images or camera poses.Source: §I · p. 1

Applications

  • Learned surface normals support post-training relighting and geometry-dependent uses, and the paper demonstrates model compression, surface extraction, and collision simulation in constructed scenes.The representation is also presented as supporting scene composition from multiple NeRF models.Source: §I · pp. 1–2

II. RELATED WORK

The related work reviews neural implicit representations, mesh-based NeRF pipelines, and robotics uses of NeRFs. It positions NeRFifyMesh among approaches that derive neural representations from meshes and supports integrating existing mesh assets into NeRF-based robotics frameworks.

A. Neural Implicit Representations

  • Neural implicit representations encode geometry or appearance continuously; cited approaches include neural SDFs, occupancy networks, and multi-resolution hash encoding.Mesh autoencoders instead operate directly on meshes, but the reviewed works focus on geometry encoding and reconstruction.Source: A. Neural Implicit Representations · p. 2

B. Mesh-based NeRF Pipelines

  • Image-supervised methods derive scene geometry and appearance from rendered multi-view images, which can leave rarely visible or occluded mesh regions inaccurate.Mesh-based pipelines also extract, edit, or accelerate NeRF representations; Mesh2NeRF trains on virtual-camera samples but does not learn normals, bakes in lighting, and uses off-the-shelf backbones.Source: A. Neural Implicit Representations · p. 2, B. Mesh-based NeRF Pipelines · p. 2
  • Mesh2NeRF is contrasted with NeRFifyMesh, which uses mesh geometry directly and learns surface normals to support relighting and geometry extraction.The cited mesh-based literature also includes methods for mesh extraction, NeRF editing, and rendering efficiency.Source: B. Mesh-based NeRF Pipelines · p. 2

C. NeRFs in Robotics Applications

  • Robotics applications of NeRFs include synthetic data generation, perception, navigation, mapping, simulation, and multi-robot agents.The reviewed work motivates mesh-to-NeRF conversion as a way to integrate existing mesh assets into NeRF-based simulation and perception frameworks.Source: C. NeRFs in Robotics Applications · p. 2

III. METHOD

NeRFifyMesh uses explicit mesh geometry and texture to create a discrete radiance field for training, without multi-view images. Its representation assumes view-independent appearance and opaque meshes, while learned normals support view-dependent shading at rendering time.

A. Overview: NeRFifyMesh Representation

  • The representation assumes view-independent mesh appearance and opaque meshes, omits ray view-direction inputs, and learns normals for view-dependent shading during rendering.Opacity is modeled as a binary function.Source: A. Overview: NeRFifyMesh Representation · p. 3
  • NeRFifyMesh samples points within a mesh’s bounding box, computes signed distance, and derives color, opacity, and point-normal assignments for each sample.Surface color comes from the associated texture, and surface normals are obtained by barycentric interpolation from vertex normals.Source: A. Overview: NeRFifyMesh Representation · p. 3
  • For sampled points, mesh color and normal values are taken from the nearest surface point; assigning colors inside the mesh helps prevent ray samples from bypassing the surface and causing unpredictable rendered colors.The nearest surface point is denoted (x′, y′, z′).Source: A. Overview: NeRFifyMesh Representation · p. 3

1) Importance Weighting:

NeRFifyMesh weights color, normal, and opacity supervision according to signed distance, then predicts the three quantities with separate networks. The weighting prioritizes interior samples near the surface.

1) Importance Weighting:

  • Distance-based weights modulate color, normal, and opacity supervision because interior samples farther from the surface are less critical and exterior points have zero opacity.The scheme is inspired by Davies et al..Source: 1) Importance Weighting · p. 3
  • With τ = 0.1, the weighting scheme prioritizes interior colors and opacities closer to the mesh surface as weights decay with distance from it.The threshold τ controls the decay rate of weights for signed distances within the mesh.Source: 1) Importance Weighting · p. 3

B. Network Architecture

  • Color, point normals, and opacity are predicted by three separate networks optimized with separate loss functions and trained simultaneously.Input points query multi-resolution surface voxel grids with hash-encoded features for opacity prediction, while positional encoding feeds the color and normal networks.Source: B. Network Architecture · p. 3, Figure 2 · p. 3

1) Color and Point Normal Networks:

Color and point normals are predicted by encoded-input SIREN networks, while opacity is learned with a mesh-initialized multiresolution hash architecture.

1) Color and Point Normal Networks:

  • Positional encoding feeds separate SIREN decoders for color and point normals, capturing high-frequency variation across the mesh.The decoders are fθ and fϕ, respectively.Source: 1) Color and Point Normal Networks · p. 3

2) Opacity Network:

  • Surface samples initialize multiresolution voxel grids whose 2D vertex latents are queried using the eight corners surrounding each input point.The corners serve as hash keys for retrieving the corresponding latent vectors.Source: 2) Opacity Network · p. 3
  • Trilinear interpolation combines retrieved latents across grid levels, and a feature decoder followed by a density network predicts opacity.Points outside a grid use a predefined default latent vector.Source: 2) Opacity Network · p. 4
  • Latents are assigned only to grid points in occupied surface voxels, while exterior points use a default latent; saved voxel indices avoid retaining the original mesh.This mesh-informed assignment minimizes trainable parameters and helps distinguish surface regions from empty space.Source: 2) Opacity Network · p. 4

C. Loss Functions

The color and normal networks use weighted losses, while opacity training combines weighted binary cross entropy with latent-vector regularization; rendering integrates predictions along camera rays.

C. Loss Functions

  • Color prediction is trained with weighted mean squared error between predicted and ground-truth colors.The weights are those defined in Eq. (2).Source: C. Loss Functions · p. 4
  • Normal prediction uses a weighted mean squared cosine-similarity loss.The weights are those defined in Eq. (2).Source: C. Loss Functions · p. 4
  • Opacity training combines weighted binary cross entropy with a regularizer that penalizes trainable latent-vector magnitudes to prevent overfitting.The total opacity loss is the weighted sum of BCE and regularization losses.Source: C. Loss Functions · p. 4

D. Rendering

  • Rendering samples points along camera rays, evaluates the model, and integrates predicted per-sample opacities and colors into pixel colors.Unlike density-predicting NeRF methods, NeRFifyMesh predicts opacity directly, which the paper notes can be derived simply from SDFs.Source: D. Rendering · p. 4

IV. EXPERIMENTS

Across three mesh datasets, NeRFifyMesh has the best reported LPIPS, competitive SSIM, and lower PSNR than image-trained NeRF baselines. The experiments also assess visual detail and describe the training setup and dataset scope.

A. Implementation Details

  • Each mesh contributes 2 million training samples; training runs for 300,000 iterations and takes roughly two hours per model on an NVIDIA P100 GPU.The batch size is 4096 samples.Source: A. Implementation Details · p. 4

B. Mesh Scene Rendering

  • Evaluation uses 25 objects across GSO, Poly Haven, and Objaverse, with image-trained baselines trained on 100 rendered views per mesh and tested on 20.The rendered training and test images use 1024×1024 resolution and spherical-coordinate camera poses.Source: B. Mesh Scene Rendering · pp. 4–5
  • Qualitative comparisons show NeRFifyMesh rendering fine mesh details with high fidelity and visually outperforming Instant-NGP and NeRF.The comparison is shown in Fig. 3.Source: B. Mesh Scene Rendering · p. 5
  • NeRFifyMesh has the lowest LPIPS across all three datasets, SSIM above NeRF but below Instant-NGP, and lower PSNR than both image-trained baselines.The comparison covers test-image rendering quality; the paper relates the PSNR difference to the methods’ distinct training objectives.Source: Table I · p. 5, B. Mesh Scene Rendering · p. 5

C. Applications

The applications section evaluates NeRFifyMesh for mesh compression and surface-normal estimation, including relighting with predicted normals. The reported results indicate compression against baselines, low normal error across three datasets, and realistic lighting effects compared with ground truth.

1) Mesh Compression:

  • NeRFifyMesh achieves significant compression ratios compared with baselines across the tested datasets.The comparison includes average mesh size (geometry, texture, and material files) and representation size (module weights, hash-table keys, and the reconstruction bounding box).Source: 1) Mesh Compression · p. 5, Table II · p. 5

2) Surface Normal Estimation:

  • Predicted surface normals have low RMSAE across three datasets, supporting the accuracy of the SIREN network with positional encoding.The method explicitly trains a module on ground-truth geometry rather than relying on density gradients, which can yield noisy and unreliable estimates.Source: 2) Surface Normal Estimation · p. 5, Table III · p. 6
  • Blinn-Phong relighting with predicted normals produces realistic lighting effects compared with ground-truth mesh renders.The comparison adjusts ambient, diffuse, and specular lighting components.Source: 3) Relighting · p. 6, Figure 4 · p. 6

3) Relighting:

NeRFifyMesh supports mesh-surface extraction and recovery, then composes transformed models into unified scenes or alongside existing NeRFs. Its reconstructions outperform NeRF and Instant-NGP on all four tested objects and NeuS on three.

3) Relighting:

  • Alpha threshold 0.95 selects high-gradient surface points and predicted normals for Poisson surface reconstruction.The resulting meshes support geometry-dependent applications such as collision interactions.Source: 4) Surface Recovery · p. 6

4) Surface Recovery:

  • NeRF and Instant-NGP produce noisy reconstructions with floating artifacts; mesh sampling preserves Conch interior features missed by all three compared methods.Limited camera-view visibility leaves the shell interior poorly reconstructed by NeRF, Instant-NGP, and NeuS.Source: Figure 5 · p. 6, 4) Surface Recovery · p. 6

3) Relighting:

  • NeRFifyMesh outperforms NeRF and Instant-NGP on Chamfer Distance for all four objects, and NeuS on three of four.The comparison uses Chamfer Distance for extracted surfaces.Source: Table IV · p. 6, 3) Relighting · p. 6

5) Scene Composition:

  • Unit-cube-normalized models are placed via inverse pose transforms, then depth-ordered back-to-front RGB blending composes unified or hybrid NeRF scenes.The pipeline resolves per-pixel occlusions and supports adding NeRFifyMesh models to standard NeRF scenes.Source: 5) Scene Composition · p. 6

D. Simulation Experiments

Collision simulations using NeRFifyMesh models more closely match ground truth than Instant-NGP simulations on the reported visual and quantitative measures. The paper also demonstrates simulations combining mesh-derived objects with static NeRF scenes.

D. Simulation Experiments

  • The simulation pipeline uses PyBullet with extracted meshes for collision geometry, then renders and composites objects using poses from the physics engine.The comparison uses original meshes for ground-truth collision geometry and mesh-based visualization; evaluation measures frame PSNR and trajectory position and rotation errors.Source: D. Simulation Experiments · pp. 6–7
  • Rendering examples include a scene formed from eight NeRFifyMesh objects and a croissant model integrated with NeRF Garden.The examples show RGB and depth renderings.Source: Figure 6 · p. 7
  • A football dropped 0.78 m onto a table demonstrates simulation of a mesh-derived object inside a static NeRF Garden scene.The static scene's collision mesh is extracted with Marching Cubes and processed to remove floating artifacts.Source: D. Simulation Experiments · p. 7
  • Both objects achieve higher PSNR and lower position and rotation trajectory errors than Instant-NGP in collision simulations.The comparison is reported in Table V and supported by visual comparison with ground truth.Source: Table V · p. 7, D. Simulation Experiments · p. 7

E. Limitations

Mesh quality and topology constrain the method: high face counts complicate even sampling, and non-watertight meshes produce inaccurate SDFs and poor renders. The conclusion emphasizes scene composition and robotics simulation applications.

V. CONCLUSION

  • Collision demonstrations include two GSO toys and a Poly Haven football dropped into a static NeRF Garden scene.The football is dropped from 0.78 m above the table.Source: Figure 7 · p. 7, Figure 8 · p. 7

E. Limitations

  • High face counts complicate even face-weighted sampling, while non-watertight meshes yield inaccurate SDFs, radiance-field errors, artifacts, and noisy renders.The illustrated non-watertight scan produces a poor-quality render.Source: E. Limitations · p. 7, Figure 9 · p. 7

V. CONCLUSION

  • NeRFifyMesh converts sampled mesh geometry into a radiance field with point normals and supports unified scenes and robotics simulations.The conclusion frames complex environments built from reusable assets as an application.Source: §V · p. 7
Loading 2610.10387v1…