Source-linked AI summary
NeRF-Editing: Geometry Editing of Neural Radiance Fields
Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, Lin Gao
TL;DR
NeRF enables realistic novel-view synthesis but existing editing methods do not provide user-controlled shape deformation. This paper transfers mesh edits into a NeRF through a tetrahedral proxy, producing edited novel views without retraining and supporting synthetic and real scenes.
Problem
NeRF-based editing methods are limited in their ability to perform user-controlled shape deformation, while accurate explicit scene geometry is difficult to obtain for real scenes.
Method
The method extracts and deforms a triangular mesh, transfers its deformation through a tetrahedral mesh proxy, and propagates it into the implicit NeRF representation.
Results
The framework produces user-controlled shape edits and novel-view renderings on synthetic data and real captured objects, including complex and local deformations.
Takeaways & Limitations
The approach combines mesh-based editing with NeRF rendering to support geometric editing of general scenes from arbitrary views.
Takeaways & Limitations
The method cannot modify color, lighting, or shadows according to the edit and does not support real-time user editing.
Abstract
from arXiv · showhide
Implicit neural rendering, especially Neural Radiance Field (NeRF), has shown great potential in novel view synthesis of a scene. However, current NeRF-based methods cannot enable users to perform user-controlled shape deformation in the scene. While existing works have proposed some approaches to modify the radiance field according to the user's constraints, the modification is limited to color editing or object translation and rotation. In this paper, we propose a method that allows users to perform controllable shape deformation on the implicit representation of the scene, and synthesizes the novel view images of the edited scene without re-training the network. Specifically, we establish a correspondence between the extracted explicit mesh representation and the implicit neural representation of the target scene. Users can first utilize well-developed mesh-based deformation methods to deform the mesh representation of the scene. Our method then utilizes user edits from the mesh representation to bend the camera rays by introducing a tetrahedra mesh as a proxy, obtaining the rendering results of the edited scene. Extensive experiments demonstrate that our framework can achieve ideal editing results not only on synthetic data, but also on real scenes captured by users.
1. Introduction
NeRF offers realistic novel-view synthesis but is difficult to edit, while explicit meshes are easier to deform yet often inaccurate for real scenes. The paper bridges these representations to enable user-controlled geometric deformation and novel-view rendering.
- NeRF supports strong novel-view synthesis, but its implicit representation is difficult for users to edit or modify.
- Explicit meshes support established deformation methods, but reconstructing accurate meshes from sparse real-scene images remains difficult.MVS-based reconstructions may be poor and produce unrealistic novel-view rendering.
- Existing NeRF editing methods mainly support color changes, object translation, or rotation rather than shape deformation.Dynamic-NeRF approaches may also restrict edits to human bodies or learn motion only from recorded videos.
- The framework targets user-controlled deformation of general-scene geometry, including animal models and man-made objects.
- The proposed method extracts a triangular mesh from a trained NeRF, lets users deform it, and transfers that deformation through a tetrahedral proxy into the implicit field.Tetrahedral interpolation completes propagation from a discrete deformation field to a continuous one.
2. Related Work
Related work spans explicit and implicit neural rendering, prior NeRF editing, and established 3D deformation methods. The paper positions its contribution at their intersection: mesh-based geometric editing combined with NeRF rendering.
- Positioning: The proposed framework combines explicit representations for local editing with implicit representations for realistic rendering effects.
- Novel view synthesis: Novel-view synthesis methods use explicit or implicit scene representations, with neural rendering applied across representations including meshes and point clouds.
- NeRF editing: Prior NeRF editing can alter image appearance or duplicate, move, and rotate objects, but does not support geometric shape changes.
- 3D deformation and editing methods: Explicit 3D deformation methods include Laplacian, Poisson, dual-Laplacian, ARAP, proxy-based, and data-driven approaches.ARAP preserves local detail by maintaining rigidity of local transformations.
3. Our Method
The method connects an editable explicit mesh to NeRF’s implicit representation, transfers mesh deformations through a tetrahedral proxy, and bends rays to render edited novel views.
- 3. Our Method: NeRF’s implicit representation supports realistic novel-view synthesis but is difficult to edit directly, motivating an editable explicit mesh representation.The method uses NeRF for rendering and a mesh for intuitive scene-content editing.
- 3.3. Deformation Transfer to Discrete Volume: A tetrahedral mesh wraps the surface mesh and transfers its deformation into a discrete volumetric deformation field using ARAP constraints and barycentric coordinates.The tetrahedral cage is offset from the surface, and each surface vertex is associated with the tetrahedron containing it.
- 3.2. Editing of Explicit Surface Mesh Representation: An explicit triangular mesh is extracted from the scene representation, with NeuS’s neural SDF reconstruction used to obtain a satisfactory editing surface.The mesh is extracted from the SDF zero-level set rather than directly relying on the often rough Marching Cubes mesh.
- 3.2. Editing of Explicit Surface Mesh Representation: Users deform the triangular mesh with ARAP, which treats user edits as boundary conditions and minimizes an energy measuring mesh rigidity.The deformation energy sums distortion over cells formed by each vertex and its 1-ring neighbors; local rotations and vertex positions are alternately optimized.
- 3.3. Deformation Transfer to Discrete Volume: During rendering, sampled ray points are mapped through corresponding tetrahedra to bend rays inside the deformed volume, while points outside it remain unchanged.The transformed samples’ density and RGB values are then accumulated with NeRF volume rendering to produce pixel colors.
4. Experiments and Evaluations
Experiments evaluate the framework on synthetic and real scenes, showing controllable edits across views, comparisons against baselines, robustness to mesh quality, and limitations in rendering and interactivity.
- Datasets and metrics: The evaluation covers synthetic datasets, including Mixamo, the Lego bulldozer, and chair, plus a real horse statue and self-captured scenes.Quantitative evaluation primarily uses Mixamo, where edited ground truths can be rendered; real scenes are evaluated mainly qualitatively because edited scenes do not physically exist.
- Editing Results: The framework supports diverse shape edits across views, including lowering a bulldozer shovel, stretching chair parts, posing and scaling a giraffe, and deforming dragon wings.Additional results include differently placed and deformed copies of a horse statue, while deformation transfer applies existing motion sequences to captured objects.
- Comparisons: The “Closest Point” and “3NN” baselines can produce discontinuities and obvious artifacts, whereas the proposed two-step deformation transfer gives more visually satisfactory results and quantitative advantages.The comparison includes synthetic Mixamo results with ground truth and a real giraffe example without ground truth.
- Comparisons: Volume rendering makes the method more tolerant of imperfect reconstructed meshes than direct mesh rendering, whose mesh artifacts appear in the rendered images.On Lego data, the reconstructed mesh is not high quality, yet the proposed method still produces desired edited results.
- Ablation Study: Editing the triangular mesh before deformation transfer performs better than directly editing the tetrahedral mesh, while mesh quality has little overall effect on the results.The NeuS mesh performs slightly better than the NeRF mesh, but the difference is small; tetrahedral-mesh editing introduces rendering artifacts.
- Limitations: The method cannot modify color, lighting, or shadows after deformation and does not support real-time user editing because NeRF rendering remains the main bottleneck.Users can select a viewing angle for image synthesis only after editing the mesh representation.
5. Conclusion
The paper introduces a method for user-controlled shape deformation of general-scene NeRF geometry by linking explicit meshes with implicit volume representations. It enables arbitrary-view visualization while supporting detailed edits across diverse scene objects.
- The method establishes correspondence between explicit mesh and implicit volume representations, transferring triangular-mesh deformations to the NeRF.This combines mesh-based editing with NeRF novel-view synthesis without requiring network retraining.
- The framework supports user-controlled deformation of general real scenes, including human bodies, animals, and man-made models.The paper describes this as the first method supporting user-controlled shape deformation in NeRF geometry.
- Users can visualize edited results from arbitrary views using NeRF's novel-view synthesis capability.
- Compared with previous NeRF editing methods, the approach offers greater editing freedom and supports detail-level scene edits.