Source-linked AI summary
SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields
Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G. Derpanis, Jonathan Kelly, Marcus A. Brubaker, Igor Gilitschenski, Alex Levinshtein
TL;DR
3D inpainting must remove objects while producing perceptually plausible, view-consistent, and geometrically valid scene content. The paper combines sparse-click multiview segmentation with perceptual NeRF optimization using 2D inpainters, and introduces a benchmark with object-free ground truth; it reports superiority over alternative 2D and 3D approaches.
Problem
NeRF scene editing, especially object removal, requires manipulating an entangled representation while maintaining perceptual realism and consistency across views.
Method
The method lifts sparse user annotations into a 3D-consistent mask, applies a pretrained 2D inpainter to multiview images, and fits an inpainted NeRF using perceptual and geometric guidance.
Results
The method outperforms alternative 2D and 3D inpainting approaches and all evaluated 2D and 3D-consistent multiview segmentation baselines.
Takeaways & Limitations
The introduced dataset provides corresponding object-free ground-truth views for more principled quantitative evaluation of multiview scene inpainting.
Takeaways & Limitations
Performance is constrained by limited camera movement during data collection, and refining depths can reduce performance because of noisy NeRF geometries.
Abstract
from arXiv · showhide
Neural Radiance Fields (NeRFs) have emerged as a popular approach for novel view synthesis. While NeRFs are quickly being adapted for a wider set of applications, intuitively editing NeRF scenes is still an open challenge. One important editing task is the removal of unwanted objects from a 3D scene, such that the replaced region is visually plausible and consistent with its context. We refer to this task as 3D inpainting. In 3D, solutions must be both consistent across multiple views and geometrically valid. In this paper, we propose a novel 3D inpainting method that addresses these challenges. Given a small set of posed images and sparse annotations in a single input image, our framework first rapidly obtains a 3D segmentation mask for a target object. Using the mask, a perceptual optimizationbased approach is then introduced that leverages learned 2D image inpainters, distilling their information into 3D space, while ensuring view consistency. We also address the lack of a diverse benchmark for evaluating 3D scene inpainting methods by introducing a dataset comprised of challenging real-world scenes. In particular, our dataset contains views of the same scene with and without a target object, enabling more principled benchmarking of the 3D inpainting task. We first demonstrate the superiority of our approach on multiview segmentation, comparing to NeRFbased methods and 2D segmentation approaches. We then evaluate on the task of 3D inpainting, establishing state-ofthe-art performance against other NeRF manipulation algorithms, as well as a strong 2D image inpainter baseline. Project Page: https://spinnerf3d.github.io
1. Introduction
The paper presents SPIn-NeRF as an integrated framework for interactive multiview segmentation and perceptually plausible 3D inpainting, addressing view consistency and geometric validity. It also introduces a benchmark dataset with object-free ground-truth views.
- Motivation: 3D inpainting must produce perceptually realistic replacements that remain consistent across views and geometrically valid.NeRF representations are entangled and difficult to manipulate, adding challenges beyond 2D image inpainting.
- Approach: The framework uses sparse user annotations, multiview images, and NeRF fitting to recover a 3D mask and replace the target object with plausible appearance and geometry.It combines interactive segmentation with full 3D inpainting in one framework.
- Approach: A semantic NeRF lifts masks initialized by a video-based model into coherent 3D segmentation, while perceptual optimization distills 2D inpainter outputs into a view-consistent NeRF.Inpainted depth images regularize the geometry of the masked region.
- Approach: The method provides a complete pipeline from minimal-burden object selection through novel-view synthesis of the inpainted scene.The pipeline is presented as a unified process rather than separate editing stages.
- Evaluation: The authors introduce a dataset containing challenging real-world scenes with corresponding object-free views as ground-truth inpaintings.These paired views support quantitative comparison of scene inpainting methods.
2. Related Work
Prior work covers 2D image inpainting and several forms of NeRF manipulation, but existing methods are limited for general, cluttered 3D scene inpainting. SPIn-NeRF builds on 2D inpainters while addressing multiview inconsistency.
- Image Inpainting: 2D inpainting methods have progressed from patch-based techniques to neural approaches targeting perceptual realism and reconstruction quality.Research directions include adversarial training, architectural advances, pluralistic outputs, multiscale processing, and perceptual metrics.
- NeRF Manipulation: NeRF manipulation research has explored training efficiency, rendering speed, reconstruction quality, data requirements, and scene editing.These efforts establish NeRFs as a basis for broader scene manipulation tasks.
- NeRF Manipulation: Existing NeRF manipulation methods generally target simple objects or tasks such as recoloring and deformation rather than general inpainting in cluttered textured scenes.Their reported scope is narrower than full scene completion after object removal.
- NeRF Manipulation: NeRF-In uses geometry and radiance priors from 2D inpainters but does not address inconsistency between views through its simple pixelwise loss.SPIn-NeRF instead focuses on transferring 2D information into a view-consistent 3D representation.
3. Background: Neural Radiance Fields
NeRF represents a scene as a function mapping spatial coordinates and viewing directions to color and density. Volumetric rendering integrates these predictions along camera rays, and training minimizes reconstruction error against observed colors.
- Scene Representation: A NeRF function f maps a 3D coordinate x and view direction d to color c and density σ.The function provides the scene representation used for rendering.
- Volumetric Rendering: Volumetric rendering divides each ray between near and far bounds into N sections and samples points to estimate its expected color.The rendering procedure uses quadrature over sampled locations along the ray.
- Volumetric Rendering: The rendered ray color depends on each sampled point’s color, density, and transmittance along the ray.The distance between adjacent samples is represented by δ_i, while c_i and σ_i denote color and density.
- Optimization: Training optimizes a reconstruction loss using ground-truth colors for rays passing through pixels in the training views.The ray batch R is sampled from the training views.
4. Method
The method first recovers 3D-consistent object masks from sparse single-view annotations, then fits an inpainted NeRF using 2D appearance and depth priors while addressing view inconsistency and computational cost.
- Multiview Segmentation: The segmentation NeRF predicts density, color, and objectness, and combines RGB reconstruction with classification supervision to obtain 3D-consistent masks.Objectness logits are rendered along rays, while detached color-rendering weights limit classification updates to the logits.
- Multiview Segmentation: Sparse annotations initialize masks through interactive and video segmentation, which are refined into coherent multiview masks with a semantic NeRF.The semantic NeRF resolves boundary inaccuracies and inconsistencies caused by treating nonadjacent training views as a video sequence.
- Multiview Inpainting: Independent RGB inpainting supplies appearance priors, while depth rendered from an object-present NeRF is itself inpainted to regularize the completed scene’s geometry.The resulting appearance and geometry priors supervise fitting a new NeRF without the undesired object.
- Multiview Inpainting: A perceptual loss optimizes masked regions while reconstruction loss preserves unmasked regions, reducing blur from inconsistent independently inpainted views.Depth guidance also helps prevent degenerate geometries such as fog forming near cameras.
- Patch-based Optimization: Patch-based optimization renders small masked-region patches instead of full views, reducing memory demands and enabling optimization on high-resolution images.Only patches inside the object-mask bounding box are used for the perceptual computation.
5. Experiments
Experiments evaluate multiview segmentation and 3D inpainting on real-world scenes, using quantitative metrics, qualitative comparisons, and a new benchmark with object-free test views. The method outperforms evaluated baselines, while performance depends on geometry guidance, input-view count, and mask refinement.
- Datasets and metrics: The benchmark contains 10 real-world forward-facing scenes with human-annotated object masks and test views captured without the unwanted object.These object-free views provide ground-truth captures for quantitative multiview inpainting evaluation.
- Datasets and metrics: Segmentation is evaluated with pixel-wise accuracy and IoU, while inpainting uses masked-region LPIPS and FID between ground-truth test views and model outputs.LPIPS and FID are calculated inside the object-mask bounding box.
- Multiview segmentation: Our multiview segmentation model outperforms both 2D and 3D-consistent baselines, with two-stage optimization further improving the masks.The task transfers a source mask to other views, and qualitative results show reduced noise, sharper edges, and improved consistency over video segmentation and NVOS.
- Multiview inpainting: Our inpainting method significantly outperforms alternative 2D and 3D approaches when given human-annotated masks, and geometry guidance improves scene quality.The method also outperforms the 2D inpainter by combining multiview information with perceptual priors.
- End-to-end evaluation: Using predicted multiview segmentation masks causes only a subtle quality decrease, while the complete system still outperforms methods fitted with human-annotated segmentations.This combines the segmentation and inpainting stages with minimal user interaction.
- Ablations and sensitivity: Inpainting quality degrades with fewer input views, and refinement gives a small but significant boost because smaller masks require less hallucination from the inpainter.On the dataset, refinement reduces mean masked area by 4.74%; noisy NeRF geometry makes depth refinement harmful, while colour refinement performs best.
6. Conclusion
The paper presents view-consistent NeRF inpainting using image and geometric priors, alongside multiview segmentation from sparse pixel clicks. It also identifies semantically consistent image priors as the main limitation and introduces a challenging benchmark dataset.
- The method enforces viewpoint-consistent NeRF inpainting using image and geometric priors from a single-view object mask.
- Sparse pixel-level clicks on and around an unwanted object are translated into a 3D mask renderable from novel views.
- The experiments evaluate the effectiveness of both the multiview segmentation and inpainting methods.
- The main limitation is the assumption that image priors are semantically consistent and may differ only in texture.
- The introduced dataset addresses the lack of challenging benchmarks for multiview inpainting and is intended to support future research.
B. Refinement Details
The refinement process projects source-view masked pixels into target views and replaces them with valid color and depth information when visibility and geometry checks pass. Iteration across views reduces the masked region while preserving source-view depth consistency.
- Mask refinement projects sampled source-ray points into target views to find unmasked pixels that can replace source-view masked pixels.
- Refinement lowers the masked area and reduces hallucination needs, supporting better inpaintings.
- A candidate point is discarded when its target pixel is masked or its target depth is inconsistent with the point’s distance to the target camera.
- Valid candidates replace the source RGB value with target-view color and update source depth using the point’s distance to the source camera.
- The process iterates until no pixels change and updates pixels with closer valid points, making results independent of target-view order.
- A source pixel is refined only when its new depth agrees with at least one of eight neighboring source-view pixels.
C. Additional Details
Implementation uses fixed loss weights, efficient NeRF components, sparse depth supervision, and forward-facing scenes captured along unstructured camera trajectories. The dataset emphasizes settings where limited visibility increases hallucination demands.
- The implementation sets λLPIPS to 0.01 and λdepth to 1, and trains on a single Nvidia RTX A6000 GPU.
- Sparse depth supervision in unmasked input regions is used to obtain more accurate scene geometries.
- The dataset contains forward-facing scenes captured by manually moving a camera along an unstructured trajectory resembling nonexpert use.
- Forward-facing scenes are more challenging because fewer views reveal regions behind objects, increasing the need for hallucination.
- All dataset images are jointly processed with Colmap to recover camera parameters in a shared coordinate system, with images sized 2268 × 4032 pixels.
- The semantic NeRF typically reaches acceptable geometry halfway through fitting, while later iterations mainly sharpen appearance.
D. Additional Qualitative Results
Additional experiments show view-consistent inpainting, coherent multiview segmentation, sharper outputs than NeRF-In, and little benefit from more than two training stages. Qualitative examples include plausible completion of unseen geometry.
- Figure 10 adds four qualitative examples of the proposed view-consistent inpainting approach from novel views.
- When parts of a statue base and ground were unseen during training, the model produced consistent plausible hallucinations completing the base’s cylinder shape.
- The perceptual loss produces a sharp grass texture in the inpainted result.
- The segmentation model consistently produces coherent masks with sharp, accurate edges across qualitative comparisons.
- Compared with NeRF-In on three dataset scenes, the proposed model produces sharper outputs.
- Two or three training stages have similar performance, indicating that two stages are sufficient.
F. Our Multiview Inpainting Dataset
The introduced benchmark contains challenging real-world scenes for quantitative evaluation of multiview inpainting, with the proposed approach outperforming other methods.
- The dataset contains 10 real-world scenes with challenging 3D inpainting segmentation and inpainting scenarios.
- The dataset is used for quantitative comparison of the proposed inpainting method against baselines.
- The proposed approach outperforms other methods on this dataset.
H. Detailed Segmentation Results
The segmentation evaluation separates forward-facing and 360° scenes, finding that the proposed model consistently outperforms the baselines in both settings.
- The evaluation breaks down multiview segmentation results into forward-facing and 360° scenes.
- The task is more challenging for 360° scenes because the single-view mask must be extrapolated to farther views.
- The proposed model consistently outperforms the baselines in both forward-facing and 360° scenarios.
I. Failure Cases
The reported failure cases arise from dependencies on the underlying NeRF and 2D inpainter, especially when independently generated views contain semantic inconsistencies.
- SPIn-NeRF inherits failure cases from its underlying NeRF and image inpainter.
- Semantic inconsistencies, such as independently inpainted views inserting novel objects, can produce blurry or artifact-filled masked regions.
- The method uses LaMa because its object-removal behavior reduces the likelihood of problematic semantic inconsistencies.
- Three-stage optimization does not significantly improve results over two-stage fitting.