PaperlayerView exact revision on arXiv

Source-linked AI summary

Decomposing NeRF for Editing via Feature Field Distillation

Sosuke Kobayashi, Eiichi Matsumoto, Vincent Sitzmann

arXiv:2205.15585v2cs.CVcs.GR

TL;DR

Editing NeRF scenes is difficult because their connectionist representations are not object-centric or compositional. DFFs distill pretrained 2D features into queryable 3D feature fields, enabling segmentation and selective editing without retraining, with quantitative segmentation evaluations and multi-view-consistent editing results.

  • Problem

    NeRF representations are not object-centric or compositional, making selective editing of specific scene regions or objects difficult.

  • Method

    DFFs distill pretrained image feature encoders into 3D feature fields that decompose scenes from text or image-patch queries without retraining.

  • Results

    DFFs achieve quantitative 3D segmentation evaluations and enable multi-view-consistent appearance and geometry edits across real-world NeRF scenes.

  • Takeaways & Limitations

    Query-conditioned DFF segmentation supports interactive, resolution-independent region selection for selective NeRF editing.

  • Takeaways & Limitations

    DFF performance is bounded by the teacher encoder and can receive noisy supervision from geometrically incorrect NeRF reconstructions.

Abstract

from arXiv · show

Emerging neural radiance fields (NeRF) are a promising scene representation for computer graphics, enabling high-quality 3D reconstruction and novel view synthesis from image observations. However, editing a scene represented by a NeRF is challenging, as the underlying connectionist representations such as MLPs or voxel grids are not object-centric or compositional. In particular, it has been difficult to selectively edit specific regions or objects. In this work, we tackle the problem of semantic scene decomposition of NeRFs to enable query-based local editing of the represented 3D scenes. We propose to distill the knowledge of off-the-shelf, self-supervised 2D image feature extractors such as CLIP-LSeg or DINO into a 3D feature field optimized in parallel to the radiance field. Given a user-specified query of various modalities such as text, an image patch, or a point-and-click selection, 3D feature fields semantically decompose 3D space without the need for re-training and enable us to semantically select and edit regions in the radiance field. Our experiments validate that the distilled feature fields (DFFs) can transfer recent progress in 2D vision and language foundation models to 3D scene representations, enabling convincing 3D segmentation and selective editing of emerging neural graphics representations.

1 Introduction

NeRFs support high-quality reconstruction and novel-view synthesis but are difficult to edit selectively because their representations are not object-centric. DFFs address this by distilling pretrained 2D features into queryable 3D fields for semantic decomposition and local editing.

  • NeRF recovers continuous density and radiance from limited observations, enabling high-quality images from arbitrary views.
  • Editing NeRF scenes is difficult because MLPs and voxel grids encode scenes implicitly rather than as object-centric, compositional structures.
  • DFFs distill pretrained image features into scene-specific 3D feature fields that support text- or image-query decomposition without retraining.
  • LSeg- and DINO-based DFFs select 3D regions through text or image-patch queries, respectively.
  • Experiments evaluate DFFs through quantitative 3D segmentation and qualitative appearance and geometry edits on real-world NeRF scenes.

2 Related Work

Prior methods impose scene, category, annotation, or representation constraints that limit flexible editing. DFF instead decomposes scene-specific NeRFs into arbitrary semantic units using text and image queries without retraining.

  • Semantic supervision can add object volumes to NeRFs, but requiring view-consistent ground-truth instance masks makes annotation expensive.
  • Category-specific and generative methods support decomposition and editing mainly in constrained domains with large datasets.
  • Voxelgrids, octrees, and unsupervised decompositions offer localized editing but impose inflexible boundaries or strong scene assumptions.
  • Domain-specialized pipelines and independently accessible object data constrain the domains or segmentation types available during or before training.
  • DFF decomposes scene-specific NeRFs into arbitrary semantic units through text and image queries without retraining.
  • Zero-shot 3D perception remains limited by insufficient effective, efficient, high-resolution architectures and large annotated datasets.

3 Preliminaries

NeRF maps 3D coordinates and viewing directions to density and color for volume-rendered novel views. Zero-shot segmentation instead aligns image features with open-set text or other queries, motivating feature-field distillation.

  • NeRF uses an MLP to map a 3D point and view direction to density and view-dependent color.
  • NeRF is optimized on posed images by minimizing a rerendering loss after volume rendering sampled points along camera rays.
  • Closed-set segmentation models predefine labels, whereas zero-shot segmentation predicts target regions from open-set queries.
  • Figure 1 depicts DFF outputs as density, color, and feature fields trained from rendered-feature and photometric differences.
  • LSeg aligns pixel-level image features with text-query features and computes label probabilities from their dot product followed by a softmax.
  • The same feature-space similarity formulation supports image, patch, or pixel queries, while LSeg uses a pretrained CLIP text encoder.

4 Distilled Feature Fields

DFF extends NeRF with a distilled 3D feature field that supports query-based, multi-view-consistent segmentation and local editing without retraining. The method combines a feature objective with NeRF’s photometric objective and supports transformations of selected regions.

  • Feature-field distillation: DFF adds a 3D feature branch to NeRF, modeling a semantic feature vector f(x) alongside density and view-dependent color.The feature field is supervised by a pretrained pixel-level image encoder through teacher-student distillation.
  • Feature-field distillation: The feature field is optimized by matching rendered features to image-encoder features while retaining NeRF’s photometric loss.The total objective combines photometric loss Lp and feature loss Lf; the feature-loss weight λ is set to 0.04, with density stop-gradient during feature rendering.
  • Query-based decomposition: DFF performs zero-shot 3D segmentation by comparing f(x) with query features and applying a softmax over candidate labels.This supports open-set text or other feature queries rather than a fixed closed-set semantic classifier.
  • Query-based decomposition: Because segmentation depends only on 3D coordinates and the query, users can change selections without retraining while preserving multi-view consistency.This query-conditioned field is computed at arbitrary 3D points and can be used with radiance-field volume rendering.
  • Editing: Selected regions can undergo geometric transformations by blending query-dependent source and transformed densities and colors during rendering.The formulation uses p(l|x) to route unselected content through the original field and selected content through the inverse-transformed field.
  • Evaluation and extensions: DFF outperforms the supervised point-cloud segmentation baseline MinkowskiNet42 on Replica semantic segmentation.The paper also reports that its decomposition enables selective updates for optimization-based editing methods such as CLIPNeRF.

5 Experiments

Experiments evaluate DFFs for semantic decomposition, segmentation, novel-view synthesis, and query-based editing. The results show strong segmentation and editable, multi-view-consistent scene manipulation while preserving view-synthesis quality under the standard feature-loss setting.

  • Experimental setup: DFFs are evaluated on Replica semantic segmentation using LSeg and on real-world editable novel-view synthesis using LSeg and DINO teachers.The benchmark uses four Replica scenes with annotated point clouds; editing experiments include LLFF and a collected dataset.
  • 3D Semantic Segmentation: LSeg-DFF achieves promising 3D semantic segmentation performance, even surpassing a supervised point-cloud model trained on ScanNet.The comparison uses MinkowskiNet42 as the point-cloud baseline.
  • Compatibility with View Synthesis: DFF preserves novel-view synthesis quality without degradation and requires only small computational and parameter overhead compared with the original NeRF.Increasing the feature-loss weight by 10× harms view synthesis without further improving segmentation.
  • Editable Novel View Synthesis: Text queries with LSeg-DFF and image-patch queries with DINO-DFF support convincing selective appearance edits of specific 3D regions.DINO patch queries can select semantically similar areas at once because its features capture region similarity and correspondences.
  • Editable Novel View Synthesis: Naive NeRF hidden-layer features do not robustly decompose complex scenes because selections leak to visually or spatially related regions and fail to fully cover targets.The observed feature space is entangled with low-level factors such as color and spatial adjacency.
  • Editable Novel View Synthesis: DFF produces plausible, multi-view-consistent object extraction and deletion, unlike the image-inpainting baseline, though deleted regions may retain floating artifacts and distant blurred volumes.The method particularly succeeds at extracting foreground objects.

6 Discussion, Limitations, and Conclusions

The paper presents DFF as a method for NeRF scene decomposition and selective editing, while identifying limitations from distillation, reconstruction geometry, and potential misuse.

  • Conclusions: DFF provides a method for NeRF scene decomposition and selective editing, with quantitative segmentation and qualitative editable novel-view evaluations.The paper also discusses future directions involving smoothness priors, unobserved regions, and applications such as 3D registration and robot teaching.
  • Limitations: Distillation limits DFF performance because students cannot largely outperform teachers, and low-resolution or weak teacher features produce coarse or unsuccessful decompositions.The limitation applies to both feature resolution and the teacher’s ability to understand queries.
  • Limitations: NeRF geometry errors, including floaters, can make the volume-rendered supervision used by DFF noisy.
  • Societal impact: Editing NeRFs can potentially be misused to create realistic fake content, while NeRF computation can entail high electricity usage.The paper mentions automatic fake detection and efficient NeRF methods as possible mitigations.

Checklist

The checklist reports that the paper addresses claims, limitations, and societal impacts, but does not provide complete reproduction code, error bars, or fully tracked compute totals.

  • Claims and ethics: The authors state that the paper’s main claims are empirically demonstrated and that its limitations and potential negative societal impacts are discussed.
  • Reproducibility: Complete code for reproducing all experimental results is not publicly available, although the authors state that reproduction is easier because the implementation modifies public code.The scene dataset is planned for public release.
  • Experimental reporting: Error bars are not reported because they would be computationally expensive and the authors expect the results to be stable.The checklist notes that most existing NeRF studies also have not reported error bars.
  • Compute reporting: The authors did not completely track and sum total experimental compute, instead reporting the setup of the main experiments.
  • Assets and data: The paper reports no human-subject or personal-data concerns for its experiments and states that existing assets are appropriately cited.

A Training and Model Architectures

The implementation trains the radiance and feature fields with specified optimization and sampling settings, then uses thresholded DFF-query similarity for hard or probabilistic selection during editing.

  • Training: Training runs for 200K iterations with Adam and a learning rate decaying linearly from 5e-4 to 8e-5, while Gaussian density noise is applied.
  • Model architecture: The NeRF uses 64 coarse and 128 fine samples, with an eight-layer 256-dimensional ReLU MLP followed by separate density, color, and feature layers.
  • Segmentation: Segmentation uses thresholded cosine similarity to assign hard probabilities of 1 or 0, with a user-specified threshold defining selection.A fixed label set is used for room-scene results in Figures 3 and 5 to avoid threshold tuning.
  • Editing: During editing, points sampled along each ray are queried in the DFF, producing a selection probability that controls whether radiance values are edited or retained.Selected and unselected results can also be mixed according to the probability instead of thresholded.

C Feature Encoders

The feature-encoder experiments distill LSeg and DINO features into DFFs, contrasting LSeg’s language-driven coarse semantics with DINO’s finer but still limited detail capture.

  • Teacher features: LSeg and DINO encode training images into resized target feature maps used to supervise the distilled feature field.
  • LSeg: LSeg uses a ViT-L/16 image encoder and CLIP ViT-B/32 text encoder with multi-scale inference across scene-dependent scales.Multi-scale inference is intended to stabilize features and reduce artifacts from discrete patch processing.
  • LSeg: Large-scale inference can increase effective feature-map resolution but may lose context and produce noisy features with incorrect semantic understanding.
  • LSeg limitations: LSeg features perform poorly for out-of-distribution regions and queries, with small objects or parts often insufficiently discriminative.The paper attributes this limitation possibly to bias in the mixed training dataset.
  • DINO: DINO uses overlapping patches and fine-grained features, but it cannot accept text queries.The implementation uses the 11th layer of dino_vits8 on 448x448-resized images and averages features with horizontally flipped inputs.
  • Feature resolution: LSeg tends to produce coarse category boundaries, whereas DINO preserves more original edges and supports a wider range of decompositions.Even DINO struggles with the finest high-resolution details, such as the ribs of a T-rex.

D Replica Dataset Experiment

The Replica experiment uses four scenes to evaluate DFF-based 3D semantic segmentation, while accounting for incomplete image coverage and reconstruction-quality limitations. Feature visualizations also require comparing patterns within, not colors across, scenes or models.

  • Dataset construction: The benchmark comprises room_0, room_1, office_3, and office_4 from Replica, with posed images and semantic labels.Unrealistically rendered camera-inside-furniture images were filtered from training and evaluation.
  • Evaluation: Because training images may not cover every scene region, evaluation also tests generalization to unobserved regions through propagation.
  • Dataset construction: Evaluation uses scanned point clouds with human annotations, but geometrically incorrect point-cloud coordinates can invalidate their labels.This issue is especially relevant when evaluating models that do not rely on point-cloud input.
  • Feature visualization: LSeg and DINO feature visualizations should be compared by their feature patterns, not by matching colors between scenes or models.

E Ablation Experiments of Variants

The ablations compare DFF architecture variants alongside segmentation and rendering metrics, finding only marginal average differences across variants. Increasing feature-loss weight does not improve performance and degrades view synthesis quality.

  • Variant comparison: Average performance differences among DFF architecture variants are marginal, although individual scenes can behave differently.
  • Sampling strategy: MLPs trained with coarse-sampling volume rendering perform better overall than MLPs using fine hierarchical sampling.The coarse sampling may implicitly regularize the MLP and promote smoothness.
  • Evaluation metrics: Table 3 reports segmentation, image-synthesis, and depth-geometry metrics for the Replica evaluation.Its metrics include mIoU, 2D agreement with the LSeg teacher, PSNR, SSIM, LPIPS, δ<1.25, and absrel.
  • Loss weighting: Increasing the feature-loss weight λ does not improve performance and hurts novel-view synthesis quality.

F Implementation of CLIPNeRF Experiment

The CLIPNeRF experiment reimplements the method, uses rendered image patches for CLIP optimization, and extends DFF editing to InstantNGP scenes. LSeg supplies text-query features, while related figures document segmentation and feature similarities.

  • CLIPNeRF implementation: Because CLIPNeRF’s official implementation was unavailable, the authors reimplemented it from the paper’s description and obtained similar behaviors despite possible nuanced differences.
  • Visual analyses: The accompanying figures include LSeg confusion and text-feature similarity matrices, plus InstantNGP editing with CLIPNeRF.
  • CLIPNeRF implementation: The implementation renders 128 × 128 patches, resizes them bilinearly, and feeds them to CLIP.Patch rays use stride 5, giving CLIP a 640 × 640 receptive field in the original image; feature rendering uses 16 importance samples for efficiency.
  • InstantNGP extension: DFF editing is also demonstrated with InstantNGP by combining an LSeg-DFF with subsequent CLIPNeRF editing.Feature rendering avoids complete volume rendering by first estimating pseudo-surface depth as a weighted average of sampled depths.
  • Teacher networks: LSeg is used as the teacher network for processing text queries, while other teacher networks are possible.
Loading 2205.15585v2…