PaperlayerView exact revision on arXiv

Source-linked AI summary

ViCA-NeRF: View-Consistency-Aware 3D Editing of Neural Radiance Fields

Jiahua Dong, Yu-Xiong Wang

arXiv:2402.00864v1cs.CV

TL;DR

Text-guided NeRF editing can be slow and inconsistent across views when it relies on iterative 2D diffusion edits. ViCA-NeRF propagates key-view edits using geometric and learned regularization, then trains NeRF on the edited dataset. The authors report faster editing with higher consistency and detail than the state of the art.

  • Problem

    NeRF editing with 2D diffusion can require iterative dataset updates and produce edits that vary across views.

    Source: §1 · p. 2

  • Method

    ViCA-NeRF uses depth-based correspondences and diffusion latent-code alignment to propagate key-view edits before NeRF training.

    Source: §1 · p. 2

  • Results

    ViCA-NeRF is reported to achieve higher consistency and detail than the state of the art, with editing 3 times faster than the state of the art.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    ViCA-NeRF supports text-guided edits ranging from stylization and weather changes to context-related operations and detailed textures.

    Source: §5 · p. 10

    Results deteriorate with inaccurate NeRF-derived depth, and edited outcomes tend to be blurrier than the original NeRF.

    Source: §4.5 · p. 10

Abstract

from arXiv · show

We introduce ViCA-NeRF, the first view-consistency-aware method for 3D editing with text instructions. In addition to the implicit neural radiance field (NeRF) modeling, our key insight is to exploit two sources of regularization that explicitly propagate the editing information across different views, thus ensuring multi-view consistency. For geometric regularization, we leverage the depth information derived from NeRF to establish image correspondences between different views. For learned regularization, we align the latent codes in the 2D diffusion model between edited and unedited images, enabling us to edit key views and propagate the update throughout the entire scene. Incorporating these two strategies, our ViCA-NeRF operates in two stages. In the initial stage, we blend edits from different views to create a preliminary 3D edit. This is followed by a second stage of NeRF training, dedicated to further refining the scene's appearance. Experimental results demonstrate that ViCA-NeRF provides more flexible, efficient (3 times faster) editing with higher levels of consistency and details, compared with the state of the art. Our code is publicly available.

1 Introduction

NeRF-based text editing is difficult because edits made independently with 2D diffusion can vary across views and require iterative dataset updates. ViCA-NeRF combines geometric and learned regularization to propagate edits from key views, then trains NeRF on the edited data.

1 Introduction

  • 2D diffusion edits can vary across views, while iterative dataset updates and NeRF fine-tuning make existing NeRF editing time-consuming.The inconsistency is especially concerning for aggressive edits.Source: §1 · p. 2
  • ViCA-NeRF uses depth-derived image correspondences to project edited pixels from key views to other views before NeRF fine-tuning.This supports user control over 3D content through selected 2D edit results.Source: §1 · p. 2
  • Latent-code alignment between edited and unedited images addresses incorrect depth correspondences and produces more homogeneous images for NeRF optimization.The alignment is implemented through a blending refinement model.Source: §1 · p. 2
  • ViCA-NeRF integrates both regularization strategies in a first editing stage, followed by NeRF training and dataset refinement.An efficient warm-up strategy adjusts the editing scale.Source: §1 · p. 2
  • 3 times faster than the state of the art, ViCA-NeRF is reported to improve editing efficiency while explicitly enforcing multi-view consistency.The authors describe it as the first work to explicitly enforce multi-view consistency in 3D editing.Source: §1 · p. 2

2 Related Work

Prior work applies diffusion models to 2D image editing and 3D generation, while text-guided NeRF editing uses 2D edits to modify scenes. ViCA-NeRF instead propagates selected key-view edits across views and updates the dataset less iteratively.

2 Related Work

  • Text-to-image editing includes SDEdit, which introduces noise into an input image and subsequently denoises it, and masked inpainting with text guidance; Dream Fusion uses SDS, while Magic3D generates higher-resolution results by a coarse-to-fine procedure.Source: §2 · pp. 2–3
  • Instruct-Pix2Pix conditions image editing on original images and text prompts, and ViCA-NeRF extends it to editing, blending, and refinement for 3D scenes.Its blending model averages multiple results in latent code before decoding a final result.Source: §2 · p. 3, Figure 3 · p. 4
  • Unlike text-to-3D generation, ViCA-NeRF edits an existing 3D representation with text prompts through 2D editing.The overview depicts key-view editing, depth-and-camera-pose projection, blending, and subsequent NeRF training.Source: §2 · p. 3, Figure 2 · p. 3
  • Earlier NeRF-editing approaches include shape and color edits, point-cloud-based editing, and CLIP-guided stylization; Instruct-NeRF2NeRF and Instrcut-3D-to-3D use language-guided diffusion edits.The latter methods iteratively use diffusion during NeRF training, and Instruct-NeRF2NeRF depends on consistent 2D edits for detailed results.Source: §2 · p. 3
  • ViCA-NeRF uses depth guidance to propagate edits from several key views, then updates the dataset at training start and refines it once.Users can personalize the final NeRF effect by choosing 2D edit results.Source: §2 · p. 4

3 Method

ViCA-NeRF uses depth-guided correspondences and diffusion-model latent alignment to propagate edits from key views across a scene. Its workflow combines sequential editing, blending, and NeRF training, with optional post-refinement.

3.1 Preliminary

  • NeRF maps spatial coordinates and viewing direction to density and radiance; Instruct-Pix2Pix denoises images conditioned on text and a guidance image.The resulting image is described as consistent with the initial image and text instruction.Source: §3.1 · p. 4
  • Figure 4 reports more detail than Instruct-NeRF2NeRF and shows ViCA-NeRF handling the “Turn him into a robot” prompt where that method fails.The comparison also reports more detailed results on NeRF-Art sequences and edits, with more substantial content changes.Source: Figure 4 · p. 5, Figure 5 · p. 7

3.2 Depth-Guided Sequential Editing

  • Depth-guided sequential editing projects prior key-view edits into later views, filters invalid correspondences by reprojection error, and mixes edits without overwriting already modified pixels.Key-view selection aims for suitable overlap; the maximum desired modified ratio ϕ is 0.3 in all experiments.Source: §3.2 · pp. 5–6

3.3 Blending Refinement Model

  • The blending model first uses the original image to clean a noisy mixup, then uses the mixup as a condition to restore its detailed characteristics.Averaging independently updated noised latent codes stabilizes the output.Source: §3.3 · p. 6

3.4 Warm-Up and Post-Refinement Strategies

  • Before editing, warm-up edits a random view and projects it to all others; a masked blend preserves some original pixels while maintaining stylization consistency.The blend uses λ to control how much of the original pixel value is preserved.Source: §3.4 · p. 7
  • After NeRF training, post-refinement uses each rendered image as input and the mixup image as a condition to refine consistency and output quality.It uses the modified Instruct-Pix2Pix model with averaging.Source: §3.4 · p. 7

4 Experiments

ViCA-NeRF is evaluated on varied scenes and prompts through qualitative comparisons, component ablations, local editing, and efficiency analysis. The experiments show faster editing, with qualitative benefits and limitations that depend on the task and depth quality.

4 Experiments

  • Experiments cover real scenes and varied prompts, comparing against Instruct-NeRF2NeRF and NeRF-Art and evaluating text alignment, consistency, and efficiency.The implementation uses NeRFStudio’s nerfacto model and Instruct-Pix2Pix; the NeRF is pretrained for 30,000 iterations and final results are obtained at 40,000.Source: §4 · p. 7, §4.1 · pp. 7–8

4.2 Qualitative Evaluation

  • ViCA-NeRF improves on Instruct-NeRF2NeRF for complex detail-heavy edits and on NeRF-Art for detail and style, while simpler edits are similar.The authors also report that key-view direction supports consistency between 2D edits and 3D renders.Source: §4.2 · p. 8, Figure 6 · p. 8

4.3 Ablation Study

  • Blending removes mixup artifacts while retaining most original content; warm-up adjusts edit scale while preserving primary geometry at minor computational cost.Post-refinement improves similarity to the rendered result, but the blended result is already sufficiently good for refinement to be optional.Source: §4.3 · p. 9
  • Omitting the reprojection check allows more extensive changes but loses original detail; sparse-view editing works when NeRF trains successfully, while ϕ=0 creates boundaries and ϕ=1 reduces efficiency.The tested key-view parameter values were 0, 0.3, and 1.Source: §4.3 · p. 9

4.4 Local Editing

  • Local editing propagates a selected object’s mask across views, enabling edits to the target while leaving the background intact.Users select the target instance with one point after SAM generates masks; depth guides progressive mask projection.Source: §4.4 · pp. 9–10

4.5 Discussion

  • 15 minutes versus Instruct-NeRF2NeRF’s 45 minutes for 10,000 iterations on the Face scene demonstrates ViCA-NeRF’s lower editing time.Both methods were tested on an RTX 4090 GPU; the authors attribute the reduction to decoupling NeRF training from dataset updates.Source: §4.5 · p. 10
  • Results deteriorate with inaccurate NeRF depth, and edited outcomes tend to be blurrier than the original NeRF.Blending refinement mitigates incorrect correspondences to some extent.Source: §4.5 · p. 10

5 Conclusion

ViCA-NeRF is a text-guided NeRF editing framework that the authors report outperforms several baselines across scenes and prompts. Further results describe its diversity, view consistency, component contributions, and warm-up effects.

5 Conclusion

  • ViCA-NeRF supports text-guided editing across varied scenes and prompts, and the authors report that it outperforms several baselines.Supported operations include human stylization, weather changes, context-related edits such as adding flowers, and detailed textures.Source: §5 · p. 10

A.1 Diversity

  • Different random seeds produce more diverse edits than Instruct-NeRF2NeRF, whose results remain similar for the same prompt.The comparison is shown between Figure 12 and Figure 6 in the main paper.Source: §A.1 · p. 13, Figure 12 · p. 13

A.2 2D Edit Consistency

  • Before NeRF tuning, ViCA-NeRF’s edits are more view-consistent than Instruct-NeRF2NeRF’s final updated edits.The comparison reports that Instruct-NeRF2NeRF edits slightly different views differently, while ViCA-NeRF’s 2D edits remain consistent.Source: §A.2 · p. 13, Figure 13 · p. 13

A.3.1 Impact of Different Components

  • Adding ViCA-NeRF components progressively improves rendered edit quality over a baseline that independently edits each view once.The baseline uses Instruct-Pix2Pix before NeRF training.Source: §A.3.1 · p. 13

A.3.2 Warm-Up Iterations

  • A few warm-up iterations greatly increase the scale of editing in the final rendered results.The comparison uses 0, 10, and 30 warm-up iterations.Source: §A.3.2 · p. 13, Figure 15 · p. 14

A.3.3 Ablation on Averaged Diffusion Process

Averaging makes 2D edits more stable and improves consistency, though it can add image smoothness; the overall 3D edit improves, including clearer wall details.

A.3.3 Ablation on Averaged Diffusion Process

  • Averaging can make 2D edits more stable, thereby improving consistency.Source: §A.3.3 · p. 14
  • Averaging may introduce some smoothness into the image.Source: §A.3.3 · p. 14
  • Despite that smoothness, the overall 3D editing result improves, including clearer wall details.Source: §A.3.3 · p. 14

A.4 More Qualitative Results

Additional results cover three edits across two outdoor scenes and show that ViCA-NeRF can edit cases where Instruct-NeRF2NeRF fails.

A.4 More Qualitative Results

  • The additional results cover three edits on two outdoor scenes.Source: §A.4 · p. 14
  • ViCA-NeRF conducts edits on these scenes where Instruct-NeRF2NeRF fails.Source: §A.4 · p. 14
  • The outdoor examples are presented as further validation of applicability to real-world scenes.Source: §A.4 · p. 14

B.1 Metrics

The evaluation uses a text-image direction score and a consistency score to assess text-image alignment and the similarity of changes between adjacent views.

B.1 Metrics

  • The text-image direction score estimates similarity between text changes and image changes using CLIP embeddings.Source: §B.1 · pp. 14–15
  • The text-image direction score is calculated using cosine similarity.Source: §B.1 · p. 15
  • The consistency score compares changes between adjacent views by comparing alterations in original renders with those in edited renders.Source: §B.1 · p. 15

B.2 Results

ViCA-NeRF outperforms Instruct-NeRF2NeRF on both metrics in simple edits and achieves significantly better scores on both metrics in the difficult setting; additional outdoor examples show edits where the baseline fails.

B.2 Results

  • ViCA-NeRF outperforms Instruct-NeRF2NeRF on text-image direction and consistency scores for simple edits, with a slight consistency improvement.Source: §B.2 · p. 15, Table 1 · p. 16
  • In the difficult setting, ViCA-NeRF achieves significantly better scores on both metrics as edits become more challenging.Source: §B.2 · p. 15
  • Additional outdoor-scene results show ViCA-NeRF editing cases where Instruct-NeRF2NeRF fails.Source: Figure 17 · p. 16

C.1 Limitations from Instruct-Pix2Pix

  • The method inherits Instruct-Pix2Pix limitations: it struggles with large spatial changes and highly detailed instructions involving interactions between image parts.Source: §C.1 · p. 16

C.2 Limitations from Current Pipeline

On difficult edits, ViCA-NeRF performs better on both reported metrics, but its diffusion blending can smooth details and produce blurry 3D edits. The paper also describes implications for controllability, editing convenience, and resource-limited settings.

C.2 Limitations from Current Pipeline

  • Diffusion processing of projection mixup can smooth high-frequency details and leave resulting 3D edits blurry, even with the clear original as guidance.The reported losses include face texture, beard, and hair detail.Source: §C.2 · p. 16
  • ViCA-NeRF performs significantly better on both metrics in difficult edits evaluated on scenes from Instruct-NeRF2NeRF.The evaluation covers 10 edits; the passage does not provide metric names or values.Source: Table 2 · p. 17

D Broader Impacts

  • The authors describe the approach as improving controllability, efficiency, and diversity, and as requiring less GPU time by avoiding diffusion integration into NeRF training.They suggest it may enable broader and more detailed editing as diffusion models advance.Source: D Broader Impacts · p. 17

E Additional Implementation Details

The implementation fine-tunes a pre-trained NeRF and varies warm-up settings by scene scale. Correspondence matching excludes pairs with reprojection error at or above 5px.

E Additional Implementation Details

  • The method fine-tunes a pre-trained nerfacto model for 10,000 iterations after 30,000 iterations of training on original data.Around 5,000 fine-tuning iterations are adequate for small scenes.Source: E Additional Implementation Details · p. 17
  • Warm-up uses 10 iterations for face-centric scenes and 30 for outdoor scenes; post-refinement is omitted for complex large-scale outdoor scenes.For those complex scenes, post-refinement does not further improve consistency.Source: E Additional Implementation Details · p. 17
  • Correspondence matching accepts pairs only when reprojection error lr is below 5px; invalid correspondences are ignored.Source: E Additional Implementation Details · p. 17
Loading 2402.00864v1…