Source-linked AI summary
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions
Ayaan Haque, Matthew Tancik, Alexei A. Efros, Aleksander Holynski, Angjoo Kanazawa
TL;DR
Editing NeRF scenes remains difficult because neural representations require specialized tools, while existing approaches offer limited control or depend on references. Instruct-NeRF2NeRF uses InstructPix2Pix to iteratively edit captured-view images and standard NeRF training to consolidate them into an edited scene. The authors demonstrate varied edits on people, objects, and large-scale scenes, while noting limitations in spatial manipulation and adding or removing content.
Problem
NeRF scene editing remains underdeveloped, often requiring specialized tools and years of training, while neural representations lack explicit surfaces.
Method
The method alternates InstructPix2Pix edits of NeRF input images with standard NeRF training to incorporate those edits into the underlying 3D representation.
Results
The authors demonstrate a wide variety of edits on people, objects, and large-scale scenes across captured NeRF scenes.
Takeaways & Limitations
Natural-language instructions provide an intuitive, accessible interface for editing pre-captured NeRF scenes while maintaining 3D-consistent results.
Takeaways & Limitations
The method inherits InstructPix2Pix limitations, including difficulty with large spatial manipulations and artifacts such as double faces from single-view diffusion editing.
Abstract
from arXiv · showhide
We propose a method for editing NeRF scenes with text-instructions. Given a NeRF of a scene and the collection of images used to reconstruct it, our method uses an image-conditioned diffusion model (InstructPix2Pix) to iteratively edit the input images while optimizing the underlying scene, resulting in an optimized 3D scene that respects the edit instruction. We demonstrate that our proposed method is able to edit large-scale, real-world scenes, and is able to accomplish more realistic, targeted edits than prior work.
1. Introduction
Instruct-NeRF2NeRF makes NeRF scene editing accessible through natural-language instructions by iteratively editing rendered training images and incorporating those edits into a 3D-consistent scene. It supports varied edits across people, objects, and large-scale scenes.
- Traditional NeRF editing remains underdeveloped and often requires specialized tools and years of training, especially because neural representations lack explicit surfaces.
- Instruct-NeRF2NeRF takes a pre-captured NeRF scene and a text instruction, producing an edited NeRF and edited versions of the input images.
- Each iteration renders a training-view image, edits it using the global instruction, replaces the dataset image, and continues NeRF training.
- Iterative Dataset Update alternates between editing NeRF input images with InstructPix2Pix and training the underlying 3D representation to incorporate those edits.
- The method is evaluated on captured NeRF scenes against ablations, DreamFusion-style SDS implementations, and a concurrent text-based stylization approach.
- The authors report a wide variety of edits on people, objects, and large-scale scenes.
2. Related Work
Prior NeRF editing methods use physical priors, spatial controls, stylization, latent representations, or reference images, while Instruct-NeRF2NeRF provides mask-free, language-based instructional editing. Its iterative process addresses viewpoint inconsistency in 2D diffusion edits by progressively improving 3D consistency.
- Physical Editing of NeRFs: Existing NeRF editing approaches impose physics-based biases, bounding boxes, or other spatial controls to modify materials, lighting, objects, geometry, or deformations.
- Artistic Stylization of NeRFs: NeRF stylization methods can produce 3D-consistent appearance changes but primarily target global stylization and usually require a reference image.
- Instruction as an Editing Interface: Instruct-NeRF2NeRF offers complementary mask-free editing through intuitive, purely language-based instructions for flexible changes to objects or entire scenes.
- Generating 3D Content: Text-to-3D generation methods can synthesize models from arbitrary prompts but lack fine-grained control and the ability to generalize beyond isolated objects.
- Instruction as an Editing Interface: The paper presents instructional guidance as a 3D editing interface intended to make complex edits accessible without specialized knowledge.
- Dataset Evolution: Initially inconsistent edited images become more 3D consistent as NeRF training and dataset updates proceed iteratively.
3. Method
Instruct-NeRF2NeRF edits captured NeRF scenes by repeatedly updating viewpoint images with InstructPix2Pix and consolidating those edits through NeRF training. The process uses image and text conditioning to produce edits that become increasingly consistent across viewpoints.
- The method takes a reconstructed NeRF, calibrated source images and camera poses, plus a natural-language editing instruction, and outputs an edited NeRF and edited input images.
- InstructPix2Pix: InstructPix2Pix provides image-conditioned diffusion guidance: it predicts noise from a noised latent conditioned on an image and text instruction, then derives an edited-image estimate.The latent output is decoded back into an RGB image before it is used in the editing pipeline.
- Editing a rendered image: InstructPix2Pix edits each viewpoint image using the unedited source image, the text instruction, and a noised rendering from the current optimization step.The noised current render supplies the evolving scene state, while the unedited image remains the diffusion model’s conditioning image.
- Iterative Dataset Update: Conditioning on unedited images keeps the repeated editing process grounded and helps prevent recurrent-synthesis drift.
- Iterative Dataset Update: Iterative Dataset Update alternates diffusion-based image updates with NeRF updates that use the changed images as supervision.Image updates occur across captured viewpoints, while NeRF updates sample rays from the full training dataset.
4. Results
The method edits diverse captured NeRF scenes, including people, objects, and large-scale environments, while targeting 3D-consistent results. Qualitative comparisons and auxiliary metrics highlight broad editability and improved consistency over simpler baselines, alongside limitations inherited from InstructPix2Pix.
- Qualitative Evaluation: The method edits environments, objects, faces, and full-body portraits captured with smartphones and mirrorless cameras.Datasets contain 50–300 images, with camera poses obtained through COLMAP or PolyCam.
- Qualitative Evaluation: It supports global and local portrait edits, contextual additions such as cowboy hats and mustaches, and partial clothing changes.Examples include transforming a portrait into a Modigliani painting or changing its face into a skull.
- Qualitative Evaluation: Edits extend to material changes, notable figures, animal transformations, large-scale scenes, seasons, weather, and time-of-day conditions.Reported examples include bronze or marble appearances, Einstein and Batman transformations, polar bears, pandas, snow, and desert scenes.
- Ablation Study: Naive per-frame editing produces view-inconsistent results, while one-time dataset updates can yield blurry, artifact-filled 3D scenes.The one-time update baseline is especially problematic when contextual objects are added to portraits.
- Quantitative Evaluation: The proposed approach achieves similar CLIP text alignment to per-frame editing while obtaining the best CLIP-space consistency across views.The auxiliary evaluation covers 10 edits across two scenes and measures text alignment and consistency between adjacent rendered frames.
- Limitations: The method fails when InstructPix2Pix cannot perform the desired 2D edit or produces correct but view-inconsistent edits that cannot be consolidated in 3D.These two failure modes are illustrated in Figure 9.
- Qualitative Evaluation: Compared with NeRF-Art, the evaluation uses that method’s scenes and edits, but its non-instructional text inputs leave the intended edit ambiguous.The authors present the comparison as a reference for readers because edit preferences are subjective.
- Limitations: Adding entirely new objects and removing objects without replacement remain challenging because diffusion outputs can be absent, misplaced, or view-inconsistent.The authors expect improvements in diffusion-based image manipulation to improve the iterative framework.
5. Conclusion
The paper presents Instruct-NeRF2NeRF as an accessible, text-driven approach to editing pre-captured NeRF scenes while maintaining 3D consistency. Results across people, objects, and large-scale scenes demonstrate a broad range of edits.
- Conclusion: Instruct-NeRF2NeRF enables intuitive NeRF scene editing for everyday users through natural text instructions.The method operates on pre-captured scenes and aims to preserve 3D consistency in the resulting edits.
- Conclusion: The method demonstrates a wide range of edits on people, objects, and large-scale scenes.These results are shown across a variety of captured NeRF scenes.
A. Additional implementation details
The method starts from a NeRF reconstruction and uses InstructPix2Pix guidance with user-selected classifier-free guidance weights. Because diffusion guidance varies during optimization, users select a termination iteration appropriate to the desired edit strength.
- Inputs and initialization: The primary input is a NeRF reconstruction of a real scene produced with NeRFStudio’s ‘nerfacto’ model.Each scene is trained for 30,000 iterations before Instruct-NeRF2NeRF begins.
- Diffusion guidance: Users specify classifier-free guidance weights to control the desired amount of change, and the method inherits these settings from InstructPix2Pix.
- Optimization behavior: The diffusion model produces temporally varying edited images, so optimization lacks a single convergence point like standard NeRF optimization.
- Optimization behavior: Users must choose when to terminate optimization because edit type and strength change over training.The paper uses 3,000–4,000 iterations for smaller scenes and 7,000–8,000 for larger scenes.
B. Limitations
The method inherits limitations from single-view diffusion editing and the Stable Diffusion autoencoder. These limitations affect spatial manipulation, view-dependent edits, texture fidelity, and visual quality during prolonged optimization.
- Inherited editing limits: The method cannot reliably perform large spatial manipulations, object binding, or adding and removing large objects.
- View-dependent artifacts: Single-view diffusion guidance can produce artifacts such as double faces on added objects.
- View-dependent artifacts: Instructions may be more relevant to some views than others, causing certain views to receive weaker or no edits.For example, views that do not prominently feature a man may respond less to an instruction to turn him into a bear.
- Texture fidelity: Edited NeRF scenes often have slightly blurrier textures because autoencoded images contain locally similar but non-identical textures that are not globally 3D-consistent.An autoencoding experiment showed that continuing NeRF training on such images gradually makes the scene blurrier.
- Optimization duration: Visual quality decreases during much longer optimization runs, which the authors largely attribute to autoencoder effects.
C. Metrics
The evaluation uses CLIP-based metrics to measure whether edits follow the requested textual direction and remain consistent across adjacent views along a novel camera path.
- Directional editing: The CLIP Directional Score measures agreement between changes in text captions and changes in images.
- Temporal consistency: The CLIP Direction Consistency Score measures cosine similarity between CLIP embeddings of adjacent frames rendered along a novel camera path.
- Temporal consistency: The temporal consistency loss compares consecutive original and edited NeRF renderings using four CLIP embeddings.It measures how the edit direction changes from frame to frame in CLIP space.