Source-linked AI summary
GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions
Junjie Wang, Jiemin Fang, Xiaopeng Zhang, Lingxi Xie, Qi Tian
TL;DR
GaussianEditor addresses the difficulty of precise localized 3D scene editing with whole-image 2D diffusion models. It extracts text-defined regions, aligns them to explicit 3D Gaussians, and restricts editing to the Gaussian RoI. The framework achieves more delicate results than Instruct-NeRF2NeRF while training within 20 minutes on a single V100 GPU.
Problem
Whole-image editing by current 2D diffusion models makes delicate, especially localized, 3D scene editing difficult.
Method
GaussianEditor extracts a text instruction RoI, aligns it to 3D Gaussians through image-space grounding, and uses the Gaussian RoI to control updates.
Results
GaussianEditor achieves more delicate editing results than Instruct-NeRF2NeRF with training within 20 minutes on a single V100 GPU.
Takeaways & Limitations
Explicit 3D Gaussians make precise control of the editing area practical for text-guided 3D scene editing.
Takeaways & Limitations
The system cannot achieve good results when grounding segmentation or the diffusion model completely fails, including drastic geometric editing.
Abstract
from arXiv · showhide
Recently, impressive results have been achieved in 3D scene editing with text instructions based on a 2D diffusion model. However, current diffusion models primarily generate images by predicting noise in the latent space, and the editing is usually applied to the whole image, which makes it challenging to perform delicate, especially localized, editing for 3D scenes. Inspired by recent 3D Gaussian splatting, we propose a systematic framework, named GaussianEditor, to edit 3D scenes delicately via 3D Gaussians with text instructions. Benefiting from the explicit property of 3D Gaussians, we design a series of techniques to achieve delicate editing. Specifically, we first extract the region of interest (RoI) corresponding to the text instruction, aligning it to 3D Gaussians. The Gaussian RoI is further used to control the editing process. Our framework can achieve more delicate and precise editing of 3D scenes than previous methods while enjoying much faster training speed, i.e. within 20 minutes on a single V100 GPU, more than twice as fast as Instruct-NeRF2NeRF (45 minutes -- 2 hours).
1. Introduction
GaussianEditor targets delicate, localized 3D scene editing by combining explicit 3D Gaussians with text-guided region control. It extracts and aligns editing regions, then constrains updates to those regions while reducing training time.
- Motivation: Traditional 3D asset creation is expensive and cumbersome, motivating editing existing captured or modeled scenes.The workflow includes labor-intensive steps such as sketching, structure building, and texture creation.
- Motivation: 2D diffusion-based editing can change real scenes from text instructions but struggles to localize edits accurately, causing unintended changes.This limitation hinders finely edited 3D scenes.
- Motivation: 3D Gaussian Splatting provides individually explicit Gaussians, making constrained manipulation of selected scene elements easier.It also offers real-time rendering with high quality and fast training.
- Method: GaussianEditor extracts instruction regions, aligns them to 3D Gaussian space through image-space grounding segmentation, and edits only the resulting Gaussian RoI.Users can provide additional controls such as Gaussian-point selection and 3D boxes.
- Results: GaussianEditor is presented as an early systematic 3D-Gaussian method for delicate scene editing and produces more delicate results than Instruct-NeRF2NeRF.The method trains within 20 minutes, compared with 45 minutes–2 hours for Instruct-NeRF2NeRF.
2. Related Work
Prior work applies diffusion models to image and 3D scene editing, but precise localization remains difficult for implicit radiance-field representations and image-based editing. GaussianEditor addresses this challenge through text-guided region extraction and lifting into 3D Gaussians.
- 2D Image Editing with Diffusion Models: Diffusion models have enabled impressive image synthesis and generation from arbitrary textual inputs.
- 3D Scene Editing of Radiance Fields: Radiance-field editing methods manipulate 3D scene geometry and appearance, but implicit NeRF representations make precise localization difficult.
- 3D Scene Editing of Radiance Fields: Previous 3D scene-editing methods have therefore primarily focused on global style transformations.
- GaussianEditor: GaussianEditor extracts a text-defined region of interest, grounds it in images, and lifts it into a 3D Gaussian region for controlled editing.
- 3D Scene Editing of Radiance Fields: Instruct-NeRF2NeRF enables text-controlled 3D editing but can cause global scene changes because it relies on 2D image-editing effects.
3. Method
GaussianEditor extracts a text-defined region of interest, aligns it from text to images and then to 3D Gaussians, and restricts diffusion-guided optimization to that region. The method combines scene-description generation, RoI lifting, optional interactive constraints, and localized editing.
- Overall Framework: The framework has three stages: text RoI extraction, 3D Gaussian RoI alignment, and delicate editing within the aligned RoI.Scene descriptions and language-model reasoning identify the target; grounding segmentation and RoI lifting map it into 3D Gaussian space.
- RoI Extraction of Text Instruction: Differentiable splatting produces multi-view image samples, a multimodal model describes them, and a language model merges those descriptions into one scene description.The merged description is combined with the user instruction before extracting the instruction RoI.
- 3D Gaussian RoI Alignment: The aligned RoI can combine trained, added, and deleted Gaussian subsets and be constrained by an optional 3D bounding box.The interface example corrects a failed grounding of “left face” by excluding the right face and limiting the region with a 3D box.
- Delicate Editing within Gaussian RoI: Random viewpoints are rendered and edited by a 2D diffusion model conditioned on the instruction and input image, after which L1 and D-SSIM losses backpropagate only through RoI Gaussians.Adam optimization over repeated training rounds produces the edited 3D scene.
4. Experiments
GaussianEditor demonstrates precise, separate editing across characters, outdoor scenes, comparisons with IN2N, and complex multi-object scenes. Quantitative and user-study results further support its editing quality, while ablations show the value of its RoI modules and scene-description generation; limitations remain for inconsistent descriptions and severe model failures.
- Implementation: GaussianEditor uses BLIP2, GPT-3.5 Turbo, Grounding DINO, SAM, and Instruct Pix2Pix to extract, ground, and edit instruction-specific regions.The grounding pipeline first obtains image boxes with Grounding DINO and then image RoIs with SAM.
- Qualitative Evaluation: Character edits include color changes, retexturing, slight geometry editing, left- and right-face localization, and accurate multi-round edits.The examples include a red nose, baldness, and edits confined to selected sides of a face.
- Qualitative Evaluation: Compared with IN2N, GaussianEditor changes pants or clothes without affecting the other garment, preserves the background, and maintains superior face rendering quality.The comparison highlights object separation and reduced unintended changes relative to the 2D-diffusion-based baseline.
- Qualitative Evaluation: GaussianEditor separately edits foreground and background content in outdoor scenes and complex scenes containing a desk, flower pot, and occluded rolling pin.The rolling pin is changed into a cucumber without affecting other objects, while the desk becomes wood and the flower pot becomes red.
- Quantitative Evaluation: GaussianEditor achieves the best results across CTIDS, IIS, FID, and training time on the bicycle scene, and receives 87.07% of user votes versus IN2N’s 12.93%.Table 1 compares GaussianEditor with IN2N and DVGO; the user study includes 21 participants.
- Ablation Study and Analysis: RoI ablations show that removing text-RoI selection broadens edits, removing RoI lifting causes facial leakage, and scene-description generation is evaluated through three setup variants.The RoI lifting module addresses noisy image RoIs during training, while the scene-description study compares no description, a random-view description, and the complete approach.
- Limitations: The system remains limited when multi-view descriptions differ substantially or when grounding segmentation or diffusion completely fails, including drastic geometric editing.The authors report that description mismatches may cause an LLM to interpret one object as multiple objects.
5. Conclusion
GaussianEditor is a systematic framework for text-guided delicate 3D scene editing with 3D Gaussians. It precisely controls editing regions, produces more delicate results than IN2N, trains within 20 minutes, and leaves dynamic-scene editing for future work.
- Conclusion: GaussianEditor uses 3D Gaussians and text instructions to achieve delicate 3D scene editing with precise control over editing areas.Its framework extracts instruction RoIs, aligns them to 3D Gaussians, and edits within the Gaussian RoI.
- Conclusion: The method achieves notably more delicate editing results than IN2N with training completed within 20 minutes versus 45 minutes–2 hours.The reported training-time comparison is for GaussianEditor and Instruct-NeRF2NeRF.
- Conclusion: Delicate editing in dynamic scenes is left as future work.The conclusion identifies dynamic Gaussian-splatting scenes as an unaddressed scope.
A.1. Additional Implementation Details
GaussianEditor reconstructs edits from a 3D Gaussian scene using 2D diffusion guidance. Training monitors rendered results and generally limits editing to within 200 rounds because repeated 2D editing can make scenes blurry.
- Additional Implementation Details: GaussianEditor takes a 3D scene reconstructed by 3D Gaussian Splatting as input and trains each scene for 30,000 iterations.Images wider than 512 pixels are resized to 512.
- Additional Implementation Details: The method uses Instruct Pix2Pix with classifier-free guidance weights for image guidance s_I and text guidance s_T.The implementation is based on 3D Gaussian splatting and uses 2D diffusion to edit rendered images.
- Additional Implementation Details: Repeated 2D editing can make scenes blurry, so training results are monitored and editing is generally limited to within 200 rounds.The stated cause is instability in the underlying 2D editing process.
A.2. Quantitative Evaluation
GaussianEditor is evaluated quantitatively against prior methods using CLIP-based editing metrics and a user study. The results show improved image-image similarity and strong user preference, while the authors identify limitations in CLIP-based evaluation.
- CLIP-Based Evaluation: GaussianEditor achieves a CLIP text-image direction similarity comparable to IN2N while substantially improving image-image similarity.The evaluation uses scenes from Fig. 5 and follows IN2N’s metrics.
- Metric Limitations: CLIP-based metrics are not reliable enough because CLIP has difficulty discriminating colors, with yellow images scoring higher than white images for corresponding color descriptions.The authors identify reliable evaluation metrics for text-guided editing as an important future direction.
- User Study: 87.07% of user-study votes favored GaussianEditor over IN2N across 7 editing cases.Twenty-one users submitted 147 total votes; IN2N received 12.93%.
A.3. Qualitative Evaluation
Qualitative comparisons show that GaussianEditor localizes edits more precisely and preserves scene details better than competing methods. It also exhibits some geometric editing capability, though drastic geometric changes remain future work.
- Comparison with IN2N and Different Backbones: GaussianEditor precisely localizes the bench and turns it red, whereas IN2N changes the bicycle, bench, and tree and DVGO struggles with localization.The comparison uses the same scene as Table 1.
- Comparison with DreamEditor: GaussianEditor edits the doll delicately while retaining hair details, whereas DreamEditor removes the hair and changes the back box.GaussianEditor also obtains the desired result using less time.
- Depth Map of Geometric Editing: The hair-editing depth map indicates a certain level of geometric editing capability, while drastic geometric changes remain future work.The result corresponds to the hair-editing example in Fig. 1.
A.4. Extension
GaussianEditor can be extended to 3D generation workflows and is compared visually with DreamEditor. Its Gaussian RoI supports isolating edited content before reintegrating it into the original scene.
- Extension: GaussianEditor integrates with GaussianDreamer to produce enhanced editing effects.The paper presents this integration as an example of the framework’s extension ability.
- Comparison with DreamEditor: DreamEditor is compared with GaussianEditor on the DTU dataset.The supplied figure caption identifies the comparison but does not report a quantitative outcome.
- Extension: The Gaussian RoI is saved individually, used to initialize the 3D-generation model, and merged back into the original scene.The edited object is generated from the edited-scene text description before reintegration.