Source-linked AI summary
GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting
Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, Guosheng Lin
TL;DR
Existing 3D editing representations trade off scene fidelity, speed, or precise control. GaussianEditor addresses this with Gaussian Splatting, semantic tracing, hierarchical organization, and 3D inpainting, while experiments report improved effectiveness, speed, and controllability; its supervision remains limited by complex prompts that current 2D diffusion models cannot guide effectively.
Problem
Meshes and point clouds struggle with complex scenes, while NeRF is slow and restricts direct modification of specific scene areas and related editing tasks.
Method
GaussianEditor uses explicit Gaussian Splatting with semantic tracing, Hierarchical Gaussian Splatting, and specialized 3D inpainting for object removal and addition.
Results
GaussianEditor surpasses previous 3D editing methods in effectiveness, speed, and controllability, with typical editing sessions taking 5-10 minutes.
Takeaways & Limitations
The method provides swift, controllable, and versatile Gaussian-Splatting-based 3D editing, including object removal and integration.
Takeaways & Limitations
GaussianEditor relies on 2D diffusion models whose guidance is ineffective for certain complex prompts, limiting 3D editing.
Abstract
from arXiv · showhide
3D editing plays a crucial role in many areas such as gaming and virtual reality. Traditional 3D editing methods, which rely on representations like meshes and point clouds, often fall short in realistically depicting complex scenes. On the other hand, methods based on implicit 3D representations, like Neural Radiance Field (NeRF), render complex scenes effectively but suffer from slow processing speeds and limited control over specific scene areas. In response to these challenges, our paper presents GaussianEditor, an innovative and efficient 3D editing algorithm based on Gaussian Splatting (GS), a novel 3D representation. GaussianEditor enhances precision and control in editing through our proposed Gaussian semantic tracing, which traces the editing target throughout the training process. Additionally, we propose Hierarchical Gaussian splatting (HGS) to achieve stabilized and fine results under stochastic generative guidance from 2D diffusion models. We also develop editing strategies for efficient object removal and integration, a challenging task for existing methods. Our comprehensive experiments demonstrate GaussianEditor's superior control, efficacy, and rapid performance, marking a significant advancement in 3D editing. Project Page: https://buaacyw.github.io/gaussian-editor/
1. Introduction
GaussianEditor targets faster, more controllable 3D editing by combining Gaussian Splatting with semantic tracing, hierarchical organization, and specialized inpainting.
- Traditional meshes and point clouds struggle to accurately render complex 3D scenes, while NeRF limits direct scene-part modification and complicates inpainting and composition.
- GaussianEditor uses Gaussian Splatting to support flexible, rapid editing with text-based edits and explicit area control through bounding boxes.
- Gaussian semantic tracing assigns semantic tags to Gaussians, continuously identifying target points so only selected areas are modified as geometry and appearance evolve.
- Hierarchical Gaussian Splatting organizes Gaussians by densification generation, constraining older generations more strongly to improve stability under random generative guidance.
- The specialized 3D inpainting algorithm removes objects, repairs boundary artifacts, and integrates new objects using prompts, masks, and 2D inpainting.
- 5-10 minutes is the typical editing-session duration, while experiments report better effectiveness, speed, and controllability than previous 3D editing methods.
2. Related Works
Prior work spans NeRF, Gaussian Splatting, and neural-field editing, but GaussianEditor adapts explicit Gaussian Splatting to pursue swift and controllable 3D editing.
- 3D Representations: NeRF supports high-quality 3D optimization with 2D supervision but remains time-consuming, and acceleration methods mainly target reconstruction rather than generation.
- 3D Representations: Gaussian Splatting offers impressive reconstruction quality and speed, efficient differentiable rendering, and fast training without spatial pruning.
- 3D Representations: GaussianEditor pioneers adapting 3D Gaussian Splatting to 3D editing, aiming to exploit its explicit representation for swift and controllable edits.
- 3D Editing: Neural-field editing methods modify shape and color through latent codes, CLIP guidance, templates, skeletons, meshes, or 2D inpainting.
- 3D Editing: Static 2D and 3D masks constrain NeRF edits, but dynamic training can make them ineffective; Gaussian semantic tracing instead tracks target Gaussians throughout training.
3. Preliminary
The preliminary framework represents scenes with explicit Gaussians and uses splatting for rendering, while 2D diffusion guidance supplies the editing loss.
- 3D Gaussian Splatting: Each Gaussian represents scene structure through a center point x and covariance matrix Σ, with covariance decomposed into rotation R and scaling S for optimization.
- 3D Gaussian Splatting: Splatting projects Gaussians onto camera planes using viewing transformation W and Jacobian J, producing the camera-coordinate covariance Σ′.
- 3D Gaussian Splatting: Gaussian attributes include position, spherical-harmonics color, opacity, rotation quaternion, and scaling factor; overlapping points are blended per pixel.
- Diffusion-based Editing Guidance: 2D diffusion guidance uses rendered 3D-model views, camera poses, and prompts to formulate an editing loss, denoted universally as D.
4. Method
GaussianEditor combines semantic tracing, hierarchical Gaussian Splatting, and specialized inpainting to enable controllable GS editing. Its method tracks target Gaussians during training, stabilizes stochastic optimization, and supports object addition and removal.
- Gaussian Semantic Tracing: Gaussian semantic tracing continuously tracks target Gaussians during training, unlike static masks that become inaccurate as 3D geometry and appearance evolve.The method unprojects 2D segmentation masks into Gaussians and propagates labels to newly densified points.
- Gaussian Semantic Tracing: Semantic masks enable selective updates, gradients, densification, pruning, and target-only rendering for controllable editing in complex scenes.The labeling process typically takes less than a second, and target-only rendering accelerates editing.
- 3D Inpainting: Object removal uses 2D inpainting and MSE supervision to repair interface artifacts, while object incorporation uses a single-view mask and prompt.The reported processing times are two minutes for removal and five minutes for incorporation.
- Hierarchical Gaussian Splatting: Hierarchical Gaussian Splatting organizes points by densification generation and applies stronger anchor constraints to older generations under stochastic guidance.Anchor loss compares current Gaussian properties with recorded anchor states, with constraint strength varying by generation.
- 3D Inpainting: The 3D inpainting formulation updates a Gaussian scene from a text prompt, camera pose, rendered view, and user-provided 2D mask.The framework supports object incorporation at a designated scene region.
5. Experiments
Experiments show that Gaussian semantic tracing confines edits to desired regions, while HGS restrains scene-wide diffusion and stabilizes editing under stochastic guidance. GaussianEditor also supports diverse scenarios, quantitative gains, and editing sessions of about 5–10 minutes.
- Ablation Studies: Semantic tracing confines editing to the desired area despite generative guidance tending to modify the entire 2D image.
- Implementation Details: Editing uses 24–96 camera poses and typically optimizes for 500–1000 steps, taking about 5–10 minutes; 3D inpainting adds objects in roughly 5 minutes plus subsecond composition.
- Qualitative Comparisons: GaussianEditor-iN2N surpasses competing methods in edit quality and controllability, while Instruct-Nerf2Nerf lacks detail and editing-area control.
- Qualitative Comparisons: GaussianEditor supports large-scale scene editing and facial swaps, using semantic tracing for controllable and realistic facial edits.
- Quantitative Comparisons: GaussianEditor-iN2N outperforms in user studies and CLIP Directional Similarity, while editing takes only 5 to 10 minutes versus more than 30 minutes for Instruct-Nerf2Nerf.
- Ablation Studies: Without HGS, Gaussians spread and densify across the scene, causing uncontrolled densification and image blurring; HGS restrains this diffusion.
6. Conclusion
GaussianEditor combines Gaussian semantic tracing, HGS, and specialized 3D inpainting to improve controllability, efficiency, and object manipulation in Gaussian Splatting. Its limitation is reliance on 2D diffusion models, which may provide ineffective guidance for complex prompts.
- GaussianEditor uses semantic tracing, HGS, and specialized 3D inpainting to provide controllable and efficient Gaussian Splatting edits.
- Current 2D diffusion models can provide ineffective guidance for certain complex prompts, limiting GaussianEditor's 3D editing capability.
A. Introduction
The supplementary material provides additional qualitative results, a WebUI, and a video demonstrating GaussianEditor operations.
- The supplementary material includes qualitative results, a WebUI with tailored 3D-editing algorithms, and a video covering tracing, editing, deletion, and object addition.
B. More Results
Additional results show that GaussianEditor delivers controllable, diverse, high-resolution 3D editing in 2–7 minutes.
- GaussianEditor provides controllable, diverse, high-resolution 3D editing with runtimes of 2–7 minutes.
C. WebUI
GaussianEditor’s WebUI combines fast rendering with interactive semantic tracing for precise, view-guided editing of specific scene regions.
- GaussianEditor’s WebUI supports 2K resolution at 60fps, addressing the low-resolution and low-frame-rate experience of NeRF-based interfaces.
- Semantic Tracing with Point-based Prompts: Interactive semantic tracing addresses cases where text alone cannot specify which instance to edit among multiple similar objects.
- Semantic Tracing with Point-based Prompts: Users click screen points in a reference view, which are back-projected into spatial points using the camera parameters and corresponding depth.
- Semantic Tracing with Point-based Prompts: The spatial points are re-projected into other views, used as prompts for SAM segmentation, and then mapped back to Gaussians.
- Semantic Tracing with Point-based Prompts: With about five user-indicated points, semantic tracing provides finer-granularity control over the regions tracked during editing.
C.2. Object Incorporation with WebUI
GaussianEditor’s WebUI supports text-specified object incorporation into designated regions, while interactive depth adjustment helps align additions with the Gaussian scene.
- Object Incorporation with WebUI: Users draw 2D masks to designate where text-specified objects should be incorporated into the Gaussian scene.
- Object Incorporation with WebUI: Sliding a depth-scale control updates the added objects’ position and size, enabling real-time scaling through small Gaussian parameter adjustments.
- Object Incorporation with WebUI: Figure 9 illustrates point-based semantic tracing for object removal, using user clicks, segmentation, and resulting removals.
- Object Incorporation with WebUI: Monocular depth estimates may misalign generated objects, so the WebUI lets users adjust the estimated depth interactively in real time.
- Object Incorporation with WebUI: GaussianEditor’s additional results require 2-7 minutes and 10-20GB of GPU memory on a single A6000 GPU, with face-editing backgrounds unchanged.