Source-linked AI summary
Gaussian Grouping: Segment and Edit Anything in 3D Scenes
Mingqiao Ye, Martin Danelljan, Fisher Yu, Lei Ke
TL;DR
Gaussian Splatting reconstructs scenes efficiently but lacks fine-grained object-level understanding. Gaussian Grouping adds compact Identity Encodings supervised by SAM masks and 3D spatial consistency, enabling grouped 3D reconstruction, segmentation, and editing. The method supports varied scene-editing applications with high visual quality, fine granularity, and time efficiency.
Problem
Gaussian Splatting focuses on appearance and geometry but lacks fine-grained object-level scene understanding for open-world 3D scenes.
Method
Gaussian Grouping augments each Gaussian with a compact Identity Encoding supervised through differentiable rendering from SAM masks and 3D spatial consistency.
Results
Gaussian Grouping jointly reconstructs and segments anything in open-world 3D scenes while supporting high-quality, fine-grained, time-efficient editing applications.
Takeaways & Limitations
Grouped discrete 3D Gaussians provide a compositional representation for object removal, inpainting, style transfer, colorization, and scene recomposition.
Takeaways & Limitations
The method is currently limited to static 3D scenes and leaves fully unsupervised 3D Gaussian grouping for future work.
Abstract
from arXiv · showhide
The recent Gaussian Splatting achieves high-quality and real-time novel-view synthesis of the 3D scenes. However, it is solely concentrated on the appearance and geometry modeling, while lacking in fine-grained object-level scene understanding. To address this issue, we propose Gaussian Grouping, which extends Gaussian Splatting to jointly reconstruct and segment anything in open-world 3D scenes. We augment each Gaussian with a compact Identity Encoding, allowing the Gaussians to be grouped according to their object instance or stuff membership in the 3D scene. Instead of resorting to expensive 3D labels, we supervise the Identity Encodings during the differentiable rendering by leveraging the 2D mask predictions by Segment Anything Model (SAM), along with introduced 3D spatial consistency regularization. Compared to the implicit NeRF representation, we show that the discrete and grouped 3D Gaussians can reconstruct, segment and edit anything in 3D with high visual quality, fine granularity and efficiency. Based on Gaussian Grouping, we further propose a local Gaussian Editing scheme, which shows efficacy in versatile scene editing applications, including 3D object removal, inpainting, colorization, style transfer and scene recomposition. Our code and models are at https://github.com/lkeab/gaussian-grouping.
1 Introduction
Gaussian Grouping addresses limitations in open-world 3D scene understanding by extending Gaussian Splatting with grouped, identity-aware 3D representations supervised from 2D masks. The resulting representation supports fine-grained segmentation and local scene editing.
- Open-world 3D understanding lacks scalable 3D supervision because existing methods depend on costly manual labels, limited datasets, or accurately scanned point clouds.
- Gaussian Splatting offers efficient explicit scene reconstruction but does not model object instances or semantic understanding.
- Gaussian Grouping learns grouped 3D Gaussians from multi-view captures and SAM-generated masks to reconstruct and segment anything across novel views.
- Identity Encoding is a compact learnable embedding attached to each Gaussian and trained through differentiable rendering for 2D identity classification.
- A 3D spatial consistency loss brings the Identity Encodings of K-nearest Gaussians closer, improving supervision for interior or heavily occluded regions.
- The framework's grouped representation enables local components to be independently identified, manipulated, or replaced for scene editing.
2 Related Works
Related work extends Gaussian Splatting beyond static reconstruction toward dynamic-scene modeling and diffusion-based 3D content creation.
- Follow-up Gaussian Splatting methods address dynamic scenes through dense-element tracking or deformation-field modeling.
- Another line of work combines Gaussian Splatting with diffusion-based models for 3D content creation and high-quality generation.
3D Gaussian Models
The paper positions grouped 3D Gaussians as an explicit representation for open-world, fine-grained scene understanding and editing, contrasting it with NeRF-based alternatives.
- Grouped 3D Gaussians are presented as an effective and flexible representation for open-world, fine-grained scene understanding and downstream editing.
- NeRF-based semantic methods incorporate semantic masks, instances, or visual features into radiance fields for novel-view scene understanding.
- Open-world feature-field approaches are limited in separating similar same-category objects and producing accurate segmentation masks, according to the paper.
- Gaussian Grouping uses SAM's automatic everything mode to pursue holistic instance-and-stuff understanding across the full scene.
- Existing NeRF-based editing methods are described as limited to simple objects rather than cluttered, textured scenes.
3 Method
Gaussian Grouping extends explicit 3D Gaussian Splatting with Identity Encodings learned from SAM-derived, cross-view-consistent masks, enabling grouped scene understanding and local editing. The pipeline jointly renders appearance and identity features, regularizes spatially neighboring Gaussians, and edits only target groups when possible.
- 3 Method: The method extends Gaussian Splatting from appearance and geometry reconstruction to fine-grained scene understanding while retaining fast training and rendering.Its discrete groups represent individual scene parts and support object-level manipulation without retraining the entire global representation.
- Input Preparation: SAM generates masks independently for each training view, and zero-shot tracking associates them into consistent multi-view mask identities.The tracking step supplies coherent segmentation labels and the total number of scene instances or stuff regions.
- 3.2 3D Gaussian Grouping: Gaussian Grouping adds compact Identity Encoding parameters to each Gaussian while preserving the original appearance, geometry, opacity, and size attributes.The encoding is a learnable 16-dimensional vector with spherical-harmonic degree 0, reflecting view-consistent instance identities.
- 3D Gaussian Rendering and Grouping: Differentiable Gaussian rendering splats Identity Encodings into 2D feature images, where a linear classifier and cross-entropy loss supervise the rendered identities.The rendered identity features are optimized alongside the scene’s other Gaussian properties.
- Grouping Loss: A 3D regularization loss brings the Identity Encodings of top k-nearest Gaussians closer, improving supervision for interior or heavily occluded regions.This spatial consistency term complements indirect 2D supervision and directly regularizes the Gaussian encodings.
- 3.3 Gaussian Grouping for Scene Editing: Grouped Gaussians support local editing by freezing most of the scene and modifying only target groups or newly added Gaussians for removal, recomposition, inpainting, colorization, and style transfer.Removal and recomposition can be performed directly; other edits adjust selected colors, positions, sizes, or newly added Gaussians.
4 Experiments
Experiments evaluate Gaussian Grouping on segmentation, reconstruction, efficiency, robustness, and editing-related ablations across several 3D datasets. The results support cross-view mask association, Identity Encoding, and 3D regularization as effective design choices.
- Datasets: The LERF-Mask dataset annotates three LERF-Localization scenes with accurate masks and averages 7.7 text queries per scene.The study also provides panoptic segmentation results on Replica and ScanNet.
- Datasets: The evaluation uses seven public Mip-NeRF 360 scenes plus diverse LLFF, Tanks & Temples, and Instruct-NeRF2NeRF cases for editing comparisons.Flowers and treehill are omitted because public access rights were unavailable.
- Ablations: Cross-view mask association simplifies training, while cost-based linear assignment slows training and produces noisier, inferior reconstruction and segmentation results.The associated-mask input requires 1 minute, and preserves appearance details such as the ceiling rope more effectively.
- Robustness: Gaussian Grouping corrects some DEVA mask-label errors during multi-view rendering through its shared 3D Gaussian representation.This remains effective even when tracking loses partial 2D mask labels.
- Identity Encoding: Identity Encoding leaves original Gaussian reconstruction quality on par with Gaussian Splatting while enabling whole-scene segmentation and downstream editing.The model trains reconstruction and segmentation components jointly end to end.
- Ablations: 16-dimensional Identity Encoding balances object separation and efficiency, whereas doubling the dimension to 32 makes training 1.3 times slower without better reconstruction quality.The selected 3D regularization setting K = 5 provides the best balance between reconstruction and object-removal accuracy.
- Ablations: Joint 2D and 3D identity supervision addresses the transparent bear issue caused by insufficient supervision of occluded interior Gaussians.The 3D regularization loss supplies additional supervision for Gaussians that are hidden during training.
- 3D Multi-view Segmentation: Gaussian Grouping doubles LERF and SA3D performance on both the figurines and ramen scenes, while producing sharper boundaries and better separation of similarly colored objects.Grounding DINO selects mask IDs for language-guided comparison because SAM does not support language prompts.
3D Panoptic Segmentation Comparison
Gaussian Grouping outperforms Panoptic Lifting in both panoptic segmentation performance and speed under the shared semantic-mask-label setting.
- 3D Panoptic Segmentation Comparison: Gaussian Grouping outperforms Panoptic Lifting in both performance and speed on novel-view panoptic segmentation.Both methods use semantic mask labels generated by Mask-DINO under the reported comparison setting.
3D Object Removal
Gaussian Grouping supports 3D object removal by separating target Gaussians from scene backgrounds, including large objects, with fewer visible artifacts than DFFs.
- 3D Object Removal: Gaussian Grouping removes large objects such as trucks from Tanks & Temples scenes with greatly reduced artifacts and without leaving a blurry background.The comparison is against DFFs.
- 3D Object Removal: For challenging large-object cases, Gaussian Grouping clearly separates the foreground object from the background, whereas DFFs can produce complete foreground removal or inaccurate removal with obvious artifacts.DFF limitations are attributed to the quality of its CLIP-distilled features.
3D Object Inpainting
Local Gaussian Editing removes target Gaussians and fills unobserved background regions for view-consistent 3D object inpainting, while supporting multi-object editing and style transfer.
- 3D Object Inpainting: 3D object inpainting detects regions invisible across all views after object deletion and inpaints those regions rather than the entire 2D object area.The goal is a photo-realistic, view-consistent natural 3D scene.
- 3D Object Inpainting: Gaussian Grouping achieves better 3D object inpainting quality than SPIn-NeRF with 1 hour of training and 20 minutes of finetuning, compared with 5 hours of training.The comparison concerns the cases shown in Figure 10.
- 3D Object Inpainting: Inpainting results better preserve spatial detail and multi-view coherence than SPIn-NeRF in the reported comparison.Each rendering view uses a 2D inpainted image to guide learning of newly introduced 3D Gaussians.
- Local Gaussian Editing: The framework supports concurrent 3D object removal and glove colorization within the same scene, alongside visual style-transfer comparisons.The style-transfer example uses the instruction “Turn the bear into a panda”.
3D Object Style Transfer
Gaussian Grouping supports efficient 3D object style transfer and produces more coherent, natural results across views than Instruct-NeRF2NeRF in the reported comparison.
- Gaussian Grouping produces more coherent and natural transferred bear appearances across views than Instruct-NeRF2NeRF.Both methods use the instruction “turn the bear into a panda” and the same InstructPix2Pix image guidance.
3D Multi-Object Editing
Grouped 3D Gaussians support concurrent editing of multiple scene objects while preserving non-interference, although the method is currently limited to static scenes.
- Grouped 3D Gaussians enable concurrent editing of several objects while maintaining non-interference among them.
- The method is limited to static 3D scenes because it lacks dynamic modeling and time-dependent updating.
5 Conclusion
Gaussian Grouping jointly reconstructs and segments anything in open-world 3D scenes using grouped Gaussians with Identity Encoding supervised by 2D SAM masks and 3D spatial consistency. The resulting representation supports versatile, high-quality, time-efficient 3D editing applications.
- Gaussian Grouping jointly reconstructs and segments anything in open-world 3D scenes with a 3D Gaussian-based representation.
- Identity Encoding is supervised by 2D SAM mask predictions and 3D spatial consistency.
- The grouped, discrete representation supports object removal, inpainting, style transfer, and scene recomposition with high-quality visual effects and good time efficiency.
6 Appendix
The appendix reports efficiency, editing, tracking, dataset, implementation, and limitation analyses, including comparisons with prior methods and detailed local editing procedures.
- Segmentation Efficiency: Gaussian Grouping jointly segments all scene objects in 9 minutes, whereas SA3D requires 35 minutes for each object.
- Mask Processing: SAM generates dense mask proposals, which are ranked by area and filtered using overlapping IoU thresholds to select varying granularities.
- Quantitative Evaluation of Editing: The editing evaluation compares inpainting, style transfer, and object removal using CLIP Text-Image Direction Similarity.
- Robustness: SAM plus tracking assigns a new instance ID when a high-confidence segmentation does not match previous objects, while sparse-view tracking can still preprocess 3-view inputs.
- Annotation Pipeline: LERF-Mask contains accurate masks for three LERF-Localization scenes, averaging 7.7 text queries per scene with ground-truth masks.
- 3D Object Inpainting: Object inpainting deletes target Gaussians, detects rendered holes for 2D inpainting, adds 200K new Gaussians, and fine-tunes only the new Gaussians.
- 3D Object Style Transfer: Style transfer fine-tunes only Gaussians associated with the selected object while freezing irrelevant Gaussians and using InstructPix2Pix to update target images.
- Limitations: Open-vocabulary segmentation can fail for complicated language prompts, and DEVA’s zero-shot 2D association accuracy limits open-world 3D segmentation performance.