Source-linked AI summary
GaussianSelector: Lightweight Human-Guided Object Selection in 3D Gaussian Splatting with Graph Optimization
Baihan Yang, Tiexin Li, Yuheng Liu, Xin Lin, Xinke Li, Xiaohui Xie, Truong Nguyen
TL;DR
Existing 3DGS object-selection methods often require dense multi-view observations, scribbles, pretrained segmentation models, or retraining. GaussianSelector directly propagates sparse scribble evidence through a Gaussian-native superpoint graph, achieving competitive performance with fewer views and lower computational cost.
Problem
Existing 3DGS interactive-selection pipelines depend on computationally expensive pretrained models, retraining, dense multi-view observations, and per-view scribbles.
Method
GaussianSelector coarsens native Gaussians into superpoints, lifts sparse scribbles with visibility-aware evidence, and solves graph-based energy minimization for object labeling.
Results
GaussianSelector achieves competitive selection quality while requiring significantly fewer interaction views, lower VRAM consumption, and substantially less computation.
Takeaways & Limitations
The framework supports lightweight human-in-the-loop 3D object selection directly in native 3DGS scenes without pretrained segmentation networks or representation retraining.
Abstract
from arXiv · showhide
Selecting a complete 3D object from a reconstructed scene with minimal user effort is essential for practical scene editing and embodied interaction. Existing 3DGS-based methods either retrain the Gaussian representation to embed per-object labels, or build dense multi-view SAM observations, both requiring heavy computation and dense viewpoint coverage that is rarely available in practice. We present GaussianSelector, a training-free framework for interactive 3D object selection from sparse views and sparse scribble guidance. Operating directly on native Gaussian primitives, we coarsen dense Gaussians into geometrically coherent superpoints and construct a continuity-weighted graph using appearance and spatial cues. Sparse user scribbles are lifted into 3D via visibility-aware transmittance coverage, and selection is solved as a global graph-cut energy minimization that propagates sparse evidence to a complete 3D object. This design naturally supports multi-round refinement, where users iteratively correct the selection from additional viewpoints to progressively improve the result. Experiments demonstrate that GaussianSelector achieves competitive selection quality against state-of-the-art multi-view SAM-based methods, while requiring significantly fewer interaction views and substantially lower computational overhead. These properties make it well suited for human-in-the-loop 3D scene editing and 3D asset extraction in real-world deployment scenarios.
1 Introduction
GaussianSelector addresses interactive object selection in reconstructed 3DGS scenes with a lightweight, training-free, Gaussian-native framework. It propagates sparse scribble guidance through a compact structural abstraction to select complete objects without pretrained segmentation networks or 3DGS retraining.
- Motivation: Interactive 3D object selection supports scene editing, asset extraction, robotic manipulation, and embodied perception, while 3DGS provides high-fidelity rendering with explicit 3D primitives.
- Limitations: Existing 3DGS methods commonly require multi-view scribbles, pretrained 2D models such as SAM, mask propagation, and sometimes 3DGS retraining, making them computationally expensive.
- Method: GaussianSelector directly operates on native 3D Gaussian primitives and propagates sparse user intent through geometrically coherent superpoints and a structured graph.
- Contributions: The framework is training-free, supports sparse scribbles from one or a few interaction views, and avoids pretrained segmentation networks and 3DGS retraining, finetuning, or refinement.
- Evaluation: On the NVOS benchmark, GaussianSelector achieves competitive selection quality against state-of-the-art multi-view SAM-based methods with fewer interaction views, lower VRAM consumption, and less computation.
2 Related Works
Prior 3DGS segmentation methods either learn semantic feature fields from 2D foundation-model supervision or lift 2D SAM results into 3D through cross-view consistency. Graph-based approaches improve spatial modeling, but existing methods either optimize fine-grained global representations or learn semantic embeddings for superpoints.
- 3DGS Segmentation by Semantic Feature Field Learning: Semantic feature-field methods distill cues from 2D foundation models into 3DGS through supervision from multi-view rendered images.Examples include LangSplat, VLGaussian, Feature3DGS, and N2F2, using CLIP, DINO, or LSeg.
- 3DGS Segmentation by SAM Lifting: SAM-lifting methods use the differentiable 3DGS renderer to transfer 2D segmentation results into 3D via cross-view consistency alignment.This direction seeks to avoid the computational and annotation costs of large-scale feature-field learning.
- 3D Segmentation via Graphs: Graph-based segmentation methods model spatial neighborhoods and include interactive object selection with graph-cut energy minimization across different 3D representations.NVOS and GaussianCut apply graph-cut formulations to interactive segmentation, but the supplied passage notes that they operate directly on global 3D representations.
- 3D Segmentation via Graphs: Superpoint graphs provide more expressive and efficient scene understanding, but existing 3DGS approaches still learn semantic embeddings to define superpoints.Open3DIS and Hui et al. target point clouds, while AG2aussian and InstanceGaussian operate in 3D Gaussian Splatting; global fine-grained optimization remains costly in earlier graph methods.
3 Method
GaussianSelector formulates 3D object selection as binary labeling over scene-native superpoints, combining scribble-derived evidence with a scene-continuity prior. It coarsens Gaussians, lifts sparse scribbles through visibility-aware coverage, and finds the globally optimal labeling using graph-cut optimization.
- MAP formulation: The method minimizes a binary submodular Potts energy whose unary costs encode scribble evidence and pairwise costs penalize disagreement across continuity-weighted edges.This energy has an exact global minimum via s-t min-cut and is instantiated through scene encoding, seed broadcasting, and global optimization.
- Scene encoding: Leiden community detection aggregates nearby, similarly appearing Gaussians into superpoints that provide low-frequency label support and place boundaries between communities.Each superpoint stores mean position, CAC appearance, and opacity, stabilizing boundaries when individual Gaussians are under-constrained or artifact-prone.
- Scene encoding: A k-NN superpoint graph weights edges by normalized spatial, CAC, and opacity distances, while gating removes edges with excessive dissimilarity.The resulting continuity prior is computed once per scene and makes cuts costly across continuous regions while concentrating them at scene discontinuities.
- Seed broadcasting: Visibility-aware transmittance coverage lifts foreground and background scribbles to Gaussian seed labels, leaves ambiguous cases unlabeled, reconciles multiple views by majority vote, and aggregates labels to superpoints.Coverage accounts for each Gaussian’s alpha-composited contribution rather than relying only on projected-center inclusion.
- Likelihood modeling: Foreground and background seed features combine standardized CAC appearance and opacity, enabling independent Gaussian mixture models to produce a calibrated log-likelihood-ratio evidence field.Positive margins favor foreground and negative margins favor background; the unary cost also softly reinforces seeded superpoints toward their designated class.
4 Experiments
GaussianSelector is evaluated on NVOS and 3D-OVS through sparse-view, single- and multi-round interaction settings, achieving strong selection quality with lightweight iterative refinement. Ablations, qualitative comparisons, and a user study further examine the method’s effectiveness and usability.
- Datasets and settings: Evaluation uses the 8-task LLFF-NVOS benchmark and 5 scenes from 3D-OVS, with initial scribbles plus one or two additional refinement rounds.The experiments compare sparse interaction settings against reported dense multi-view SAM-lifting-based results and include qualitative studies across reconstruction settings.
- Quantitative results: 85.3 mIoU is achieved from a single NVOS-scribble view, outperforming NVOS at 70.1 mIoU without multi-view mask propagation or pretrained 2D segmentation models.This single-round setting uses only spatially sparse benchmark scribbles as input.
- Quantitative results: 89.6 mIoU follows one additional refinement view, while 92.2 mIoU is reached after two additional novel-view interaction rounds.The resulting performance is described as comparable to strong baselines built on dense-view priors.
- Ablation study: 33.2 mIoU with scribbles alone rises to 61.0 mIoU after graph propagation, demonstrating the contribution of graph connectivity to sparse-evidence propagation.The ablation separates unary evidence modeling from graph propagation under the NVOS benchmark.
- Qualitative analysis: Additional human-in-the-loop refinement improves boundary consistency and reduces artifacts, while GaussianCut is more sensitive to sparse or less informative inputs.The qualitative comparison examines progressively enriched sparse-view conditions.
- User study: A 12-participant user study evaluates intent matching, blind visual satisfaction, waiting acceptability, and SUS, finding more precise control and shorter response time for scribble-based interaction.Participants rated the first three criteria on 7-point Likert scales and completed the System Usability Scale questionnaire.
5 Conclusion
GaussianSelector is a plug-and-play, neural-network-free framework that performs interactive object selection directly in native 3DGS space through graph-based abstraction and evidence modeling. It matches state-of-the-art 3DGS-based methods while reducing computational cost and VRAM usage.
- Framework: GaussianSelector reformulates interactive 3DGS object selection as graph-based abstraction and evidence modeling in native 3DGS space.The framework is plug-and-play and does not use neural networks.
- Evaluation: GaussianSelector achieves comparable performance with state-of-the-art 3DGS-based methods in human-in-the-loop workflows.The comparison concerns interactive object selection performance.
- Efficiency: GaussianSelector uses significantly less computational cost and VRAM than the compared 3DGS-based methods.The efficiency advantage is reported for both computational cost and VRAM usage.