PaperlayerView exact revision on arXiv

Source-linked AI summary

SplatScout: Query-Driven Part-Level Referring Segmentation in Gaussian Scenes

Yihan Zang, Da Li, Shinkyu Park, Ivan Viola, Dominik Engel

arXiv:2610.37098v1cs.CV

TL;DR

Existing Gaussian-scene referring segmentation largely targets whole objects and relies on predetermined candidates that may miss query-specific parts. SplatScout lets a VLM control observation and segmentation in a feedback loop; it substantially outperforms existing methods on PartRef-ScanNet++ and scores higher on four existing referring benchmarks.

  • Problem

    Gaussian-scene referring methods primarily target whole objects, while fixed candidate sets may omit fine-grained parts needed as targets or cues for identifying objects.

    Source: §1 · p. 1

  • Method

    SplatScout uses a query-driven closed loop in which the VLM selects observations, directs segmentation, constructs 3D instances, and refines predictions using feedback.

    Source: §1 · p. 2

  • Results

    SplatScout substantially outperforms existing methods on PartRef-ScanNet++ and achieves higher scores on four of five existing referring benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PartRef-ScanNet++ evaluates varied part-level referring expressions with mesh-derived ground truth, and the experiments show benefits from iterative interaction and complementary feedback from perceptual stages.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Existing referring segmentation methods on Gaussian scenes primarily target whole objects, relying on segmenter-generated candidates from which a vision-language model (VLM) selects. Such predetermined fixed candidates may omit fine-grained parts needed as targets or cues for identifying objects. We introduce SplatScout, a query-driven, closed-loop approach that places perception under VLM control. The model actively selects views, directs segmentation, and inspects rendered 3D results to refine its predictions. For part-level evaluation, we introduce PartRef-ScanNet++, comprising 261 manually verified queries across 12 ScanNet++ scenes with mesh-derived ground truth. On PartRef-ScanNet++, SplatScout achieves 0.538 mIoU, substantially outperforming existing methods. Ablations demonstrate the benefits of iterative refinement, while comparisons across VLM backbones highlight the importance of effective tool-feedback utilization. SplatScout also achieves higher scores on four existing referring benchmarks. Code and the PartRef-ScanNet++ benchmark are available at https://github.com/kaust-violab/SplatScout.

1 Introduction

Existing Gaussian-scene referring methods focus on whole objects, while practical queries may require parts at query-dependent granularity. SplatScout puts perception under VLM control in a closed loop, and the paper reports improved referring segmentation across part-level and existing benchmarks.

1 Introduction

  • Existing Gaussian-scene referring methods primarily target whole objects, although queries may require parts as targets, discriminators, or anchors at query-dependent granularity.Grounding also requires query-dependent views and segmentation regions that may become apparent only during perception, making fixed scene decompositions unreliable for supplying all needed candidates.Source: §1 · p. 1
  • SplatScout transfers perception control to the VLM, which decides where to look, what to segment, and at what granularity instead of selecting from predetermined instances.The Navigator selects observations, the Segmenter produces query-directed masks, and the Lifter constructs 3D Gaussian instances; rendered lifted instances provide feedback for error correction.Source: §1 · p. 2
  • A closed loop lets the VLM inspect intermediate results and revise decisions when observations, masks, or lifted 3D instances expose errors.The next action may depend on previous results, so the model can progressively refine the target rather than commit to its initial predictions.Source: §1 · p. 2
  • SplatScout substantially improves referring segmentation on PartRef-ScanNet++ and achieves higher scores on four of five existing referring benchmarks.Controlled experiments investigate closed-loop perception and how VLM capabilities use perceptual feedback.Source: §1 · p. 2
  • PartRef-ScanNet++ provides 261 manually verified referring expressions across 12 ScanNet++ scenes, with ground truth verified on laser-scanned meshes.It covers whole objects, parts used in different referring roles, and multiple-target queries.Source: §1 · p. 2
  • The paper experimentally investigates fixed segmentation candidate pipelines against VLM-guided perception with visual feedback, particularly for part-level referring expressions.The motivation is that selecting among candidates cannot recover a target part that is missing or segmented at the wrong granularity.Source: §1 · p. 2

2 Related Work

Prior Gaussian-scene language methods attach semantic features or use VLMs to select among segmenter-generated candidates. Part-level methods and benchmarks often rely on predefined regions or targets, whereas SplatScout and PartRef-ScanNet++ support query-adaptive segmentation and varied part roles.

Grounding without a VLM

Grounding with a VLM

Part-level grounding

  • Part-aware Gaussian methods commonly use pre-generated whole-object, part, and sub-part masks, limiting granularity to regions identified during preprocessing [Qin et al., 2024, Xie et al., 2026, Chacko et al., 2025].Point-cloud part methods use 2D detectors and segmenters or fixed superpoints, while other work evaluates bounding-box grounding or supplies detailed part annotations [Corsetti et al., 2025, Wang et al., 2025b] [Wu et al., 2025] [Wang et al., 2025a] [Halacheva et al., 2025, Mao et al., 2022].Source: §2 · p. 3
  • PartRef-ScanNet++ covers parts as targets, discriminators, and relational anchors with mesh-level ground truth, while SplatScout supports these roles through query-adaptive part segmentation.Existing benchmarks primarily treat parts as predefined targets, with limited coverage of their roles in identifying other objects.Source: §2 · p. 3

3 Method

SplatScout uses a VLM to coordinate scene exploration, query-directed segmentation, and 3D instance construction, adapting its actions to the query and prior feedback. Its tools return intermediate results for inspection and refinement rather than relying on precomputed semantic candidates.

3.1 Preliminaries

  • SplatScout represents each scene with planar 2D Gaussians and returns only lifted instances as answers.A group is masks of one thing across one or more frames before lifting; an instance is a subset of Gaussians obtained by lifting.Source: §3.1 · p. 3

3.2 Method Overview

  • The VLM coordinates the Navigator, Segmenter, and Lifter to retrieve views, produce query-directed masks, and integrate them into 3D Gaussian instances.Each tool returns intermediate results the VLM can inspect before choosing its next action.Source: §3.2 · p. 4, §3.4 · p. 5

3.3 Closed-Loop Grounding

  • At each step, the VLM chooses an action using the referring expression and previous feedback, then may explore views, revise masks, merge observations, or inspect lifted instances.The adaptive interaction constructs target instances during query execution, without precomputed semantic candidates.Source: §3.3 · p. 4, §3.2 · p. 4

3.4 Perception Tools

  • The perception tools provide the VLM with scene observations, query-directed 2D masks, and 3D Gaussian subsets for inspection.The Navigator retrieves observations, the Segmenter constructs masks, and the Lifter converts masks into Gaussian subsets.Source: §3.4 · p. 5

3.4.1 Navigator: Active Scene Exploration

  • The Navigator supports global exploration for locating regions and local inspection for higher-resolution, complementary views.Global exploration greedily selects frames that reveal the largest weighted portion of the scene not yet observed; local inspection retrieves nearby views with moderate overlap.Source: §3.4.1 · pp. 5–6

3.4.2 Segmenter: Query-Adaptive Segmentation

  • The Segmenter generates query-specific masks from VLM-selected frames, boxes, and text prompts, and can constrain part segmentation to a parent object.It returns overlays, cut-outs, multi-view results, and quality indicators so the VLM can decide whether to revise, explore, or lift.Source: §3.4.2 · p. 6

3.4.3 Lifter: Multi-View Gaussian Instance Construction

  • The Lifter groups multi-view masks and aggregates their geometric evidence to identify Gaussian primitives corresponding to each segmentation group.When requested, it refines a lift with complementary views and additional segmentation; rendered instances are returned for inspection rather than automatically accepted.Source: §3.4.3 · pp. 6–7

4 Experiments

The experiments evaluate part-level grounding on a mesh-grounded benchmark and compare SplatScout with existing methods, VLM backbones, and ablations. SplatScout leads on PartRef-ScanNet++, while results across benchmarks, model choices, and oracle tests qualify where performance gains arise.

4.1 PartRef-ScanNet++ Dataset and Evaluation Metrics

  • PartRef-ScanNet++ contains 261 manually verified queries across 12 scenes, with mesh-verified ground truth spanning objects, part roles, and multiple targets.Evaluation averages IoU over eligible held-out frames; five queries have insufficient target visibility and are excluded.Source: §4.1 · p. 7

4.2 3D Visual Grounding Experiments

  • 0.538 mIoU versus 0.253 for the strongest baseline, with substantial gains across all PartRef-ScanNet++ query types.Table 1 groups results by Obj (76), Part (151), and Multi (29).Source: §4.2 · p. 8, Table 1 · p. 8
  • SplatScout outperforms published baselines on four of five external benchmarks; on GR-ScanNet it achieves 24.0 mIoU versus ZeroSplat’s 41.2.The authors identify a likely granularity mismatch: GR-ScanNet uses category-level labels, while SplatScout predicts query-specific instances.Source: §4.2 · p. 8, Table 2 · p. 8

4.3 Effect of Model Capability

  • SplatScout improves from 0.128 to 0.450 mIoU across the compared VLM backbones, while ZeroSplat stays around 0.09 and OpenGround ranges from 0.161–0.180.With Qwen3.5-397B, SplatScout also outperforms both baselines when they use ground-truth information.Source: §4.3 · pp. 8–9, Table 3 · p. 9

4.4 Ablations and Oracle Experiments

  • Restricting SplatScout to one observation and segmentation round reduces mIoU from 0.450 to 0.209; removing visual exploration causes the largest feedback-ablation drop, to 0.319.Suppressing segmentation and lifting feedback yields 0.412 and 0.440 mIoU, respectively.Source: §4.4 · p. 9, Table 4 · p. 9
  • Ground-truth boxes raise mIoU to 0.571, frame selection to 0.578, and replacing segmentation masks with ground truth to 0.689.The authors identify accurate query-directed segmentation as an important remaining source of improvement.Source: §4.4 · p. 9, Table 4 · p. 9

5 Conclusion

SplatScout uses VLM-controlled, closed-loop perception to construct query-specific Gaussian instances without precomputed semantic candidates. It substantially outperforms existing methods on its part-level benchmark and achieves higher scores on four external benchmarks.

5 Conclusion

  • The paper concludes that SplatScout’s iterative interaction benefits referring segmentation, with feedback from different perceptual stages playing complementary roles.The method places observation selection, instance construction, and refinement under VLM control; PartRef-ScanNet++ supplies mesh-derived ground truth for diverse part-level expressions.Source: §5 · p. 9

A Closed-Loop Grounding Implementation

The implementation describes the available VLM tools and a closed-loop grounding algorithm. The algorithm incorporates tool feedback into the interaction history and terminates with an answer or failure report.

A Closed-Loop Grounding Implementation

B.1 Benchmark Comparison and Statistics

PartRef-ScanNet++ broadens referring-benchmark coverage to part-level roles and mesh-derived segmentation, while SplatScout scores higher than OpenGround on sampled OpenTarget queries. The section also describes benchmark construction and its held-out evaluation protocol.

B.1 Benchmark Comparison and Statistics

  • PartRef-ScanNet++ covers objects and parts as targets, discriminators, and relational anchors, with mesh-level segmentation ground truth, unlike box-evaluated articulated-part OpenTarget.OpenTarget additionally includes whole objects, nonarticulated parts, and multi-target expressions in PartRef-ScanNet++.Source: §B.1 · p. 14
  • SplatScout achieves higher box accuracy under both IoU thresholds on 300 sampled OpenTarget queries from 12 shared scenes.Both methods use Qwen3.5-397B; two queries blocked by API filtering are excluded.Source: Table 7 · p. 14, §B.1 · p. 14
  • PartRef-ScanNet++ has 261 queries across 12 scenes: 78 object targets, 92 part targets, 32 part discriminators, 30 part anchors, and 29 multi-target queries.154 queries (59%) involve parts as targets, discriminators, or anchors; Fig. 3 shows the role distribution.Source: §B.1 · p. 14, Figure 3 · p. 15

B.2 Dataset Construction

  • The 12 ScanNet++ scenes provide laser-scanned meshes and posed DSLR images; annotations comprise 1,616 object instances and Articulate3D-derived parts supplemented with 24 manually verified parts.Annotations are represented as vertex subsets of the same meshes.Source: §B.2 · p. 15
  • Annotators reviewed 357 initial queries, retaining 254, revising nine, and discarding 94; after deduplication, the benchmark contains 261 queries.AI-assisted candidates were reviewed for naturalness, unique targets, qualifying-condition necessity, target visibility, and annotation validity; viewpoint-dependent expressions were excluded.Source: §B.2 · p. 15

B.3 Evaluation Protocol

  • Ground-truth masks are ray-traced from laser-scanned meshes, assigning pixels only when the first ray intersection lies on the annotated target surface.Privacy-masked regions are excluded, and only frames with at least 300 visible ground-truth pixels are evaluated.Source: §B.2 · p. 15, §B.3 · p. 15
  • Evaluation uses 216 held-out test frames excluded from all methods’ perception and preprocessing; 256 of 261 queries are evaluable because five lack sufficient target visibility.Predictions are scored by mean IoU across queries, plus Acc@0.25 and Acc@0.5.Source: §B.3 · pp. 15–16
  • The evaluator accepts binary Gaussian subsets, per-Gaussian soft weights, or mesh vertex subsets; rendered Gaussian membership is composited with original opacities for occlusion-aware scoring.A pixel is foreground when target contribution and total accumulated opacity each exceed 0.5; a triangle in a vertex-based submission is included when at least two vertices are selected.Source: §B.3 · p. 15

C Reproduction of Unreleased Baselines

Because ZeroSplat and GaussDet implementations were not publicly available at the time, the authors reimplemented them and compared their Ref-LERF results with published scores. The reproduced implementations were used for PartRef-ScanNet++ comparisons.

C Reproduction of Unreleased Baselines

  • The authors reimplemented ZeroSplat and GaussDet because their implementations were not publicly available, and used those reproductions on PartRef-ScanNet++.The reproduced methods were evaluated on four Ref-LERF scenes to document differences from published results; other benchmark results were taken from the original papers.Source: C Reproduction of Unreleased Baselines · p. 16
  • ZeroSplat and GaussDet reproduce at 28.4 and 40.2 mIoU, respectively, versus published results of 32.7 and 45.9.Differences vary by scene, with larger discrepancies on Teatime for ZeroSplat and Figurines for GaussDet.Source: C Reproduction of Unreleased Baselines · p. 16, Table 8 · p. 16

D External Benchmark Evaluation Protocols

SplatScout is evaluated on five existing benchmarks using their original protocols and training frames, without changes to its perception or grounding pipeline. The section distinguishes 2D LERF scoring from fixed-point ScanNet scoring and specifies handling of zero- and multi-target queries.

D External Benchmark Evaluation Protocols

  • Across Ref-LERF, GR-LERF, GR-ScanNet, Causal-LERF, and Causal-ScanNet, evaluation uses training frames, the original protocols, and no changes to SplatScout’s pipeline.Baseline results are taken from the respective publications.Source: D External Benchmark Evaluation… · p. 16
  • Ref-LERF, GR-LERF, and Causal-LERF score rendered predictions against provided masks using 2D mIoU on four LERF scenes.The benchmark-specific scoring protocols are followed.Source: D External Benchmark Evaluation… · p. 16
  • GR-ScanNet and Causal-ScanNet use point-level 3D mIoU with one Gaussian per input point and no densification or position updates, preserving direct label correspondence.This setup avoids nearest-neighbor matching or label transfer.Source: D External Benchmark Evaluation… · p. 16
  • For GR-LERF and GR-ScanNet, empty predictions score one on zero-target queries and zero otherwise; remaining queries score the union of predicted target instances.Source: D External Benchmark Evaluation… · p. 16

E VLM and Perception Analysis

The analysis tests ground-truth interventions and examines how VLM backbones use SplatScout’s feedback. The results distinguish gains from candidate or prompt changes from the stronger model’s more effective tool use.

E VLM and Perception Analysis

  • Across VLM backbones, SplatScout uses the same prompts, tools, and execution budgets; API filtering reduces some runs from 256 evaluable queries to 254–255.All methods use training frames and fixed method-specific configurations; some ZeroSplat decoding settings differ between runs.Source: E VLM and Perception Analysis · p. 17
  • GT-guided prompting gives ZeroSplat the correct category and ground-truth boxes while retaining its segmentation and lifting pipeline, reaching 0.293 mIoU.Replacing segmentation outputs with GT masks alone decreases performance because incorrect instance selections can still yield accurate distractor masks.Source: E VLM and Perception Analysis · p. 17
  • OpenGround reaches 0.424 mIoU with official ScanNet++ candidates and 0.435 mIoU under oracle selection from an augmented static table.The oracle selects one candidate per query, including for multi-target queries, and the official table has limited part coverage.Source: E VLM and Perception Analysis · p. 17
  • The 30B model reaches its execution budget without a valid terminal action on 65 queries, versus none for 397B; re-segmentation after failure is 52% versus 69%.The authors report these observations as helping explain the stronger backbone’s larger gains under closed-loop perception.Source: E VLM and Perception Analysis · p. 17

F Ablation and Oracle Details

Feedback ablations show that visual exploration, segmentation feedback, and lifting feedback contribute differently to tool use and part-level accuracy. Progressive oracle settings replace model decisions with ground truth to isolate sources of error.

Feedback ablations

  • The one-look, one-cut setting limits the VLM to four overview images and one frame-selection and segmentation decision, while retaining automatic multi-view lifting.Feedback ablations suppress one visual feedback channel at a time while preserving the other two and allowing permitted tool use and revision.Source: F Ablation and Oracle Details · p. 17
  • Without visual exploration, segmentation calls rise from 2.5 to 4.6 per query, and queries reported as not found rise from 15 to 62.The VLM also makes 58 attempts to specify boxes in frames not previously visualized through the Navigator.Source: F Ablation and Oracle Details · p. 17, Table 9 · p. 18
  • Without segmentation feedback, lifting calls rise from 1.8 to 4.9 per query, interaction length from 11.0 to 13.4 steps, and token use from 118k to 131k per query.Lifting feedback partially substitutes for direct mask inspection but requires additional interaction.Source: F Ablation and Oracle Details · p. 17, Table 9 · p. 18
  • Removing lifting feedback lowers part-level mIoU from 0.453 to 0.429, though segmentation feedback still lets the VLM inspect and revise 2D predictions.The effect on overall accuracy is described as smaller.Source: F Ablation and Oracle Details · p. 17

Progressive oracle experiments

  • Progressive oracle settings replace predicted boxes and parent scopes, then frame choices, then SAM3 masks with mesh-derived ground truth to isolate segmentation quality.GT masks retains the selected frames and replaces SAM3 outputs with mesh-projected ground-truth masks, including during multi-view refinement.Source: F Ablation and Oracle Details · pp. 17–18

Related papers

Loading 2610.37098v1…