PaperlayerView exact revision on arXiv

Source-linked AI summary

ClusterSplat: Semantic Cluster Selection for 3D Visual Grounding in Gaussian Splatting

Jaesung Lee, Hyeontaek Hwang, Gabriel Manalu, Daeyoung Kim

arXiv:2610.36462v1cs.CV

TL;DR

Primitive-level relevance scores vary across scenes and prompts, making heuristic filtering sensitive for explicit 3D target selection. ClusterSplat instead selects among scene-adaptive clusters using query-conditioned scoring, with cluster scores rendered as a 2D mask. It reports the strongest results among compared 3DGS baselines across ScanRefer and ScanNet v2, with complementary fine-grained validation on Ref-LERF.

  • Problem

    Primitive-level relevance scores vary with scene context and prompt complexity, making thresholding and top-k filtering sensitive for selecting explicit 3D targets.

    Source: §1 · p. 2

  • Method

    ClusterSplat builds scene-adaptive clusters from instance-aware Gaussian features and uses query-conditioned cluster scoring to select the explicit target and render its 2D mask.

    Source: §1 · p. 2, §3.1 · p. 5, §3.3 · p. 6

  • Results

    ClusterSplat achieves state-of-the-art ScanRefer referring segmentation among compared 3DGS baselines, with fine-grained validation on Ref-LERF and strong rendered masks and competitive 3D matching on ScanNet v2.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The results support scene-adaptive cluster selection for referring-expression grounding in 3DGS.

    Source: §6 · p. 10

    Experiments cover ten indoor ScanNet v2 scenes and four Ref-LERF scenes; broader coverage and transfer to unseen scenes remain to be evaluated.

    Source: §5 · p. 10

Abstract

from arXiv · show

We study 3D visual grounding in 3D Gaussian Splatting (3DGS), where a referring expression should identify a target object as both a rendered 2D mask and a set of explicit 3D Gaussians. Prior methods typically extract 3D targets by applying heuristic thresholding or top-ratio filtering to dense primitive-level relevance scores. However, these scores often vary substantially across scenes and prompts, making such heuristics sensitive and leading to unstable targeting. ClusterSplat addresses this by reframing 3D target selection as cluster-level selection over scene-adaptive candidates. The method first learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges neighboring seed voxels when a description-length criterion decreases, producing a scene-specific set of cluster candidates. Given a referring expression, a query-conditioned cluster scorer ranks the cluster candidates with a lightweight MLP, and the selected cluster directly defines the explicit 3D target while its cluster score is rendered for the 2D mask. On ScanRefer, ClusterSplat achieves the best rendered 2D segmentation and explicit 3D target metrics among the compared 3DGS baselines. It also obtains the best average Ref-LERF scores, supporting the same trend under fine-grained referring descriptions. On ScanNet v2 class name queries, ClusterSplat achieves the strongest rendered 2D results and the highest 3D matching accuracy, while maintaining competitive 3D IoU. Our project page is available at https://jaesung8.github.io/clustersplat/ .

1 Introduction

Primitive-level relevance filtering can produce unstable 3D targets because score distributions vary with scene context and prompt complexity. ClusterSplat selects scene-adaptive clusters and evaluates grounding across rendered masks and explicit 3D targets.

1 Introduction

  • Dense primitive-level relevance scores vary across scenes and prompts, making thresholding or top-k filtering sensitive and explicit 3D targeting unstable.Fixed-ratio filtering can scatter selected primitives across an object and nearby distractors.Source: §1 · p. 2
  • ClusterSplat replaces primitive filtering with scene-adaptive cluster selection, avoiding per-query thresholds while letting candidate granularity follow scene structure.It learns instance-aware Gaussian features, forms feature-aware seed voxels, and merges them using a description-length criterion before query-conditioned scoring.Source: §1 · p. 2
  • The selected cluster defines the explicit 3D target, while its cluster-level scores are rendered as the 2D mask.Source: §1 · p. 2
  • The evaluation protocol combines ScanNet v2 annotations with ScanRefer expressions to report class-name grounding and referring-expression segmentation on the same scenes.Source: §1 · p. 3
  • ClusterSplat achieves state-of-the-art referring segmentation on ScanRefer, with consistent fine-grained gains on Ref-LERF and strong rendered masks and competitive 3D matching on ScanNet v2.The evaluation reports class-name grounding and referring-expression segmentation on the same ScanNet v2 scenes.Source: §1 · p. 3

2 Related Work

Prior 3DGS language-grounding methods use semantic features and often discretize continuous relevance scores, while referring tasks require identifying a coherent single 3D target. ClusterSplat groups primitives into scene-adaptive candidates and selects among them for 2D and 3D grounding.

2.1 3D Gaussian Splatting and Language Grounding

  • Semantic feature fields and language-grounding methods extend 3DGS beyond rendering, but target extraction often still relies on thresholding or top-k selection of continuous relevance scores.These rules are sensitive to scene- and prompt-dependent score scales.Source: §2.1 · p. 3
  • Referring expressions require selection of one coherent target, motivating grouped candidates as the decision domain rather than view-wise masks or primitive-level scores.ClusterSplat uses the selected cluster as the explicit 3D target.Source: §2.1 · p. 3

2.2 3D Visual Grounding

  • 3D visual grounding includes comprehension tasks that predict coarse locations and segmentation tasks that predict precise point- or mask-level regions.ScanNet v2 and ScanRefer provide RGB-D scenes and compositional language descriptions used in this area.Source: §2.2 · p. 3
  • In 3DGS, unstructured rendering-optimized Gaussians are not semantic units, so pixel-level parsing can conceal errors in the underlying 3D target set.ClusterSplat’s overview shows instance-aware feature learning, scene-adaptive clustering, query-conditioned scoring, and cluster-level mask rendering.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

3 Method

ClusterSplat selects a referred target from scene-adaptive Gaussian clusters rather than filtering primitive-level relevance scores. It learns instance-aware features, constructs clusters, scores them against the query, and uses cluster scores to render the 2D mask.

3.1 Problem Definition and Target Selection

  • The selected query-specific cluster defines the explicit 3D target, while its cluster-level scores provide renderable evidence for the 2D mask.The formulation selects one target from Ks scene-specific cluster candidates.Source: §3.1 · pp. 4–5

3.2 Instance-Aware Gaussian Feature Learning and Scene-Adaptive Clustering

  • Instance-aware Gaussian features initialize feature-aware seed voxels, which are greedily merged when an MDL-inspired description-length cost decreases.The resulting cluster count adapts to each scene, and assignments remain fixed during Stage 2 training.Source: §3.2 · p. 5

3.3 Query-Conditioned Cluster Scoring

  • A query-conditioned scorer contextualizes cluster tokens, attends to relevant query tokens, and ranks candidates without primitive-level score thresholds.The scorer combines each cluster token with its query summary, their elementwise product, and their absolute difference in a two-layer MLP.Source: §3.3 · p. 6

3.4 Cluster-Level Mask Rendering

  • Each Gaussian inherits its assigned cluster's score, which is rasterized with opacity-aware Gaussian rendering to produce the 2D mask.The explicit 3D prediction remains the selected Gaussian cluster while cluster scores are learned through 2D mask supervision.Source: §3.4 · p. 6

4 Experiments

ClusterSplat improves ScanRefer referring segmentation over compared 3DGS baselines, with gains also reported on Ref-LERF and ScanNet v2 class-name evaluation. The experiments analyze candidate quality, multi-cluster selection, and scorer configuration.

4.1 Experimental Setup

  • The evaluation pairs ScanNet v2 scenes and annotations with ScanRefer expressions, assessing referring segmentation and class-name grounding on the same reconstructed scenes.The ScanNet v2 and ScanRefer evaluation uses the common 10-scene subset adopted by OpenGaussian.Source: §4.1 · p. 7

4.2 Benchmark results

  • On ScanRefer, ClusterSplat improves both rendered 2D segmentation and explicit 3D target selection over the compared 3DGS baselines.The larger improvement in 3D target metrics suggests particular usefulness for queries with attributes or spatial relations.Source: §4.2 · p. 7
  • On ScanNet v2 class-name queries, ClusterSplat substantially improves rendered 2D segmentation and achieves the highest 3D matching accuracy while remaining close to the strongest 3D IoU baselines.Class-name prompts have limited ability to test specific-object targeting when multiple same-category objects appear.Source: §4.2 · pp. 7–8
  • On Ref-LERF, ClusterSplat reaches 31.14% average mIoU and 41.70% mAcc@0.25, versus 26.59% and 36.35% for ReferSplat with our loss.The comparison uses the same human expressions and pseudo-masks.Source: §4.2 · p. 8
  • Qualitative ScanRefer examples show more concentrated target support and fewer distractors for ClusterSplat than for score-discretized baselines.The examples include queries using attributes and spatial relations.Source: Figure 3 · p. 8, §4.2 · p. 9

4.3 Design analysis

  • Scene-adaptive clustering achieves the highest Coverage@1 and Oracle Top-1 IoU despite lower purity than fixed KMeans, whose purity rises as K increases while coverage and oracle IoU fall.Stage 2 can rank candidate clusters but cannot repair their boundaries.Source: §4.3 · p. 9, Table 2 · p. 9
  • For scene-adaptive clusters, ScanRefer 3D mIoU falls from 23.33% at Top-1 to 14.96% at Top-5, and no tested union exceeds Top-1.Top-k predictions are cumulative unions of the k highest-scoring clusters from the same trained model.Source: §4.3 · p. 9, Table 3 · p. 9
  • CLIP achieves 23.33% 3D mIoU versus 14.62% for BERT; caption-source results range from 20.01% for Qwen3-VL-8B to 23.33% for GPT-4.1.Both local caption generators outperform the compared ScanRefer baselines, while gain magnitude varies by caption source.Source: §4.3 · pp. 9–10

5 Discussion and Limitations

ClusterSplat's Top-1 selection depends on the quality of its scene-specific candidate clusters, and the formulation targets single-object expressions. Evaluation and preparation are limited to a small set of scenes, with transfer to unseen scenes not yet evaluated.

5 Discussion and Limitations

  • Over-merging similar nearby objects or splitting one object across clusters can constrain conservative Top-1 selection and leave exact 3D IoU stricter than matching accuracy.Future work may address fragmented candidates with uncertainty-aware multi-cluster selection.Source: §5 · p. 10
  • The formulation handles single-object expressions; grounding a plural expression with a variable number of instances remains future work.Separate queries can select several objects.Source: §5 · p. 10
  • Experiments cover ten indoor ScanNet v2 scenes and four Ref-LERF scenes, so broader scene coverage and transfer to unseen scenes remain unevaluated.Each scene requires offline preparation before queries, and training relies on annotated instance masks and generated descriptions.Source: §5 · p. 10

6 Conclusion

ClusterSplat replaces primitive-level score filtering with discrete selection over scene-adaptive Gaussian clusters, achieving strong results across the reported 3DGS grounding evaluations. The supplementary details describe how those clusters and query-conditioned masks are built and evaluated.

6 Conclusion

  • ClusterSplat achieves state-of-the-art ScanRefer referring segmentation among compared 3DGS baselines, with complementary fine-grained Ref-LERF validation and strong ScanNet v2 masks and competitive 3D object matching.Remaining errors suggest future work on uncertainty-aware multi-cluster selection while preserving explicit target sets.Source: §6 · p. 10

A.3 Scene-Adaptive Clustering and Mask Rendering Details

  • The method learns instance-aware Gaussian features, constructs scene-adaptive clusters with a description-length criterion, ranks them by query-conditioned scores, and renders cluster-level scores for 2D masks.Clustering is performed once per scene before query-conditioned training, without evaluation queries, CLIP text features, object IDs, or a manually fixed cluster count.Source: §6 · p. 10, §A.3 · p. 16

A.1 System Setup and Hyperparameters

  • The reported setup uses frozen CLIP token features projected to dimension 64 and a two-layer cluster scorer; scene cluster counts range from 53 to 143, averaging 87.3 on the ten-scene subset.Training uses Adam at a fixed learning rate of 1 × 10−4 for 10,000 iterations, without cluster-entropy regularization.Source: §A.1 · p. 15

A.2 Instance-Aware Gaussian Feature Learning Details

  • Stage 1 learns instance-aware Gaussian features from multi-view SAM mask supervision, with an objective combining within-mask cohesion and between-mask separation.This feature-learning supervision is separate from the ScanNet v2 masks used for pseudo-caption and evaluation-label construction.Source: §A.2 · pp. 15–16

A.3 Scene-Adaptive Clustering and Mask Rendering Details

  • Feature-aware seed voxels intentionally over-segment scenes before candidate merges, while a whitened feature-position graph limits which neighboring prototypes can be tested.The description-length objective decides whether an allowed pair should merge; the final cluster count varies across scenes.Source: §A.3 · pp. 16–17
  • The fixed Stage 2 cluster assignments give each Gaussian its cluster score, which is rendered as a logit map so the scorer is trained through 2D mask supervision while the explicit 3D output remains the selected cluster.The mask objective combines focal and Dice losses with equal weights.Source: §A.3 · p. 17

A.4 Data Preparation

  • Evaluation uses every 20th frame with an offset of 10 as held-out views; ScanRefer evaluates referred instances, while ScanNet v2 uses class-name queries.For ScanRefer 3D evaluation, the target object ID defines foreground so same-category distractors are not counted as target foreground.Source: §A.4 · p. 17

A.5 Automated Pseudo Label Pipeline

  • Training-frame instance masks anchor GPT-4.1 generation of five diverse pseudo-captions per instance, while held-out evaluation masks and official referring expressions are excluded from caption generation and optimization.The protocol uses annotations for training supervision, so it is not annotation-free, but avoids leakage from evaluation views and official referring expressions.Source: §A.5 · p. 18

A.6 Baseline Implementation Details

The baseline comparison uses reproduced official implementations and an oracle top-ratio sweep to reduce the effect of under-tuned discretization choices.

A.6 Baseline Implementation Details

  • LangSplat, OpenGaussian, Dr.Splat, and ReferSplat are evaluated using their official implementations; ReferSplat receives the same training-frame mask and pseudo-caption supervision as ClusterSplat.OpenGaussian and Dr.Splat continuous relevance scores use the same top-ratio sweep, while Dr.Splat’s Mahalanobis label propagation is used only for evaluation pseudo-ground truth.Source: §A.6 · p. 20
  • For score-based baselines, the evaluation reports each method’s best performance from a per-method top-ratio grid search over 0.01 to 0.10, reducing sensitivity to under-tuned discretization.The protocol is an oracle within the swept range, not a single fixed global ratio.Source: §A.6 · p. 20

B Additional Results

  • The additional-results section covers selection heuristics, model design, cluster candidates, caption sources, lexical overlap, per-scene Ref-LERF results, and qualitative comparisons.Source: B Additional Results · p. 20

B.1 Sensitivity Analysis of Selection Heuristics

Primitive-level top-ratio selection is highly scene-dependent: small changes from the optimum can sharply reduce 3D IoU, motivating cluster-level Top-1 extraction without ratio calibration.

B.1 Sensitivity Analysis of Selection Heuristics

  • Optimal top-ratios vary substantially by scene, and small deviations from the peak can sharply reduce 3D IoU for primitive-level targeting.Source: §B.1 · p. 20
  • Dr.Splat peaks at ratio 0.04 in scene0070, whereas scene0645 shows a decreasing trend across the sweep.Source: §B.1 · p. 20
  • ClusterSplat’s Top-1 cluster extraction avoids per-query and per-scene score-threshold calibration during 3D target extraction.Source: §B.1 · p. 20

B.2 Design Choices

The design analyses examine encoder choice, candidate clusters, scorer modules, and prototype dimension, alongside downstream selection protocols.

B.2 Design Choices

  • In the paired encoder comparison, CLIP scores higher than BERT across all reported metrics, supporting CLIP for this scorer; the preferred encoder should be evaluated within each grounding architecture.Source: §B.2 · p. 21, Table 4 · p. 21
  • Scene-adaptive candidates are fixed before Stage 2, allowing the query-conditioned model to rank renderable clusters and select a Top-1 target.Table 2 evaluates the candidate pool before query-conditioned training.Source: §B.2 · p. 21
  • The cluster scorer’s full combination of position encoding, cluster self-attention, cross-attention, and token scoring gives the best reported ScanRefer 2D accuracy, 3D mIoU, 3D mAcc@0.25, and Hit@1.Cross-attention alone does not improve the 3D metrics, and the token-scorer baseline already gives strong 3D mIoU.Source: §B.2 · p. 21, Table 5 · p. 22
  • The 64-dimensional prototype setting provides the best 3D target selection while retaining competitive 2D segmentation; 128 dimensions gives the highest rendered 2D scores, but lower explicit 3D target metrics.At 256 dimensions, 3D mIoU, mAcc@0.25, mAcc, and Hit@1 fall further.Source: §B.2 · p. 22, Table 6 · p. 22

B.3 Cluster Evaluation Protocols

  • Candidate-pool evaluation distinguishes purity, Coverage@1, Oracle Top-1 IoU, and Frag@0.9, measuring cluster object dominance, single-cluster object coverage, best possible single-cluster IoU, and clusters needed to cover 90% of an object.Source: §B.3 · p. 22
  • Cumulative Top-k predictions are unions of the query-ranked clusters, not the kth cluster alone; Table 3 reports mean ScanRefer 3D mIoU over ten scenes using the same trained model for each construction.Source: §B.3 · p. 22

B.4 Caption Source and Evaluation-Language Analysis

Caption-source comparisons show that ClusterSplat’s gains persist with local caption generation, although scores vary by captioner. A separate analysis finds evaluation expressions include wording beyond their scene’s training captions.

B.4 Caption Source and Evaluation-Language Analysis

  • InternVL3.5-8B and Qwen3-VL-8B score lower than GPT-4.1, yet both outperform the compared ScanRefer baselines.The comparison uses the same scene features, cluster candidates, CLIP encoder, and training settings.Source: §B.4 · pp. 22–23, Table 7 · p. 23
  • The median nearest-caption token-set Jaccard similarity is 0.391 after normalization, lowercasing, and punctuation removal, without stopword removal or stemming.Each evaluation expression is compared with all training captions from the same scene.Source: §B.4 · p. 23
  • 286 of 958 ScanRefer expressions (29.85%) contain a content word absent from their scene’s training-caption vocabulary.The lexical analysis does not measure grounding accuracy separately for this subset.Source: §B.4 · p. 23

B.5 Ref-LERF Quantitative Results

On Ref-LERF’s four small-scale scenes and fine-grained referring expressions, ClusterSplat achieves the highest average mIoU and mAcc@0.25.

B.5 Ref-LERF Quantitative Results

  • ClusterSplat achieves the highest average mIoU and mAcc@0.25 across the evaluated Ref-LERF scenes.Table 8 reports quantitative results in percentages and summarizes the per-scene results.Source: §B.5 · p. 23, Table 8 · p. 23
  • Ref-LERF complements room-scale ScanRefer evaluation with four small-scale scenes and fine-grained referring expressions.The reported gains on scenes such as teatime suggest more stable object support under detailed descriptions.Source: §B.5 · p. 23
  • Table 8 marks the best and second-best distinct scores in each column, with ties receiving the same marking.The table caption states that ClusterSplat improves average referring segmentation scores across the evaluated scenes.Source: Table 8 · p. 23

B.6 Efficiency Analysis

ClusterSplat trades higher one-time per-scene preparation cost for lower repeated query latency, with timing reported on an NVIDIA RTX A6000.

B.6 Efficiency Analysis

  • 19.55 ms per query is about 6.1 times faster than ReferSplat, while ClusterSplat achieves higher grounding accuracy.Online targeting includes text encoding, cluster scoring, and target Gaussian selection; 2D rendering is timed separately.Source: §B.6 · p. 24, Table 9 · p. 24
  • 125.3 min of method-specific preparation per scene is 10.2% more than ReferSplat.Preparation is incurred once per scene, while queries are repeated during interaction.Source: §B.6 · p. 24, Table 9 · p. 24
  • Offline costs average ten scenes and exclude shared RGB reconstruction and cache preparation.ClusterSplat’s preparation total sums separately timed instance-feature learning, cluster construction, and Stage 2 training with different scene representations.Source: §B.6 · p. 24, Table 9 · p. 24

B.7 Additional Qualitative Results

This section presents additional qualitative results across fine-grained Ref-LERF examples, ScanRefer scenes, and ScanNet v2 class visualizations, alongside checklist information on methodological limitations and reproducibility. It also documents the use of GPT-4.1 to generate pseudo-captions from training-frame instance masks and the stated exclusions from that process.

B.7 Additional Qualitative Results

  • On fine-grained Ref-LERF descriptions, ClusterSplat produces more concentrated masks than baselines that merge targets with background, omit parts, or respond to distractors.The examples include appearance attributes, surface cues, and spatial relations.Source: §B.7 · p. 25, Figure 6 · p. 25
  • Across all 10 ScanRefer evaluation scenes, ClusterSplat generally concentrates rendered masks and explicit top-down 3D target selections on referred objects while reducing distractor responses.The visualizations include attribute-based descriptions and spatial relations.Source: §B.7 · p. 25, Figure 7 · p. 26, Figure 8 · p. 27
  • The ScanNet v2 visualization covers 19 classes across all 10 evaluation scenes, showing label construction and class-level scene predictions from 3DGS baselines and ClusterSplat.Annotated point-cloud labels and Gaussian pseudo labels are shown alongside the method comparisons.Source: §B.7 · p. 25, Figure 9 · p. 28

NeurIPS Paper Checklist

  • The checklist identifies limitations in cluster merging and fragmentation, single-target grounding, limited scene coverage, reliance on annotated supervision, and per-scene preparation costs.The paper reports these limitations in Section 5.Source: NeurIPS Paper Checklist · p. 29
  • The checklist says reproduction details cover architecture, data preparation, splits, baselines, hyperparameters, and evaluation, with an anonymized supplementary code package and instructions planned.It also identifies the RTX 5090 and RTX A6000 hardware used and points to reported preparation time, query latency, and rendering throughput.Source: NeurIPS Paper Checklist · pp. 30–32
  • GPT-4.1 is used as a VLM to generate pseudo-captions from training-frame instance masks.The appendix describes the model inputs used for generation.Source: NeurIPS Paper Checklist · p. 35
  • Official ScanRefer expressions, evaluation masks, and evaluation target IDs are not used to generate pseudo-captions.Source: NeurIPS Paper Checklist · p. 35

Related papers

Loading 2610.36462v1…