PaperlayerView exact revision on arXiv

Source-linked AI summary

DSSR-3D: Decoupled Reasoning for View-Dependent Referring in 3D Gaussians

Thanh-Khoi Nguyen, Thien-Phuc Tran, Minh-Triet Tran

arXiv:2610.00040v1cs.CV

TL;DR

Existing 3D referring fields encode language features in globally view-invariant spaces, limiting observer-centric spatial grounding. DSSR-3D separates semantic localization from pose-conditioned spatial reasoning on continuous 3D Gaussian fields, using training-free inference-time mechanisms. The paper reports consistent gains across structurally distinct frozen backbones and evaluates viewpoint-dependent grounding on six scenes and eight directional relations.

  • Problem

    Existing 3D referring methods use globally view-invariant semantic spaces, making observer-centric relations that depend on camera pose difficult to resolve.

    Source: §1 · pp. 1–2

  • Method

    DSSR-3D decouples pose-invariant semantic localization from pose-conditioned spatial reasoning through training-free interfaces on continuous 3D Gaussian fields.

    Source: §6 · p. 13

  • Results

    DSSR-3D achieves consistent gains across structurally distinct frozen backbones and is validated across six scenes and eight directional relations.

    Source: §1 · p. 3, §6 · p. 13

  • Takeaways & Limitations

    The framework composes with structurally distinct frozen semantic fields without adaptation or backbone-specific learning.

    Source: §6 · p. 13

    The continuous, bounding-box-free formulation trades exact boundary precision for possible directional bias on elongated or topologically complex objects, and is scoped to single-anchor queries.

    Source: §6 · p. 13

Abstract

from arXiv · show

Recent advances in 3D Gaussian Splatting have enabled open-vocabulary and referring segmentation by distilling semantic knowledge from 2D foundation models into 3D representations. However, existing referring fields embed language features in a globally view-invariant space, making them fundamentally unable to resolve observer-centric spatial relations (e.g., "to the left of") that depend on camera pose. We propose DSSR-3D, an inference-time framework for view-dependent referring segmentation on continuous 3D Gaussian fields, formalized as two interfaces - pose-invariant semantic localization and pose-conditioned spatial reasoning - such that any pair of functions satisfying these constraints yields a valid instantiation, requiring no retraining of the underlying semantic field and no reliance on discrete geometric proxies such as bounding boxes. We instantiate the two interfaces with a temperature-sharpened softmax localization mechanism and a projection-based directional scoring function, fused via a lightweight, training-free step, and show they transfer zero-shot to structurally distinct semantic fields without adaptation. We further propose ViewRef-GS, a benchmark isolating view-dependent segmentation on 3D Gaussian fields, evaluated jointly with an augmented Ref-LERF to provide a comprehensive testbed for viewpoint-dependent spatial grounding. Experiments show consistent gains over existing 3DGS-based referring methods, with no additional training beyond the base semantic field

1 Introduction

DSSR-3D targets observer-centric spatial queries that globally view-invariant semantic fields struggle to resolve. It separates semantic localization from pose-conditioned reasoning and introduces inference-time mechanisms and a benchmark for evaluating viewpoint-dependent grounding.

1 Introduction

  • Existing 3D referring fields encode semantics in globally view-invariant spaces, while isolated-frame 2D models lack depth and 3D structure for occluded targets.Figure 1 illustrates failures under viewpoint ambiguity and occlusion; static scene-graph relations also remain view-invariant.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Earlier viewpoint-aware approaches use point clouds, additional viewpoint training, or discrete bounding boxes, leaving a gap for inference-time reasoning on continuous 3DGS.The introduction identifies annotation cost and quantization errors as concerns with box-based reasoning.Source: §1 · p. 2
  • Temperature-Sharpened Softmax provides scale-adaptive, bounding-box-free localization, while Direction-Aware Scoring uses projection and spatial decay to resolve pose-conditioned relations.Both mechanisms operate at inference time without task-specific retraining.Source: §1 · pp. 2–3
  • DSSR-3D decouples view-dependent spatial grounding into pose-invariant localization and pose-conditioned relational scoring, forming a training-free plug-and-play layer.The decomposition is designed to work across structurally distinct, frozen 3DGS backbones without per-backbone tuning.Source: §1 · p. 3
  • ViewRef-GS and an augmented Ref-LERF form a testbed for viewpoint-dependent spatial grounding across diverse observer-centric configurations.The introduction describes ViewRef-GS as a synthetic benchmark.Source: §1 · p. 3

2 Related Work

Prior 3D semantic fields and referring methods provide open-vocabulary querying and relational modeling, but encode semantics in globally view-invariant spaces. Viewpoint-aware methods address observer-centric relations, yet DSSR-3D instead targets continuous 3D Gaussian fields through training-free interfaces and introduces a dedicated benchmark.

2.1 3D Semantic and Referring Segmentation

  • 3D semantic fields distill CLIP/SAM features into Gaussians, and ReferSplat formalized referring segmentation on 3DGS with Ref-LERF.Later work adds graph alignment, relational embeddings, or cross-view logit alignment, while retaining globally view-invariant semantic spaces.Source: §2.1 · p. 3

2.2 Viewpoint-Aware Spatial Reasoning

  • Viewpoint-aware work includes predicted-speaker-view grounding and a benchmark for observer-centric relations; DSSR-3D instead uses training-free pose-invariant localization and pose-conditioned scoring on continuous 3DGS.Other approaches cited operate on point clouds or meshes, use boxes, require viewpoint-specific training, or treat relations as view-invariant; ViewRef-GS isolates view-dependent referring segmentation on 3DGS.Source: §2.2 · pp. 3–4

3 Method

DSSR-3D separates frozen-field semantic localization from camera-conditioned relational scoring, then fuses their outputs into a rendered referring mask. Localization uses adaptive soft weighting, while spatial scoring is pose-aware and locality-aware.

3.1 Preliminary: 3D Gaussian Splatting

  • For referring segmentation, each Gaussian’s color is replaced by a query-alignment score that produces a query-conditioned response map.Source: §3.1 · p. 4

3.2 Problem Formulation

  • The method separates pose-invariant reference localization from pose-conditioned relational scoring, constraining cameras to near-planar views and queries to a single anchor.Either interface pair satisfying the stated constraints can transfer across frozen semantic fields without retraining.Source: §3.2 · pp. 4–5

3.3 Pipeline Overview

  • The inference-time pipeline rewrites a query into target, direction, and anchor, then localizes the anchor, scores the relation under the camera pose, and renders a mask.Camera dependence is isolated in directional scoring.Source: §3.3 · p. 5

3.4 Pose-Invariant Semantic Localization

  • Localization thresholds anchor scores to form a candidate set and temperature-softmax weights their Gaussian means into a reference guaranteed inside the semantic support’s convex hull.The soft aggregation avoids fixed-cardinality selection, which can discard support for large objects or absorb background noise for compact ones.Source: §3.4 · p. 6

3.5 Pose-Conditioned Directional Scoring

  • Directional scoring projects the anchor-relative offset onto the camera’s local horizontal XZ-plane along the parsed direction, then applies distance decay to favor nearby candidates.A tanh transformation bounds the score to [−1, 1] while preserving its geometric sign.Source: §3.5 · pp. 6–7

3.6 Semantic-Spatial Fusion

  • Fusion adds the spatial score as a symmetric residual bias to the semantic score, then splats composite scores into a rendered response map that is binarized.The spatial weight balances geometry against semantic confidence.Source: §3.6 · p. 7

4 Experiment

DSSR-3D is evaluated on augmented Ref-LERF and the six-scene ViewRef-GS suite, with results showing gains over 3DGS baselines and transfer across frozen semantic backbones. The experiments also report qualitative mask behavior and compare inference latency and localization accuracy.

4.1 Experimental Setup

  • ViewRef-GS adds four synthetic scenes to two augmented Ref-LERF scenes, spanning over 19,000 queries and eight directional relations.The synthetic scenes contain over 700 calibrated views across 50 object instances; prompts commonly run 8–12 words.Source: §4.1 · pp. 7–8
  • The evaluation reports per-query IoU and mean IoU, and excludes GroundingSAM from quantitative comparisons on ViewRef-GS because it generated those scenes’ ground-truth masks.The two real-world Ref-LERF scenes use new human-verified directional queries, while two official pre-trained scenes are included without retraining baseline models.Source: §4.1 · pp. 7–8

4.2 Main Results

  • DSSR-3D outperforms 3DGS and 2D baselines on Ramen, exceeds ReferSplat and GS-Grouping on Waldo Kitchen, and beats all 3DGS baselines across four synthetic scenes.The largest absolute performance on ViewRef-GS is observed on Sofa.Source: Table 1 · p. 9, §4.2 · p. 9, Table 2 · p. 9
  • 16.94% to 21.02% average mIoU (+4.08) for LangSplat and 23.60% to 28.43% (+4.83) for GS-Grouping hold across all six scenes without backbone adaptation.Gains occur on every individual scene.Source: Table 3 · p. 10, §4.2 · p. 10
  • DSSR-3D generally produces more spatially consistent masks, better boundary alignment, and fewer missed regions, though occlusion and similar distractors still yield false positives.GS-Grouping can propagate first-view false positives into incorrect Gaussian groupings across later views.Source: §4.2 · p. 10

5 Ablation Study

The ablations compare inference latency, anchor localization, and spatial-decay settings. They report faster repeated querying than GroundingSAM and more consistent localization and decay performance from adaptive choices.

5.1 Time Comparison with Grounding SAM

  • Approximately 0.08s per query versus GroundingSAM’s 0.6–0.7s yields an approximately 8× speedup for repeated queries within a known scene.DSSR-3D amortizes feature extraction offline and performs spatial reasoning in the pre-embedded Gaussian space.Source: §5.1 · pp. 10–11

5.2 Anchor Localization Strategies

  • Temperature-Sharpened Softmax beats the strongest Top-K result by 3.11 mIoU points on Ramen and 4.02 on Waldo Kitchen, while retaining gains on synthetic scenes.It weights candidates by semantic confidence, reducing low-confidence contributions while adapting to object extent; Top-K performance is sensitive to K.Source: §5.2 · pp. 11–12, Table 5 · p. 12

5.3 Robustness of Spatial Decay

  • Adaptive σd outperforms the unpenalized baseline and fixed radii across all six scenes, improving by up to 12.31 points on Sport and at least 3.50 on Ramen.The adaptive radius is the median distance of top semantic candidates from the anchor; the best fixed radius varies by scene.Source: Table 6 · p. 12, §5.3 · p. 12

6 Conclusion

DSSR-3D separates view-invariant semantic localization from view-conditioned spatial reasoning in a training-free framework that transfers across distinct frozen semantic fields. Its instantiation is validated on six scenes and eight directional relations, with boundary precision and single-anchor scope remaining limitations.

6 Conclusion

  • DSSR-3D decouples view-invariant semantic localization from view-conditioned spatial reasoning and generalizes across structurally distinct baselines without adaptation.The framework is instantiated through soft continuous aggregation and algebraic projection.Source: §6 · p. 13
  • Validation covers six scenes and eight directional relations on Ref-LERF and the new ViewRef-GS benchmark.Source: §6 · p. 13
  • The continuous, bounding-box-free formulation trades exact boundary precision for possible directional bias on elongated or topologically complex objects.The pipeline is scoped to single-anchor queries; future work targets multi-anchor relationships and larger, more diverse benchmarks.Source: §6 · p. 13
Loading 2610.00040v1…