R4DGS: Referring Segmentation in 4D Gaussian Splatting

1 Introduction

• R4DGS requires dynamic Gaussian systems to produce temporally grounded, set-valued masks for complex referring expressions.Its query categories include temporal variation, multi-target or reasoning queries, and zero-target or distractor queries.Source: §1 · p. 2, Figure 2 · p. 2
• Existing Gaussian methods emphasize rendering or semantic outputs, leaving temporal, relational, multi-target, and absent-referent queries unsupported within a unified referring protocol.Such queries require temporal reasoning, entity disambiguation, set-level constraints, and uncertainty-aware rejection.Source: §1 · p. 2
• ReferGaussian forms persistent Gaussian entities from mask-supported multiview evidence and resolves structured referring constraints over an EntityBank without scene-specific semantic training.MLLMs decompose queries while Grounded-SAM supplies 2D mask evidence.Source: §1 · p. 2
• R4D-Bench-QA introduces structured query annotations and pixel-level spatiotemporal supervision for evaluating R4DGS.Its construction converts source videos into trackable entities and verified entity-centric query annotations.Source: §1 · p. 2, Figure 3 · p. 3

2 Related Work

• Dynamic Gaussian methods advance view synthesis, but complex language grounding over temporal states and set-level entity constraints remains distinct from their rendering objective.Existing dynamic semantic methods enable time-sensitive querying but primarily focus on semantic segmentation, with limited coverage of complex referring.Source: §2 · p. 3
• Figure 4 summarizes R4D-Bench-QA vocabulary, query-category composition, and expression length across two scene sources.The benchmark includes both concise references and longer expressions involving temporal or set-level constraints.Source: Figure 4 · p. 3
• ReferGaussian uses vision-language and segmentation foundation models as modular components for query parsing and visual evidence extraction.This decomposition allows query-decomposition, mask-evidence, and entity-association errors to be analyzed separately.Source: §2 · p. 3

3.1 Referring 4D Gaussian Splatting (R4DGS)

• R4DGS returns frame-aligned masks for a single entity, multiple entities, or an empty set according to spatiotemporal, relational, and set-level query constraints.Its queries include state changes, conjunction, exclusion, quantification, relations, and cases with no valid referent.Source: §3.1 · p. 3

3.2 R4D-Bench-QA

• R4D-Bench-QA contains 89 densely annotated sentence-level queries across 12 scenes from HyperNeRF and Neu3D, covering dynamic referring and temporal and spatial grounding.The benchmark includes six HyperNeRF scenes with 47 queries and six Neu3D scenes with 42 queries.Source: §3.2 · pp. 3–4
• Benchmark annotations specify evaluated frames, target entities, and frame-aligned masks, with multi-target masks formed by union and zero-target masks empty throughout.Annotations associate expressions with complete target sets and their valid temporal support.Source: §3.2 · p. 4

3.3 Evaluation Metrics

• Acc measures frame-wise temporal activation, while vIoU combines temporal localization and mask overlap; zero-target queries require empty predictions for a perfect vIoU.The protocol also reports category-specific Acc and vIoU across both source datasets.Source: §3.3 · pp. 4–5

4 Method

• ReferGaussian addresses the gap between dynamic Gaussian primitives and language-referred entities by constructing persistent entities and selecting referents without scene-specific semantic optimization.It begins from a precomputed 4D Gaussian scene and uses multiframe visual evidence.Source: §4 · p. 5

4.1 Preliminaries

• 4D Gaussian Splatting conditions Gaussian primitives on time, while LangSplat renders per-Gaussian language features using the color-rendering weights.A common dynamic formulation predicts position, rotation, and scale offsets with a temporal deformation function.Source: §4.1 · p. 5

4.2 Framework

• ReferGaussian uses mask tracks to lift phrase-conditioned 2D evidence into Gaussian-supported candidates, stores them in an EntityBank, and selects entities under structured constraints.The three stages produce frame-aligned masks and shift reasoning from pixel- or frame-wise matching to persistent Gaussian entities and globally consistent set-valued answers.Source: §4.2 · p. 6

4.3 Mask-Supported Persistent Entity Construction

• Mask-supported entity construction iteratively ranks Gaussian support and validates candidates by rendering masks back to sampled frames.The procedure runs at most ten rounds, stops at mean render-back IoU of 0.70 or after three rounds without improvement, and retains the best validated candidate; entities preserve track identity and temporal support.Source: §4.3 · p. 6

4.4 EntityBank Construction and Constraint-Aware Querying

• The Refer-Planner converts queries into validated structured constraints, then closed-set selection checks identity, time, relations, cardinality, exclusion, and empty-target conditions over EntityBank candidates.Selected Gaussian supports are rasterized only in chosen temporal segments, and masks from multiple entities are combined frame by frame; cached records and visual evidence can be reused across queries.Source: §4.4 · pp. 6–7

5.1 Experimental Setup

• All methods use the same scene inputs, query annotations, and protocols; ReferGaussian uses Qwen3-VL-8B and Grounded SAM 2 without additional training or scene-specific semantic optimization.The planner analysis also evaluates Qwen3-VL-4B, Qwen2.5-VL-7B, Qwen2-VL-7B, and InternVL3-8B.Source: §5.1 · p. 7

5.2 Main Results

• ReferGaussian reaches 76.5% Acc and 34.4% vIoU on R4D-Bench-QA, surpassing both baselines, and 93.01% average Acc and 62.95% vIoU on the four-scene protocol, with Acc 2.18 points above 4D LangSplat.The evaluation covers temporal, multi-target, and distractor queries; the 4D LangSplat comparison is reported without per-scene semantic fitting.Source: §5.2 · p. 7

5.3 Qualitative Results

• Qualitative examples show coherent transition timing and clean exclusion, including tracking an active liquid-filling interval while excluding the hand applying force.Figure 6 compares RGB input, ground truth, ReferGaussian, Segment then Splat, and 4D LangSplat.Source: §5.3 · pp. 7–8

5.4 Ablation and Analysis

• Qwen3-VL-8B gives the strongest overall planner profile, with stable trends across HyperNeRF and Neu3D for the three query categories.Figure 7 reports Acc and vIoU separately across six scene-source/category axes in each metric panel.Source: §5.4 · p. 8, Figure 7 · p. 8
• Iterative lifting raises vIoU from 12.4 to 34.4 over one-pass voting, while mask erosion, dilation, and proposal dropping retain 32.1–33.9 vIoU.Merge and split errors yield 19.4 and 22.7 vIoU, respectively, and can alter entity identity and propagate to later constraints; Table 4 reports vIoU for lifting variants and controlled mask perturbations.Source: §5.4 · p. 8, Table 4 · p. 8
• EntityBank reuse cuts runtime from about 600 to 200 seconds and local MLLM calls from six to three.The initial query is dominated by mask detection and entity construction; cached queries reuse tracks, visual evidence, and entity descriptions. Figure 8 segments the costs by pipeline component.Source: §5.4 · p. 8, Figure 8 · p. 8
• Remaining failures involve crowded contact events, ambiguous long queries, breakage, and severe occlusion, which combine rapid topology changes with uncertain identity boundaries.The analysis identifies stronger temporal association and calibrated uncertainty as motivations for addressing these cases.Source: §5.4 · p. 8

• R4DGS defines grounding complex natural-language expressions in dynamic Gaussian scenes, and R4D-Bench-QA evaluates temporal-state, multi-target, and zero-target referring.Source: §6 · p. 8
• ReferGaussian organizes dynamic Gaussian primitives into persistent, language-addressable entities and selects referents through EntityBank without scene-specific semantic optimization.Source: §6 · p. 8
• Experiments on R4D-Bench-QA and the 4D LangSplat protocol demonstrate effective entity-level reasoning for complex spatiotemporal grounding.The conclusion describes the combined task, benchmark, and framework as a foundation for language-guided interaction and structured reasoning over dynamic 4D scene representations.Source: §6 · p. 8