PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussianDS: Depth-supervised Semantic Gaussian Splatting for Scene Understanding

Yufei Zhang, Chenlu Zhan, Hongwei Wang

arXiv:2609.27850v1cs.CV

TL;DR

View-dependent 2D foundation-model predictions can conflict across views and provide weak geometric grounding for semantic lifting into 3D. GaussianDS jointly learns appearance, depth, and compact semantics using pose-aware mask propagation and depth-based constraints. It reports strong open-vocabulary and boundary-localization results on LERF and 3D-OVS while maintaining real-time rendering.

  • Problem

    View-dependent 2D semantic predictions can conflict across views, producing supervision misalignment and weak geometric grounding for 3D semantic boundaries.

    Source: §I · p. 1, B. Consistent Semantic Learning and… · p. 4

  • Method

    GaussianDS jointly optimizes appearance, rendered depth, and compact semantics, using pose-aware pseudo-video mask propagation and depth-edge constraints.

    Source: §I · p. 2

  • Results

    GaussianDS reports 60.5% mIoU on LERF and 97.79% mIoU with 90.28% mBIoU on 3D-OVS while maintaining real-time rendering.

    Source: §I · p. 2

  • Takeaways & Limitations

    The framework provides a geometry-grounded semantic Gaussian field for open-vocabulary 3D scene parsing and downstream editing.

    Source: §VI · p. 10

Abstract

from arXiv · show

3D Gaussian Splatting provides an efficient representation for 3D reconstruction, and recent extensions attach semantic attributes to Gaussians for open-vocabulary scene understanding. However, lifting view-dependent 2D foundation-model outputs into 3D space introduces cross-view inconsistencies and weak geometric grounding, leading to severe semantic drift and boundary leakage. We propose GaussianDS, a depth-supervised semantic 3DGS framework that treats semantic lifting as a supervision-alignment problem and jointly optimizes RGB appearance, rendered depth, and compact semantics from scratch. Specifically, GaussianDS organizes unordered multi-view images into a pose-aware pseudo-video trajectory to propagate view-consistent masks via SAM2. During joint optimization, scale-shift-aligned monocular depth supervision and depth total-variation regularization stabilize Gaussian geometry, while a depth-edge-aware refinement loss explicitly anchors semantic transitions onto physical geometric discontinuities. Extensive evaluations show that our end-to-end framework not only retains high-fidelity 3D reconstruction and real-time rendering, but also establishes superior semantic understanding. GaussianDS sets new state-of-the-art performance on LERF (60.5% mIoU) and 3D-OVS (97.79% mIoU, 90.28% mBIoU) by mitigating semantic leakage, while seamlessly facilitating downstream 3D object removal.

I. INTRODUCTION

Lifting view-dependent 2D semantic predictions into 3D can create conflicting supervision and semantic boundaries that drift across physical surfaces. GaussianDS jointly optimizes geometry and semantics with pose-aware masks and depth-based constraints, and reports strong benchmark results while retaining real-time rendering.

I. INTRODUCTION

  • Unaligned, view-dependent 2D predictions can send conflicting gradients to shared Gaussians, causing semantic drift and boundary leakage.The problem is especially consequential for discrete, boundary-sensitive semantic supervision.Source: §I · p. 1
  • Scale-shift-aligned monocular depth and depth-edge-aware refinement regularize Gaussian geometry and anchor semantic boundaries to physical discontinuities.The approach uses relative depth structure rather than requiring absolute metric depth.Source: §I · pp. 1–2
  • GaussianDS jointly optimizes RGB appearance, rendered depth, and compact latent semantics from scratch rather than freezing geometry before semantic distillation.This replaces cascaded two-stage pre-training with joint optimization.Source: Figure 1 · p. 1, §I · p. 2
  • Pose-aware pseudo-video ordering lets SAM2 propagate spatially coherent masklets across overlapping viewpoints, suppressing inter-view mask jitter.The sequence is built from unordered multi-view captures.Source: §I · p. 2
  • The paper frames unaligned 2D foundation-model supervision as a supervision-alignment bottleneck in semantic 3DGS.The stated issue is conflicting gradients on shared Gaussian primitives.Source: §I · pp. 1–2
  • 60.5% mIoU on LERF and 97.79% mIoU with 90.28% mBIoU on 3D-OVS, with real-time rendering.The reported evaluations cover open-vocabulary query accuracy and boundary localization.Source: §I · p. 2

II. RELATED WORK

Prior work addresses Gaussian geometry, 2D semantic supervision, and open-vocabulary 3D fields through distinct strategies. GaussianDS combines geometric depth cues with pose-aware mask propagation and joint geometry-semantic optimization.

A. 3D Gaussian Splatting and Geometric Regularization

  • GaussianDS routes scale-shift-aligned depth into semantic optimization, using physical depth discontinuities to regularize 3D semantic boundaries.The cited geometric methods primarily use depth and geometric priors in the appearance reconstruction pathway.Source: A. 3D Gaussian Splatting and Geometric… · p. 2

B. 2D Segmentation and Semantic Lifting

  • Ordering unordered viewpoints by spatial proximity lets SAM2 track masks across views before semantic features are lifted into 3D.Direct use of SAM2 on unordered multi-view collections is described as ill-posed.Source: B. 2D Segmentation and Semantic Lifting · p. 2

C. Open-Vocabulary 3D Gaussian Understanding

  • Unlike prevailing approaches that freeze Gaussian geometry before semantic distillation, GaussianDS jointly optimizes appearance, geometry, and semantics from scratch.It anchors semantic transitions onto physical geometric depth edges.Source: C. Open-Vocabulary 3D Gaussian… · p. 2

III. PRELIMINARY: 3D GAUSSIAN SPLATTING

3DGS represents scenes with anisotropic Gaussian primitives and renders them through differentiable rasterization. GaussianDS uses posed images to jointly optimize color, depth, geometry, and compact semantic features, with pose-aware mask propagation and depth-edge constraints.

III. PRELIMINARY: 3D GAUSSIAN SPLATTING

  • 3DGS represents a continuous scene with anisotropic Gaussians defined by centers, covariance matrices, opacity, and spherical-harmonic appearance coefficients.The covariance is factorized into rotation and scaling components to maintain positive semi-definiteness.Source: §III · p. 3
  • Differentiable rasterization projects and depth-sorts Gaussians, then uses visibility-ordered alpha-compositing to compute rendered color.The appearance reconstruction loss is introduced after this rendering formulation.Source: §III · p. 3

IV. METHODOLOGY

  • GaussianDS learns an open-vocabulary semantic Gaussian field from posed images and 2D foundation-model supervision without dense 3D annotations.It jointly optimizes spatial geometry, appearance, and compact latent semantics from scratch.Source: §IV · p. 3
  • Pose-aware pseudo-video mask propagation and joint geometric-semantic optimization address cross-view inconsistency and ambiguous 2D boundaries.The method orders camera poses by spatial proximity for SAM2 propagation and uses depth regularization and depth-edge constraints to anchor semantic boundaries to scene geometry.Source: §IV · p. 3

A. Depth Rendering and Geometric Regularization

Rendered camera-space depth supplies geometric supervision alongside appearance reconstruction, while monocular estimates provide relative depth guidance. Scale-shift alignment and rendered-depth total variation constrain geometry without smoothing away genuine discontinuities.

A. Depth Rendering and Geometric Regularization

  • Vanilla 3DGS lacks direct pixel-level geometric constraints, so Gaussian locations can depart from surfaces, especially in sparse or textureless regions.Camera-space z-depth is introduced as an auxiliary supervisory signal alongside appearance reconstruction.Source: A. Depth Rendering and Geometric… · pp. 3–4
  • Depth Anything supplies estimated relative depth for each training view, while GaussianDS renders camera-space depth by compositing depth attributes along each viewing ray.Rendered depth accumulates the depth attributes of contributing Gaussians.Source: A. Depth Rendering and Geometric… · p. 4
  • Because monocular depth is scale-ambiguous and shift-variant, GaussianDS optimizes view-specific scale and shift before computing the aligned depth loss.The alignment parameters are solved per iteration using a closed-form least-squares solution across valid depth pixels, without gradient backpropagation.Source: A. Depth Rendering and Geometric… · p. 4
  • Rendered-depth total variation penalizes high-frequency depth noise while preserving true geometric discontinuities.It encourages piecewise smoothness on locally under-constrained surfaces and works with aligned depth to support semantic-field learning.Source: A. Depth Rendering and Geometric… · p. 4

B. Consistent Semantic Learning and Depth-Edge-Aware Refinement

GaussianDS addresses cross-view semantic conflicts and boundary leakage with coherent mask propagation and geometry-aware semantic refinement. It uses compact semantic targets and depth edges to align 3D semantic boundaries with physical discontinuities.

B. Consistent Semantic Learning and Depth-Edge-Aware Refinement

  • Pose-aware ordering lets SAM2 propagate instance masks across overlapping views, preserving object identities before 2D semantics are lifted into 3D.Camera viewpoints are sorted by spatial proximity to form a pseudo-video trajectory.Source: B. Consistent Semantic Learning and… · p. 4
  • The semantic targets are compact latent maps derived from masked CLIP features, while each Gaussian carries a learnable semantic vector rendered into pixel features.The passages describe the target map and per-Gaussian attributes used in semantic rendering.Source: B. Consistent Semantic Learning and… · pp. 4–5
  • A depth-edge-aware loss weights semantic discrepancies at physical depth discontinuities, guiding 3D semantic boundaries to align with scene geometry.The binary edge mask is obtained by applying Canny to the scale-shift-aligned monocular depth map.Source: B. Consistent Semantic Learning and… · p. 5

C. Joint Optimization Objective

GaussianDS combines its appearance, depth, and semantic supervision in a single-stage joint optimization objective. Balancing hyperparameters govern the geometric regularization and semantic alignment terms.

C. Joint Optimization Objective

  • The objective unifies supervision in a single-stage joint optimization from scratch rather than separating semantic distillation from geometric reconstruction.It is a weighted combination of appearance, depth, and semantic losses.Source: C. Joint Optimization Objective · p. 5
  • The balancing hyperparameters govern geometric regularization and semantic alignment.Source: C. Joint Optimization Objective · p. 5

D. Open-Vocabulary Query and Downstream Editing

GaussianDS supports text-query semantic segmentation and 3D object removal through its rendered semantic features. The evaluation covers open-vocabulary segmentation, reconstruction, and object-removal scenarios across the stated datasets.

D. Open-Vocabulary Query and Downstream Editing

  • Text queries are encoded into the latent semantic space, then compared by cosine similarity with Gaussian attributes or rendered pixel features.Source: D. Open-Vocabulary Query and… · p. 5
  • Rendered semantic features are compared pixel-wise with the query embedding, and thresholding produces a binary 2D novel-view segmentation mask.Source: D. Open-Vocabulary Query and… · p. 5
  • Target-object Gaussians are identified by semantic similarity to query embeddings or class prototypes and removed by pruning those primitives.The remaining Gaussians retain their appearance and structural properties.Source: D. Open-Vocabulary Query and… · p. 5

A. Experimental Setup

  • Evaluation uses LERF, 3D-OVS, Tanks and Temples, and Deep Blending, with reconstruction, semantic segmentation, boundary localization, and object-removal measures.Reconstruction is assessed with PSNR, SSIM, and LPIPS; segmentation with mIoU; and 3D-OVS boundaries with mBIoU.Source: A. Experimental Setup · p. 5
  • Implementation uses SAM2 mask propagation, a frozen autoencoder compressing 512-dimensional CLIP descriptors to d = 3, and joint optimization for 30K iterations on one RTX 4090.Source: A. Experimental Setup · p. 5

B. Multi-view 3D Reconstruction

GaussianDS pairs depth-constrained reconstruction with compact semantics, reporting stronger reconstruction than vanilla 3DGS and Gaussian Grouping across most evaluated scenes and the highest overall LERF segmentation accuracy. Its reconstruction remains robust to standard depth-noise perturbations, with motion blur the most damaging.

B. Multi-view 3D Reconstruction

  • Reconstruction metrics remain robust to Gaussian, Poisson, and salt-and-pepper depth noise, while motion blur causes the most noticeable degradation.Motion blur smooths depth gradients and weakens Canny-detected geometric edges used by the boundary refinement loss.Source: B. Multi-view 3D Reconstruction · p. 6
  • GaussianDS outperforms vanilla 3DGS and Gaussian Grouping on most evaluated reconstruction scenes, with gains especially pronounced on 3D-OVS.Compact d = 3 semantics add negligible per-primitive memory overhead, and average real-time rendering speed remains comparable to vanilla 3DGS; high-dimensional identity embeddings compromise reconstruction quality and frame rates.Source: B. Multi-view 3D Reconstruction · pp. 5–6, Table I · p. 7

C. Open-Vocabulary 3D Semantic Segmentation

D. Boundary-Aware Semantic Segmentation

GaussianDS reports strong region- and boundary-level segmentation on 3D-OVS, while ablations examine mask ordering, depth alignment, loss components, semantic capacity, and loss-weight sensitivity.

D. Boundary-Aware Semantic Segmentation

  • Boundary-aware evaluation pairs region-level mIoU with boundary-level mBIoU because mIoU is less sensitive to semantic bleeding along object margins.Qualitative comparisons report fragmented or dilated baseline masks, while GaussianDS produces sharp boundaries aligned with physical extents.Source: D. Boundary-Aware Semantic Segmentation · p. 7, Figure 7 · p. 8
  • 97.79% overall mIoU and 90.28% overall mBIoU on 3D-OVS exceed LangSplatV2 by 3.19% and 1.88%, respectively.GaussianDS has the highest mBIoU on every evaluated scene; Table III summarizes results across the five scenes.Source: D. Boundary-Aware Semantic Segmentation · p. 7, Table III · p. 8

E. Ablation Studies

  • Pose-nearest frame ordering reduces adjacent-camera distance from 5.00 to 1.50 and raises adjacent-mask IoU from 33.92% to 53.21% versus random ordering.The comparison covers five 3D-OVS scenes.Source: Table IV · p. 8, E. Ablation Studies · p. 8
  • On bench, scale-shift alignment reaches 24.81 dB PSNR and 91.99% mBIoU; metrics of 21.31 dB PSNR and 59.69% mBIoU are attributed to scale discrepancies.Source: Table V · p. 8, E. Ablation Studies · p. 8
  • On bench, depth supervision adds +6.44 dB PSNR, while depth-edge refinement adds +37.30% mIoU; combining all three losses gives the strongest joint results.The losses are scale-shift depth, rendered-depth total variation, and depth-edge semantic refinement.Source: E. Ablation Studies · p. 8, Table VI · p. 9
  • The loss-weight sweep reports a negative correlation between λdepth and semantic mIoU (r = −0.89), while λsem correlates positively (r = +0.69).The reported trade-off is best at λdepth ∈[0.01, 0.02] and λsem ∈[4.0, 5.0]; Fig. 8 visualizes the sensitivity sweep.Source: Figure 8 · p. 9, E. Ablation Studies · p. 9
  • On waldo kitchen, d = 3 retains 97.7% of peak mIoU, while d = 32 reaches 61.32% mIoU and exceeds 210 minutes of training.The d = 3 configuration requires 39 minutes and 4.2 GB VRAM.Source: E. Ablation Studies · pp. 8–9, Table VII · p. 9

F. 3D Object Removal

GaussianDS supports zero-shot 3D object removal by pruning query-matched Gaussian primitives, with qualitative results showing clean removal while preserving supporting background surfaces.

F. 3D Object Removal

  • Query-matched Gaussian primitives are pruned directly, leaving unselected primitives’ appearance and geometry unchanged.Membership is determined by cosine similarity between each primitive’s latent semantic attribute and the target query embedding or class prototype.Source: F. 3D Object Removal · p. 9
  • Object removal produces sharp contours without residual foreground floaters or collateral erosion of adjacent supporting structures.The reported visualizations attribute this precision to depth-edge refinement and cross-view mask propagation anchoring semantic transitions to physical boundaries.Source: F. 3D Object Removal · p. 9

VI. CONCLUSION

GaussianDS jointly optimizes appearance, aligned depth, and compact semantics with pose-aware mask propagation and depth-edge constraints; the conclusion reports improvements across reconstruction, query accuracy, and boundary localization while retaining real-time rendering. Its depth-discontinuity prior provides weaker constraints for planar, texture-only semantic transitions.

VI. CONCLUSION

  • The framework addresses supervision inconsistency through pose-aware pseudo-video mask propagation and joint optimization of appearance, aligned depth, and compact semantics.Depth-edge constraints are used to ground semantic boundaries in physical geometry.Source: §VI · pp. 9–10
  • Across evaluated benchmarks, GaussianDS improves reconstruction fidelity, open-vocabulary query accuracy, and boundary localization over multi-stage distillation baselines.The conclusion also reports preserved real-time rendering and downstream scene-editing support.Source: §VI · p. 10
  • Planar, texture-only semantic transitions remain a weaker case because they lack geometric depth variation.The conclusion identifies joint appearance-geometry boundary modeling as future work.Source: §VI · p. 10
Loading 2609.27850v1…