Source-linked AI summary
Hybrid Gaussians for Robust Open-Vocabulary 3D Segmentation with Multi-View Object Association and Boundary Refinement
Xueqi Qiu, Yueming Sun, Tianyu Zhang, Yuxuan Xia, Yang Long
TL;DR
Incomplete or noisy 2D cues hinder consistent open-vocabulary 3D segmentation across views. Hybrid Gaussians combine object identities and language-aligned semantics with multi-view association and boundary refinement; experiments report strong results on LERF and 3D-OVS, including 59.1% mIoU on LERF.
Problem
Incomplete, noisy, or intermittent 2D cues hinder consistent open-vocabulary 3D segmentation across views.
Source: §1 · p. 1Method
Hybrid Gaussians jointly represent object identities and language-aligned semantics, using Observation Fusion and Semantic Contrastive Learning for multi-view association and Boundary Reconstruction Optimization for contour refinement.
Source: §1 · p. 1, §2.2 · p. 3, §2.3 · p. 4Results
59.1% mIoU on LERF and 96.1% mIoU on 3D-OVS are the best average results, exceeding the second-best results by 7.0 and 0.1 points, respectively.
Source: §3.2 · p. 4Takeaways & Limitations
The framework improves identity consistency and semantic discriminability while refining contour quality and local spatial precision.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Open-vocabulary 3D segmentation localizes objects from free-form text queries, but remains challenging in real image sequences: incomplete or noisy 2D supervision destabilizes multi-view identity assignment, while full-scene semantic learning weakens object-level discriminability. We introduce Hybrid Gaussians, a unified 3D representation jointly modeling object association and language-aligned semantics. Its Multi-View Object Association mechanism combines Observation Fusion and Semantic Contrastive Learning to improve identity consistency and semantic discrimination. Boundary Reconstruction Optimization further refines local boundary structure to improve contour quality. Experiments on LERF and 3D-OVS demonstrate strong quantitative and qualitative performance. Our method achieves 59.1\% mIoU on LERF, yielding a 13.4\% relative gain over the baseline. Project page: https://nora202.github.io/hybridgaussians.
1. INTRODUCTION
Hybrid Gaussians jointly represent object identity and language-aligned semantics for object-conditioned rendering and learning. Multi-view association and boundary refinement target cross-view consistency, semantic discrimination, and contour quality.
1. INTRODUCTION
- Prior approaches lift 2D foundation-model supervision into point-based 3D representations, but incomplete, noisy, or intermittent cues hinder consistent segmentation across views.Source: §1 · p. 1
- Each Gaussian jointly carries an object identity and a learnable language-aligned semantic embedding for object-conditioned rendering and learning.Source: §1 · p. 1
- Multi-view Object Association addresses inconsistent cross-view object association and weak object-level semantic discrimination.It combines Observation Fusion with Semantic Contrastive Learning.Source: §1 · p. 1
- Boundary Reconstruction Optimization uses a lightweight objective to improve contour quality.Source: §1 · p. 1
- Experiments on LERF and 3D-OVS demonstrate strong qualitative and quantitative performance.Source: §1 · p. 1
2. METHOD
Hybrid Gaussians combine object identity and language-aligned semantics in each Gaussian. Multi-view Object Association refines identity and semantic discrimination, while Boundary Reconstruction Optimization focuses reconstruction on object contours.
2.1. Hybrid Gaussians
- Each Gaussian carries an object assignment and learnable language-aligned embedding, forming a shared representation for object-conditioned rendering and learning.The representation follows 3DGS and augments its Gaussian primitives with identity and semantic attributes.Source: §2.1 · pp. 1–2
2.2. Multi-view Object Association
- Observation Fusion aggregates object evidence across views using learnable view reliability, then converts the fused evidence into soft associations and hard identities.For Gaussians without valid observations, the preceding iteration’s soft association and hard identity are retained.Source: §2.2 · p. 2
- Semantic Contrastive Learning uses object-conditioned semantic features to improve within-object consistency and separate different foreground objects.Features are aggregated within propagated object masks; another valid view of the same object provides a positive pair, while other foreground objects provide negatives.Source: §2.2 · pp. 2–3
2.3. Boundary Reconstruction Optimization (BRO)
- Boundary Reconstruction Optimization applies gradient reconstruction in a narrow contour neighborhood to encourage sharper, more accurately localized object transitions.Its boundary bands combine predicted opacity and propagated object masks, covering both the target contour and potentially misaligned predicted boundaries.Source: §2.3 · pp. 3–4
3. EXPERIMENTS
Hybrid Gaussians achieves the best average mIoU on both benchmarks, while component ablations show complementary contributions from association, contrastive learning, and boundary refinement.
3.1. Implementation Details
- Evaluation uses LERF and 3D-OVS, SAM-initialized masks propagated with SAM2, and OpenCLIP ViT-B/16 features pretrained on LAION-2B.Experiments were conducted on an NVIDIA RTX 4090 GPU.Source: §3.1 · p. 4
3.2. Results and Analysis
- 59.1% LERF mIoU and 96.1% 3D-OVS mIoU are best average scores, exceeding second-best results by 7.0 and 0.1 points, respectively.The evaluation covers LERF and 3D-OVS.Source: §3.2 · p. 4
- The method produces more complete object regions and cleaner boundaries, including better preservation of fine-grained queried objects with less interference from surrounding regions.Examples include “tea in a glass” and “sake cup and chopsticks.”Source: §3.2 · p. 4
3.3. Ablation Studies
- The full OF, SCL, and BRO combination achieves the best reported metrics: 59.13% and 96.12% mIoU, and 60.54% and 88.27% mBIoU on LERF and 3D-OVS, respectively.OF consistently improves both benchmarks; BRO notably improves boundary quality, while SCL alone is mixed but complementary in combination.Source: Table 3 · p. 4, §3.3 · p. 4
3.2. Results and Analysis
- Training is second-fastest on both datasets, with time reduced versus OpenGaussian by 10.4% and 15.7%, despite added inference and memory costs.The paper describes this as a practical trade-off between computational overhead and segmentation performance.Source: §3.2 · p. 4
4. CONCLUSION
The conclusion presents Hybrid Gaussians as a unified representation for object-conditioned rendering and learning, combining multi-view association with boundary refinement.
4. CONCLUSION
- Hybrid Gaussians unifies object-conditioned rendering and learning with object identities and language-aligned semantics.Observation Fusion and Semantic Contrastive Learning improve identity consistency and semantic discriminability; Boundary Reconstruction Optimization refines contour quality and local spatial precision.Source: §4 · p. 4
- Observation Fusion and Semantic Contrastive Learning improve identity consistency and semantic discriminability.Source: §4 · p. 4
- Boundary Reconstruction Optimization further refines contour quality and local spatial precision.Source: §4 · p. 4