PaperlayerView exact revision on arXiv

Source-linked AI summary

Stream3Dv2: Geometric-Semantic Fusion Enhanced Streaming Zero-Shot 3D Scene Understanding

Jie Xu, Na Zhao

arXiv:2608.21136v1cs.CV

TL;DR

Open-vocabulary zero-shot 3D understanding remains difficult to deploy on streaming RGB-D data because of computational latency and noisy 2D masks. Stream3Dv2 addresses these issues with nested local-to-historical processing, geometric-semantic fusion, and manifold-based refinement, outperforming existing baselines across streaming segmentation and detection while supporting language-driven scene understanding with an LLM agent.

  • Problem

    Zero-shot 3D perception must handle sequential RGB-D inputs efficiently while mitigating noise and ambiguity in lifted 2D segmentation masks.

  • Method

    Stream3Dv2 uses nested local multi-view and local-to-historical processing with semantic-guided mask fusion and manifold-aware point-cloud refinement.

  • Results

    20.6% semantic AP versus 15.5% for MV3DIS on ScanNet200, while Stream3Dv2 also surpasses competitors across ScanNet++ and MatterPort3D metrics.

  • Takeaways & Limitations

    The framework provides training-free open-vocabulary streaming 3D perception and extends to language-driven 3D scene understanding through an LLM-based agent.

Abstract

from arXiv · show

Recently, open-vocabulary zero-shot 3D scene understanding using vision foundation models has emerged as a promising alternative to data-intensive supervised methods. However, deploying these models in real-world scenarios is severely hindered by their inability to efficiently handle streaming RGB-D inputs and their inherent vulnerability to noise 2D segmentation masks. To address these critical limitations, we propose Stream3Dv2, a novel training-free framework designed for robust streaming 3D perception. Stream3Dv2 processes sequential data through an original nested local-to-historical architecture, capturing multi-view consistency while circumventing the high computational overhead so as to support timely responses. At its core, we introduce a comprehensive geometric-semantic fusion mechanism that resolves geometric noise and semantic ambiguity by explicitly utilizing semantic guidance and formulating 3D segmentation as solving point-and-set merging and partitioning problems. Furthermore, we present an innovative manifold-distance-based point cloud refinement strategy. This approach leverages local manifold graphs for point-to-manifold optimization that mitigates the boundary delineation failures caused by Euclidean-distance metrics, and employs geometric bounding boxes to dynamically activate and update historical instances for achieving rapid manifold-to-manifold refinement. Extensive experiments on public datasets demonstrate that Stream3Dv2 consistently outperforms existing baselines in foundational open-vocabulary streaming 3D segmentation and detection. Finally, we show that integrating our framework with an LLM-based agent enables advanced language-driven 3D scene understanding, underscoring its potential for open-world embodied intelligence. Code will be updated at https://github.com/SubmissionsIn/Stream3D.

I. INTRODUCTION

Stream3Dv2 targets the streaming and noise-robustness limitations of open-vocabulary zero-shot 3D scene understanding. Its nested local-to-historical framework combines geometric-semantic fusion and manifold-based refinement for training-free streaming perception.

  • Motivation: Supervised 3D scene-understanding methods depend on costly annotated point clouds and generalize poorly to unseen instances or scenes with domain gaps.These methods are typically trained on curated datasets with paired point clouds and human-annotated labels.
  • Motivation: Zero-shot 3D methods still struggle with streaming RGB-D inputs and noisy 2D segmentation masks caused by cross-modal discrepancies, sensor noise, and view-dependent semantics.Many existing approaches require offline access to the full scene or historical frames, limiting online deployment.
  • Framework: Its geometric-semantic fusion filters noisy masks and uses semantic-driven mask merging and partitioning to combine complementary grid-prompted and semantic-prompted 2D masks.The 3D segmentation problem is formulated through point-and-set merging and partitioning, jointly supporting mask denoising and fine- and coarse-grained fusion.
  • Framework: Stream3Dv2 decomposes streaming instance segmentation into nested local multi-view and local-to-historical tasks, exploiting multi-view consistency while maintaining time-efficient processing.The framework is designed for sequential perception without requiring the full scene at once.
  • Framework: Point-to-manifold optimization on local manifold graphs addresses Euclidean metric limitations, while manifold-to-manifold updates dynamically activate historical masks for efficient refinement.The approach targets inaccurate point-cloud boundaries and progressive errors during subsequent merging and updating.
  • Results: Stream3Dv2 is training-free and zero-shot, with experiments reporting superior foundational segmentation and detection performance and advanced language-driven understanding when integrated with an LLM agent.The framework is presented as scalable for open-world 3D perception and embodied-intelligence applications.

A. Stream3Dv2: Geometric-Semantic Fusion Framework for Open-Vocabulary Streaming Zero-Shot 3D Segmentation

Stream3Dv2 formulates streaming zero-shot 3D segmentation as nested local multi-view and local-to-historical tasks. It combines geometric denoising, multi-view mask selection and merging to process sequential RGB-D inputs while limiting irrelevant views and computational latency.

  • Streaming formulation: Streaming zero-shot 3D segmentation recursively updates masks using the current RGB-D frame, local point cloud and historical scene state.The streaming formulation contrasts with full-sequence processing, which assumes access to globally accumulated point clouds and masks.
  • Nested architecture: Stream3Dv2 decomposes the task into local multi-view segmentation and local-to-historical segmentation within a nested framework.A sliding window retains the most recent k frames for local processing before merging results into the historical scene.
  • Geometric processing: Grid-prompted masks provide broad class-agnostic coverage but are coarse and noisy, motivating geometry-based denoising and multi-view filtering.The local pipeline includes single-view mask denoising, set-covering-problem filtering and IoU-based key-mask merging.
  • Geometric processing: Single-view denoising retains spatially coherent points and suppresses boundary noise by enforcing intra-instance continuity and inter-instance separation.Points violating either Euclidean-distance constraint are discarded from each view’s mask.
  • Multi-view fusion: The set-covering formulation selects the smallest mask subset that covers all key points, filtering redundant or noisy masks across local views.Farthest-point-sampled key points represent the local point cloud before mask selection.

C. Semantic-Driven Fine-Grained 3D Segmentation

The semantic-driven stage combines semantic-prompted masks with coarse geometric masks to resolve ambiguity and refine class-agnostic instances. It formulates mask refinement as a set-partitioning problem that can split coarse masks and retain disjoint novel instances.

  • Semantic guidance: Semantic-prompted segmentation supplies explicit class information but is limited by prompt vocabulary and may not cover complete instances.The method therefore combines semantic masks with grid-prompted coarse masks rather than relying on either source alone.
  • Semantic-driven partitioning: Semantic masks are assigned to the coarse mask with the largest inclusion degree when overlap exceeds the threshold 1 − α.The formulation tolerates boundary leakage when α < 1, which accommodates noisy mask boundaries.
  • Semantic-driven partitioning: The resulting cropped semantic parts are pairwise disjoint subsets of each coarse mask and can partition it into finer-grained instances.Disjointness prevents duplicated point assignments among semantic masks.
  • Semantic-driven partitioning: The set-partitioning formulation refines coarse-grained masks by selecting semantic-mask decompositions or retaining original masks and disjoint novel instances.Binary variables indicate whether a coarse mask is decomposed or a semantic mask is retained as a new instance.
  • Refined output: The refined mask set integrates semantic masks, undecomposed coarse masks and semantic masks retained as novel instances.This produces more informative 3D instance segmentation while splitting class-agnostic coarse masks according to fine-grained semantic evidence.

D. Point Cloud Manifold Refinement

Stream3Dv2 refines local point clouds with a manifold-aware graph instead of relying solely on Euclidean distances. Point-to-manifold assignment propagates geodesic costs from mask seeds, preserving surface geometry during label assignment.

  • Manifold-aware representation: Euclidean distance can misassign points across curved or thin structures because it does not capture non-Euclidean surface geometry.The method addresses this limitation through point-to-manifold assignment on a local manifold graph.
  • Manifold-aware representation: The manifold graph represents voxelized super-points using centroids, regularized covariances and accessibility edges.Each node corresponds to a subset of local points, while covariance regularization supports near-planar or degenerate shapes.
  • Manifold-aware metric: Symmetric Mahalanobis edge weights penalize normal displacement more strongly than tangential displacement, encouraging paths to follow intrinsic surface geometry.The metric makes tangential motion comparatively inexpensive while discouraging off-surface movement.
  • Point-to-manifold assignment: For each mask, an Eikonal distance field propagates from mask seeds and records the minimum graph cost to every node.Each node is assigned to the mask whose geodesic front reaches it first.
  • Point-to-manifold assignment: Iterative Bellman relaxation simultaneously updates the coupled mask distance fields by propagating minimum accumulated costs over the super-point graph.After R iterations, nodes receive the label with the smallest finite distance, while unreachable nodes remain unassigned.
  • Refined output: Assigned labels are mapped from super-points back to their constituent points to update the refined local mask set.Previously unassigned points inherit a super-point label when one is available.

E. Local-to-Historical Streaming 3D Instance Segmentation

The local-to-historical module propagates freshly extracted local masks into historical instances for efficient streaming segmentation. It uses bounding-box retrieval, overlap-based instance updates, and a union of static, dynamic, and newly detected instances.

  • Streaming outcome: The local-to-historical procedure supports dynamic streaming scene understanding while exploiting local-view consistency for robust perception.It updates historical masks using freshly extracted local masks rather than recomputing the full sequence.
  • Spatial retrieval: Bounding boxes retrieve only spatially overlapping historical masks before precise point-level intersection, reducing unnecessary comparisons.Historical AABBs are pre-indexed in a spatial grid for constant-time candidate lookup.
  • Instance division: The module classifies historical instances as dynamic when they overlap a local mask and static otherwise.A local mask with no overlapping historical instance is treated as newly detected.
  • Mask update: A local mask merges into the dynamic instance with the largest overlap; otherwise, the update preserves inter-instance separation.The resulting historical segmentation combines static, dynamic, and newly detected instances.

A. Experimental Setup

Experiments evaluate Stream3Dv2 on indoor-scene benchmarks using segmentation, detection, and streaming comparisons against open-vocabulary baselines. The setup covers class-agnostic and semantic performance under zero-shot and streaming conditions.

  • Datasets: Experiments use ScanNet200, ScanNet++, and MatterPort3D indoor-scene benchmarks for 3D instance segmentation.The validation sets contain 312 ScanNet200 scenes and 50 ScanNet++ scenes; MatterPort3D uses its testing set.
  • Metrics: Segmentation is evaluated with AP, AP25, and AP50 for both class-agnostic and semantic 3D instance segmentation.AP averages precision across IoU thresholds from 0.5 to 0.95, while AP25 and AP50 use IoU thresholds of 0.25 and 0.5.
  • Baselines: Comparisons include full-sequence, streaming, and zero-shot 3D segmentation methods, including Stream3D and locally adapted baselines.The “+Loc2his” suffix denotes local segmentation followed by naive local-to-historical streaming merging.
  • Detection setup: Open-vocabulary 3D detection is compared with OpenM3D, Zoo3D1, and Zoo3D0.These are the three baselines selected for the detection evaluation.
  • Implementation: Implementation uses Open3D, CLIP with ViT-H, SAM2 for grid prompts, and SAM3 for semantic prompts.The reported settings include 20 local frames, manifold distance 0.05, and overlap threshold 0.2 for IoU-based key-mask merging.

B. Main Results

Stream3Dv2 improves zero-shot streaming segmentation and detection across indoor-scene benchmarks, while supporting language-driven grounding, reasoning, navigation, and fuzzy queries through an LLM agent.

  • Qualitative results: Stream3Dv2’s visualizations compare RGB-D scenes, Stream3D, Stream3Dv2, and ground truth across several indoor environments.The examples are presented for ScanNet200, ScanNet++, and MatterPort3D scenes.
  • Segmentation results: 8.5% and 7.5% semantic AP are obtained by OVIR-3D+Loc2his and MaskClustering+Loc2his on ScanNet200 under streaming zero-shot constraints.These results illustrate the difficulty of maintaining segmentation fidelity when both constraints apply.
  • Segmentation results: 20.6% versus 15.5% semantic AP on ScanNet200 shows Stream3Dv2 outperforming the full-sequence zero-shot baseline MV3DIS while operating in streaming mode.On ScanNet++ and MatterPort3D, Stream3Dv2 surpasses the best competitors across all reported segmentation metrics.
  • Language-driven understanding: An LLM agent equipped with Stream3Dv2 supports 3D instance grounding, scene reasoning, path navigation, and fuzzy open-vocabulary queries.The agent receives RGB values and Stream3Dv2’s 3D segmentation point-cloud files.
  • Per-class analysis: 38.3% average semantic AP50 on ScanNet200 exceeds Stream3D’s 20.2% and MaskClustering’s 18.7% across the first 20 categories.Reported gains include office chair 22.6% →56.6% and bed 26.8% →60.0%.
  • Detection results: 14.2% AP50 and 24.8% AP25 are achieved for open-vocabulary 3D detection under simultaneous zero-shot and streaming settings.The comparison evaluates Stream3Dv2 on ScanNet200 against zero-shot detection baselines.

C. Model Analysis

Stream3Dv2 is evaluated under degraded 2D masks, streaming scenes, prompting choices, ablations, and parameter variations. The combined modules improve segmentation quality while maintaining scalable, near-real-time processing and extending downstream applicability.

  • Robustness from 2D to 3D: Severe 2D mask degradation includes temporal inconsistency, over-segmentation, and semantic errors under real-world conditions.These cases motivate robustness from 2D mask generation through 3D perception.
  • Streaming 3D segmentation: The streaming process incrementally segments newly discovered instances and refines partially observed objects as frames arrive.The approach is illustrated across room-level and large-building-level cases.
  • Streaming 3D segmentation: Local segmentation remains below 100 ms, while mask-pool memory and local-to-historical processing grow linearly with the number of RGB-D frames.This supports scaling from single rooms to large building environments.
  • Ablation study: 27.1% class-agnostic AP and 20.6% semantic AP on ScanNet200 are achieved when MVF, SDS, and PMR are jointly integrated.The ablation describes these modules as complementary and reports the joint configuration as highest across almost all metrics.
  • Prompting analysis: Semantic prompts provide class guidance but limited spatial coverage, whereas grid prompts improve whole-scene object identification and support near-real-time streaming perception.Pure semantic prompts achieve only 14.0% class-agnostic AP on ScanNet200 in the cited comparison.
  • Parameter analysis: The framework is insensitive to local-frame window size across dense and sparse frame conditions, with k = 20 used by default.The analyzed parameters also include key-point rate γ, geometry distance δ, and overlap threshold α.
  • Conclusion: Evaluations across ScanNet200, ScanNet++, and MatterPort3D report performance above streaming baselines and highly competitive with state-of-the-art offline methods.The extensions particularly improve multi-view semantic consistency, geometric manifold refinement, and open-vocabulary grounding.
  • Conclusion: Scaling streaming 3D perception to unconstrained, ultra-large outdoor environments with dynamic occlusions remains an open challenge.Future work targets outdoor scene understanding and coupling with physical embodied agents for robotic manipulation.

APPENDIX

The appendix details Stream3Dv2’s geometric denoising, multi-view mask filtering, and set-partitioning procedures. These operations combine spatial constraints, greedy coverage selection, and a closed-form partitioning solution.

  • Implementation details: The appendix covers geometry-based single-view mask denoising, SCP-based multi-view noise-mask filtering, and closed-form set partitioning.It also includes implementation details for the overall Stream3Dv2 algorithm.
  • Geometry-based single-view mask denoising: Single-view denoising enforces intra-instance continuity and inter-instance separation using a point-cloud distance threshold δ.The constraints retain spatially coherent manifolds and suppress boundary noise caused by projection inaccuracies.
  • SCP-based multi-view noise mask filtering: Multi-view filtering selects a key-mask subset that covers all key points by approximately solving an NP-hard set-covering problem greedily.Masks are ordered by size, and each iteration adds the mask covering the most currently uncovered points.
  • SCP-based multi-view noise mask filtering: The greedy filtering algorithm stops when all key points are covered or no remaining mask provides positive marginal gain.Selected masks are added to M while covered points are removed from the uncovered set U.
  • Closed-form solution for set partitioning: Set-partitioning constraints separate across coarse class-agnostic and mutually disjoint semantic masks, enabling independent optimization.The resulting closed-form solution maximizes the objective globally by selecting the better feasible option for each mask.

B. Complexity Analysis

The complexity analysis shows that manifold refinement scales linearly in the number of points at the pipeline level and adds limited overhead. Detection-based localization further reduces historical mask comparisons when overlap is sparse.

  • Complexity of manifold refinement: Manifold refinement is the heaviest component but exhibits linear growth with respect to the number of input points at the complete-pipeline level.The analysis decomposes refinement into super-point graph construction and Bellman relaxation.
  • Complexity of manifold refinement: Bellman relaxation aggregates neighbor information over super-points and masks, while assigning original points to enclosing super-point masks takes O(n_∆).The experiments fix the relaxation count at R = 5, making this cost negligible relative to nearest-neighbor search.
  • Complexity of manifold refinement: Nearest-neighbor estimation contributes O(n_∆log n_∆), which dominates refinement because V ≪ n_∆ and E = O(V).The refinement therefore adds only a small constant overhead to the complete processing pipeline.
  • Complexity of manifold refinement: Refinement space complexity is O(n_∆ + V + E + C_∆V), with O(n_∆) dominated by storage of the original point cloud.Auxiliary graph and mask structures occupy only a fraction of the memory.
  • Detection-based fast mask localization: Detection-based localization first performs constant-cost 3D intersection tests and applies detailed point comparisons only to the α_t fraction of surviving overlapping pairs.This contrasts with normal localization, which compares every point pair for every historical-local mask pair.
  • Detection-based fast mask localization: When α_t is small for most frames, detection-based localization yields an asymptotic improvement of roughly O(k_t^2) over normal localization.The efficiency gain becomes more significant as frame and mask counts increase.

C. Convergence Analysis

The convergence analysis identifies the manifold-distance iteration as a monotone, lower-bounded Bellman relaxation. On the finite graph, it stabilizes after finitely many steps at the exact shortest-path distance.

  • Preliminaries and notation: For each mask, the anisotropic geodesic distance is defined as the minimum path length from a graph node to any source super-point.The source super-points are those intersecting the corresponding mask.
  • Monotonicity and lower bound: The Bellman relaxation operator preserves order and is deflationary, so each update cannot increase any distance component.These properties follow from taking minima over the current value and non-negative neighbor-distance candidates.
  • Monotonicity and lower bound: Every iterate remains component-wise above the true geodesic distance, establishing a lower bound for the decreasing relaxation sequence.The proof uses initialization, induction, and monotonicity of addition and the minimum operation.
  • Finite convergence: Because the graph is finite and edge weights are fixed, the decreasing sequence becomes constant after finitely many iterations.Its limit satisfies the Bellman fixed-point equation.
  • Fixed-point optimality: The fixed point equals the exact shortest-path distance, establishing fixed-point optimality for the manifold refinement iteration.The result is stated as Theorem 1: ϕ(∞) = ϕ∗.

3) Finite-step convergence via dynamic programming:

The Bellman-relaxation solver reaches exact geodesic-distance convergence after finitely many iterations bounded by the graph’s source-relative hop diameter. In practice, the compact super-point graph supports a fixed five-iteration setting, while disconnected nodes receive a sentinel label.

  • Finite-step convergence: The Bellman-Ford analysis shows that R ≥ D iterations suffice for exact convergence of all distance fields.D is the maximum number of edges on a simple shortest path from any node to a source set.
  • Practical setting: Because the super-point graph is typically smaller than approximately five hops in diameter, the implementation fixes R = 5 for every scene.This setting is used as the default in experiments.
  • Disconnected components: Nodes unreachable from every source retain infinite distance and receive sentinel label −1.This handles disconnected graph components without requiring special processing.
  • Order independence: The convergence guarantee is independent of node-visit order, although the implementation uses voxel-grid order to accelerate spatial propagation.Any permutation produces the same final result.
Loading 2608.21136v1…