PaperlayerView exact revision on arXiv

Source-linked AI summary

ProFuse: Efficient Cross-View Context Fusion for Open-Vocabulary 3D Gaussian Splatting

Yen-Jen Chiou, Wei-Tse Cheng, Yuan-Fu Yang

arXiv:2601.04754v2cs.CV

TL;DR

Open-vocabulary 3D scene understanding must recover geometry while assigning language-based semantics without fixed labels, and existing registration pipelines do not impose cross-view consistency or intra-mask cohesion. ProFuse uses dense correspondence to initialize Gaussians and form cross-view context proposals, then fuses proposal features during registration. It reports accurate open-vocabulary selection and point-level understanding, with semantic attachment taking about five minutes per scene, 2× faster than prior SOTA.

  • Problem

    Open-vocabulary 3D understanding must recover accurate geometry and meaningful language-based semantics, while registration pipelines do not impose cross-view consistency or intra-mask cohesion.

    Source: §1 · p. 2

  • Method

    ProFuse uses dense correspondence to initialize Gaussians and form cross-view 3D Context Proposals, whose global language features are associated with Gaussians during registration.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    ProFuse reports accurate open-vocabulary selection and point-level understanding on LERF and ScanNet, with semantic attachment taking about five minutes per scene, 2× faster than prior SOTA.

    Source: §5 · p. 8, §4.4 · p. 8

  • Takeaways & Limitations

    Correspondence-guided context association provides an efficient path to semantic association in 3DGS without render-supervised semantic training.

    Source: §5 · p. 8

    The method remains bounded by SAM segmentation quality and CLIP embedding errors, particularly for similar objects and uncommon text prompts.

    Source: §B.1 · p. 11

Abstract

from arXiv · show

We present ProFuse, an efficient context-aware framework for open-vocabulary 3D scene understanding with 3D Gaussian Splatting (3DGS). The pipeline enhances cross-view consistency and intra-mask cohesion within a direct registration setup, adding minimal overhead and requiring no render-supervised fine-tuning. Instead of relying on a pretrained 3DGS scene, we introduce a dense correspondence-guided pre-registration phase that initializes Gaussians with accurate geometry while jointly constructing 3D Context Proposals via cross-view clustering. Each proposal carries a global feature obtained through weighted aggregation of member embeddings, and this feature is fused onto Gaussians during direct registration to maintain per-primitive language coherence across views. With associations established in advance, semantic fusion requires no additional optimization beyond standard reconstruction, and the model retains geometric refinement without densification. ProFuse achieves strong open-vocabulary 3DGS understanding while completing semantic attachment in about five minutes per scene, which is two times faster than SOTA. Additional details are available at our project page https://chiou1203.github.io/ProFuse/.

1. Introduction

Open-vocabulary 3D understanding must combine accurate geometry with language-based semantics, but existing approaches leave gaps in semantic consistency. ProFuse adds correspondence-guided context proposals and fuses their semantics directly into 3D Gaussian Splatting.

1. Introduction

  • Earlier render-supervised methods transfer 2D vision-language predictions into 3D, while direct 3D methods assign language features to Gaussians for querying.Render-and-composite supervision can mismatch the original language embedding, while view-specific acquisition and querying make reasoning less direct and stable.Source: §1 · p. 2
  • Registration-based semantic fields bypass render-supervised semantic training, but the registration pipeline does not itself impose cross-view consistency or intra-mask cohesion.ProFuse aims to add those constraints directly to the representation.Source: §1 · p. 2
  • ProFuse introduces cross-view consistency and intra-mask cohesion directly into registration-based 3D Gaussian Splatting, without render-supervised semantic training.The framework also avoids gradient-based fine-tuning or backpropagation of language loss during semantic registration.Source: §1 · p. 2
  • Dense multi-view correspondence initializes Gaussians with accurate geometry and groups cross-view observations of the same object into 3D Context Proposals.These proposals represent objects across views rather than isolated frame masks, providing semantics aligned across viewpoints.Source: §1 · p. 2
  • Each proposal supplies a global language feature from its mask members, which is associated with corresponding Gaussians during feature registration.The paper describes this as a render-free semantic association that maintains per-primitive language coherence across views.Source: §1 · p. 2
  • ProFuse is presented as a compact, training-free route to consistent open-vocabulary 3D scene understanding built on correspondence-driven registration.The reported experiments cover object selection, point-cloud understanding, and training efficiency on existing benchmarks.Source: §1 · p. 2

2. Related Work

Related work spans explicit 3D representations and render-supervised or direct 3D language-feature methods. ProFuse builds on dense correspondence for cross-view grouping and 3D Gaussian initialization.

2. Related Work

  • Prior work includes render-supervised transfer of 2D vision-language signals into 3D and direct language-feature retrieval on Gaussians or points.Other work promotes cross-view consistency and semantic cohesion, while dense correspondence supports multi-view grouping and correspondence-driven Gaussian initialization.Source: §2 · p. 2
  • Dense correspondence provides wide-baseline matches and confidences for multi-view grouping and correspondence-driven 3D Gaussian initialization.ProFuse builds on this direction to couple context association with a registration-based semantic field.Source: §2 · p. 2
  • Figure 2 depicts pre-registration: matched pixels are triangulated into Gaussian seed points, while passing warped-IoU mask pairs form bipartite-graph edges.For each reference view, neighbors are selected through view clustering before dense matching.Source: Figure 2 · p. 3

3. Method

ProFuse builds a queryable semantic 3D Gaussian scene by pairing dense-correspondence pre-registration with context-guided feature registration. The pipeline groups cross-view masks into proposals, transfers their pooled language features to Gaussians, and supports direct 3D text queries without render-supervised semantic training.

3. Method

  • ProFuse combines dense-correspondence pre-registration and context-guided registration to produce a 3D Gaussian representation searchable directly with text queries.The method targets cross-view consistency and intra-mask cohesion without render-supervised semantic training.Source: §3 · p. 3

3.1. Dense Correspondence Pre-registration

  • Dense correspondences triangulate Gaussian centers and link masks across views, producing an initialized scene and mask links for semantic grouping.Geometric refinement adjusts the initialized primitives without needing to create a large number of new Gaussians.Source: §3.1 · pp. 3–4

3.2. 3D Context Proposals

  • Mutual overlap and box gates link masks across views; graph connected components become proposals, with small components filtered by member count and view support.The proposal graph is built from cross-view mask agreement under dense correspondence.Source: §3.2 · p. 4

3.3. Feature Registration

  • A mass-weighted, normalized pool of member-mask embeddings forms each proposal descriptor, which renderer contribution weights accumulate onto Gaussians.Registration uses renderer outputs and batched gather–scatter operations.Source: §3.3 · p. 5

3.4. Inference Procedure

  • At query time, PQ codes are decoded for cosine scoring, shortlisted Gaussian descriptors are re-scored, and active Gaussians are selected directly in 3D without render-based fine-tuning.The text query is encoded and normalized before similarity scoring.Source: §3.4 · p. 5

4. Experiments

ProFuse is evaluated on LERF-OVS object selection and ScanNet point-cloud understanding, with qualitative comparisons, runtime measurements, and a Top-K ablation. Its reported semantic attachment takes about five minutes per scene and is 2× faster than prior SOTA.

4.1. Implementation

  • Experiments use all four LERF scenes and ten sampled ScanNet scenes; the LERF object-selection scores are averaged per scene and then across scenes.The ScanNet point-cloud evaluation uses fixed hyperparameters and averages scores across the sampled scenes.Source: §4.1 · p. 5, Table 1 · p. 6, §4.3 · p. 8

4.2. Open-Vocabulary 3D Object Selection

  • On LERF-OVS, ProFuse isolates queried objects with fewer background activations than Dr. Splat, whose spillovers include clutter and specular reflections.Figure 4 compares qualitative object-level semantic queries; its caption describes cleaner retrieval and sharper correspondence between the text query and selected 3D content.Source: Figure 4 · p. 6, §4.2 · p. 7

4.3. Open-Vocabulary Point Cloud Understanding

  • On ScanNet, ProFuse feature visualizations show more coherent color regions and cleaner boundaries than Dr. Splat, particularly at furniture edges and object–wall contacts.The point-cloud evaluation reports mIoU and mAcc for 19-, 15-, and 10-class settings, but the supplied passages do not give the scores.Source: Table 2 · p. 7, §4.3 · p. 8

4.4. Training Efficiency

  • About five minutes per scene and 2× faster than prior SOTA, ProFuse has the fastest reported semantic-attachment runtime.The authors attribute the reduction to correspondence-guided initialization without densification and lightweight proposal-level feature fusion.Source: §4.4 · p. 8, Table 4 · p. 8

4.5. Ablation Study

  • On ScanNet, ProFuse reaches maximum accuracy at K=10, while registration-based baselines typically require K=40 to saturate.The authors attribute this to global proposal features concentrating semantic mass on leading Gaussians, with correspondence-initialized geometry reducing long-tail ambiguity.Source: §4.5 · p. 8, Table 5 · p. 8

5. Conclusion

ProFuse uses dense correspondences to establish cross-view semantic consistency in 3DGS without render-supervised semantic learning. Evaluations on LERF and ScanNet support accurate open-vocabulary selection and point-level understanding, alongside efficient semantic association.

5. Conclusion

  • ProFuse enforces cross-view semantic consistency in 3DGS without render-supervised semantic learning.Dense correspondences generate 3D Context Proposals, and visibility-weighted fusion yields a coherent semantic field.Source: §5 · p. 8
  • Experiments on LERF and ScanNet confirm accurate open-vocabulary selection and point-level understanding.The conclusion identifies correspondence-guided geometry as an efficient path to semantic association in 3DGS.Source: §5 · p. 8
  • The supported scope is open-vocabulary 3DGS understanding through correspondence-guided semantic association.The conclusion names object selection and point-level understanding as demonstrated tasks.Source: §5 · p. 8

A. Reproducibility and Code Release

The authors state that ProFuse implementation code will be released in a repository, including code for its three main stages.

A. Reproducibility and Code Release

B.1. Limitation

The appendix notes that ProFuse remains bounded by SAM and CLIP quality, including segmentation and embedding errors that can contaminate pooled proposal features. It also describes the method’s societal scope, representation background, related systems, and implementation settings.

B.1. Limitation

  • SAM segmentation quality and CLIP embedding errors constrain ProFuse, and a mixed mask can contaminate the pooled global feature for its grouped masks.The passage gives an example in which an 80% large-object and 20% small-object mask is grouped with masks of the large object under a 0.5 IoU threshold.Source: §B.1 · p. 11

B.2. Societal Impact

  • The stated application settings include interactive exploration, searchable 3D captures, indoor robotics, digital twins, and assistive guidance.The authors also recommend consent for capture and established governance for storage and sharing.Source: §B.2 · p. 11

C.1. 3D Gaussian Splatting

  • 3DGS represents scenes with anisotropic Gaussians whose covariance is factorized into rotation and diagonal scale.The appendix describes projection of covariance to image space and front-to-back alpha compositing for rendering.Source: §C.1 · p. 11

C.2. Product Quantization (PQ)

  • Product Quantization stores subvector centroid indices and supports similarity lookup through summed table entries.The appendix says this shifts computation from high-dimensional arithmetic to indexed retrieval while retaining correlation with true distances within known quantization bounds.Source: §C.2 · pp. 11–12

D.1. Correspondence Driven Gaussians

  • ProFuse initializes Gaussians from dense cross-view correspondences and uses pruning-only optimization without densification.The described setup samples reference views and neighbors, triangulates correspondences to seed Gaussians, then applies standard splat optimization.Source: §D.1 · p. 12

D.2. The ProFuse Framework

  • The semantic path forms per-Gaussian language descriptors from masks and text features, enabling open-vocabulary queries without render-supervised training.The appendix says mask features are fused using top-K ray contributions, descriptors are shared across views, and Product Quantization stores them compactly.Source: §D.2 · p. 12

D.3. Compare Model Settings

  • The related-work discussion contrasts ProFuse with LangSplat, LEGaussians, OpenGaussian, and Dr. Splat on feature learning, smoothing, association, and registration designs.It describes render-supervised feature learning for LangSplat, LEGaussians, and OpenGaussian, and direct feature registration for Dr. Splat.Source: §D.3 · pp. 12–13

D.4. Training Details

  • Implementation details include cross-view mask clustering, proposal-level feature registration, and sparse contribution-weighted accumulation.The clustering uses warped-mask overlap and visibility filtering; registration attaches descriptors in a frozen-scene pass, with sparse-matrix blocks available for memory control.Source: §D.4 · pp. 13–14

D.5. Evaluation Details

The appendix specifies the LERF object-selection and ScanNet point-cloud evaluation protocols, along with the shared experimental resources and settings.

D.5. Evaluation Details

  • Object selection uses cosine similarity to activate Gaussians, LERF-style reranking, contribution-weighted masks, and scene-averaged mIoU and mAcc@0.25.The activation threshold is grid-searched in 0.01 steps and fixed per method across scenes; reranking temperature is 3.0 and contribution threshold is 0.025.Source: §D.5 · p. 14, Table 5 · p. 14
  • ScanNet point-cloud evaluation transfers Gaussian-to-class cosine logits to mesh vertices with an opacity-weighted, rotation- and scale-aware Mahalanobis kernel.The protocol uses 64 candidates, a gating radius of 3.0, and temperature 1.0; points outside the gate use the nearest valid Gaussian.Source: §D.5 · p. 14

D.6. Computing Resource Configuration

  • Experiments use one NVIDIA A100 80 GB GPU, 170–210 ScanNet images per scene, and 30k scene-optimization iterations.Registration-based methods share a PQ codebook with sub-vector size 128, and each method uses its own best threshold.Source: §D.6 · p. 14

E.1. Gaussian Scene Experiments

These experiments compare Gaussian-scene initialization and correspondence-neighborhood size, assessing semantic performance, reconstruction, and computation. The default non-densified correspondence-guided scene offers the strongest reported semantic performance with competitive reconstruction quality and lower training time than its densified alternative.

E.1. Gaussian Scene Experiments

  • Raw correspondence seeds already capture major-object layout; pruning optimization sharpens geometry and appearance, with later steps refining shading and details.The qualitative progression is shown before optimization, after 7k pruning steps, and after 30k iterations.Source: §E.1 · p. 15
  • The non-densified correspondence-guided scene achieves the best mean LERF object-selection performance, while densification has mixed scene-level effects.The comparison reports mIoU and mAcc@0.25; the authors attribute densification's variability to scene-dependent similarity-threshold curves under one shared threshold.Source: Table 6 · p. 15, Table 7 · p. 15, §E.1 · p. 15
  • The densified correspondence-guided scene achieves the highest mean PSNR, while the default non-densified scene is slightly lower.The result indicates densification still benefits reconstruction quality despite accurate initial geometry.Source: Table 8 · p. 15, §E.1 · p. 15
  • Non-densified correspondence-guided scenes take roughly 14 minutes to train on average, versus about 22 minutes with densification.The default configuration is presented as a compromise: strongest semantic performance, competitive reconstruction, and lower pretraining time.Source: Table 9 · p. 15, §E.1 · p. 15

E.2. Neighbor per Reference

  • For Waldo kitchen, mIoU rises as neighborhoods grow while mAcc can fall under the shared threshold, despite per-configuration tuning exceeding mIoU 46 and mAcc 68.The reported explanation is that the mIoU peak shifts toward lower thresholds as neighborhood size increases.Source: §E.2 · p. 17
  • Three neighbors per reference offers a favorable trade-off between semantic gains and dense-correspondence initialization time.Initialization cost rises steadily because each added neighbor requires extra matcher evaluation and mask projection.Source: §E.2 · p. 17, Table 12 · p. 17
  • Adding three neighbors improves mIoU over zero neighbors, and larger neighborhoods improve several scenes further, though Figurines becomes less stable.Zero neighbors removes cross-view mask clustering and 3D Context Proposals; the reported gain over that setting is attributed to the proposals' global feature.Source: Table 10 · p. 17, §E.2 · p. 17

E.3. Additional Ablations

Additional ablations examine semantic concentration, correspondence thresholds, and inference efficiency. ProFuse's proposal-based registration concentrates contribution mass on fewer Gaussians and supports faster retrieval with a smaller scene representation.

More Details on Top-K Choice

  • ProFuse reaches its best or near-best ScanNet point-cloud mIoU and mAcc within top-10 Gaussians, while Dr. Splat usually needs K=40 to approach its peak.The authors relate this difference to useful semantic mass being concentrated on fewer Gaussians along each ray.Source: §E.3 · p. 17
  • Per-view top-10 proposals account for 0.87–0.99 of proposal mass; the top 1% of Gaussians carry 41–68% of accumulated mass and the top 0.1% carry 14–42%.The authors use this skewed distribution to explain why ProFuse saturates at K=10 while Dr. Splat needs larger K for comparable point-cloud performance.Source: §E.3 · p. 18

Warped IoU

  • A warped-IoU threshold of 0.2 is adopted because it retains reliable cross-view links while discarding clearly inconsistent warps.Scene-level mIoU curves are non-monotonic; the mutual-best-neighbor rule removes many noisy correspondences before the IoU gate.Source: §E.3 · p. 18

Inference Efficiency

  • On ScanNet, ProFuse uses 2.7 times fewer Gaussians than Dr. Splat and completes point-cloud inference in about 59 seconds versus about 99 seconds.The authors report semantic and geometric metrics as competitive or improved despite the smaller Gaussian set.Source: §E.3 · p. 18, Table 13 · p. 18

E.4. More Qualitative Results

Qualitative visualizations show how ProFuse's global features affect language-query activations and per-Gaussian feature coherence across views.

E.4. More Qualitative Results

  • For text queries, global features sharpen activation on the queried object, suppress background clutter, and maintain activation within the object's extent.The examples include cosine-similarity activations for coffee mug, old camera, nori, and spoon queries.Source: §E.4 · pp. 18–19, Figure 3 · p. 19
  • PCA visualizations show more coherent ProFuse feature regions aligned with objects and surfaces, stable across viewpoints compared with Dr. Splat.The comparisons cover ScanNet and LERF scenes.Source: §E.4 · pp. 18, 20, Figure 4 · p. 20
  • Additional ScanNet feature visualizations are presented as examples of ProFuse's context understanding.The section characterizes ProFuse as carrying context-level interpretation and consistency in 3D scene understanding.Source: §E.4 · pp. 18–20
Loading 2601.04754v2…