PaperlayerView exact revision on arXiv

Source-linked AI summary

EviSplat: Preserving Multi-View Evidence in 3D Gaussian Splatting for Open-Vocabulary Segmentation

Sungho Moon, Kota Shimomura, Junwoo Park, Wonhyeok Choi, Seunghun Lee, Takayoshi Yamashita, Sunghoon Im

arXiv:2609.34853v1cs.CVcs.AI

TL;DR

Open-vocabulary 3D segmentation must localize objects from free-form text, yet pre-query consolidation can discard cues relevant to later queries. EviSplat preserves individual observations for query-guided instance and Gaussian scoring, improving results across three datasets and rendered-view and point-level evaluations.

  • Problem

    Pre-query consolidation can discard view-specific cues that a later text query needs to select, despite observations varying in relevance and quality.

    Source: §1 · pp. 1–2

  • Method

    EviSplat retains observation features in class-agnostic instance banks and combines query-relevant instance scores with reliable Gaussian-local evidence.

    Source: §1 · p. 2, §4.2 · p. 6

  • Results

    Across three datasets, EviSplat improves over corresponding baselines in rendered-view and point-level evaluations; its LERF-OVS mean mIoU is 70.7.

    Source: §5.2 · p. 7, §6 · pp. 9–10

  • Takeaways & Limitations

    Different queries can draw on different cues from the same preserved evidence, with complementary gains from instance selection and Gaussian-local evidence.

    Source: §6 · pp. 9–10

    EviSplat remains dependent on source-mask quality and frozen CLIP features, which may limit attribute-combination and spatial-relation understanding.

    Source: §6 · p. 10

Abstract

from arXiv · show

Open-vocabulary 3D scene understanding enables object localization and segmentation from free-form text queries without a fixed category vocabulary. Many recent methods build on 3D Gaussian Splatting and consolidate multi-view observations, such as masked crops from individual views, into language features or compact object descriptors before the query is known. However, observations of the same object vary across viewpoints and are not equally informative: some reveal cues relevant to a particular query, whereas others provide incomplete or misleading evidence. Pre-query consolidation can therefore suppress cues on which a later query depends. We introduce EviSplat, which preserves individual observation features as evidence for later text queries. EviSplat retains individual observation features within class-agnostic 3D instances that represent objects, object parts, or background regions. It also learns, for each Gaussian, a distribution describing which visual appearances its observations support. Given a text query, EviSplat scores each instance using its most relevant observations. It then computes a score for each Gaussian by combining instance-level relevance with locally supported evidence, weighted by how often and how unambiguously that Gaussian was observed. Different queries can thus draw on different visual cues from the same preserved evidence. Experiments across diverse datasets and evaluation protocols demonstrate state-of-the-art performance, supporting the benefit of preserving multi-view evidence until query time and aggregating it according to the query.

1 INTRODUCTION

Open-vocabulary 3D segmentation localizes text-referred objects, but pre-query consolidation can discard view-specific cues that matter to later queries. EviSplat preserves those observations for query-guided scoring and improves segmentation across evaluated benchmarks.

1 INTRODUCTION

  • Viewpoint and mask-granularity changes produce different cues, while occlusion, blur, and inaccurate masks can make observations incomplete or misleading.Pre-query feature consolidation can replace the individual observations, preventing a later query from selecting among them.Source: §1 · pp. 1–2
  • EviSplat retains observation features in class-agnostic instance banks and learns per-Gaussian appearance evidence for query-guided instance and local scoring.Local evidence is weighted by observation reliability.Source: §1 · p. 2
  • LERF-OVS mean IoU rises from 63.6 to 70.7, an 11.2% relative gain over the LaGa baseline.Source: §1 · p. 2
  • EviSplat improves segmentation across multiple benchmarks and rendered-view and point-level evaluations, with complementary gains from observation selection and Gaussian-local evidence.Source: §1 · p. 2

2 RELATED WORK

Prior work incorporates language features into 3D scenes and groups scene elements into class-agnostic instances. Methods also differ in how they associate multi-view observations and aggregate their semantic information.

Language fields and instance grouping

  • Language-field methods distill or compress pretrained features into scene representations, including NeRF and 3DGS approaches such as LERF, LangSplat, and LEGaussians.LangSplatV2 uses sparse dictionary coefficients, while VLGS uses cross-modal rasterization and camera-view blending.Source: §2 · p. 2
  • Class-agnostic grouping links scene elements without semantic labels; examples include identity encodings from associated SAM masks (Ye et al., 2024) and scale-gated affinity features.OpenGaussian, GALA, and OpenGaFF (Li et al., 2026a) connect instance representations to language features.Source: §2 · pp. 2–3

Multi-view association and semantic aggregation

  • Multi-view methods associate observations and aggregate their information using adaptive granularity, attention, rendering-weighted aggregation, visibility gating, or object-level descriptor fusion.LaGa (Cen et al., 2025b) clusters and reweights descriptors; Arafa and Stricker (Arafa & Stricker, 2025) retain per-view embeddings and average top-k query-relevance scores.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries describe 3D Gaussian Splatting rendering and the common pipeline for locating text queries with language features. Query relevance compares image features with a text embedding and canonical negative phrases.

3D Gaussian Splatting

  • 3D Gaussian Splatting represents scenes as anisotropic Gaussians and renders pixels by front-to-back alpha blending, with each Gaussian’s contribution weighted along the ray.A Gaussian denoted g′ ≺v g lies in front of g along the ray in view v.Source: §3 · p. 3

Language features and query relevance

  • Language-feature localization segments source views at multiple mask granularities, encodes masked crops with frozen CLIP, consolidates features before the query, then scores them against the text.Consolidated features may be attached per Gaussian or stored as object descriptors.Source: §3 · p. 3
  • The relevance function compares a feature with the query’s unit-norm CLIP embedding and fixed canonical negative phrases; a score above 0.5 means it exceeds every negative phrase’s similarity.Source: §3 · p. 3

4 METHOD

EviSplat preserves separate multi-view observations in class-agnostic instances and Gaussian-local appearance distributions, then aggregates their relevance for each text query. Its two-stage method uses query-dependent instance selection and reliable local evidence to score Gaussians for segmentation.

4.1 MULTI-VIEW EVIDENCE PRESERVATION

  • Class-agnostic instances retain separate masked-crop features across views and may represent objects, object parts, or background regions.Observations are clustered across views using affinity features; an instance is not assumed to be a complete object.Source: §4.1 · p. 4
  • Each Gaussian’s evidence field models a weighted distribution over observed visual prototypes, approximated by a learned latent and scale-specific decoder.Only the latents and decoders are trained; the scene, CLIP encoder, and affinity field remain frozen, without text queries or a label vocabulary.Source: §4.1 · p. 5

4.2 QUERY-GUIDED EVIDENCE AGGREGATION

  • For each query, instance relevance pools the strongest observation scores and transfers them to Gaussians through query-independent affinity assignments.The mean of the top ⌈βn⌉ scores interpolates between average pooling and max pooling; CLIP features in each bank remain separate.Source: §4.2 · p. 6
  • Gaussian-local evidence selects the most relevant fixed fraction of prototype probability mass and can only raise the instance prior, scaled by observation reliability.Reliability accounts for observation count relative to the scene average and assignment ambiguity; the correction is smaller when local evidence is less reliable.Source: §4.2 · pp. 6–7
  • EviSplat averages Gaussian scores across semantic scales, thresholds them, and renders the selected Gaussians as the segmentation mask.Representations and reliability are built once without text and shared across queries.Source: §4.2 · p. 7

5 EXPERIMENTS

Experiments evaluate EviSplat on rendered-view and point-level segmentation benchmarks, with component analyses testing evidence selection, local evidence, and storage trade-offs. Results show strong performance across datasets and quantify where the components help or trade off.

5.1 EXPERIMENTAL SETUP

  • LERF-OVS uses rendered masks, whereas ScanNet evaluates 3D points; LERF-OVS reports mAcc@0.25 and ScanNet reports class-averaged mAcc.The LERF-OVS setup uses SAM masks and OpenCLIP ViT-B/16 features, with three semantic scales and shared aggregation parameters across four scenes.Source: §5.1 · p. 7

5.3 COMPONENT ANALYSIS

  • The instance-only prior raises LERF-OVS mIoU from 63.6 to 66.2, while top-β observation selection reaches 66.2 versus 62.8 for averaging all observations and 60.0 for taking the maximum.Adding reliability-weighted Gaussian-local evidence yields a further 4.5-point gain to 70.7 mIoU, though RAMEN falls from 65.1 to 62.9.Source: §5.3 · pp. 7–8, Table 3 · p. 9

5.2 MAIN RESULTS

  • 70.7 mean mIoU on LERF-OVS, 5.1 points above the best prior mean; EviSplat also reaches 97.4 mIoU on 3D-OVS versus 97.1 for the best prior.On LERF-OVS it has the highest mIoU on three of four scenes and mAcc@0.25 of 86.5 versus 82.9; on ScanNet it leads several class-protocol mIoU and mAcc results.Source: §5.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8

5.3 COMPONENT ANALYSIS

  • Fixed-probability-mass aggregation reaches 70.7 mIoU, versus 68.2 for mean observation features, 67.5 for averaging prototype relevances, and 44.2 for maximum prototype relevance.Mean observation features perform better on RAMEN; the comparisons support retaining multiple appearances and considering their probability support.Source: §5.3 · p. 8, Table 3 · p. 9
  • At D = 16, the learned field matches sparse empirical distributions at 70.2 versus 70.1 mean mIoU while reducing local storage by 43.7%, from 307.7 to 173.3 MiB.With D = 8, reducing K from 256 to 128 lowers mean mIoU from 69.3 to 67.8; storage figures exclude shared components.Source: §5.3 · pp. 8–9, Table 3 · p. 9
  • For “coffee,” EviSplat reaches IoU 0.909 against the annotated mug, compared with 0.347 for the instance-only variant and 0.276 for the baseline.For “tea in a glass,” EviSplat gives more complete annotated-target coverage while the instance-only masks remain similar to baseline.Source: §5.3 · p. 9, Figure 3 · p. 9

6 CONCLUSION

The conclusion presents EviSplat as preserving multi-view observations for query-dependent selection and refining instance scores with Gaussian-local evidence. It reports gains across datasets and evaluation protocols, while noting dependence on source-mask quality and frozen CLIP features.

6 CONCLUSION

  • The approach retains multi-view features in instance observation banks and learns a compact distribution of appearances supported at each Gaussian.Query-guided aggregation forms an instance prior from relevant observations and refines Gaussian scores with reliable local evidence.Source: §6 · p. 9
  • Across three datasets, EviSplat improves over corresponding baselines under rendered-view and point-level evaluations, with complementary gains from instance observation selection and Gaussian-local evidence.The method preserves individual observation features and uses query-guided aggregation without retraining the representation.Source: §6 · pp. 9–10
  • EviSplat remains dependent on source-mask quality and frozen CLIP features, which may not overcome limitations in attribute combinations and spatial relations.Refining fixed Gaussian-to-instance assignments and improving compositional query understanding remain future directions.Source: §6 · p. 10
  • The paper reports using generative AI for editing, code assistance, and figure-draft formatting, while stating that authors verified numerical values, designs, derivations, and citations.The authors retain responsibility for the manuscript, results, code, and figures.Source: AI USE STATEMENT · p. 10

A.1 OBSERVATION PREPARATION AND TRAINING

A.1 specifies observation filtering, confidence weighting, target construction, and optimization; A.2 records shared scoring conventions and post-processing.

A.1 OBSERVATION PREPARATION AND TRAINING

  • LERF-OVS retains masks over 400 pixels and removes cache masks overlapping finer-scale masks at IoU ≥0.8, unless that would remove every mask at a scale.The bank reuses baseline mask filtering; noise-labeled observations enter no bank. The bank and association cache retain different subsets, and codebook construction uses the cache subset.Source: §A.1 · p. 13
  • Assignment confidence is normalized within each view, preventing views with many associated Gaussians from automatically receiving greater total weight.A view with all-zero assignment margins receives zero weight, with no uniform fallback.Source: §A.1 · pp. 13–14
  • When a Gaussian’s evidence has zero total weight, its target is an all-zero placeholder and it is never sampled or contributes loss at that scale.Training errors if a scale has no positive-weight rows; the shared latent may still be trained through other scales, and zero observation reliability leaves fusion at the instance prior.Source: §A.1 · p. 14
  • The evidence field is trained on empirical evidence distributions before queries; sparse Adam updates Gaussian latents, while AdamW updates scale-specific decoders.Only decoder learning rates use cosine decay; stored observation features and prototypes are reused at query time without another image-encoder pass.Source: §A.1 · p. 14

A.2 SCORING IMPLEMENTATION AND POST-PROCESSING

  • Scoring uses a relative association margin with ε = 10^-12 and clipping to [0, 1]; one-candidate views receive margin one, and relevance logits are scaled by 10.Per-view weight normalization follows Eq. A.1.Source: §A.2 · p. 14
  • Soft assignments and observation reliability are computed once and reused across queries; prototype relevances are sorted once per query and scale for all Gaussians.These computations implement scoring terms defined in §4.2.Source: §A.2 · p. 14
  • For LERF-OVS, fused scores are percentile-clipped, min–max normalized, cosine-gated, thresholded at 0.7, rendered, and mask-thresholded at 0.3.The inherited cosine gate uses the highest-affinity instance at each scale, and no additional 3D smoothing is applied.Source: §A.2 · p. 14

B EVALUATION PROTOCOL

The evaluation uses consistent averaging and settings across comparisons, with repeated runs varying evidence-field training seeds while holding other components fixed. LERF-OVS reports 86.54% mAcc@0.25 for all three main-setting seeds.

B EVALUATION PROTOCOL

  • LERF-OVS and 3D-OVS scene mIoU averages annotated frame–class pairs, then reports an unweighted mean across scenes.All rows within each comparison use this convention.Source: B EVALUATION PROTOCOL · p. 14
  • Table C.1 extends the LERF-OVS comparison; its upper and lower groups distinguish rendered-feature-map queries from Gaussian selection and geometry rendering.The table caption marks reproduced baseline results with † and LightSplat reevaluations with ‡.Source: Table C.1 · p. 15
  • Three-run statistics vary evidence-field training seeds while keeping the photometric scene and affinity field fixed; ablations reuse affinity-field checkpoints.The varied seed affects codebook initialization, Gaussian sampling, and latent initialization.Source: B EVALUATION PROTOCOL · p. 15
  • 86.54% mAcc@0.25 is reported for all three evidence-field seeds in the main LERF-OVS setting, although their masks differ.The metric is the fraction of annotated frame–query pairs whose mask IoU exceeds 0.25, averaged across four scenes.Source: B EVALUATION PROTOCOL · p. 15

C ADDITIONAL EXPERIMENTS ON LERF-OVS

The additional LERF-OVS experiments examine evidence-selection and representation choices, observation quality, and appearance variation. The extended comparison distinguishes rendered-feature-map querying from Gaussian selection and geometry rendering.

C ADDITIONAL EXPERIMENTS ON LERF-OVS

  • The additional experiments examine observation selection, Gaussian-local evidence, representation size, observation quality, and cross-view appearance differences.Unless varied explicitly, experiments use the main LERF-OVS setting (β, γ, T) = (0.5, 0.25, 0.07); aggregation and sensitivity ablations fix one evidence-field seed.Source: C ADDITIONAL EXPERIMENTS ON LERF-OVS · p. 15
  • Aggregation and sensitivity ablations use a fixed evidence-field checkpoint across inference variants.This keeps the field fixed while testing the varied inference choices.Source: C ADDITIONAL EXPERIMENTS ON LERF-OVS · p. 15
  • The extended LERF-OVS comparison separates rendered-feature-map methods from methods that select Gaussians and render selected geometry.Mask success rates are reported only for the latter protocol.Source: C ADDITIONAL EXPERIMENTS ON LERF-OVS · p. 15

C.1 OBSERVATION QUALITY

The observation-quality experiments perturb source-view features or masks while holding scene geometry and grouping fixed. EviSplat is more robust than the baseline to tested feature replacements and improves mean IoU under estimated test-view occlusion.

C.1 OBSERVATION QUALITY

  • At 30% feature replacement, EviSplat’s mIoU drops 1.91 points, compared with 4.78 for the baseline and 2.07 for instance-only.Replacement features come from the same scene and semantic scale, but are not guaranteed to come from a different semantic class.Source: §C.1 · p. 16, Figure C.1 · p. 16
  • Under mask erosion, EviSplat’s accuracy stays within 0.13 points of its matched control, while all three variants change by at most 0.60 points.The control and erosion conditions re-encode the same 30% view subset, separating mask shrinkage from re-encoding effects.Source: §C.1 · p. 16, Table C.2 · p. 16
  • With more than 30% estimated test-view occlusion, EviSplat’s mean-IoU gain over the baseline is 10.42 points, versus 5.52 points at 30% or less.The higher-occlusion group contains 19 frame–query pairs, and gains vary across scenes; TEATIME contributes one pair to that group.Source: §C.1 · p. 17, Table C.3 · p. 17

C.2 QUERY-GUIDED EVIDENCE AGGREGATION

Query-guided selection improves target–competitor separation, and Gaussian-local evidence is associated with larger gains when observed appearances differ more. The representation-size experiments compare storage and accuracy trade-offs, storage growth with source views, and local query costs.

C.2 QUERY-GUIDED EVIDENCE AGGREGATION

  • Selecting the top 50% of observation scores raises the mean target–competitor relevance gap from 0.065 to 0.078, with gains in all four scenes.The gap is target relevance minus the strongest competing-instance relevance, not segmentation IoU.Source: §C.2 · p. 17
  • Cross-view feature distance is more strongly associated with additional Gaussian-local evidence gains (Spearman r = 0.388) than with instance-selection gains (r = 0.055).After adjustment for scene, log observation count, and target-mask size, the coefficients are 0.231 and 0.106, respectively; the association varies across scenes.Source: §C.2 · p. 18, Figure C.2 · p. 18
  • A probability-weighted mean prototype feature gives 67.67 mIoU, versus 70.59 for fixed-mass aggregation, supporting selection of query-relevant prototype support.The matched single-run comparison uses the same instance prior and learned distribution at β = 0.5.Source: §C.2 · p. 18, Table C.5 · p. 19
  • Positive-only fusion yields 70.59 mIoU versus 70.30 for signed correction; reliability-weighting variants differ by less than 0.4 points in this run.The combined reliability weight accounts for observation count and assignment confidence; signed correction can lower the instance prior.Source: §C.2 · pp. 18–19
  • Across three evidence-field seeds, β = 0.5 averages 70.67 ± 0.10 mIoU versus 69.89 ± 0.04 at β = 0.25, but scene gains are not uniform.The paired gains are 0.65, 0.92, and 0.78 points; this sweep is not a held-out test of the parameter choice.Source: §C.2 · pp. 19–20

C.3 REPRESENTATION SIZE, STORAGE, AND COMPUTATIONAL COST

  • D = 16 uses 173.3 MiB rather than 307.7 MiB for sparse storage, with comparable mean mIoU of 70.2 versus 70.1.D = 8 reduces local payload by 71.8% but lowers mean mIoU to 69.29; D = 32 uses 346.7 MiB, exceeding sparse storage.Source: §C.3 · p. 20
  • On FIGURINES, sparse payload grows from 17.4 to 46.5 MiB over 29–299 source views, while the D = 16 field remains at 22.0 MiB.The figure’s continuation beyond 299 views is hypothetical; shared components are excluded, so fixed field storage does not mean fixed total system storage.Source: §C.3 · p. 20, Figure C.3 · p. 21
  • Sparse aggregation takes 6.90 ms per local query, versus 12.13 ms for D = 8 and 13.34 ms for D = 32 in the measured comparison.The field decodes K logits per Gaussian, while sparse aggregation visits occupied bins; these timings exclude other pipeline costs.Source: §C.3 · pp. 20–21, Table C.7 · p. 21

D QUALITATIVE RESULTS

Qualitative results on LERF-OVS and ScanNet compare the baseline, instance prior only, and EviSplat, showing that gains vary by query and can depend on Gaussian-local evidence.

D QUALITATIVE RESULTS

  • For “rubics cube” and “pour-over vessel” on LERF-OVS, the instance prior alone falls below the baseline, while Gaussian-local evidence recovers the target.The figures compare target coverage and false-positive regions as instance observations and Gaussian-local evidence are introduced.Source: D QUALITATIVE RESULTS · p. 22
  • The LERF-OVS and ScanNet figures compare the baseline, instance prior only, and EviSplat on selected qualitative examples.Figure D.1 reports rendered-view mask IoU; Figure D.2 reports query-specific point IoU over the full scene.Source: D QUALITATIVE RESULTS · p. 22, Figure D.1 · p. 22, Figure D.2 · p. 23

Related papers

Loading 2609.34853v1…