PaperlayerView exact revision on arXiv

Source-linked AI summary

Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation

Dominik Engel, Yousef Rajeh, Leon Sick, Ivan Viola

arXiv:2610.30908v1cs.CVcs.AI

TL;DR

Scene-lifted vision-language features can exhibit scene-dependent directional concentration that raw cosine comparison does not account for. The paper calibrates fixed features and text at query time, and reports consistent segmentation improvements across lifting methods and datasets in both multiclass and single-query settings.

  • Problem

    Lifted scene features can have a pronounced scene-dependent direction, while raw cosine comparisons do not account for the observed scene distribution.

    Source: §1 · p. 2

  • Method

    The method partially centers visual features and text prototypes, whitens text using a fixed generic-term reference bank, and applies soft clustering or a reduced-CSLS relevance margin by task.

    Source: §3.1 · p. 4, §3.5 · p. 7, §5 · p. 9

  • Results

    Calibration consistently improves mIoU across evaluated lifting methods and four datasets in both multiclass and single-query segmentation.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Scene centering improves class-relative feature separation in every evaluated diagnostic scene, and calibration makes use of fixed lifted representations without retraining or re-lifting.

    Source: §1 · p. 2

    Evaluation is limited to indoor scenes, and single-query tests do not establish reliable rejection of entirely absent targets or arbitrary natural-language queries.

    Source: §5 · p. 9

Abstract

from arXiv · show

Lifting vision-language features such as CLIP into 3D Gaussian reconstructions enables open-vocabulary segmentation through natural-language queries. Although CLIP represents a broad range of semantic concepts, the lifted features of individual scenes can exhibit strong directional concentration, while raw cosine similarity does not account for the scene-specific feature distribution. We show that substantial segmentation gains can be obtained by changing how a fixed lifted feature field is compared with text. Our method partially subtracts a shared scene direction from visual features and text prototypes and whitens text prototypes using a fixed reference vocabulary of generic terms. It then uses soft clustering for multiclass prediction and CSLS with a relevance margin against the reference vocabulary for single-query prediction. Beyond reducing directional concentration, scene centering improves class-relative feature separation in every evaluated scene. Across seven lifting methods and four datasets, the resulting query-time calibration consistently improves mean intersection over union in both multiclass and single-query segmentation. These gains require no retraining, source-image re-encoding, or feature re-lifting, demonstrating the value of scene-conditioned comparison for making better use of existing lifted representations.

1 INTRODUCTION

Scene-lifted CLIP features support open-vocabulary queries, but their scene-dependent geometry is not accounted for by raw cosine comparison. The paper calibrates fixed lifted features and text for multiclass and single-query segmentation, improving results across lifting methods and datasets without retraining or re-lifting.

1 INTRODUCTION

  • A scene’s lifted features can share a pronounced scene-dependent direction, and raw cosine comparisons do not explicitly account for that scene’s observed distribution.The paper motivates adapting comparisons to the scene rather than assuming global embedding geometry is equally informative everywhere.Source: §1 · p. 2
  • Single-query segmentation requires rejecting regions that do not match the target, a capability that multiclass competition among supplied labels does not establish.The paper therefore evaluates single-query segmentation separately from multiclass prediction.Source: §1 · p. 2
  • The method partially subtracts a shared scene direction from visual features and text prototypes, whitens text using a fixed generic-term bank, and uses soft clustering or reduced CSLS by task.The bank supplies negative competitors for a query margin and is independent of the requested class list.Source: §1 · p. 2
  • Calibration consistently improves multiclass and single-query segmentation across seven lifting methods and four datasets without retraining, source-image re-encoding, or feature re-lifting.The method operates on fixed lifted fields.Source: §1 · p. 2
  • Scene centering improves class-relative feature separation in every evaluated diagnostic scene, while component ablations examine feature calibration, score adjustment, and reference-vocabulary contributions.The full method combines these components.Source: §1 · p. 2

2 RELATED WORK

Related work improves language-aware 3D representations, adapts embedding geometry, or calibrates scores and predictions. This paper’s setting is query-time correction of already-lifted features, without retraining or repeating reconstruction or lifting.

2 RELATED WORK

  • Language-aware reconstruction methods embed or distill language features into radiance fields or Gaussian representations, while other methods organize instance identities or semantic scales.These approaches improve the 3D representation constructed from images.Source: §2 · p. 3
  • Dense language-feature methods improve text and image features directly, whereas efficient lifting methods construct language-aware fields through registration, multiview aggregation, inverse problems, or compact semantic indices.Their feature-construction strategies can complement postlifting calibration without repeating reconstruction or lifting.Source: §2 · p. 3
  • Other dense and 3D segmentation methods learn proposal or bias corrections, fuse multiview features, associate proposals with language evidence, or balance known and novel classes.The paper instead targets label-free correction of an already available field without learning a segmentation or confidence model.Source: §2 · p. 3
  • Embedding-geometry work motivates dominant-direction removal and covariance transforms, while scene-local statistics and a reference vocabulary determine this paper’s calibration of a lifted field.Other adaptations may instead use image access, labeled support, or learned parameter updates.Source: §2 · p. 3
  • Distribution-conditioned methods estimate corrections from unlabeled visual collections, selected predictions, or predicted class structure, providing alternatives on fixed features.They compete with scene centering and reference-text whitening by deriving corrections from visual distributions or class structure.Source: §2 · p. 3
  • Reference-population score normalization includes inverted softmax, CSLS, and querybank normalization; target-distribution methods instead adapt predictions jointly across samples or classes.The paper uses soft assignment refinement without graph regularization for multiclass prediction and a class-side CSLS offset with a query-versus-reference margin for single-query prediction.Source: §2 · p. 4

3 METHOD

The method calibrates a fixed lifted scene by centering visual and text features and whitening text with a fixed reference vocabulary, then adjusts scores for multiclass or single-query prediction.

3.1 SCENE CENTERING

  • Scene centering subtracts a partial shared scene direction from primitive and text features, then normalizes them.The row-wise feature calibration uses λ = 0.3.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 TEXT-QUERY WHITENING

  • Text queries are whitened using a mean and transform fitted only from the scene-centered reference bank, independently of the requested class list.Visual features remain scene-centered and are not whitened.Source: §3.2 · p. 5

3.3 REFERENCE VOCABULARY

  • The fixed reference vocabulary contains 45 generic terms and supplies text-whitening statistics and background competitors for single-query segmentation.The bank is selected from a pool of 64 generic words and phrases using held-out calibration scenes, then fixed.Source: §3.3 · p. 5

3.4 MULTI-CLASS SCORE CALIBRATION

  • For multiple requested classes, calibrated similarities initialize soft assignments that are refined with class centers and shared diagonal variance before each primitive receives its highest-weight class.Centers are initialized from each class’s eight highest-probability primitives.Source: §3.4 · pp. 5–6

3.5 SINGLE-QUERY SCORE CALIBRATION

  • For a single query, reduced CSLS adjusts query and reference scores, and a thresholded margin over the strongest reference score determines where the query is present.The margin compares each primitive’s corrected query score with its highest corrected reference-bank score.Source: §3.5 · pp. 6–7

4 EVALUATION

The evaluation examines feature geometry, segmentation across datasets and lifting methods, alternative query-time adaptations, and the contributions of the calibration components. Calibration improves mIoU broadly in multiclass and single-query settings, while centering also increases class-relative separation.

4 EVALUATION

  • The evaluation covers four datasets and seven lifted-feature methods, using mIoU, precision, and recall for open-vocabulary segmentation.Eligible Gaussians are assigned ground-truth points using each method’s filtering strategy and minimum Mahalanobis distance.Source: §4 · p. 7

4.1 INVESTIGATING THE EFFECTS OF FEATURE CALIBRATION

  • Centering reduces mean vMF concentration κ from 1600 to 517 and the leading-axis share from 76.8% to 47.3% across twelve ScanNet++ SFS scenes.Participation rank rises from 1.69 to 4.15, indicating feature magnitude is distributed across more axes rather than merely losing its shared direction.Source: §4.1 · p. 7
  • Class-relative separation rises in every evaluated setting: 1.820 to 2.056 on ScanNet, 2.854 to 3.163 on ScanNet++, and 2.414 to 2.738 on ScanNet++.These ratios compare between-class scatter with within-class scatter.Source: §4.1 · p. 7

4.2 OPEN-VOCABULARY SEGMENTATION

  • Calibration improves mIoU across datasets and lifting methods, with VALA and OpenGaussian mostly unchanged on LERF-OVS; single-query calibration improves IoU in all cases.Single-query results are generally below multiclass results; the authors suggest independently thresholded masks can reuse primitives, unlike multiclass argmax assignments.Source: §4.2 · p. 8, Table 1 · p. 8

4.3 COMPARISON TO OTHER QUERY-TIME ADAPTATIONS

  • The full calibration outperforms the compared query-time adaptations, while PTC and TransCLIP variants also improve segmentation and the Frolic variant improves only on ScanNet.These comparisons are multiclass-only; the alternatives require multiclass queries and are unsuitable for single-query calibration.Source: Table 2 · p. 8, §4.3 · p. 9

4.4 CONTRIBUTION ABLATIONS

  • The full method yields the largest mIoU gain on both datasets; Feature Calibration alone hurts on ScanNet++ but helps in every combination with other components.The ablation evaluates Feature Calibration, soft-clustering Score Calibration, and the generic Reference Vocabulary.Source: §4.4 · p. 9, Table 3 · p. 9

5 CONCLUSION

With geometry and lifted features held fixed, scene-conditioned calibration combines centering, text-prototype whitening, and score adjustment to improve mIoU across evaluated methods and datasets. The evaluation and proposed future directions remain bounded by indoor-scene testing and the use of a fixed generic reference bank.

5 CONCLUSION

  • Combining scene centering, regularized text-prototype whitening, and scene-relative score adjustment improves mIoU across evaluated lifting methods on four datasets in multiclass and single-query settings.The evaluation holds reconstructed geometry and lifted features fixed; multiclass prediction uses soft clustering and single-query prediction uses a reduced-CSLS margin against the reference vocabulary.Source: §5 · p. 9
  • The evaluation is limited to indoor scenes, with transfer to other environments and embedding models untested.Shared centering and whitening parameters were selected using SFS results on held-out indoor scenes.Source: §5 · p. 9
  • Single-query evaluation uses GT-present classes and held-out-selected thresholds, so it does not establish reliable rejection of absent targets or arbitrary natural-language queries.The suitability of the requested vocabulary and reference bank across different scenes and queries is also not established.Source: §5 · p. 9
  • The authors identify context-informed negative queries and evaluation beyond indoor environments as open directions.They suggest prior scene knowledge or previously queried concepts could inform the reference vocabulary.Source: §5 · p. 9
  • The paper releases source code and plans to provide reconstructed scenes, lifted feature sets, and a script reproducing the segmentations reported in Table 1.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL RESULTS

Additional results include illustrative single-query maps, dataset-wide segmentation metrics, and single-query component ablations. The supplied passages describe their scope and setup but do not provide table values or state comparative outcomes.

A ADDITIONAL RESULTS

  • Figure 4 shows one SFS query map from each of eight ScanNet++ scenes using present classes from a 100-class vocabulary.Raw and fully calibrated maps are independently min–max normalized, so the comparison concerns spatial response patterns rather than absolute score magnitudes.Source: A ADDITIONAL RESULTS · p. 19
  • Table 5 reports mIoU, precision, and recall across datasets, including 10-class ScanNet and 20-class ScanNet++ results.LERF-OVS and 3D-OVS use rendered 2D evaluation under their respective protocols.Source: A ADDITIONAL RESULTS · p. 19
  • Table 4 reports SFS single-query component ablations on ScanNet and ScanNet++ with thresholds optimized independently.Whitening and CSLS are no-ops without the reference bank; bank ablations use a query-versus-maximum-reference margin, while no-bank ablations threshold the query score directly.Source: Table 4 · p. 19, A ADDITIONAL RESULTS · p. 19

B REFERENCE VOCABULARY BANK

The reference vocabulary bank is shared across scenes, datasets, and lifting methods, and its statistics are fitted once from 45 words. The fitted calibration remains fixed as requested queries change.

B REFERENCE VOCABULARY BANK

  • The reference bank uses bare phrases shared across scenes, datasets, and lifting methods, without exact case-insensitive matches to evaluated class names.Broad semantic overlap with evaluated classes is not excluded; Table 6 presents the reference vocabulary.Source: B REFERENCE VOCABULARY BANK · p. 19, Table 6 · p. 19
  • Whitening mean and covariance are fitted once from 45 words, so adding, removing, or regrouping requested queries does not change the statistics.Requested queries are transformed using this fixed reference calibration.Source: B REFERENCE VOCABULARY BANK · p. 19
  • For N > 1, bank rows are discarded before class scoring; for N = 1, bank scores remain available for the target-minus-maximum-bank rejection margin.The single query is excluded from the mean and covariance estimate.Source: B REFERENCE VOCABULARY BANK · p. 19

C HELD-OUT THRESHOLDS

The main single-query evaluation applies thresholds selected on held-out scenes and fixed across evaluation scenes and queries. Raw cosine scores and reference-only relevance margins use separate thresholds.

C HELD-OUT THRESHOLDS

  • For the main N = 1 evaluations, thresholds are determined on held-out scenes and kept fixed across evaluation scenes and queries.Threshold values vary by dataset, method, and whether Raw or Ours is used.Source: C HELD-OUT THRESHOLDS · p. 19
  • Table 5 presents raw-versus-calibrated segmentation results for each dataset pair, reporting multi-query results before single-query results.Source: Table 5 · p. 20
  • Raw cosine scores and reference-only relevance margins have separate thresholds, applied without refitting or imposing a recall constraint during evaluation.Source: C HELD-OUT THRESHOLDS · p. 21

D SENSITIVITY OF λ AND α

The selected λ and α were chosen using held-out scenes and were generally robust to slight variations, although the selected pair was not the highest-scoring displayed test combination.

D SENSITIVITY OF λ AND α

  • Table 7 reports SFS multiclass mIoU (%) for ScanNet’s 19 classes and ScanNet++’s 100 classes.ScanNet values are shown on top and ScanNet++ values on the bottom of each cell; the bold cell marks the held-out selection.Source: Table 7 · p. 21
  • The selected λ and α are generally robust to slight variations, though the selected pair is not the highest-scoring displayed test combination.The bold cell marks the held-out selection, not the maximum among the displayed multiclass scores.Source: D SENSITIVITY OF λ AND α · p. 21, Table 7 · p. 21
  • The selected (λ, α) = (0.30, 0.40) was chosen by averaging multiclass and single-query mIoU on held-out scenes.LERF-OVS and OVS-3D use the same parameters.Source: D SENSITIVITY OF λ AND α · p. 21

E EVALUATION POPULATION AND CORRESPONDENCE PROTOCOLS

The evaluation retains active ground-truth points and assigns each to an eligible Gaussian, addressing protocols that omit points when predictions are filtered. Reported LangSplat results use a fixed-field adaptation rather than its complete rendered-image pipeline.

E EVALUATION POPULATION AND CORRESPONDENCE PROTOCOLS

  • OpenGaussian’s official ScanNet configuration fixes initial point locations and disables densification, while its evaluator ignores labels at Gaussian opacity below 0.1.Dr. Splat instead describes unconstrained Gaussian geometry and Gaussian-level evaluation with Mahalanobis pseudo-label assignment and significance weighting.Source: E EVALUATION POPULATION AND… · p. 21
  • The evaluation retains every active ground-truth point and assigns it to an eligible Gaussian using Mahalanobis distance before semantic evaluation.Eligible Gaussians have opacity of at least 0.1 and valid features; Table 8 contrasts this protocol with prior rules that omit ground-truth points.Source: E EVALUATION POPULATION AND… · p. 21, Table 8 · p. 22
  • Prior protocols can omit substantial numbers of ground-truth points when associated predictions are removed by opacity or feature-validity filtering.The resulting evaluated population is method-dependent; Table 8 reports omitted or unsupported ground-truth points under prior rules.Source: E EVALUATION POPULATION AND… · p. 21, Table 8 · p. 22
  • The association uses geometric covariance and opacity only for candidate eligibility, not as a distance factor; labels do not enter the rule.The same correspondences are used for raw and calibrated predictions, and multiple ground-truth points may select one Gaussian.Source: E EVALUATION POPULATION AND… · p. 21
  • LangSplat results use its decoded 512-dimensional level-3 language field as a fixed representation, not its complete rendered-image pipeline.The adaptation omits query-dependent level selection and spatial smoothing, which the passage says interfere with the calibrations.Source: E EVALUATION POPULATION AND… · p. 22

Related papers

Loading 2610.30908v1…