PaperlayerView exact revision on arXiv

Source-linked AI summary

PROVE: Probabilistic Visual Grounding via Patch-Level Evidence for Vision-Language Models

Mayank Nautiyal, Tianru Zhang, Li Ju, Ekta Vats, Prashant Singh

arXiv:2610.32440v1cs.CVcs.LG

TL;DR

Deterministic CLIP grounding maps do not indicate localisation reliability or whether the referent is absent. PROVE models per-patch membership with Beta distributions on frozen CLIP features, and remains competitive in localisation while supporting selective prediction and absent-referent detection.

  • Problem

    Deterministic CLIP grounding maps do not show whether a localisation is reliable or whether the referent is absent.

    Source: §1 · p. 1

  • Method

    PROVE adds a post-hoc evidential adapter to frozen CLIP features, modeling per-patch membership with Beta distributions and using prior pull to keep absent examples near an uninformative prior.

    Source: §1 · p. 2

  • Results

    PROVE remains competitive in localisation, its vacuity supports selective prediction, and prior pull supports absence detection (testA AUROC 0.83 versus 0.56 without Lnt).

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    A shared per-patch posterior provides grounding, spatial reliability, and an image-level absence score in a single forward pass.

    Source: §1 · p. 2

    Some localisation gain stems from supervised adaptation, performance is bounded under distribution shift, and PROVE inherits biases and localisation errors from frozen CLIP features and its Grad-ECLIP cue.

    Source: §6 · p. 8

Abstract

from arXiv · show

Visual grounding with vision-language models such as CLIP typically produces a single deterministic relevance map for an image-query pair, without indicating whether the localisation is reliable or whether the referent is present at all. We introduce \textbf{PROVE} (\textbf{Pro}babilistic \textbf{V}isual Grounding via Patch-Level \textbf{E}vidence), a post-hoc evidential adapter that maps frozen CLIP patch features to per-patch Beta distributions without modifying the backbone. PROVE casts grounding as per-patch membership: each patch accumulates positive and negative evidence for whether it belongs to the referent, so a query may match zero, one, or many patches. We train PROVE with an evidential objective on soft patch-coverage targets, and define a prior-pull objective that keeps absent-referent examples near an uninformative prior, so that vacuity serves as an absence signal without a dedicated presence classifier. During inference, a single forward pass yields, in closed form, a posterior-mean grounding map, a spatial vacuity map, and an image-level absence score. We evaluate PROVE on referring-expression benchmarks: it is competitive in localisation with state-of-the-art CLIP grounding methods, its uncertainty tracks localisation error and supports selective prediction under abstention, and its image-level score detects when the referent is absent altogether. Our evidential patch-membership modelling offers a simple, drop-in mechanism for probabilistic verification in frozen vision-language grounding systems.

1. Introduction

Deterministic CLIP grounding maps cannot indicate whether a localisation is reliable or whether a referent is absent. PROVE adds patch-level probabilistic membership to frozen CLIP features, while related methods either provide deterministic maps or uncertainty at the global image–text level.

1. Introduction

  • CLIP grounding maps provide neither localisation reliability nor a way to distinguish a valid match from an unsupported response when the referent is absent.CLIP’s contrastive objective does not represent a referent absent from an otherwise matched scene.Source: §1 · pp. 1–2
  • PROVE models each patch’s referent membership with a Beta distribution, yielding localisation, spatial reliability, and absence detection from a shared representation.The prior-pull objective keeps no-target examples near the uninformative prior, so absence requires no dedicated presence classifier.Source: §1 · p. 2

2. Related Work

  • Existing CLIP grounding methods return deterministic relevance maps, whereas PROVE predicts patch-level distributions that express both relevance and its reliability.Unlike global image–text uncertainty, patch-level uncertainty indicates where the query is supported.Source: §2 · pp. 2–3
  • PROVE keeps CLIP frozen and trains a lightweight post-hoc adapter, rather than finetuning an entire vision-language backbone.It uses Grad-ECLIP as an input cue and refines it into an evidential map with per-patch uncertainty.Source: §2 · p. 2
  • PROVE’s Beta-based patch membership adapts evidential uncertainty to language-conditioned grounding, where accumulated evidence across patches can also signal an absent referent.Prior Beta-based evidence models address classification, dense prediction, and bounded regression.Source: §2 · p. 3

3. Method

PROVE maps frozen CLIP patch and text features, alongside a Grad-ECLIP cue, to per-patch Beta distributions for membership. Fractional coverage targets and evidential objectives train the adapter; one forward pass provides grounding, uncertainty, and absence readouts.

3.1. Problem Setup and Overview

  • Frozen CLIP patch and text features plus a Grad-ECLIP localisation cue form the inputs to PROVE’s patchwise probabilistic grounding model.CLIP and Grad-ECLIP remain frozen; relevance is not normalised across patches, allowing a referent to occupy zero, one, or many patches.Source: §3.1 · p. 3

3.2. Patchwise Beta Relevance

  • Each patch’s membership probability is modeled by a Beta distribution, whose parameters accumulate positive and negative evidence over an uninformative Beta(1, 1) reference.The relevance field factorises into independent per-patch Betas under the stated conditional-independence assumption.Source: §3.2 · p. 3

3.3. Patch-Level Predictive Quantities

  • The Beta mean gives patch relevance, while vacuity and predictive variance distinguish predictions with similar means but different evidential support.Vacuity is highest at the uniform prior and decreases as evidence accumulates; at fixed mean, variance decreases with concentration.Source: Figure 2 · p. 3, §3.3 · p. 4

3.4. Evidential Parameterisation

  • The evidential head represents each Beta parameter as a unit reference plus non-negative evidence, bounded by a capacity γ.The capacity prevents unbounded concentration and fixes an otherwise underdetermined evidence scale because supervision constrains mean relevance, not concentration.Source: §3.4 · p. 4

3.5. Training Objective

  • Training aligns the Beta mean with fractional patch coverage and penalizes evidence in the channel unsupported by the target.Fractional targets preserve partial boundary coverage; the prior-pull term keeps absent-referent patches near the reference prior, while the full objective anneals the regularizer weight.Source: §3.5 · pp. 4–5

3.6. Inference, Abstention, and Absence Detection

  • One forward pass yields a posterior-mean grounding map and foreground-focused uncertainty, while absence is scored by the lack of any high-evidence patch.High foreground uncertainty indicates weak support and can rank predictions for abstention; the absence score uses patch evidence without a separate classifier.Source: §3.6 · p. 5

4. Experiments

Experiments evaluate PROVE’s localisation, uncertainty, and absent-referent detection on gRefCOCO and transfer datasets. They also examine whether vacuity ranks localisation quality and supports selective prediction.

4.1. Experimental Setup

  • Training uses ∼209k gRefCOCO examples, with evaluation on testA, testB, Flickr30k Entities, and PhraseCut using a frozen CLIP ViT-B/16 backbone.Source: §4.1 · p. 5

4.2. Evaluation Protocol

  • On gRefCOCO people, IoU@k is 0.616 versus 0.299 for the best baseline, though PROVE is supervised while the baselines are training-free.Under transfer, PROVE attains the best IoU@k on Flickr30k and remains competitive elsewhere, while saliency baselines lead on PG and EBPG.Source: Table 1 · p. 6, §4.2 · p. 6
  • For three-zebra queries, PROVE’s mean map covers all referents without spatially normalizing relevance; for absent queries, vacuity is high almost everywhere while baselines still peak.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
  • AUROC reaches 0.849 on testA and 0.813 on testB, highest versus Grad-ECLIP (0.682/0.746) and the control (0.823/0.789); the threshold rejects about two-thirds of absences while retaining T-acc ≥0.86.Source: Table 2 · p. 7, §4.2 · p. 7
  • Across four datasets, IoU@k falls monotonically across vacuity bins, with the lowest-vacuity bin localising best and the highest worst.Source: Figure 4 · p. 7, §4.2 · p. 7
  • As uncertain predictions are rejected, retained-set IoU@k rises monotonically; on gRefCOCO testA people, it increases from ∼62% to ∼75% at 50% rejection.Source: Figure 4 · p. 7, §4.2 · p. 7

5. Ablation Study

The ablations compare PROVE with a deterministic adaptation and isolate the roles of its two auxiliary loss terms. They attribute localisation gains over training-free baselines to supervised adaptation, while the auxiliary terms support absent-query detection and evidence calibration.

5. Ablation Study

  • PROVEdet shares PROVE’s post-hoc architecture up to the output layer but uses a sigmoid logit trained with binary cross-entropy on soft coverage targets, without either auxiliary term.Source: §5 · p. 7
  • PROVEdet matches PROVE on PG, is slightly higher on IoU@k, and clearly higher on EBPG across four datasets, indicating supervised adaptation—not the evidential head—accounts for gains over training-free baselines.PROVE remains within a small margin on PG and IoU@k while providing vacuity for selective grounding and a stronger absence signal than the deterministic control.Source: §5 · pp. 7–8
  • Removing the prior-pull term drops AUROC from 0.83 to 0.56 on testA and from 0.80 to 0.57 on testB, while balanced accuracy falls to 0.50 on both splits.Localisation is unchanged or slightly improved, so the prior-pull term supplies the absence signal at negligible localisation cost.Source: §5 · p. 8
  • Removing the unsupported-evidence regularizer lowers IoU@k by about 0.02 to 0.03 and reduces T-acc from 0.95 to 0.85 on testA and 0.90 to 0.83 on testB.The ablation indicates that this term stabilizes localisation, evidence scale, and the thresholded absence decision.Source: §5 · p. 8

6. Limitations

PROVE’s localisation gains partly reflect supervised adaptation, and its capability remains bounded under distribution shift. Its reliance on frozen CLIP features and a Grad-ECLIP cue also carries their biases and localisation errors forward.

6. Limitations

  • Some localisation gain comes from supervised adaptation rather than the evidential parameterisation alone, and capability remains bounded under distribution shift.The reported degradation on Flickr30k and PhraseCut is modest, and PROVE remains competitive on those transfer sets.Source: §6 · p. 8
  • Table 3 reports PROVE’s mean over five seeds; higher values are marked per metric, while PG is described as comparable across methods.Source: Table 3 · p. 8
  • Table 4’s loss ablation covers gRefCOCO testA and testB, with single-seed runs and one auxiliary term removed per row.Source: Table 4 · p. 8
  • PROVE inherits biases and localisation errors from frozen CLIP features and its Grad-ECLIP saliency prior.Stronger backbones or saliency priors could address these inherited limitations.Source: §6 · p. 8

7. Conclusion

The conclusion presents PROVE as a post-hoc evidential adapter for probabilistic grounding with frozen CLIP features. Across referring-expression benchmarks, it reports competitive localisation, useful uncertainty for selective prediction, and detection of absent referents.

7. Conclusion

  • PROVE predicts per-patch Beta distributions over referent membership from frozen CLIP features instead of producing only a deterministic map.Source: §7 · p. 8
  • Per-patch membership allows a query to match zero, one, or many patches, while prior-pull keeps absent-referent examples near an uninformative prior.The resulting vacuity signal indicates absence without a dedicated presence classifier.Source: §7 · p. 8
  • A single forward pass yields a grounding map, spatial uncertainty map, and image-level absence score in closed form.Source: §7 · p. 8
  • Across referring-expression benchmarks, PROVE is competitive in localisation, supports selective prediction through uncertainty, and detects absent referents better than a deterministic control trained with the same supervision.Source: §7 · p. 8

S1. Datasets

The study constructs training and evaluation splits from three public referring-expression datasets. gRefCOCO supplies training and in-domain evaluation data, while Flickr30k Entities and PhraseCut are used for zero-shot transfer evaluation.

S1. Datasets

  • The splits use three public referring-expression datasets: gRefCOCO, Flickr30k Entities, and PhraseCut.Source: S1. Datasets · p. 11
  • gRefCOCO includes single-target, multi-target, and no-target expressions; its train split is the sole training source, val supports model selection and absence-threshold calibration, and testA/testB provide in-domain evaluation.Source: S1. Datasets · p. 11
  • Flickr30k Entities contributes 9,905 phrase-region examples for zero-shot transfer evaluation, retained with valid chain mappings and expressions of at least two words.Source: S1. Datasets · p. 11
  • PhraseCut contributes 10,004 test examples for zero-shot transfer evaluation, retained when polygon annotations and images load successfully.Source: S1. Datasets · p. 11

S2. Backbones and Feature Extraction

Experiments use frozen OpenAI CLIP encoders, with ViT-B/16 as the main backbone and ViT-L/14 as a larger-backbone replication. The backbone-specific patch grids and feature dimensions differ, while the rest of the pipeline is unchanged.

S2. Backbones and Feature Extraction

  • The main results use frozen CLIP ViT-B/16, and the findings are replicated with ViT-L/14; Table S1 reports localisation across backbones.Table S1 reports PROVE results as mean±std over five seeds for ViT-B/16 and four for ViT-L/14.Source: S2. Backbones and Feature Extraction · p. 11, Table S1 · p. 11
  • ViT-B/16 produces 196 patch tokens on a 14 × 14 grid, while ViT-L/14 produces 256 on a 16 × 16 grid; only the head’s input projection and positional embeddings adapt to their dimensions.Both use 224 × 224 images, and the frozen backbone features, text embedding, and Grad-ECLIP prior are precomputed and cached.Source: S2. Backbones and Feature Extraction · p. 11

S3. Results with a Larger Backbone

With ViT-L/14, PROVE improves localisation over ViT-B/16 while preserving absent-referent detection and uncertainty behaviour. The absence-score comparison favors total strength, which uses foreground and background evidence and performs similarly to top-k pooled strength.

S3. Results with a Larger Backbone

S4. Absence Score Design

  • Total strength uses either channel: foreground evidence marks target patches, background confidence can preserve weakly localised present images, and prior-pulled absent images remain low.Membership-only scoring discards background evidence, pushing some present images with weak foreground evidence toward the absent cluster and retaining fewer.Source: S4. Absence Score Design · p. 12
  • The strength score (S1) has the best AUROC and highest T-acc; top-k strength matches it within <0.003 AUROC, while membership-only trails on T-acc.The top-k result supports absence as a distributed lack of evidence rather than a signal from a single patch.Source: S4. Absence Score Design · p. 12, Table S3 · p. 13

S5. Evidence Capacity

Across the tested evidence capacities, PROVE’s PG, IoU@k, and absence AUROC remain stable, while EBPG increases with capacity. Additional qualitative examples show vacuity concentrated around ambiguous boundaries and competing referents.

S5. Evidence Capacity

  • Across γ values {5, 10, 20, 40, 100}, PG, IoU@k, and absence AUROC are stable, while EBPG increases; the default γ=20 lies in the stable region.The authors attribute the EBPG trend to greater capacity letting the mean concentrate relevance mass more tightly on the referent.Source: S5. Evidence Capacity · p. 13

S6. Additional Qualitative Results

Related papers

Loading 2610.32440v1…