PaperlayerView exact revision on arXiv

Source-linked AI summary

ProtoLIP: From Sentence-Level to Object-Level Evidence Disentanglement

Yan Zhu, Yongbo Chen, Zhengming Ding, Rebecca Faust

arXiv:2609.16284v2cs.CVcs.AI

TL;DR

Fine-grained queries do not necessarily yield object-specific evidence, and localization maps alone do not show that evidence constitutes a prediction. ProtoLIP routes queries through semantic families and localized prototypes, improving evidence localization and separation while constructing matching scores from those contributions.

  • Problem

    Fine-grained VLM objectives do not ensure object-level evidence is separated within sentences or spatially from co-occurring entities and context, and localization alone does not establish predictive influence.

    Source: §1 · pp. 1–2

  • Method

    ProtoLIP adds a prototype-mediated layer to a frozen VLM, using semantic families to constrain prototype eligibility and the complete query to determine localized evidence contributions to matching scores.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    ProtoLIP improves object-specific localization and evidence separation across settings, with localization gains transferring to several independently pretrained VLMs.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Because localized prototype contributions form the matching score itself, the prediction can be exactly decomposed across prototypes and semantic families rather than explained by a separate post-hoc layer.

    Source: §1 · p. 2, §5 · p. 10

    For single-family queries the overlap loss is zero, and unmatched queries use all prototypes and are excluded from spatial regularization.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Query-conditioned vision-language models enable fine-grained interpretation by revealing which visual content supports a given textual query and how this evidence changes across queries. However, semantically, sentence-level evidence does not necessarily decompose into object-specific contributions, while spatially, object-level evidence can remain entangled with co-occurring objects and surrounding scene context. Across multiple VLM architectures and independent benchmarks, we observe persistent object-level evidence entanglement. Moreover, exposed evidence maps do not necessarily correspond to the evidence that directly constitutes the model's prediction. To disentangle visual evidence at both semantic and spatial levels, we introduce ProtoLIP, a lightweight prototype-mediated evidence layer that organizes reusable visual prototypes into text-derived semantic families and uses coarse-to-fine evidence routing, where semantic families constrain prototype eligibility and the complete query determines fine-grained prototype contributions. Our studies show that ProtoLIP improves evidence localization and separation across query granularities, achieving average relative gains of 29% in Pointing and 43% in Energy across four object- and phrase-level OOD benchmarks. Its localization gains also transfer to independently pretrained VLMs, with larger improvements observed in several transfer settings. On the primary backbone, ProtoLIP also improves image-text matching discrimination while remaining competitive with a spatially supervised grounding model in object-level localization. Crucially, ProtoLIP constructs its image-text matching score directly from localized prototype evidence, enabling exact decomposition across prototypes, semantic families, and spatial evidence without spatial annotations or backbone retraining.

1 INTRODUCTION

Fine-grained text conditioning does not ensure that visual evidence is separated by object or spatially from surrounding context. ProtoLIP adds prototype-mediated evidence to a frozen VLM, improving localization and separation while making matching scores decomposable from localized evidence.

1 INTRODUCTION

  • Existing fine-grained VLM objectives do not require object-level evidence to be disentangled within sentences, so object queries can retain evidence from co-occurring objects and context.Figure 1 contrasts ItemizedCLIP's retained evidence from the woman and background with ProtoLIP's object-specific evidence.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Evidence disentanglement separates predictive evidence across semantic and spatial dimensions, and that evidence must connect to the matching score to explain a prediction.Localization alone does not establish that connection because attention weights need not reflect predictive influence, and gradient-based attribution has faithfulness and completeness limitations.Source: §1 · p. 2
  • ProtoLIP uses semantic families to select prototypes and the complete query to determine their contributions, making localized prototype evidence the matching score's additive terms.The layer operates on a frozen VLM and enables object-, phrase-, and sentence-level evidence with exact decomposition into prototype and semantic-family contributions.Source: §1 · pp. 2–3
  • ProtoLIP improves object-specific localization and evidence separation across settings, with localization gains transferring to several independently pretrained VLMs.It remains competitive with a spatially supervised grounding model in object-level localization while improving image–text matching on the primary backbone.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work has refined vision–language alignment through finer interactions, richer textual supervision, and text-conditioned visual representations, but its textual units need not correspond to individual objects. ProtoLIP addresses this granularity gap and links localized prototype evidence directly to the matching score.

2 RELATED WORK

  • Fine-grained alignment advances include token-wise visual–textual interactions, region alignment, richer caption supervision, text-conditioned pooling, and independent textual-item representations.These approaches refine alignment or conditioning, but textual items can still contain multiple objects, attributes, and relations.Source: §2 · p. 3
  • Separating descriptions that contain multiple objects does not require their constituent objects to have separated visual evidence; ProtoLIP resolves descriptions into localized object evidence.Its semantic families constrain prototype eligibility, while the complete query determines prototype contributions to matching and spatial evidence.Source: §2 · p. 3, Figure 2 · p. 4
  • Training-free localization methods derive maps from pretrained representations, whereas dedicated grounding models use explicit detection or grounding supervision; ProtoLIP uses no spatial annotations and keeps its backbone frozen.Source: §2 · p. 3
  • Plausible attribution maps and prototype similarities need not faithfully reflect predictive evidence, whereas ProtoLIP builds its score from localized prototype contributions for exact decomposition.Source: §2 · p. 3

3 METHOD

ProtoLIP routes queries through text-derived semantic families to select visual prototypes, then uses query-conditioned prototype evidence to construct matching scores and spatial explanations. Spatial regularization encourages compact, separated family evidence, and the training objective combines matching with spatial losses.

3.1 TEXT-DERIVED SEMANTIC FAMILIES

  • Text-derived families constrain prototype eligibility, while the complete query determines prototype contributions; routing uses a training-description taxonomy and falls back to all prototypes when no family matches.The taxonomy is built offline from localizable concepts and aliases; the prototype bank contains 512 trainable vectors distributed across 39 families.Source: §3.1 · p. 4

3.2 FAMILY-CONSTRAINED PROTOTYPE EVIDENCE

  • Prototype assignments pool image patches into prototype-specific evidence, which a gated residual MLP transforms before eligible prototypes are scored against the complete query.Each prototype’s weighted query compatibility contributes directly to the matching score, making that score decomposable into prototype contributions.Source: §3.2 · pp. 4–5

3.3 QUERY-SPECIFIC PREDICTION EVIDENCE

  • The spatial map assigns every prototype contribution to image locations, preserving the full matching score; its positive part is normalized for localization and visualization.The decomposition is exhaustive, with positive and negative values indicating evidence that supports or opposes the image–text match; Figure 3 illustrates family contributions for a compositional sentence query.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 SPATIAL EVIDENCE REGULARIZATION

  • Spatial regularization aggregates active-family prototype assignments into normalized family maps and discourages diffuse evidence and excessive sharing between families.The losses penalize excessive area and pairwise overlap without forcing semantically related families into disjoint regions.Source: §3.4 · pp. 5–6
  • Unmatched queries are excluded from spatial regularization, while single-family queries have no overlap penalty and rely on query-conditioned prototype scoring for within-family distinctions.Source: §3.4 · p. 6

3.5 LEARNING OBJECTIVE

  • Training combines symmetric contrastive image–text matching with area and overlap regularizers, applying spatial regularization only to positive diagonal pairs and using the same score at training and inference.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments evaluate ProtoLIP’s localization, matching, and transfer across query granularities and backbones. They report stronger object-specific evidence, compare against external localization methods, and ablate the layer’s components.

4.1 EXPERIMENTAL SETUP

  • ProtoLIP trains only its evidence layer on Itemized-CC0.3M with a frozen ItemizedCLIP backbone, then evaluates zero-shot on external benchmarks.Evaluation covers object-, phrase-, and caption-level evidence; metrics include Pointing, Energy, AUC, balanced accuracy, and retrieval Recall@k.Source: §4.1 · pp. 6–7

4.2 EVIDENCE DISENTANGLEMENT ACROSS QUERY GRANULARITIES

  • ProtoLIP improves Pointing and Energy on VOC20 object and ADE20K class queries, and improves object-query localization and matching on COCO and Flickr30K while preserving or modestly improving caption localization.Figure 5 reports the OOD-benchmark results; ADE20K gains are more modest, consistent with its broader long-tail vocabulary and greater mismatch with the training corpus.Source: §4.2 · p. 7, Figure 5 · p. 8
  • Direct ProtoLIP retains exact decomposition but moderately reduces caption retrieval; 50/50 fusion recovers the gap and improves retrieval over the frozen backbone while retaining spatial evidence, at the cost of exact decomposition.Direct ProtoLIP is the primary model because its localized prototype evidence directly corresponds to the matching prediction.Source: Table 1 · p. 7, §4.2 · p. 8

4.3 GENERALIZATION ACROSS VISION–LANGUAGE BACKBONES

  • ProtoLIP improves Pointing and Energy in all eight backbone–dataset combinations; matching improves for SigLIP-WebLI and remains broadly stable for FLAIR.These results transfer the localization gains beyond ItemizedCLIP to independently pretrained backbones.Source: §4.3 · p. 8, Table 2 · p. 9
  • Across same-data retrained Multi-positive SigLIP and FLAIR, localization gains average 66% in Pointing and 37% in Energy across three evaluation datasets; DreamLIP localization does not improve.The authors suggest transfer also depends on the underlying patch–text representations.Source: §4.3 · p. 8

4.4 COMPARISON WITH EXTERNAL LOCALIZATION METHODS

  • ProtoLIP beats both fine-grained VLM baselines on all eight comparisons and NACLIP on six of eight, while remaining competitive with MM Grounding DINO despite its spatial supervision and training-data overlap.Against MM Grounding DINO, ProtoLIP is stronger on COCO and VOC20 under both metrics, but trails on ADE20K and Flickr30K.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5 ABLATION STUDY

  • A flat prototype layer improves COCO object localization, semantic-family organization adds the largest gain, and combining area and overlap constraints yields the strongest localization.ProtoLIP adds 2.23M trainable parameters, or 1.48% of the frozen backbone.Source: §4.5 · pp. 9–10, Table 3 · p. 9

5 CONCLUSION

ProtoLIP addresses the mismatch between fine-grained text conditioning and precise object-level evidence by using localized prototypes for exact score decomposition. The appendix describes constructing and auditing the semantic-family taxonomy from training descriptions and their extracted concepts.

5 CONCLUSION

  • ProtoLIP improves query-specific localization and separation without spatial supervision or backbone retraining, and directly builds matching scores from localized prototypes for exact semantic-family and prototype decomposition.The conclusion emphasizes that finer text conditioning alone does not necessarily yield precise object-level evidence.Source: §5 · p. 10
  • Generative AI assisted offline taxonomy construction and manuscript editing, and the authors reviewed and verified all AI-assisted outputs.The taxonomy prompts use sampled training descriptions, not images, evaluation labels, downstream test data, or model outputs.Source: AI USE STATEMENT · p. 11, A LLM PROMPT TEMPLATES FOR TAXONOMY… · p. 14

A LLM PROMPT TEMPLATES FOR TAXONOMY CONSTRUCTION

  • The taxonomy workflow extracts localizable visual concepts from training descriptions, groups concepts into coarse semantic families, and audits assignments against the original inventory.It normalizes lexical variants, excludes nonlocalizable concepts, and checks for missing, duplicate, unsupported, or visually incoherent assignments.Source: A LLM PROMPT TEMPLATES FOR TAXONOMY… · pp. 14–15

B EVALUATION PROTOCOL AND IMPLEMENTATION DETAILS

ProtoLIP is evaluated across object-query datasets and VLM backbones using a fixed caption-linked COCO manifest and a common spatial representation for detector comparisons. Semantic-family routing yields the largest localization gains on matched queries, while the all-prototype fallback largely preserves performance on unmatched queries.

B EVALUATION PROTOCOL AND IMPLEMENTATION DETAILS

  • The prototype layer is a gated two-layer residual MLP, and training uses AdamW for 6,000 steps with a frozen backbone.Prototypes are initialized from similar training patch features, then optimized as learnable parameters; reported results are averaged over three runs.Source: B EVALUATION PROTOCOL AND… · pp. 15–16
  • MM Grounding DINO detections are converted to normalized 64 × 64 heatmaps for Pointing and Energy, providing a common spatial representation for comparison.The comparison is against a spatially supervised grounding model, whereas ProtoLIP uses no spatial annotations and keeps its VLM backbone frozen.Source: B EVALUATION PROTOCOL AND… · p. 16
  • The caption-linked COCO manifest contains 681 object queries across 557 images, with matched-category instances combined into union targets.Pointing tests whether the top evidence patch falls inside any instance box; Energy measures evidence mass within the union.Source: B EVALUATION PROTOCOL AND… · p. 16

B.1 ADDITIONAL COCO-80 EVALUATION

  • ProtoLIP improves both Pointing and Energy over frozen ItemizedCLIP on the complete COCO-80 category set.This evaluation complements the caption-linked COCO object-query benchmark.Source: §B.1 · p. 16, Table 4 · p. 17

C NATURAL FALLBACK AND SEMANTIC-FAMILY COVERAGE

  • Matched queries show the largest localization gains, while the all-prototype fallback largely preserves unmatched-query localization on ADE20K and Flickr30K Entities.The evaluation separates unique-query coverage from sample coverage and reports Pointing and Energy improvements over frozen ItemizedCLIP.Source: C NATURAL FALLBACK AND SEMANTIC-FAMILY… · pp. 16–17, Table 5 · p. 17

C.1 BEHAVIOR ON NATURALLY UNMATCHED QUERIES

Naturally unmatched queries use the complete prototype bank and retain AUC and balanced accuracy close to matched queries. However, matched and unmatched queries are different semantic populations, and separate comparisons suggest a modest strict top-1 retrieval cost associated with family-constrained routing.

C.1 BEHAVIOR ON NATURALLY UNMATCHED QUERIES

  • Naturally unmatched queries retain AUC and balanced accuracy close to matched queries, but these semantically different populations support descriptive rather than causal comparisons.Standard inference and forced fallback are identical for naturally unmatched queries because they already use the complete prototype bank.Source: §C.1 · pp. 17–18
  • Forcing matched queries to use the complete prototype bank leaves balanced accuracy essentially unchanged while increasing I@1.This is a controlled intervention on routing for matched queries.Source: §C.1 · p. 17
  • Family-constrained routing carries a consistent, modest strict I@1 cost relative to the frozen baseline, despite leading on AUC, BAcc, and broader retrieval cutoffs.The passage relates this comparison to a matched-query intervention and describes the cost as traded for localization, separation, and matching gains.Source: §C.1 · p. 18
  • The fallback prevents unmatched queries from being discarded or assigned to arbitrary semantic families.Unmatched queries already use the full prototype bank during standard inference.Source: §C.1 · pp. 17–18

D TAXONOMY GRANULARITY ANALYSIS

The 39-family taxonomy produces modest matching, retrieval, and localization changes relative to the 23-family taxonomy on the same held-out evaluation protocol. Its small advantage in localization and broader semantic coverage motivates using K39 as the main taxonomy.

D TAXONOMY GRANULARITY ANALYSIS

  • The comparison retrains only the ProtoLIP layer and evaluates both taxonomies on the same held-out Itemized-CC0.3M split and 1,200-pair evaluation set.The compared taxonomies contain 23 and 39 semantic families.Source: D TAXONOMY GRANULARITY ANALYSIS · p. 18, Table 7 · p. 18
  • K39 yields modestly stronger spatial localization than K23, with differences within one point on all reported metrics; it is adopted for its modest localization benefit and broader semantic coverage.Matching and retrieval changes are also modest, and BAcc remains essentially unchanged.Source: Table 8 · p. 18, D TAXONOMY GRANULARITY ANALYSIS · p. 18
  • K39 slightly improves AUC and several retrieval cutoffs over K23, while balanced accuracy remains essentially unchanged.The small matching and localization gap indicates results are not sensitive to this taxonomy-granularity choice.Source: D TAXONOMY GRANULARITY ANALYSIS · p. 18, Table 7 · p. 18

E PARAMETER BUDGET

The prototype variants have identical trainable parameter budgets, so the reported improvements from routing and spatial regularization do not arise from increased model capacity.

E PARAMETER BUDGET

  • The parameter-budget analysis tests whether improvements in the main ablation could be explained by differences in trainable capacity.The reported comparison is presented in Table 9.Source: E PARAMETER BUDGET · p. 18, Table 9 · p. 19
  • Flat Prototype, Family Prototype, and Full ProtoLIP use identical trainable parameter budgets, ruling out increased capacity as the source of their reported improvements.The backbone remains frozen, and the prototype variants each add 2.23M trainable parameters, or 1.48% of the backbone.Source: E PARAMETER BUDGET · p. 18, Table 9 · p. 19
  • Family routing uses fixed masks, while the area and overlap objectives add no trainable parameters.The parameter-budget comparison covers the prototype ablation variants.Source: Table 9 · p. 19

F SAME-DATA IN-DOMAIN CONTROLS

Same-data controls test whether ProtoLIP’s matching gains persist when pretraining data are held constant. ProtoLIP leads on AUC and balanced accuracy, though ItemizedCLIP retains strict Recall@1.

F SAME-DATA IN-DOMAIN CONTROLS

G MATCHED-DATA BACKBONE TRANSFER AND PATCH–TEXT ALIGNMENT

Matched-data transfer varies by backbone, and patch–text alignment tracks matching performance. Localization and matching transfer can diverge, leaving the determinants of localization transfer unresolved.

G MATCHED-DATA BACKBONE TRANSFER AND PATCH–TEXT ALIGNMENT

  • Transfer remains backbone-dependent across frozen backbones trained on Itemized-CC0.3M under the full evaluation protocol.Table 11 covers COCO object queries and full validation splits for ADE20K and VOC20.Source: G MATCHED-DATA BACKBONE TRANSFER AND… · p. 19, Table 11 · p. 20
  • Patch Top-8 AUC near 0.5 and Patch I@1 near 0.0025 indicate little text-discriminative information in DreamLIP and Multi-positive SigLIP patch tokens; FLAIR scores 0.782 AUC and 0.308 Patch I@1.The diagnostic uses 400 held-out image–text pairs; Patch Top-8 AUC measures pairwise discrimination and Patch I@1 measures image-to-text retrieval.Source: G MATCHED-DATA BACKBONE TRANSFER AND… · p. 19, Table 12 · p. 20
  • +0.243 Pointing for Multi-positive SigLIP-0.3M accompanies −0.177 retrieval, its largest loss; DreamLIP-0.3M degrades on both.Table 13 summarizes average changes from adding ProtoLIP across COCO object, ADE20K full validation, and VOC20 full validation.Source: G MATCHED-DATA BACKBONE TRANSFER AND… · p. 20, Table 13 · p. 21
  • Patch–text alignment tracks matching but does not separate DreamLIP-0.3M from Multi-positive SigLIP-0.3M (0.513 vs 0.496), whose localization changes in opposite directions; localization-transfer determinants remain for future work.Matching performance is retained only where patch–text alignment is present, and disabling family routing does not recover lost matching performance.Source: G MATCHED-DATA BACKBONE TRANSFER AND… · pp. 20–21

Related papers

Loading 2609.16284v2…