PaperlayerView exact revision on arXiv

Source-linked AI summary

Half-Truths Break Similarity-Based Retrieval

Bora Kargi, Arnas Uselis, Seong Joon Oh

arXiv:2602.23906v1cs.CV

TL;DR

CLIP-style dual encoders can reward plausible but incorrect additions to otherwise correct image descriptions, revealing weak supervision on caption parts. CS-CLIP adds entity- and relation-level foil supervision during fine-tuning while preserving standard dual-encoder inference. It reaches 69.3% Half-Truth Accuracy and improves average performance across 16 compositional benchmarks by 5.7 percentage points over CLIP.

  • Problem

    CLIP-style dual encoders often increase similarity when a plausible but incorrect entity or relation is appended, because contrastive training weakly supervises individual caption units.

  • Method

    CS-CLIP parses captions into entity and relation units, contrasts each correct unit with a minimally edited foil during fine-tuning, and preserves standard dual-encoder inference.

  • Results

    CS-CLIP reaches 69.3% Half-Truth Accuracy and improves average performance across 16 compositional benchmarks by 5.7 percentage points over CLIP.

  • Takeaways & Limitations

    Targeted supervision on caption parts makes retrieval more sensitive to compositional changes while retaining the standard dual-encoder architecture.

  • Takeaways & Limitations

    The approach relies on text-only LLM parsing and COCO fine-tuning, which may miss visual details and trade some zero-shot accuracy for compositional sensitivity.

Abstract

from arXiv · show

When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP

1. Introduction

CLIP-style dual encoders can assign higher similarity to descriptions containing plausible but incorrect entity or relation details. CS-CLIP addresses this half-truth vulnerability with unit-level supervision and improves both diagnostic and compositional performance.

  • Problem: Adding a plausible but incorrect detail can increase image-text similarity instead of decreasing it.The paper formalizes this failure as the half-truth vulnerability.
  • Diagnostic: 40.6% overall Half-Truth Accuracy and 33.2% for relation additions show that CLIP often fails to penalize incorrect details.NegCLIP reaches 56.5% overall but 48.3% for relation additions; SigLIP and SigLIP2 reach 45.7% and 54.6% overall.
  • Mechanism: Contrastive training aligns full captions but provides weak supervision for individual entities and relations, allowing coarse overlap to dominate similarity.This weakness is especially consequential for relations and role-sensitive structure.
  • Method: CS-CLIP parses captions into entity and relation units, contrasts each correct unit with a minimally edited foil, and retains standard dual-encoder inference.The unit-level supervision is applied during fine-tuning rather than at test time.
  • Results: 69.3% Half-Truth Accuracy is achieved by CS-CLIP, compared with 40.6% for CLIP and 56.5% for NegCLIP.The model correctly penalizes incorrect additions in the diagnostic.
  • Results: 5.7 percentage points of improvement over CLIP is reported across 16 compositional benchmarks.CS-CLIP also achieves the best average Image-to-Text accuracy at 57.8% and the best Group Accuracy among evaluated models.

2. Related Work

Prior work improves compositional sensitivity through hard negatives, perturbations, synthesis, or architectural changes. CS-CLIP complements these approaches by adding targeted supervision on entity and relation units while preserving the standard dual-encoder architecture.

  • Contrastive pretraining: CLIP and ALIGN support efficient retrieval through a shared image-text embedding space and a single similarity score.Prior analyses find that CLIP-style models can under-use linguistic structure under minimal meaning-changing edits.
  • Prior methods: Prior methods use sentence-level hard negatives, scene-graph perturbations, or caption synthesis to strengthen compositional sensitivity.Other approaches modify the model architecture.
  • Positioning: CS-CLIP parses entity and relation units and contrasts each with a minimally edited foil, adding direct compositional pressure without changing the standard dual-encoder architecture.This approach builds on sentence-level negatives with targeted unit-level supervision.

3. The Half-Truth Vulnerability

The half-truth diagnostic tests whether models penalize a plausible but incorrect detail appended to a correct description. CLIP-style models often fail, especially when the added detail expresses a relation.

  • Motivation: The vulnerability reflects weak supervision on individual caption units despite strong full-sentence alignment.Coarse overlap can dominate similarity, particularly when verifying relations and role-sensitive composition.
  • Diagnostic construction: Half-truths append one plausible but incorrect entity or relation detail to a correct anchor description.The anchor describes an image-supported entity, while the appended unit introduces exactly one targeted error.
  • Diagnostic results: 52.9% entity accuracy and 32.9% relation accuracy show that standard CLIP often fails to reject incorrect additions.Relation additions fall below the 50% random-choice level, so incorrect relations are frequently preferred over the anchor.
  • Diagnostic results: 69.8% entity accuracy and 48.3% relation accuracy show that NegCLIP improves entity rejection but leaves relations near chance.SigLIP reaches 56.9% entity and 38.8% relation accuracy, while SigLIP2 reaches 69.9% and 45.2%, respectively.
  • Motivation: Unit-level supervision is motivated by the diagnostic’s single-unit edit, which isolates the specific detail that changed.The paper therefore contrasts each correct unit with a minimally edited foil rather than relying only on sentence-level negatives.

4. Method: CS-CLIP

CS-CLIP fine-tunes CLIP with explicit supervision on entity and relation units against matched foils while retaining global caption alignment. Its unit loss teaches image embeddings to prefer correct units over foils and other in-batch units.

  • Unit extraction and sampling: CS-CLIP parses captions into entity and relation units and generates a minimally edited matched foil for each unit.Entity foils can change objects or attributes, while relation foils can change predicates, arguments, or entities.
  • Unit-level loss: The unit-level loss uses normalized image and text embeddings with cosine similarity and contrasts correct units against matched foils and in-batch units.A symmetric unit-to-image term provides bidirectional regularization, while foils are excluded from that symmetric term because each is tied to a specific image-unit pair.
  • Unit-level supervision: The training pipeline samples one unit–foil pair per image, encodes the image and unit texts, and applies unit-level supervision.The unit loss pulls the image embedding toward the correct unit and pushes it away from its foil and other in-batch units.
  • Training objective: The final objective combines global image-caption contrastive training with unit-level supervision.Sentence-level hard negatives are retained in parallel, preserving full-caption alignment while adding compositional pressure.
  • Inference and design: CS-CLIP preserves standard dual-encoder inference and cosine scoring while making similarity more sensitive to caption structure.The model is trained to distinguish correct caption units from matched foils without directly training on anchor-versus-half-truth comparisons.

5. Experiments

Experiments show that CS-CLIP improves half-truth robustness and compositional benchmark performance through unit-level supervision, while preserving competitive downstream retrieval and classification.

  • 5.1. Half-Truth Vulnerability on COCO: 69.3% overall AccHT and a +0.017 mean similarity gap make CS-CLIP the strongest evaluated method for penalizing incorrect additions.It improves over zero-shot CLIP’s 40.6% AccHT by 28.7 percentage points.
  • 5.1. Half-Truth Vulnerability on COCO: 75.4% entity-addition accuracy ties FSC-CLIP, while CS-CLIP reaches 65.5% relation accuracy versus FSC-CLIP’s 47.9%.Relation additions remain harder, but CS-CLIP substantially improves performance in this category.
  • 5.2. Compositional Understanding: 57.8% compositional I2T and 69.3% AccHT are the best results across models, with the two metrics positively correlated.This pattern is reported as evidence that the gains extend beyond the half-truth setup.
  • 5.2. Compositional Understanding: CS-CLIP improves over NegCLIP on 14/16 compositional benchmarks and achieves the best VL Checklist score of 79.2%.The method uses additional unit-level supervision without changing the dual-encoder architecture or test-time scoring.
  • 5.3. Downstream Performance: CS-CLIP improves average T2I and Group Accuracy relative to CLIP and NegCLIP, while achieving the best average Group Accuracy.COCO fine-tuning modestly lowers zero-shot classification accuracy from 63.6 to 59.9 at Acc@1 and from 86.5 to 84.6 at Acc@5.
  • 5.4. Ablations: Full dual-encoder fine-tuning is important for relations, and increasing λu mainly improves Half-Truth accuracy without disrupting compositional or downstream metrics.Text-only and image-only fine-tuning reach 41.9 and 49.3 relation accuracy, versus 65.5 for full fine-tuning; relation accuracy rises from 61.3% to 67.8% as λu increases from 0.10 to 0.75.
  • 5.4. Ablations: Matched unit foils produce the largest isolated Half-Truth improvement, while combining global negatives, unit supervision, and foils yields the best compositional I2T and Half-Truth results.The combined variant reaches 57.8 compositional I2T, 69.3 overall AccHT, and 65.5 relation AccHT.

6. Conclusion

The paper identifies half-truth failures in CLIP-style retrieval and addresses them with unit-level supervision over entity and relation foils. CS-CLIP improves half-truth robustness and compositional matching, while remaining subject to parsing, data, fairness, and zero-shot limitations.

  • 6. Conclusion: CS-CLIP contrasts entity and relation units with minimally edited foils during fine-tuning while retaining the standard dual-encoder architecture.This directly targets compositional changes in caption parts.
  • 6. Conclusion: 69.3% Half-Truth Accuracy, 57.8% compositional I2T accuracy, and the highest Group Accuracy are reported for CS-CLIP.The paper attributes these results to targeted supervision on caption parts.
  • 6. Conclusion: The approach relies on text-only LLM parsing, may miss visual details absent from captions, and trades some zero-shot accuracy for compositional sensitivity.It also does not guarantee factual correctness or demographic fairness.

Impact Statement

The work aims to make vision-language retrieval more reliable when users refine queries with additional details, while recognizing that broader safety properties remain unresolved. Its text-only pipeline extracts grounded entities and relations, generates minimally edited foils, and filters malformed or semantically redundant outputs.

  • Impact: The intended impact is more reliable retrieval under query refinement for search, accessibility tools, and dataset curation.The goal is to reduce spuriously confident matches caused by incorrect added details.
  • Method: CS-CLIP adds unit-level supervision while retaining standard dual-encoder inference and cosine scoring at test time.Global caption alignment is combined with supervision on individual caption units and matched foils.
  • Unit extraction: The pipeline parses captions into entity noun phrases and directed relations grounded in those entities.Entity units retain bound attributes and quantifiers; relation arguments must match extracted entity strings exactly.
  • Foil generation: Matched foils change one entity or relation component while preserving fluency, context, and the unit’s changed meaning.Entity edits alter objects or attributes; relation edits alter predicates, arguments, or asymmetric roles.
  • Filtering: Rule-based filters remove duplicates, near-synonyms, malformed outputs, multi-edit strings, and invalid symmetric relation swaps.These filters help preserve the intended minimal-edit contrast.

B.7. Usage in Training and Evaluation

The same unit-and-foil pipeline supports both CS-CLIP training and half-truth evaluation. Training samples one caption unit and contrasts it with a matched foil, while evaluation appends a foil from another unit to an anchor.

  • Training: For each image-caption pair, CS-CLIP samples one entity or relation unit and one matched foil for training.The sampled unit comes from the union of extracted entity and relation units.
  • Evaluation: The half-truth diagnostic appends exactly one foil from a different unit to an anchor entity unit.This construction tests whether an incorrect added detail increases similarity.

C. Half-Truth Diagnostic: Construction and Additional Results

The diagnostic constructs nearly identical truthful and half-truth refinements by replacing one caption-supported addition with a minimally edited foil. It separates entity and relation corruptions and complements anchor-versus-half-truth evaluation with truthful-completion comparisons.

  • C.1. Construction and Notation: Each evaluation caption is parsed into grounded entity units and directed relation triplets, with matched minimal foils constructed for both types.Entities retain attributes and quantifiers, while relation arguments refer to extracted entities.
  • C.2. Half-Truth Construction: Half-truth refinements preserve the anchor and most tokens while replacing exactly one truthful added unit with a matched foil.This isolates verification of the added content rather than topic drift.
  • Entity half-truths: Entity corruptions include wrong objects, wrong attributes, and random distractors added to an otherwise fixed anchor.The taxonomy is +Obj, +Attr, and +Rand.
  • Relation half-truths: Relation corruptions include predicate changes, role swaps, argument changes, and attribute or object edits inside relation arguments.These are grouped as Ant, Swap, Subj/Obj, and Attr/Obj conditions.
  • Additional Results: 69.3% Half-Truth Accuracy is achieved by CS-CLIP, while corruption breakdowns show relation edits are harder for many baselines.Random distractor entities are comparatively easier, whereas attribute/object and relation edits require tighter binding.
  • C.3. Truthful Completion vs. Matched Foil: The truthful-completion metric measures whether similarity prefers a correct added unit over its minimally edited foil, rather than whether it beats the shorter anchor.Many models can distinguish longer completions yet still fail anchor-versus-half-truth constraint monotonicity; CS-CLIP improves both diagnostics.
  • Qualitative Examples: Figures 8 and 9 organize half-truths by entity and relation corruption types and compare CLIP, NegCLIP, and CS-CLIP scores.CS-CLIP more consistently lowers similarity for incorrect refinements in the illustrated examples.

D.3. Benchmark Subset Mapping

The benchmark mapping organizes compositional evaluations by capability, while the evaluation protocol measures ranking accuracy for correct captions, images, or matched groups. Across 16 benchmarks, CS-CLIP reaches the highest average performance, with gains alongside a dataset-overlap caveat.

  • D.3. Benchmark Subset Mapping: The mapping groups benchmark subsets into relational structure, attribute binding, linguistic syntax, coreference, and negation capabilities.Relational subsets include predicate and role sensitivity; binding subsets include attribute and multi-object tests.
  • Results: 57.8% average I2T accuracy is achieved by CS-CLIP across 16 compositional benchmarks, with the highest overall average.It performs strongly on tight binding and spatial reasoning subsets.
  • Evaluation Protocol: I2T accuracy is the fraction of images where the correct caption ranks above all foil captions.With one foil, the metric becomes a pairwise win rate.
  • Evaluation Protocol: Group Accuracy requires the matched image-caption pair to receive the highest similarity over mismatched pairs in both directions.This evaluates consistency across groups of matched pairs.
  • Results: COCO fine-tuning improves compositional robustness, but overlap with COCO-derived benchmarks creates favorable evaluation conditions.Improvements also appear on benchmarks with independent image sources, suggesting gains beyond memorization.
  • Results: Performance remains weaker on spatial relations and group-level consistency than on color attributes.Reported ranges are 39–44% for What’sUp, 4–16% for MMVP, 20–33% for Winoground, and 84–93% for ColorFoil.

E.4. Per-Capability Results

CS-CLIP improves compositional sensitivity across capability-level, bidirectional retrieval, and standard vision-language evaluations. Its strongest gains concern role-sensitive matching, while counting, negation, and fine-grained attributes remain challenging.

  • Per-capability results: 40.1% Role Sensitivity is CS-CLIP’s best capability-level score, exceeding CLIP’s 34.6% and matching or exceeding other COCO-trained methods.CS-CLIP also ranks second on Attribute Binding, Object Recognition, Existence, and Syntax.
  • Remaining weaknesses: 24–39% counting and 26–37% negation accuracy reveal systematic weaknesses, while Object Recognition reaches the highest absolute performance at 55–72%.These results indicate stronger entity-level grounding than relational or linguistic understanding across models.
  • Training-data comparison: 47.6% average capability performance for CS-CLIP trained on COCO exceeds FSC-CLIP’s 44.5% despite FSC-CLIP using 30× more data.The comparison suggests compositional robustness depends more on alignment-signal quality than raw dataset size.
  • Bidirectional compositional retrieval: 27.3% average Group Accuracy is the best result on seven paired compositional datasets, requiring both I2T and T2I retrieval to succeed.CS-CLIP therefore improves bidirectional matching rather than only one retrieval direction.
  • Standard vision-language tasks: 59.9% Acc@1 and 84.6% Acc@5 preserve competitive zero-shot classification, although CLIP reaches 63.6% and 86.5%.The reported Acc@1 gap is 3.7 points, alongside compositional-suite and Half-Truth Accuracy gains.
  • Standard retrieval: 56.8% I2T and 71.7% T2I retrieval are best among evaluated models, exceeding CLIP by 11.6 and 9.0 points respectively.On Flickr8k, CS-CLIP reaches 67.3% I2T and 81.5% T2I, where captions are more descriptive and compositional structure is more salient.

F. Additional Ablations

Additional ablations show that unit construction and optimization choices trade off Half-Truth Accuracy against retrieval and zero-shot performance. The selected baseline balances relational grounding, compositional sensitivity, and downstream performance.

  • Unit construction choices: 69.8 overall and 66.0 relation Half-Truth Accuracy from N=1 units slightly improve half-truth scores but reduce retrieval to 70.7 I2T.Using 3–4 units yields diminishing returns, as additional units may add noise or redundancy.
  • Unit construction choices: 78.7 entity accuracy with p=0.0 contrasts with 55.2 relation accuracy, while increasing the relation-unit fraction improves relations at the cost of entity performance.The chosen p=1.0 configuration maximizes relational grounding because entity-level understanding is already strong in the CLIP initialization.
  • Foil construction: 71.7 versus 69.4 I2T retrieval favors swap-style over add/replace foils despite similar overall Half-Truth Accuracy.The results support swap operations for teaching role assignments relevant to fine-grained retrieval.
  • Optimization hyperparameters: 73.6 overall and 70.8 relation Half-Truth Accuracy at LR=2×10−5 reduce zero-shot classification to 56.5 and retrieval to 69.0 I2T.The baseline LR=5×10−6 balances 69.3 Half-Truth Accuracy with 59.9 zero-shot and 71.7 retrieval performance.
  • Optimization hyperparameters: ±0.5-point variation across WD=10−3 to WD=5×10−2 shows that weight decay has minor overall effects.The baseline WD=10−2 provides slightly better retrieval performance.
  • Optimization hyperparameters: 70.0 overall and 66.3 relation Half-Truth Accuracy at BS=32 improve half-truth performance but reduce retrieval to 69.8 I2T.Larger batches improve retrieval, likely through more diverse negative samples; BS=128 is selected for the best retrieval performance with competitive other metrics.
Loading 2602.23906v1…