PaperlayerView exact revision on arXiv

Source-linked AI summary

Know-Thy-Not: Training-Free Targeted Negation in Dual-Encoder Vision-Language Models

DoYoung Kim, Changkyu Choi, Adín Ramírez Rivera, YunSeok Choi, Jee-Hyong Lee

arXiv:2610.17440v1cs.CVcs.AI

TL;DR

Dual-encoder VLMs often ignore negation, and the paper argues that single-embedding scoring is structurally limited to diluting rather than rejecting a negated concept. It proposes training-free KTN and COCO-Confusion; across nine backbones, KTN improves negation handling, including by +9.6 mAP over Embedding Correction on COCO-Confusion.

  • Problem

    Dual-encoder VLMs struggle with negation, while a single-embedding representation cannot express the rejection signal needed to penalize a negated concept’s presence.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    KTN is a training-free method that scores queries with separate text-side affirmation and image-side detection and penalization of the negated concept.

    Source: Proposed method · p. 2, §4 · p. 5

  • Results

    Across 9 backbones, KTN improves COCO-Confusion by +9.6 mAP over Embedding Correction.

    Source: Proposed method · p. 2

  • Takeaways & Limitations

    COCO-Confusion makes genuine rejection, rather than merely ignoring negation, necessary for success.

    Source: Proposed method · p. 2

    KTN’s image-side rejection can still produce false positives when the negated concept is absent, despite adaptive thresholding.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Dual-encoder vision-language models systematically fail at negation, scoring ''a cat without a dog'' nearly identically to ''a cat with a dog.'' We show that this failure is inherent to the single-embedding framework. Under contrastive training, single-embedding methods can at best ignore the negated concept (dilution) but never penalize its presence (rejection). We propose Know-Thy-Not (KTN), a training-free method that introduces an image-side rejection dimension to penalize the negated concept directly in the visual representation. We also introduce COCO-Confusion, the first benchmark requiring genuine rejection to succeed. Across 9 backbones, KTN improves COCO-Confusion mAP by up to +25.6 over vanilla VLMs and +9.6 over the strongest existing method.

1 Introduction

Dual-encoder VLMs often ignore negation, undermining reliable retrieval and grounding. The paper argues single-embedding methods can dilute but not reject a negated concept, and introduces KTN and COCO-Confusion to address and measure rejection.

1 Introduction

  • VLMs treat negated queries as if negation were absent; on VOC2007, CLIP scores 82% on affirmative queries but only 3% on negated ones.The paper argues this failure undermines VLMs as reliable retrieval and grounding systems.Source: §1 · p. 1
  • Embedding correction removes the negated concept’s direction from the text embedding, but does not verify whether that concept is present in an image.The paper distinguishes this dilution from rejection: a system must actively penalize the negated concept’s presence.Source: §1 · p. 1, Proposed method · p. 2
  • The paper’s formal analysis says natural visual data drives the optimal text coefficient on a negated concept to zero, bounding approaches that operate only on text embeddings.KTN scores queries along independent text-side affirmation and image-side rejection dimensions.Source: §1 · p. 1, Proposed method · p. 2
  • Across 9 backbones, KTN improves COCO-Confusion by +9.6 mAP over Embedding Correction, on a benchmark designed to require genuine rejection.COCO-Confusion puts the negated concept in every gallery, unlike existing benchmarks where it rarely co-occurs with the affirmed concept.Source: Proposed method · p. 2

2 Preliminaries

The paper frames retrieval as ranking gallery images by similarity to a text query, then defines negation queries and the image sets needed to evaluate faithful retrieval.

2 Preliminaries

  • A dual-encoder VLM maps text and images into ℓ2-normalized embeddings in a shared space and ranks gallery images by their inner product with the query.The encoders map text and images into the same d-dimensional space.Source: §2 · pp. 2–3
  • A negation query combines an affirmed concept c+ with a negated concept c−, typically phrased as “c+ without c−.”The paper uses a binary indicator to denote whether a concept is present in an image.Source: §2 · p. 3
  • The gallery is partitioned into correct images that satisfy c+ and exclude c−, confusing images that contain both, and irrelevant images that fail c+.These are the correct set I+, confusing set I±, and irrelevant set I−.Source: §2 · p. 3
  • Faithful negation retrieval ranks the correct set I+ above I±, the most challenging distractor set.For the example, c+ is “car” and c− is “traffic light.”Source: §2 · p. 3

3 Why single text embeddings cannot reject negation

The paper argues that single text embeddings can dilute a negated concept but cannot reject its visual presence. Its formal analysis and COCO-Confusion example motivate separate image-side rejection.

3.2 A structural barrier in single-embedding negation handling

  • Contrastive training provides no gradient toward the negative coefficient needed for rejection, while post-hoc correction reaches only a zero negated-concept coefficient.The paper concludes that single-embedding approaches are bounded by dilution, not rejection.Source: §3 · p. 3, §3.2 · p. 4

3.1 Existing approaches only dilute, not reject

  • Training-based approaches augment data with negated captions but yield marginal improvements despite substantial retraining cost.Embedding correction instead projects the negated concept direction out of the query embedding.Source: §3.1 · p. 3
  • Embedding correction modifies the text embedding without grounding c− in visual features, so it dilutes the concept rather than penalizing its presence.The paper argues that faithful negation requires rejection, which a conventional single-embedding format cannot express.Source: §3.1 · p. 3

3.2 A structural barrier in single-embedding negation handling

  • Under standard ℓ2-normalized contrastive training, positive alignment gives no signal toward the negative coefficient required to rank correct images above confusing ones.The analysis uses a linear concept model and assumes non-negative concept activations.Source: §3.2 · pp. 3–4
  • With a unit-norm constraint, the optimal negated-concept text coefficient is zero, leaving correct and confusing images tied at the optimum.The paper also says hard-negative supervision can collapse to a generic negation offset rather than concept-specific rejection, and post-hoc correction remains insufficient.Source: §3.2 · p. 4

3.3 The limit of dilution-based approaches

  • ≈0.15: vanilla SigLIP2’s top five results are all confusing; after correction scores fall to ≈0.12, but correct and confusing images remain interleaved.The example uses the query “a car is present but no traffic light” on COCO-Confusion.Source: Figure 3 · p. 5, §3.3 · p. 5
  • The example illustrates the dilution ceiling: KTN instead separates text-side affirmation from an image-side signal that penalizes c−.The paper identifies multi-dimensional scoring as the needed alternative to single-embedding scoring.Source: §3.3 · p. 5

4 Proposed method: Know-Thy-Not

KTN replaces single-embedding negation scoring with separate text-side affirmation and image-side rejection dimensions. The text dimension alone remains limited to dilution, while image-conditioned detection supplies the rejection signal needed to penalize a negated concept’s presence.

4 Proposed method: Know-Thy-Not

  • KTN uses a corrected text embedding to affirm c+ and a separate image-side computation to detect and penalize c− in the image.The image-side score does not pass through the text encoder’s unit-norm budget, so it can supply the rejection signal unavailable to a single dimension.Source: §4 · p. 5

4.1 Text-side affirmation

  • The text-side affirmation step extracts affirmed and negated concepts, then projects the c− direction out of the query embedding.This makes the corrected embedding orthogonal to the negated-concept direction and sets its coefficient to zero, which remains insufficient for rejection.Source: §4.1 · pp. 5–6

4.2 Image-side rejection

  • Image-side rejection computes a relative c− detection score against a per-image reference anchored to affirmed-concept similarity.The score uses soft thresholding and discounts cases where semantic correlation between c+ and c− could otherwise create a spurious penalty.Source: §4.2 · p. 6

4.2.1 Why KTN avoids the constraint

  • A separate scoring axis avoids the text-side unit-norm bound, and the matched-pair test requiring a negative ranking signal is satisfied by Dimension 2.The method divides affirmation, which receives positive data signal, from rejection, for which direct image-side detection supplies the missing signal.Source: §4.2.1 · p. 6

5 Experiments

The experiments introduce COCO-Confusion to test rejection of images containing negated concepts, then evaluate KTN across multiple VLM backbones and tasks. KTN improves confusion retrieval, and ablations attribute substantial gains to its image-side rejection dimension.

5.1 Introducing COCO-Confusion: A benchmark for negation rejection

  • COCO-Confusion fills galleries with images containing the negated concept, making it possible to distinguish rejection from dilution in a way unfiltered benchmarks cannot.The benchmark pairs correct images containing c+ but not c− with confusing images containing both; it includes 950 2C pairs and 1,740 3C pairs, for 2,690 verified pairs total.Source: §5.1 · pp. 6–7

5.2 Experimental setup

  • Nine frozen pretrained VLMs are tested on MCQ, standard retrieval, and COCO-Confusion retrieval; baselines include Raw VLM and Embedding Correction, plus NegCLIP and NegationCLIP for CLIP ViT-B/32.These tasks measure negation comprehension, handling in natural galleries where confusing images are rare, and rejection, respectively.Source: §5.2 · p. 7

5.3 Main results

  • On CLIP ViT-B/32, KTN reaches 82.8% mAP on Conf-2C and 75.2% on Conf-3C, improving over Embedding Correction by +9.6 and +4.4 points.Across backbones, KTN reaches 81–87% on Conf-2C and 74–79% on Conf-3C, with a best result of 87.1% mAP on SigLIP2 SO400M.Source: Table 2 · p. 8, §5.3 · p. 8
  • Across backbones, MCQ accuracy improves over the raw baseline by +42 to +57 points on COCO, +51 to +71 on VOC2007, and +31 to +37 on MSRvtt.Standard retrieval preserves R@5 while R-Neg@5 gains range from +1.5 to +4.7 on COCO and +1.5 to +4.2 on MSRvtt.Source: §5.3 · p. 8

5.4 Ablation study

  • On SigLIP2 SO400M, adding Dimension 2 (KTN) beyond Proj adds +25.9pp on MCQ and +8.5pp on COCO-Confusion; the standard retrieval R-Neg@5 gap is +1.3pp.The ablation identifies the image-side rejection dimension as the dominant source of improvement, while dilution alone suffices more often in galleries with few confusing images.Source: §5.4 · pp. 8–9, Table 3 · p. 9

6 Related work

Prior work documents negation failures in VLMs and explores retraining or text-embedding correction. These approaches have limitations that motivate adding a separate rejection dimension.

6 Related work

  • Benchmarks report near-chance negation performance; on VOC2007, CLIP scores 82% on affirmative queries but only 3% on negated ones.Prior accounts attribute failures to bag-of-words behavior, sparse negation in training captions, or limits of joint embedding spaces.Source: §6 · p. 9
  • Training-based methods such as ConCLIP [16] and NegationCLIP retrain on negation captions, while embedding correction projects the negated concept direction out of the text embedding.The cited work reports limited generalization for retraining approaches and describes embedding correction as reducing bias through dilution rather than rejection.Source: §6 · p. 9

7 Conclusion

The conclusion argues that single-embedding scoring is bounded to dilution and presents KTN and COCO-Confusion as a method and benchmark for rejection. Across nine backbones, experiments show improvements across comprehension, retrieval, and confusion rejection.

7 Conclusion

  • Across nine backbones, experiments show consistent improvements in MCQ, negation retrieval, and confusion rejection, while COCO-Confusion makes rejection measurable beyond dilution.The conclusion attributes the single-embedding limit to the non-negative nature of visual concept activations, which drives the negated-concept text coefficient to zero.Source: §7 · p. 9

Limitations

The analysis argues that single-embedding negation is structurally limited to dilution, while KTN’s image-side rejection still risks false positives and has a stated scope boundary. It further explains how absent-concept margins are compressed and why generic offsets cannot provide concept-specific rejection.

Limitations

  • KTN’s cosine-based image-side rejection can score the negated concept highly even when absent, and adaptive per-image thresholding does not fully remove false positives.The method targets dual-encoder VLMs that score through a single dot product; extending it to generative VLMs is identified as a next step.Source: Limitations · p. 9

A.1 Per-image budget compression

  • The optimal negated-concept coefficient reaches zero, and each absent concept’s margin is compressed at O(1/|C|), regardless of training objective.The analysis attributes this to optimization and representation geometry, with the number of visually present concepts typically much smaller than the vocabulary.Source: §A.1 · p. 12

A.2 Hard-negative supervision and frequency collapse

  • A query-independent generic offset penalizes the negated concept and unmentioned allowed concepts alike, so it cannot provide concept-specific rejection.The analysis links hard-negative frequency collapse to global conservatism: conventional dual encoders dilute negated concepts rather than specifically rejecting them.Source: §A.2 · p. 12

B Implementation details

Implementation uses shared hyperparameters and specified procedures for concept embeddings, negation parsing, and image or video preprocessing. The section also describes the embedding-correction reimplementation and introduces the detailed COCO-Confusion construction appendix.

B.1 Hyperparameter settings

  • All KTN hyperparameter values are fixed across backbones and tasks, with N set to 4 for multiple-choice questions and |G| for retrieval.No hyperparameter is tuned per backbone or dataset.Source: §B.1 · p. 12, Table 4 · p. 13

B.2 Concept embedding

  • Each concept embedding is the L2-normalized average of four prompt-template embeddings.The templates vary the phrasing around the concept, including “a photo of a {c}” and “an image showing a {c}.”Source: §B.2 · p. 13

B.3 Negation parsing

  • Negation parsing identifies cues by dependency arc or cue-set membership, assigns the cue subtree’s head noun to c−, and uses remaining root-attached noun phrases as C+.The negation-stripped text removes the cue subtree; for “no traffic light,” the negated concept is “traffic light” and “car” is affirmed.Source: §B.3 · p. 13

B.4 Image and video preprocessing

  • Images are encoded alongside horizontal mirrors and combined by an L2-normalized mean; MSR-VTT videos use four evenly spaced frames, likewise mean-pooled after normalization.Image preprocessing follows the standard CLIP resize, center-crop, and normalize pipeline at each backbone’s input resolution.Source: §B.4 · p. 13

C Embedding correction reimplementation

  • The Embedding Correction baseline is reimplemented by projecting the negated-concept direction out of the query embedding with λ = 1.9, anchored to a neutral text embedding.The cited prior work reports only CLIP ViT-B/32 results and does not release code.Source: C Embedding correction reimplementation · p. 13

D COCO-Confusion: Construction details

D.1 Motivation: Why a new benchmark is neeeded

COCO-Confusion addresses the inability of unfiltered retrieval galleries to distinguish negation dilution from genuine rejection. It constructs matched correct and confusing image sets, verifies labels, and evaluates retrieval across those pairs.

D.1 Motivation: Why a new benchmark is neeeded

  • Unfiltered galleries rarely contain images with both affirmed and negated concepts, so dilution can perform competitively; COCO-Confusion instead tests whether methods separate matched correct and confusing images.Each query’s gallery contains images with the affirmed concept, partitioned by whether the negated concept is present.Source: §D.1 · p. 14

D.2 Concept pair selection

  • COCO-Confusion selects two- and three-object partitions from COCO val2017, retaining pairs with at least 10 images per subset and sampling up to 50 from each.Coverage is 76 of 80 categories for 2-object pairs and 62 of 80 for 3-object pairs; omitted categories lack sufficient co-occurrence for matched-pair construction.Source: §D.2 · p. 14, Table 5 · p. 14

D.3 OWLv2 verification

  • OWLv2 verifies image labels using a 0.1 confidence threshold; pair-level retention is 99.9% for 2-object pairs and 97.1% for 3-object pairs.The low threshold favors recall for partially visible negated objects, and filtering primarily refines image lists rather than discarding pairs.Source: §D.3 · p. 15, Table 6 · p. 15

D.4 Evaluation protocol

  • The benchmark computes mAP within each query’s verified correct-and-confusing gallery, then macro-averages per-pair mAP across 950 Conf-2C or 1,740 Conf-3C pairs.Source: §D.4 · p. 15

D.5 Qualitative examples

  • The qualitative examples show verified correct images in the top row and confusing images in the bottom row for both two- and three-object queries.The examples contrast cat without dog and apple plus banana without bowl.Source: §D.5 · p. 15

D.6 What makes COCO-Confusion different

COCO-Confusion uses curated galleries with a high density of confusing images and cross-verifies labels, making it a rejection stress test. Its examples hold affirmed concepts constant while varying the negated concept.

D.6 What makes COCO-Confusion different

  • COCO-Confusion galleries contain 20–50% confusing images, compared with fewer than 1% in standard retrieval, and use OWLv2 to double-verify labels.Standard retrieval remains useful for measuring dilution-level negation handling, while COCO-Confusion stresses rejection.Source: §D.6 · p. 15
  • In the cat-without-dog example, all ten images contain a cat; compare rows by dog presence to see whether a method separates correct from confusing images.Source: Figure 4 · p. 16
  • In the apple-and-banana-without-bowl example, all shown images contain apple and banana, while only the bottom row also contains a bowl.Source: Figure 5 · p. 16

Related papers

Loading 2610.17440v1…