Source-linked AI summary
Vision-Language Models Do Not Understand Negation
Kumail Alhamoud, Shaden Alshammari, Yonglong Tian, Guohao Li, Philip Torr, Yoon Kim, Marzyeh Ghassemi
TL;DR
Vision-language models are widely used in multimodal applications, but their ability to understand negation is underexplored despite its practical importance. The paper introduces NegBench and synthetic negation datasets to evaluate and improve this capability, finding substantial failures and reported gains after fine-tuning.
Problem
The paper asks how well current VLMs understand negation, a capability needed for queries specifying both present and absent objects or conditions.
Method
The paper introduces NegBench for retrieval and multiple-choice evaluation, then fine-tunes CLIP-based models on large-scale synthetic negation datasets.
Results
VLMs frequently confuse affirmative and negated captions, while fine-tuning yields a 10% increase in recall on negated queries and a 40% boost in multiple-choice accuracy.
Takeaways & Limitations
Negation remains a significant weakness of current VLMs, and synthetic negation data can improve performance on the evaluated tasks.
Takeaways & Limitations
CLIP training data lacks explicit negation examples, while ConCLIP training overfits to a single hybrid linguistic template, limiting generalization across negation structures.
Abstract
from arXiv · showhide
Many practical vision-language applications require models that understand negation, e.g., when using natural language to retrieve images which contain certain objects but not others. Despite advancements in vision-language models (VLMs) through large-scale training, their ability to comprehend negation remains underexplored. This study addresses the question: how well do current VLMs understand negation? We introduce NegBench, a new benchmark designed to evaluate negation understanding across 18 task variations and $79$k examples spanning image, video, and medical datasets. The benchmark consists of two core tasks designed to evaluate negation understanding in diverse multimodal settings: Retrieval with Negation and Multiple Choice Questions with Negated Captions. Our evaluation reveals that modern VLMs struggle significantly with negation, often performing at chance level. To address these shortcomings, we explore a data-centric approach wherein we finetune CLIP models on large-scale synthetic datasets containing millions of negated captions. We show that this approach can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions.
1. Introduction
The paper introduces NegBench to evaluate negation understanding in vision-language models through retrieval and multiple-choice tasks. It finds that joint-embedding VLMs often confuse affirmative and negated statements, while synthetic negation data improves performance.
- Benchmark motivation: NegBench evaluates VLM negation understanding using coarse-grained retrieval and fine-grained multiple-choice tasks.The benchmark targets multimodal applications involving both presence and absence criteria.
- Benchmark motivation: Retrieval-Neg tests whether models retrieve images satisfying affirmative and negative criteria, such as beaches with no people.It serves as the coarse-grained retrieval component, including top-5 retrieval scenarios.
- Benchmark motivation: MCQ-Neg asks models to choose among closely related captions that differ in which elements are affirmed or negated.The task is designed to expose subtle failures, including opposite medical diagnoses expressed through similar language.
- Findings: Joint-embedding VLMs frequently collapse affirmative and negated captions into similar embeddings, treating “a dog” and “no dog” as nearly indistinguishable.This affirmation bias was not sufficiently addressed by earlier benchmarks such as CREPE and CC-Neg.
- Data-centric solution: 10% increase in recall on negated queries and 40% boost in accuracy on multiple-choice questions followed fine-tuning CLIP-based models on synthetic negation datasets.The approach uses CC12M-NegCap and CC12M-NegMCQ to improve negation comprehension.
2. Related Work
Related work shows that negation remains insufficiently evaluated in vision-language models. NegBench broadens evaluation beyond rigid templates by using natural-sounding negated captions across diverse multimodal domains.
- Negation understanding: Prior language-model studies report sub-optimal negation understanding, while this work identifies a more severe affirmation bias in VLMs.The paper describes VLMs as failing to differentiate affirmative from negative captions.
- Existing benchmarks: Existing VLM negation benchmarks such as CREPE and CC-Neg provide limited assessments because they rely on rigid linguistic templates.The paper argues that these templates do not reflect the varied forms of negation in natural user queries.
- NegBench contribution: NegBench uses LLM-generated natural-sounding captions spanning broader negation types and contexts across images, videos, and medical datasets.This design aims to uncover challenges and failure cases not fully addressed by prior benchmarks.
3. The Negation Benchmark (NegBench)
NegBench transforms image, video, and medical datasets into evaluations of affirmative and negative concepts. Its retrieval and multiple-choice tasks use generated negations, paraphrases, hard negatives, and domain-specific controls to test fine-grained understanding.
- Evaluation tasks: NegBench evaluates coarse-grained retrieval and fine-grained question answering across multiple tasks.Retrieval requires matching inclusions and exclusions, while question answering selects among closely related descriptions.
- Dataset construction: The benchmark adapts COCO, VOC2007, MSR-VTT, and CheXpert by identifying present positive elements and absent negative elements.This covers images, video, and specialized medical imaging domains.
- Retrieval-Neg: Retrieval-Neg inserts negated statements into captions and uses LLaMA 3.1 paraphrases to increase linguistic diversity.Negated clauses may appear before or after the original caption.
- MCQ-Neg: MCQ-Neg generates affirmation, negation, and hybrid templates for image descriptions.Each question contains one correct answer and three hard-negative distractors based on presence or absence of elements.
- Medical evaluation: CheXpert evaluation tests affirmative and negated descriptions of Lung Opacity in separate binary classification tasks.The negation task distinguishes whether the medical condition is present or absent.
- Retrieval findings: 23.0% drop in recall@5 was observed for NegCLIP on HardNeg-Syn when captions were negated rather than original.HardNeg-Syn contains hard negatives requiring stronger negation reasoning.
- Synthetic data: HardNeg-Syn synthesizes image pairs differing by a single object’s presence or absence to isolate negation sensitivity.The dataset also expands visual concept diversity beyond the limited object inventories of COCO and VOC2007.
4. NegBench Evaluations: Results and Insights
NegBench evaluations show that CLIP-like VLMs struggle broadly with negation: performance remains near chance, scaling and newer architectures do not resolve the problem, and medical tasks show substantial drops. Analyses attribute these failures to affirmation bias and collapsed or shortcut embedding representations.
- MCQ-Neg Results: Most CLIP models perform at or only slightly above random guessing on MCQ-Neg, with CLIP reaching 39% accuracy on COCO.The task is four-way multiple choice, where random chance is 25%.
- MCQ-Neg Results: Scaling from ViT-B/32 to ViT-L/14 and ViT-H/14, or using SigLIP and AIMV2, does not improve negation understanding over baseline CLIP.AIMV2’s failure is notable despite its state-of-the-art performance on many other vision-language tasks.
- MCQ-Neg Results: 15.8% and 33.2% performance drops occur for BioMedCLIP and CONCH, respectively, when negation is introduced in CheXpert MCQ-Neg.The result underscores the difficulty of interpreting presence and absence correctly in medical diagnostics.
- MCQ-Type Analysis: CLIP achieves 82% on Affirmation MCQs but only 3% on Negation MCQs for VOC2007, revealing severe affirmation bias.All models perform poorly on Negation MCQs, with ConCLIP as the exception to the stated affirmation-bias pattern.
- MCQ-Type Analysis: Models often select negation templates that incorrectly negate existing objects, suggesting shortcut strategies that ignore words such as “no.”Because 67% of MCQs contain no correct Affirmative option, biased models may default to statements like “This image does not include {pos}.”
- Embedding Analysis: PCA shows distinct embedding failures: CLIP and NegCLIP overlap affirmative and negated captions, while ConCLIP collapses negative captions across object types.A Sentence Transformer provides a reference with separation along both object-type and negation dimensions; hybrid-caption analysis further shows ConCLIP compression along the negation axis.
5. A Data-Centric Approach for Improving Negation Understanding
The paper addresses CLIP-based negation failures by creating synthetic negation-enriched data and combining contrastive and multiple-choice supervision. Fine-tuning on these datasets substantially improves retrieval and MCQ performance.
- Motivation: Training data limitations may cause CLIP’s linguistic shortcuts because standard data lacks explicit negation, while ConCLIP overfits one hybrid template.These limitations restrict distinguishing affirmed from negated concepts and generalizing across negation structures.
- Synthetic datasets: CC12M-NegCap provides three negated captions per image, totaling approximately 30 million captions.CC12M-NegMCQ provides one correct caption and three hard negatives per image, totaling around 40 million captions.
- Training objectives: The contrastive objective aligns each image with its correct caption while lowering similarity to incorrect captions in batches from CC12M-NegCap.Each image-caption pair describes both present and absent objects.
- Training objectives: The MCQ objective trains models to select one correct caption over three hard negatives using cosine-similarity logits and cross-entropy.For the experiments, each image has four caption options.
- Training objectives: The combined objective weights CC12M-NegCap contrastive loss and CC12M-NegMCQ loss by α to balance retrieval and fine-grained MCQ learning.Increasing α emphasizes the original CLIP objective, whereas decreasing it emphasizes MCQ supervision.
- Results: On COCO, CLIP fine-tuning with CC12M-NegCap raises R-Neg@5 from 48.0% to 57.8% and MCQ accuracy from 39.2% to 47.3%.With CC12M-NegFull, CLIP and NegCLIP reach MCQ accuracies of 54.4% and 56.2%, respectively; similar trends hold on MSR-VTT.
6. Discussion and Conclusions
The discussion argues that negation-sensitive VLMs require more diverse language data and complementary learning objectives. The reported results support combining broad contrastive retrieval with fine-grained MCQ supervision.
- Results: Negation-enriched fine-tuning significantly improves R-Neg and MCQ performance across COCO and MSR-VTT.Table 1 compares fine-tuning datasets on retrieval and MCQ metrics, highlighting MCQ increases of +8 or more.
- Implications: Negation-aware pretraining should include diverse language constructs, including nuanced expressions and complex syntactic structures.The paper contrasts this need with datasets dominated by straightforward affirmative statements.
- Implications: Combining contrastive learning with MCQ supervision can improve coarse-grained retrieval and fine-grained negation understanding.Varying α reveals a performance tradeoff between the two objectives.
A. Evaluating LLaVA on NegBench MCQs
LLaVA improves on CLIP models for negation MCQs, likely benefiting from its language model text encoder. However, its pairwise generation-based design is poorly suited to scalable retrieval.
- LLaVA evaluation: LLaVA significantly outperforms CLIP models on MCQ-Neg despite using a CLIP ViT-L/14 vision encoder.The passage attributes this advantage potentially to Vicuna’s exposure to affirmations and negations in diverse text.
- Limitations: LLaVA’s retrieval approach requires presenting all candidate captions with each image, limiting scalability and preventing text-to-image retrieval.CLIP instead supports efficient retrieval through a shared image-text embedding space and cosine similarity.
- Template selection: Figure 8 measures how often models select Affirmation, Negation, or Hybrid templates regardless of answer correctness.This analysis is intended to expose systematic decision biases.
- Template selection: Most CLIP-based models overselect Negation templates even when Affirmation or Hybrid is correct.This behavior is consistent with models ignoring “not” and collapsing positive and negative statements in embedding space.
- Embedding analysis: Figure 9 broadens PCA embedding analysis across CLIP models differing in size and pretraining data, complementing the earlier embedding visualization.The projections represent caption embeddings and examine separation across VLMs.
D.1 How does this work fit into the broader landscape of negation and compositionality research?
The paper extends prior template-based and compositionality-focused negation benchmarks with naturalistic, multimodal evaluation and synthetic training data. Scaling joint-embedding models alone does not resolve their negation failures.
- Relation to prior work: Prior benchmarks such as CREPE and CC-Neg provide limited, template-based negation tests, while NegBench spans broader naturalistic contexts.NegBench covers retrieval and MCQ tasks across images, video, and medical domains.
- Contributions: NegBench contains 79K examples, and the paper generates more than 70M synthetic negation-enriched training samples.The samples support both contrastive and multiple-choice learning objectives.
- Scaling experiments: Larger CLIP, SigLIP, and AIMv2 models still struggle on MCQ-Neg despite stronger standard retrieval performance.The comparisons indicate that increased capacity alone does not meaningfully close the negation gap.
- Data construction: The paper constructs realistic negated prompts by selecting absent objects that commonly co-occur with objects present in densely annotated images.This co-occurrence strategy aims to keep negations visually grounded rather than artificial.
D.4 What is the significance of the medical experiment, despite its simplicity?
Despite its simple binary design, the medical retrieval experiment reveals a substantial negation failure in domain-specialized VLMs, with direct clinical significance.
- Up to 33% performance drops occur under negation for BioMedCLIP and CONCH.The experiment contrasts “has pneumonia” with “does not have pneumonia” in a binary decision setup.
- The binary setup provides a clean, interpretable upper bound on model capability.
- The failure has real-world clinical implications because affirming or negating medical conditions requires precision.
D. Dataset and Task Summary for NegBench
NegBench spans diverse image, video, and medical datasets through retrieval and MCQ tasks, with templated and LLM-paraphrased variants. Its benchmark also includes targeted synthetic hard negatives for studying negation understanding.
- NegBench evaluates both Retrieval-Neg and MCQ-Neg, generally in templated and LLM-paraphrased versions.CheXpert is templated only because its tasks directly evaluate diagnostic capabilities involving negation words.
- COCO, VOC2007, MSR-VTT, and CheXpert provide complementary retrieval, MCQ, video, and medical-imaging evaluations.VOC2007 lacks retrieval because it has no retrieval-style captions, while original retrieval captions are used for COCO and MSR-VTT.
- CheXpert contributes Affirmation Control and Negation Understanding tasks in both MCQ and binary-classification formats.
- 18 task variations and 79,239 samples comprise NegBench across its datasets and task versions.Table 3 summarizes the included datasets, tasks, versions, and sizes.
- HardNeg-Syn contains 10,000 synthetic images constructed as hard negatives for targeted negation evaluation.Each image pair differs by a single object, supporting controlled study of negation understanding.
- The synthetic-data pipeline combines object labels, LLaMA 3.1 scene descriptions and paraphrases, Stable Diffusion image pairs, and OWL-ViT verification.Negative images exclude object B in the negative prompt, while captions use templates and LLaMA 3.1 paraphrasing.
E. Visualizing the NegBench Evaluation Tasks
The visualized NegBench examples cover retrieval and MCQ tasks across natural-image, medical, synthetic, and video settings. They illustrate how negation is tested through object absence, diagnostic statements, paired images, and absent actions.
- Figures 11–14 visualize NegBench retrieval and MCQ samples across diverse visual domains and real-world applications.
- COCO and VOC2007 examples include Retrieval-Neg and MCQ-Neg tasks.
- CheXpert examples show Affirmation Control and Negation tasks in medical MCQ settings.
- HardNeg-Syn pairs differ by a single object, such as a cactus, making them suitable for controlled negation evaluation.
- MSR-VTT examples cover negated captions for complex water-rescue scenes and choices about the presence or absence of actions.