Source-linked AI summary
Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning
Junhao Xiao, Zhiyu Wu, Hao Lin, Yi Chen, Yahui Liu, Xiaoran Zhao, Zixu Wang, Zejiang He
TL;DR
CLIP struggles to distinguish negated from affirmative visual descriptions, while fine-tuning-based fixes require costly annotated data and risk degrading general-purpose performance. CLIPGLASSES addresses this with non-invasive Lens and Frame modules that add negation-aware repulsion to similarity computation, achieving competitive in-domain accuracy, stronger cross-domain generalization, and low-resource robustness while preserving zero-shot capabilities.
Problem
CLIP often matches negated descriptions with corresponding visual content, while fine-tuning-based adaptations require large annotated datasets and risk catastrophic forgetting.
Method
CLIPGLASSES uses a syntax-semantic Lens to disentangle negated semantics and a Frame to model context-aware repulsion without modifying CLIP parameters.
Results
CLIPGLASSES achieves competitive in-domain accuracy, state-of-the-art cross-domain generalization, and low-resource robustness while preserving CLIP’s native zero-shot capabilities.
Takeaways & Limitations
The framework provides a transferable negation-sensitive extension of CLIP that avoids the general-purpose performance costs associated with fine-tuning.
Takeaways & Limitations
Current methods, including CLIPGLASSES, remain limited in handling non-visual negations such as “not authentic.”
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) like CLIP struggle to understand negation, often embedding affirmatives and negatives similarly (e.g., matching "no dog" with dog images). Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting. In this work, we propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions. CLIPGlasses adopts a dual-stage design: a Lens module disentangles negated semantics from text embeddings, and a Frame module predicts context-aware repulsion strength, which is integrated into a modified similarity computation to penalize alignment with negated semantics, thereby reducing false positive matches. Experiments show that CLIP equipped with CLIPGlasses achieves competitive in-domain performance and outperforms state-of-the-art methods in cross-domain generalization. Its superiority is especially evident under low-resource conditions, indicating stronger robustness across domains.
Introduction
CLIP struggles with negation because negation expressions are sparse in pretraining, while fine-tuning approaches require costly data and risk catastrophic forgetting. CLIPGLASSES addresses this with frozen-parameter modules that disentangle negated semantics and dynamically penalize their image-text alignment, improving cross-domain and low-resource robustness.
- CLIP incorrectly matches negated concepts with corresponding visual content because negation expressions comprise only 0.7% of pretraining corpora.
- Fine-tuning methods require large negation-annotated datasets and risk catastrophic forgetting of general-purpose performance.
- CLIPGLASSES uses frozen CLIP parameters, with Lens disentangling negated semantics and Frame predicting context-dependent repulsion strength for modified similarity computation.
- 34.51% on Neg-COCO-MCQ versus 25.70% for CoN-CLIP demonstrates stronger cross-domain generalization, while CoN-CLIP reaches 99.70% versus 96.56% on CC-Neg-val.
- Under 5K-image conditions, CLIPGLASSES surpasses CoN-CLIP by 27.45 points on CC-Neg-val and 5.29 points on Neg-COCO-MCQ.
- The framework preserves CLIP’s native zero-shot abilities while achieving a trade-off between in-domain accuracy and cross-domain generalization.
Related Work
Related work presents CLIP as a broadly capable cross-modal model while emphasizing its limitations in fine-grained semantic understanding. The section situates negation modeling within these broader VLM deficiencies.
- CLIP supports retrieval, captioning, visual question answering, and generation through representations learned from large-scale noisy image-text pairs.
- VLMs exhibit critical deficiencies in fine-grained semantic understanding, including over-reliance on shallow statistical cues and failures in compositional reasoning.
- The section introduces preliminary discussion of two-stage modeling for negation understanding.
Negation Understanding
CLIP places affirmative and negative semantics close together, but structured separability and human two-stage processing motivate extracting negation semantics before suppressing alignment with negated concepts.
- Layer-specific encoding creates potential to disentangle negation-related information from CLIP text embeddings.
- Human negation processing first identifies the negated concept and then inverts its semantic implication.
- The proposed computational strategy extracts negation semantics first, then modulates image-text similarity by suppressing alignment with negated concepts.
- CLIP’s t-SNE features separate different sentence pairs while keeping positive and negative forms within each pair closely positioned.
Methodology
CLIPGlasses extends CLIP with Lens and Frame modules that separate negated semantics, estimate context-dependent repulsion, and adjust image-text similarity to penalize negated alignment.
- Lens: Lens disentangles negated semantics from CLIP text embeddings using syntax-semantic processing across hierarchical representations.The module uses early-layer features for syntactic information and final-layer features for global semantic context.
- Lens: Hierarchical attention combines multi-level syntactic representations with global semantics, while residual gating limits semantic drift by blending attended features with original CLIP features.The gate acts as a soft selector, favoring original features unless structural adjustments are needed.
- Image-Text Matching: The architecture uses Lens-derived negated concepts and Frame-predicted λ to compute S = SI2T −λ · Sneg, reducing alignment with negated content.The modified score preserves affirmative image-text alignment while subtracting a context-scaled negated-semantic component.
- Frame: Frame predicts repulsion strength λ from fused text and cross-modal image-text context, allowing negation strength to vary with linguistic and visual conditions.Joint self-attention provides bidirectional cross-modal dependencies before the text-centric representation is used to estimate λ.
- Image-Text Matching: The final similarity combines CLIP’s base alignment with a negation-aware repulsion term that penalizes agreement with negated concepts.The repulsion term is conditionally activated for negated inputs, preserving standard alignment behavior for affirmative cases.
Comparative Experiments
CLIPGlasses is evaluated against fine-tuning baselines, standard zero-shot benchmarks, and component ablations. The results emphasize cross-domain and low-resource robustness, preservation of general abilities, and contributions from Lens, Frame, and adaptive repulsion.
- Comparative performance: CLIPGlasses achieves 96.56% accuracy on CC-Neg-val and surpasses CoN-CLIP by 8.81 points on Neg-COCO-MCQ, despite slightly lower in-domain accuracy.CoN-CLIP scores 99.70% on CC-Neg-val, while CLIPGlasses reaches 34.51% versus 25.70% on Neg-COCO-MCQ.
- Comparative performance: Under low-resource training with 5K images, CLIPGlasses outperforms CoN-CLIP by 27.45 points on CC-Neg-val and 5.29 points on Neg-COCO-MCQ.The corresponding scores are 93.36% versus 65.91% on CC-Neg-val and 35.90% versus 30.61% on Neg-COCO-MCQ.
- Zero-shot retention: CLIPGlasses retains near-original zero-shot performance on ImageNet and Caltech101, matching or surpassing vanilla CLIP while CoN-CLIP degrades, particularly on ImageNet.These evaluations test whether negation improvements compromise standard non-negation visual-language alignment.
- Ablation analysis: Removing the syntactic stream reduces accuracy by 2.47% and FAR by 3.33 points, while removing the semantic stream reduces accuracy by 1.63% and FAR by 1.73 points.The ablations associate syntactic features with locating negation cues and semantic features with interpreting their sentence-level targets.
- Ablation analysis: Predicted repulsion weight λ decreases consistently with weaker negation, indicating adaptive modulation of repulsion intensity according to linguistic cues.The controlled negation-strength analysis evaluates the module’s hypothesized role in calibrating repulsion.
Conclusion
CLIPGLASSES addresses CLIP’s negation limitations without modifying pretrained parameters, while maintaining competitive in-domain accuracy, strong cross-domain generalization, and low-resource robustness. The paper identifies non-visual negations as a remaining limitation for current methods.
- CLIPGLASSES uses a non-intrusive design with a Lens module for disentangling negated semantics and a Frame module for context-aware repulsion.
- The method achieves competitive in-domain accuracy, state-of-the-art cross-domain generalization, and low-resource robustness while preserving CLIP’s native zero-shot capabilities.
- Current methods remain limited in handling non-visual negations such as “not authentic,” motivating future integration of commonsense knowledge.