PaperlayerView exact revision on arXiv

Source-linked AI summary

CS-CLIP: Compositional Scene Graph-guided CLIP for Robust Compositional Reasoning

SeongJun Jeong, Minjoon Jung, Woo Suk Choi, Youwon Jang, Byoung-Tak Zhang

arXiv:2609.08242v1cs.CVcs.AI

TL;DR

Existing compositionality-aware VLMs show element-specific biases that controlled tests expose more clearly than prior benchmarks. CS-CLIP uses scene graphs and a generate-and-rank process to create element-targeted, semantically contradictory hard negatives, achieving robust compositional reasoning while preserving general capabilities and sample efficiency.

  • Problem

    Existing compositionality-aware VLMs exhibit element-specific biases, including underperformance on certain compositional elements, and their behavior remains unclear under isolated element-wise evaluation.

  • Method

    CS-CLIP parses captions into scene graphs, targets compositional elements with hard negatives, and uses MLM generation plus NLI ranking to select fluent contradictions.

  • Results

    CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across reasoning types while preserving cross-modal retrieval and downstream visual reasoning.

  • Takeaways & Limitations

    Element-targeted contradictory negatives enable compositional gains alongside preserved general vision-language capabilities and strong sample efficiency.

  • Takeaways & Limitations

    CS-CLIP relies on off-the-shelf scene graph, MLM, and NLI modules, whose linguistic biases may propagate to generated negatives.

Abstract

from arXiv · show

Vision-language models (VLMs) demonstrate strong performance across compositional reasoning benchmarks, which require reasoning over semantic perturbations of objects, attributes, relations, and their interactions. However, our controlled analysis reveals that existing compositionality-aware VLMs exhibit element-specific biases, often underperforming vanilla CLIP on certain compositional elements. To address this, we propose Compositional Scene Graph-guided CLIP (CS-CLIP), which uses scene graphs to identify compositional elements and construct structured negatives via selective masking. We further retain negatives that are most contradictory to the original caption, forcing the model to rely on compositional structure rather than surface cues. CS-CLIP achieves state-of-the-art compositional reasoning with robust performance across compositional elements. It also preserves general vision-language capabilities such as cross-modal retrieval and downstream visual reasoning, while requiring fewer training samples than prior methods.

1 Introduction

Existing compositionality-aware VLMs can exhibit element-specific biases, while CS-CLIP targets each compositional element with structured hard negatives and achieves robust compositional reasoning with preserved general capabilities.

  • Motivation: Controlled evaluations reveal element-specific biases in existing compositionality-aware VLMs, including underperformance relative to vanilla CLIP on some elements.The analysis uses test cases that isolate individual compositional elements and measures both accuracy and sensitivity.
  • Method: CS-CLIP identifies compositional elements through scene graph parsing and constructs element-targeted hard negatives with a generate-and-rank pipeline.An MLM generates fluent candidates, while an NLI model selects candidates that contradict the original caption.
  • Results: CS-CLIP achieves state-of-the-art average performance with the lowest type-wise deviation on compositional reasoning benchmarks.The reported pattern indicates robust performance across reasoning types.
  • Results: The method preserves competitive cross-modal retrieval and transfers to zero-shot visual reasoning while using substantially fewer training samples than prior methods.These capabilities are reported alongside the compositional reasoning gains.
  • Contribution: The paper presents CS-CLIP as a way to mitigate element-specific biases while preserving general vision-language capabilities.This is stated as a central contribution of the work.

2 Related Work

VLMs align images and text in shared embedding spaces and generalize across downstream tasks, but often rely on salient keywords rather than fine-grained compositional structure.

  • Vision-Language Models: Large-scale contrastive learning aligns image and text representations into a shared embedding space for broad downstream generalization.The related work describes this as the foundation of modern vision-language models.
  • Compositional Reasoning: VLMs often exhibit bag-of-words behavior, prioritizing salient keywords over fine-grained compositional structure.This limitation affects reasoning over objects, attributes, and relations.

3 Controlled Analysis

The controlled analysis isolates object, attribute, and relation changes, evaluates accuracy and sensitivity, and exposes inconsistent compositional reasoning in existing VLMs.

  • Construction of Test Cases: The test cases target objects, attributes, and relations using vocabularies extracted from scene graph annotations.Each hard negative modifies exactly one compositional element at a time.
  • Evaluating Discriminative Confidence: The evaluation uses accuracy and sensitivity to assess both correct positive-negative discrimination and confidence of separation.Sensitivity is measured with Cohen’s d using positive and negative similarity scores.
  • Results on Test Cases: The controlled evaluation compares vanilla CLIP with NegCLIP, StructureCLIP, and TripletCLIP across compositional element changes.The comparison is reported in Figure 2.
  • Results on Test Cases: Existing compositionality-aware VLMs do not improve consistently across element changes and often underperform vanilla CLIP for attributes and relations.NegCLIP falls from 63.6 to 54.4 on the reported comparison and also shows substantially lower sensitivity.
  • Hard Negative Generation: The hard-negative framework identifies elements with scene graphs, generates candidates with an MLM, and selects contradictory captions using an NLI model.These three stages are summarized in Figure 3.

4 Method

CS-CLIP targets compositional elements precisely by parsing captions into scene graphs and generating fluent, semantically contradictory hard negatives. It trains CLIP with element-targeted supervision using contrastive and margin-based objectives.

  • Element targeting: Scene graphs localize objects, relations, and attributes to caption spans, enabling element-level targeting.Each caption is parsed into G = (O, R, A), and the localized spans become targets for hard-negative generation.
  • Candidate generation: Masked language modeling generates fluent candidate replacements for targeted spans while filtering vocabulary by grammatical category.The pipeline uses K = 16 candidate tokens and removes sub-word artifacts before semantic ranking.
  • Semantic ranking: Natural language inference ranks candidates by contradiction with the original caption, excluding semantically equivalent paraphrases.The original caption is treated as the premise, and candidate captions are evaluated for contradiction.
  • Augmentation types: Replace, add, and swap augmentations perturb objects, attributes, or relations independently to construct diverse element-targeted negatives.Replace uses the full generate-and-rank pipeline, add introduces an additional object or attribute, and swap rearranges same-type spans without MLM or NLI.
  • Contrastive learning: CS-CLIP trains CLIP on positive image-caption pairs and hard-negative triplets using InfoNCE plus a hinge loss.The hinge objective enforces a positive-image similarity advantage of at least ω = 0.1 over each hard negative.

5 Experiments

Experiments show that CS-CLIP delivers robust compositional reasoning across benchmarks and element types while preserving retrieval and downstream visual reasoning. Ablations indicate that scene-graph-guided placement and targeted hard negatives are central to these gains, with strong sample efficiency.

  • Robust compositional reasoning across element changes: CS-CLIP achieves the highest accuracy and sensitivity across all three compositional elements, while baselines underperform vanilla CLIP on relations.The same pattern appears on VG, where CS-CLIP reaches state-of-the-art on VG-Relation and remains competitive on VG-Attribute.
  • Gains across reasoning and element types: CS-CLIP achieves the highest average accuracy and lowest type-wise standard deviation on SugarCrepe, performing best or second-best on every reasoning type.Lower standard deviation indicates more uniform performance across the seven reasoning types.
  • Generalization beyond Compositional Benchmarks: CS-CLIP preserves competitive COCO cross-modal retrieval and achieves the highest accuracy among baselines on zero-shot GQA visual reasoning.TripletCLIP degrades substantially on COCO retrieval despite its compositional gains.
  • Ablations: Pipeline ablations raise average accuracy from 76.3 to 88.5 while reducing type-wise standard deviation from 9.6 to 5.2.Scene-graph guidance drives the largest accuracy gain, while MLM-based generation and NLI-based ranking further reduce deviation.
  • Sample efficiency: CS-CLIP is trained across scales from 1K to 1M hard negatives, enabling a controlled within-method sample-efficiency analysis against baseline checkpoints.The comparison uses baseline checkpoints at their reported training scales.
  • Ablations: Structural placement yields a substantial gain over random placement when both variants share the same generated vocabulary.This supports precise element-level control as the primary source of the placement benefit rather than vocabulary alone.
  • Qualitative analysis: CS-CLIP distinguishes relation and attribute changes that mislead other models, including cases involving reversed actions or distracting background colors.Qualitative predictions show a clearer margin between positive and negative captions than CLIP and TripletCLIP.

6 Conclusion

The paper identifies element-specific biases in existing compositionality-aware VLMs and proposes CS-CLIP to address them with targeted, fluent, contradictory hard negatives. Across benchmarks, CS-CLIP achieves robust compositional reasoning, preserves cross-modal capabilities, transfers to downstream visual reasoning, and uses fewer training samples than prior methods.

  • Conclusion: CS-CLIP constructs element-targeted hard negatives through scene graph parsing, masked language modeling, and natural language inference.The pipeline targets compositional elements while making negatives fluent and semantically contradictory to positives.
  • Conclusion: CS-CLIP achieves robust performance across reasoning types, preserves cross-modal capability, transfers to downstream visual reasoning, and shows strong sample efficiency.The paper states that code and trained models will be released for future research.

7 Limitations

The study’s framework depends on external scene graph parser, MLM, and NLI modules, whose linguistic biases may propagate into generated negatives. Its negative generation uses only original captions rather than visual features, although the authors describe this as an efficient alternative.

  • Limitations: CS-CLIP relies on off-the-shelf scene graph, MLM, and NLI modules, so their inherent linguistic biases may propagate to generated negatives.The modular design can incorporate future improvements to these modules without architectural changes.
  • Limitations: Negative generation uses only original captions; incorporating visual features could further refine the process.The authors state that caption-derived negatives remain visually relevant and that the current approach is efficient and effective.

8 Ethical Considerations

CS-CLIP is evaluated against representative vanilla and compositionality-aware VLMs, while the paper notes that CLIP-derived models may retain societal biases from web-crawled training data.

  • CS-CLIP may inherit societal, racial, or gender biases present in large-scale web-crawled training data.The approach focuses on compositional accuracy and does not explicitly filter or mitigate these underlying biases.
  • The authors recommend caution when deploying CS-CLIP in sensitive real-world applications.
  • The study compares CS-CLIP with CLIP, BLIP-2, NegCLIP, StructureCLIP, and TripletCLIP using released checkpoints or official implementations.

B Benchmark Details

The benchmarks test whether models distinguish images from captions differing in fine-grained objects, attributes, relations, or visually grounded answers. Controlled negatives alter one compositional element while preserving other content and linguistic plausibility.

  • SugarCrepe is a binary image-to-text retrieval benchmark pairing each image with a correct caption and a minimally perturbed hard negative.Its hard-negative families include object, attribute, and relation replacement, object and attribute swapping, and object and attribute addition.
  • VG-Relation tests relation order using captions that swap the subject and object around a relation.
  • VG-Attribute tests whether models bind properties to the correct entities by swapping attributes between two objects.
  • The controlled VG tests modify exactly one object, relation, or attribute using vocabularies containing 899 objects, 91 relations, and 541 attributes.Synonyms and near-duplicates are manually filtered so substitutes remain semantically distinct.
  • The GQA benchmark presents a scene-consistent answer against a fluent but visually incorrect distractor in a zero-shot image-to-text retrieval task.Distractor answers are drawn from the same image’s scene graph while excluding entities referenced by the question and gold answer.
  • Quality filtering removes yes/no answers, near-duplicate replacements, and overly similar positive-negative sentences before randomly sampling 10,000 instances.

D.3 Evaluation Protocol

The evaluation encodes images and two candidate texts, then determines correctness by comparing their cosine similarities; Figure 6 illustrates representative CS-CLIP hard negatives.

  • Each image is encoded with positive and negative texts, and the instance is correct when the positive text has higher cosine similarity.Accuracy is reported against a 50% chance level.
  • Figure 6 presents examples of CS-CLIP’s hard negatives.

E Effectiveness on Different Backbones

Hard-negative augmentation improves performance across both CLIP and StructureCLIP backbones, indicating that its benefits are not restricted to one encoder design. The examples show negatives that change only the targeted compositional element while preserving fluency and sentence structure.

  • 88.5% average accuracy for CLIP and 88.0% for StructureCLIP follow hard-negative augmentation, versus 76.3% and 83.8% without it.
  • The augmentation provides consistent supervision across different backbone architectures rather than depending on a specific encoder design.
  • CS-CLIP hard negatives change only the targeted compositional element while preserving fluency and the rest of the sentence structure.Figure 6 includes examples spanning all seven augmentation types.
Loading 2609.08242v1…