PaperlayerView exact revision on arXiv

Source-linked AI summary

CF-VLM:CounterFactual Vision-Language Fine-tuning

Jusheng Zhang, Kaitong Cai, Yijia Fan, Jian Wang, Keze Wang

arXiv:2506.17267v1cs.LGcs.AI

TL;DR

VLMs still struggle with fine-grained discrimination and causal reasoning, as existing methods can rely on superficial correlations rather than visual-textual causal logic. CF-VLM uses targeted counterfactual samples and complementary objectives to address this gap, and experiments report stronger compositional reasoning and generalization performance, with reduced hallucinations. Its current limitation is reliance on synthetic counterfactual generation, which may introduce bias and limit coverage of real-world variation.

  • Problem

    VLMs remain limited in fine-grained discrimination and deep causal reasoning because they may rely on superficial correlations instead of underlying visual-textual causal logic.

  • Method

    CF-VLM fine-tunes VLMs with targeted counterfactual samples, combining joint image-text alternatives with minimally edited images and causal-discrimination objectives.

  • Results

    CF-VLM outperforms CLIP-based and LLM-augmented VLMs across compositional reasoning and hallucination benchmarks, with reported gains on ConMe, ARO, and VL-Checklist and generalization to untrained benchmarks.

  • Takeaways & Limitations

    Counterfactual supervision improves factual grounding and reduces hallucinations while supporting generalization across vision-language evaluation settings.

  • Takeaways & Limitations

    The framework relies primarily on synthetic image and text generation, which may introduce bias and fail to capture the full diversity of real-world variations.

Abstract

from arXiv · show

Recent advances in vision-language models (VLMs) have greatly improved cross-modal semantic understanding, yet significant limitations remain in fine-grained discrimination and deep causal reasoning tasks. Existing VLMs often rely on superficial statistical correlations, lacking the ability to capture the underlying causal logic between visual and textual content. To address this, we propose CounterFactual Vision-Language Fine-tuning (CF-VLM), a novel framework that enhances the causal reasoning capabilities of VLMs through the targeted use of counterfactual samples. CF-VLM introduces three complementary training objectives: maintaining foundational cross-modal alignment, reinforcing the uniqueness and stability of factual scene representations against coherent counterfactuals, and sharpening the model's sensitivity to minimal but critical causal edits. Extensive experiments demonstrate that CF-VLM consistently outperforms strong baselines and state-of-the-art methods on compositional reasoning and generalization benchmarks. Furthermore, it shows promise in mitigating visual hallucinations, indicating improved factual consistency. Our CF-VLM provides a robust foundation for deploying VLMs in high-stakes, real-world scenarios requiring reliable reasoning and interpretability.

1 Introduction

Existing VLMs improve cross-modal understanding but remain weak at fine-grained discrimination and causal reasoning because they often rely on superficial correlations. CF-VLM addresses this gap by using counterfactual samples and objectives that distinguish factual scenes from coherent alternatives and detect minimal causal edits.

  • Current VLMs struggle with fine-grained visual-language tasks and deep reasoning, limiting their applicability in high-stakes environments.
  • Triplet-style methods optimize whether pairs match, but do not explicitly model the causal attributes or relations behind semantic shifts.
  • CF-VLM fine-tunes VLMs around counterfactual samples created through minimal, semantically crucial edits to images or causal relations.
  • Complete counterfactual scenarios jointly modify image and text, encouraging factual anchor representations to remain more similar than logically consistent alternatives.
  • Minimal causal-edit training contrasts a factual image with a minimally altered image paired with the same text, emphasizing the semantic shift caused by the intervention.

2 Preliminary

CF-VLM extends cross-modal representation learning with controlled counterfactual samples that preserve scene proximity while changing decisive semantics. Its sample types include jointly edited alternatives and minimally edited images used as hard negatives.

  • Figure 1 contrasts CF-VLM’s focus on fine-grained causal decision points with Triplet CLIP’s coarse matching distinction.
  • VLMs learn shared image-text embeddings in which aligned pairs have high similarity and unrelated pairs have low similarity.
  • CF-VLM defines counterfactual samples through precise, targeted interventions designed to expose causal factors underlying image-text alignment.
  • Complete counterfactual scenario pairs jointly edit image and text into logically coherent but factually distinct alternatives.
  • Minimally edited counterfactual images apply one critical intervention and retain the original text as a hard negative for fine-grained discrimination.

3 Methodology

CF-VLM refines pretrained vision-language representations with structured factual and counterfactual samples, combining foundational alignment with scenario-level and minimal-edit causal discrimination. Its pipeline targets both semantic precision and sensitivity to causal changes.

  • Training Data: CF-VLM organizes each training unit around an anchor factual image-text pair, complete counterfactual scenarios, and minimally edited counterfactual images.Complete scenarios jointly modify image and text, while minimal edits target a single critical semantic or causal change.
  • Foundational Cross-Modal Alignment Loss: The foundational alignment objective preserves image-text matching by jointly optimizing image-to-text and text-to-image similarities with a symmetric InfoNCE loss.The loss encourages each image to match its paired text and each text to match its paired image within the batch.
  • Counterfactual Scenario Discrimination Loss: Counterfactual scenario discrimination separates the factual anchor from logically coherent but semantically different complete counterfactual scenarios.Its hinge formulation penalizes counterfactual pairs whose similarity approaches or exceeds the anchor, using a margin-based separation.
  • CF-VLM Training Pipeline: The complete pipeline generates complete and minimally edited counterfactual images from factual image-text anchors using a fine-tuned SDXL model.The three objectives are cross-modal alignment, counterfactual scenario discrimination, and fine-grained causal discrimination.
  • Fine-Grained Causal Discrimination Loss: Fine-grained causal discrimination focuses on the hardest minimally edited counterfactual to detect subtle semantic changes that alter image-text matching.The objective compares the anchor similarity against the highest-similarity edited image and applies a hinge penalty unless a margin is maintained.

4 Experiment

Experiments evaluate CF-VLM across compositional reasoning, standard vision-language generalization, ablations, and hallucination benchmarks. The results show broad performance gains, with complementary counterfactual types and objectives contributing to accuracy while larger counterfactual ratios eventually yield diminishing returns.

  • Main Results: CF-VLM outperforms CLIP-based baselines on ConMe, Visual Genome, and VL-Checklist, reaching 59.13% on ConMe and 88.4% on VL-Checklist.It also achieves 88.4% on VG-Rel and 90.3% on VG-Attr.
  • Main Results: On Qwen-VL 7B, CF-VLM reaches 87.57% on ConMe, improving over Standard Fine-tuning by 4.9 points and TextNeg Fine-tuning by 3.4 points.It reaches 91.8% on VG-Rel and 94.6% on VG-Attr, and transfers well to LLaVA-1.5.
  • Main Results: CF-VLM generalizes to zero-shot ImageNet-1k classification and cross-modal retrieval, with top-1 8.83%, top-5 19.54%, and Recall@5 of 13.80% on MSCOCO and 26.58% on Flickr30k.These results outperform or match CLIP-based state-of-the-art methods and relevant baselines.
  • Ablation Studies: The full model using both image and text counterfactuals achieves the best results across ablation benchmarks, including ConMe 59.1 and ARO 89.35.The comparison includes non-causal, text-only counterfactual, and image-only counterfactual baselines.
  • Ablation Studies: All three loss components are necessary for the best ConMe performance: the full model averages 59.1, while removing fine-grained causal discrimination lowers average accuracy to 57.9.The full model scores 60.4 on Replace-Obj, 60.4 on Replace-Attr, and 56.6 on Replace-Rel; removing fine-grained causal discrimination reduces Replace-Rel by 1.5 points.
  • Counterfactual Supervision: Removing attribute-, object-, or relation-level counterfactuals lowers the corresponding subtask accuracy by 2.2, 2.9, and 3.3 points, respectively.Average accuracy also declines when any counterfactual category is omitted.
  • Counterfactual Supervision: ConMe average accuracy rises from 54.43% with 0% counterfactuals to 59.13% with 100%, then begins plateauing at approximately 60%–80%.The plateau indicates diminishing marginal gains and a performance–computational-cost tradeoff.
  • Hallucination Evaluation: CF-VLM improves hallucination-sensitive evaluation, adding 0.5–1.2 percentage points across POPE accuracy, precision, recall, and F1 and gaining 2.0 on MME Color and 1.4 on Position.The MME gains are reported relative to Standard Fine-tuning on hallucination-sensitive subtasks.

5 Related Work

Prior VLM methods improve cross-modal discrimination but often rely on superficial correlations and do not fully model causal relations. CF-VLM addresses this gap through joint image-text counterfactuals and complementary objectives targeting causal decision points.

  • Limitations of Existing VLMs: Existing VLMs perform well on cross-modal tasks but remain limited in fine-grained discrimination and causal reasoning, often relying on superficial statistical correlations.These limitations motivate methods that capture deeper relationships between visual and textual content.
  • Contrastive Learning: Contrastive methods improve binary matching decisions but do not directly explain how minimal counterfactual differences cause semantic shifts.Their emphasis is whether pairs match rather than which causal attributes determine the match.
  • Counterfactual VLM Methods: Earlier counterfactual approaches typically use single-modality counterfactuals and singular objectives, whereas CF-VLM jointly modifies images and text.CF-VLM combines cross-modal alignment, counterfactual scenario discrimination, and fine-grained causal discrimination.
  • CF-VLM: CF-VLM targets causal decision points rather than only coarse-grained similarity, achieving superior benchmark accuracy while reducing visual hallucinations.The reported benchmarks include ARO, ConMe, and VL-Checklist.

6 Conclusion and Limitations

CF-VLM uses minimally edited image-text pairs and three complementary objectives to improve semantic granularity and causal reasoning. It outperforms competing approaches across compositional reasoning and hallucination benchmarks, including untrained benchmarks, while future work targets richer editing, broader tasks, and interpretability.

  • Conclusion: CF-VLM combines minimally edited image-text pairs with counterfactual scenario discrimination and fine-grained causal discrimination objectives.The framework is designed to improve semantic granularity and causal reasoning in VLMs.
  • Conclusion: Evaluations on ARO, ConMe, VL-Checklist, POPE, and MME report improved performance, factual grounding, and reduced hallucinations, including on untrained benchmarks.The paper presents this as evidence of generality for counterfactual supervision.
  • Future Work: Future work includes human-in-the-loop counterfactual editing, applying CF-VLM to VQA and image editing, and using counterfactual sensitivity for interpretability.These directions aim to provide richer counterfactuals, broaden task coverage, and make VLM behavior more transparent.

A Dataset Details

CF-VLM is evaluated with counterfactual training data and benchmarks designed to test fine-grained semantic sensitivity, compositional reasoning, and cross-modal generalization. Qualitative comparisons further examine attribute recognition, causal action modeling, relational understanding, and complex event semantics.

  • Training Data: Training uses cleaned CC12M and CC3M corpora with standardized preprocessing and a fixed 1:4 factual-to-counterfactual batch ratio.The datasets contain 8.6M and 2.6M image-text pairs, respectively.
  • Counterfactual Generation: Counterfactual generation creates minimally perturbed image-text pairs by changing one attribute or causal relationship while preserving other sentence elements.The rewriting procedure supports attribute edits, causal-relation reversals, and multiple distinct outputs.
  • Benchmarks: Evaluation spans VL-Checklist, zero-shot ImageNet-1k matching, MSCOCO and Flickr30k retrieval, ConMe compositional reasoning, and ARO attribute-relation-object alignment.ConMe tests negation, sequence, and conjunction, while ARO targets attributes, relationships, and object identity.
  • Counterfactual Design: CF-VLM applies controlled attribute, object, relationship, and location substitutions to train sensitivity to semantic variations and causal decision points.These edits provide structured changes to properties, identities, and relations in visual-textual scenes.
  • Qualitative Results: Qualitative results show improved attribute-aware recognition, action-outcome modeling, role distinction, and tracking of temporal and relational changes across complex scenes.Examples include identifying a dog’s frisbee-catching outcome, distinguishing a tennis player from an observer, and tracking bridge-crossing states.
  • Mechanistic Comparison: CF-VLM jointly processes image and text counterfactuals, unlike related approaches that primarily use text-only counterfactuals or image perturbations.The framework combines jointly edited image-text scenarios with minimally edited images paired with original text.

C.2 Analysis of CF-VLM’s Uniqueness and Integrative Advantages

CF-VLM’s integrative design combines controlled counterfactual data, multiple complementary objectives, causal decision-point supervision, and an end-to-end fine-tuning pipeline. Its experimental setup emphasizes consistent comparisons across base models and reproduced baselines.

  • Integrative Advantages: CF-VLM combines diverse minimal edits and complete counterfactual scenarios to provide multi-level contrastive supervision.This design is presented as a central integrative strength of the framework.
  • Integrative Advantages: Lalign, Lcsd, and Lfcd jointly maintain alignment, distinguish factual scenes, and detect causal edits.The objectives are described as complementary components of CF-VLM’s multi-objective design.
  • Integrative Advantages: CF-VLM focuses on causal decision points that explain why small semantic edits change image-text matching beyond feature matching.This focus distinguishes the framework from approaches centered mainly on general discrimination or robustness.
  • Integrative Advantages: The framework integrates SDXL and LLM-based generation into an end-to-end pipeline intended to retrofit existing VLMs.Experiments fine-tune pretrained Qwen2.5-VL and LLaVA-1.5 models.
  • Experimental Control: Baseline comparisons use consistent hardware and software conditions, with implemented fine-tuning baselines largely matching CF-VLM’s optimizer and learning-rate settings.Reported baselines include zero-shot, standard fine-tuning, text-negative fine-tuning, TripletCLIP*, and other related methods.

D.5 Counterfactual Sample Generation Parameters

CF-VLM generates counterfactual samples through a controlled multimodal pipeline using language-model rewriting and diffusion-based image generation. The procedure maintains a fixed factual-to-counterfactual ratio during relevant training stages.

  • Generation Parameters: Counterfactual text descriptions are generated with LLaMA-3-70B-Instruct, while counterfactual images use SDXL 1.0 Base plus Refiner.The image generation schedule uses 40+15 steps.
  • Generation Parameters: Each factual image-text pair produces four counterfactual samples, maintaining a 1:4 factual-to-counterfactual training ratio.Samples may modify the image, the text, or both in complete counterfactual scenarios.
  • Generation Parameters: Figure 8 illustrates counterfactual examples generated through controlled edits to attributes, objects, relationships, and locations.These edit categories correspond to the framework’s structured intervention strategy.

E Large-Scale Counterfactual Image Dataset

The large-scale counterfactual image dataset supplies structured visual perturbations for CF-VLM pretraining, fine-tuning, and ablation studies. Experiments identify a counterfactual density of K = 4 as the strongest trade-off for ConMe compositional generalization.

  • Dataset Scale: The dedicated dataset contains approximately 2 million synthetic counterfactual images generated under structured attribute interventions.The images are produced with a high-resolution diffusion model.
  • Dataset Construction: The dataset is derived from filtered CC12M, MSCOCO, and Flickr30k sources, with edits targeting one semantic attribute such as color, pose, quantity, or emotion.Selection emphasizes visual diversity, semantic clarity, and attribute-editability.
  • Dataset Structure: Each counterfactual unit includes an original image-caption pair plus four counterfactual images and corresponding counterfactual texts.The resulting pairs can be used at 1:1 or higher ratios with factual data.
  • Applications and Release: The dataset supports pretraining, fine-tuning, and image-level counterfactual ablations, with release planned pending copyright and license validation.The release plan is intended to support future research and reproducibility.
  • Density Ablation: K = 4 achieves peak ConMe accuracy, with Replace-Obj 60.4, Replace-Attr 60.4, and Replace-Rel 56.6.Performance improves monotonically from K = 1 to K = 4 before plateauing or slightly regressing.
  • Density Ablation: A 4:1 counterfactual density offers the strongest compositional-generalization trade-off and is recommended as a default for lightweight CLIP-based models.The recommendation is framed for settings with limited training budgets.

G Analysis of Training Cost and Efficiency

CF-VLM improves benchmark performance with moderate, controllable computational overhead. Its counterfactual design combines scene-level and minimal-edit supervision to target causal, fine-grained distinctions beyond conventional discrimination.

  • Performance and cost: 87.57% CONME accuracy on Qwen2.5-VL-7B improves over 79.5% with standard fine-tuning, a +8.1 point gain.The improvement surpasses the modest gains of TextNeg FT.
  • Performance and cost: 60.4% Replace-Obj accuracy with CF-VLM improves over 55.7% with standard fine-tuning on ViT-B/32.
  • Performance and cost: Training steps increase roughly 20% and GPU-hours less than 25% on ViT-B/32, while performance improves by over 5 points.TextNeg FT provides only marginal benefits at nearly the same cost as standard fine-tuning.
  • Counterfactual ratio: Replace-Obj accuracy rises from 57.9% to 60.4% as counterfactual ratio K increases from 1 to 4, then plateaus beyond K = 4.Data size increases fivefold and GPU-hours scale proportionally, making K = 4 a cost-effective choice.
  • Model-size efficiency: ViT-B/32 uses roughly one-third to one-quarter the per-step latency and memory of Qwen2.5-VL-7B, with several-fold higher throughput.The larger model exceeds 90% accuracy on key benchmarks but requires substantially higher training cost.
  • Mechanistic efficiency: CF-VLM integrates complete scene-level counterfactuals with minimal targeted interventions across modalities, supplying richer supervision for subtle semantic differences.Its objectives preserve alignment, distinguish factual scenes from coherent alternatives, and detect causal edits.

H.2 Analysis of CF-VLM’s Uniqueness and Integrative Advantages

CF-VLM’s distinctive contribution is an integrated, intervention-aware framework combining multi-level counterfactual data with three complementary objectives. Its synthetic counterfactuals show strong diversity, semantic consistency, distributional similarity, and downstream utility, while synthetic generation remains a stated limitation.

  • Counterfactual design: CF-VLM combines complete jointly edited image-text scenarios with minimally edited image counterfactuals to provide multi-level semantic contrasts.The edits target attributes such as color, category, and state, or causal relations such as action consequences.
  • Integrated objectives: Its three objectives preserve image-text alignment, stabilize factual representations against coherent counterfactuals, and detect semantic shifts from minimal causal edits.This integration spans foundational matching, scene-level discrimination, and low-level causal sensitivity.
  • Causal emphasis: CF-VLM explicitly targets causal decision points whose alteration changes whether an image-text pair semantically matches.This makes the approach intervention-aware rather than relying only on generic hard negatives or augmentation.
  • Practical framework: CF-VLM provides an end-to-end pipeline combining controllable counterfactual generation with integrated learning objectives.The pipeline uses SDXL for image generation and Qwen2-72B-Instruct for text generation.
  • Limitation: The framework relies primarily on synthetic counterfactual generation, which may introduce synthetic-data biases and limit coverage of real-world variation.The authors identify human-in-the-loop editing and stronger validation as future directions.
  • Synthetic-data evaluation: CF-VLM outperforms synthetic baselines across diversity, bias control, semantic consistency, and downstream task performance.Its object-category entropy is 3.90, close to the 4.10 reported for real counterfactuals.
  • Synthetic-data evaluation: CF-VLM achieves 0.10 KL divergence, 0.92 average CLIP cosine similarity, 58.9% ConMe accuracy, and 88.5% ARO accuracy.These results are reported as outperforming the listed synthetic baselines and approaching real counterfactuals.

J.1 Hyperparameter Sensitivity Analysis on the Qwen-VL Backbone

CF-VLM remains broadly stable under Qwen-VL hyperparameter changes, although emphasizing causal loss or tightening margins can trade task gains for optimization instability. The reported accuracy fluctuations are generally moderate, supporting controllable tuning.

  • Loss weights: Reducing α to 0.8 lowers ConMe accuracy to 86.8%, while ARO and VL-Checklist remain 92.8% and 90.1%.The ConMe decrease is 0.8 percentage points from the default.
  • Loss weights: Raising α to 1.2 yields 87.1% ConMe and 92.7% ARO accuracy, with loss variance reduced to 0.048.The passage describes performance as nearly unchanged and training as slightly more stable.
  • Causal-loss weight: Increasing γ to 0.70 raises VL-Checklist from 89.7% to 90.6% but lowers ConMe to 86.6% and increases loss variance to 0.062.The trade-off indicates greater optimization difficulty when causal distinctions receive more weight.
  • Hinge margins: Reducing m1 to 0.15 raises ConMe accuracy to 87.3% but lowers ARO and VL-Checklist performance while increasing loss variance.
  • Overall sensitivity: Most accuracy metrics fluctuate within ±0.5% to 1.0% under Qwen-VL hyperparameter changes, while training variance remains manageable.The authors describe optimization as requiring balance between training stability and convergence efficiency.
  • Backbone transfer: The same sensitivity trends appear on CLIP, where increasing γ improves VL-Checklist to 87.9% but raises loss variance to 0.092.The CLIP analysis reports a similar trade-off between causal-task gains and training instability.
Loading 2506.17267v1…