PaperlayerView exact revision on arXiv

Source-linked AI summary

What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models

Letian Zhang, Xiaotong Zhai, Zhongkai Zhao, Yongshuo Zong, Xin Wen, Bingchen Zhao

arXiv:2310.06627v4cs.CLcs.CVcs.LG

TL;DR

The paper asks whether current multi-modal language models can reason about counterfactuals, an ability that existing VQA benchmarks do not directly measure. It builds C-VQA by adding counterfactual presuppositions to real and synthetic visual questions. Evaluations find that no model consistently handles these questions.

  • Problem

    Existing VQA benchmarks do not directly test whether MLLMs can imagine what-if scenarios that differ from what actually happened.

    Source: §1 · pp. 1–2

  • Method

    The authors introduce C-VQA, a benchmark of real and synthetic visual questions modified with counterfactual presuppositions.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    No model in the current multi-modal model literature consistently handles C-VQA's counterfactual questions.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors release C-VQA and its code as a benchmark for evaluating progress in counterfactual reasoning.

    Source: §5 · p. 8

    The number of images in C-VQA is a limiting factor, and the authors plan to gather more image annotations and evaluate more models.

    Source: F. Limitations, future works, and… · p. 15

Abstract

from arXiv · show

Counterfactual reasoning, a fundamental aspect of human cognition, involves contemplating alternatives to established facts or past events, significantly enhancing our abilities in planning and decision-making. In light of the advancements in current multi-modal large language models, we explore their effectiveness in counterfactual reasoning. To facilitate this investigation, we introduce a novel dataset, C-VQA, specifically designed to test the counterfactual reasoning capabilities of modern multi-modal large language models. This dataset is constructed by infusing original questions with counterfactual presuppositions, spanning various types such as numerical and boolean queries. It encompasses a mix of real and synthetic data, representing a wide range of difficulty levels. Our thorough evaluations of contemporary vision-language models using this dataset have revealed substantial performance drops, with some models showing up to a 40% decrease, highlighting a significant gap between current models and human-like vision reasoning capabilities. We hope our dataset will serve as a vital benchmark for evaluating the counterfactual reasoning capabilities of models. Code and dataset are publicly available at https://bzhao.me/C-VQA/.

1. Introduction

The paper asks whether contemporary multi-modal language models can reason about counterfactual scenarios, which existing VQA benchmarks do not directly test. It introduces C-VQA, a benchmark that adds counterfactual presuppositions to questions, and reports that evaluated models struggle with these questions.

1. Introduction

  • The paper frames counterfactual ability as a human cognitive capacity supporting decision-making, moral reasoning, and problem-solving, and asks whether contemporary MLLMs possess it.The question follows recent MLLM advances in image recognition, image-based dialogue, and language grounding.Source: §1 · p. 1
  • Earlier VQA benchmarks assess abilities such as visual recognition, external knowledge, and compositional reasoning, while CF-VQA and VQA-CP test train–test property-distribution shifts rather than imagined what-if scenarios.The paper identifies this as a gap in evaluating MLLMs' counterfactual reasoning.Source: §1 · pp. 1–2
  • C-VQA adds counterfactual presuppositions to visual questions, requiring models to reason about image content after an assumed event or change.For example, if the TV were off, a cat shown on its screen would no longer be visible.Source: §1 · p. 2
  • No evaluated model family consistently handles C-VQA questions; models show large performance drops, and GPT-4V cannot solve the benchmark.The experiments also report that neuro-symbolic models perform worse than end-to-end models on complex counterfactual reasoning and find systematic bias on gender-related counterfactual questions.Source: §1 · p. 2
  • C-VQA combines real and synthetic image-question pairs, with questions spanning different difficulties and including numerical and boolean queries.The paper builds on VQAv2 and also creates synthetic images and questions to probe out-of-distribution ability.Source: §1 · p. 2

2. Related Works

Related work covers visual question answering, reasoning benchmarks, and general multi-modal benchmarks. The paper distinguishes C-VQA by adding counterfactual presuppositions directly to visual questions, targeting reasoning about what-if scenarios.

Visual Question Answering

  • VQA benchmarks span daily-life questions, compositional reasoning, and image understanding, with datasets including COCO-QA, VQA, CLEVR, GQA, and OKVQA.CLEVR uses synthetic data, while GQA uses scene graphs to generate reasoning questions on real-world images.Source: §2 · pp. 2–3
  • C-VQA modifies VQAv2 questions and answers with counterfactual presuppositions to test scene parsing and reasoning about the observed world after the presupposition.The authors describe this as a complementary evaluation scenario for MLLMs.Source: §2 · p. 3

Evaluation of Reasoning Abilities

  • CF-VQA and VQA-CP test counterfactuals defined by differing property distributions between training and test data, unlike C-VQA's counterfactual presuppositions in question text.The paper also contrasts its target with IfQA and LLM evaluations of counterfactual reasoning.Source: §2 · p. 3

Multi-Modal LLM Benchmarks

  • Other MLLM benchmarks cover perception and cognition, multiple ability dimensions, image and video questions, or emergent vision-language tasks; the paper presents counterfactual reasoning as an orthogonal evaluation direction.Examples include MME, MM-Bench, SEED-Bench, and MM-Vet.Source: §2 · p. 3

3. Dataset

C-VQA combines real and synthetic image-question pairs with counterfactual presuppositions, covering numerical and boolean questions. Its construction includes manually checked real questions and procedurally generated synthetic tasks.

3. Dataset

  • C-VQA contains 3,144 real pairs from VQAv2 and 3,000 automatically generated synthetic pairs, with numerical and boolean questions in both parts.Each synthetic image has an original and a counterfactual question.Source: §3 · p. 3

3.1. Annotation

  • Real questions use direct numerical changes, indirect changes requiring more reasoning, or boolean presuppositions designed to reverse the fact and answer.For example, turning off a TV removes the cats shown on its screen from the imagined scene.Source: §3.1 · p. 3
  • Real annotations begin with 200 manually labeled examples per group, then use ChatGPT prompts; manual review checks image relevance, answer reasonability, ambiguity, and indirect numerical questions.Boolean generation flips the answer first because random counterfactual presuppositions often fail to change it; at least two people verify each image.Source: §3.1 · p. 4, §3.2 · p. 4

3.3. Implementation of C-VQA-Synthetic

  • Synthetic C-VQA uses six templates with 500 instances each; flower tasks combine color and polygon regions, while dot tasks include counterfactual removals and counting.The flower questions test spatial relationships alongside counterfactual changes; the six templates span flower-counting and dot-counting tasks.Source: §3.3 · p. 5, Table 2 · p. 5

3.4. Dataset Statistics

  • C-VQA-Real has 3,144 questions, including 2,014 numerical and 1,130 boolean questions; synthetic questions use multiple-choice answers, with options balanced near 25% each.Real boolean answers are 55.13% “no” and 44.87% “yes”; real and synthetic counterfactual questions average 13.15 and 25.33 words, respectively.Source: §3.4 · pp. 5–6
  • All evaluated models in Table 3 show significant performance drops on counterfactual questions across three C-VQA-Real subgroups.The caption names ViperGPT, VisProg, BLIP2, InstructBLIP, MiniGPT4, LLaVA, CogVLM, and Qwen-VL.Source: Table 3 · p. 6

4. Experiments

Across real and synthetic C-VQA evaluations, current model families struggle to handle counterfactual questions consistently. The experiments also report performance differences across reasoning approaches, synthetic inputs, and gender subgroups.

4.1. Experiments on C-VQA-Real

  • End-to-end models perform worse on counterfactual questions across all three groups, while chain-of-thought prompting leaves performance unchanged or lowers it.The accuracy gap is smaller for numerical direct questions, which require only one simple reasoning step.Source: §4.1 · p. 6
  • Across code-generation LLMs in ViperGPT, counterfactuals produce consistent performance drops; CodeLlama's decrease is moderate, while WizardCoder performs worst in both settings.GPT-3.5-turbo generally performs best in the original setting, but also drops substantially on counterfactuals.Source: §4.1 · p. 7, Table 6 · p. 8
  • No model family—end-to-end or neuro-symbolic—consistently handles counterfactual questions well.The authors characterize C-VQA as challenging and call for further exploration.Source: §4.1 · p. 7
  • GPT-4V shows over 10% lower performance for all three counterfactual types on a randomly selected 10% subset of C-VQA-Real.The table caption cautions that these results are not directly comparable with other tables.Source: §4.1 · p. 7, Table 5 · p. 7

4.2. Experiments on C-VQA-Synthetic

  • On C-VQA-Synthetic, InstructBLIP (Vicuna-7B) scores at random level (25%), while InstructBLIP (FlanT5XXL) and BLIP2 (FlanT5XXL) score below random.InstructBLIP (Vicuna) also gives random answers when asked merely to count flowers, indicating it cannot recognize flowers in the synthetic images.Source: §4.2 · p. 7, Table 7 · p. 8
  • Neuro-symbolic models perform better in reasoning but lack available APIs for some synthetic-image queries, while ViperGPT can often generate correct code with counterfactual presuppositions.For example, ViperGPT lacks tools to check whether a flower is in a polygon; the authors conclude that current models do not generalize beyond real-image training domains.Source: §4.2 · p. 7

4.3. Bias Analysis

  • In gender-subgroup analysis, end-to-end models have larger counterfactual accuracy drops for female than male images, a pattern reported as systematic gender-related bias.The analysis covers 717 images: 510 male and 207 female; Figure 5 describes end-to-end bias toward the male subgroup and neuro-symbolic bias toward the female subgroup. Figure 4 shows InstructBLIP answering a male instance correctly but a similar female instance incorrectly.Source: §4.3 · p. 8, Figure 5 · p. 8, Figure 4 · p. 8

5. Conclusion

The paper introduces C-VQA to evaluate counterfactual reasoning in multimodal language models using real and synthetic images and questions. Its evaluations find that no current model in the literature consistently handles the benchmark's counterfactual questions.

5. Conclusion

  • C-VQA combines real and synthetic images and questions to evaluate multimodal models, and no model in the current literature consistently handles its counterfactual questions.The authors released the code and dataset to support further work toward human-level multimodal intelligence.Source: §5 · p. 8

A. Overview

The supplementary material provides question-generation prompts, qualitative model results, and additional dataset statistics. It includes examples from both end-to-end and ViperGPT evaluations.

A. Overview

  • The supplement provides ChatGPT prompts for generating numerical-direct and boolean questions, qualitative end-to-end and ViperGPT results, and further C-VQA statistics and noun and verb top words.Source: A. Overview · p. 11

B. Prompt for ChatGPT

The appendix describes separate ChatGPT prompting strategies for numerical-direct and boolean counterfactual questions. Both use stepwise generation, but the boolean strategy changes after the shared approach often fails to flip the answer.

B. Prompt for ChatGPT

  • For both numerical-direct and boolean questions, the authors use chain-of-thought prompting and in-context examples to generate counterfactual modifications.They use different prompt patterns for the two question groups.Source: B. Prompt for ChatGPT · p. 11

B.1. Numerical Direct Group

  • For numerical-direct questions, ChatGPT is asked to change the item count and work out the resulting answer through simple calculations.The new counterfactual answer is intended to differ from the original answer.Source: §B.1 · p. 11
  • The numerical-direct prompt requests an original question and answer, a count change, its effect, and a counterfactual question with a single-number answer.Source: §B.1 · p. 11

B.2. Boolean Group

  • For boolean questions, the shared numerical-direct prompt often fails to flip the original answer, so the authors change the chain-of-thought strategy.Source: §B.2 · p. 12
  • The revised boolean strategy first flips the yes/no answer, then constructs a counterfactual situation that would make the flipped answer true.The prompt asks ChatGPT to produce the modified question and answer.Source: §B.2 · pp. 12–13
  • Boolean prompt examples turn a changed state into a hypothetical condition, such as asking whether the sky would be clear if it were cloudy.Source: §B.2 · p. 14

C. Qualitative Result of End-to-end Models

Most models fail to answer correctly when counterfactuals are added. LLaVA-13B shows an unusual pattern: it answers some counterfactual questions correctly while missing the corresponding original questions.

C. Qualitative Result of End-to-end Models

  • Most models fail to provide correct answers when counterfactuals are added.Figure 6 presents common end-to-end model failure cases.Source: C. Qualitative Result of End-to-end… · p. 14, Figure 6 · p. 14
  • LLaVA-13B (Vicuna-13B) scores 31.2% correct on both original and counterfactual numerical direct questions, despite often getting the counterfactual answer right and the original answer wrong.The authors inspect its results and show examples in Figure 7.Source: C. Qualitative Result of End-to-end… · p. 14
  • Figure 6 collects common failure cases of end-to-end models on C-VQA.The caption identifies added counterfactual presuppositions in bold.Source: Figure 6 · p. 14

D. Qualitative Result of Neuro-symbolic Models

ViperGPT's generated code sometimes misunderstands or ignores counterfactuals, while its synthetic-data code can encode them correctly despite a specific visual-checking limitation.

D. Qualitative Result of Neuro-symbolic Models

  • ChatGPT-generated ViperGPT code sometimes misunderstands or entirely ignores counterfactuals.Figure 9 contrasts logically wrong codes for C-VQA-Real with logically correct codes for C-VQA-Synthetic.Source: D. Qualitative Result of… · p. 14
  • ViperGPT cannot check whether a flower lies inside a polygon in C-VQA-Synthetic.Despite this limitation, its generated code can often be correct when counterfactual presuppositions are added.Source: D. Qualitative Result of… · p. 14
  • The synthetic-data code examples are described as inspiring because they can often represent added counterfactual presuppositions correctly.Figure 9 shows examples of the generated codes.Source: D. Qualitative Result of… · p. 14
  • LLaVA-13B examples answer counterfactual questions correctly but fail on the original questions.The examples are shown in Figure 7.Source: Figure 7 · p. 15

E. Additional Detailed Statistics

C-VQA has a higher noun and verb type-token ratio than other common datasets; the paper attributes this to duplicated wording between presuppositional statements and main clauses.

E. Additional Detailed Statistics

F. Limitations, future works, and broader impact

The authors identify C-VQA's image count as a limitation, while reporting a performance drop on its real-image and synthetic data and proposing broader annotation and model coverage as future work.

F. Limitations, future works, and broader impact

Loading 2310.06627v4…