PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Language Models Cannot Reason About Physical Transformation

Dezhi Luo, Yijiang Li, Maijunxian Wang, Tianwei Zhao, Bingyang Wang, Siheng Wang, Pinyuan Feng, Pooyan Rahmanzadehgervi, Ziqiao Ma, Hokin Deng

arXiv:2603.07109v2cs.AI

TL;DR

Whether VLMs genuinely understand physical transformations remains unclear, despite the importance of this ability for reasoning in dynamic environments. ConservationBench evaluates this ability across paired conserving and non-conserving scenarios, and its results show that current VLMs struggle to maintain transformation-invariant representations across dynamic scenes.

  • Problem

    It remains unclear whether VLMs can reason about physical transformations, where properties may remain invariant or change despite altered appearance.

    Source: §1 · p. 1

  • Method

    ConservationBench tests four quantitative properties using paired conserving and non-conserving scenarios, evaluating 112 VLMs across 23,040 trials.

    Source: §1 · p. 2

  • Results

    VLMs struggle to track conserved properties across dynamic scenes, and conservation-task gains often accompany worse performance on matched non-conserving controls.

    Source: §1 · p. 2, §4.3 · p. 5

  • Takeaways & Limitations

    Current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.

    Source: §6 · p. 9

    The evaluation covers four quantitative properties under controlled laboratory conditions; more complex scenarios remain for future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Understanding physical transformations is fundamental for reasoning in dynamic environments. While Vision Language Models (VLMs) show promise in embodied applications, whether they genuinely understand physical transformations remains unclear. We introduce ConservationBench evaluating conservation -- whether physical quantities remain invariant under transformations. Spanning four properties with paired conserving/non-conserving scenarios, we generate and evaluate 23,040 questions across 112 VLMs. Results reveal systematic failure: performance remains near chance with improvements on conservation tasks accompanied by drops on controls. Control experiments show strong textual priors favoring invariance, yet models perform worse with actual visual content when performance is balanced across conserving and non-conserving scenarios. Neither temporal resolution, prompting, nor curated sampling helps. These findings show that current VLMs fail to maintain transformation-invariant representations of physical properties across dynamic scenes.

1. Introduction

The paper asks whether VLMs can track physical properties that may remain invariant across transformations, a capability relevant to embodied physical reasoning. It introduces ConservationBench and finds that models struggle to track conserved properties across dynamic scenes.

1. Introduction

  • Existing evaluations leave underexplored whether VLMs can reason about properties that remain invariant—or change—during physical transformations.Prior work emphasizes video generation or physical properties in static scenes.Source: §1 · p. 1
  • Conservation requires recognizing that physical quantities remain invariant despite changes in appearance, using visual as well as conceptual representations.The paper gives pouring water into a differently shaped glass without changing its volume as an example.Source: §1 · p. 1
  • ConservationBench evaluates four properties—number, length, volume, and size—with matched conserving and non-conserving tasks across 60 conditions and 23,040 trials.The benchmark varies frame extraction, temporal resolution, and prompting strategy.Source: §1 · p. 2
  • VLMs consistently fail to integrate temporal information to track conserved properties across dynamic scenes.High conservation-task accuracy can reflect heuristics that reverse on non-conserving scenarios; prompting and higher temporal resolution do not help.Source: §1 · p. 2

2. Related Works

Prior work assesses quantity understanding in static scenes and physical understanding through outcome prediction or descriptive inference, leaving transformation-invariant reasoning underexplored. The paper frames conservation as a cognitive capacity involving stable representations despite perceptual change.

2. Related Works

  • Quantity benchmarks often test counting or individuation in static scenes, not numerical invariance across transformations.The paper distinguishes perceptual counting from representing quantity as transformation-invariant.Source: §2 · p. 2
  • Conservation is presented as a foundational capacity for higher-level physical reasoning, requiring transformation-invariant representations and suppression of misleading perceptual cues.The cited developmental and behavioral work also links performance to sensorimotor grounding and inhibitory control.Source: §2 · p. 2
  • Physical-understanding studies emphasize outcome prediction or descriptive inference, leaving open whether VLMs track sequential evidence while maintaining stable property representations.The paper identifies this ability as the target of conservation tasks.Source: §2 · pp. 2–3

3. Experimental Design

The experimental design uses videos of physical transformations, matched quantity-changing controls, and varied frame inputs and selection strategies to assess VLM conservation reasoning.

3.1. Conservation Tasks

  • Each conservation video moves from an initial state through an observable transformation to a final state with altered appearance, across number, length, volume, and size tasks.The design varies object count, size, color, layout, container shape, and transformation direction across 192 videos.Source: §3.1 · p. 3
  • Volume and size tasks require witnessing the transformation, whereas number and length tasks can be judged from initial and final states.This distinction diagnoses whether performance may rely on static cues rather than representations of the process.Source: §3.1 · p. 3

3.2. Non-conserving Tasks

  • Matched non-conserving controls change the target quantity while keeping irrelevant features and visual contexts constant.The 192 control videos pair with conservation tasks to test sensitivity to quantity changes rather than invariance defaults.Source: §3.2 · p. 3

3.3. Adaptation to Multi-frame Input

  • Frame-count conditions vary temporal detail, from three frames to 5, 7, and 9 frames and a 16-frame condition; all include initial and final states.The conditions test whether more temporal information helps models extract transformation-relevant cues.Source: §3.3 · pp. 3–4
  • The study compares uniform sampling, human-selected intermediate frames from 18 annotators, and model-based keyframe selection with a BLIP-2 Localizer.The model-based strategy selects query-relevant frames before an Answerer makes its inference.Source: §3.3 · p. 4

4. Experiments

Across 112 VLMs, ConservationBench exposes weak transformation reasoning, strong invariance bias, and failures that persist across prompting, temporal-resolution, and scaling tests. Human accuracy exceeds 98%, while mechanistic analyses associate dominant errors with anchoring on the initial frame and failing to update after transformations.

4.1. Inference and Evaluation

  • The 112-model evaluation used reference configurations and shuffled, counterbalanced answer choices, with free-form responses mapped and scored in two stages.Unambiguous outputs were matched to options first; unresolved outputs were adjudicated by a constrained LLM judge, and persistent high-FAIL models were excluded.Source: §4.1 · p. 4

4.3. Main Results

  • Across 112 VLMs, accuracy ranges from ∼20% to ∼69%, mostly near 33.3% chance; 82/112 (73.2%) score below chance on strict pairwise accuracy, and only three exceed chance.Better conservation performance tends to accompany worse performance on matched non-conserving controls; textual priors favor invariance, while real visual content disrupts rather than enables conservation judgments.Source: Figure 2 · p. 5, §4.3 · pp. 5–6

4.2. Human Baseline

  • Humans reached 98.35% accuracy on 864 balanced questions, validating the benchmark’s human adaptation.Six participants remained after one was excluded for chance-level performance on a consecutive block.Source: §4.2 · pp. 4–5

4.4. Different Prompting Strategies, Frame Numbers, and Sampling Methods

  • Across prompt, frame-count, and sampling tests, extra frames and prompting generally did not help; uniform sampling outperformed curated frames on Volume & Size tasks.Continuous prompts modestly helped Number & Length, while chain-of-thought prompting hurt; prompt and frame-count null effects persisted among the top 15 models.Source: Figure 4 · p. 7, §4.4 · pp. 7–8

4.5. Does Scaling of Model Size Help?

  • At seven frames, model size predicted conservation accuracy poorly (R2 = 0.019) and non-conserving accuracy modestly (R2 = 0.239); physically oriented post-training did not resolve the failures.The size analysis covered models ranging from 1B to 76B parameters; the non-conserving relationship accounted for less than 24% of variance.Source: Figure 5 · p. 8, §4.5 · p. 8

4.6. Relationship to Other Physical Reasoning Benchmarks

  • ConservationBench correlated substantially with all nine evaluated physical-reasoning and video-understanding benchmarks, which the authors interpret as a shared general factor.The authors describe its controlled design and richer contextual benchmarks as complementary.Source: Figure 6 · p. 8, §4.6 · p. 8

4.7. Mechanistic Analysis of Failure Modes

  • On non-conserving trials, confident “same” errors were linked to greater attention to Frame 1 and less attention to later frames, consistent with anchoring on the initial state.The mechanistic analysis is preliminary, uses one model, and lacks causal verification through targeted interventions.Source: §4.7 · pp. 8–9

5. Discussion

ConservationBench reveals systematic failures in transformation reasoning, including heuristic reversals between conserving and non-conserving scenarios. The authors argue that robust progress requires models to represent visual states over time.

5. Discussion

  • Models show heuristic reversals, and control experiments indicate that textual priors dominate while visual evidence fails to correct them.Increased temporal resolution, targeted prompting, and human-curated frame sampling do not induce robust transformation reasoning.Source: §5 · p. 9
  • The tasks can serve as enduring sanity checks because their failure modes persist even as models advance on high-level physical reasoning benchmarks.Source: §5 · p. 9
  • The authors suggest predictive, state-based visual abstractions rather than static semantic features, while noting that further mechanistic investigation is needed.They caution that whether conservation deficits impair planning, tool use, or robotic manipulation remains an open empirical question.Source: §5 · p. 9

6. Conclusion and Limitation

The paper introduces ConservationBench and identifies limitations in both its controlled evaluation scope and the evidence about underlying mechanisms and downstream effects.

6. Conclusion and Limitation

  • ConservationBench evaluates whether physical quantities remain invariant under transformations despite changes in appearance.The benchmark is presented as an enduring diagnostic test for transformation reasoning.Source: §6 · p. 9
  • The evaluation covers four quantitative properties under controlled laboratory conditions; occlusions, deformable objects, noisy observations, and ambiguous transformations are reserved for future work.Source: §6 · p. 9

A. Data Curation

  • Curation used three college-trained annotators, two independent cross-review passes per video, and removal or revision of items that failed design criteria.Source: A. Data Curation · p. 16
  • Videos used a fixed camera, consistent lighting and backgrounds within each property category, and scripted transformations intended to be clear, reproducible, and minimally ambiguous.Source: A. Data Curation · p. 16
  • The item design targets discriminativeness while minimizing confounds and textual shortcuts, requiring multimodal reasoning.Source: A. Data Curation · p. 16

B. Task Design

The benchmark pairs conservation tasks with matched controls, tests prompt strategies for engaging with visual dynamics, and provides a seven-frame example input. It evaluates a broad set of models using reference implementations.

B. Task Design

  • Task descriptions pair conservation scenarios with matched non-conserving controls across four quantitative properties.Table 2 presents the paired descriptions and corresponding illustrations in Figure 2.Source: B. Task Design · p. 16, Table 2 · p. 16

C. Prompting Strategy

  • Four prompt types progressively cue the continuous transformation, including sequential frame-by-frame processing and an explicit continuous-process framing.Sequential and chain-of-thought prompts direct attention to frame-wise evidence; the continuous prompt offers a conceptual cue for conservation reasoning.Source: C. Prompting Strategy · p. 16, Table 3 · p. 17

D. Example Input

  • The example uses uniform extraction, seven frames, and a direct-question prompt, with image placeholders marking the frames in the textual input.Source: D. Example Input · p. 17
  • The example presents seven chronologically ordered frames of coins, showing one row spread out while the number of coins remains unchanged.The direct question asks whether the final rows have the same number of coins.Source: D. Example Input · pp. 17–18, Figure 7 · p. 17

E. Model Inference

  • Evaluation spans 112 models, including proprietary and open-source systems from 1B to 76B parameters, using official configurations and implementations.The inference framework also verifies that models receive the intended media input through human-reviewed modality checks.Source: E. Model Inference · p. 18

F. Evaluation

The evaluation combines deterministic answer matching with LLM adjudication for unresolved outputs and uses counterbalanced task variations across the four properties.

F. Evaluation

  • Hybrid Matching prioritizes deterministic template extraction, then uses four LLM judges when needed and accepts a mapping only with agreement from at least three.The passage describes this approach as balancing template precision with flexibility for varied response styles.Source: F. Evaluation · p. 18

G. Counterbalancing Conditions

H. Complete Model Results Aggregated Across Domains and Conditions

Models consistently perform worse on non-conserving controls than on conservation tasks, and neither conceptual prompting nor caption-based input yields balanced performance. Control-condition accuracy correlates positively with conservation accuracy, including when models can abstain.

H. Complete Model Results Aggregated Across Domains and Conditions

I. Combined Model Performance By Quantitative properties

J. Additional Prompting and Input Modality Controls

K. Model response under empty image and text control

  • For 62 VLMs, conservation accuracy correlates with Empty Image Control accuracy (r = 0.578) and Text Control accuracy (r = 0.475).The authors interpret these correlations as evidence of textual priors favoring quantity invariance when visual content is removed.Source: Figure 12 · p. 25

L. Empty Image and Text Control with “I Don’t Know” Option

M. Prompting, Frame Count, and Extraction Method Analysis on Top-15 Models

Among the top 15 models, neither prompt type nor frame count has a significant effect on either task category; the extraction-method effect on Volume and Size persists.

M. Prompting, Frame Count, and Extraction Method Analysis on Top-15 Models

  • No significant prompt-type or frame-count effects appear for either task category among the top 15 models; SeViLA yields significantly worse Volume & Size performance (p < 0.01).The extraction-method effect observed in the full sample is preserved in this restricted analysis.Source: Figure 14 · p. 26, M. Prompting, Frame Count, and… · p. 26

N. Evaluation of Cosmos Reason

Cosmos Reason, a model post-trained with physical common-sense and embodied-reasoning data, shows the same conservation/non-conservation performance asymmetry as other models.

N. Evaluation of Cosmos Reason

  • Cosmos Reason has near-zero non-conserving accuracy on Number (0.003) and Length (0.045), and below-chance accuracy on Volume (0.229) and Size (0.108).The characteristic asymmetry between conservation and non-conserving performance persists despite physical common-sense post-training.Source: Figure 15 · p. 26, N. Evaluation of Cosmos Reason · p. 26

O. Mechanistic Analysis: Confidence and Attention

Mechanistic analyses link non-conserving errors to greater confidence and attention anchored on the initial frame. A qualitative example likewise shows frame-by-frame matching instead of reasoning about the transformation as a continuous process.

O. Mechanistic Analysis: Confidence and Attention

  • On Qwen2.5-VL-7B-Instruct, non-conserving “same” errors are more confident than correct non-conserving predictions, consistent with overriding a default invariance prior being difficult.Decision confidence is measured as the log-probability margin between the top two answer choices.Source: Figure 16 · p. 27, O. Mechanistic Analysis: Confidence… · p. 27
  • For Qwen2.5-VL-7B-Instruct, non-conserving “same” errors attend more to Frame 1 and less to later frames in layers 20–27, unlike other difference maps.The authors identify this initial-frame anchoring as specific to the dominant failure mode.Source: Figure 17 · p. 28, O. Mechanistic Analysis: Confidence… · p. 28

P. Qualitative Analysis of Model Reasoning Traces

  • In a non-conserving volume example, Claude Sonnet 4.5 compares liquid levels frame by frame and misses that partial transfer should determine whether the quantity changed.The example illustrates static perceptual matching rather than representing the transfer as a continuous physical process.Source: P. Qualitative Analysis of Model… · p. 29, Figure 18

Related papers

Loading 2603.07109v2…