PaperlayerView exact revision on arXiv

Source-linked AI summary

Probing Mechanical Reasoning in Large Vision Language Models

Haoran Sun, Qingying Gao, Haiyun Lyu, Dezhi Luo, Yijiang Li, Hokin Deng

arXiv:2410.00318v4cs.AIq-bio.NC

TL;DR

Mechanical reasoning is an important human capability, but evidence on VLMs’ performance across its domains is limited. This paper evaluates 26 VLMs on 155 cognitive tasks and finds that models lag humans across domains, with little scaling in gear systems and liquid mechanics.

  • Problem

    Mechanical reasoning is central to human activity, and evaluating it offers a way to assess VLM capabilities and limitations in visuospatial model-based reasoning.

    Source: §1 · p. 1

  • Method

    The study evaluates 26 VLMs using 155 single-image cognitive tasks spanning six mechanical-reasoning domains and compares their performance with a human baseline.

    Source: §1 · pp. 1–2, §2.1 · p. 2, §2.2 · p. 3

  • Results

    VLMs underperform humans across all domains, while Gear Systems and Liquid Mechanics show very weak scaling with model size.

    Source: §3 · p. 4, §3.2 · p. 6, §4 · p. 6

  • Takeaways & Limitations

    The findings suggest current architectures may not support some mechanisms required for mechanical reasoning, particularly in gear systems and liquid mechanics.

    Source: §4 · p. 6, §5 · p. 7

    The proposed model-based-reasoning account does not by itself explain why models perform relatively well on pulley tasks, which also rely on such reasoning.

    Source: §4 · pp. 6–7

Abstract

from arXiv · show

Mechanical reasoning is a hallmark of human intelligence, defined by its ubiquitous yet irreplaceable role in human activities ranging from routine tasks to civil engineering. Embedding machines with mechanical reasoning is therefore an important step towards building human-level artificial intelligence. Here, we leveraged 155 cognitive experiments to test the understanding of system stability, gears and pulley systems, leverage principle, inertia and motion, and fluid mechanics in 26 Vision Language Models (VLMs). Results indicate that VLMs consistently perform worse than humans on all domains, while demonstrate significant difficulty in reasoning about gear systems and fluid mechanics. Notably, their performance on these tasks do not improve as number of parameters increase, suggesting that current attention-based architecture may fail to grasp certain underlying mechanisms required for mechanical reasoning, particularly those pertaining to mental simulations.

1 INTRODUCTION

Mechanical reasoning supports human activity across routine and complex systems, and evaluating it in VLMs can inform discussion of their visuospatial model-based reasoning. This study evaluates VLMs across six mechanical-reasoning domains and reports human superiority and weak scaling in some domains.

1 INTRODUCTION

  • Mechanical reasoning concerns relations and interactions among physical objects and supports human activity from routine tasks to engineering.The paper frames its evaluation as a way to assess VLM capabilities and limitations in this area.Source: §1 · p. 1
  • Mental simulation involves constructing and manipulating internal world models, and people use visuospatial representations to infer how gears and pulleys behave.The study uses 155 single-image tasks adapted from cognitive psychology across six domains: system stability, pulleys, gears, leverage, inertia and motion, and liquid mechanics.Source: §1 · p. 1
  • The paper evaluates 26 open- and closed-source VLMs against a human baseline, reporting consistent human superiority and weak scaling in gear systems and liquid mechanics.Its discussion links performance limitations to deficits in model-based reasoning while also considering architectural constraints and shortcut exploitation.Source: §1 · p. 2

2 METHODS

The study uses single-image adaptations of cognitive tasks to assess six mechanical-reasoning domains, evaluates VLMs in zero-shot image-and-text tasks, and establishes a human baseline using participant responses and a question-screening procedure.

2.1 EXPERIMENT DESIGN

  • The 155 single-image tasks cover stability, pulleys, gears, leverage, inertia and motion, and liquid mechanics.Tasks probe domain-specific physical judgments, including stability, force and motion, gear rotation, balance, object trajectories, and liquid behavior.Source: §2.1 · pp. 2–3

2.2 EXAMINED VISION LANGUAGE MODELS

  • The evaluation includes 26 VLMs across open- and closed-source and multi-image and single-image categories, tested zero-shot on images and text.Models were asked to explain their answers after responding.Source: §2.2 · pp. 3–4

2.3 HUMAN BASELINE

  • Nine English-proficient college students formed the human baseline, skipping questions they found ambiguous or too complex within 90 seconds.Questions fewer than 80% of participants could answer in time were marked failed; otherwise, questions were modified and tested by new annotators.Source: §2.3 · p. 4

3 RESULTS

Humans outperform VLMs overall and across every evaluated domain, with the largest reported gap in pulley systems. Larger models tend to perform better in many domains, but gear systems and liquid mechanics show almost no scaling relationship.

3 RESULTS

  • Human participants significantly outperformed models overall and in every domain; pulley accuracy was nearly 90% for humans versus about 50% for models.The overall difference had p < 0.0001, and each domain comparison had p < 0.001.Source: §3 · p. 4, Figure 3 · p. 5

3.1 HUMAN VS. MODEL PERFORMANCE ACROSS EACH DOMAIN

  • Human performance differed significantly across domains, with pulley performance higher than the other five, while model accuracy showed no significant category differences except for liquid mechanics.Human category differences had F = 11.8111, p < 0.0001; model category differences had F = 1.6809, p = 0.1430.Source: §3.1 · pp. 4–5, Figure 4 · p. 5

3.2 RELATIONSHIP BETWEEN MODEL PERFORMANCE AND MODEL SIZE

  • Larger models tended to improve accuracy in most domains, but scaling was weak for Gear Systems (slope 0.0010, R² = 0.000023) and Liquid Mechanics (slope 0.0169, R² = 0.003097).Both reported slopes had p-values close to one: 0.981628 for Gear Systems and 0.787137 for Liquid Mechanics.Source: §3.2 · p. 6, Figure 5 · p. 6

4 DISCUSSIONS

VLMs lag human performance across mechanical-reasoning domains, with gear and liquid-mechanics performance near chance and unresponsive to model size. The discussion considers domain-specific physical primitives and shortcut learning as possible explanations for differences across tasks.

4 DISCUSSIONS

  • Gear Systems and Liquid Mechanics remain near chance and do not improve with model size, despite larger models tending to achieve higher overall accuracy.The authors suggest these patterns may reflect limits in mechanisms required for mechanical reasoning.Source: §4 · p. 6
  • The discussion cautions that the model-based-reasoning account does not by itself explain why pulley performance differs from other domains.Pulley systems are also described as relying on model-based reasoning.Source: §4 · pp. 6–7
  • Different domains recruit distinct intuitive-physics primitives, so models may show some model-based reasoning yet fail to apply the appropriate primitives.The authors contrast liquid-specific priors with those governing solids.Source: §4 · p. 7
  • Pulley performance may appear stronger because educational materials and simpler systems could support training-data pattern matching instead of genuine simulation.The authors note that very large models, such as GPT-4o, perform particularly well on pulley tasks, while smaller models lag.Source: §4 · p. 7

5 CONCLUSION

The paper assesses VLM mechanical reasoning and finds incomplete competency across domains, with some tasks showing no improvement as model size increases. It identifies model-based reasoning as a highlighted limitation while noting that the mechanisms remain unclear.

5 CONCLUSION

  • Current VLMs do not fully acquire the competencies needed for mechanical reasoning across the assessed domains.The authors describe this as the finding of their comprehensive assessment.Source: §5 · p. 7
  • Performance on certain tasks does not improve with increased model size, suggesting potential limitations in existing model architectures.Source: §5 · p. 7
  • The authors highlight model-based reasoning as a relevant limitation, but say the precise mechanisms underlying VLM limitations remain unclear.They identify further exploration of those mechanisms as potentially informative for architecture development.Source: §5 · p. 7

A.1 SYSTEM STABILITY

The supplied figure captions identify Figure 6 as covering system-stability reasoning and Figure 7 as covering pulley systems. The accompanying results describe cases where models identify objects or mechanical states yet still answer incorrectly.

A.1 SYSTEM STABILITY

  • Figure 6 is identified as showing VLM performance on mechanical system stability in MechBench.Source: Figure 6 · p. 13
  • Models can identify objects and recognize mechanical states but still fail to infer the correct mechanical outcome.In the stool example, the model reverses the relation between wider leg angles and stability; in the bottle example, it misses the distinction between standing and rolling.Source: §A.1 · p. 13
  • Figure 7 is identified as showing VLM performance on pulley systems in MechBench.Source: Figure 7 · p. 14

A.2 PULLEY SYSTEMS

VLMs struggle with pulley systems, especially identifying movable pulleys and predicting whether suspended objects rise or fall.

A.2 PULLEY SYSTEMS

  • VLMs struggle to identify movable pulleys and predict whether objects move up or down through pulley systems.The authors describe these as recurring failures in pulley reasoning.Source: §A.2 · p. 14
  • In Experiment 7A, models selected the wrong pulley system, and in 7D they gave the opposite of the correct movement prediction.The authors hypothesize that movement errors may stem from difficulty recognizing movable pulleys, but say further experiments are needed.Source: §A.2 · p. 14
  • People with some mechanical experience can identify simple pulley systems from basic diagrams more readily than VLMs.The comparison particularly applies to Experiments 7A and 7D.Source: §A.2 · p. 14

A.3 GEAR SYSTEMS

VLMs make accurate gear and conveyor-belt judgments, including determining one pulley’s rotation from another’s direction.

A.3 GEAR SYSTEMS

  • VLMs give highly accurate judgments on gear and conveyor-belt problems, including the rotation direction of one pulley from another’s.The authors report this contrast with pulley-system performance.Source: §A.3 · p. 15
  • The authors speculate that fixed, straightforward gear layouts reduce complexity and can be represented as logical diagrams.They present this as one possible reason for stronger gear-system performance than pulley-system performance.Source: §A.3 · p. 15
  • Gear reasoning can use two stated rules: adjacent gears rotate oppositely, and smaller gears rotate faster than larger ones.The authors suggest these rules can be applied as fixed logical principles.Source: §A.3 · p. 15

A.4 SEESAW-LIKE SYSTEMS AND LEVERAGE PRINCIPLE

Across leverage, inertia and motion, and liquid-mechanics tasks, VLMs showed uneven performance, handling some simpler scenes while struggling with more complex inference.

A.4 SEESAW-LIKE SYSTEMS AND LEVERAGE PRINCIPLE

  • VLMs handled several simple leverage tasks well but failed on two tasks the authors associate with multi-step reasoning.In Experiment 9B, they failed to infer that sliding backward on a seesaw increases leverage and is needed to level it.Source: §A.4 · p. 16

A.5 INERTIA AND MOTION

  • VLMs could identify inertia-and-motion situations but often struggled to predict what would happen next.The section contrasts straightforward single-change extrapolations with more complex tasks involving additional objects, principles, or force changes.Source: §A.5 · p. 17, Figure 10 · p. 17
  • In the pendulum task, the model confused the marked positions; the proposed perceptual-constancy explanation remains unconfirmed.The authors suggest it may have mistaken a half-arc for a full pendulum motion and call for further experimentation.Source: §A.5 · p. 17

A.6 LIQUID MECHANICS

  • Liquid-mechanics experiments covered fluid flow, buoyancy, and volume, and the authors report persistent difficulty with complex inference alongside strong scene understanding.Figure 11 presents VLM performance on liquid-mechanics reasoning.Source: §A.6 · p. 18, Figure 11 · p. 18
  • In Experiment 11B, the model missed that water stops flowing when it reaches the bucket’s wall hole; in 11C, its explanation was incorrect despite a correct answer.The authors attribute the 11B error to missing the hole’s location.Source: §A.6 · p. 18
  • The model captured detailed measuring-cup features in 11D and used gas and liquid density in its explanation of 11F.The authors cite these as examples of strong image comprehension and detail recognition.Source: §A.6 · p. 18

Related papers

Loading 2410.00318v4…