Source-linked AI summary
Evaluation of Small Vision-Language Models on Qualitative Mechanical Problems
Henry Fordjour Ansah, Shreya Banerjee, Pranish Ghimire
TL;DR
The paper examines whether small vision-language models can reliably solve qualitative mechanical problems requiring visual, spatial, commonsense, and qualitative reasoning. It evaluates Gemma and Qwen on 30 benchmark problems, finding that accurate and flawless reasoning remains limited while flawed reasoning can still produce correct answers.
Problem
Small vision-language models need evaluation on visually presented qualitative mechanical problems requiring integrated perception and reasoning for broader real-time deployment.
Method
The authors evaluate Gemma and Qwen on 30 curated Bennett Mechanical Comprehension Test problems, comparing elicited reasoning chains and final answers with verified solutions.
Results
Accurate and flawless reasoning occurred in 30% of Gemma cases and 40% of Qwen cases, while both models also produced correct answers despite flawed reasoning in 6.67% of cases.
Takeaways & Limitations
Answer-level accuracy alone is insufficient for assessing understanding and reliability, motivating verifiable justifications and improvements to both perceptual and reasoning capabilities.
Takeaways & Limitations
The models hallucinated nonexistent dependencies and misapplied physical laws, making subsequent inferences inaccurate and exposing brittleness in qualitative reasoning.
Abstract
from arXiv · showhide
Qualitative mechanical problem-solving (QMPS) refers to solving qualitative problems from the mechanical domain. Qualitative problems can be solved with minimal discipline-specific information, without any robust quantitative calculation, generally by using qualitative reasoning and commonsense knowledge. QMPS is a vital aspect of human intelligence that allows us to tackle a wide range of tasks, from simple everyday ones such as turning on a tap to complex tasks in highly demanding and well-paying jobs in various fields, e.g., emergency medicine, plumbing, driving, etc. Employers often use the Bennett Mechanical Comprehension Test (BMCT) to evaluate job candidates' ability to solve such problems. In this work, we assess two state-of-the-art multimodal models, Gemma-3 and Qwen-VL, on their ability to interpret mechanical problem images by eliciting a step-by-step chain of thought (CoT) and a final answer. Each image inherently encodes ground-truth qualitative facts, such as contact points in gears, support relations, and relative weights, which we use to evaluate each model's spatial and commonsense reasoning capabilities. We assess each chain for coherence, completeness, and logical progression to assess each model's thought process, and final answers are compared to verified solutions to measure accuracy.
1 Introduction
Qualitative mechanical problem-solving uses qualitative reasoning and commonsense knowledge instead of robust quantitative calculation, supporting everyday and professional tasks. The work evaluates small vision-language models by examining both answer correctness and chain-of-thought reasoning.
- Qualitative Mechanical Problem-Solving: Qualitative problems rely on attribute qualities and comparisons rather than actual quantities or robust quantitative calculation.Examples include comparisons of size, appearance, symmetry, volume, and length.
- Qualitative Mechanical Problem-Solving: QMPS applies qualitative reasoning and commonsense knowledge to mechanical tasks without requiring equations, theorems, or formulae from mechanics textbooks.The paper connects this capability to tasks ranging from hanging a picture frame to emergency medicine, plumbing, and hydraulic machinery.
- Study Focus: The paper evaluates Qwen-VL and Gemma-3 on qualitative mechanical problems while examining answer correctness alongside chain-of-thought coherence and validity.This evaluation is conducted on a single A100-40GB GPU.
2 Motivation and Background
The study motivates benchmarking small vision-language models on visually presented mechanical concepts because real-world QMPS requires integrated language, visual, and spatial reasoning. It uses BMCT problems as human-level mechanical comprehension tests and examines a statistical vision-language approach.
- Motivation: Small, efficient vision-language models need evaluation on specialized tasks that integrate visual perception, language understanding, and mechanical reasoning.The motivation emphasizes broader accessibility and deployment in real-time environments.
- Benchmark: BMCT problems test mechanical-concept understanding through diagrams, labels, annotations, textual information, and multiple-choice answers.The test is used by companies as a recruitment and training tool for roles requiring mechanical acumen.
- Related Work: Prior work addressed Bennett problems with analogical and formal logicist approaches, whereas this paper evaluates a purely statistical vision-language approach.The comparison concerns methodological approaches to the same benchmark family.
3 Methodology
The methodology curates a 30-problem BMCT benchmark and prompts vision-language models to produce step-by-step reasoning alongside final answers. The benchmark spans multiple mechanical scenarios and difficulty levels.
- Dataset or Test Cases: The benchmark contains 30 curated BMCT problems spanning hydraulics, gears and belt drives, acoustics, force and mechanics, and heat and radiation.The scenario counts are hydraulics (7), gears and belt drives (6), acoustics (5), force and mechanics (6), and heat and radiation (6).
- Prompting: Each benchmark instance is embedded in a template that asks models to reason step by step before giving an answer.The instances contain graphical depictions, questions, and fixed answer choices.
3.2 Categories of Failure and Success Cases
The paper categorizes failures by whether models infer unsupported facts, misuse physical concepts, omit crucial visual information, or overlook geometric relations. These categories explain wrong answers and poor spatial reasoning in the benchmark.
- Case 1: Flawed Reasoning Leading to Wrong Answer: Some wrong answers follow chains of thought that mention relevant visual facts but introduce unsupported assumptions, misuse physical concepts, or contain invalid deductions.Examples include assuming unequal lever arms when arms are equal and mishandling force equilibrium or torque balance.
- Case 2: Omission of Crucial Visual Information - Poor Spatial Reasoning: Other wrong answers result from omitting essential image facts, including contact points, force vectors, pressure levels, gear relations, or tooth counts.These omissions prevent the chain of thought from representing crucial spatial or mechanical information.
- Case 2: Omission of Crucial Visual Information - Poor Spatial Reasoning: Overlooked geometric relations, such as incline angles or lever-arm lengths, form another category of poor spatial reasoning.The missing relations concern relative distances and angles encoded in the diagrams.
Case 3: Correct Answer Despite Flawed Reasoning
Some models reach the correct final answer despite flawed chains of thought, either by guessing or by using only a minimal set of essential visual facts.
- Models can produce a correct final answer even when their chains of thought contain clear reasoning flaws.
- Lucky guessing occurs when a model hallucinates intermediate facts or misapplies concepts but reaches the correct answer by chance.
- Partial reasoning can suffice when a model correctly captures enough essential facts for a valid qualitative judgment despite an incomplete or partly incorrect chain.
Case 4: Correct Answer with Flawless Reasoning
The evaluation also includes a case in which both models produce the correct answer together with coherent and logically sound chains of thought.
- Both models are analyzed in a case where they provide a correct answer and coherent, logically sound chains of thought.
3.3 Results and Analysis
Table 1 summarizes Qwen-VL and Gemma’s performance across 30 BMCT problems, combining answer accuracy with reasoning quality and spatial-reasoning failures.
- Table 1 reports both models’ overall accuracy across all 30 BMCT problems.
- The table separates flawless and flawed reasoning from failures attributed to poor spatial reasoning.
- The analysis examines demonstrations from each category to relate reasoning quality and spatial understanding to overall success or failure.
Case 1: Flawed Reasoning Leading to Wrong Answer
A gear-speed example shows both models reaching the same incorrect conclusion because their chains of thought misuse gear relationships and hallucinate unsupported dependencies.
- Case 1: Flawed Reasoning Leading to Wrong Answer: The problem asks which of gears A and B rotates faster when a central gear drives both sides through equal tangential speeds.
- Case 1: Flawed Reasoning Leading to Wrong Answer: The correct reasoning is that both smaller gears have equal tangential speed and radius, so their angular speeds are equal; rigidly attached outer wheels therefore turn equally fast.
- Case 1: Flawed Reasoning Leading to Wrong Answer: Both Gemma and Qwen-VL nevertheless conclude incorrectly that Gear B rotates faster than Gear A.
- Case 1: Flawed Reasoning Leading to Wrong Answer: Gemma incorrectly treats Gear A as directly driving Gear B and applies inverse driver–driven speed proportionality and tooth-count reasoning.
- Case 1: Flawed Reasoning Leading to Wrong Answer: Qwen-VL invents a tooth-count difference between the identical driven gears and infers that the right gear rotates faster.
- Case 1: Flawed Reasoning Leading to Wrong Answer: These unsupported dependencies and misused physical laws compromise subsequent inferences and produce an inaccurate final answer.
Case 2: Omission of Crucial Visual Information - Poor Spatial Reasoning
In the acoustics problem, both models omit crucial visual information and answer incorrectly by inventing motion or distance cues instead of using medium-dependent sound speed.
- Problem setup: The acoustics problem asks which submarine detects the same blast first when propagation occurs through air or water.Water transmits sound at approximately 1,482 m/s, compared with 343 m/s in air, so Scenario B is correct.
- Gemma’s reasoning: Gemma invents submarine motion and applies irrelevant approach-versus-recession reasoning to static illustrations.This misreading replaces the actual comparison of sound speeds with an unsupported constant-speed kinematic interpretation.
- Qwen-VL’s reasoning: Qwen-VL invents unequal blast distances, treating Scenario A as closer despite identical horizontal distances in both images.It consequently emphasizes proximity rather than the medium-dependent propagation speed and selects Scenario A without justification.
- Interpretation: The elicited chains of thought expose which visual features each model attends to and how those cues shape its qualitative reasoning.This provides insight beyond final-answer accuracy by revealing the assumptions extracted from each image.
Case 3: Correct Answer Despite Flawed Reasoning
In the hydraulics case, Qwen-VL reaches the correct answer despite flawed reasoning, while the broader cases show that both models succeed mainly when visual cues and governing principles are straightforward.
- Hydraulics problem: Qwen-VL correctly identifies Turbine C as fastest, but explains the result using water-jet speed rather than torque from the jet’s position.The model’s conclusion is correct, but its stated mechanism differs from the rigorous torque analysis.
- Hydraulics problem: The correct analysis assigns opposing, zero, and reinforcing moments to Turbines A, B, and C respectively, making C the fastest runner.A’s jet opposes the river torque, B’s jet adds no moment, and C’s jet supplies extra driving moment.
- Implication: Lucky hits in which flawed internal logic produces a correct answer demonstrate the brittleness and unpredictability of language models in qualitative reasoning.Answer correctness alone can therefore conceal an invalid reasoning process.
- Easy cases: Both models produce coherent, correct chains when visual cues are unambiguous and the underlying geometric principles are straightforward.Under these easy conditions, they reliably map pixels to facts and apply familiar rules.
- Limitation: When visual inference becomes subtle or physics more nuanced, the models’ spatial reasoning and qualitative deductions can fail.The contrast with ideal conditions identifies a practical boundary for their reliability.
4 Discussion, Conclusion, and Future Work
Gemma and Qwen-VL show limited but distinct success on qualitative mechanical reasoning, with errors arising from flawed reasoning and poor spatial interpretation. The paper recommends improving both perceptual and reasoning capabilities through specialized training, enhanced visual encoders, and neuro-symbolic systems.
- Results: Gemma achieved accurate and flawless reasoning in 30% of cases, while Qwen-VL achieved it in 40%.Gemma succeeded in 9 of 30 cases and Qwen-VL in 12 of 30.
- Results: Both models produced correct answers with flawed chains of thought in 2 of 30 cases each (6.67%).These cases included inaccurate physical principles, minor logical missteps, or subtle misapplications despite correct final selections.
- Results: Gemma failed because of flawed reasoning in 40% of cases, compared with 33.33% for Qwen-VL.Reported causes include incorrect assumptions, hallucinated concepts, and invalid deductive steps.
- Results: Poor spatial reasoning caused errors in 23.33% of Gemma cases and 20% of Qwen-VL cases.The models misperceived spatial relationships or missed critical image features, producing incorrect premises and outcomes.
- Future Work: The paper recommends a multi-pronged approach combining visuo-spatial fine-tuning, richer visual encoders, and domain-specific neuro-symbolic knowledge.The proposed directions include physics- and engineering-focused datasets, more powerful vision backbones, and atomic physical facts embedded in hybrid reasoning systems.