PaperlayerView exact revision on arXiv

Source-linked AI summary

MechReason: Benchmarking Multi-Image Multi-Hop Reasoning in Mechanical Engineering

Tengyue Wang, Kang An, Chenxu Du, Zhongyu Yang, Yuanchi Zhu, Xinqi Yang, Hebao Zhu, Ziliang Wang, FaQiang Qian, Yunli Yang, Qibing Ren

arXiv:2609.16012v1cs.CV

TL;DR

Existing benchmarks rarely test multi-hop mechanical reasoning across heterogeneous evidence, images, and engineering constraints. MechReason turns scientific-paper arguments into verified questions while withholding posterior validation evidence. Even GPT-5.5 achieves only 62.89% accuracy, and CoT supervision does not resolve diagnostic and grounding bottlenecks.

  • Problem

    Existing mechanical benchmarks rarely combine heterogeneous visual evidence, multi-image inputs, complex engineering claims, and explicit reasoning supervision, leaving multi-hop mechanical reasoning largely unexplored.

    Source: Introduction · p. 1

  • Method

    MechReason decomposes paper claims into prerequisites, reasoning, conclusions, and corroboration, then generates anti-shortcut questions and applies multimodal quality verification.

    Source: Introduction · p. 2, Question Generation · p. 4

  • Results

    Even GPT-5.5 achieves only 62.89% accuracy, and failures are dominated by chain, grounding, and compression errors rather than simple hallucination.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    CoT-based supervised fine-tuning improves performance, but substantial gaps remain.

    Source: Introduction · p. 2

    CoT supervision does not improve Limit Diagnosis and slightly decreases Deviation Attribution, while evidence grounding and mechanical mechanism understanding remain broader bottlenecks.

    Source: Chain-of-Thought Supervision Analysis · p. 7

Abstract

from arXiv · show

Despite significant progress in general visual question answering and cross-modal understanding, multimodal large language models still face a pronounced gap in evaluation for complex reasoning within the mechanical engineering domain. Existing benchmarks predominantly focus on rudimentary tasks such as drawing recognition, CAD interpretation, or single-chart querying, falling short of assessing whether models can integrate multiple images, textual conditions, physical principles, and engineering constraints to perform multi-step reasoning when confronted with authentic, intricate mechanical problems. To address this, we introduce MechReason, a benchmark derived from real mechanical engineering papers, comprising 12k question-answer pairs with explicit reasoning-chain annotations and 21k visual materials spanning nine evidence types, including statistical charts, parameter tables, engineering drawings, microscopic images, simulation images, system architectures, real mechanical scene photos, CAD model images and manufacturing flowcharts. MechReason covers eight task types across four reasoning dimensions: explanation, prediction, design, and diagnosis. We devise a four-stage construction pipeline: we first extract core engineering claims and decompose their supporting evidence into premises, reasoning processes, conclusions, and corroborative evidence; we then generate shortcut-preventing questions by masking posterior verification information; finally, we apply multimodal quality validation to ensure task quality and multi-hop nature. Extensive experimental results demonstrate that MechReason is highly challenging, with even the most advanced models achieving only 62.89\% accuracy.

Introduction

Existing benchmarks leave complex, multi-image mechanical reasoning largely unexplored. MechReason builds a paper-derived benchmark and evaluates current models, finding that even the best-performing model remains below 63% accuracy.

  • Existing mechanical benchmarks emphasize localized perception or shallow tasks and rarely combine heterogeneous evidence, multi-image inputs, complex claims, and explicit reasoning supervision (Doris et al. 2025; Mallis et al. 2026; Doris et al. 2026; Kou et al. 2026).Source: Introduction · p. 1
  • MechReason contains 12,257 question-answer pairs with explicit CoT supervision and 21,842 visual materials across nine evidence types, organized into Explanation, Prediction, Design, and Diagnosis.Source: Introduction · p. 1
  • Its pipeline identifies core engineering claims, decomposes their arguments, generates questions that withhold posterior verification evidence, and applies multimodal quality validation.Directly prompting models to generate questions can expose final performance tables, simulation maps, or experimental figures, enabling lookup rather than reasoning.Source: Introduction · pp. 1–2
  • 62.89% accuracy was the best-performing model’s result, with errors mainly reflecting flawed reasoning chains, grounding failures, and lossy compression; CoT fine-tuning improved performance but left substantial gaps.Source: Introduction · p. 2

Related Work

Prior mechanical-engineering benchmarks and general multi-hop datasets address parts of the problem, but leave mechanical reasoning across heterogeneous sources and paper-specific argument structures insufficiently covered.

LLMs for Mechanical Engineering

  • Mechanical-engineering benchmarks largely evaluate CAD, drawing, and document tasks, while most lack multi-hop reasoning across heterogeneous engineering information.Existing work includes CAD rule extraction and program generation, manufacturability analysis, and real-world part or assembly drawings.Source: LLMs for Mechanical Engineering · p. 2

Multi-hop Reasoning Question Construction

  • General multi-hop datasets combine text, tables, images, or scientific documents, but are not specifically designed for mechanical-engineering scenarios.Source: Multi-hop Reasoning Question… · p. 2
  • Scientific-paper conclusions may be answered through posterior experimental figures or tables instead of reconstructed reasoning, so MechReason separates reasoning evidence from verification evidence.This targets a shortcut inherent in scientific argument structures, beyond lexical or local shortcuts addressed by other dataset approaches.Source: Multi-hop Reasoning Question… · p. 2

MechReason Benchmark

MechReason is a paper-derived benchmark spanning diverse mechanical-engineering evidence and reasoning tasks. Its construction pipeline turns scientific arguments into verified questions while withholding posterior corroboration that could provide answer shortcuts.

Benchmark Overview

  • MechReason derives from published papers and spans thermal-fluid systems, manufacturing processes, machine tools, robotics, materials processing, structural analysis, and control systems.Source: Benchmark Overview · p. 2
  • MechReason contains 12,257 QA samples, including 9,994 training and 2,263 test samples; test questions require visual information, with 2.87 images per question on average.Source: Benchmark Overview · pp. 2–3
  • Its 21,842 unique visual materials cover nine evidence types, from charts and parameter tables to microscopy, simulation, real scenes, CAD, and manufacturing flowcharts.Source: Benchmark Overview · p. 3
  • The benchmark groups questions into explanatory, predictive, design, and diagnostic dimensions, each further divided into two subcategories.The dimensions cover causes, responses to changed conditions, constrained design choices, and diagnoses of limits or failures.Source: Benchmark Overview · p. 3

Question Generation

  • Question generation parses papers into text and visual materials, then selects mechanically meaningful claims supported by multiple pieces of evidence and non-trivial reasoning.Source: Question Generation · p. 4
  • For each claim, the pipeline separates preconditions, derivation, conclusion, and posterior corroboration, keeping the logical basis distinct from later validation evidence.Corroboration can include experimental measurements, simulations, or performance comparisons that verify a conclusion after it is derived.Source: Question Generation · p. 4
  • Questions expose necessary preconditions but omit derivations and corroboration, which are retained as reasoning supervision rather than visible input.This prevents answering by looking up final experimental results, performance tables, or verification figures.Source: Question Generation · pp. 4–5
  • Multimodal verification retains samples only when they have sufficient prerequisites, visual dependency, reasoning complexity, and mechanical relevance.The pipeline does not directly extract QA pairs; it reconstructs arguments, creates anti-shortcut questions, and filters low-quality samples.Source: Question Generation · p. 5

Experiments

The experiments find a substantial performance gap between the strongest proprietary and open-weight models, with multi-image mechanical reasoning still challenging even for the top model. Error analysis points to faulty reasoning chains and grounding, while CoT fine-tuning improves some tasks but leaves important bottlenecks.

Experimental Setup

  • Models receive question text and corresponding images and must generate short answers, without external tools, retrieval, or additional paper context.Accuracy uses a judge model to assess semantic consistency rather than exact string matching.Source: Experimental Setup · p. 5

Main Results

  • 62.89% overall accuracy for GPT-5.5 leads the MechReason results; even this best model fails on more than one third of test samples.Claude Opus-4.8 reaches 58.07%, while GLM-4.1V is the strongest open-weight model at 42.11%.Source: Table 2 · p. 6, Main Results · p. 6

Open Source Models

  • Category-wise accuracy is calculated over the test samples belonging to each reasoning category.The evaluation permits reasonable answer variations through semantic-consistency judging.Source: Open Source Models · p. 6

Main Results

  • Predictive reasoning is generally easier, while explanation, design, and diagnosis require harder evidence integration and remain more challenging.Open-weight models particularly struggle with limit diagnosis and failure diagnosis, involving boundary conditions, physical constraints, and model-assumption failures.Source: Main Results · p. 6

Failure Mode Analysis

  • 37.12%–44.65% of failures are Chain errors across models, while Qwen3.5-9B has 44.86% Grounding errors.Chain errors reflect incorrect multi-hop reasoning despite partial evidence recognition; Qwen's grounding errors point to difficulty aligning heterogeneous visual evidence.Source: Failure Mode Analysis · p. 6, Table 3 · p. 7
  • Compression errors account for 27.80% of GPT-5.5 failures and 23.25% of Gemini-3.1-Pro failures; hallucination is relatively rare.Compression errors occur when a partly reasonable trace yields a generic final answer or drops a key qualifier.Source: Table 3 · p. 7, Failure Mode Analysis · p. 7

Chain-of-Thought Supervision Analysis

  • 36.63% accuracy after CoT fine-tuning exceeds Qwen3.5-9B's original 32.35% by 4.27 points, versus 33.27% without CoT.The comparison suggests gains stem more from learning intermediate engineering reasoning processes than from answer-format or terminology adaptation.Source: Figure 4 · p. 7, Chain-of-Thought Supervision Analysis · p. 7
  • CoT gains are largest for Process Extrapolation (+10.53 points), Performance Prediction (+8.43), and Entity Design (+6.46), but Limit Diagnosis does not improve.Deviation Attribution decreases slightly; textual CoT alone is difficult for learning visual localization, boundary conditions, and modeling assumptions.Source: Table 4 · p. 7, Chain-of-Thought Supervision Analysis · p. 7

Conclusion

MechReason covers four reasoning intentions across nine types of mechanical visual evidence, with approximately 12K QA pairs and chain-of-thought supervision. Experiments find that even GPT-5.5 achieves only 62.89% accuracy, with failures concentrated in reasoning chains, evidence grounding, and answer compression.

Conclusion

  • MechReason includes approximately 12K QA pairs with CoT supervision across nine types of mechanical visual evidence and four reasoning intentions.The four intentions are Explanation, Prediction, Design, and Diagnosis.Source: Conclusion · p. 7
  • 62.89% accuracy is achieved by GPT-5.5, indicating that current MLLMs remain far from reliable on MechReason.The paper describes complex multi-image mechanical reasoning as highly challenging.Source: Conclusion · p. 7
  • Failures are dominated by chain errors, grounding errors, and compression errors rather than simple hallucination.Source: Conclusion · p. 7
  • CoT supervision improves tasks with clearer derivation paths but brings limited gains for diagnosis and deviation attribution.Source: Conclusion · p. 7
  • Future models need stronger abilities to integrate multi-image evidence, recover mechanical mechanisms, and reason under engineering constraints.Source: Conclusion · p. 8
Loading 2609.16012v1…