PaperlayerView exact revision on arXiv

Source-linked AI summary

EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning

Jinchang Zhang, Yingda Tao, Jiakai Lin, Guoyu Lu

arXiv:2609.33261v1cs.CVcs.SE

TL;DR

Static multimodal engineering benchmarks leave open whether models can use an understood system state to reason about relations, constraints, and design-change consequences. EngIntervene evaluates that capability across four levels and seven domains; results show that stronger grounding does not reliably yield stronger diagnosis or intervention.

  • Problem

    Existing multimodal engineering benchmarks largely assess static understanding, leaving whether models can reason about system relations, constraints, and design-change consequences under-evaluated.

    Source: §1 · pp. 1–2

  • Method

    EngIntervene evaluates 3,229 questions across seven engineering domains in four levels, from state grounding through relational reasoning and diagnosis to constraint-preserving intervention.

    Source: §1 · p. 2

  • Results

    Stronger grounding does not consistently yield stronger diagnosis or intervention, while visual evidence matters, fine-tuning improves T1 but not T2–T4, and partial revision success may not produce a valid change.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Engineering reasoning should be measured by whether a recovered state supports constraint-aware decisions about how a system should change.

    Source: §5 · p. 10

    Domain-specific T4 scores do not measure identical task compositions because intervention subtypes and role coverage vary across domains.

    Source: §6.4.4 · p. 19

Abstract

from arXiv · show

Multimodal engineering benchmarks largely evaluate static understanding, such as recognizing components, interpreting diagrams, or answering technical questions. This leaves a missing middle between engineering perception and full design generation: whether a model can use an understood system state to reason about relations, constraints, and the consequences of design changes. We introduce \textsc{EngIntervene}, a benchmark for this capability. It contains 3,229 questions across seven engineering domains and organizes evaluation into four levels: state grounding (T1), relational and mechanistic reasoning (T2), constraint-aware diagnosis (T3), and intervention reasoning (T4), which asks whether a proposed modification achieves its target while preserving required constraints. The tasks instantiate a unified engineering state representation spanning objects, relations, constraints, and design objectives, and T2--T4 are scored against structured reference answers with atomic criteria. Across open- and closed-weight multimodal models, stronger grounding does not reliably translate into better diagnosis or intervention, and the best open-weight model trails the best closed model by 14.7 percentage points on the T2--T4 average. Removing or shuffling visual evidence consistently degrades performance, while benchmark-specific supervised fine-tuning improves T1 but not T2--T4. T4 further exposes a large gap between satisfying individual revision criteria and producing a fully valid intervention. Engineering reasoning thus requires not only recovering the current state, but also reliably using it to reason about constraints and post-intervention consequences. Code and benchmark artifacts are available at https://github.com/changcv2021/EngIntervene

1 INTRODUCTION

Engineering systems require models to reason beyond component recognition, because local design changes can affect system-wide relations, constraints, and objectives. EngIntervene evaluates this missing middle between static understanding and design generation.

1 INTRODUCTION

  • Engineering reasoning must represent system entities, relationships, constraints, and objectives while predicting how local modifications affect the overall state.A seemingly reasonable change can still fail in the broader system context.Source: §1 · pp. 1–2
  • Existing multimodal and engineering benchmarks cover adjacent capabilities, but leave open whether models consistently reason about system relations, constraints, and local-change consequences.Evaluating only a final answer or artifact makes it difficult to locate whether failures arise from recognition, relation modeling, diagnosis, or side-effect prediction.Source: §1 · p. 2
  • EngIntervene contains 3,229 questions across seven domains and evaluates four levels, from engineering understanding to whether changes achieve objectives while preserving constraints.The authors report that stronger state understanding does not reliably translate into diagnosis or intervention gains, and fine-tuning improvements in basic understanding do not automatically transfer to T2–T4.Source: §1 · p. 2

2 ENGINTERVENE

EngIntervene frames intervention validity as achieving a specified objective while preserving required constraints, then instantiates that reasoning across four capability levels and seven engineering domains. Its questions, structured answers, and atomic criteria are built from traceable engineering sources and reviewed for quality.

2.1 BENCHMARK OVERVIEW AND UNIFIED PROBLEM FORMULATION

  • A valid intervention must achieve its objective in the modified state while keeping required constraints satisfied.The unified state is represented as objects and attributes, relations, constraints, and objectives; regressions are failures to preserve a required constraint.Source: §2.1 · p. 3
  • The four capability levels progress from grounding objects and functions to relational reasoning, constraint-aware diagnosis, and intervention reasoning.The levels describe capability types; individual samples are not required to be solved sequentially.Source: §2.1 · pp. 3–4

2.2 FOUR TASKS AND CROSS-DOMAIN INSTANTIATION

  • T4 covers selecting or verifying revisions, proposing constrained local changes, and prioritizing interventions or trade-offs.These operations respectively choose, assess, construct, or rank design changes against objectives and constraints.Source: §2.2 · p. 4

2.3 DATA COLLECTION, ANSWER REPRESENTATION, AND QUALITY CONTROL

  • Questions draw on 357 source documents, with construction limited to source-supported facts, constraints, and engineering conclusions.The pipeline includes source discovery, visual matching, task assignment, reference preparation, structured scoring-target construction, automated checks, and human semantic review.Source: §2.3 · pp. 4–5
  • Structured answers align with task objectives, and T2–T4 criteria atomize required reasoning; an independent audit reviewed 500 stratified questions across five dimensions.Before revision, 94.8% of audited questions met all applicable criteria; all identified failures were revised and re-reviewed, and all 240 T4 questions received a dedicated audit.Source: §2.3 · p. 5

2.4 COVERAGE AND DIVERSITY OF THE ENGINEERING REASONING SPACE

  • The 3,229 questions span seven domains, with each domain contributing 450–505 questions and T2 and T3 coverage balanced across domains.The benchmark includes 2,275 distinct images and 3,936 image references; 483 questions require reasoning jointly over multiple images.Source: §2.4 · pp. 5–6
  • Across domains, T3–T4 questions average 3.2 constraint categories, and shared constraint types recur across different engineering systems.Function / Performance constraints cover 63.0%–87.8% of T3–T4 questions in every domain; Safety / Reliability is concentrated in Aerospace, Building Services, and Power and Energy.Source: §2.4 · p. 6
  • The 240 T4 questions include 132 source-supported constructed items with explicit modification objectives and acceptance conditions.Reference modifications, protected constraints, and regression conditions undergo a dedicated T4 audit.Source: §2.4 · p. 6

3 EXPERIMENT

Across EngIntervene, stronger state grounding does not reliably translate into better diagnosis or intervention, and complete revision validity remains much harder than partial correctness. Visual evidence supports performance across tasks, while benchmark-specific fine-tuning improves T1 but not T2–T4.

3.1 MAIN RESULTS

  • 14.66 percentage points separate Claude Opus 4.7’s 61.30% T2–T4 average from Qwen3.8-27B’s 46.64%; similar T1 scores can still accompany substantially different open-ended performance.Claude Opus 4.7 leads the evaluated models on the T2–T4 average; T4 Atomic is higher than T2 and T3 for all but one model.Source: §3.1 · p. 6, Table 4 · p. 7

3.2 IS VISUAL EVIDENCE NECESSARY?

  • 10.67% is the five-model mean T2 score under Shuffled-image, down from 26.73% under Text-only; removing correct images also lowers every model’s four-task macro average.Across the five models, removing correct images costs 7.84–13.33 percentage points on the four-task macro average; shuffled images cause additional losses and can prompt descriptions of nonexistent components and relations.Source: §3.2 · p. 7, Table 5 · p. 7

3.3 FINE-GRAINED ANALYSIS OF DESIGN-CHANGE REASONING

  • 17.92% Strict Success for Qwen3.5-27B remains far below its 52.12% T4 Atomic score, illustrating that partial revision credit often fails strict acceptance.Scaling improves revision capabilities unevenly: within Qwen3.5, Mechanism and Goal rise with model size, while Protection is non-monotonic and active Regression checking remains nearly unchanged. Even responses satisfying all Goal criteria can leave Protection or Mechanism requirements incomplete.Source: §3.3 · pp. 8–9

3.4 CAN SFT NARROW THE ENGINEERING REASONING GAP?

  • For all three evaluated models, Industry-SFT improves T1 but performs below the corresponding Base model on T2, T3, and T4 Atomic.The four-task macro average decreases for each model, and T4 Strict does not improve under the current LoRA-based training setting.Source: §3.4 · p. 9, Table 6 · p. 9

4 RELATED WORK

Related benchmarks address engineering problem solving, tool use, design generation, or multimodal reasoning, while EngIntervene targets reasoning between understanding and generation.

4 RELATED WORK

  • EngIntervene measures state recovery, constraint diagnosis, and local intervention reasoning separately from state recovery, unlike benchmarks focused on problem solving, tool operation, design generation, or other multimodal reasoning tasks.Its four reasoning levels span visual grounding through design intervention within one engineering setting.Source: §4 · p. 9

5 CONCLUSION

EngIntervene evaluates whether models can use an understood engineering state to reason about relations, constraints, and design interventions. Results show that grounding alone does not ensure reliable diagnosis or valid changes.

5 CONCLUSION

  • EngIntervene’s 3,229 questions across seven engineering domains span four levels from state grounding to post-intervention validity.Results show that visual evidence matters beyond recognition, fine-tuning improves T1 but not T2–T4, and partial revision correctness often fails to yield a valid design change.Source: §5 · pp. 9–10

AI USE STATEMENT

The authors used generative AI for language assistance and evaluated large multimodal and language models as experimental systems.

AI USE STATEMENT

  • Generative AI supported translation and limited English polishing; authors reviewed and revised all AI-assisted text.Large multimodal and language models were also evaluated on EngIntervene, with model-based scoring for open-ended T2–T4 tasks under the paper’s fixed protocol.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The benchmark uses traceable engineering sources and excludes intentional collection of personal or sensitive information. Safety-relevant answers are not substitutes for professional review or certification.

  • Benchmark materials come from traceable sources, including examinations, textbooks, technical manuals, and design guidelines.Source: ETHICS STATEMENT · p. 10
  • The benchmark does not intentionally collect personal or sensitive information and focuses on engineering systems rather than individuals.Source: ETHICS STATEMENT · p. 10
  • Safety-relevant benchmark answers should not replace professional engineering review or certification.Source: ETHICS STATEMENT · p. 10
  • Materials will be distributed according to applicable source licenses and redistribution constraints.Non-English materials are translated during normalization while preserving engineering meaning and source provenance.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The authors plan to release the benchmark and the materials needed to reproduce its evaluation, including a fixed model-based protocol for T2–T4.

  • The planned release includes the benchmark, evaluation code, scoring scripts, prompts, structured reference answers, and atomic rubrics.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Dataset splits, task and domain metadata, model inference settings, and metric and ablation scripts are also slated for release.Source: REPRODUCIBILITY STATEMENT · p. 10
  • For T2–T4, the release will provide the fixed model-based evaluation protocol and scoring configuration used in the experiments.Source: REPRODUCIBILITY STATEMENT · p. 10

6 APPENDIX

Figure 6 is identified as a qualitative example.

6.1 QUALITATIVE EXAMPLE

6.2 TASK DEFINITIONS AND REFERENCE ANSWER REPRESENTATIONS

The appendix defines task-specific reference representations for understanding, relational reasoning, diagnosis, and intervention. Together, these representations specify what answers must identify and, for interventions, what constraints and consequences must be considered.

6.2.1 T1: MULTIMODAL DESIGN UNDERSTANDING

  • T1 evaluation uses images, a question, and candidate options, with stable option identifiers for correct answers.References may identify relevant visual regions and source explanations; questions cover component categories, roles, local-feature purposes, and design intent.Source: §6.2.1 · p. 13

6.2.2 T2: MECHANISTIC AND RELATIONAL REASONING

  • T2 references specify required entities, critical relations, paths or sequences, and critical errors such as reversed directions or incorrect connections.Source: §6.2.2 · p. 13

6.2.3 T3: CONSTRAINT-AWARE DESIGN DIAGNOSIS

  • T3 represents diagnosis as location or object, problem type, violated constraint or principle, and potential consequence.When explicit numerical values appear, the reference also records observed and required values; scoring covers only judgments supported by the source or question conditions.Source: §6.2.3 · pp. 13–14
  • Table 7 presents the benchmark's domain-by-task and T4-subtype distribution.Source: Table 7 · p. 14

6.2.4 T4: CONSTRAINT-PRESERVING DESIGN INTERVENTION REASONING

  • T4 references capture a design objective or problem, proposed modification and intended effect, protected conditions, and critical side effects.Its subtypes evaluate selection, verification and regression, constrained local proposals, or prioritization and trade-offs; equivalent answers must achieve the target while satisfying protected conditions.Source: §6.2.4 · p. 14

6.3 COMPLETE DATASET STATISTICS

The dataset statistics describe EngIntervene’s intervention taxonomy, source construction, visual inputs, constraints, and quality controls across seven engineering domains. The breakdowns show that T4 differs from simpler tasks in both construction and item structure.

6.3.1 COMPLETE DOMAIN, TASK, AND T4 SUBTYPE DISTRIBUTION

  • All 30 Automotive T4 questions are Revision Selection, while Revision Verification and Constrained Local Revision Proposal each span six domains.Revision Prioritization and Trade-off has 25 questions across four domains, so the four types are a unified taxonomy, not equally sized sub-benchmarks.Source: §6.3.1 · p. 14

6.3.2 CONSTRUCTION AND ADAPTATION STATISTICS

  • 55.0% of T4 questions are source-supported constructed items, compared with 7.8%–11.0% for T1–T3.Across all tasks, 2,850 questions are normalized, translated, or cropped source questions and 379 are source-supported constructed questions.Source: §6.3.2 · p. 14

6.3.3 SOURCE TYPE COMPOSITION

  • Official qualification or licensing examinations account for 59.7% of source documents and 47.8% of questions.The source set also includes university examinations, manuals, technical documents, and a human-verified question–answer compilation contributing 249 questions (7.7%).Source: §6.3.3 · p. 15

6.3.4 SOURCE LANGUAGE DISTRIBUTION

  • All benchmark inputs are in English, with non-English source materials translated during normalization.The 357 source documents include English, German, Japanese, Chinese, and mixed-language materials.Source: §6.3.4 · p. 15

6.3.5 VISUAL ASSETS AND MULTI-IMAGE INPUTS

  • The benchmark references 2,275 distinct images; 483 questions use multiple images, representing 15.0% of the benchmark.A further 644 images are shared by two or more questions.Source: §6.3.5 · p. 15

6.3.6 INPUT STATISTICS BY TASK

  • T4 questions have a median length of 114 characters, compared with 35 for T1, while all four tasks remain dominated by single-image inputs.The longer T4 prompts specify objectives, candidate modifications, editable boundaries, or constraint conditions.Source: §6.3.6 · p. 15

6.3.7 SOURCE CONCENTRATION AND INDEPENDENCE

  • The 3,229 questions are associated with 357 source documents organized into 227 source groups, and splits are constructed at the source-group level.The largest source contributes 249 questions, or 7.7% of the benchmark.Source: §6.3.7 · p. 16

6.3.8 DEDICATED AUDIT OF T4 DESIGN INTERVENTIONS

  • Before revision, 218/240 T4 questions (90.8%) satisfied all applicable audit criteria.Protected-constraint completeness passed at 95.4% and regression-condition completeness at 93.8%; failed items were revised and rechecked.Source: §6.3.8 · p. 17

6.4 DETAILED T4 DIAGNOSTIC RESULTS

The T4 diagnostics separate criterion-level success from fully valid interventions and examine how performance varies across roles, intervention types, and domains. Their comparisons require attention to eligibility and task composition.

6.4.1 ROLE-WISE T4 PERFORMANCE

  • Goal, Mechanism, Protection, and Active Regression criteria are observable in 237, 218, 159, and 46 T4 items, respectively.Atomic and Strict metrics cover all 240 items and are not averages of the four role scores.Source: §6.4.1 · p. 18, Table 19 · p. 20

6.4.2 FAILURES AFTER GOAL SATISFACTION

  • Among responses satisfying all available Goal criteria, many still fail complete Mechanism or Protection requirements.Qwen3.5-27B fails Mechanism on 29 of 84 eligible responses and Protection on 19 of 55; Claude Opus 4.7 fails them on 38 of 114 and 25 of 71, respectively.Source: §6.4.2 · p. 19

6.4.3 PERFORMANCE ACROSS T4 INTERVENTION TYPES

  • Qwen3.8-27B has higher overall T4 performance than Qwen3.5-27B, but lower Selection Atomic performance (35.93% vs. 43.31%).The model performs better on Verification and Proposal; Prioritization has only five items and is reported descriptively.Source: §6.4.3 · p. 19

6.4.4 DOMAIN-LEVEL T4 PERFORMANCE AND ROLE COVERAGE

  • Domain-level Strict Success differences are descriptive, not intrinsic difficulty rankings, because subtype composition and role observability vary across domains.All 30 Automotive T4 items are Selection, while Aerospace, Building Services and MEP, and Compact Electronics lack independently scored positive Active Regression criteria.Source: §6.4.4 · p. 19

6.4.5 COMMON-SCOPE SENSITIVITY CHECK

  • On the common 134-item subset, stronger models still score substantially higher on Goal and Mechanism, while Strict Success remains below Atomic performance.Some model rankings change on this restricted subset, supporting diagnostic rather than leaderboard interpretations of role-specific averages.Source: §6.4.5 · pp. 19–20

6.5 COMPLETE DATASET STATISTICS

The complete dataset statistics characterize the benchmark’s task taxonomy, source materials, inputs, constraint annotations, and T4 audit. They show that higher-level intervention items rely more heavily on controlled construction and have distinct domain and representation profiles.

6.5.1 COMPLETE DOMAIN, TASK, AND T4 SUBTYPE DISTRIBUTION

  • All 30 Automotive T4 questions are Revision Selection, while Revision Verification and Constrained Local Revision Proposal each cover six domains.The 25 Prioritization and Trade-off questions are distributed across four domains, making the four types a unified taxonomy rather than equally sized sub-benchmarks.Source: §6.5.1 · p. 20, Table 25 · p. 23

6.5.2 CONSTRUCTION AND ADAPTATION STATISTICS

  • 55.0% of T4 questions are source-supported constructed questions, compared with 7.8%–11.0% for T1–T3.The constructed items are grounded in traceable engineering materials and explicit acceptance conditions.Source: §6.5.2 · pp. 20–21, Table 26 · p. 24

6.5.3 SOURCE TYPE COMPOSITION

  • Official qualification or licensing examinations account for 59.7% of source documents and 47.8% of questions.The largest single source is a human-verified question–answer compilation, contributing 249 questions (7.7%).Source: §6.5.3 · p. 21, Table 27 · p. 24

6.5.4 SOURCE LANGUAGE DISTRIBUTION

  • All benchmark inputs are in English; non-English source materials were translated into English during normalization.The 357 source documents include English, German, Japanese, Chinese, and mixed-language materials.Source: §6.5.4 · p. 22, Table 28 · p. 25

6.5.5 VISUAL ASSETS AND MULTI-IMAGE INPUTS

  • The questions reference 2,275 distinct images, with 483 multi-image questions accounting for 15.0% of the benchmark.There are 3,936 image references in total, and 644 images are shared by multiple questions.Source: §6.5.5 · p. 22, Table 29 · p. 25

6.5.6 INPUT STATISTICS BY TASK

  • T4 questions have a median length of 114 characters, compared with 35 for T1, and all tasks are dominated by single-image inputs.T4 prompts specify objectives, candidate modifications, editable boundaries, or constraint conditions.Source: §6.5.6 · p. 22, Table 30 · p. 26

6.5.7 SOURCE CONCENTRATION AND INDEPENDENCE

  • The 3,229 questions link to 357 source documents organized into 227 source groups, with experimental splits constructed at the source-group level.The largest source contributes 249 questions (7.7%); the ten largest sources contribute 24.7%.Source: §6.5.7 · pp. 22–23, Table 31 · p. 26

6.5.8 DEDICATED AUDIT OF T4 DESIGN INTERVENTIONS

  • Before revision, 218/240 T4 questions (90.8%) passed all applicable audit criteria.Protected-constraint completeness and regression-condition completeness were 95.4% and 93.8%; revised items were rechecked.Source: §6.5.8 · p. 23, Table 32 · p. 26
Loading 2609.33261v1…