Source-linked AI summary
SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation
Ruiqi Zhang, Jiahao Wang, Mingxuan Li, Haichen Luo, Chaoting Wang, Guoyu Mou, Keyu Lai, Hanchao Lv, Jiaxu Wang, Yibo Zheng, Aijun Yang, Xiaohua Wang
TL;DR
Existing Simulink benchmarks mainly test whether generated models execute or resemble a reference, not whether they satisfy engineering requirements. SimuVerity evaluates 101 tasks across ten domains using profile-grounded native simulation and hierarchical, multidimensional scoring. Across six agent systems, the highest overall score is 42.86, with bottlenecks both before and after engineering qualification.
Problem
Existing benchmarks do not center whether generated Simulink models satisfy engineering requirements across prescribed operating conditions.
Source: §1 · p. 2Method
SimuVerity profiles reference systems for 101 tasks across ten domains, grounds task-specific simulation scenarios in those profiles, and hierarchically evaluates qualified models across six dimensions.
Source: §1 · p. 2Results
42.86 overall is the highest score among six agent systems, with capability bottlenecks in engineering qualification and subsequent multidimensional performance.
Source: §5 · p. 9Takeaways & Limitations
SIMUVERITY provides a systematic basis for assessing agents’ engineering capabilities and diagnosing failures in executable model generation.
Source: §5 · p. 10The benchmark relies on MATLAB/SIMULINK, and native simulation-based evaluation can be slow.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.
1 INTRODUCTION
Existing Simulink benchmarks do not establish whether generated models meet engineering requirements across prescribed operating conditions. SimuVerity addresses this with profile-grounded native simulation and hierarchical evaluation, then assesses six agent systems.
1 INTRODUCTION
- 42.86 overall is the highest score among six agents; artifact-delivery, executability, and qualification rates are 94.72%, 86.47%, and 76.90%, respectively.Experts classified eight of 31 candidates scoring at least 70 from the highest-scoring agent as severely disordered.Source: Figure 1 · p. 1, §1 · p. 2
- Existing benchmarks emphasize artifact delivery, technical execution, or similarity to a reference, rather than requirement satisfaction across prescribed operating conditions.A reference may have errors, and structurally different implementations can still be valid.Source: §1 · p. 2
- Its evaluation uses four families of task-specific native simulation scenarios and six performance dimensions, following checks for delivery, executability, and engineering qualification.This hierarchical design supports multidimensional diagnosis, valid structural diversity, and detection of poor performance in reference-aligned models.Source: §1 · p. 2, Figure 2 · p. 3
- SIMUVERITY comprises 101 text-to-executable tasks across ten engineering domains, each grounded in an executable-system profile and engineering prompt.Tasks use reconstructed or independently built reference systems.Source: §1 · p. 2
2 RELATED WORK
Prior work increasingly evaluates executable engineering artifacts through native execution and requirement-level evidence. Simulink benchmarks, however, have often focused on tool faults, reconstruction, control validation, or reference similarity.
2 RELATED WORK
- Executable-engineering benchmarks increasingly use native execution and requirement-level evidence beyond artifact formation and basic validity checks.This motivates asking how execution evidence can support hierarchical, multidimensional assessment of dynamical-system compliance.Source: §2 · p. 3
- Prior Simulink studies address toolchain faults, block-diagram reconstruction, or automotive control validation; SimuBench spans domains but relies mainly on completeness, executability, and reference similarity.SIMUVERITY instead uses profile-grounded scenarios and hierarchical multidimensional evaluation.Source: §2 · p. 3
- Reference-based criteria can miss both structurally distinct valid implementations and engineering failures in candidates aligned with the reference.Native MATLAB/Simulink access further broadens the space of valid implementations.Source: §2 · p. 3
3 SIMUVERITY
SIMUVERITY frames text-to-executable Simulink generation as a build-to-specification task, then grounds its prompts and simulation tests in executable-system profiles. Its hierarchical evaluator applies prerequisite gates before scoring qualified models across six engineering dimensions.
3.1 TASK DEFINITION AND CROSS-DOMAIN COVERAGE
- SIMUVERITY defines model generation as delivering a complete native .slx model that meets a natural-language engineering specification, while leaving implementation structure unrestricted.Its 101 tasks span ten engineering domains, with 9–11 tasks per domain and near-balanced coverage.Source: §3.1 · pp. 3–4, Table 1 · p. 4
3.2 DATA CURATION
- Experts curate executable references, build observable system profiles, derive profile-consistent prompts, and validate each task’s native execution and evaluator.Quality control includes source-lineage tracking, duplicate screening, and withholding task-specific test conditions and scoring logic from evaluated agents.Source: §3.2 · p. 4
3.3 TASK-SPECIFIC SIMULATION TEST DESIGN
- Four scenario families probe operating envelopes, interactions and faults, temporal processes, and required mechanisms or causal paths through native simulations.Candidates are loaded, updated, compiled, and simulated in MATLAB/SIMULINK, with execution status, trajectories, engineering quantities, and mechanism checks used as evidence.Source: §3.3 · pp. 4–5
3.4 HIERARCHICAL MULTIDIMENSIONAL PERFORMANCE EVALUATION FRAMEWORK
- Three prerequisite gates screen delivery, native execution, and engineering qualification; qualified models receive six dimension scores mapped from scenario evidence and combined through task-specific aggregation.The dimensions are objective accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response.Source: §3.4 · pp. 5–6
4 EXPERIMENTS AND RESULTS
Across six agent systems, even the highest overall score is 42.86, and prerequisite success does not guarantee engineering qualification. Structural similarity also fails to reliably order performance, while domain bottlenecks, poor visual layouts, and tool restrictions expose distinct weaknesses.
4.1 EXPERIMENTAL SETUP
- All six systems were tested on the same 101 tasks under common MATLAB/SIMULINK conditions, with three fresh-session runs per task–system pair.The evaluation used identical specifications, delivery requirements, workspaces, and environments; reference models and scoring materials were hidden from agents.Source: §4.1 · p. 7
4.2 OVERALL PERFORMANCE
- 42.86 is the highest overall score among six systems, and the top system remains far from fully satisfying benchmark engineering requirements.Claude Opus 4.8 scores 42.86, followed by GPT-5.5 at 41.72 and DeepSeek-V4-Pro at 28.40.Source: §4.2 · p. 7
- 84.43% of task–system runs deliver an artifact, 68.65% execute natively, and 47.85% pass engineering qualification.Delivery and native executability alone do not guarantee a qualified implementation.Source: §4.2 · p. 7
- Output quality leads the six end-to-end dimensions at 38.46, while robustness and dynamic response are lowest at 27.69 and 27.82.Table 3 reports end-to-end and post-G scores across A/Q/M/C/R/D, averaged over each dimension’s applicable tasks.Source: §4.2 · p. 7, Table 3 · p. 8
4.3 STRUCTURAL SIMILARITY AND ENGINEERING PERFORMANCE
- Reference proximity does not reliably order engineering performance across Reference-Aligned, Partially Similar, and Structurally Distinct models.Of 24 top-quartile candidates, 12 are Partially Similar or Structurally Distinct; seven Reference-Aligned candidates receive zero, including two that pass G.Source: §4.3 · p. 7, Table 4 · p. 8
4.4 CROSS-DOMAIN CAPABILITY GAPS
- Claude Opus 4.8 shows qualification-dominated, post-qualification, and mixed capability gaps across the ten engineering domains.Power electronics and fluid systems are qualification-dominated; battery and aerospace show post-qualification bottlenecks; mechanical and communication systems show mixed bottlenecks.Source: §4.4 · p. 8
4.5 VISUAL LAYOUT QUALITY OF HIGH-PERFORMING SIMULINK MODELS
- Among eight severely disordered high-scoring models, layout-only reorganization eliminated overlap in every case and reduced line-crossing density by 22.7–85.1%.Routing-detour ratio fell in seven cases; block positions and existing line routes changed while implementations remained unchanged.Source: §4.5 · p. 8, Figure 4 · p. 9
4.6 ABLATIONS OF NATIVE SIMULATION FEEDBACK AND TOOL ACCESS
- On ten selected tasks, removing simulation feedback reduced the overall score from 78.46 to 7.88, while removing MCP reduced it to 18.23.Under No-Simulation MCP, delivery, executability, and qualification fell from 100.00% to 53.33%, 33.33%, and 13.33%; 8 of 10 Batch-only tasks scored lower.Source: §4.6 · p. 9
5 CONCLUSION
The conclusion frames SIMUVERITY as a native-simulation benchmark for engineering performance, finding that structural similarity and successful execution alone do not establish engineering quality. It also notes that native simulation can be slow and the benchmark depends on MATLAB/SIMULINK.
5 CONCLUSION
- SIMUVERITY evaluates 101 text-to-executable tasks across ten domains using native simulation and six engineering dimensions.The best of six evaluated systems scores 42.86, with bottlenecks both in qualification and in performance after qualification.Source: §5 · p. 9
- Structural similarity does not reliably reflect engineering quality, and high-scoring models may still fail requirements or have disordered layouts.The benchmark is presented as a systematic basis for assessing agents’ engineering capabilities and diagnosing executable-model-generation failures.Source: §5 · pp. 9–10
- Native simulation-based evaluation can be slow, and the benchmark relies on MATLAB/SIMULINK.The authors identify evaluation-pipeline efficiency and scalability as future work.Source: §5 · p. 10
A CLOSEST-WORK COMPARISON
- Table 6 summarizes the scope and evaluation criteria of the most relevant related studies.Its caption identifies the comparison as covering executable-engineering and SIMULINK benchmarks.Source: A CLOSEST-WORK COMPARISON · p. 13, Table 6 · p. 13
B TASK EXAMPLES
The examples illustrate profile-grounded tasks with explicit engineering goals, constraints, inputs, outputs, and validation scenarios. The scoring appendix explains scenario coverage, dimension applicability, qualification checks, and weighted aggregation for an electric-axle candidate.
B TASK EXAMPLES
- XAERO-04 specifies a closed-loop tiltrotor model spanning hover, transition, and fixed-wing flight, with six validation scenarios.The required connected paths couple propulsion, tilt actuation, aerodynamic control surfaces, vehicle dynamics, and feedback control.Source: B TASK EXAMPLES · p. 13
- XAERO-07 tests an electric-aircraft powertrain by varying battery capacity and payload across five scenarios.The specification requires battery capacity to affect stored energy and mass, payload to affect power demand, and electrical and mechanical paths to participate in execution.Source: B TASK EXAMPLES · p. 14
- XAUTO-07 couples closed-loop motor control, an electric drivetrain, and vehicle dynamics for acceleration and regenerative braking.Its five scenarios include nominal operation, reduced demands, cyclic operation, and asymmetric pedal excitation; prescribed or replayed trajectories are disallowed.Source: B TASK EXAMPLES · p. 14
C.1 SCENARIO COVERAGE
- The 595 formal scenarios are each assigned one primary category, with cross-category contributions not counted twice.Table 7 summarizes scenario counts by primary category.Source: §C.1 · p. 14, Table 7 · p. 14
C.2 DIMENSION APPLICABILITY
- Domain experts select applicable dimensions from each task’s engineering requirements and system characteristics, applying the same set to every agent on that task.Table 8 summarizes task counts for the six dimensions.Source: §C.2 · p. 14, Table 8 · p. 14
- The engineering-qualification gate evaluates only delivered, natively executing candidates and checks required functional roles, signal-path connectivity, and valid evaluation evidence.The gate returns pass or fail and is designed by human experts from the task specification and executable-system profile.Source: §C.2 · p. 15
C.3 ELECTRIC-AXLE SCORING EXAMPLE
- For XAUTO-07, five scenarios cover all four scenario categories, and Table 9 associates scenarios with engineering checks and performance dimensions.All scenarios undergo output-completeness checks and contribute to dimension scores and cross-scenario evaluation.Source: §C.3 · p. 15, Table 9 · p. 15
- The electric-axle score uses expert-set weights in a weighted geometric mean, so one strong dimension cannot fully offset a critical weakness in another.The Opus Run 1 candidate passes all three prerequisite checks and scores 88.03 using unrounded dimension scores.Source: §C.3 · p. 16
C.4 EXPERT VALIDATION OF EVALUATOR VALIDITY
Evaluator rankings align closely with independent expert judgments, while system tiers and tool-ablation comparisons remain stable across task-level analyses.
C.4 EXPERT VALIDATION OF EVALUATOR VALIDITY
- Evaluator scores show strong agreement with expert ratings: across 30 models, mean absolute difference is 0.37 points and weighted Cohen’s κ is 0.90.Experts matched the evaluator in 21 cases, differed by 1 point in 7, and by 2 points in 2; the κ 95% CI is [0.79, 0.96].Source: §C.4 · pp. 16–17, Table 11 · p. 17
D.1 IMPLEMENTATION SETTINGS
- Anthropic-compatible systems share the Claude Code harness, GPT-5.5 uses Codex, and agents and MATLAB sessions run with outbound connections blocked.Reference files, scenario definitions, and scoring scripts remain inaccessible to agents during runs.Source: §D.1 · p. 17
D.2 STATISTICAL RELIABILITY
- Uncertainty is estimated by bootstrapping tasks, while run-to-run variation is reported separately.Task scores average three independent runs; pairwise comparisons use paired task-level differences and the Wilcoxon signed-rank test.Source: §D.2 · p. 17
- The six systems form four tiers, with significant differences between adjacent tiers and consistent qualification outcomes across 81.19–92.08% of tasks.Only Claude Opus 4.8 and GPT-5.5 exchange positions across runs; within-tier differences are not significant.Source: §D.2 · p. 18, Table 13 · p. 18, Table 14 · p. 18
- System ordering below the top pair is unchanged under three alternative aggregation rules, with Kendall’s τ of 0.87, 1.00, and 0.87.These are the arithmetic mean, equal-weight geometric mean, and minimum rules, respectively.Source: §D.2 · p. 18, Table 15 · p. 18
- Full MCP exceeds No-Simulation MCP by 70.58 points and Batch-only by 60.23 points; those restricted conditions score lower than Full MCP on 10/10 and 8/10 tasks, respectively.Differences were significant (No-Simulation: 95% CI [58.02, 78.63], Wilcoxon p = 0.002; Batch-only: 95% CI [37.13, 77.58], p = 0.010).Source: §D.2 · p. 18
D.3 CROSS-SYSTEM DOMAIN RESULTS
The systems show distinct domain profiles, and substantial differences remain after engineering qualification; the appendix also reports six-dimensional scores for four qualified candidates.
D.3 CROSS-SYSTEM DOMAIN RESULTS
- Systems have different domain-performance profiles end-to-end, and substantial cross-system differences persist after engineering qualification.Table 17 reports end-to-end and post-G overall scores across ten domains for six systems.Source: §D.3 · p. 19, Table 17 · p. 19
- GPT-5.5 is relatively stronger end-to-end in thermal, robotics, biomedical, and fluid domains, while its qualified candidates are stronger in fluid and thermal.DeepSeek-V4-Pro is relatively stronger in robotics and fluid, while Qwen3.8-Max’s end-to-end scores concentrate in mechanical, robotics, thermal, and biomedical domains.Source: §D.3 · p. 19
E.1 SIX-DIMENSIONAL PERFORMANCE
- Table 19 reports six-dimensional scores for four representative candidates, all of which pass the engineering-qualification gate.An em dash marks a non-applicable dimension.Source: Table 19 · p. 21
E.2 STRUCTURAL FEATURES AND ENGINEERING OUTCOMES
Structural relations are classified qualitatively using model architecture and signal routing, while supplementary interventions isolate visual layout changes from model content.
E.2 STRUCTURAL FEATURES AND ENGINEERING OUTCOMES
- Structural relations are classified from block choices and abstraction, topology, subsystem decomposition, and signal and feedback routing.The relations are mutually exclusive; layout coordinates, colors, labels, parameter values, solver settings, and block-count differences alone do not determine a label.Source: §E.2 · p. 21, Table 20 · p. 21
- Representative cases in the structural-similarity analysis illustrate valid structural diversity and deceptive structural similarity.Table 21 presents the cases, but the supplied passages do not specify their individual outcomes.Source: §E.2 · p. 21, Table 21 · p. 22
F ADDITIONAL VISUAL LAYOUT INTERVENTIONS
- The seven supplementary layout comparisons hold model content and simulation settings fixed, changing only block and subsystem positions and existing line routing.The fixed properties include blocks, parameters, ports, connectivity, hierarchy, configuration, and solver.Source: F ADDITIONAL VISUAL LAYOUT INTERVENTIONS · p. 22
F.1 SUPPLEMENTARY METRIC CONVENTIONS •
- Layout metrics count positive-area block overlaps, traceable directed-path detours, and internal crossings between different line networks, with specified exclusions.Boundary contact alone, zero-distance or undirected paths, shared endpoints, same-network branches, and collinear overlap are excluded.Source: §F.1 · p. 22
F.2 LAYOUT-ONLY INTERVENTION RESULTS
The layout-only intervention changes positions and line routes while preserving model and simulation content; supplementary figures show seven example systems.
F.2 LAYOUT-ONLY INTERVENTION RESULTS
- Experts reorganize eight high-performing but severely disordered models by changing layout only, and all three indicators favor lower values.Blocks, parameters, ports, connectivity, hierarchy, simulation settings, and solver configuration remain unchanged.Source: §F.2 · p. 22, Table 22 · p. 23
F.3 SEVEN SUPPLEMENTARY FIGURES
- The supplementary figures depict a QPSK communication link, wave-energy converter, CPU cooling system, ventilation system, two-level inverter, hybrid-electric vehicle, and three-level NPC inverter.Source: Figure 6 · p. 23, Figure 7 · p. 24, Figure 8 · p. 24
G.1 FIXED TASKS AND TOOL CONDITIONS
The appendix specifies three MATLAB tool conditions and reports task-level scores averaged across three runs. It also provides supplementary six-dimensional results for Claude Opus 4.8 with Claude Code.
G.1 FIXED TASKS AND TOOL CONDITIONS
- Full MCP reuses the main experiment’s three-run mean results for Opus 4.8 on the ten tasks.Source: §G.1 · p. 26
- No-Simulation MCP retains MATLAB MCP tools for model work and compilation but blocks agent-side simulation selfchecks during generation.The server rejects simulation calls, including sim invocations through the code-execution tool.Source: §G.1 · p. 26
- Batch-only removes MATLAB MCP access but retains standard file and shell tools, allowing stateless MATLAB batch scripts through shell calls.Source: §G.1 · p. 26
- Table 23 reports task-level engineering scores across ten cross-domain tasks, with each entry averaged over three runs.The caption does not provide the numerical scores.Source: Table 23 · p. 26
G.2 SUPPLEMENTARY SIX-DIMENSION PERFORMANCE
- Table 24 reports end-to-end six-dimensional tool-ablation results for Claude Opus 4.8 with Claude Code across ten cross-domain tasks.The caption does not provide the numerical results.Source: Table 24 · p. 26