PaperlayerView exact revision on arXiv

Source-linked AI summary

Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs

Xingang Guo, Yaxin Li, Xiangyi Kong, Yilan Jiang, Xiayu Zhao, Zhihua Gong, Yufan Zhang, Daixuan Li, Tianle Sang, Beixiao Zhu, Gregory Jun, Yingbing Huang, Yiqi Liu, Yuqi Xue, Rahul Dev Kundu, Qi Jian Lim, Yizhou Zhao, Luke Alexander Granger, Mohamed Badr Younis, Darioush Keivan, Nippun Sabharwal, Shreyanka Sinha, Prakhar Agarwal, Kojo Vandyck, Hanlin Mai, Zichen Wang, Aditya Venkatesh, Ayush Barik, Jiankun Yang, Chongying Yue, Jingjie He, Libin Wang, Licheng Xu, Hao Chen, Jinwen Wang, Liujun Xu, Rushabh Shetty, Ziheng Guo, Dahui Song, Manvi Jha, Weijie Liang, Weiman Yan, Bryan Zhang, Sahil Bhandary Karnoor, Jialiang Zhang, Rutva Pandya, Xinyi Gong, Mithesh Ballae Ganesh, Feize Shi, Ruiling Xu, Yifan Zhang, Yanfeng Ouyang, Lianhui Qin, Elyse Rosenbaum, Corey Snyder, Peter Seiler, Geir Dullerud, Xiaojia Shelly Zhang, Zuofu Cheng, Pavan Kumar Hanumolu, Jian Huang, Mayank Kulkarni, Mahdi Namazifar, Huan Zhang, Bin Hu

arXiv:2509.16204v2cs.CEcs.HCcs.RO

TL;DR

Practical engineering design requires domain-knowledge synthesis, trade-off management, and functional validation, but existing benchmarks do not capture these demands. ENGDESIGN evaluates design tasks across nine engineering domains through executable, simulation-based pipelines. The benchmark remains challenging for current models and exposes limitations that traditional benchmarks miss.

  • Problem

    Existing benchmarks do not capture practical engineering design’s open-ended, multidisciplinary demands for domain-knowledge synthesis, trade-off navigation, and simulator-based validation.

    Source: §1 · p. 2

  • Method

    ENGDESIGN benchmarks functional engineering designs across nine domains using task-specific executable pipelines for simulation-based validation and partial-credit grading.

    Source: §1 · p. 2

  • Results

    The top-performing model, o3, achieves 34.38% overall pass rate, while ENGDESIGN exposes model limitations invisible to traditional benchmarks.

    Source: §1 · p. 3, §3.1 · p. 7

  • Takeaways & Limitations

    ENGDESIGN provides a standardized, reproducible testbed for evaluating end-to-end design synthesis, constraint satisfaction, and simulation-based validation.

    Source: §5 · p. 10

    Iterative-design results cover a resource-limited subset of 71 tasks and four models, with pass rates slightly below Table 2.

    Source: §3.3 · p. 9

Abstract

from arXiv · show

Modern engineering, spanning electrical, mechanical, aerospace, civil, and computer disciplines, stands as a cornerstone of human civilization and the foundation of our society. However, engineering design poses a fundamentally different challenge for large language models (LLMs) compared with traditional textbook-style problem solving or factual question answering. Although existing benchmarks have driven progress in areas such as language understanding, code synthesis, and scientific problem solving, real-world engineering design demands the synthesis of domain knowledge, navigation of complex trade-offs, and management of the tedious processes that consume much of practicing engineers' time. Despite these shared challenges across engineering disciplines, no benchmark currently captures the unique demands of engineering design work. In this work, we introduce EngDesign, an Engineering Design benchmark that evaluates LLMs' abilities to perform practical design tasks across nine engineering domains. Unlike existing benchmarks that focus on factual recall or question answering, EngDesign uniquely emphasizes LLMs' ability to synthesize domain knowledge, reason under constraints, and generate functional, objective-oriented engineering designs. Each task in EngDesign represents a real-world engineering design problem, accompanied by a detailed task description specifying design goals, constraints, and performance requirements. EngDesign pioneers a simulation-based evaluation paradigm that moves beyond textbook knowledge to assess genuine engineering design capabilities and shifts evaluation from static answer checking to dynamic, simulation-driven functional verification, marking a crucial step toward realizing the vision of engineering Artificial General Intelligence (AGI).

1 Introduction

Practical engineering design requires LLMs to synthesize domain knowledge, manage trade-offs, and validate functional designs, yet existing benchmarks do not capture these demands. ENGDESIGN evaluates open-ended design tasks across nine domains using executable, simulation-based assessment.

1 Introduction

  • Practical engineering design demands domain-knowledge synthesis, trade-off navigation, and simulator-based validation, but existing benchmarks do not capture its open-ended, multidisciplinary demands.Source: §1 · p. 2
  • ENGDESIGN introduces a benchmark for practical engineering design across nine domains, requiring domain-knowledge synthesis, constraint-based reasoning, and functional, objective-oriented solutions.Its tasks include operating system, computer architecture, control system, mechanical, structural, digital hardware, analog integrated circuit, robotics, and signal processing design.Source: §1 · p. 2
  • Each task uses a domain-specific evaluation pipeline to validate designs through simulations and other functional checks rather than linguistic pattern matching.Examples include SPICE simulations, finite element analysis, and the MATLAB Control System Toolbox.Source: §1 · p. 2
  • ENGDESIGN uses partial-credit rubrics to quantify incremental progress on incomplete solutions, rather than relying only on binary scoring.Source: §1 · p. 2
  • Evaluations across 10+ state-of-the-art models show that ENGDESIGN exposes over-reliance on textual patterns and difficulty handling trade-offs that traditional benchmarks miss.Source: §1 · p. 3

2 ENGDESIGN

ENGDESIGN contains 101 design tasks across nine engineering domains and uses simulation-based evaluation for practical design problems. Each task includes design goals and constraints, with review and validation steps during benchmark construction.

2 ENGDESIGN

  • ENGDESIGN contains 101 tasks across 9 domains and 473 gradable items; 23 tasks include images, and the benchmark is released as ENGDESIGN-OPEN.Source: §2 · pp. 3, 5

2.1 Task Structure

  • Each task pairs a detailed design prompt with a rubric, automated evaluation pipeline, and validated reference design; rubrics award partial credit for incomplete designs.Source: §2.1 · p. 5

2.2 Evaluation Pipeline

  • Evaluation parses structured design fields, runs task-specific simulations against rubrics, and returns pass/fail results, 0–100 scores, and evaluation logs.Source: §2.2 · pp. 5–6

2.3 Construction of ENGDESIGN

  • Contributors propose complete tasks; o4-mini pretests prompt sufficiency and code, reviewers assess feasibility and rubric quality, domain experts vet technical soundness, and accepted tasks are standardized.Source: §2.3 · pp. 6–7

3 Experiments

ENGDESIGN evaluates LLMs on engineering design tasks, with no model exceeding a 35% overall pass rate. Iterative refinement improves performance, while failure analysis highlights persistent domain-knowledge and constraint-handling challenges.

3.1 Main Results

  • No model exceeds 35% overall pass rate: o3 reaches 34.38%, GPT-4o and Gemini-2.0-Flash attain around 15%, and all models score 0% on Analog IC design.The benchmark requires domain-specific expertise, multi-step reasoning, and design trade-offs.Source: §3.1 · p. 7
  • Reasoning models including o3, o4-mini, and DeepSeek-R1 outperform general-purpose chat models across most engineering domains, except Claude models show similar pass rates.Claude-3.7-Sonnet and Claude-3.7-Thinking exhibit similar pass rates despite different reasoning emphasis.Source: §3.1 · pp. 7–8
  • 0.62, 0.61, and 0.57 are robustness scores for o1, o3, and o4-mini-high; the reasoning/chat robustness gap reaches 3× versus a 2× pass-rate gap.Gemini-2.0-Flash and DeepSeek-v3 have robustness scores of 0.20 and 0.35, respectively.Source: Figure 4 · p. 8, §3.1 · p. 8
  • o1 generates significantly more completion tokens than GPT-4o to achieve performance gains, making the latency–performance trade-off relevant to deployment.Figure 5 presents average token consumption across evaluated models.Source: Figure 5 · p. 8, §3.1 · p. 8

3.2 Iterative Design

  • For iterative design, models receive their previous design and evaluation feedback, including scores, performance metrics, and diagnostic logs, before generating an improved design.The iterative prompt is shown in Figure 6.Source: §3.2 · p. 8, Figure 6 · p. 9
  • o3 achieves almost a 60% pass rate after ten iterations, but models still fail to meet requirements on Analog IC design tasks.These iterative experiments cover 71 tasks and four representative models.Source: Figure 6 · p. 9, §3.2 · p. 9, §3.3 · p. 9

3.3 Failure Analysis

  • Failures were manually annotated into six categories, with multi-label tagging for multiple failure modes; Table 3 summarizes three representative models.The models are o4-mini, Gemini-2.5-Pro, and Claude-3.7-Sonnet.Source: §3.3 · p. 9, Table 3 · p. 9
  • Domain-knowledge errors and constraint violations account for roughly 55–67% of failures; prior-knowledge overreliance and hallucinations contribute 25–30%, while computation errors are ≤9%.The analysis identifies numerical computation as a less frequent failure category than the other reported modes.Source: §3.3 · p. 10

4 Related Work

Prior LLM research covers general benchmarks and emerging engineering applications, but engineering studies often center on QA or a single domain. ENGDESIGN instead combines multiple engineering domains with simulation-based evaluation of generative designs.

General Purpose Benchmarks for LLMs

LLMs in Engineering Domains

  • Engineering studies include curated QA datasets and domain-specific design workflows, but many focus on factual recall or one subfield.Examples include ControlAgent [Guo et al., 2024], AnalogCoder [Lai et al., 2024], and SPICED [Chaudhuri et al., 2024].Source: §4 · p. 10
  • ENGDESIGN spans multiple engineering domains and combines LLM-generated outputs with simulation-based evaluation in realistic design settings.The paper frames this as a distinct evaluation approach compared with prior work.Source: §4 · p. 10

5 Conclusion

The paper presents ENGDESIGN as a comprehensive benchmark for realistic, multi-domain engineering design. Its results show that even frontier models achieve only modest performance, and the authors propose the benchmark as a reproducible testbed for future research.

5 Conclusion

  • ENGDESIGN evaluates end-to-end design synthesis, constraint satisfaction, and simulation-based validation across diverse engineering disciplines.The conclusion characterizes it as a comprehensive benchmark for realistic, multi-domain engineering design scenarios.Source: §5 · p. 10
  • Even the most capable frontier models achieve only modest performance on ENGDESIGN.The conclusion presents the benchmark as a substantial challenge for current LLMs.Source: §5 · p. 10
  • The authors envision ENGDESIGN as a standardized, reproducible testbed to support research at the intersection of language, reasoning, and engineering intelligence.They hope it will accelerate progress toward general-purpose AI engineers.Source: §5 · p. 10

NeurIPS Paper Checklist

The checklist states that the paper’s main claims reflect its contributions and scope, with dataset details and experimental results identified as supporting material. It also provides submission instructions and guidance for answering checklist questions.

Checklist Instructions

  • Checklist responses should use Yes, No, or NA and include a short justification immediately after each answer.NA means the question is inapplicable or relevant information is unavailable.Source: NeurIPS Paper Checklist · p. 18

Checklist Guidelines

  • Claims should reflect the paper’s results, scope, assumptions, and limitations; aspirational goals are acceptable when clearly distinguished from achieved outcomes.The checklist asks authors to identify supporting sections for Yes answers.Source: NeurIPS Paper Checklist · p. 18

Checklist

  • The authors answer that the paper’s main claims accurately reflect its contributions and scope, citing dataset details in Section 2 and experimental results in Section 3.The stated contribution is the ENGDESIGN benchmark for evaluating current LLM engineering-design capabilities.Source: NeurIPS Paper Checklist · p. 18

2. Limitations

The authors state that the work’s limitations are discussed in Appendix A and report open access to benchmark data and evaluation resources. They also point to Appendix D.1 for experimental details and report three evaluation trials per task.

2. Limitations

  • The authors say the work’s limitations are discussed in Appendix A.Source: §2 · p. 19
  • Benchmark data, evaluation logs, and the task evaluation pipeline are made openly accessible.Source: §2 · p. 20
  • Experimental setup and required compute resources are described in Appendix D.1.Source: §2 · p. 21
  • Each task was evaluated in three independent trials, with mean and standard deviation reported.Source: §2 · p. 21

Limitations

ENGDESIGN evaluates end-to-end engineering design across domains, while the authors note limits in domain coverage, evaluated models, and structured-output adherence. The supplement also reports prompt lengths, benchmark construction, and broader-impact cautions.

Limitations

  • The benchmark covers 9 engineering domains, but not the full breadth of engineering; its model selection is also not exhaustive, and some models violate the expected output format.The authors note that instructor remains under active development and may need greater robustness and compatibility.Source: Limitations · p. 26

Broader Impact

  • LLMs are not presented as replacements for engineers, and safety-critical designs require validation and safety assessment by qualified human experts.The authors describe potential assistance with routine tasks and early-stage design, particularly where expert resources are limited.Source: Broader Impact · p. 26

B More on Related Work

  • ENGDESIGN emphasizes end-to-end design reasoning and simulation-based validation, unlike benchmarks focused on partial tasks.Source: B More on Related Work · p. 27

C.1 Prompt Token Length Comparison

  • 778.71 tokens is ENGDESIGN’s average prompt length, versus 61.76 for MMLU-Pro, 250.03 for HLE, and 58.46 for GSM8K.The authors attribute the longer prompts to the contextual and structural complexity of engineering design tasks.Source: §C.1 · p. 27, Table 4 · p. 28

C.2 More on construction of ENGDESIGN

  • Over 40 contributors proposed 186 tasks; 101 were accepted and 85 discarded after two review phases.Rejected proposals had issues such as insufficient clarity, infeasibility, or misalignment with the benchmark’s design goals.Source: §C.2 · p. 28

C.3 Word Clouds of ENGDESIGN

The supplement uses word clouds to show frequent terms across ENGDESIGN prompts and engineering domains, and identifies the benchmarking pipeline as the focus of the following evaluation section.

C.3 Word Clouds of ENGDESIGN

  • Word clouds summarize frequently occurring terms across all prompts and within each engineering domain, highlighting design vocabulary and contextual diversity.Source: §C.3 · p. 28, Figure 7 · p. 28

D More on Evaluation

D.1 Detailed Experimental Setup

The experimental setup uses default LLM configurations and API access to proprietary models, while structured responses support automated parsing of design choices and code. The response format separates reasoning from task-specific configuration.

D.1 Detailed Experimental Setup

  • Evaluations use each LLM’s default configuration and API access to proprietary models, without local GPU infrastructure or model training.The authors report that inference-time API usage is the primary computational cost.Source: §D.1 · p. 28

D.2 LLM Response Structure

  • The response structure has reasoning and config keys; a task-specific ConfigFile guides the model to provide design components in a predefined format.This structure facilitates automated parsing of design choices or code snippets.Source: §D.2 · p. 30

D.3 More on Evaluation Metrics

ENGDESIGN aggregates results across independent trials and weights tasks equally. It also defines Reasoning Robustness to capture whether solvable tasks are passed consistently across repeated attempts.

D.3 More on Evaluation Metrics

  • Each task is tested over 3 independent trials, with metrics aggregated per task and then summarized across tasks using equal weighting.The overall pass rate is the mean of per-task pass rates, reported with its standard deviation across tasks.Source: §D.3 · pp. 30–31
  • For each task, Average Score averages trial scores; the overall score is the mean of those per-task scores, with a corresponding standard deviation.The trial score is denoted x_i,j.Source: §D.3 · p. 31
  • Reasoning Robustness measures consistency as the proportion of tasks passed in all trials among tasks passed at least once.A value of 1 indicates consistent success whenever a task is solvable; lower values indicate greater instability across attempts.Source: §D.3 · p. 31

D.4 Additional Evaluation Results

Additional results show that reasoning-focused models score highest, while score differences are less pronounced than pass-rate differences. The appendix also examines token consumption, filtering prompts, and chain-of-thought results and examples.

D.4 Additional Evaluation Results

  • Reasoning-focused models o4-mini-high, o3-high, and o3 achieve the highest average scores, with less pronounced differences than in pass rates.This suggests that models can earn substantial credit for partially correct or near-complete solutions without consistently meeting strict pass criteria.Source: §D.4 · p. 32
  • The appendix also reports text-only and ENGDESIGN-OPEN pass-rate and score results, along with results using error bars.Table 6 reports average scores across domains.Source: §D.4 · p. 32

D.5 Token Consumption

D.6 Prompt

  • The initial filtering prompt asks o4-mini to identify a task’s engineering domain, summarize its objective, and judge whether its description is sufficient, using JSON output.The prompt is described as part of ENGDESIGN’s second-stage task filtering.Source: §D.6 · pp. 32, 37

D.7 Prompt Optimization and Variants

  • Chain-of-thought prompting generally improves both evaluation metrics, except for signal processing and operating-system score results.Source: §D.7 · p. 38, Table 13 · p. 38
  • In WJ_01, CoT adds bilateral filtering to a median-filter pipeline, while the simpler two-step solution achieves higher accuracy and efficiency.Source: §D.7 · p. 38, Table 14 · p. 38
  • For operating-system tasks CY_03 and XW_04, CoT cases include incorrect API assumptions, over-analysis, or no implementation.Source: §D.7 · p. 38

E More on Failure Analysis

This section analyzes LLM failures in circuit, control, and structural design tasks, including misapplied domain knowledge, ignored constraints, unsupported computation claims, and shortcuts behind a hallucinated compliance estimate. The examples show that plausible or coherent responses do not necessarily yield complete, compatible, or reliable designs.

E More on Failure Analysis

  • In loop shaping, o4-mini correctly formed the controller but normalized its transfer function, altering the designed gain through generic manipulation rather than preserving gain as required.The analysis classifies this as both a domain knowledge error and prior-knowledge overreliance.Source: E More on Failure Analysis · p. 39
  • In the Hybrid A* task, o4-mini reported a 49.96 m path without using map obstacles or executing the algorithm, fabricating a result from intuition.It provided no calculation process, search trace, or steering behavior to support the claimed path length.Source: E More on Failure Analysis · p. 40
  • The SLC-3 design violated explicit hardware constraints, including setting LDR’s ld_cc to 0, omitting required memory-operation states, and failing to return several instructions to s_18.The analysis identifies these as structural and architectural errors that compromise correctness, not minor oversights.Source: E More on Failure Analysis · p. 41
  • The model’s folded-cascode OTA response was structurally flawed, often omitting or misimplementing the topology, omitting required transistor parameter m, and violating formatting constraints.The analysis attributes this to inaccurate structural parameters, reliance on simplified examples and estimates, and failure to account for transistor interactions.Source: E More on Failure Analysis · pp. 45–46
  • The reported C_y_hat ≈0.061 is a hallucination: it misapplies 1D beam theory and assumes stiffness scales linearly with volume fraction, unlike stress-path material layouts.The LLM estimated compliance by treating the 2D topology-optimization problem as an equivalent beam and scaling stiffness with volume fraction.Source: E More on Failure Analysis · p. 47

F ENGDESIGN Examples

  • The control-system example asks for a reactor model derived from step-response timing and a feedback controller meeting specified stability-margin, overshoot, and settling-time requirements.The task gives t1 = 23 s and t2 = 36 s and requires gain margin ≥7 dB, phase margin ≥60 degrees, overshoot ≤10%, and settling time ≤150 s.Source: F ENGDESIGN Examples · pp. 47–48
  • The structural-design example asks for a minimum-maximum-stress layout under a 0.25 volume-fraction cap, with stress, volume fraction, and x-symmetry reported.Source: F ENGDESIGN Examples · p. 48
  • The analog-circuit example requires sizing a five-transistor OTA by changing only device widths, lengths, and multipliers, then verifying its gain, frequency, rejection, and input-range targets.The specifications include AV0 ≥40 dB, fUGF ≥50 MHz, CMRR ≥80 dB, and ICMR ≥800 mV, with additional endpoint checks at VSS and VDD.Source: F ENGDESIGN Examples · pp. 48–49

G Task-Level Breakdown of the ENGDESIGN

This section inventories ENGDESIGN’s tasks, describing their design challenges and associated references across diverse engineering domains. The examples show tasks requiring concrete designs, calculations, and constraint satisfaction.

G Task-Level Breakdown of the ENGDESIGN

  • Table 15 summarizes each ENGDESIGN task with a concise design-challenge description and its relevant references.The task inventory provides a granular view of the benchmark’s scope and structure.Source: Table 15 · p. 51, G Task-Level Breakdown of the ENGDESIGN · p. 51
  • Computer-system tasks include virtio block-driver implementation, page-table design, SSD-layout tuning, and resource-constrained accelerator execution planning.The accelerator examples include partitioning matrix multiplication across IPU cores and allocating NPU architecture and cluster resources for serving SLOs.Source: G Task-Level Breakdown of the ENGDESIGN · pp. 52–53
  • Control-system tasks include state-feedback and observer design, response analysis, loop-shaping filters, and stability assessment.Other listed tasks cover gain selection for an absolutely stable Lurye interconnection and exploration–exploitation parameter ranges in a two-phase bandit algorithm.Source: G Task-Level Breakdown of the ENGDESIGN · pp. 53–55
  • Mechanical-design tasks include topology and material optimization under structural constraints, plus collision-free robot trajectories around static and moving obstacles.The optimization examples target compliance under volume-fraction constraints; the trajectory tasks impose grid, timing, and speed constraints.Source: G Task-Level Breakdown of the ENGDESIGN · pp. 55, 56, 59
  • Digital-hardware tasks range from decoding a Tetris pattern to specifying an SLC-3 CPU and implementing a parameterizable bubble-sort module.Analog-circuit tasks include sizing folded-cascode and five-transistor OTAs, and designing a common-source amplifier to meet specified performance requirements.Source: G Task-Level Breakdown of the ENGDESIGN · pp. 57–59
  • Other tasks cover antenna design, polynomial fitting, optical measurement, image filtering and reconstruction, and radar detectability analysis.The listed requirements include antenna resonance and bandwidth targets, Stokes-vector measurement, edge detection, and polarization-image interpolation.Source: G Task-Level Breakdown of the ENGDESIGN · pp. 60–61
  • Additional signal-processing work asks models to select and parameterize denoising filters according to detected noise types.The benchmark also includes a task to design and implement an automated test stand using polarizers, a quarter-wave plate, and rotary motors.Source: G Task-Level Breakdown of the ENGDESIGN · p. 61
Loading 2509.16204v2…