PaperlayerView exact revision on arXiv

Source-linked AI summary

EngiWorld: What Can Frontier Agents Deliver in Professional Engineering Environments?

Hongcheng Gao, Hailong Qu, Yu Lei, Henghui Sun, Haoyang Li, Yipeng Wei, Naihao Xue, Xiaohan Yu, Zhuo Tao, Yihe Zang, Yajiao Wang, Jingyi Tang, Yi Li, Jingjing Zhou, Jie Luo, Bohan Zeng, Chengyu Shen, Hao Jiang, Chong Chen, Bowen Qu, Olive Huang, Zeqiang Wang

arXiv:2609.37686v1cs.AIcs.CL

TL;DR

Professional engineering automation remains difficult to assess because workflows combine artifact constraints, long operations, and dependencies across software stages. EngiWorld benchmarks the complete design loop and evaluates artifacts with domain-specific verification; its results show that frontier agents still perform poorly on engineering workflows.

  • Problem

    Existing computer-use benchmarks do not fully assess engineering artifact integrity, long-horizon interactions, native tool feedback, or workflows spanning the complete design loop.

    Source: §1 · p. 1

  • Method

    EngiWorld provides 1,301 expert-curated tasks across six engineering domains and 26 platforms, with GUI and CLI interfaces and artifact-centric domain verification.

    Source: §1 · p. 2

  • Results

    Claude Opus 5 achieves the highest overall EngiScore at 44.3, while only 3.6% of multi-software attempts succeed across seven frontier agents.

    Source: §4.2 · p. 7, §5 · p. 10

  • Takeaways & Limitations

    EngiWorld provides a rigorous foundation for research on agents operating professional engineering software end to end.

    Source: §1 · p. 3, §5 · p. 10

Abstract

from arXiv · show

Autonomous agents have made rapid progress in general-purpose computer use, but reliable automation of professional industrial engineering remains out of reach, as engineering workflows demand reasoning over geometric and physical constraints and dependencies preserved across software and design stages. We present EngiWorld, the first benchmark structured around the complete design loop: 1,301 expert-curated tasks spanning 6 engineering domains (CAD, CAE, CAM, BIM, EDA, and 3D visualization) and 26 professional software platforms, with both GUI and CLI interfaces and 6 task types ranging from software-selection to open-ended tasks. We further introduce an artifact-centric evaluation methodology built on a unified domain-verifier suite, which programmatically checks the geometric validity, physical feasibility, and rule compliance of final and intermediate artifacts, and scores quantitative design tasks continuously by specification attainment rather than binary success. Evaluation of seven frontier models reveals a substantial capability gap: the strongest model achieves an EngiScore of only 44.3, and just 3.6% of multi-software attempts succeed. EngiWorld provides the first rigorous foundation for measuring progress toward agents that operate professional engineering software end to end.

1 INTRODUCTION

The introduction identifies gaps in evaluating professional engineering workflows and presents EngiWorld as a complete-design-loop benchmark with artifact-centric evaluation. Its evaluation finds that frontier agents still fall short of industrial-grade engineering demands.

1 INTRODUCTION

  • Existing benchmarks often check final states rather than artifact integrity, cover only one platform or task format, limit interaction depth, or omit native tool feedback.Engineering work also requires maintaining parametric logic across long operation sequences and completing the full design loop.Source: §1 · pp. 1–2
  • EngiWorld comprises 1,301 expert-curated tasks across six engineering domains and 26 platforms, using GUI and CLI interaction across six task types covering the design process.Tasks include long-horizon and multi-software workflows, as well as software selection, quantitative design, and open-ended tasks.Source: §1 · p. 2
  • A unified domain-verifier suite checks final and intermediate artifacts for geometric accuracy, physical feasibility, and rule compliance, while quantitative design tasks receive continuous specification-attainment scores.Quantitative tasks are gated by feasibility checks rather than judged only by binary completion.Source: §1 · p. 2
  • Across seven agents, frontier models fall short of industrial-grade standards, with failures in cross-application coordination, geometric and physical constraints, verification, and attention to solver residuals.The authors characterize this as a profound gap between general intelligence and professional engineering demands.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans computer-use benchmarks, engineering-agent approaches, and domain-specific artifact evaluation. Existing benchmarks assess different workflow and artifact requirements, while the comparison framework distinguishes cross-software work, execution feedback, software choice, and engineering objectives.

Computer-Use Benchmarks

  • Computer-use benchmarks cover software engineering, web interaction, desktop operation, and professional data or scientific workflows, with newer work addressing long-horizon tasks and verifiable deliverables.OSWorld2 studies extended tasks, while Agents’ Last Exam includes engineering and 3D applications.Source: §2 · p. 3

Engineering Agents and Benchmarks

  • Engineering-agent work ranges from parametric CAD and hardware-description generation to tool-using approaches with planning, execution feedback, and professional GUI interaction datasets.The cited benchmarks and methods span CAD, electronic design, and numerical simulation.Source: §2 · p. 3
  • Engineering benchmarks check artifacts through functional simulation, executable constraints, geometric or topological criteria, and visual comparisons across domains.The comparison with existing benchmarks tracks cross-software workflows, execution feedback, software selection, and engineering performance targets.Source: §2 · p. 3, Table 1 · p. 3

3 THE ENGIWORLD BENCHMARK

ENGIWORLD frames engineering work as interactive tasks in native software and evaluates the resulting artifacts against engineering requirements. It combines GUI and CLI interaction, broad expert-calibrated task coverage, and artifact-based verification across six domains.

3.1 BENCHMARK OVERVIEW AND TASK FORMULATION

  • Each task specifies an objective, inputs, software environment, available actions, required deliverables, and acceptance criteria, then proceeds through a closed-loop sequence of actions and feedback.The interaction model treats the full software state as partially observable and terminates when the agent declares DONE or FAIL or reaches its decision limit.Source: §3.1 · p. 4

3.2 ENVIRONMENTS AND INTERACTION

  • Tasks run in isolated Windows or Ubuntu workspaces, and GUI or CLI interaction shares the same artifact-based evaluation protocol.Multi-software tasks transfer intermediate artifacts across applications; software-selection tasks let agents choose among permitted applications.Source: §3.2 · p. 5

3.3 TASK CONSTRUCTION AND COVERAGE

  • Experts identify representative workflows and convert them into executable specifications; sampled tasks and executions are reviewed, then every task is run and reviewed in its designated environment.Specifications include instructions, initial files, reference artifacts, required deliverables, and task-specific verifiers.Source: §3.3 · pp. 5–6
  • ENGIWORLD contains 1,301 tasks across six engineering domains and 26 software platforms, with 691 CLI tasks and 610 GUI tasks.Its 1,723 task–software associations count each application in a workflow and all permitted tools in software-selection tasks.Source: Figure 3 · p. 5, §3.3 · p. 6
  • The six task types range from single-software construction to open-ended work, covering image-based modeling, tool selection, quantitative design, and cross-application delivery.The partition includes 931 single-software, 60 multi-software, 140 software-selection, 40 quantitative-design, 120 image-based-modeling, and 10 open-ended tasks.Source: §3.3 · p. 6

3.4 ARTIFACT-CENTRIC EVALUATION

  • Verifiers reopen submitted files and check structural or tolerance-bounded criteria across engineering properties; multi-software tasks also check intermediate artifacts and cross-stage consistency.Checks cover geometry, simulation quantities, schematic connectivity, building-model entities and relationships, and 3D scene structure.Source: §3.4 · p. 6
  • Quantitative design tasks receive a quality score only after engineering feasibility passes, while binary tasks require all acceptance criteria to be satisfied.Infeasible quantitative submissions receive a quality score of zero.Source: §3.1 · p. 5, §3.4 · p. 6

4 EXPERIMENTS

EngiWorld evaluates seven frontier models across a stratified sample of professional engineering tasks, using artifact-based scores and measures of interaction cost. Results show low overall performance, sharp difficulty at cross-application transitions, and model-dependent effects from observation and history choices.

4.1 EXPERIMENTAL SETUP

  • The evaluation tests seven models in a zero-shot setting on the same stratified 300-task subset, with 152 CLI tasks and 148 GUI tasks drawn from all six domains.Episodes run in clean Windows or Ubuntu virtual machines with interface-specific decision limits; EngiScore aggregates binary success and quantitative design quality, weighting tasks equally on a 0–100 scale.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 MAIN RESULTS

  • Claude Opus 5 scores 44.3 overall, ahead of GPT-5.6 Sol at 38.0; all seven models score zero on 128 of 300 tasks.Claude and GPT perform similarly on CLI, while Claude has an advantage on GUI tasks.Source: §4.2 · p. 7
  • Only 6 of 168 Multi-software attempts succeed across all models, while Claude completes roughly half of Single-software tasks and two thirds of Image-based modeling tasks.Claude solves only one quarter of Software-selection tasks; Quantitative design and Open-ended results are reported in Appendix C.Source: §4.2 · pp. 7–8
  • Claude leads in CAD, CAE, EDA, and BIM and ties GPT in CAM, while Qwen3.8 Max leads in 3D visualization; CAM is hardest for every model.Tasks involving multiple domains contribute to each relevant domain.Source: §4.2 · p. 8, Figure 4 · p. 8
  • Models average 54.9–120.5 interaction steps per task; GPT uses the fewest at 54.9, while Claude attains the highest score at nearly twice GPT’s API cost.DeepSeek and Gemini score approximately 25, but DeepSeek costs less than half as much while generating 9.9K output tokens per step against Gemini’s 0.7K.Source: §4.2 · p. 8

GUI CLI

  • Reference drawings in the task message improve GUI performance for all three tested models, but the benefit is smaller on CLI and reverses for Kimi.GPT rises from 15.0 to 31.7, yet five previously successful tasks become failures; runtime image access helps Gemini on both CLI conditions but lowers Kimi’s score in both settings.Source: GUI CLI · pp. 8–9, Table 3 · p. 8, Table 4 · p. 8
  • Native 1920 × 1080 screenshots yield the highest score for all three models, while the best interaction-history length varies by model.Reducing resolution to 0.6× cuts GPT’s cost by 30.8% while preserving most of its score, but degrades Kimi more; Kimi’s 10-to-15-turn history extension lowers EngiScore and raises cost by 53.0%.Source: GUI CLI · p. 9, Figure 6 · p. 9

4.4 FAILURE ANALYSIS

  • Declared completion with zero score and decision-turn exhaustion account for 94.1% of main-evaluation failures, but reflect different breakdowns: validation versus construction.DONE is especially common among retained GPT and Claude failures, while turn exhaustion is more common on GUI than CLI and accounts for 55.3% and 36.6% of failures, respectively.Source: Figure 5 · p. 8, §4.4 · p. 9

5 CONCLUSION

EngiWorld is a complete-design-loop benchmark for professional engineering agents, pairing expert-curated tasks with artifact-centric evaluation. Across seven frontier agents, the strongest EngiScore is 44.3 and only 3.6% of multi-software attempts succeed.

5 CONCLUSION

  • EngiWorld covers six engineering domains, 26 software platforms, 1,301 expert-curated tasks, six task types, and both GUI and CLI workflows.Its tasks span individual tools and multi-stage workflows.Source: §5 · p. 10
  • The unified domain-verifier suite checks final and intermediate artifacts for engineering validity and preserves continuous quality scores for quantitative designs.Checks include geometric, physical, connectivity, simulation, and cross-stage consistency.Source: §5 · p. 10
  • Only 3.6% of multi-software attempts succeed, and the strongest of seven frontier agents reaches an EngiScore of 44.3.Failures concentrate in unsatisfied engineering constraints and deliverables not verified before completion.Source: §5 · p. 10

A DATASET CONSTRUCTION

EngiWorld’s dataset is built through expert-led stages that turn representative engineering scenarios into specified tasks, validated instances, and task-specific verification criteria.

A DATASET CONSTRUCTION

  • The expert-led, AI-assisted construction pipeline standardizes tasks and artifacts while experts review and iteratively refine each stage.The pipeline comprises task construction, task-instance construction, and evaluation-criteria construction.Source: A DATASET CONSTRUCTION · p. 15, Figure A.1 · p. 15

A.1 TASK CONSTRUCTION

  • Experts select software environments and representative scenarios from real workflows, then define result-oriented specifications covering objectives, inputs, deliverables, and constraints.AI assists with consistent wording and structure, and experts review and finalize each specification.Source: §A.1 · p. 15

A.2 TASK INSTANCE CONSTRUCTION

  • Experts prepare the initial software state and a validated reference realization, with multiple valid realizations allowed when they satisfy the task requirements.For multi-software tasks, intermediate artifacts preserve dependencies between workflow stages; open-choice tasks fix requirements while allowing tool and execution-order flexibility.Source: §A.2 · pp. 15–16

A.3 EVALUATION CRITERIA AND VERIFICATION

  • Task-specific criteria define successful artifacts using engineering properties and tolerances, which AI-assisted verifiers extract and check from final and, when needed, intermediate submissions.Reference realizations and controlled variants test verifier acceptance and failure boundaries; experts review the complete task package and revise and revalidate identified inconsistencies.Source: §A.3 · p. 16

B ENGINEERING EVALUATION

EngiWorld evaluates agents in isolated engineering-software environments and scores deliverables through artifact checks, feasibility constraints, and verifier validation. In the main evaluation, 128 of 300 tasks receive zero from all seven models, including many single-software tasks.

B.1 SOFTWARE ENVIRONMENTS

  • Each task runs in an isolated Windows or Ubuntu VM; 37 environments comprise 26 software-specific, 10 composite, and one Open-ended environment.Cross-application tasks share a workspace, and the harness restores a clean snapshot and prepares task inputs before each episode.Source: §B.1 · p. 16, Table B.1 · p. 17

B.2 ARTIFACT CHECKS AND SCORING

  • Verifiers inspect submitted artifacts with application data models or format-specific parsers, applying task-specific numerical tolerances and exact checks for properties such as connectivity and required references.Multi-software checks track consistency across intermediate and final artifacts, and a missing upstream artifact fails dependent criteria; infeasible quantitative outputs score zero.Source: §B.2 · pp. 16–18

B.3 VERIFIER VALIDATION

  • All 96 reference artifacts in the completed audit group pass after review and repair; alternative-valid submissions are accepted and empty or metrics-only submissions are rejected.Tolerance-boundary tests are outside the completed audit populations.Source: §B.3 · p. 18, Table B.3 · p. 19

C.1 TASKS MISSED BY ALL MODELS

  • 128 of 300 tasks receive zero from all seven models; the all-zero set includes many single-software tasks, so the gap is not limited to cross-application workflows.The set also includes 22 Multi-software, 18 Software-selection, and 12 Quantitative design tasks.Source: §C.1 · pp. 18–19

C.2 QUANTITATIVE DESIGN RESULTS

Quantitative design scores distinguish quality among feasible outputs while validity checks gate credit. Partial scores are uncommon, and score values can reflect either improvement over a baseline or direct design quality.

C.2 QUANTITATIVE DESIGN RESULTS

  • 53 of 231 quantitative-design runs receive partial scores, nine receive 1.0, and 169 receive zero; KiCad layout and ZBrush surface tasks account for 38 partial scores.These runs cover 33 tasks and seven models; validity checks determine whether an artifact can receive credit.Source: §C.2 · p. 19
  • GPT-5.6 Sol scores 0.892 on ZBrush task 04 versus Gemini 3.7 Flash’s 0.804, with lower height-field RMSE: 0.07650 versus 0.15098 OBJ units.Both outputs are valid and have full evaluation-grid coverage; the initial design scores 0.880, so only GPT improves on it.Source: §C.2 · p. 19
  • Gemini scores 0.982 on KiCad task 01 versus the baseline’s 0.647; weighted cost falls from 64.2085 to 3.2135, while route length falls from 53.1618 to 40.1688 mm.The remaining cost comes from route length, so the score gap to 1.0 does not represent unmet requirements as a fraction.Source: §C.2 · p. 19
  • A ZBrush task-04 output with RMSE 0.18097 OBJ units scores zero because its mesh is invalid; GPT’s task-05 output also scores zero with 94.75% coverage against a 98% requirement.RMSE cannot compensate for invalid topology or insufficient coverage.Source: §C.2 · p. 19

C.3 OPEN-ENDED VERSUS MULTI-SOFTWARE TASKS

Matched tasks share engineering targets but differ in workflow and delivery requirements. On task 09, several models succeed when free to choose their tools, while all fail the prescribed Multi-software version.

C.3 OPEN-ENDED VERSUS MULTI-SOFTWARE TASKS

  • Both task pairs share an engineering target, but Multi-software prescribes KiCad, OpenSCAD, FreeCAD, and Blender plus 18 artifacts, whereas Open-ended permits any tools and requires one final STEP assembly.The Multi-software sequence includes intermediate handoffs, reports, a rendered review, and a toolchain invocation log.Source: §C.3 · pp. 19–20, Figure C.2 · p. 21
  • On task 09, all seven models fail the Multi-software version, while Claude Opus 5, Gemini 3.7 Flash, Qwen 3.8 Max, and DeepSeek V4.1 Flash succeed on Open-ended; task 02 remains unsolved by all seven.The paired outcomes are binary: 1 indicates success.Source: §C.3 · p. 21
  • On task 09, Gemini’s 478-byte invocation log misses the 500-byte minimum, while Qwen reaches its 150-step limit with the required file missing; both fail delivery checks before geometry is assessed.By contrast, Gemini and Qwen construct the Open-ended assembly through FreeCAD scripting and submit STEP files that pass the final geometry evaluator; they finish in 41 and 30 steps, respectively.Source: §C.3 · p. 21
  • The paired outcomes separate valid assembly construction from required handoffs and documentation; GPT-5.6 Sol also leaves 3.698 mm3 of material in the task-09 MH1 bore, above the 0.1 mm3 tolerance.That geometric error is a separate failure mode from delivery requirements.Source: §C.3 · p. 21

C.4 ADDITIONAL ABLATION ANALYSIS

Ablations show that effects vary by model and task: image placement, screenshot resolution, and history length can change both success and score. Four illustrated GUI workflows pass their artifact checks with full scores.

C.4 ADDITIONAL ABLATION ANALYSIS

  • Reference-drawing placement improves GUI EngiScore for all three models, but Kimi’s CLI score falls from 40.0 to 36.7; within FreeCAD, task 14 succeeds while task 20 becomes unsuccessful.For GPT, the GUI change turns 15 failures into successes and five successes into failures across 60 tasks.Source: §C.4 · pp. 21–23
  • Native-resolution screenshots produce the highest EngiScore for all three models; GPT rises from 28.98 at 0.6× to 29.84 at 1.0× despite ten tasks losing a combined 9.140 points.Kimi gains eleven nonzero outcomes and loses none; GPT’s new KiCad success requires repairing closely spaced QFN-48 features, while a Blender regression involves 100 disconnected brick islands.Source: §C.4 · p. 23
  • Increasing history from ten to fifteen turns lowers Kimi’s EngiScore from 26.21 to 20.12, while GPT changes from 36.61 to 36.74; Kimi’s decline is dominated by tasks losing all credit.GPT’s near-constant mean combines seven tasks gaining nonzero scores and seven formerly full-score tasks losing points; its improved ZBrush task-03 score remains below the initial design’s 0.909292.Source: §C.4 · p. 23

D.1 SUCCESSFUL WORKFLOW CASES

  • Four GUI workflows in KiCad, AutoCAD, SolidWorks, and Blender retain required artifact properties, pass task-specific checks, and score 1.0.KiCad verifies new data connections and preserved circuitry; AutoCAD uses offset arrays to create 23 staggered holes; SolidWorks changes relative part placement while preserving geometry; Blender packs the baked ambient-occlusion image into the native file.Source: §D.1 · p. 23, Figure D.1 · p. 24

D.2 FAILURE CASES

Failure analysis shows that decision-turn exhaustion is more common among GUI than CLI failures, while artifact checks expose zero-credit outcomes after apparent completion. The cases include failures in load representation, file schemas, import scale, and action targeting.

Failure taxonomy

  • Across experiments, 3,347 distinct zero-score runs were classified into five terminal-outcome categories; reused runs counted once, while required-field and missing-file failures were excluded.The categories are DONE with zero score, decision-turn exhaustion, explicit FAIL, environment timeout, and agent-initiated desktop closure or restart.Source: §D.2 · p. 24, Table D.1 · p. 25

Failure by interaction interface

  • Decision-turn exhaustion accounts for 233 of 636 CLI failures (36.6%) and 471 of 851 GUI failures (55.3%).The breakdown covers analyzed failures; completion without acceptance is more common on CLI, while explicit failure declarations and runtime-limit failures are smaller components in both settings.Source: §D.2 · p. 24, Figure D.2 · p. 25

Artifact checks in declared successes

  • Independent artifact checks identify geometric and cross-stage violations in four runs that end with DONE, and all four receive zero scores.The checks use delivered artifacts even when agent-written reports claim the requirements are met; Figure D.3 compares recorded actions, submitted data, and observed outcomes with reference artifacts.Source: §D.2 · p. 24, Table D.3 · p. 25, Figure D.3 · p. 26

Load representation mismatch

  • In the SolidWorks–Abaqus case, a completed analysis uses −11.111 MPa pressure, equivalent to approximately 1 kN tension on the specified 30×3 mm face, but fails the load check.The evaluator checks force or displacement parameters without converting pressure to resultant force; the task specifies a 1 kN tensile load.Source: §D.2 · pp. 25–26

Import scale errors

  • An AutoCAD–SolidWorks–Abaqus import-scale error recurs after a crash, leaving an elongated model unlike the 80×60×21 mm, ten-fin reference and no solved thermal ODB.Repeated volume-meshing attempts continue until the 300-turn limit.Source: §D.2 · p. 26

Incorrect action targeting

  • A mis-targeted Ctrl+A/Delete clears the drawing; the saved DXF retains diagonal GUIDE lines, lacks eight required windows, and the workflow ends with FAIL.Subsequent attempts to launch FreeCAD do not advance the workflow.Source: §D.2 · p. 26

CSV schema mismatch

  • In the KiCad–OpenSCAD–FreeCAD–Blender CLI case, all 18 files are submitted, but an unaccepted CSV field name makes the evaluator read the MH1 diameter as None.The submitted field is hole_dia_mm, while the ground truth uses hole_diameter_mm; evaluation stops before downstream geometry checks.Source: §D.2 · p. 26
Loading 2609.37686v1…