PaperlayerView exact revision on arXiv

Source-linked AI summary

Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization

Yizhe Chi, Deyao Hong, Dapeng Jiang, Tianwei Luo, Kaisen Yang, Boshi Zhang, Zhe Cao, Xiaoyan Fan, Bingxiang He, Han Hao, Weiyang Jin, Dianqiao Lei, Qingle Liu, Houde Qian, Bowen Wang, Situ Wang, Youjie Zheng, Yifan Zhou, Calvin Xiao, Eren Cai, Qinhuai Na

arXiv:2604.12290v2cs.AIcs.CL

TL;DR

Many agent evaluations emphasize one-shot or binary outcomes, while engineering optimization seeks better feasible designs under constraints. Frontier-Eng benchmarks budget-aware generative optimization across 47 engineering tasks. Current agents can optimize simple controllers and algorithmic kernels, but struggle with complex physical systems.

  • Problem

    Many agent evaluations emphasize one-shot or binary completion, while engineering asks agents to improve feasible designs under constraints and finite budgets.

    Source: §1 · pp. 1–3

  • Method

    Frontier-Eng formalizes generative optimization and benchmarks 47 engineering tasks across five categories using unified interfaces, executable verifiers, and representative search methods.

    Source: §1 · p. 3

  • Results

    Current agents optimize simple controllers and algorithmic kernels effectively, but struggle with multi-objective trade-offs and long-horizon reasoning in complex physical systems.

    Source: §5 · p. 17

  • Takeaways & Limitations

    The paper suggests progress toward engineering artificial general intelligence depends not only on larger models but also on search strategies integrating domain knowledge with structured executable feedback.

    Source: §5 · p. 17

    The general formulation assumes agents can submit candidates and receive grounded feedback; stochastic evaluators approximate expected scores through repeated runs with fixed seeds.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Current LLM agent benchmarks, which predominantly focus on binary pass/fail tasks such as code generation or search-based question answering, often neglect the value of real-world engineering that is often captured through the iterative optimization of feasible designs. To this end, we introduce Frontier-Eng, a human-verified benchmark for generative optimization -- an iterative propose-execute-evaluate loop in which an agent generates candidate artifacts, receives executable verifier feedback, and revises them under a fixed interaction budget -- spanning $47$ tasks across five broad engineering categories. Unlike previous suites, Frontier-Eng tasks are grounded in industrial-grade simulators and verifiers that provide continuous reward signals and enforce hard feasibility constraints under constrained budgets. We evaluate eight frontier language models using representative search frameworks, finding that while GPT 5.4 achieves the most robust performance, the benchmark remains challenging for all models. Our analysis suggests a dual power-law decay in improvement frequency ($\sim$ 1/iteration) and magnitude ($\sim$ 1/improvement count). We further show that although width improves parallelism and diversity, depth remains crucial for hard-won improvements under a fixed budget. Frontier-Eng establishes a new standard for assessing the capacity of AI agents to integrate domain knowledge with executable feedback to solve complex, open-ended engineering problems.

1 Introduction

Engineering often derives value from iterative improvement of feasible designs, whereas prevailing search and coding benchmarks emphasize binary outcomes. Frontier-Eng addresses this gap with a benchmark for budget-aware generative optimization across real-world engineering tasks.

1 Introduction

  • Engineering's iterative cycle of requirements, prototyping, constraint testing, and refinement makes optimization—not initial solution creation—the main locus of practical value.The paper illustrates this with incremental improvements in charging time, material use, and factory makespan.Source: §1 · p. 1
  • Unlike binary pass/fail tasks, engineering optimization begins from feasible artifacts and asks how good an improvement an agent can find within a finite budget.Many engineering problems have no known theoretical optimum, or an optimum that is unreachable in practice.Source: §1 · pp. 1–2
  • Recent work motivates generative optimization: models iteratively propose solutions and improve them on a fixed problem under finite interaction budgets.The paper describes this as self-evolution on a fixed problem instance.Source: §1 · p. 2
  • Existing efforts target narrow domains or synthetic settings, leaving no comprehensive benchmark of retrieval, constrained synthesis, refinement, and realistic engineering feedback.The paper identifies this missing benchmark as a gap across real engineering disciplines.Source: §1 · p. 3
  • Frontier-Eng packages 47 tasks across five categories with feasible artifacts, executable verifiers, and baselines, while formalizing budget-aware improvement and evaluating representative search methods across frontier models.Its contributions include a unified interface supporting cross-task comparison and an evaluation of evolutionary search, sample-efficient evolution, and tree-based exploration.Source: §1 · p. 3

2 Introducing the Frontier-Eng Benchmark

This section introduces Frontier-Eng as a diverse 47-task benchmark for iterative engineering optimization with executable feasibility and score feedback, safeguards, and a task-independent evaluation protocol. It also compares nine models across the tasks under OpenEvolve.

2.1 Task formulation and generative optimization

  • Agents iteratively revise a feasible starting artifact under a fixed budget, using an immutable evaluator’s feasibility and score feedback to seek the best feasible result.Tasks pair readable context with an editable artifact and evaluator; the solution space and evaluator can vary widely, while LLM proposals are separated from the search strategy that selects prior attempts for the prompt.Source: §2.1 · p. 4

2.2 Benchmark construction and integrity

  • Frontier-Eng’s 47 tasks come from five sources, and each admitted task has a self-contained specification, editable artifact, runnable evaluator, and verified-feasible start.Sources include competitions, academic suites, coursework, industrial-grade simulators, and domain-expert contributions; automated review and human maintainer review check code, execution, engineering soundness, evaluator correctness, and constraints.Source: §2.2 · pp. 4–5

2.3 Benchmark composition and coverage

  • The benchmark spans computing and quantum information, operations research, robotics and energy, optics and communications, and physical sciences and engineering design.Its 47 tasks cover diverse executable artifacts and engineering objectives, including code and circuits, combinatorial decisions, dynamic-system policies, physics-based optical metrics, and simulation-grounded design.Source: §2.3 · pp. 5, 7
  • Tasks also vary in artifact languages, verifier types, optimization structures, and compute requirements, preventing any single search or prompting approach from being universally advantageous.The benchmark includes Python, C, C++, and CUDA/Triton artifacts; simulators, solvers, emulators, and reference checks; and CPU-only, GPU-mandatory, and Docker-isolated evaluations.Source: §2.3 · p. 7

2.4 Evaluation protocol

  • To compare incompatible task scores, Frontier-Eng uses average rank as its primary metric, performance profiles to recover relative magnitude, and win rate to measure improvement frequency.It also reports category-level metrics; the protocol applies independently of the search algorithm, keeping the benchmark stable as methods evolve.Source: §2.4 · pp. 7–9
  • Performance profiles translate scores per task before forming ratios, supporting mixed-sign scores while preserving within-task ordering.The translation means profile ratios are relative to a shifted scale rather than the original score scale.Source: §2.4 · pp. 8–9

2.5 Case study: SUSTAINABLEDATACENTERCONTROL

  • The SustainableDataCenterControl case evaluates a policy coordinating workload shifting, cooling, and battery dispatch against a feasible no-op baseline across four fixed scenarios.The verifier runs 192 hourly steps per scenario and checks action bounds and runtime execution; its score combines carbon and water reductions while penalizing overdue work.Source: §2.5 · pp. 9–10
  • Across 768 hours, strong performance requires balancing temporal energy decisions, carbon savings against overdue work, and robustness across seasonal scenarios.The no-op baseline is marginally beatable, but high scores require a coordinated strategy that maintains service-level guarantees across all scenarios.Source: §2.5 · p. 10
  • 24/47 tasks (51.1%) were best for GPT-5.4, which was within the 1.1 near-best band on 33/47 tasks (70.2%).The diagnostics use retest results in place of GPT-5.4’s original results; the figure also reports win rate over baseline as the fraction of tasks where a model’s best score exceeds its initial score.Source: Figure 4 · p. 11

3 Experiments

Experiments compare models and search frameworks across 47 tasks, then examine improvement trajectories and the allocation of a fixed search budget. GPT-5.4 has the best average rank in the updated nine-model comparison, while deeper chains outperform wider parallel searches under equal budgets.

3.1.1 Setup and Metrics

  • Models share the same search framework, initial programs, and frozen task verifiers; average rank is primary, with performance profile and win rate as diagnostics.The comparison uses a fixed 100-iteration OpenEvolve budget; gpt-5.4 is represented by its full 47-task retest.Source: §3.1.1 · p. 11

3.1.2 Results and Analysis

  • 3.54 average rank places gpt-5.4 first across 47 tasks, narrowly ahead of claude-opus-4.6 at 3.63 after the retest.The retest shifts gpt-5.4 from its original average rank of 5.68 to the top position.Source: §3.1.2 · p. 11, Table 2 · p. 12

3.2.1 Aggregate Comparison

  • Across all three frameworks, claude-opus-4.6 has lower average rank and more task wins than gpt-oss-120b.The largest rank gap is under OpenEvolve, with ShinkaEvolve close behind.Source: §3.2.1 · p. 12, Table 3 · p. 13

3.2.2 Trajectory Tendencies Across Frameworks

  • Under ABMCTS, gains concentrate early; OpenEvolve permits more later updates, while ShinkaEvolve shows later improvements and shorter plateaus.For ShinkaEvolve, gpt-oss-120b has last-improve 0.64, plateau 0.45, and 5 median best-updates; claude-opus-4.6 has median gain 72.9%.Source: §3.2.2 · p. 13, Table 4 · p. 13

3.2.3 Model Behavioral Differences

  • claude-opus-4.6 ranks better across frameworks, while gpt-oss-120b records more best-updates under OpenEvolve and ShinkaEvolve, with shorter plateaus.Under ShinkaEvolve, claude-opus-4.6 reaches its final improvement earlier; gpt-oss-120b tends to improve later, narrowing the gap on refinement-heavy tasks.Source: §3.2.3 · p. 14

3.2.4 Summary

  • ABMCTS is most conservative, OpenEvolve most structurally aggressive, and ShinkaEvolve most conducive to sustained refinement.claude-opus-4.6’s advantage is most pronounced when early accepted candidates strongly influence final outcomes; gpt-oss-120b narrows the gap when late-stage gains remain possible.Source: §3.2.4 · p. 14

3.3.1 Improvement Dynamics Follow a Power Law

  • Across 47 tasks and 500 iterations, improvement frequency decays as t^-1 and improvement magnitude as k^-1, with both fits having R^2 > 0.83.Every task improves at least once, with a median of 7 improvements per task; marginal returns approach zero after about 50–100 iterations.Source: §3.3.1 · p. 14, Figure 5 · p. 15

3.3.2 Depth Dominates Width under Fixed Budget

  • On the equal-budget diagonal, normalized best-of-n scores fall from 1.00 to 0.91 as parallel chains increase from 1 to 16, favoring depth over width.Width improves scores when depth is held fixed, but under a fixed total budget a single deep chain dominates.Source: §3.3.2 · p. 14

4 Related work

Prior agent benchmarks establish long-horizon tool use and engineering evaluations test technical knowledge or executable design, but Frontier-Eng combines cross-domain engineering tasks with constrained optimization and budget-aware best-found performance.

Verifier-guided search and evaluation

  • Frontier-Eng combines cross-domain engineering tasks, executable constraints, and limited interaction budgets, evaluating best-found feasible performance rather than only validity or task completion.Table 5 compares adjacent benchmarks on dimensions including hard constraints and best-found-under-budget evaluation.Source: §4 · pp. 15–16, Table 5 · p. 17

Agent benchmarks for open-world problem solving

  • Open-world agent benchmarks evaluate planning, tool use, and iterative correction, but typically measure digital task completion rather than constrained engineering-design improvement.Frontier-Eng adopts this long-horizon, tool-using perspective while grounding evaluation in executable engineering tasks.Source: §4 · p. 15

Engineering-grounded benchmarks and systems

  • Engineering-knowledge benchmarks remain largely answer-centric, while executable engineering benchmarks and systems couple design tasks with simulation or engineering tools.The latter include EngDesign, EngiBench, BikeBench, BuildArena, ControlAgent, AnalogCoder, and SPICED.Source: §4 · pp. 15–16

Verifier-guided search and evaluation

  • Verifier-guided methods use feedback to iteratively improve candidate solutions, from tool-using search frameworks to executable program discovery and refinement.The cited examples include ReAct, Reflexion, Tree of Thoughts, LATS, OPRO, FunSearch, AlphaEvolve, and Self-Refine.Source: §4 · p. 16
  • Best-found-under-budget evaluation refers to assessing the best feasible solution found within a fixed interaction budget.This shifts the question from producing one correct response to searching effectively across engineering domains.Source: §4 · p. 17

5 Conclusion

Frontier-Eng evaluates generative optimization across 47 engineering tasks using unified interfaces, industrial-grade simulators and solvers, and anti-hack safeguards. The evaluation finds emerging iterative-improvement capacity alongside substantial difficulty on complex physical systems.

5 Conclusion

  • Frontier-Eng packages 47 tasks across five engineering categories with unified metadata-driven interfaces, simulator and solver feedback, and strict anti-hack safeguards.The benchmark formalizes optimization as a context, initial solution, and executable evaluator, emphasizing budget-aware search for high-quality feasible solutions.Source: §5 · p. 17
  • Agents can optimize simple controllers and algorithmic kernels, but often struggle with multi-objective trade-offs and long-horizon reasoning in complex physical systems.The conclusion points to search strategies that better integrate domain knowledge with structured executable feedback.Source: §5 · p. 17
  • The benchmark is intended to support cross-disciplinary comparison and research on agents that participate in engineering’s define-build-test-refine cycle.The authors hope future expansions will add more diverse domains and higher-fidelity simulations.Source: §5 · p. 17

A Detailed Task Catalog

The catalog organizes Frontier-Eng’s 47 tasks into five engineering categories and specifies each task’s objective, scoring mechanism, and evaluation-integrity safeguards.

A Detailed Task Catalog

  • The catalog describes each of the 47 tasks by its objective, scoring mechanism, and safeguards for evaluation integrity.Tasks are organized into five engineering categories.Source: A Detailed Task Catalog · p. 22

A.1 Computing & Quantum Information (10 tasks)

The catalog spans computing, quantum information, and operations research tasks, with objectives scored through performance or solution quality and evaluators that apply correctness, feasibility, or anti-manipulation checks.

A.1 Computing & Quantum Information (10 tasks)

MLA (Multi-Head Latent Attention)

  • MLA attention optimization scores geometric-mean kernel latency subject to correctness verification against a reference implementation.Evaluation runs in a sandboxed environment with independent correctness checking.Source: MLA (Multi-Head Latent Attention) · p. 22

TriMul (Triangular Multiplicative Update)

  • TriMul, allocator, and hashing tasks use masked GPU operations or C implementations, with isolated correctness checks, randomized inputs, or throughput and utilization scoring.The SHA3-256 evaluator parses pass/fail counts to detect partial-correctness edge cases.Source: TriMul (Triangular Multiplicative… · p. 22

Routing QFTEntangled

  • QFT-entangled circuit routing minimizes normalized gate-count and depth costs on IBM Falcon topology, using evaluator-owned transpilation and reference circuits.Source: Routing QFTEntangled · p. 23

Clifford+T Synthesis

  • Clifford+T synthesis minimizes T-gate count, two-qubit gates, and circuit depth against MQT Bench optimization anchors.The verifier code is read-only.Source: Clifford+T Synthesis · p. 23

Cross-Target QAOA

  • Cross-target QAOA optimization scores mean normalized cost across IBM Falcon and IonQ Aria, discouraging overfitting to one device topology.Source: Cross-Target QAOA · p. 23

A.2 Operations Research & Decision Science (9 tasks)

  • Operations-research tasks cover supply-network inventory, portfolio rebalancing, ordering policies, and job-shop scheduling, using objective, cost, service, robustness, or makespan scores.Evaluators use independent models, simulations, reference solutions, or schedule-constraint validation.Source: §A.2 · pp. 23–24
  • Job-shop scheduling tasks score makespan against known optima or across large instance families, with schedule validation and restrictions on external solvers.The listed families include ABZ, SWV, and Taillard instances.Source: §A.2 · p. 24

A.3 Robotics, Control & Energy Systems (8 tasks)

The task set spans robotics, control, and energy problems as well as optics and communications, with objectives evaluated through task-specific scoring and verifier constraints.

A.3 Robotics, Control & Energy Systems (8 tasks)

  • Robotics tasks cover obstacle navigation, quadrotor control, quadruped gait, arm cycle time, and UAV inspection, with feasibility checked through physics-based evaluation.Objectives include safe, fast motion, stable gait, and inspection coverage under wind; constraints include collision, actuator, joint, and acceleration limits.Source: §A.3 · p. 24
  • Charging tasks balance time, aging, plating, and thermal performance, while a data-center policy jointly manages load shifting, cooling, and battery control across four scenarios.The charging evaluations use frozen or read-only physics models; the data-center score includes penalties for dropped or overdue tasks.Source: §A.3 · p. 25

A.4 Optics & Communication Systems (10 tasks)

  • Adaptive-optics tasks evaluate corrupted-slope fusion and sequential control, including trade-offs between wavefront correction quality and command smoothness.Evaluation spans 320 turbulence scenarios for fusion; sequential control uses delayed or noisy slopes with a hidden deformable-mirror model.Source: §A.4 · p. 25
  • Optics and communications tasks include assigning users to WDM channels and setting transmit power, with satisfaction, BER, spectral efficiency, or SNR-based scoring.The evaluators independently recompute communication metrics and enforce assignment, power, and MCS constraints.Source: §A.4 · pp. 25–26
  • Holography tasks optimize phase layers or patterns for target focal spots and energy distributions, scored using efficiency, ratio matching, shape fidelity, or dark-zone suppression.Scores are derived from simulated wave propagation or an independent forward model.Source: §A.4 · pp. 25–26
  • Additional optics tasks optimize a variance-controlled BER estimator and binary phase transitions for uniform Dammann-grating orders.The estimator trades accuracy against runtime, while the grating score combines uniformity, efficiency, and balance across orders −3 to +3.Source: §A.4 · pp. 25–26

A.5 Physical Sciences & Engineering Design (10 tasks)

The section covers structural design, chemical optimization, trajectory planning, and data-driven engineering tasks; it also introduces trajectory-level case studies to complement aggregate statistics.

A.5 Physical Sciences & Engineering Design (10 tasks)

  • Structural-design tasks minimize truss or tower weight and beam compliance under engineering constraints, with independent or built-in finite-element evaluation.The MBB beam task enforces a volume fraction of ≤0.50, and the tower task uses discrete section selection.Source: §A.5 · p. 26
  • Chemical-design tasks optimize reaction yield or Pareto trade-offs in yield, environmental factor, and turnover number using benchmark emulators.The SnAr task uses 24 experiments per seed; the Suzuki task scores 2D hypervolume normalized by yield and turnover.Source: §A.5 · pp. 26–27
  • Trajectory planning maximizes delivered lunar payload subject to time, fuel-bookkeeping, and altitude constraints, with numerical integration and physics-based validation.Source: §A.5 · p. 27
  • Data-driven tasks assess sensor placement for pressure-field reconstruction and prediction of ADT modality from RNA measurements against held-out data.The sensor task uses a frozen Transolver model on held-out CFD cases; the modality task scores correlation and RMSE.Source: §A.5 · p. 27
  • One task combines seven heterogeneous sub-problems, including drivers, denoising, CPU logic, robotics, and topology, by averaging their independent sub-scores.The evaluator re-solves or re-simulates selected components and verifies ISA behavior point by point.Source: §A.5 · p. 27

B Case Study

  • The case study examines four representative tasks to show how model capability and search mechanisms interact across different task structures.Its running-best staircase plots show all evaluated candidates in gray and the best score so far in green.Source: B Case Study · p. 27

B.1 ComputerSystems_MallocLab

MallocLab requires a structural shift from a naive allocator to production-grade memory management. Claude reaches the same high score through different search paths, while GPT-OSS benefits from incremental refinement.

B.1 ComputerSystems_MallocLab

  • 97 for Claude under both openevolve and shinkaevolve; GPT-OSS scores 53 after large rewrites but reaches 86 with shinkaevolve.The running-best staircase plots show the MallocLab trajectories.Source: Figure 7 · p. 28, §B.1 · p. 28
  • Claude makes a one-shot shift to a segregated-list design, whereas GPT-OSS reaches that structural transition through atomic refinements from headers to coalescing and segregated classes.Source: §B.1 · p. 28
  • The case study characterizes search granularity as a reasoning stabilizer for mid-tier models facing architectural thresholds.The authors contrast incremental scaffolding with frontier models’ ability to make direct architectural jumps.Source: §B.1 · p. 28

B.2 EnergyStorage_BatteryFastChargingProfile

BatteryFastChargingProfile is a boundary-sensitive control problem: agents must exploit voltage and temperature headroom without violating hard safety limits. Claude with openevolve finds a higher-scoring profile than incremental shinkaevolve runs.

B.2 EnergyStorage_BatteryFastChargingProfile

  • 120.80 for Claude with openevolve, compared with 87.58 for incremental shinkaevolve runs.The running-best staircase plot presents the task trajectories.Source: §B.2 · p. 28, Figure 8 · p. 29
  • Claude with openevolve switches from a conservative four-stage baseline to an aggressive six-stage profile, citing low-SOC voltage headroom; shinkaevolve adjusts switch points without raising current levels.Source: §B.2 · p. 29
  • The authors interpret the result as occupying the near-optimal safety boundary, which incremental methods did not replicate in this task.Source: §B.2 · p. 29

B.3 InventoryOptimization_general_meio

The case studies link engineering progress to task-specific search dynamics: substantial rewrites or policy shifts can outperform incremental refinement, while model and framework strengths vary with task structure. Across the four cases, Claude is stronger than GPT-OSS, and no framework is uniformly best.

B.3 InventoryOptimization_general_meio

  • Inventory optimization requires a unified network-level policy because local heuristics fail under coupled supply-chain dynamics.The analysis frames complex-system engineering as policy discovery: replacing local heuristics with a solver-aligned structure may be a regime shift that incremental methods fail to trigger in time.Source: §B.3 · pp. 29–30
  • 0.9929 for Claude + openevolve follows an early logic replacement, while Claude + shinkaevolve reaches 0.9694 through refinement.The strongest trajectory uses a custom two-phase procedure with explicit local holding costs and node-specific stockout penalties.Source: §B.3 · p. 29

B.4 Cryptographic_SHA3-256

  • SHA3-256 fixes the algorithm, making C++ implementation density and efficiency the optimization variables.The task favors implementation-level rewrites, although the reported results show a competitive band rather than a massive separation.Source: §B.4 · p. 30
  • 28.90 for Claude + openevolve exceeds 24.42 for GPT-OSS + abmcts and 22.12 for Claude + shinkaevolve on SHA3-256.The leading rewrite moves to mmap-based file handling and scalarized 64-bit Keccak rounds; multiple combinations improve on baseline, but only the strongest rewrites reach the top tier.Source: §B.4 · p. 30

B.5 Cross-Case Synthesis

  • Across four tasks, Claude is systematically stronger than GPT-OSS, while model capability, framework search dynamics, and task structure jointly shape performance.Task structure ranges from structural thresholds and boundary-sensitive control to policy search and implementation-level engineering.Source: §B.5 · pp. 30–31

C Raw Data for Model Comparison

Table 7 presents raw per-task scores for the model comparison and points to the rank results in Table 2.

C Raw Data for Model Comparison

Loading 2604.12290v2…