PaperlayerView exact revision on arXiv

Source-linked AI summary

PLE: Physical-Engineering's Last Exam

Rujun Han, Yufan Zhuang, Arijit Ray, Han Yu, Zoey CuiZhu, George Lee, Yifei Ming, Binz Roy, Varun Sakalkar, Vishy Tirumalashetty, Burak Gokturk, Tomas Pfister, Chen-Yu Lee

arXiv:2610.30537v1cs.AI

TL;DR

Existing benchmarks do not comprehensively test whether agents can deliver realistic, economically valuable professional physical-engineering work. PLE introduces 85 tasks across four engineering disciplines, curated through expert authorship and engineer-agent co-authoring. Across 14 models and seven harnesses, only one configuration exceeds 50%, highlighting substantial remaining headroom.

  • Problem

    Existing engineering benchmarks mainly cover textbook questions, isolated tools, or toy designs rather than complete professional work across realistic project corpora.

    Source: §1 · p. 2

  • Method

    PLE contains 85 tasks across four physical-engineering disciplines, curated through expert-authored and engineer-briefed, agent-built tasks validated by experts.

    Source: §1 · p. 2, §6 · p. 14

  • Results

    Only one configuration exceeds 50% across 14 frontier models and seven harnesses: Claude Opus 5.5 in Claude Code, at 50.6%.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The main obstacle remains producing designs that satisfy every hard constraint while optimizing cost, leaving substantial headroom for future agents.

    Source: §6 · p. 14

    Rubric-only scoring cannot distinguish a better design from a merely feasible one when both meet the same criteria.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

Frontier language models now score above 90% on popular knowledge benchmarks and are increasingly deployed as agents that carry out multi-hour professional work. Yet no existing benchmark tests whether they can perform the real-world, economically valuable work of a professional physical engineer, such as modelling physics and legal code requirements across a multi-format project corpus, optimising material and installation costs, and issuing 2D/3D engineering design packages that a licensed reviewer would sign off on. Existing engineering benchmarks focus on textbook question answering, isolated tool calls or self-contained toy design problems. As a result, most of these benchmarks are already saturated by leading agents and solving them is far away from delivering real engineering design. We introduce PLE, a benchmark of 85 long-horizon, agentic tasks in professional physical engineering consisting of electrical, civil, structural and mechanical engineering, where solving a task means completing a realistic, billable engineering deliverable. PLE combines two complementary curation paths. Expert-Authored tasks are created end to end by practising engineers from a professional data provider's expert pool. Co-Authored tasks start from problem statements written by practising engineers; an adversarial Creator-Solver agent pipeline constructs each task's realistic project environment, executable grader and reference deliverables before sending them to expert engineers for validation and difficulty intensification. We evaluate a variety of frontier LLMs under seven publicly available agent harnesses. Even the strongest configuration reaches a mean score of only 50.6%, while spending 27M tokens and 63 minutes per task on average, demonstrating the substantial gap between current agents and autonomous physical engineering design.

1. Introduction

PLE targets professional physical-engineering work that existing benchmarks do not comprehensively measure, using realistic tasks and deliverables across four engineering domains. The paper introduces two task-curation routes and reports that the strongest evaluated configuration scores only 50.6%.

1. Introduction

  • Existing benchmarks mainly test textbook questions, toy designs, or isolated tools, rather than complete billable work across realistic, messy, multi-format project corpora.Rubric checks alone may also fail to distinguish compliant designs with different installation costs.Source: §1 · p. 2
  • PLE comprises 85 professional tasks across electrical, civil, structural, and mechanical engineering, with realistic work products checked against governing documents and physics.Tasks span work such as roof-deck design, site design, low-voltage cable sizing, and cooling-tower life-cycle cost analysis.Source: §1 · p. 2
  • PLE tasks place agents in realistic project file systems and require cited, production-format deliverables using document, spreadsheet, and code-execution tools.Deliverables include engineering memos, calculation workbooks, equipment schedules, and 2D/3D drawings.Source: §1 · p. 3
  • Practising engineers create Expert-Authored tasks end to end, while Co-Authored tasks begin with engineer-written briefs and are built by an adversarial agent pipeline before expert validation and difficulty intensification.Co-Authored tasks combine rubric-style checks with an economic or performance objective.Source: §1 · p. 3
  • The paper evaluates 14 frontier models across seven public agent harnesses and presents the benchmark as a basis for an open competition and leaderboard.The listed contributions also include analysis of discipline difficulty, run-to-run variance, and performance-cost trade-offs.Source: §1 · pp. 3–4
  • 50.6% is the mean score of the strongest configuration, Claude Opus 5.5 in Claude Code, on PLE.The figure caption reports a cost of $17.18 per task for this configuration.Source: §1 · p. 3, Figure 2 · p. 3

2. Related Work

Related benchmarks cover digital engineering, professional work, or limited slices of physical engineering. PLE adapts workplace-realistic evaluation to physical engineering and adds objective measures of design quality and cost efficiency.

Agentic benchmarks for software and ML engineering

  • Digital engineering benchmarks established sandboxed, tool-calling evaluation, but their artifacts are digital code, models, and test suites rather than physical designs for licensed review.Their success measures include unit-test passes and held-out metrics.Source: §2 · pp. 4–5

Benchmarks of professional work

  • Professional-work benchmarks use authentic deliverables and expert evaluation, and PLE brings that workplace focus to physical engineering with expert rubrics and programmatic design-quality and cost-efficiency measures.The comparison describes GDPval, APEX, APEX-Agents, HealthBench, and PaperBench as related efforts.Source: §2 · p. 5

Professional physical engineering

  • Prior physical-engineering benchmarks cover narrow or abstracted tasks and show high performance on some suites, while PLE tests complete jobs from project documents to deliverables checked against codes, physics, and economic optimality.The passage reports that the best agent beats Frontier-Eng’s baseline on 97.9% of its tasks and leading models exceed 80% at EngiBench Level 2.Source: §2 · p. 5

3. Data Curation

PLE tasks model professional assignments with a project brief, realistic file system, deliverable specification, and grader; two complementary routes create these tasks. Practising engineers author one route end to end, while the other combines engineer-written briefs with agent construction and expert review.

3. Data Curation

  • Each PLE task combines a professional brief, realistic project file system, production deliverable specification, and weighted expert rubric or executable verifier.Tasks run in Archipelago-style environment containers with document, spreadsheet, PDF, mail, calendar, chat, filesystem, and code-execution tools.Source: §3 · p. 5
  • The MEP coordination example requires reconciling plans, schedules, sections, and codes to route four overhead systems around 128 cabinets.The production-ready 3D model must connect every cabinet, avoid clashes, meet clearances, and keep routes economical.Source: §3 · p. 6, Figure 3 · p. 6

3.1. Expert-Authored Tasks

  • Practising engineers create expert-authored tasks as realistic colleague or client requests, then build project environments containing drawings, calculations, schedules, reports, estimates, and correspondence.The 44 tasks average 63 files across six native formats; their weighted rubrics contain 941 binary checks, averaging 21.4 per task.Source: §3.1 · pp. 6–7
  • Expert-authored tasks use weighted binary verifiers, with high weights for load-bearing numbers and low weights for presentation items.Scores are the weight-normalized fraction of checks passed, judged by an LLM reading the deliverable.Source: §3.1 · p. 7

3.2. Co-Authored Tasks

  • Co-authored tasks begin with engineer-written briefs; a Creator builds the task, environment, grader, and reference deliverable, while sandboxed Solver runs calibrate difficulty before expert validation.Creators inspect solver trajectories and may intensify difficulty by changing objectives or adding grader constraints; experts verify realism and consistency and may further intensify tasks.Source: §3.2 · pp. 7, 9, 10
  • Co-authored tasks use real, licensable artifacts and require engineered solutions rather than retrieved answers, with deliverables checked against independent physical, regulatory, interface, and operational constraints.Deep research builds multi-format corpora with distractors and information distributed across documents, requiring cross-referencing.Source: §3.2 · pp. 7–8
  • Rubric-only scoring cannot distinguish better designs that satisfy the same checks, so co-authored tasks combine constraint compliance with an objective score and a hard-constraint gate.The objective compares design quality against a task-calibrated naive baseline; the gate prevents unsafe or infeasible designs from gaining objective credit.Source: §3.2 · pp. 8–9
  • Co-authored environments add CAD/BIM models, schematics, imagery, web portals, and code to document-heavy inputs, and are 12.6× larger on disk per task.Expert-authored environments are document-centric; co-authored tasks issue multi-file deliverable packages rather than mostly one report or workbook.Source: Figure 6 · p. 10, Figure 7 · p. 10

4. Data Analysis

PLE combines two task-curation routes across four engineering disciplines, using large project environments and deliverables that range from single reports to multi-file engineering packages.

4. Data Analysis

  • Across 53 environments, PLE’s 85 tasks involve 5,170 environment files, averaging 61 per task and reaching up to 207; 44 tasks are expert-authored and 41 co-authored.Expert-authored tasks are graded against 941 rubric items (21.4 per task); the 37 constraint-graded co-authored tasks average 29.9 constraints per task.Source: §4 · p. 10
  • Mechanical (30 tasks) and electrical (27) are the largest disciplines, followed by civil (18) and structural (10).All ten structural tasks and the building-systems design work in other disciplines come from the co-authored route.Source: §4 · p. 11
  • Expert-authored environments are document-centric, while co-authored environments include CAD/BIM and other formats and average 38.1 MB per task versus 3.0 MB.Both routes include distractors such as superseded revisions and unrelated sheets; their 85 tasks require 249 deliverable files, with co-authored tasks averaging 4.9 files versus 1.1 for expert-authored tasks.Source: §4 · p. 11

5. Results

PLE evaluates frontier models across agent harnesses on 85 tasks, with scores remaining modest and varying across tasks, models, and harnesses. Analyses also show substantial run-to-run variability on individual tasks and differences in cost and grading outcomes.

5.1. Experimental Setup

  • Evaluation crosses 14 models with seven harnesses, runs each model-task-harness combination three times, and reports mean task scores.The agent sees no rubric or score feedback; expert-authored rubric items are judged by Claude Opus 5.5, while co-authored tasks use frozen executable graders, with some narrative items judged by Claude Opus 5.5.Source: §5.1 · pp. 11–12, Table 3 · p. 12

5.2. Main Results

  • 50.6% is the highest mean score, achieved by Claude Opus 5.5 in Claude Code across the 85 tasks.Claude Opus 5.5’s median task score is 43.5%, and it scores below 50% on 48 tasks; even the best harness per task leaves 35 tasks below 50%.Source: §5.2 · p. 12
  • Claude Opus 5.5 scores 40.5% on mechanical and 46.6% on structural tasks, versus 69.9% on civil tasks; co-authored tasks score 48.0% versus 53.1% for expert-authored tasks in Claude Code.Source: §5.2 · p. 12
  • Scores across each model’s best harness span from 50.6% for Claude Opus 5.5 to below 7% for Qwen3 Coder 480B, MiniMax M2, Kimi K2 Thinking, and Llama 4 Maverick.For a fixed model, the harness spread is 0.7–12.1 points, with a median of 3.0; the model spread is 50 points.Source: §5.2 · p. 13
  • Claude Opus 5.5’s scores by harness range from 47.3% to 50.6%, while harness rankings vary across models.For Claude Opus 5.5, Claude Code is best for 4 models, while other models’ best harnesses include Antigravity, Hermes, Codex, and OpenCode.Source: §5.2 · p. 13

5.3. Analysis

  • 105 of 595 task-harness cells span at least 30 points across three runs, although Claude Opus 5.5 harness means have standard deviations of 0.3–1.8 points.The mean range of a task-harness cell is 15.4 points; the passage recommends averaging per-task results over repeated runs.Source: §5.3 · p. 13
  • Claude Code ($17.18/task, 50.6%) and OpenHands ($14.82/task, 49.9%) lie on the performance-cost frontier for Claude Opus 5.5.OpenCode and Pi score 47.4% and 47.3% at lower cost, while Codex costs $35.15/task and scores 49.3%.Source: §5.3 · pp. 13–14
  • Claude Opus 5.5 in Claude Code scores 0% in 11 of 255 runs, while on co-authored tasks its deliverable items satisfy 91% of constraints on average but only 41.3% satisfy every hard constraint.Claude Code remains first under each of the three judges used in the regrading analysis, despite judge-related score differences on rubric-graded tasks.Source: §5.3 · p. 14

6. Conclusion

PLE is a benchmark of realistic professional physical-engineering work, with tasks curated through expert authorship and engineer-agent co-authorship. Results show that current agents still struggle to produce designs that satisfy all hard constraints while optimizing cost.

6. Conclusion

  • PLE’s 85 agentic tasks assess whether models can perform realistic, economically valuable professional physical-engineering work.Tasks require navigating multi-format project corpora, reconciling physical and code requirements, and producing production-ready deliverables.Source: §6 · p. 14
  • Expert-authored tasks are created end to end by practising engineers, while co-authored tasks are built from engineers’ briefs by an adversarial agent pipeline and then validated and intensified by experts.For design tasks, a hybrid score rewards cost-effective, fully compliant designs rather than merely admissible ones.Source: §6 · p. 14
  • Only one of 14 model-harness configurations exceeds 50%: Claude Opus 5.5 in Claude Code at 50.6%.The model matters more than the harness, but harness choice still shifts scores and costs substantially.Source: §6 · p. 14

A. Evaluation Configuration

Evaluation results use a shared model proxy, repeated runs, and grading outside the agent container.

A. Evaluation Configuration

  • Every model-harness pair runs every task three times with a six-hour agent budget per run.A model proxy pins each model’s highest reasoning setting and is the only endpoint accessible to the agent container.Source: A. Evaluation Configuration · p. 18
  • After the agent exits, Claude Opus 5.5 judges rubric items and frozen executable graders score co-authored tasks outside the agent container.Source: A. Evaluation Configuration · p. 18

B. Results for Every Model and Harness

Table 5 reports the mean scores of every model–harness pair across all 85 PLE tasks and underlies the model and harness comparisons in section 5.

B. Results for Every Model and Harness

C. Expert Seed Nomination Interface

Practising engineers nominate co-authored tasks through a structured interface that specifies the project context, inputs, goals, constraints, verification, and deliverables. These nominations guide task construction and verifier synthesis.

C. Expert Seed Nomination Interface

  • Practising engineers use the Task Nomination Interface to submit seed problems intended to reflect authentic, economically valuable engineering work.The interface replaces free-form prompts with six structured specification fields.Source: C. Expert Seed Nomination Interface · p. 20, Figure 9 · p. 20
  • The six fields specify the Situation, starting Inputs, quantifiable Goal, physical and code Constraints, deterministic Verification logic, and Expected Output deliverables.Examples of expected formats include XLSX schedules, DXF drawings, and IFC4 BIM models.Source: C. Expert Seed Nomination Interface · p. 20, Figure 9 · p. 20
  • The submitted nomination becomes the Expert Seed Prompt guiding the Task Creator agent’s research, environment construction, and executable verifier synthesis.Source: C. Expert Seed Nomination Interface · p. 20

Related papers

Loading 2610.30537v1…