PaperlayerView exact revision on arXiv

Source-linked AI summary

Agents' Last Exam

Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, Chris Duroiu, Haoyun Zhang, Jeffrey Lin, Weishu Zhang, Tyler Zeng, Ying Yan, Bo Liu, Hanson Wen, Mingyang Xu, Xiaoyuan Liu, Zimeng Chen, Weiyan Shi, Amanda Dsouza, Vincent Sunn Chen, Patrick Bryant, Carl Boettiger, Yamini Rangan, Bradley Rothenberg, Kyle Steinfeld, Arvind Rao, Tapio Schneider, Georgios Yannakakis, Laure Zanna, Kaan Ozbay, Ida Sim, Tarek Zohdi, George Em Karniadakis, Jack Gallant, Teresa Head-Gordon, Yushan Li, Wenxi Deng, Tao Sun, Huiqi Wang, Zhun Wang, Justin Xu, Chris Yuhao Liu, Yafei Cheng, Rongwang Hu, Aras Bacho, Shengcao Cao, Zengyi Qin, Yixiong Chen, Hengduan Fan, Hao Liu, Lin Zeng, Shashank Muralidhar Bharadwaj, Litian Gong, Yingxuan Yang, Maojia Song, Ruheng Wang, Zongzheng Zhang, Honglin Bao, Shuo Lu, Jianhong Tu, Zhonghua Wang, Zheng Zhang, Zijiao Chen, Yanqiong Jiang, Zhendong Li, Bohan Lyu, Chang Ma, Peiran Xu, Benran Zhang, Shangding Gu, Haoyue Hua, Haoyang Li, Wanzhe Liao, Chengzhi Liu, Junbo Peng, Haoran Sun, Zechen Xu, Bo Chen, Jiayi Cheng, Yi Jiang, Keying Kuang, Yuan Li, Youbang Pan, Ziyan Rao, Alexander Schubert, Yifan Shen, Vincent Siu, Xiatao Sun, Kangqi Zhang, Xiaopan Zhang, Yuchen Zhu, Ishaan Singh Chandok, Lei Ding, Jingxuan Fan, Andrew Glover, Jiaming Hu, Yiran Hu, Wenbo Huang, Zixin Jiang, Haoran Jin, Lukas Kim, Ming Liu, Yang Liu, Alireza Rafiei, Xuhuan Shen, Kunyang Sun, Sophia Sun, Ting Sun, Eric Wang, Yixin Wang, Hanwen Xing, Sihan Xu, Yuzheng Xu, Zhongxing Xu, Zhiling Yan, Boqin Yuan, Ruiqi Zhang, Yifan Zhang, Zibo Zhao, Liana, Santanu Bosu Antu, Haoyue Bai, Carlo Bosio, Joseph Cavanagh, Patricia Cavazos-Rehg, Tianxing Chen, Xuewen Chen, Yipu Chen, Chenyu Zhu, Chen Dai, Stefano De Castro, Yunfu Deng, Kaustubh Dhole, Jiayuan Ding, Chenchen Du, Zhehang Du, Hao Fan, Run-Ze Fan, Hengyu Fu, Shi Gu, Yifan Gu, Charlie Guo, Baihe Huang, Baixiang Huang, Rimika Jaiswal, Zhihan Jiang, Ran Jin, Erin Kasson, Xin Lan, Joseph Lee, Deren Lei, Chenyu Li, Daofeng Li, Haitao Li, Hongwei Li, Jingyan Li, Xiao Li, Yi Li, Yinsheng Li, Yuangang Li, Zhixu Li, Wenyu Liang, Longtai Liao, Kevin Qinghong Lin, Andy Zeyi Liu, Che Liu, Jiaming Liu, Kaiyuan Liu, Xuan Liu, Pan Lu, Wenbo Lv, Yicheng Lyu, Qiuyang Mang, Kyle Montgomery, Yuzhou Nie, Ruoxi Ning, Jorin Overwiening, Xu Pan, Layna Paraboschi, Core Francisco Park, Justin Purnomo, Swati Rajwal, Scott Rankin, Bixuan Ren, Yiren Rong, HaoYang Shang, Ventus Shaw, Fiona Shen, Jiawei Shen, Minqi Shi, Shi Qiu, Huaxiu Yao, Tianneng Shi, Jonah So, Vladislav Susoy, Hannah Szlyk, Haocheng Wang, Jialu Wang, Wei Wang, Xinyu Wang, Zehao Wang, Dowling Wong, Angela Wu, Dehao Wu, Fangyu Wu, Mengyuan "Millie" Wu, Yu Wu, Yuchen Wu, Yuhao Wu, Qingpo Wuwu, Weihang Xiao, Yongyi Xiong, Fan Xu, Ruiling Xu, Mingxuan Yan, Benjamin Yang, Jirong Yang, Sen Yang, Xiaoli Yang, Yushi Yang, Haoran Ye, Xiaohu Yu, Zhengming Yu, Chenlong Zhang, Chi Zhang, Hanning Zhang, Hanwen Zhang, Junge Zhang, Kunpeng Zhang, Song Zhang, Wenjin Zhang, Wenshuo Zhang, Ying Zhang, Yizhi Zhang, Brian Zhao, Qijian Zhao, Yimin Zhao, Yuhaohua Zheng, Liwei Zhou, Tianyue Zhou, Sichen Zhu, Siqi Zhu, Yan Zhu, Yishu Zhu, Jierui Zuo, Chonghao Cai, Helena Casademunt, Wenjia Chen, Cheng Cheng, Nawen Deng, Rao Fu, Tianfu Fu, Yifan Han, He Ren, Zhenyu He, Qiao Jin, Langlang Li, Yuetai Li, Sylvia Liu, Lu Lu, Luqing Zhou, Subhabrata Mukherjee, Yunqi Ouyang, Yin Ren, Dawei Shi, Haoran Wu, Zhiyue Wu, Hannah Yao, Zhuoran Yi, Jenny Yu, Rhea Zhan, Hang Zhou, Blake Zhu, Junfan Zhu, Alan Yuille, Yang Liu, Russell Alan Poldrack, Jiachen Li, Zhenglu Li, Molei Tao, Jing Huang, Wenqi Shi, Costas Spanos, Lichao Sun, Chenguang Wang, Orson Xu, Zhen Dong, Hector Gomez, Aylin Caliskan, Ali Emami, Haimin Hu, Zhi Li, Lihui Liu, Murphy Niu, Yi Shao, Jianxin Sun, Mikko Tolonen, Ting Wang, Sanjiv Das, Yanjun Gao, Wenbo Guo, Erika J Schneider, Zhiyong Lu, Yian Ma, Mark Mueller, Radha Poovendran, Somayeh Sojoudi, Yinglun Zhu, Dawn Song

arXiv:2606.05405v2cs.AIcs.CLcs.LG

TL;DR

Strong benchmark results have not been matched by measurable transformation in core industries, while evaluations provide limited evidence about agents’ sustained performance on economically valuable professional work. ALE benchmarks agents on expert-authored professional workflows. Frontier agents clear only a small fraction of ALE, which remains far from saturated.

  • Problem

    Widely used benchmarks provide limited measurement of agents’ ability to perform long-horizon, economically valuable work in real professional environments, despite strong benchmark results and muted broader economic impact.

    Source: §1 · p. 2

  • Method

    ALE benchmarks agents on expert-authored workflows sourced from professional practice and scored through deterministic checks and structured rubrics.

    Source: §1 · p. 3, §6 · p. 10

  • Results

    Frontier agents clear only a small fraction of ALE today, and the benchmark remains far from saturated.

    Source: §1 · p. 4, §6 · p. 10

  • Takeaways & Limitations

    ALE saturation would signal that agents can sustain the long-horizon, tool-intensive work required by professional practice.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a benchmark designed to evaluate AI agents on long horizon, economically valuable, real world tasks with verifiable outcomes. Developed in collaboration with 250+ industry experts, ALE covers non-physical industries defined with reference to O*NET / SOC 2018 (the U.S. federal occupational taxonomy). It is organized around a task taxonomy with 55 sub fields grouped into 13 industry clusters covering 1K+ tasks. Current results show that the hardest tier remains far from saturated: across mainstream harness and backbone configurations, the average full pass rate is below 1%. ALE is designed as a living benchmark: its task pool grows continuously as new workflows and industries are onboarded. More broadly, ALE is intended not merely as another leaderboard, but as an instrument for closing the gap between benchmark success and GDP relevant impact.

1 Introduction

ALE targets the gap between benchmark gains and muted economic impact by measuring sustained, economically valuable work in real professional workflows. It combines broad, expert-built industry coverage with verifiable outcomes, and current results show substantial difficulty remains.

Motivation

  • AI benchmark victories have outpaced measurable transformation in core industries, while evaluations for economically central sectors such as finance, law, electrical engineering, and manufacturing remain underdeveloped.Benchmarks shape research attention and engineering targets; widely used, verifiable evaluations have often accelerated progress and preceded deployment.Source: §1 · p. 2
  • Real workflows are difficult to collect, cover broadly, and verify across heterogeneous outputs, so existing benchmarks often trade away realism, breadth, or verifiability.Source: §1 · p. 3

Purpose

  • “Last Exam” names both a readiness threshold for sustained professional work and a difficulty frontier at the edge of reliable current capability, with ALE aiming to connect benchmark progress to GDP-relevant impact.Source: §1 · pp. 3–4

Benchmark design

  • 1K+ task instances span 55 subfields and 13 industry clusters, built with 250+ experts from completed professional projects, quality-controlled, and requiring combined GUI and CLI use.Expert advisory committees map workflow landscapes using the O*NET / SOC 2018 occupational taxonomy; admission includes first-pass review, engineer dry-runs, and expert peer review.Source: §1 · p. 3

Evaluation

  • ALE makes heterogeneous real-world outputs verifiable without human judges through structured deliverable-based or milestone-based checks against expert references and rubrics.Source: §1 · p. 4

Results

  • Below 50% on ALE’s easiest tier and under 10% on its hardest, Codex with GPT-5.5 (82% on Terminal-Bench) contrasts with mainstream agents’ near-zero hardest-tier pass rates.The passage identifies Codex with GPT-5.5 as the strongest configuration and says most mainstream agents, including Claude Code, have near-zero pass rates at that difficulty.Source: §1 · p. 4

2 Benchmark Design and Dataset Construction

ALE selects realistic, expert-scale workflows with verifiable outputs, organizes them in a taxonomy that exposes prior-benchmark coverage gaps, and builds a rigorously reviewed task pool with limited public release.

2.1 Benchmark Design Principles: What Tasks are We Looking for?

  • ALE admits workflows that reflect professional software practice, require substantial end-to-end work, and yield deterministically checkable outputs or measurable artifacts.For example, converting a 2D blueprint into a 3D model should use software architects would typically choose, such as SolidWorks or Rhino.Source: §2.1 · p. 4

2.2 Benchmark Scope and Taxonomy: Which Domains Do We Cover?

  • Grounded in SOC 2018 and O*NET, ALE groups workflows into 13 domains and 55 subdomains; even 16 prior benchmarks together leave 13 subdomains uncovered.Prior benchmarks’ published categories are mapped onto the shared taxonomy using an LLM-assisted classifier.Source: §2.2 · p. 4

2.3 Task Construction Pipeline: How are the Tasks Created?

  • A five-gate pipeline sources domain experts, specifies proposals, reviews them, implements runnable assets and evaluation logic, then peer-checks references, scoring bounds, and context before admission.Proposals specify a natural-language description, input files, target software, expected deliverable, and evaluation specification; engineer dry-runs can send gaps back to experts.Source: §2.3 · p. 5, Figure 4 · p. 5
  • ALE releases 150 of 1,490 task instances publicly and periodically rotates private tasks into the public set while replacing retired public tasks to limit contamination.The public subset is about 10% of the pool; the paper reports that Appendix D.1 verifies its representativeness.Source: §2.3 · p. 5, Figure 5 · p. 5

3 Evaluation Pipeline

ALE’s evaluation pipeline separates task specifications, agents, and environments through interchangeable interfaces. It defines a generalist agent class for workflows requiring both GUI and CLI capabilities, and scores heterogeneous outputs with task-specific evaluation modes.

3 Evaluation Pipeline

  • A task workflow is an end-to-end professional procedure, while a task instance is one runnable input-output case sharing that workflow’s evaluate() function.The paper’s unqualified “task” generally refers to the runnable instance.Source: §3 · p. 6

3.1 Pipeline Architecture

  • Each instance uses a decoupled executable task specification, agent, and virtual-machine environment, enabling interchangeable agents and deployment across cloud VMs or local containers.The specification’s load(), start(), and evaluate() functions define requirements, provision a deterministic state, and score outputs; the agent acts through screenshots, shell, mouse, keyboard, file edits, and APIs. The agent can write only to output/, while reference/ contains hidden scoring artifacts.Source: §3.1 · p. 6, Figure 6 · p. 6, §3 · p. 6

3.2 Agent Architecture: From CLI/GUI-agents to Generalist CUA

  • ALE evaluates Generalist CUA-agents with full capability across Brain, Eyes, Body, Hands, and Feet, because its workflows require the union of GUI and CLI capabilities.CLI agents lack visual perception, while GUI agents have limited orchestration, tool use, and runtime access, which restricts their ability to write code, manage files, and sustain long workflows.Source: §3.2 · p. 6
  • Modern CLI-native harnesses can be extended to GCUA by adding GUI-as-Tool or GUI-as-SubAgent, with the primary benchmark evaluation using GUI-as-Tool.GUI-as-SubAgent is reserved for models without native vision input; the representative harness also includes a main loop, modular prompt builder, unified tools, sub-agent dispatch, and context compaction.Source: §3.2 · p. 7

3.3 Evaluation Modes

  • ALE scores heterogeneous deliverables using task-specific artifact comparisons and combines their signals through gate-and-score or checklist and per-file averaging.Gate-and-score is the most common pattern: a binary precondition must pass before continuous quality scoring, and failing it sets the task score to 0.Source: §3.3 · p. 7
  • ALE rejects scoring workflows that rely only on holistic LLM judgments when deterministic checks are possible, and uses narrow evidence-anchored yes/no probes for cases requiring an LLM judge.Task instances share one evaluate() function while differing in input and reference data.Source: §3.3 · p. 7

4 Experiment

ALE experiments evaluate agents on authentic professional workflows that mix shell, GUI, file, and web tasks, using a unified GCUA setup. Analysis finds low GUI use relative to task demand, domain-dependent performance, and failures dominated by knowledge and approach gaps.

4 Experiment

  • All systems were evaluated in GCUA configuration because ALE workflows interleave shell, GUI, file, and web work across all five functional layers.GUI-as-Tool exposes 14 desktop-action tools through a unified CUA MCP bridge, letting one foundation model reason over shell output and visual feedback in one action loop.Source: §4 · p. 8, §4.1 · p. 8

4.1 Main Results

  • The 3.8% overall timeout rate occurred under a five-hour run cap, ranging from ∼1% for lightweight harnesses to 5.7% for OpenClaw.Table 1 reports mean fine-grained task scores, full-pass rates, costs, and wall-clock times; its lower panel covers 105 Linux-only ALE-CLI tasks alongside GCUA references.Source: §4.1 · pp. 8–9

4.2 Analysis

  • Computational mathematics and agriculture/environment scored highest (∼55–85%), business and legal scored ∼50–55%, and education remained below 25%.Claude Fable 5 and GPT-5.5 showed similar domain profiles; the authors suggest capability imbalances and uneven training exposure may contribute.Source: §4.2 · p. 9, Figure 9 · p. 9
  • Although 34% of public tasks designate graphical software as the primary tool, GUI use remained small across most configurations as agents substituted Bash/CLI.The tool-call mix varied with both the harness and the model.Source: Figure 9 · p. 9, §4.2 · p. 10
  • Understanding and Approach failures made up roughly three quarters of failed Claude Code + Opus 4.7 tasks, pointing to domain knowledge rather than execution as the dominant bottleneck.Agents lacking specialized knowledge often used ad-hoc scripts instead of intended domain software, reinforcing GUI underuse.Source: Figure 9 · p. 9, §4.2 · p. 10
  • Among well-engineered systems, foundation-model choice accounted for roughly 3× the performance spread of harness choice, while greater resource use did not reliably improve performance.The appendices also provide task-instance score heatmaps across the three tiers.Source: §4.2 · p. 10

5 Related Work

ALE is positioned as a benchmark for real professional workflows across all 55 SOC/O*NET industries, with deterministic automated verification. It addresses limitations in topical, agentic, and economically grounded benchmarks.

Related Work

  • ALE covers all 55 SOC/O*NET industries, draws every task from a real project completed by one of 300+ practitioners, and uses deterministic rubric-based automated verification.This contrasts with benchmarks that cover only narrower domains or depend on human grading.Source: §5 · p. 10
  • GDPval and RLI evaluate economically grounded, project-scale work but leave 16 and 14 of 55 industries untested, respectively, and rely on expensive human grading.Source: §5 · p. 10
  • MMLU [20], GPQA, and HLE are topically broad knowledge and exam-style benchmarks, but test what models know rather than what they can do.Source: §5 · p. 10
  • SWE-bench, OSWorld [51], WebArena [56], and GAIA [25] add multi-step interaction and tool use, but cover few software-centric domains and typically use curator-authored tasks.Source: §5 · p. 10

6 Conclusion … B.1.6 Robotics & Autonomous Systems

ALE is a benchmark for testing whether agents can sustain long-horizon, tool-intensive professional work, using verifiable outcomes to help close the gap between benchmark success and GDP-relevant impact.

6 Conclusion

  • ALE comprises 960 expert-authored task workflows (1,490 task instances) across 55 digital industries, scored with deterministic checks and structured rubrics.The workflows are sourced from work experts have already shipped; frontier agents currently clear only a small fraction.Source: §6 · p. 10

A Authors

  • Contributor affiliations are listed in full and keyed by superscript to a numbered institution list.Source: A Authors · p. 16

B.1.1 Taxonomy Definition

  • The final taxonomy contains 55 workflow-level subdomains grouped into 13 domains, including 51 SOC-anchored subdomains and a frontier supplement.The supplement adds four frontier subdomains and seven extensions; subdomains are formed by grouping occupational evidence into artifact-producing workflows, with expert review of borderline assignments.Source: §B.1.1 · p. 18

B.1.2 Industry Landscape Review

  • Subdomain workflow landscapes organize task collection by summarizing work settings, practitioner roles, digital inputs, dependencies, and output artifacts.They draw on field references, workflow documentation, LLM-assisted research, and expert review to check task realism and verifiable success criteria.Source: §B.1.2 · p. 18

B.1.3 Manufacturing & Industrial Operations

  • Manufacturing workflows turn CAD, materials, specifications, and production targets into plans, code, control files, and inspection packages, while tracking consistency across digital and physical processes.When inspection finds a feature out of specification, its record can route work back to fixture, parameter, or toolpath revision.Source: §B.1.3 · p. 18

B.1.4 Biomolecular Structure & Design

  • Biomolecular design workflows produce ligand rankings, structure models, sequence designs, codon-optimized constructs, and plasmid maps for experimental handoff.They track consistency between computational models and biological material, including protonation states, MSA depth, codon usage, and assembly-junction fidelity.Source: §B.1.4 · pp. 18–19

B.1.5 3D, Animation & Interactive Media

  • 3D and interactive-media workflows move assets from scripts and storyboards through geometry, rigging, animation, lighting, compositing, and engine integration.The asset graph can help locate where scale, color, or timing drift entered the pipeline.Source: §B.1.5 · p. 19

B.1.6 Robotics & Autonomous Systems

  • Robotics workflows translate task specifications and environment models into robot descriptions, perception calibration, controllers, planners, safety logic, and simulation assets.They track sim-to-real assumptions; when hardware diverges from simulation, the mismatch can inform updates to the simulator, controller, or policy.Source: §B.1.6 · p. 19

B.2 Task Construction Pipeline Details · B.3 Industry Task Cards and Metadata

ALE turns expert-submitted workflows into benchmark instances through a staged process of sourcing, specification, review, implementation, and quality control. Tasks are checked for executable environments, reproducibility, and evaluation integrity before acceptance.

B.2 Task Construction Pipeline Details

  • The construction protocol converts expert workflow submissions into benchmark instances through five gates: sourcing, submission and editing, first-pass review, implementation, and final QC and acceptance.This appendix expands the protocol summarized in Section 2.3 and depicted in Figure 4.Source: §B.2 · p. 19
  • An advisory committee anchors targeted recruitment of domain specialists who perform complex software workflows in their daily practice, supporting coverage across the defined taxonomies.The committee comprises leading industry professionals and expert practitioners.Source: §B.2 · p. 19
  • Practitioners submit past projects through a web portal and use AI-assisted tools to refine specifications covering the task description, inputs, tools, deliverable, and evaluation.Experts are asked to upload projects that historically took days or weeks; source-grounded tasks retain records for papers, datasets, standards, or workflow documentation that define assets or references [19].Source: §B.2 · pp. 19–20
  • First-pass review screens submissions using conference-style decisions, with revision-based decisions sending proposals back to experts for further editing.Decision categories include major or minor revision, borderline accept, accept, and strong accept.Source: §B.2 · p. 20
  • During implementation, engineers turn written specifications into runnable assets, provision software containers, codify evaluation logic, and dry-run tasks; gaps trigger logs to experts for resolution.An engineering review checks task execution and dependencies.Source: §B.2 · p. 20
  • Before acceptance, expert-committee QC checks reference-output correctness, calibrated evaluation bounds, sufficient problem context, reproducibility, and evaluation integrity; identified issues prompt final adjustments.The gate assesses whether the task can reach its final state without impossible or spuriously permissive evaluation bounds.Source: §B.2 · p. 21

B.3.1 Representative Task Cards

Representative task cards make ALE’s task construction and evaluation protocol concrete at the individual-instance level.

B.3.1 Representative Task Cards

  • Each card summarizes an executed task’s request, inputs, deliverables, rubric, score, and outcome; all shown trajectories used the Claude Code harness with Claude Opus 4.7.The cards complement the paper’s aggregate coverage and performance results.Source: §B.3.1 · p. 21

Task Card 1: Injection Mold-Flow Analysis PARTIAL

The task evaluates whether an agent can run a Moldex3D injection-mold simulation and extract solver-derived metrics into a results file. The agent completed the setup workflow but earned only partial numeric-result credit.

Task Card 1: Injection Mold-Flow Analysis PARTIAL

  • The task requires opening project 230057, applying process_spec.json, running fill, pack, cooling, and warp analyses, and writing solver-derived metrics to output/results.json.Required values include pressure, force, time, volume, and weight.Source: Task Card 1: Injection Mold-Flow… · p. 22
  • Hard gates reject missing, malformed, or all-null results.json; numeric fields are compared with hidden solver references at 1 percent relative tolerance.Generated project artifacts can provide limited partial credit when the main metrics are weak.Source: Task Card 1: Injection Mold-Flow… · p. 22
  • 0.4762 score: the agent followed the Moldex3D setup workflow, but submitted metrics were not reliably extracted from completed solver outputs.Some values were estimated rather than measured; successful GUI setup was insufficient without closed-loop numerical verification.Source: Task Card 1: Injection Mold-Flow… · p. 22

Task Card 2: Orchestral Music Transcription FAIL

The orchestral transcription task required a complete score, MIDI, and screenshot package; the agent received a zero after exporting only the MIDI, with the PDF and screenshot unconfirmed.

Task Card 2: Orchestral Music Transcription FAIL

  • The task required converting an audio brief into a readable full score and multitrack MIDI in Dorico, following the Prelude / Akinola / tempo 140 / 27-instrument specification and providing an overview screenshot.Source: Task Card 2: Orchestral Music… · p. 23
  • After artifact gates pass, pitch and rhythm each count 30%, dynamics 20%, instrument assignment 10%, and score layout 10%, with MIDI compared after track pairing and quantization.Source: Task Card 2: Orchestral Music… · p. 23
  • 0.0 was the score: the agent exported transcription.mid, but transcription.pdf and overview.png were unconfirmed, triggering the complete-bundle hard gate before musical accuracy was evaluated.Missing or invalid required artifacts cause a hard zero.Source: Task Card 2: Orchestral Music… · p. 23

Task Card 3: Reference-Based Chroma Key Compositing FAIL … C Evaluation Pipeline Details

Three task-card runs illustrate uneven completion: chroma-key compositing scored 0.0, skeletal animation 0.4285, and chest-X-ray adjudication 0.3333, with failures involving reference grounding, motion fidelity, and visual adjudication.

Task Card 3: Reference-Based Chroma Key Compositing FAIL

  • The chroma-key task required keying a bird clip in DaVinci Resolve, matching a reference image, and exporting a composited video.The expected output had to preserve foreground edges, remove the green screen, and match the reference composition.Source: Task Card 3: Reference-Based Chroma… · p. 24
  • 0.0 was the chroma-key score because the output failed to reproduce the reference-based foreground/background relationship despite using Resolve and exporting an MP4.The failure was attributed to visual task grounding rather than software operation alone.Source: Task Card 3: Reference-Based Chroma… · p. 24

Task Card 4: Skeletal Animation Reproduction PARTIAL

  • The animation task required rigging a character, reproducing reference motion with 19 exact bone names, and submitting editable and replayable artifacts.Validity gates required both files and a non-trivial rig; scoring also considered video match, replay consistency, skeleton coverage, and vision checks.Source: Task Card 4: Skeletal Animation… · p. 25
  • 0.4285 was the skeletal-animation score: required artifacts and animation were present, but the preview lasted about 4 seconds versus a 13.6-second reference.The motion was damped to reduce mesh distortion, limiting motion fidelity despite file-level validity.Source: Task Card 4: Skeletal Animation… · p. 25

Task Card 5: MicroDicom Chest X-Ray Adjudication PARTIAL

  • The adjudication task required reviewing nine DICOM cases, choosing between two reader annotations per case, and producing three fixed-schema TSV files.The box output had to select the hidden-supported reader and reach IoU of at least 0.50 against the gold box.Source: Task Card 5: MicroDicom Chest X-Ray… · p. 26
  • 0.3333 was the chest-X-ray adjudication score: all three required TSVs covered nine cases, but decisions only partially matched hidden references.Detailed image review appeared in only a few cases; later choices relied on coordinates and heuristics rather than systematic visual adjudication.Source: Task Card 5: MicroDicom Chest X-Ray… · p. 26

B.3.2 Executed Task Inventory

  • The executed-task inventory lists 150 selected public tasks with names, identifiers, domains, and brief descriptions under the ALE-V1 Benchmark Splits tab.The inventory is browsable online under the Benchmark Splits (ALE-V1, 2026/06) tab.Source: §B.3.2 · p. 27

C.1 Pipeline Architecture Details … C.3.4 Judge Types and the LLM-Judge Helper Layer

ALE’s evaluation pipeline separates task definitions, agents, and execution environments, then scores agent outputs against references or rubrics. Its evaluation favors reproducible checks and uses narrowly targeted LLM judgments only when objective scoring is unavailable.

C.1 Pipeline Architecture Details

  • The pipeline decouples task specifications, agents, and virtual-machine environments, enabling any interface-conforming agent to run on a task and the same task to move across environment backends.Specifications encode the task description, assets, software, and evaluation criteria; agents observe and act through GUI, shell, files, or APIs, while references remain hidden from the agent.Source: §C.1 · p. 27

C.2 Task Specification Protocol

  • The task protocol uses load() to declare task information, start() to prepare a deterministic environment, and evaluate() to score outputs against references or rubrics on a normalized [0, 1] scale.The phases are designed to make evaluation deterministic and reproducible.Source: §C.2 · pp. 27–28

C.3 Evaluation Modes: Full Taxonomy and Worked Examples

  • The appendix details where scoring runs, the artifact comparison modes, score-composition patterns, and the helper layer supporting LLM-as-judge evaluations.It points readers to concrete task implementations for source inspection.Source: §C.3 · p. 28

C.3.1 Execution Locale

  • Scoring usually runs on the host after artifacts are transferred; when required software cannot be moved, a VM-side verifier returns a score through stdout JSON without writing to output/.Host-side scoring is preferred because its code is easier to review, version-control, and rerun offline.Source: §C.3.1 · p. 28

C.3.2 Artifact Modes

  • Artifact comparisons include exact or hashed short answers, structured tabular or numeric fields scored against tolerances, and geometric or spatial deliverables evaluated against references.Exact/hash scoring is common in cybersecurity and some mathematics workflows; structured comparisons are common in finance, accounting, and clinical-data standards, while geometric scoring can use sampled surface distances.Source: §C.3.2 · pp. 28–29
  • Other modes assess visual appearance with a vision LLM, replay behavioral or world state, grade free-text reports by rubric, or run executable artifacts against held-out tests or an oracle.Source: §C.3.2 · p. 29

C.3.3 Score-Composition Patterns

  • Scores combine through gate-and-score, weighted rubrics, binary-checklist averaging, or pairwise file aggregation; gates can force zero before continuous scoring, and pairwise aggregation averages scores across matched files.The manufacturing/gcode example gates geometric similarity on passing collision and gouge checks; the weighted gcode rubric combines fractions within 0.3 mm and 2.0 mm.Source: §C.3.3 · p. 29

C.3.4 Judge Types and the LLM-Judge Helper Layer

  • ALE avoids LLM judges when deterministic checks are possible; unavoidable LLM workflows use narrow, evidence-anchored probes, with code handling gating, weighting, and aggregation.The helper layer supports targeted yes/no visual questions, binary checklists, and graded or JSON-structured rubrics; these probes check identifiable artifact features rather than asking for an abstract quality score.Source: §C.3.4 · pp. 30–31

C.3.5 Empirical Distribution of Judge Types … D.1 Public-Subset Representativeness

The appendix describes ALE’s scoring and task organization, documents its agent harness, and finds that public-subset cluster pass rates track those on the full pool.

C.3.5 Empirical Distribution of Judge Types

  • Code-based judges score 93.2% of workflows versus 6.8% LLM-as-judge; host-side scoring covers 88.5% versus 11.5% VM-side verifiers.VM-side workflows commonly require software stacks such as CAD/CAM, licensed financial workbooks, or headless 3D rendering; uploaded per-workflow verifiers run on the VM and return JSON scores.Source: §C.3.5 · p. 31, Table 4 · p. 31
  • Static analysis finds most evaluate() bodies gate with early zero returns, while explicit weighted-rubric counts are a lower bound because scripts may encode weights.The estimate finds at least two early [0.0] return sites in most bodies; the static count misses weighted aggregation in per-workflow scoring scripts.Source: §C.3.5 · p. 31

C.3.6 Reference Isolation and Robustness

  • Evaluators isolate references, award zero for missing references or malformed or absent outputs, and support reproducibility through deterministic code judges or recorded LLM judge models, prompts, and artifacts.Reference-path sanity checks also prevent misconfigured runs from silently inflating scores.Source: §C.3.6 · p. 32

C.3.7 Workflows and Task Instances

  • ALE’s current release contains 960 task workflows and 1,490 task instances, with instance scores averaged into workflows, then industries, then clusters.Instances within a workflow share one evaluate() implementation while carrying instance-specific configuration.Source: §C.3.7 · p. 32

C.4 Agent Harness Internals

  • The harness repeats a six-phase loop of initialization, context building, model call, action-or-delivery decision, tool-result collection, and overflow checking with compaction when needed.The loop ends when the model delivers rather than taking another action.Source: §C.4 · p. 32

C.4.1 Tool Surface and Terminology

  • Harness tools are grouped into a common taxonomy, while a three-tier context manager clears stale results, summarizes older exchanges, and truncates to hard context limits.The taxonomy maps differing harness tool names into categories including Bash, File, GUI, Web, planning/delegation, and Other.Source: §C.4.1 · pp. 32–33
  • ALE-Claw ports OpenClaw’s agent loop to Python, adapting it for CUA and enabling in-place harness ablations; removing production-assistant components reduces the system prompt by ∼65%.The port adds a composite computer-use tool and vision-driven GUI sub-agent while retaining OpenClaw’s context-management primitives.Source: §C.4.1 · pp. 33–34

D.1 Public-Subset Representativeness

  • Pearson r=0.89 (p<0.001) between public-subset and full-pool cluster pass rates supports representativeness, although the full pool scores higher because it contains more Near-Term tasks.Most clusters lie above the diagonal because the private pool contains proportionally more easier Near-Term tasks.Source: Figure 11 · p. 34, §D.1 · p. 34

D.2 Timeout Analysis … D.6 Per-Task Instance Score Heatmaps

The appendix describes the timeout and failure-analysis methods and reports that failures cluster in approach errors, while performance varies more with backbone than harness. Cost, runtime, and token consumption do not reliably predict scores.

D.2 Timeout Analysis

  • 3.8% of evaluated runs reached the five-hour cap; capped runs averaged 20.7 versus 33.2 for runs ending earlier.At the cap, the harness stops the agent and the evaluator scores the artifacts then present in the output directory.Source: §D.2 · p. 34

D.3 Failure Taxonomy Classification

  • The appendix documents a two-stage pipeline for producing the failure root-cause taxonomy presented in Figure 9.Source: §D.3 · p. 35

D.3.1 Stage 1: Trajectory analysis

  • For each failed task, OpenAI Codex analyzes run artifacts and creates a five-part Markdown card covering verdict, task, correct and incorrect actions, and scoring.Claims must cite artifact files and fields; the prompt excludes the full transcript to bound generation cost.Source: §D.3.1 · p. 35

D.3.2 Stage 2: Taxonomy classification

  • The study classifies each analysis card into a two-level failure taxonomy: Understanding (knowledge gaps or fabrication), Approach (wrong plans), Execution (flawed implementation), and Infrastructure (external constraints unrelated to agent capability).GPT-4o performs the classification at temperature 0.Source: §D.3.2 · pp. 35–36

D.3.3 Distribution

  • 47% of classifiable failures are Approach errors, compared with 31% Understanding and 22% Execution errors.Wrong strategy accounts for 30% and premature abandonment for 17%; timeout and resource-exhaustion cases are excluded from this breakdown.Source: §D.3.3 · p. 36

D.4 Model vs. Harness Effect

  • 16.8 percentage points separate backbones under a fixed OpenClaw harness, versus 4.9–7.2 points across harnesses under fixed backbones.GPT-5.5 spans 4.3% (Grok) to 21.1%; the passage attributes the larger variation primarily to foundation-model reasoning and domain knowledge.Source: Figure 12 · p. 36, §D.4 · p. 36

D.5 Cost, Time, and Token Efficiency

  • 45.8% at $326 makes ALE-Claw with GPT-5.5 the highest-scoring configuration, while ALE-Claw with Opus 4.7 costs $1 164 and scores 40.5%.Runtime and token volume also fail to track scores consistently: ALE-Claw (GPT-5.5) scores highest in about 51 hours, while Cursor (Opus 4.7) scores 41.8% using 460M tokens versus ALE-Claw (Opus 4.7) at 40.5% using 1 373M.Source: Figure 13 · p. 37, §D.5 · p. 37

D.6 Per-Task Instance Score Heatmaps

  • Figures 14–16 visualize per-task instance mean scores across evaluated systems for the Near-Term, Full-Spectrum, and Last-Exam tiers.Rows are sorted by average score across systems, columns by average score within each harness; gray cells indicate missing runs.Source: §D.6 · p. 37, Figure 14 · p. 38, Figure 15 · p. 39

Related papers

Loading 2606.05405v2…