PaperlayerView exact revision on arXiv

Source-linked AI summary

cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh

arXiv:2609.40284v1cs.LGcs.AIcs.CL

TL;DR

CUA speed and cost are deployment barriers, while differing evaluation environments make speed comparisons difficult. cua-speedrun standardizes infrastructure and task sets and evaluates agents across benchmarks; its results show that reasoning and environment I/O can affect task time in unexpected ways.

  • Problem

    CUA speed and deployment cost are barriers to adoption, and varying infrastructure makes fair speed and cost comparisons difficult.

    Source: §1 · p. 1

  • Method

    cua-speedrun standardizes evaluation infrastructure and agent interfaces, and selects representative task subsets to benchmark CUA performance, speed, and cost.

    Source: §1 · p. 2

  • Results

    More reasoning can reduce task time by avoiding repeated unsuccessful actions, while faster environment I/O can increase it through additional interaction.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The findings support studying reasoning and interaction together when developing faster agents.

    Source: §5 · p. 10

    Reported times and costs reflect inference conditions during evaluation, and task subsets may need revalidation as agent capabilities change.

    Source: A LIMITATIONS · p. 16

Abstract

from arXiv · show

Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at https://cuaspeedrun.com.

1 INTRODUCTION

CUA speed and cost are important deployment barriers, but inconsistent infrastructure and interfaces make fair benchmarking difficult. cua-speedrun standardizes evaluation and task selection, and its analyses show that reasoning and environment latency can affect speed in counterintuitive ways.

1 INTRODUCTION

  • Different execution environments, deployment setups, interfaces, hyperparameters, harnesses, and benchmark implementations hinder consistent comparisons of CUA speed and cost.CUAs have strong benchmark capabilities, but speed and deployment cost remain adoption barriers.Source: §1 · pp. 1–2
  • cua-speedrun standardizes infrastructure and provides a common agent interface so one implementation can operate across benchmarks.Evaluations use the same execution pipeline and virtual-machine setup; the common interface supports portable harnesses.Source: §1 · p. 2
  • An energy-minimization framework selects representative task subsets to reduce evaluation time while preserving result signal and relative agent ordering.On OSWorld, the approach reduces evaluation time by 84.9% while maintaining high correlation with full-set evaluations.Source: §1 · p. 2
  • The platform benchmarks frontier agents across OSWorld-Verified, OSWorld 2.0, CUA-World, and MyPCBench, including long-horizon and personalized computer use.It supports live Pareto rankings of cost-effectiveness, speed, and capability.Source: §1 · pp. 2–3
  • More reasoning can improve scores while reducing task time by avoiding repeated unsuccessful actions, whereas faster environment I/O can increase task time.The evaluated open-weight models also do not lie on the performance–time or performance–cost Pareto frontier of any benchmark.Source: §1 · p. 3
  • The authors argue that studying efficiency alongside success can support research toward faster, cheaper, and more practical CUAs.They hope public release encourages work on efficiency and wider real-world adoption.Source: §1 · p. 3

2 METHOD

The method decouples agents, benchmarks, infrastructure, and agent loops for more independent evaluation. It selects small representative task sets and provides a standardized runtime, instrumentation, and interfaces for consistent execution.

2 METHOD

  • The framework standardizes measurements and infrastructure, selects representative tasks, and executes and times agents consistently.These are the method’s three components: measurement standardization, subset selection, and consistent agent execution and timing.Source: §2 · p. 3

2.1 STANDARDIZED COMPUTER-USE EVALUATION

  • Evaluations hold benchmark and infrastructure fixed when comparing agents, and hold agent and infrastructure fixed when comparing benchmarks.Tasks specify an initial desktop state, natural-language instruction, and verifier; timing covers agent task execution and separates environment-operation time from agent-operation time.Source: §2.1 · p. 3

2.2 SELECTING A REPRESENTATIVE SET

  • Task subsets are selected by minimizing energy distance between full-benchmark and subset score distributions, using partial and exact task-completion scores.The objective favors subsets that represent full-benchmark performance patterns while avoiding redundant tasks; selection uses leave-one-agent-out validation.Source: §2.2 · p. 4
  • The smallest subset is chosen when rank correlation with full-benchmark results meets the 0.95 threshold at K−1, K, and K+1 tasks.Selected sets contain 50 of 295 OSWorld tasks, 52 of 63 OSWorld2 tasks, 26 of 143 CUA-World tasks, and 38 of 184 MyPCBench tasks.Source: §2.2 · p. 4

2.3 INFRASTRUCTURE

  • The infrastructure hosts isolated agent and environment sandboxes in Modal and uses fixed L40S GPUs for self-hosted open-weight inference.CUA-AutoDebug tests input and screenshot behavior and uncovered input errors in widely used CUA tools, which the infrastructure corrects.Source: §2.3 · pp. 4–5
  • FastCUA reduces action-to-observation latency from roughly 2–3 seconds to 2–28 ms, enabling evaluation of whether lower infrastructure latency reduces wall time.At that speed, screenshots can be captured before the application updates its interface in response to an action.Source: §2.3 · p. 5
  • A single Python agent-loop file and standard templates support implementation across hosted or local evaluations through one CLI call.The infrastructure standardizes keyboard and mouse actions and desktop screenshot observations.Source: §2.3 · p. 5

3 RESULTS

Across OSWorld and OSWorld2, the performance–time and performance–cost frontiers vary by benchmark, and reasoning effort, I/O, harnesses, and batching alter the trade-offs. The analysis also offers practical guidance for choosing configurations against workload and budget.

3.1 EXPERIMENTAL SETUP

  • The evaluation covers 56 OSWorld and 21 OSWorld2 configurations on representative task sets, with repeated GPT-6 Astra (low) runs scoring 90.8 ± 1.8% in 89.5 ± 2.2 s.Selected evaluations also compare harnesses, I/O modes, and single versus batched tool calls; Astra with Codex exceeds 90% rubric scores on MyPCBench and CUA-World.Source: §3.1 · p. 5, §3.2 · p. 8

3.2 ANALYSIS

  • On OSWorld, Gemini 3.8 Flash (low) matches the highest score at nearly GPT-6 Astra’s time for $0.11 rather than $0.71 per task, while the frontiers shift on OSWorld2.OSWorld2’s time frontier consists entirely of Astra configurations, while its cost frontier also includes Gemini 3.8 Flash and Muse Spark 1.3.Source: §3.2 · pp. 5–6, Figure 2 · p. 6
  • On OSWorld, Gemini 3 Flash Preview at medium effort scores 57.6% in 268 s, versus 33.6% in 492 s at low effort, despite generating more tokens.Low effort generated 1,598 tokens and took 56.0 steps, compared with 4,125 tokens and 32.5 steps at medium effort; low-effort trajectories included repeated unsuccessful actions.Source: §3.2 · pp. 6–7
  • Reasoning effort has benchmark-dependent returns: Gemini 3.8 Flash’s medium setting doubles OSWorld time without improving score, but raises OSWorld2 performance from 35.3% to 59.9%.On OSWorld2, moving from medium to high adds 0.2 percentage points while increasing time by 49% and cost by 72%.Source: §3.2 · p. 7, Figure 4 · p. 7
  • For GPT-6 Astra (low), fast I/O cuts mean environment processing from 12.68 s to 0.34 s per task but increases total task time from 89.5 s to 99.0 s.The authors attribute the increase to more steps: fast I/O can return screenshots before an application updates, and current models do not use waits optimally.Source: §3.2 · pp. 7–8, Figure 5 · p. 7
  • Harness choice changes the speed–performance trade-off: GPT-5.6 Luna at low effort scores 75.6% in 145 s with Codex versus 61.8% in 74 s through the direct API.At medium and high effort, the direct-API agent is both more accurate and faster, so neither harness is uniformly better.Source: §3.2 · p. 8
  • Restricting GPT-6 Astra (xhigh) to one action per response keeps its score at 91.6% but raises mean task time from 127 s to 183 s.The benefit of batching depends on how well a model plans multi-action sequences.Source: §3.2 · p. 8

3.3 PRACTICAL SUGGESTIONS FOR BUILDING CUAS

  • The authors recommend selecting frontier configurations for the target workload and budget, tuning effort for measured gains, batching when intermediate screenshots are unnecessary, and optimizing harnesses, tokens, and steps.Their guidance cautions that faster generation or I/O helps only if it does not lead to longer outputs or additional steps.Source: §3.3 · p. 8

4 RELATED WORK

Computer-use benchmarks have progressed from synthetic interfaces to realistic websites, live Internet tasks, and dynamic desktop environments, with newer suites emphasizing long horizons and personal-assistant use. Related work also studies computer-use models and methods for more reproducible, efficient evaluation.

4 RELATED WORK

  • Benchmark settings progressed from synthetic interfaces to realistic website analogs, static real-site tasks, live-Internet evaluations, and dynamic desktop environments.Recent benchmarks also emphasize long-horizon tasks and personalized computer use.Source: §4 · pp. 8–9
  • Computer-use models include frontier general models and specialized systems such as UI-TARS, MolmoWeb, Fara-1.5, ScaleCUA, and Qwen-CUA.Source: §4 · p. 9
  • Evaluation research addresses noisy or unreliable results and reduces benchmark cost through task subsets selected with item response theory, sparse optimization, human validation, or cheap proxy evaluations.The cited work includes studies of inflated web-agent success and unreliable single-run claims.Source: §4 · p. 9

5 CONCLUSION

The conclusion presents cua-speedrun as a standardized framework for evaluating computer-use agent performance, speed, and cost. Its OSWorld and OSWorld2 evaluations identify changing Pareto frontiers and show that reasoning and interaction can affect task time in opposing ways.

5 CONCLUSION

  • Cua-speedrun evaluates computer-use agent performance, speed, and cost under standardized infrastructure.Source: §5 · p. 9
  • Across OSWorld and OSWorld2, the performance–time and performance–cost Pareto frontiers change across benchmarks.Source: §5 · p. 9
  • More reasoning can reduce task time by avoiding repeated unsuccessful actions, whereas faster I/O can increase it through additional interaction.The authors conclude that reasoning and interaction should be studied together when developing faster agents.Source: §5 · pp. 9–10

A LIMITATIONS

The evaluation constrains reported times and costs to API conditions at evaluation time, and broad coverage is limited by evaluation cost. The infrastructure standardizes sandbox execution and provides a common cross-benchmark agent interface, including a faster I/O implementation.

A LIMITATIONS

  • Reported times and costs reflect API inference conditions at evaluation time; subset scores approximate full-benchmark performance and may need revalidation as agents change.Evaluation cost also limits coverage of every model, harness, reasoning-effort, and benchmark combination.Source: A LIMITATIONS · p. 16

B.1 AGENT AND ENVIRONMENT SANDBOXES

  • Separate agent and environment sandboxes let the same agent implementation run across benchmarks while task initialization and verification remain independent of the agent.The environment contains the desktop, applications, and initial task state; it scores the trajectory or final state when the agent finishes or reaches its limit.Source: §B.1 · p. 16

B.2 MODEL SERVING, RESOURCE ALLOCATION, AND PARALLEL EVALUATION

  • Tasks start from fresh environments, and their clocks run only after agent and environment readiness, excluding waits for an execution slot.Model weights are reused across tasks; model initialization and weight loading finish before timing begins.Source: §B.2 · p. 16

B.3 FAST I/O IMPLEMENTATION

  • Fast I/O combines action execution and screenshot delivery, with background screenshot encoding and reusable network connections.It can return a screenshot before the application has updated its interface.Source: §B.3 · p. 16

B.4 ACTION LATENCY: FAST I/O, GYM-ANYTHING, AND OSWORLD

  • 2.4–27.6 ms per call: Fast I/O is substantially faster than existing implementations, which take 2.5–3.5 s for the tested actions.A click takes 27.6 ms versus 2.8 s with Gym-Anything; typing 100 characters takes 15.0 ms versus 3.5 s.Source: §B.4 · p. 17, Table 1 · p. 17

B.5 LATENCY BREAKDOWN

Fixed post-action sleeps account for most baseline latency, while Fast I/O completes the tested commands in under 50 ms. Across five evaluations, its environment processing remains low; the following sections explain agent setup and use.

B.5 LATENCY BREAKDOWN

  • 2-second fixed sleeps account for most baseline latency; after subtracting them, Gym-Anything retains 804–1524 ms and OSWorld 524–769 ms.Gym-Anything also captures and transfers PNGs, while most remaining OSWorld time falls in action and screenshot requests.Source: §B.5 · p. 17, Table 2 · p. 17
  • Under 50 ms: all 80 tested Fast I/O commands complete within this interval, with median times of 38.5 ms for clicks and 12.8–29.5 ms for the other actions.Communication averages 4.6–5.0 ms; image preparation takes 0.02–0.27 ms.Source: §B.5 · p. 18, Table 3 · p. 18

B.6 FAST I/O CONSISTENCY ACROSS FIVE SEEDS

  • 0.336 ± 0.015 s per task: environment processing excluding requested waits across five Fast I/O runs; mean task time is 99.0 ± 3.9 s and score is 90.4 ± 1.8%.Requested waits account for 4.739 s of 5.075 s mean environment time per task.Source: §B.6 · p. 18, Table 4 · p. 18

C IMPLEMENTING AND EVALUATING AN AGENT

  • To evaluate an agent, users implement its model calls, prompts, and interaction history; the common interface handles setup, desktop interaction, and verification across benchmarks.Templates are available for direct model APIs, Codex, Claude Code, and vLLM-served open-weight agents; initialization occurs before task timing.Source: C IMPLEMENTING AND EVALUATING AN AGENT · p. 18, §C.1 · p. 19

C.2 MINIMAL AGENT EXAMPLE

  • The minimal agent loop requests screenshots, chooses actions from the instruction, screenshot, and interaction history, and ends when the agent signals completion.The benchmark verifier, not the agent, determines the score.Source: §C.2 · p. 19

C.3 HOSTED AND LOCAL EVALUATION

The evaluation interface supports hosted or local runs and records settings, seeds, scores, and timings. The paper defines interaction and token measures, then compares configurations and their frontiers across OSWorld and OSWorld2.

C.3 HOSTED AND LOCAL EVALUATION

  • Hosted runs use Modal, while local runs place agent and environment VMs on the evaluator host; evaluation records settings, seeds, task scores, timings, logs, and screenshots.Local desktop evaluation requires a Linux host with the benchmark’s VM runtime and hardware.Source: §C.3 · p. 20

D INTERACTION AND TOKEN MEASUREMENTS

  • Mean verifier score retains partial credit, while mean task time and model cost aggregate each agent’s task outcomes.The timing decomposition separates environment time, including requested waits, from agent time, which includes model requests and agent-side processing.Source: D INTERACTION AND TOKEN MEASUREMENTS · p. 20

D.1 MODEL CALLS, ACTION BATCHES, AND INDIVIDUAL ACTIONS

  • Model responses and desktop actions are counted separately; an action batch can contain several individual actions, and typing a string counts as one action.For example, clicking, typing, and pressing Enter in one request is one batch but three actions.Source: §D.1 · p. 20

D.2 TOKEN ACCOUNTING

  • Generated tokens include reasoning and other output, and the effective token rate divides total generated tokens by total agent time.Provider-supplied reasoning-token counts are reported separately.Source: §D.2 · p. 21

D.3 FULL INTERACTION AND TOKEN COMPARISONS

  • Tables report per-task interaction and token means, including unsuccessful tasks, for OSWorld and OSWorld2 configurations.Tokens include reasoning and other generated output.Source: §D.3 · p. 21, Table 5 · p. 21, Table 6 · p. 22

D.4 PERFORMANCE–INTERACTION AND PERFORMANCE–TOKEN FRONTIERS

  • OSWorld and OSWorld2 plots compare performance against action batches, individual actions, and generated tokens, marking nondominated configurations for each measure.The comparisons reuse trajectories from the time and cost analyses and average over all tasks.Source: §D.4 · p. 23, Figure 7 · p. 23, Figure 8 · p. 24

E.1 MODEL, HARNESS, AND REASONING-EFFORT CONFIGURATIONS

  • Configuration tables report mean score, time, cost, and token use by model, harness, and reasoning setting; the accompanying plots identify their performance–time and performance–cost comparisons.The evaluations include Luna through the direct API and Codex, and Kimi with single-call and batched-call harnesses.Source: §E.1 · p. 25, Table 7 · p. 25, Table 8 · p. 26

E.2 OSWORLD: PERFORMANCE–TIME AND PERFORMANCE–COST FRONTIERS

Across these benchmarks, the preferred reasoning effort and performance–time–cost trade-offs vary by model and task set. Five-seed OSWorld results show little variation in score and task time under the tested setup.

E.2 OSWORLD: PERFORMANCE–TIME AND PERFORMANCE–COST FRONTIERS

  • With generation time normalized to 100 tokens/s, Astra is faster than Opus; measured task times are 89.5 s for Astra and 85.9 s for Opus before normalization.Opus scores 87.6% and generates 1,795 tokens per task; Astra scores 90.8% and generates 1,579 tokens per task.Source: §E.2 · p. 27

E.3 OSWORLD2: PERFORMANCE–TIME AND PERFORMANCE–COST FRONTIERS

  • Astra defines the entire OSWorld2 time frontier, while Gemini 3.8 Flash and Muse Spark 1.3 offer lower-cost choices on the cost frontier.The comparison covers all 21 OSWorld2 configurations.Source: §E.3 · p. 28, Figure 12 · p. 28

E.4 OSWORLD2: JOINT PERFORMANCE–TIME–COST FRONTIER

  • The OSWorld2 joint frontier includes Astra at all four reasoning settings, Gemini 3.8 Flash at all three, and Muse Spark 1.3 at minimal effort.A configuration is included when no evaluated alternative improves one of score, time, or cost without worsening another.Source: §E.4 · p. 29, Figure 13 · p. 29

E.5 REASONING EFFORT ON MYPCBENCH

  • 93.6%: xhigh achieves the highest MyPCBench score, while medium has the lowest mean time and cost, at 495 s and $4.33 per task.The comparison uses the 38-task representative set and a 100-action-batch limit.Source: §E.5 · p. 30, Table 10 · p. 30, Figure 14 · p. 30

E.6 REASONING EFFORT ON CUA-WORLD

  • 94.2%: high effort outperforms xhigh’s 93.0% on CUA-World, while taking less time (1,108 vs. 1,515 s) and costing less ($11.11 vs. $15.06 per task).All four settings use Astra through Codex on the same 26 long-horizon tasks.Source: §E.6 · p. 30, Table 10 · p. 30, Figure 14 · p. 30

F RUN-TO-RUN VARIATION

  • The five-seed evaluation varies task seeds while holding the agent prompt and 500-step limit fixed for GPT-6 Astra at low effort through Codex on 50 OSWorld tasks.It measures variation in scores, task times, and action sequences, including whether the same tasks succeed across runs.Source: F RUN-TO-RUN VARIATION · p. 31, §F.1 · p. 31

F.2 SCORES AND TIMES ACROSS SEEDS

  • 90.82 ± 1.79% score and 89.54 ± 2.16 s mean task time: both vary little across five runs; action batches are also stable at 5.86 ± 0.06 per task.The values are means ± standard deviations across runs.Source: §F.2 · p. 31, Table 11 · p. 31

F.3 TASK-LEVEL SCORE AGREEMENT

Repeated runs produce highly consistent task scores, while different valid action sequences can still complete the same task with different numbers of batches and times.

F.3 TASK-LEVEL SCORE AGREEMENT

  • 42 tasks received full credit in all five standard-I/O runs, and partial-credit scores matched across runs on 48 of 50 tasks.With Fast I/O, 40 tasks received full credit in all five runs, while partial scores matched on 45 tasks.Source: §F.3 · p. 31
  • Standard-I/O pass@1 was 87.2%, pass@5 was 88.0%, and best-of-five partial score was 91.62%.Pass@k measures whether a set of k runs contains a full success; best-of-k partial credit uses the highest score within that set.Source: §F.3 · p. 31, Table 12 · p. 31

F.4 VARIATION IN ACTION SEQUENCES

  • In the movie-title spreadsheet task, the two-batch trajectory took 40.3 s versus 97.4 s for a seven-batch trajectory, with both receiving full credit.The faster trajectory applied one formula to the selected rows at once.Source: §F.4 · p. 32

G REPRESENTATIVE TASK SELECTION

The section develops and validates representative task subsets intended to preserve benchmark scores, rankings, and speed–performance comparisons while reducing evaluation effort. It also describes task-population screening and checks that runtime and harness actions are executed correctly.

G.2 INITIALIZATION, OPTIMIZATION, AND TASK COUNT

  • The selection procedure represents task score patterns using distances between task profiles, then minimizes a normalized energy objective for a subset of size K.It uses 100 balanced initializations and one-task swaps, choosing the smallest K whose held-out rank correlations meet the threshold at K−1, K, and K+1.Source: §G.2 · pp. 33–34

G.3 HELD-OUT SCORE AND RANKING ACCURACY

  • At 50 OSWorld tasks, held-out partial-score error fell from 4.02 to 2.16 percentage points; selected sets also preserved rankings and recovered two of three frontier configurations.At 50 OSWorld tasks, estimated-frontier precision was 100% and recall was 66.7%; pairwise dominance agreed for 91.7% of model pairs.Source: §G.3 · p. 35, Table 14 · p. 35, §G.4 · p. 36

G.1 MODELS AND TASKS USED FOR SELECTION

  • The selected evaluation sets contain 50 OSWorld tasks, 52 OSWorld2 tasks, 26 CUA-World tasks, and 38 of MyPCBench’s 184 canonical tasks.The OSWorld and OSWorld2 selected sets preserve both partial-score and exact-completion rankings above 0.95 at the chosen task counts.Source: §G.1 · p. 33, §G.3 · p. 35

G.4 PRESERVING THE SPEED–PERFORMANCE FRONTIER

  • Selecting 50 of 295 OSWorld tasks reduces the task count by 5.9×; these tasks account for 15.1% of recorded trajectory time, a 6.61× reduction.The time calculation excludes setup and verification.Source: §G.4 · p. 36

G.5 COMPARING TASK-SELECTION METHODS

  • At a 32-task OSWorld budget, energy-based selection’s mean error was 4.42 percentage points, versus 5.08 for difficulty-stratified random selection and 8.08 for the IRT-inspired method.Energy-based selection estimates each agent’s benchmark score as its average score on the selected tasks.Source: §G.5 · p. 37, Table 15 · p. 37

H.1 INCLUSION RULE

I.2 INPUT FAILURES DETECTED BY THE RUNTIME TESTS

  • Runtime tests found 11 failures among 94 supported input cases through SSH and PyAutoGUI, including Unicode and named-key cases; the corrected runtime passed all supported cases in five repetitions.The tests compare requested keyboard and mouse inputs with the events received.Source: §I.2 · p. 40, Table 19 · p. 40

J.1 TASK DEFINITION

  • The task interface pairs an initial environment state with an instruction and verifier; agents observe desktop screenshots and produce actions until termination or a task limit.The infrastructure separates the agent from the task environment, with benchmark verification scoring the resulting trajectory or state.Source: §J.1 · p. 41, §J.2 · p. 41

J.3 TASK LIMITS AND EXECUTION

The evaluation fixes task and execution conditions, applies benchmark-specific verification, and accounts for model-request costs. Controlled comparisons show that batching and harness choice can change both speed and score.

J.3 TASK LIMITS AND EXECUTION

  • Task time runs from delivery of the instruction until agent termination or the task limit, excluding setup, initialization, and verification; the benchmark’s own verification procedure scores completion.Evaluation conditions fix the task set, agent, reasoning setting, environment resources, and parallel-task count.Source: §J.3 · p. 42, §J.4 · p. 42

J.5 MODEL COST

  • Model cost sums task-level request charges or token usage at applicable API prices, excluding hosting, model-server rental, setup, and separate verification calls.The analysis examines batching, harnesses, and reasoning effort through within-model comparisons.Source: §J.5 · p. 42, K ADDITIONAL AGENT ANALYSES · p. 42

K.1 CONTROLLED BATCHING COMPARISONS

  • At Astra xhigh, batching preserved the 91.62% score while reducing mean task time from 182.7 s to 126.8 s.The batched setting used 7.14 action batches and generated 2,060 tokens, compared with 16.50 batches and 2,559 tokens when restricted to one action per screenshot.Source: §K.1 · p. 43, Table 22 · p. 43
  • At Kimi maximum effort, batched tools reduced mean task time from 606.3 to 443.7 s and raised score from 85.62% to 89.62%; at high effort, batching increased time and lowered score.The high-effort comparison changed from 305.1 to 358.1 s and from 81.62% to 79.62%.Source: §K.1 · p. 43, Table 22 · p. 43

K.2 HARNESS COMPARISONS ACROSS REASONING EFFORTS

  • At low effort, Codex raised GPT-5.6 Luna’s score from 61.8% to 75.6% but increased mean task time from 74 to 145 s.At medium and high effort, the direct-API agent was both more accurate and faster than Codex.Source: §K.2 · p. 44, Figure 17 · p. 44

K.3 COMPARING REASONING EFFORTS ON THE SAME SUCCESSFUL TASKS

On tasks completed at every compared setting, higher reasoning effort can coincide with longer times and more output; a separate search for task-selection methods sharply reduced validation error without improving held-out-model error. The results caution against relying on validation gains alone to predict generalization.

K.3 COMPARING REASONING EFFORTS ON THE SAME SUCCESSFUL TASKS

  • Gemini 3.8 Flash took 3,024 s per task at high effort versus 1,291 s at low on five tasks completed at all three efforts, while generating 73,578 versus 29,596 tokens.Astra xhigh took 1,132 s on 13 tasks completed at all four settings, versus 792 s at low and 753 s at medium.Source: §K.3 · p. 45, Table 23 · p. 45

L.1 SEARCH AND EVALUATION

  • The coding agent developed candidate task-selection and score-prediction methods using six models, with a seventh model withheld for separate evaluation.Each candidate selected 32 tasks; the agent received six-fold leave-one-agent-out validation error during search.Source: §L.1 · p. 46, L AUTOMATED SEARCH FOR TASK-SELECTION… · p. 46

L.2 VALIDATION GAINS DO NOT GENERALIZE TO THE HELD-OUT MODEL

  • Across 34 task-selection methods, validation error fell from 2.43 to 0.01 percentage points, while held-out-model error changed from 3.90 to 4.00.Repeated selection guided by validation errors allowed the search to overfit the six models used during development.Source: §L.2 · p. 46, Figure 18 · p. 46
Loading 2609.40284v1…