PaperlayerView exact revision on arXiv

Source-linked AI summary

CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

Prince Zizhuang Wang, Chenhao Liang, Zelong Xu, Aojie Yuan, Xiaolin Zhou, Haiyue Zhang, Yue Zhao, Xiyang Hu, Shuli Jiang

arXiv:2609.32600v1cs.AIcs.SE

TL;DR

Coding agents and computer-use agents are often studied separately, despite software repair requiring visual inspection and interaction to connect runtime behavior with code. CUA-SWE combines coding and computer use in a benchmark with deterministic verification; across four domains, frontier models achieve higher aggregate task success with hybrid access than code-only access.

  • Problem

    Coding and computer-use agents are largely studied separately, leaving underexplored how agents use visual feedback to diagnose, repair, and verify software and recover requirements from running applications.

    Source: §1 · p. 2

  • Method

    CUA-SWE combines editable projects, development commands, GUI interaction, and screenshot observations in one environment, with deterministic tests checking requested behavior and preserved functionality.

    Source: §1 · p. 2

  • Results

    Every frontier model in the four-domain comparison achieves higher aggregate task success with hybrid access; GPT-6-Astra gains 48.6 percentage points, with gains concentrated on application-material-dependent tasks.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    CUA-SWE supports measuring and expanding agents’ ability to translate visual runtime evidence into verified software changes.

    Source: §7 · p. 10

    Whether CUA-SWE provides a verifiable learning signal and how to develop scalable RLVR training remain future work.

    Source: §4.4 · p. 9, §6 · p. 10

Abstract

from arXiv · show

Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works. Existing coding agents and computer-use agents are largely studied in isolation, leaving this integrated development process underexplored. Diagnosing a runtime interaction failure requires agents to connect visual observations with the responsible code, then use the application again to verify the repair. We introduce CUA-SWE, a benchmark, environment, and evaluation pipeline for software engineering with computer use. Beyond studying how GUI feedback supports diagnosis and repair, we ask whether agents can complete software engineering tasks when required specification or operational information is available only through the running application's visual interface. CUA-SWE spans four software engineering domains and requires agents to modify code and configuration, execute commands, interact with running software, and inspect visual feedback within the same task. Each task includes deterministic, task-specific tests that verify whether the resulting software satisfies the requirements and preserves specified behavior. Our evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes. We examine performance across domains and task information requirements, alongside the development behaviors associated with successful repairs. CUA-SWE provides a unified testbed for studying how agents use visual feedback and interaction to guide software engineering, with executable correctness criteria for the resulting software.

1 Introduction

Interactive software repair requires agents to connect visual runtime behavior with code changes and then check the effects in the running application. CUA-SWE combines coding and computer use across development tasks, with deterministic tests and an evaluation of frontier agents.

1 Introduction

  • Agents must reproduce interaction failures, connect observed behavior to responsible code, and replay repairs while checking preserved behavior.A game may build and appear correct in a static screenshot while still failing when a character lands after a jump.Source: §1 · p. 2
  • Prior benchmarks often assess coding and computer use separately, motivating two questions about GUI-guided repair and visually supplied task requirements.The questions are how agents use GUI feedback to diagnose, repair, and verify software, and whether they can complete tasks whose information is available only through the running application.Source: §1 · p. 2
  • CUA-SWE lets agents edit projects, run commands, interact with running software, inspect screenshots, and revise code within one development episode.Agents choose when to use these tools, so visual observations can guide changes and later interaction can check their effects.Source: §1 · p. 2
  • Deterministic task-specific tests check requested behavior and preservation of existing functionality through execution and interaction.The tests provide consistent criteria for task success across four software engineering domains.Source: §1 · p. 2
  • The evaluation examines frontier agents across domains and application families, including tasks requiring recovery of visual-interface specifications or operational information.It compares code-only execution with hybrid access to screenshots and graphical actions and studies development trajectories.Source: §1 · p. 3

2 When CUA Meets Visual Software Engineering

CUA-SWE models software development as an agent modifying an editable codebase while observing a running application through partial visual and execution feedback. The agent can combine coding and computer-use actions, and its submitted code is judged by a deterministic verifier.

2 When CUA Meets Visual Software Engineering

  • Each task gives the agent an instruction, editable codebase, running application access, and a finite action budget before submission.The verifier checks requested functionality, preserved behavior, and permitted-change constraints.Source: §2 · p. 3
  • Observations may include file contents, command output, or screenshots, providing only a partial view of code and runtime state.Visible effects can depend on prior inputs and whether the application has loaded an edit.Source: §2 · p. 3
  • The action space combines code inspection, editing, and development commands with screenshots, mouse and keyboard input, and application controls.A finish action submits the final codebase for verification.Source: §2 · pp. 3–4
  • Patch correctness and trajectory validity are checked separately, and the reported task outcome multiplies their binary scores.The terminal repair reward is the verifier’s correctness result, which supplies the post-training signal.Source: §2 · p. 4
  • The policy selects both coding and computer-use actions to maximize expected terminal repair reward.Because the reward is binary, the objective maximizes the probability that the final codebase passes verification.Source: §2 · p. 4

3 The CUA-SWE benchmark

CUA-SWE builds tasks around visually observable software behavior and independently verified repairs, then varies task demands to examine agent capabilities. Its benchmark spans four domains and validates both the repair criteria and the interactive development path.

3 The CUA-SWE benchmark

  • Task construction exposes relevant runtime evidence, judges repair correctness independently of the agent trajectory, and supports controlled variations in task demands.LLM-assisted tasks are admitted only after human review and executable validation of requirements, runtime behavior, reference solutions, and verifiers.Source: §3 · p. 4

3.1 Tasks and domain coverage

  • The benchmark contains 105 tasks: 36 Web, 29 Game, 20 DevOps, and 20 Mobile, with domains linking runtime observations to different implementation demands.Tasks may be source-specified or application-material-dependent; both require diagnosis, implementation, and verification.Source: §3.1 · p. 5

3.2 Task construction and validation

  • Task authors start from reproducible runtime behavior and create instructions, reference repairs, negative repairs, and protected tests, followed by human review and validation.Verifiers check requested behavior, preserved functionality, and permitted modifications; they must reject plausible partial fixes, while interactive-path validation checks observable evidence and reproducibility.Source: §3.2 · pp. 5–6

3.3 Capability-oriented task expansion

  • Construction-time agent trials calibrate difficulty and inform related task variants, while each released variant receives its own review and executable validation.Task families vary particular demands while retaining application and repair lineage, and released conditions are fixed before evaluation.Source: §3.3 · pp. 6–7

4 Experiments

The experiments compare code-only and hybrid computer-use access across four software-engineering domains, examining task success, development behavior, and supervised post-training. They assess how agents use visual runtime evidence to diagnose and repair software and recover requirements available through the running application.

4.1 Experimental setup

  • Both access conditions allow source edits, commands, and output inspection, while hybrid CUA adds screenshots and graphical interaction with the running application.Task success combines patch correctness with evidence that the agent followed the assigned access and observation rules; the four-domain mean weights domains equally.Source: §4.1 · p. 7

4.2 Frontier performance and computer-use access

  • 59.9% mean task success, 17.7 percentage points above GPT-5.6-Sol, gave GPT-6-Astra the four-domain lead and highest hybrid success in every domain.Its advantage spans graphical editing, game state, service behavior, and mobile application integration.Source: §4.2 · p. 7
  • 48.6 percentage points: GPT-6-Astra's aggregate task-success gain with hybrid CUA; every frontier model improved, with gains concentrated in material-dependent tasks.Source-specified subsets had near-zero domain-average changes and substantial variation across models.Source: §4.2 · pp. 7–8

4.3 Development behavior and reliability

  • In the nested-selection task, GPT-6-Astra used feedback after resizing to detect and repair a connected selection defect that GPT-5.6-Sol and Claude-Opus-5 left unresolved.It dragged the object beyond its frame, observed lost selection, extended hit testing to selectable descendants, and repeated the interaction.Source: §4.3 · p. 8
  • Across Mobile gear, sprite animation, and DevOps tasks, GPT-6-Astra recovered relationships or operating rules and implemented their consequences across application interactions.Examples include reconstructing 23 gear-driving relations and probing endpoint behavior to update an inferred deployment mode.Source: §4.3 · p. 8
  • GPT-6-Astra averaged 4.3 minutes per attempt versus GPT-5.6-Sol's 3.7, yet used fewer responses on 83.3% of shared Web successes with Grok-4.6 and 72.2% with Claude-Opus-5.Figure 6(a) pairs success rates with mean episode times; Figure 6(b) compares completed responses on shared Web tasks.Source: Figure 6 · p. 8, §4.3 · p. 9
  • 34.5% of Game tasks were solved by GPT-6-Astra in all three attempts, versus 3.4% for GPT-5.6-Sol, distinguishing repeated success from coverage added by retries.Later attempts also added a sprite repair for GPT-5.6-Sol and a schedule repair for GPT-6-Astra on Mobile.Source: §4.3 · p. 9

4.4 Supervised post-training

  • 11.1% to 33.3%: Qwen3.8-27B's native-verifier pass rate on Web tasks after SFT on verified reference-replay action prefixes.The study reports patch correctness, while benchmark task success additionally checks trajectory validity; scalable RLVR training remains future work.Source: §4.4 · p. 9, Figure 7 · p. 9

5 Related work

Related work covers coding benchmarks, computer-use benchmarks, and efforts to combine coding with graphical interaction. CUA-SWE places development across four domains within that combined setting and provides deterministic correctness criteria.

Software engineering and coding benchmarks

Computer-use benchmarks and agents

Connecting coding and computer use

  • Programming with Pixels combines visual IDE interaction, coding tools, and verification, while WeaveBench evaluates hybrid GUI, CLI, and coding workflows.CUA-SWE focuses on four-domain software engineering and deterministic correctness criteria for evaluation and post-training feedback.Source: §5 · p. 9

6 Limitations

The authors identify current domain coverage and scalable reinforcement learning with verifiable rewards as areas for future development.

Coverage and future extensions

  • The benchmark currently spans Web, Game, DevOps, and Mobile; expanding to more domains, applications, and tasks would broaden its coverage of visual software engineering.The authors also identify richer workflows and longer tasks spanning multiple tools and applications as possible extensions.Source: §6 · p. 10

Learning signal and RLVR

  • Whether CUA-SWE provides a verifiable learning signal and how to develop scalable RLVR training remain open; efficient parallel rollouts, stable optimization, and credit assignment are needed.The initial SFT study offers early positive indications of a learning signal from gold-patch replays.Source: §6 · p. 10

7 Conclusion

CUA-SWE integrates code editing and execution with visual observations, graphical interaction, and deterministic verification across four software-engineering domains. The paper uses it to evaluate how agents apply visual runtime evidence to produce verified software changes.

Contribution

  • CUA-SWE is a benchmark, environment, and evaluation pipeline for visual software engineering across Web, Game, DevOps, and Mobile.It connects code editing and execution with screenshots, GUI actions, and deterministic verification.Source: §7 · p. 10

Evaluation

  • The evaluation compares code-only and hybrid CUA agents to study diagnosis, repair, and recovery of requirements available only through application interfaces.The benchmark supports measuring agents' ability to translate visual evidence into verified software changes.Source: §7 · p. 10

Verification

  • Each task uses deterministic tests to check whether the resulting software meets requirements and preserves specified behavior.Source: §7 · p. 10

AI use statement

AI tools assisted across the writing and research process, while the authors retained responsibility for the manuscript’s claims and evidence.

AI use statement

  • AI tools assisted with writing and literature searches.They also supported code inspection and conceptual and experimental design.Source: AI use statement · p. 10
  • AI tools assisted with result interpretation and figure and table preparation.An image-generation model also assisted with figure layout studies.Source: AI use statement · p. 10
  • The authors are responsible for the manuscript’s claims, citations, and experimental evidence.Source: AI use statement · p. 10

Reproducibility statement

The paper points readers to sections and appendices documenting the environment, tasks, tests, evaluation, workload accounting, and post-training.

Reproducibility statement

  • Sections 2 and 3 describe the environment, tasks, and tests, while Section 2 specifies the problem formulation.Source: Reproducibility statement · p. 10
  • Appendices document evaluation protocols, condition comparisons, workload accounting, repeated-attempt metrics, statistical resampling, task implementations, and task listings.Appendix A covers protocols and workload accounting; Appendices B, D, and F cover implementation and examples, tasks, and repeated-attempt methods.Source: Reproducibility statement · p. 10
  • Section 4.4 reports supervised post-training, with appendices describing SFT data, training configuration, and RLVR extensions.Figure 7 reports SFT loss history and training-task verifier outcomes for base and trained checkpoints.Source: Reproducibility statement · p. 10

Ethics statement

The evaluation combines controlled software changes, permitted agent interactions, and behavior-based checks, with procedures for constructing and testing task variants. The paper also identifies limits on what benchmark success establishes.

Ethics statement

  • Task materials include adapted or original applications, but passing benchmark tests does not guarantee deployment safety, and the study does not measure human performance.Source and asset attribution must be preserved when distributing task materials.Source: Ethics statement · p. 10

A.1 Patch correctness and trajectory validation

  • Patch correctness checks requested functionality, preserved behavior, and permitted changes, while trajectory validation checks interface use and required observation evidence.Web and Mobile require screenshot observations for CUA success; DevOps verifies screenshot use through execution records, while Game records it as a trajectory measure.Source: §A.1 · p. 14

A.2 Agent capabilities and access rules

  • Code-only agents receive source-level tools and command feedback, whereas hybrid CUA adds screenshots and graphical actions while excluding DOM, accessibility-tree, and browser-script access.Both conditions receive the same task instruction and protected tests; Mobile’s imported product materials are inspectable through the hybrid application interface.Source: §A.2 · p. 14

A.3 What visual runtime evidence contributes

  • Visual runtime evidence exposes displayed information and interaction states, and M labels tasks whose specification partly depends on application materials or bundled graphical references.M includes received gear drawings and live service contracts; five Boxel tasks receive M because their restart rule is on a graphical card not displayed by the code-only interface.Source: §A.3 · pp. 14–15

A.4 Condition comparisons

  • Condition comparisons pair the same tasks, while repeated-attempt pass@k counts tasks solved in any of the first k executions.Four-domain comparisons use equally weighted domains; Game’s repeated-attempt rounds use the same 29 task definitions and protected verifiers.Source: §A.4 · p. 15, §A.5 · p. 15

A.6 Execution and workload accounting

  • Workload reporting tracks episode time, steps, screenshots, GUI actions, and tokens, with four-domain summaries equally weighting domain-level episode measurements.Agent minutes per success divides mean episode time by full-inventory success probability.Source: §A.6 · pp. 15–16

A.7 Task sources and adaptations

  • Tasks draw on documented behaviors and adapted or purpose-built applications, with lineage records preserving sources, revisions, initial conditions, triggers, and outcomes.The four domains cover distinct task families, and controlled variants extend difficulty within a family.Source: §A.7 · p. 16

A.8 Seeded task construction and difficulty expansion

  • Task construction uses LLM-authored seeds, human review, executable checks, and reviewed agent trials to validate tasks and expand difficulty ladders.Variants change one declared demand while retaining the application and behavioral requirement; adjacent successes and reviewed failures locate a capability boundary for the tested agent and budget.Source: §A.8 · pp. 17–18, Figure 8 · p. 17

C Development examples

The examples trace how agents inspect running software, revise code, and check behavior, while independent verifiers distinguish complete repairs from attempts that still leave defects. The appendices describe task coverage, supervised replay training, and a proposed verifiable-reward extension whose completed experiment is supervised fine-tuning.

C Development examples

  • The examples contrast a successful code-only 2048 repair and verified chart and Threadloop reference repairs with interactive traces that exposed unfinished work.The 2048 attempt passed without browser observation; the chart replay showed the ring fitting its card, and Threadloop tests checked reveal-or-preserve behavior and Quill runtime use.Source: C Development examples · p. 19
  • Vector Relay’s repair transferred or canceled scheduled events according to the change in event ownership, and the independent evaluator recorded a pass.The browser replay showed “Circuit complete” and a score of 500; source checks covered cancellation, settlement after two transfers, and repeated contact.Source: C Development examples · p. 20
  • Hextris required further inspection and retries after the first replay remained at nine; the later cascade reached 27 and passed evaluation, while other traces expose incomplete repairs.Letter Arc passed 24 browser checkpoints despite no recorded successful replay of its final patch; Captain Callisto and Flappy Bird failed, Core Ball’s custom replay passed while independent tests rejected defects, and recipient browser interactions still showed failure.Source: C Development examples · pp. 19–21

D Complete task list

  • The 105 tasks comprise 36 Web, 29 Game, 20 DevOps, and 20 Mobile tasks, labeled by specification basis as application-material dependent (M) or source-specified (S).Source: D Complete task list · p. 21

E.1 SFT data construction across domains

  • Verified reference replays generate action-prefix training examples, with supervision applied to the final assistant action and preceding messages, observations, and screenshots used as context.Each accepted replay yields five cumulative prefixes; N accepted replays produce 5N prefix positions and 44N weighted training rows.Source: §E.1 · pp. 23–24

E.2 Supervised fine-tuning

  • 33.3% versus 11.1% native-verifier pass rate for the SFT checkpoint and base model, respectively.Both checkpoints were evaluated on the same tasks with one greedy attempt per task; these figures measure patch correctness, not task success including trajectory compliance.Source: §E.2 · p. 24

E.3 Verifiable rewards and the trajectory-level GRPO objective

  • The RLVR extension scores complete development episodes with verifier-based repair rewards and group-relative advantages, while observations condition the policy without prediction loss.Groups in which all repairs succeed or all fail receive zero advantage; the completed post-training experiment reported in the paper is SFT.Source: §E.3 · pp. 24–25

E.4 Rollout and update interface

  • The rollout interface runs independent development episodes in fresh workspaces, but scaling RLVR to long hybrid-CUA trajectories remains future work.The authors identify rollout throughput, optimization stability, and credit assignment as challenges; the reported completed experiment is supervised fine-tuning.Source: §E.4 · p. 25, Figure 11 · p. 26

F Repeated-attempt evaluation

Repeated attempts expand task coverage while distinguishing reliable competence from successes reached on only some trajectories. The evaluation defines cumulative pass@k and uses paired resampling to assess uncertainty in Web and Game comparisons.

F Repeated-attempt evaluation

  • 75.0% Web, 62.1% Game, 90.0% DevOps, and 60.0% Mobile are GPT-6-Astra’s pass@3 rates after three attempts.On Game, its cumulative pass@1, pass@2, and pass@3 are 37.9%, 58.6%, and 62.1%, respectively.Source: F Repeated-attempt evaluation · p. 27, Table 4 · p. 27
  • 34.5% of Game tasks are solved by GPT-6-Astra in all three attempts, compared with 3.4% for GPT-5.6-Sol.Ten of GPT-6-Astra’s 18 solved tasks succeed every time; the other eight succeed in one or two attempts.Source: F Repeated-attempt evaluation · p. 27, Table 6 · p. 28
  • An additional attempt lets GPT-5.6-Sol complete the Mobile sprite task and GPT-6-Astra one schedule variant, while the other schedule variants remain unsolved.The repeated-evaluation distinction is between consistently completed tasks and tasks reached through another development trajectory.Source: F Repeated-attempt evaluation · p. 27

F.2 Task-sampling uncertainty

  • 31.0 points is GPT-6-Astra’s Game pass@3 advantage over GPT-5.6-Sol, with a 95% task-bootstrap interval of [13.8, 48.3].Resampling whole game families gives [11.5, 50.0], preserving the same ordering under variation in source-game composition.Source: F Repeated-attempt evaluation · p. 27, Table 7 · p. 29

F.1 Metrics and aggregation

  • Pass@k is the percentage of tasks solved at least once in the first k attempts; pass@1 matches the single-attempt comparison and pass@3 covers all three attempts.The reported profile uses three attempts per task, with Web and Game task-level outcomes available in recorded order.Source: §F.1 · p. 27

F.2 Task-sampling uncertainty

  • Task bootstrap intervals resample paired task outcomes, while family-level sensitivity resamples whole Web task families or Game source games and retains task weighting.The reported intervals quantify variation under resampling of evaluated tasks or benchmark-family composition.Source: §F.2 · p. 29, Table 7 · p. 29

G Domain-level CUA performance

Across four equally weighted domains, GPT-6-Astra has the highest mean CUA success rate and leads in each individual domain. Results also show task-family-specific strengths and complementary capabilities among agents.

G Domain-level CUA performance

  • 59.9% is GPT-6-Astra’s mean CUA task success across four equally weighted domains, ahead of GPT-5.6-Sol at 42.2%, Grok-4.6 and Codex with GPT-5.6-Sol at 41.9%, and Claude-Fable-5 at 39.9%.Table 8 reports the complete domain-level results.Source: G Domain-level CUA performance · p. 30, Table 8 · p. 30
  • GPT-6-Astra has the highest observed CUA success in each domain: 66.7% Web, 37.9% Game, 80.0% DevOps, and 55.0% Mobile.Its Web advantage over GPT-5.6-Sol is concentrated in Canvas interactions after geometry changes; Mobile results distinguish precise gear relationships and sprite-pose assignments. Across Game, the union of successful CUA repairs covers 58.6% of tasks.Source: G Domain-level CUA performance · p. 30, Figure 15 · p. 34

H Task-family performance

  • Individual tasks reveal complementary strengths: Claude-Opus-4.8 uniquely solves an alert-routing task, while Grok-4.6 and Codex with GPT-5.6-Sol solve a log-normalization task that GPT-6-Astra misses.These task-family comparisons expose different demands on identifying which application facts determine a correct repair.Source: G Domain-level CUA performance · p. 30, Figure 15 · p. 34

G Domain-level CUA performance

  • Codex with GPT-5.6-Sol scores 55.6% Web, 17.2% Game, and 55.0% DevOps; Claude Code with Claude-Opus-5 scores 55.6%, 20.7%, and 65.0% in those domains.Compared with corresponding frontier models, Codex is 6.9 percentage points higher on Game and Claude Code is 5.0 points higher on DevOps; these results combine model, tools, and execution strategy.Source: G Domain-level CUA performance · p. 30, Figure 13 · p. 32

I Task-level outcomes and paired differences

Hybrid CUA adds screenshots and graphical interaction to source-level execution and command feedback. Its largest reported advantages occur on tasks requiring application materials, while successful trajectories show agents using visual observations to guide code changes and verify repairs.

I Task-level outcomes and paired differences

  • Additional successes connect application access to implementation requirements such as received drawings and displayed service behavior, and to testing changed behavior such as game restart state or Web selection after resizing.In the Web nested-selection trajectory, a drag after resizing reveals a second interaction defect; the paired-outcome figure partitions matched tasks by code-only and CUA outcomes.Source: I Task-level outcomes and paired… · p. 35, Figure 16 · p. 36, Figure 20 · p. 40
  • Web success rises from 19.4% to 58.3% for GPT-5.6-Sol and from 22.2% to 55.6% for Claude-Opus-5 under hybrid access; Game success rises from 3.4% to 20.7% for Claude-Opus-4.8 and 17.2% to 37.9% for GPT-6-Astra.The paired comparison holds task membership constant within each model and domain.Source: I Task-level outcomes and paired… · p. 35, Figure 20 · p. 40
  • On DevOps, nine frontier models average 50.0% hybrid success versus 0.6% code-only; on Mobile, GPT-5.6-Sol scores 0.0% versus 45.0% and GPT-6-Astra 0.0% versus 55.0%.The tasks involve inspecting service interfaces or visual materials, exploring behavior through graphical controls, and translating observations into code changes.Source: I Task-level outcomes and paired… · p. 35, Figure 18 · p. 38, Figure 19 · p. 39
  • On application-material-dependent tasks, mean hybrid advantages across the eight models are 42.0 points Web, 23.6 Game, 48.1 DevOps, and 38.8 Mobile; on source-specified tasks, differences are 0.0 Web and −0.6 Game.Application-material-dependent tasks account for 14 Web tasks, 9 Game tasks, and all 20 tasks in each of DevOps and Mobile; rates compare the same tasks across access conditions.Source: I Task-level outcomes and paired… · p. 35, Figure 20 · p. 40, Table 9 · p. 41

J Interaction workloads

  • GPT-6-Astra has 59.9% four-domain success and the lowest agent time per success among eight frontier models, at 7.2 minutes; its hybrid episodes average 46.3 responses and 145.8 output tokens per response.GPT-5.6-Sol uses 37.7 responses and 203.7 tokens per response, with 8.7 agent minutes per success.Source: J Interaction workloads · p. 43, Figure 21 · p. 43

Effort on shared successful tasks

  • On 18 shared successful Web tasks, GPT-6-Astra uses fewer responses on 83.3% of tasks shared with Grok-4.6 and 72.2% shared with Claude-Opus-5.Median response counts are 28.5 versus 43.5 and 21.5 versus 27.5, respectively; each comparison uses its own shared-success set.Source: J Interaction workloads · p. 43, Figure 23 · p. 46, Figure 24 · p. 47

M Successful repair trajectories

  • The Web repair successively fixes resizing, tests a drag beyond the containing frame, extends descendant hit testing, and passes the task evaluator.The Game repair combines lift-phase recovery, camera restoration in course coordinates, and support-relative player attachment, verified by deterministic state replay.Source: M Successful repair trajectories · pp. 51–52, Figure 28 · p. 51, Figure 29 · p. 52
  • The DevOps repair infers endpoint update modes from a pulse–pulse–settle sequence, while Mobile repairs translate visual structures into 23 gear relations and 28 approved sprite-pose assignments.The gauge repair resets modes each epoch and preserves receipt order; the Mobile repairs implement associated phase, angle, editing, timing, and transport behavior.Source: M Successful repair trajectories · pp. 53–54, Figure 30 · p. 53, Figure 31 · p. 54

N Patch correctness and task success

The trajectory analysis separates repair correctness from task success and finds final-edit GUI re-verification associated with higher success, while interaction volume alone does not consistently track outcomes.

N Patch correctness and task success

O.1–O.5 Coding, computer-use, and hybrid benchmarks

  • Prior benchmarks separately evaluate repository engineering, computer-use grounding, visual development, or hybrid interfaces; CUA-SWE ties interaction to code changes and executable outcomes.Its related-work discussion distinguishes evaluation of user-task completion or interface grounding from software changes verified against task requirements.Source: §O.1 · p. 58, §O.2 · p. 58, §O.3 · p. 58

P.1 Evaluation sample and annotation procedure

  • The trajectory review sampled up to two failures and one success per model–domain–condition group, annotated five overlapping failure stages, and retained unknown labels.It reviewed 214 trajectories from a larger collection of 2,040 attempts; outcome associations weighted each attempt by its stratum’s population-to-sample ratio.Source: §P.1 · p. 60, §P.2 · p. 60

P.2 Failure stages across models and domains

  • Among 84 sampled hybrid failures, implementation errors or incomplete repairs appeared in 61, observation gaps in 59, and verification gaps in 82.Verification gaps also appeared in 36 of 41 sampled hybrid successes, indicating incomplete coverage beyond a repaired interaction.Source: §P.2 · p. 60, Table 12 · p. 60

P.3 Observation, editing, and GUI re-verification

  • 45.8% vs. 5.7% Web task success, 16.6% vs. 12.8% Game, 68.1% vs. 5.9% DevOps, and 83.7% vs. 10.6% Mobile were associated with observed versus absent final-edit GUI re-verification.These are descriptive associations among edited hybrid trajectories; task difficulty, model, and trajectory validation may also contribute, and the small outcome strata do not support significance tests.Source: §P.3 · p. 63, Table 15 · p. 63

P.4 Interaction counts for successful and unsuccessful repairs

  • More interaction did not uniformly accompany success: GPT-6-Astra Web successes had medians of 7.5 image deliveries and 6 GUI actions, versus 19 and 17.5 for failures.Observed image counts are lower bounds for 80 streams lacking a recognized terminal marker.Source: Table 16 · p. 64, §P.4 · p. 65

P.5 Patch correctness and paired task outcomes

  • Across 101 paired tasks in three model–domain groups with lower hybrid task success, passing patches rose from 9 to 26 while task successes fell from 9 to 3.The decomposition identifies 23 hybrid passing patches with invalid trajectory validation; losses included incorrect patches and passing patches with invalid tool use.Source: §P.5 · p. 65, Table 17 · p. 65

P.6 Annotation consistency

  • Reviewer agreement was 100% for acquiring runtime information and observing relevant GUI behavior, but lower for interpretation and diagnosis, so stage labels are exploratory.Agreement was 90.6% for observation before the first edit and 87.5% for both GUI re-verification measures; adjudication changed 17 of 110 differing fields.Source: §P.6 · p. 66, Table 18 · p. 66
Loading 2609.32600v1…