PaperlayerView exact revision on arXiv

Source-linked AI summary

OS-Helix: Unifying GUI Interaction and Code Execution in Native Computer-Using Agents

Bowen Yang, Kaiming Jin, Zichen Ding, Zhenyu Wu, Zehao Li, Zhoumianze Liu, Kanzhi Cheng, Fangzhi Xu, Yanan Sun, Qiushi Sun, Yu Qiao

arXiv:2610.22234v1cs.LGcs.AIcs.CV

TL;DR

GUI and code have complementary roles in desktop automation, but simply adding code execution does not teach an agent how to route between them. OS-Helix trains a unified policy using synthesized executable desktop tasks, hybrid SFT, and online RL; on OSWorld-Verified, its RL model reaches 66.44% success, a 42.9% improvement over its matched baseline.

  • Problem

    GUI and code offer complementary capabilities, but prompt-level tool composition does not teach a single policy when to route between them and can degrade GUI performance.

    Source: §1 · p. 2

  • Method

    OS-Helix trains a unified GUI/code policy using executable desktop task synthesis, filtered hybrid SFT, and online agentic RL with Step-balanced GSPO.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    On OSWorld-Verified, OS-Helix-27B-RL improves 42.9% over its matched baseline to reach 66.44% success.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The reported analyses associate learned modality routing with fewer interactions and broader task coverage, while code access alone does not reproduce that effect.

    Source: §5 · p. 9

    The isolated experimental setting does not resolve deployment-time safety and privacy risks from screen access, local-file modification, code execution, or prompt injection.

    Source: ETHICS STATEMENT · p. 12

Abstract

from arXiv · show

Real desktop automation is neither purely visual nor purely programmatic. Agents must understand user-facing interfaces while efficiently manipulating files, scripts, and system state. However, native GUI/code coordination cannot be obtained by prompt-level tool composition. In our preliminary experiments, naively adding a code tool degrades a GUI agent, revealing that GUI/code routing must be internalized through training on high-quality hybrid data. We present OS-Helix, an end-to-end trained multimodal computer-using agent that unifies GUI actions and code execution in one action space. To construct the required supervision, we introduce application-artifact composition, pairing desktop applications with authentic file artifacts to instantiate executable Ubuntu states, and a decoupled instruction/evaluator generation pipeline that produces environment-grounded tasks whose final states can be verified regardless of the solution modality. We cold-start the policy with SFT on filtered hybrid data and further improve it through online RL in live desktop environments. For RL, we propose Step-balanced GSPO, which aggregates importance weight by action step rather than raw assistant tokens to prevent long code actions from dominating heterogeneous-action optimization. On OSWorld-Verified, OS-Helix-27B and OS-Helix-35B-A3B achieve relative improvements of 35.0% and 33.7% over their matched Qwen3.5 hybrid baselines, while OS-Helix-27B-RL further improves by 42.9% to reach 66.44% success. Ablations provide evidence consistent with contributions from learned GUI/code coordination, scalable verifiable supervision, and action step balanced optimization, while demonstrating cross-platform transfer on WindowsAgentArena. Our project homepage is at https://os-copilot.github.io/OS-Helix/.

1 INTRODUCTION

GUI interaction and code execution offer complementary strengths, but a single agent does not learn when to route between them simply by adding a code tool. OS-Helix trains one policy to coordinate both modalities using executable desktop tasks, hybrid data, and online RL.

1 INTRODUCTION

  • GUI offers visual access to arbitrary applications, while code enables efficient, precise manipulation of programmatic state.Source: §1 · p. 2
  • Naively exposing code execution reduced Qwen3-VL-32B-Instruct’s OSWorld-Verified success from over ∼34% with GUI actions alone to roughly ∼32%.Source: §1 · p. 2
  • OS-Helix unifies GUI interaction and code execution within a single end-to-end trained policy.Source: §1 · p. 2
  • Native GUI/code hybrid computer use is framed as an end-to-end policy-learning problem; controlled experiments show that merely exposing a GUI policy to code execution fails to yield reliable routing, motivating hybrid training.Qwen3-VL-32B-Instruct achieves over ∼34% success on OSWorld-Verified with GUI actions alone, but falls to roughly ∼32% when code execution is naively exposed.Source: §1 · p. 2
  • Application–artifact composition and decoupled instruction/evaluator synthesis create reproducible executable desktop states and high-quality hybrid training data.Source: §1 · p. 2
  • Step-balanced GSPO aggregates importance ratios over action steps rather than raw assistant tokens to prevent long code actions from dominating optimization.Source: §1 · p. 2
  • The released collection includes approximately 9K generated instructions, 83K unfiltered SFT samples, and about 31K successful filtered samples.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work explores hybrid control, desktop task synthesis, and interactive-agent training, but leaves challenges in learning native GUI/code routing and scaling reproducible, verifiable desktop environments. OS-Helix combines executable state construction, task synthesis, hybrid SFT, and agentic RL with step-balanced optimization.

Hybrid GUI/Code Computer-Using Agents

  • Framework, tool-call, and model-fusion approaches combine GUI and code through external orchestration, predefined interfaces, or predefined designs rather than coordination learned within one policy.Source: §2 · p. 3

Environment Scaling and Task Synthesis for Computer-Using Agents

  • Existing task-synthesis pipelines vary in how they scale applications, authentic artifacts, executable initial states, and verifiable terminal conditions.Source: §2 · p. 3

Training Paradigms for Computer-Using Agents

  • CUA training spans SFT, offline preference optimization, grounding RL, and agentic RL, while stable native GUI/code routing across heterogeneous action granularities remains underexplored.Source: §2 · pp. 3–4
  • OS-Helix’s pipeline constructs executable desktop environments, synthesizes verifiable tasks, cold-starts with teacher-generated hybrid trajectories, and applies online RL in live sandboxes.Source: Figure 2 · p. 4

3 OS-HELIX

OS-Helix trains GUI interaction and code execution as complementary ways to change the same digital state, with environments, tasks, trajectories, and reinforcement signals designed for hybrid execution. Its pipeline composes executable desktop states, synthesizes verifiable tasks, cold-starts the policy with filtered SFT trajectories, and refines it through online RL.

3.1 OVERVIEW

  • OS-Helix builds a unified pipeline for hybrid GUI/code execution rather than merely adding a shell tool to a GUI model.The policy selects actions from a shared action space; GUI actions, code actions, and episode-level metaactions are supported.Source: §3.1 · p. 4

3.2.1 APPLICATION–ARTIFACT COMPOSITION FOR DESKTOP ENVIRONMENTS

  • Authentic artifacts expand executable desktop-state diversity by linking file formats to compatible applications and persistent local state.Application configurations record supported input types, versions, launch commands, and initialization settings; resulting states run in reproducible Ubuntu virtual machines.Source: §3.2 · p. 5, §3.2.1 · p. 5

3.2.2 ENVIRONMENT-GROUNDED TASK SYNTHESIS

  • Task synthesis samples an application and compatible artifacts, then explores the instantiated environment to propose realistic instructions and observable success conditions.A global application memory tracks previously exercised functions to encourage broader coverage over time.Source: §3.2.2 · p. 5
  • Verification specifies terminal states without prescribing whether tasks are completed through GUI, code, or their combination.An independent evaluator is synthesized after instruction acceptance; rule-based checks handle programmatically observable conditions, while VLM-based evaluation handles visual conditions that cannot be reliably recovered programmatically.Source: §3.2.2 · pp. 5–6

3.3.1 SFT COLD START

  • Filtered SFT trajectories cold-start the policy by teaching unified action semantics and a coarse GUI/code routing policy.Teacher trajectories are evaluator-scored for rejection sampling, then filtered for malformed calls, schema mismatches, and evaluator or quality failures.Source: §3.3.1 · p. 6

3.3.2 AGENTIC RL WITH STEP-BALANCED GSPO

  • Online RL uses live desktop rollouts and evaluator scores as the sole outcome reward, assigning zero reward to invalid action-format responses.An asynchronous pipeline, capability-frontier task sampling, and bounded sliding-window visual-context segmentation help manage the cost of desktop interaction.Source: §3.3.2 · p. 6
  • Step-balanced GSPO aggregates importance weights by action step rather than token count, with equal weights for GUI and code steps in this setting.This addresses standard GSPO's tendency to let long code actions dominate sample-level importance ratios and clipping decisions; the change does not generally equalize full parameter-gradient norms or variances across action lengths.Source: §3.3.2 · pp. 6–7

4 EXPERIMENTS

OS-Helix is evaluated on desktop benchmarks and compared with matched-backbone and other agent baselines. Hybrid training improves benchmark results, while RL and ablations examine where further gains arise and how learned coordination and action-step balancing contribute.

4.1 EXPERIMENTAL SETUP

  • The experiments use 361 OSWorld-Verified tasks across five Ubuntu domains, report mean success, and also evaluate WindowsAgentArena for Windows transfer.Episodes are capped at 50 interaction steps; OSWorld is the paper's name for OSWorld-Verified.Source: §4.1 · pp. 7–8

4.2 MAIN RESULTS

  • 16.25 and 14.43 absolute points: OS-Helix-27B and OS-Helix-35B-A3B improve over their matched Hybrid baselines, with the largest gains in Office and Work.Office gains are +33.3 and +29.1 points, and Work. gains are +16.0 and +14.1 points, respectively.Source: §4.2 · p. 8
  • 23.05 points: OS-Helix-27B-RL improves over the Qwen3.5-27B GUI baseline on WindowsAgentArena, providing evidence of cross-platform transfer.In broader, non-controlled comparisons, it is competitive with Agent-S3 and OS-Symphony, while newer and substantially larger general models remain stronger overall.Source: Table 2 · p. 8, §4.2 · p. 8
  • 66.44%: OS-Helix-27B-RL raises average success from 62.75%, with gains concentrated in Professional, OS, and Daily rather than uniformly across domains.Office and Work. do not improve further over their already strong SFT results.Source: §4.2 · p. 8

4.3 ABLATION STUDY

  • Both teacher-derived SFT datasets outperform the unadapted Hybrid backbone; removing samples from nine OSWorld application domains, reducing the set from approximately 31K to 17K, shifts success from 62.75% to 62.31%.Kimi K2.5 and Claude Sonnet 4.6 trajectories were collected from the same 800 synthesized tasks. Retained performance after domain removal is consistent with capabilities that transfer beyond evaluated application domains.Source: §4.3 · pp. 8–9
  • SFT provides the main improvement over the unadapted Hybrid policy, while RL after SFT reaches 66.44%; standard GSPO lowers success to 60.47% and raises code-action rate to 40.0%.Step-balanced GSPO raises success while keeping the code-action rate at 26.1%, close to the SFT policy's 27.9%.Source: §4.3 · p. 9, Table 4 · p. 9
  • With code access and the 10K-sample budget fixed, hybrid SFT adds 3.05 points over GUI-only SFT, while the same policy retains 52.13% success with code disabled.Enabling code for that policy adds 10.62 points, supporting retained GUI competence alongside code complementarity.Source: §4.3 · p. 9, Table 5 · p. 9

5 DISCUSSION

OS-Helix improves execution efficiency and task coverage, while Step-balanced GSPO stabilizes code-action behavior and strengthens repeated-sampling coverage. Scaling and instruction-distribution analyses show gains as supervision grows and limited instruction-level overlap with OSWorld.

Accuracy versus efficiency

  • OS-Helix-27B uses fewer interactions while covering more successes under small step budgets across full, successful, and outcome-matched task sets.The matched subsets control for task-composition differences; code access alone at inference time does not reproduce the effect.Source: §5 · p. 9, Figure 3 · p. 10

Why step balance matters

  • Step-balanced GSPO keeps code-action frequency lower and more stable than standard GSPO, while OS-Helix-27B-RL raises Pass@1 to 66.44% and Pass@3 to 76.44%.Relative to SFT, the Pass@1 gain is 3.69 points and the Pass@3 gain is 5.56 points; the Pass@3 gain has a paired-bootstrap 95% interval of [2.24,8.97].Source: Figure 4 · p. 10, §5 · p. 10

Training scaling and data distributions

  • Performance improves overall as hybrid supervision grows and as the general-purpose multimodal-data fraction rises from 0% to 100%.Cross-corpus instruction similarity to OSWorld is comparable to OSWorld’s internal similarity, while the synthesized corpus has a higher but broadly spread internal-similarity distribution.Source: §5 · p. 10

6 CONCLUSION

OS-Helix combines executable desktop task synthesis, hybrid SFT, and Step-balanced GSPO online RL in one policy for GUI and code actions. On OSWorld-Verified, it outperforms matched Qwen3.5 hybrid baselines, and the RL model also transfers to WindowsAgentArena.

6 CONCLUSION

  • OS-Helix is trained end-to-end through scalable executable desktop task synthesis, hybrid SFT, and Step-balanced GSPO online RL.The policy is designed to coordinate heterogeneous GUI and code actions.Source: §6 · p. 10
  • 35.0% and 33.7% relative improvements over matched Qwen3.5 hybrid baselines are achieved by OS-Helix-27B and OS-Helix-35B-A3B, respectively; OS-Helix-27B-RL reaches 66.44% success after a 42.9% relative improvement.OS-Helix-27B-RL also achieves 63.87% success on WindowsAgentArena and surpasses GPT-5-based framework methods on both benchmarks.Source: §6 · p. 11
  • Ablations and behavioral analyses support contributions from hybrid supervision, learned routing, and step-balanced optimization, including broader successful-trajectory discovery with RL.The conclusion reports improved execution efficiency and task coverage beyond code access alone, and mitigation of standard GSPO’s drift toward code-intensive behavior.Source: §6 · p. 11

AI USE STATEMENT

The authors used LLMs for synthetic dataset generation, manuscript writing assistance, and preliminary mathematical-formula derivation, while retaining responsibility for the research and its presentation.

AI USE STATEMENT

  • LLMs assisted with synthetic datasets, manuscript drafting and refinement, and preliminary derivation of formulas in Appendix E.The authors reviewed and take responsibility for the research design, experiments, analysis, claims, and final presentation.Source: AI USE STATEMENT · p. 12

ETHICS STATEMENT

The work’s experiments use isolated Ubuntu virtual machines with fixed applications and curated local files, but this does not resolve deployment risks from GUI and code access. The authors recommend layered permissions, safeguards, and oversight for real-device use.

ETHICS STATEMENT

  • Experiments and evaluation run in controlled, isolated Ubuntu VMs with fixed application versions and a curated local file library separated from host files and credentials.The setup reduces the risk of exploratory actions modifying host state or accessing unauthorized resources.Source: ETHICS STATEMENT · p. 12
  • Real-device deployment can expose private or credential-bearing content and increase the impact of erroneous or malicious code workflows.The isolated experimental setting does not resolve these deployment-time risks.Source: ETHICS STATEMENT · p. 12
  • The authors recommend least-privilege permissions, confirmation for high-impact actions, sandboxing, audit logs, verification, and data minimization.They also recommend red teaming, interpretable action auditing, and reliable human takeover mechanisms for future CUA research.Source: ETHICS STATEMENT · p. 12

A LIMITATIONS AND FUTURE WORK

OS-Helix remains limited by desktop-environment coverage, imperfect verification, and constrained RL resources. The authors also identify benchmark and safety gaps, while proposing self-bootstrapping and hybrid-aware evaluation as future directions.

A LIMITATIONS AND FUTURE WORK

  • Desktop environments cannot cover the full diversity of real machines, private artifacts, application versions, and personalized configurations.The authors propose expanding the application and artifact ecosystem and broadening state and configuration randomization while preserving reproducibility.Source: A LIMITATIONS AND FUTURE WORK · p. 20
  • Automatic verification remains brittle: rule checks can depend on application versions or export formats, while VLM checks may miss fine visual discrepancies.Inconsistent strictness can score semantically near-correct work as zero; RL practice typically excludes VLM-based scores from rewards, constraining task diversity.Source: A LIMITATIONS AND FUTURE WORK · p. 20
  • RL training covered only the dense 27B model, used conservative settings and limited steps, and did not systematically investigate scaling behavior.Current resources also make stable RL training difficult for the 35B-A3B MoE model.Source: A LIMITATIONS AND FUTURE WORK · p. 20
  • The authors propose model-generated training trajectories as a future self-bootstrapping direction for agents with meaningful general computer-use capability.They also suggest adapting GDPval and Agents’ Last Exam to GUI settings, which they describe as naturally involving authentic artifacts and complex reasoning.Source: A LIMITATIONS AND FUTURE WORK · p. 20
  • GUI-centered benchmarks may miss valid hybrid solutions, and success rate alone does not measure modality efficiency or hybrid-execution safety risks.The authors call for benchmarks designed for hybrid use, with multi-path references and joint evaluation of correctness, efficiency, modality selection, and safety.Source: A LIMITATIONS AND FUTURE WORK · pp. 20–21

B.1 TASK SYNTHESIS COMPARISON

This material describes OS-Helix’s artifact-grounded task synthesis, evaluator design, and training-data preparation. It also documents current limits in cross-application task coverage and the representation of final states.

B.1 TASK SYNTHESIS COMPARISON

  • OS-Helix combines K/E/C grounding, scalable application configuration, authentic artifact-backed states, and executable evaluation; its compatibility graph organizes valid application–artifact pairings.The comparison covers task grounding, application scaling, authentic artifacts, cross-application support, and executable evaluation, rather than data scale or maturity.Source: §B.1 · p. 21
  • Cross-application synthesis remains a limited prototype; expanding and characterizing it is a stated direction for future work.The passage notes Qwen-CUA and Qwen-UI-Agent have more extensive coverage of long-horizon and cross-application workflows at scale.Source: §B.1 · p. 21

B.2 APPLICATIONS AND ARTIFACTS DETAILS

  • The Ubuntu infrastructure includes 45 applications, with artifact types and file formats covered by the pipeline; long-tail artifacts use lightweight sources rather than large-scale collection.These alternatives include curated examples, bundled demo assets, conversions, and manually constructed seed files.Source: §B.2 · pp. 21–22

B.3 FINAL STATE MATERIALIZATION

  • Final-state materialization externalizes heterogeneous goals into VM-resident files when semantics are preserved, but serialized files cannot represent every visual, transient, or application-internal condition.Evaluator choice follows observability: deterministic rules for inspectable states, VLM checks for essential visual evidence, or hybrid checks when the evidence is complementary.Source: §B.3 · p. 22

B.4 INSTRUCTION AND EVALUATOR SYNTHESIS DETAILS

  • The instruction pipeline explores initialized states and proposes goals with success conditions, while an independent Evaluator Agent turns frozen conditions into checks.Exploration can inspect interfaces without keyboard or free-form text input; the pipeline filters invalid proposals and repairs evaluator failures before removing a proposal.Source: §B.4 · pp. 23–25
  • Rule-based judges score fine-grained completion items from VM-resident artifacts or commands, while VLM checks cover cases requiring visual evidence; SFT filtering thresholds both separately, but RL uses rule-based tasks only.Near-duplicate instructions are removed, and the RL pool retains moderately difficult tasks after SFT-model rollouts.Source: §B.4 · p. 25

B.5 INSTRUCTION AND ACTION DISTRIBUTIONS

  • The data summary covers 8,880 generated tasks, 83,701 desktop SFT samples from 5,416 filtered tasks, and 40,729 tool actions in successful samples.The sample pool includes successful and failed trajectories plus three invalid-sample categories.Source: §B.5 · p. 26

B.6 GENERAL DATA MIXTURE

  • The SFT mixture combines approximately 31K filtered desktop samples with 40K general-purpose samples spanning GUI understanding, coding, mathematics, OCR, charts, and dialogue.The paper says the general-purpose data helps stabilize training and reduce over-specialization to desktop trajectories.Source: §B.6 · p. 26

C TRAINING DETAILS

The training details specify full-parameter SFT, Step-balanced GSPO for RL, and online rollout sampling, segmentation, and reward handling. RL sampling favors tasks near the current capability frontier, while rewards come from environment evaluators.

C.1 SFT CONFIGURATION

  • SFT uses full-parameter training for one epoch, a 32,768-token maximum sequence length, and a 1 × 10−5 learning rate.The setup also uses sample packing, bfloat16 precision, and tensor parallelism of 4.Source: §C.1 · p. 26

C.2 GSPO RL CONFIGURATION

  • RL uses Step-balanced GSPO with equal GUI and code step weights, 64 rollout samples per collection, and a maximum of 50 GUI interaction steps.The configuration disables KL divergence and samples eight trajectories for each of eight prompts per collection.Source: §C.2 · p. 27

C.3 ONLINE RL SAMPLING, SEGMENTATION, AND REWARDS

  • Task sampling favors tasks with estimated success rates near 0.5 and downweights tasks estimated as very easy or very difficult.The task success-rate estimates are updated using an exponential moving average, and the sampling weights use a Gaussian kernel with bandwidth 0.25.Source: §C.3 · pp. 27–28
  • Rollouts retain a 5–10 screenshot history; when it exceeds 10, the system saves a segment, drops the five oldest screenshots, and keeps the five newest.Only responses generated with 5–10 screenshots in the retained history contribute to loss.Source: §C.3 · p. 28
  • Rewards come only from the environment evaluator, and invalid tool-call or action-format responses receive zero reward even if later recovery could earn a positive task score.Valid trajectory and segment evaluator scores are real values in [0, 1].Source: §C.3 · p. 28

D.1 TRANSFER TO WINDOWSAGENTARENA

OS-Helix improves matched GUI baselines on WindowsAgentArena despite no Windows-specific training data, under a Python-only code-action adaptation. Additional analyses report slightly lower pure-grounding scores and identify command-layer and infeasible-task limitations.

D.1 TRANSFER TO WINDOWSAGENTARENA

  • Failed WindowsAgentArena trajectories show high-level strategies can still break at the system-command layer, including Bash-like patterns and brittle PowerShell-style logic.The authors say this pattern is compatible with the absence of Windows-specific command data despite Python-only prompt adaptation.Source: §D.1 · p. 28
  • Roughly 10% of OSWorld and WindowsAgentArena tasks are infeasible, and the synthesis pipeline does not naturally generate them, leaving the model liable to continue until its interaction budget expires.Some GUI-infeasible tasks may still be solvable by code but judged incorrect when the benchmark expects infeasibility recognition.Source: §D.1 · p. 29
  • On WindowsAgentArena, OS-Helix-27B lifts matched GUI 40.82% to 57.95% (+17.13 points), while RL reaches 63.87%, +23.05 over GUI and +7.46 over Hybrid.Code actions were restricted to Python, with system calls issued through Python when Cmd or PowerShell commands would otherwise be needed.Source: §D.1 · p. 28, Table 12 · p. 30

D.2.1 GROUNDING ABILITY CHECK

  • On OSWorld-G, Qwen3-VL-32B-Instruct scores 61.35 versus OS-Helix-32B at 57.21; on ScreenSpot-V2, they score 95.05 versus 93.47, respectively.The authors attribute the decrease to training for hybrid desktop execution, where code-capable actions can avoid brittle pixel-level interactions in end-to-end tasks.Source: Table 13 · p. 30, §D.2.1 · pp. 30–31

D.2 ADDITIONAL ANALYSIS

  • The additional diagnostic uses Qwen3-series backbones with the same data recipe as above and SFT only, so it does not test online RL’s contribution.The base and OS-Helix variants differ from the Qwen3.5-series backbones used in the preceding experiments.Source: §D.2 · p. 30

D.2.2 BACKBONE REPLACEMENT ANALYSIS

Backbone-replacement experiments show that SFT improves hybrid execution with both Instruct and Thinking backbones. Gains span multiple domains, though Thinking has one domain-level decline.

D.2.2 BACKBONE REPLACEMENT ANALYSIS

  • 50.56% (+17.88 points) for OS-Helix-32B over the Instruct Hybrid baseline, and 51.67% (+8.79 points) for OS-Helix-32B-Thinking over the Thinking Hybrid baseline.The unadapted Instruct Hybrid baseline falls below GUI-only performance, while the Thinking Hybrid baseline exceeds its GUI-only counterpart.Source: §D.2.2 · p. 31
  • Instruct gains span all five domains; Thinking gains span OS, Office, Daily, and Work., while Prof. decreases by 2.83 points.For Instruct, gains range from 12.24 points in Prof. to 19.76 points in Work.; for Thinking, the other four domain gains are positive.Source: §D.2.2 · p. 31
  • The authors interpret the gains as evidence for learned coordination and routing, rather than a benefit from merely adding a tool.The Instruct baseline slightly worsens when Code is exposed without hybrid training, while trained gains are strongest on domains requiring coordination across GUI observations, executable operations, and persistent task state.Source: §D.2.2 · p. 31

D.2.3 TRAINING RECIPE AND CONTEXT ANALYSIS

This section analyzes training choices and step-balanced GSPO alongside executable tasks verified component by component. It also presents case studies of GUI/code coordination across applications, including cross-modal verification or repair and evaluation through persistent application state and project metadata.

D.2.3 TRAINING RECIPE AND CONTEXT ANALYSIS

  • 50.56% falls to 37.74% without training CoT, 37.82% without inference CoT, 43.22% without historical CoT, and 47.06% with rejected failure data mixed in.The similar drops from removing training and inference CoT support its role as both a training target and an execution-time state representation; historical CoT removal is less damaging, while the smaller decline with failure data suggests data quality matters. Work-category success also falls from 38.54% to 28.87%, 31.43%, and 31.30% under those three CoT ablations, respectively.Source: §D.2.3 · pp. 31–32, Table 15 · p. 32

E STEP-BALANCED GSPO DETAILS

  • Standard GSPO aggregates token-level log-importance weights across a sample, so a long code action can dominate its ratio and clipping decision despite being one action step.The issue is that sample-level aggregation does not equalize action-step contributions within a multi-action sample.Source: E STEP-BALANCED GSPO DETAILS · p. 32, §E.1 · p. 33

E.1 GSPO

  • In standard GSPO, an action step’s implicit weight in the sample-level log-ratio is determined by its valid assistant-token count.Consequently, a longer code action can disproportionately affect the shared sample ratio and clipping decision.Source: §E.1 · p. 33

E.2 STEP-BALANCED GSPO

  • Standard GSPO weights action steps by token count, whereas Step-balanced GSPO uses explicit action-level weights and recovers standard GSPO when those weights are proportional to token counts.The method preserves GSPO’s token-level importance weights and clipping structure while changing sample-level aggregation.Source: E STEP-BALANCED GSPO DETAILS · p. 32, §E.2 · p. 34

E.3 GRADIENT CONTRIBUTION

  • Step-balanced GSPO removes token-count dependence from an action step’s aggregate scalar gradient coefficient while retaining token-level importance weights.This addresses one source of length-driven drift toward code actions, not a guarantee of correct modality routing; the full gradient also depends on token-specific directions, clipping, and reward correlations.Source: §E.3 · p. 34

F CASE STUDY

  • Two filtered synthesis tasks illustrate application–artifact composition, environment-grounded instructions, and reviewed rule-based evaluators that award partial terminal credit across independently verifiable components.The examples span an office/code workflow and a media workflow, with task metadata, instructions, and selected evaluator code provided.Source: F CASE STUDY · p. 35

F.1 SYNTHESIS TASK CASE STUDY 1: OFFICE AND CODE WORKFLOW

  • The LibreOffice Calc–VSCode task requires categorizing spreadsheet rows and creating a runnable Python audit script; its evaluator separately scores spreadsheet correctness, data-derived code, and executed output.The task is labeled complex and estimated at 30 interaction steps; its three terminal checks are averaged.Source: §F.1 · p. 35

F.2 SYNTHESIS TASK CASE STUDY 2: MEDIA WORKFLOW

  • The OBS Studio–Shotcut task transfers recording settings into a matching editing project and requires a two-line record of the resolution and frame rate.The task is labeled medium and estimated to require 22 interaction steps.Source: §F.2 · p. 37
  • The media-workflow evaluator checks OBS settings, Shotcut project compatibility, and a text handoff record, awarding partial credit for incomplete configurations.It scores persistent application state and project metadata rather than replaying a prescribed action sequence; Shotcut receives full credit for a matching 720p/30 profile, or partial credit when two of three core fields match.Source: §F.2 · p. 37

F.3 AGENT BEHAVIOR CASE STUDY: GUI–CODE COORDINATION

  • OS-Helix uses GUI-observed context to guide code execution and combines code-based spreadsheet filtering with GUI-based e-mail recipient entry.It infers a lecture-file URL pattern from an opened slide before downloading PDFs in a loop, and identifies unpaid tuition records in a spreadsheet before entering recipients in a mail client.Source: §F.3 · p. 38

F.4 AGENT BEHAVIOR CASE STUDY: CROSS-MODALITY VERIFICATION

  • Cross-modality checks let OS-Helix detect and correct mismatches, using GUI feedback to repair a code-generated chart and code inspection to validate GUI-edited slides.In the chart case, GUI observation reveals a missing COGS series before code repairs the visualization. In the slide case, code checks that the saved presentation contains the requested note and title formatting.Source: §F.4 · pp. 38–39

F.5 FAILURE ANALYSIS: EVALUATION COVERAGE AND CROSS-MODALITY HANDOFFS

This section examines evaluation coverage and GUI/code handoffs as hybrid-agent challenges, including task-grounded handoffs and cross-modal verification or repair in a representative trajectory subset. It also describes prompts that strengthen task and evaluator quality through explicit review criteria, actionable repair, and verification against the intended task state.

F.5 FAILURE ANALYSIS: EVALUATION COVERAGE AND CROSS-MODALITY HANDOFFS

  • In the Calc-to-Writer failure, code conversion lost alternating row shading, and the agent misread the GUI mismatch as missing currency symbols.The example exposes a specification–evaluation coverage gap: “preserve the original formatting” includes rendered attributes such as row banding, while code serialization requires formatting to be reconstructed explicitly.Source: §F.5 · p. 39, Figure 8 · p. 40
  • In the PowerPoint failure, a code-side save omitted the earlier GUI edit, and choosing “Don’t Save” discarded that update.Reliable execution requires reconciling GUI-resident and code-side artifact states before handoff and maintaining consistent save-scope and commit-status semantics.Source: §F.5 · pp. 39–40, Figure 8 · p. 40

F.6 TRAJECTORY AUDIT

  • 27/30 audited trajectories contained grounded GUI/code handoffs, including 22/30 with cross-modal verification or repair.The audited set included 21 successful, 4 partially successful, and 5 failed trajectories; four of the five failures transferred state correctly, while the remaining failure replaced a GUI-observed PDF source with an unverified URL. The audit covers a representative task subset and does not establish that routing decisions are optimal.Source: §F.6 · pp. 40–41

G.6 INSTRUCTION QUALITY REVIEW PROMPT

  • Instruction review rejects unrelated multi-application actions, scores realism, complexity, verification, and leakage, and ranks accepted proposals by rationality subject to quality gates.When multiple applications are genuinely required, the dependency chain must specify a verifiable source-to-operation-to-target relationship; rejections receive a reason and actionable repair.Source: §G.6 · p. 46

G.7 EVALUATOR QUALITY REVIEW PROMPT

  • Evaluator review checks coverage of task conditions, concrete VM getters, internal structure proving completion, and resistance to initial, partial, stale, or superficially similar states.A vm_file getter must identify a local cached path for rule code to open or parse, whereas vm_command_line returns stdout text.Source: §G.7 · p. 47
  • Evaluator checks prefer deterministic rules over VLM evidence when possible and avoid unsafe or nondeterministic behavior; material defects or failed preflight require repair.The review scores evaluator quality primarily, using coverage, getter correctness, structure validation, and false-positive resistance to explain the score.Source: §G.7 · p. 47

Related papers

Loading 2610.22234v1…