PaperlayerView exact revision on arXiv

Source-linked AI summary

Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields

Liya Zhu, Jingzhe Ding, Jian Zhang, Jianbo Xue, Shihao Liang, Ge Zhang, Yi Zhu, Duju Zeng, Xiang Gao, Qingshui Gu, Mailun Gao, Huimin Che, Yan Zhao, Peiheng Zhou, Haojun Wang, Chaobo Xian, Lili Le, Chi Wu, Yiwei Liu, Shengda Long, Jiale Yang, Fangzhi Xu, Sijin Wu, Haodong Duan, Chao He, Zhaojian Li, Minchao Wang, Huan Zhou, Jiani Hou, Chuqian Yu, Weiran Shi, Hongwan Gao, Jiamin Chen, Guanhong Chen, Tingqin Luo, Kaiyuan Zhang, Zhixin Yao, Qing Hua, Yuhao Jiang, Jin Chen, Pu Chen, Zhenyu Hu, Xingyu Li, Zhengxuan Jiang, Meng Cao, Tianfeng Long, Haozhe Wang, Mingzhang Wang, Yichen Zhang, Yiming Dai, Chenchen Zhang, Jiaying Wang, Xinying Liu, Xingzu Liu, Lingling Zhang, Xinjie Chen, Yujia Qin, Wangchunshu Zhou, Zhiyong Wu, Yang Liu, Jiaheng Liu, Lei Zhang, Shen Yan, Wenhao Huang, Zaiyuan Wang, Xiaolong Chang

arXiv:2606.11042v4cs.AI

TL;DR

Existing GUI benchmarks provide limited coverage of long-horizon professional workflows. Workflow-GYM evaluates GUI agents on such tasks, and experiments show that even the strongest model achieves a 30% success rate.

  • Problem

    Existing GUI benchmarks mainly cover short-horizon, general-purpose tasks, leaving long-horizon professional computer-use workflows rarely evaluated.

    Source: §1 · p. 2

  • Method

    Workflow-GYM benchmarks GUI agents on end-to-end, domain-specific workflows in realistic work environments, evaluating the final state or artifact.

    Source: §1 · p. 2

  • Results

    Long-horizon professional GUI workflows remain highly challenging; even the strongest model achieves a 30% success rate.

    Source: §1 · p. 3, §6 · p. 17

  • Takeaways & Limitations

    Workflow-GYM offers a testbed for advancing GUI agents toward more capable, reliable, and practically useful computer-use systems.

    Source: §6 · p. 17

    Most current GUI agents use snapshot-based observation, leaving intermediate execution states inaccessible for real-time monitoring or correction.

    Source: §5.3 · p. 14

Abstract

from arXiv · show

Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple applications, and short-horizon tasks, leaving it largely unknown whether modern agents can follow user instructions to autonomously operate domain-specific professional software and accomplish economically valuable work in an end-to-end manner. To bridge this gap, we introduce Workflow-GYM, a benchmark for long-horizon GUI tasks centered on professional domains and specialized software environments. Through extensive experiments on state-of-the-art models, we find that even the strongest models achieve only slightly above 30% success rates, highlighting that professional long-horizon GUI workflows remain highly challenging for current GUI agents. Further analysis reveals that current agents struggle to maintain long-horizon workflow consistency, frequently exhibiting workflow stage omission, error propagation, objective drift, and insufficient understanding of professional software environments. Our findings provide important insights into the limitations of current agent systems and suggest key directions for the next generation of GUI-agent research.

1 Introduction

Workflow-GYM evaluates whether GUI agents can complete economically meaningful, long-horizon workflows in specialized professional software. The introduction presents the benchmark’s design and construction and reports that such tasks remain difficult for current agents.

1 Introduction

  • Existing GUI benchmarks emphasize short-horizon general-purpose applications, while long-horizon economically valuable benchmarks largely omit GUI interaction with real software.This leaves few benchmarks at the intersection of professional expertise, real software interaction, and long-horizon execution.Source: §1 · p. 2
  • Workflow-GYM frames real-world professional GUI work as goal-directed workflows and evaluates diverse-domain tasks from natural-language instructions against automatically verifiable final states.Each task runs in a configured environment with professional software installed; agents receive no intermediate guidance or additional input hints.Source: §1 · pp. 2–3
  • 30% was the strongest model’s success rate; agents also omit workflow stages, propagate errors, drift from objectives, lack software knowledge, and face a discrete-action/human-interaction mismatch.The evaluation covered state-of-the-art models under different agent frameworks; repetitive action looping is also identified as a key challenge.Source: §1 · pp. 2–3
  • A reusable task-construction pipeline combines expert-driven design, multi-stage validation, and environment-grounded execution to support scalable generation of high-quality, verifiable workflows.Source: §1 · p. 3

2 Related Works

Prior GUI benchmarks span varied devices and scenarios but largely miss long-horizon professional workflows, while GUI-agent research explores frameworks and training methods for more capable end-to-end interaction.

2.1 Benchmarks for GUI Agents and Long-Horizon tasks

  • Benchmarks vary by interaction device and task scenario, yet remain largely short-horizon and miss end-to-end professional workflows involving domain-specific software and complex multi-stage execution.The device categories are PC-based and mobile; task scenarios range from general-purpose operations to domain- or software-specific work.Source: §2.1 · p. 3

2.2 MLLM-Driven GUI Agents

  • End-to-end GUI tasks better reflect practical GUI ability than GUI question answering; Mobile-Agent-v2 [2], UI-TARS [18], and Step-GUI [21] develop approaches for long-horizon interaction.Mobile-Agent-v2 uses specialized planning, decision, and reflection agents; UI-TARS unifies reasoning-action-observation with sandboxed execution and multi-turn reinforcement learning; Step-GUI uses calibrated step-level rewards and a self-evolving training pipeline.Source: §2.2 · p. 3

3 Workflow-GYM

Workflow-GYM evaluates professional GUI workflows through authentic, domain-specific, long-horizon tasks with objective success criteria, built and validated through a multi-stage pipeline.

3.1.1 Workflow Selection

  • Experts source tasks from daily professional practice, filter for realism, specialized software and knowledge, at least 30 actions, and objective verifiability, yielding over 1,000 candidates.Source: §3.1.1 · pp. 4–5

3.1.2 Environment Setup

  • A two-level setup uses 56 reusable software-specific virtual machines, then injects task artifacts at runtime into pre-launched, interaction-ready states for controlled, comparable evaluation.Removing irrelevant login or authentication steps avoids extraneous interactions.Source: §3.1.2 · p. 5

3.1.3 Task Construction

  • Each instance pairs self-contained instructions with explicit artifact-or-GUI-state targets and binary deterministic criteria, plus expert atomic-interaction procedures that solve tasks without revealing answers.Source: §3.1.3 · p. 5

3.1.4 Task Validation

  • Tasks undergo environment execution, expert and LLM instruction review, and SOTA-agent trials with human trajectory review; failures trigger revision until all three validation stages pass.Source: §3.1.4 · p. 6

3.2 Benchmark Statistics

  • Workflow-GYM contains 338 tasks across 6 top-level and 23 fine-grained domains, with procedures requiring 30–110 execution steps.Execution-step count serves as a proxy for task complexity.Source: §3.2 · p. 6

3.3 Automatic Evaluation for Workflow-GYM Tasks

  • Evaluation checks artifact files against requirements or scores post-task GUI screenshots with a VLM, using rule-based or LLM-based methods as appropriate.Source: §3.3 · p. 6

4 Experiments

Experiments show low success on Workflow-GYM even for the strongest evaluated models, with performance worsening as task difficulty rises and varying across domains; reliable GUI execution also depends on model-specific agent-framework alignment.

4.1 Experimental Settings

  • Evaluation gave each model one instruction to complete an isolated GUI workflow unaided, allowed up to 400 interaction rounds, ran three trials per task, and used Seed-1.8[13] for non-rule-based scoring.The evaluation used model-specific agent frameworks and aggregated results across independent trials.Source: §4.1 · p. 7

4.2 Main Results

  • 30.67% average pass rate for Gemini-3.1-pro and 41.42% pass@3 for Kimi-k2.6 mark the strongest reported results; most models’ pass@3 rates stay below 30%.Table 2 reports results averaged over 338 × 3 = 1,014 trials.Source: §4.2 · p. 7, Table 2 · p. 8
  • All models’ performance falls as task difficulty rises, because interdependent-step failures propagate and longer workflows demand sustained reasoning plus error-prone dragging and coordinated UI operations.The passages identify both long-range sequential reasoning and reliable execution of complex interactions as bottlenecks.Source: §4.2 · p. 8
  • Kimi-k2.6 leads Data Analysis and Multimedia & Creative, while Gemini-3.1-pro performs particularly well in Geography & Environment, Engineering & Design, and Scientific Computing.The passages suggest coding and multimodal perception may explain Kimi-k2.6’s strengths, while Gemini-3.1-pro’s strengths may relate to scientific and reasoning capabilities.Source: §4.2 · pp. 8–9, Table 3 · p. 9
  • Agents perform better on structured, standardized GUIs than on visually open-ended environments such as multimedia creation.Structured interfaces tend to have explicit elements, stable layouts, and discrete interactions; open-ended environments demand spatial manipulation, visual grounding, and fine-grained control.Source: §4.2 · p. 9
  • GUI performance depends on model-specific agent-framework alignment, including the agent loop, tool design, coordinate transformations, action representations, and interfaces.For example, Gemini may output coordinates on a normalized [0,1000] grid, requiring a transformation layer; deploying models outside aligned configurations can significantly degrade performance.Source: §4.2 · p. 9

5 Analysis and Discussions

This section analyzes barriers to long-horizon workflow completion and examines how procedural guidance and video tutorials affect model performance. It also discusses the limits of video demonstrations on unfamiliar, complex creative software, where interface differences can lead to unsuccessful attempts and extra steps.

5.1 Outcome-level Failure Decomposition

  • Pearson r = −0.97 links workflow incompletion and overall performance, as stronger models complete workflows more often and increasingly fail through execution inaccuracies.Weaker models often fail before the final stage, while stronger models more often reach it but produce an incorrect result.Source: §5.1 · pp. 9–10, Figure 5 · p. 10

5.2 Behavior-level Failure Analysis

  • Behavior-level failures divide into long-horizon breakdowns in consistency, progress tracking, and completion verification, and execution failures in visual grounding, software knowledge, and complex GUI manipulation.The analysis distinguishes observable unsuccessful outcomes from the process-level mechanisms that cause them.Source: §5.2 · p. 10

5.2.1 Long-Horizon-Level Failures

  • Long-horizon failures include propagated errors from invalid states, omitted mandatory workflow stages, and objective drift toward intermediate interface states or local goals.These deviations can leave later actions locally plausible while making the task unrecoverable or diverging from its intended result.Source: §5.2.1 · pp. 10–11

5.2.2 Execution-Level Failures

  • Professional-software execution failures include insufficient software-specific knowledge and repetitive loops, defined as ten identical consecutive actions; over 100 loops were identified across trajectories.Geography & Environment tasks accounted for 27.20% of looping cases versus 10.95% of tasks (2.48× enrichment); Scientific Computing showed 1.38× and Data Analysis 0.53×.Source: §5.2.2 · p. 11, Figure 8 · p. 14

5.3 From Continuous Interaction to Discrete Observation: A Fundamental Bottleneck of Current GUI Agents

  • Snapshot-based agents observe only before and after actions, preventing real-time monitoring or correction during continuous GUI operations and making visually precise tasks reliant on trial and error.In a design-software example requiring a line exactly 152 cm long, agents lacked feedback during drawing and had to check the length after each attempt.Source: §5.3 · pp. 13–14

5.4 Ablation Study: Effects of Procedural Guidance

  • The procedural-guidance ablation compares task-only instructions with textual step-by-step tutorials and video tutorials to test their effects on long-horizon execution and complementary visual information.The main experiment supplied target outcomes and basic requirements; the ablations added procedural guidance.Source: §5.4 · pp. 14–15

5.4.1 Effect of Textual Step-by-Step Procedures

  • Explicit step-by-step text substantially improves performance by mitigating operation-order errors, cascading errors, and misconfigurations, but inaccurate clicks, early stopping, and skipped steps persist.Seed-2.0-lite and GPT-5.4 (and mini) show large gains; structured procedures can mitigate premature termination, but some models still deviate autonomously from them.Source: §5.4.1 · p. 15

5.4.2 Additional Benefits of Video Demonstrations

  • 31/100 for Seed and 35/100 for Gemini 3.1 Pro were achieved with video tutorials, exceeding text-only results of 28 tasks for each model.Video added +3 tasks for Seed (28 to 31) and +7 for Gemini 3.1 Pro (28 to 35), with gains concentrated in procedural applications such as JASP, GeoGebra, and HomeBank, where demonstrations convey click locations and order.Source: §5.4.2 · p. 16
  • Video tutorials lift current models to their best performance, but do not make already-solvable tasks more step-efficient; extra steps go to unsuccessful attempts on hard tasks.The modality comparison evaluates pass@1 under a 400-step budget on 100 tasks sampled across professional-software categories.Source: §5.4.2 · p. 16, Table 5 · p. 17
  • On unfamiliar, high-degree-of-freedom software such as Blender and GIMP, even detailed video demonstrations may not suffice for long-horizon tasks.Failures involve tracking whether actions took effect, re-localizing controls after interface deviations, recovering from failing strategies, and performing fine-grained spatial or parameter manipulation; replaying demonstrated coordinates can cause mis-click loops that exhaust the step budget.Source: §5.4.2 · p. 16
  • 165 steps is the median for Gemini 3.1 Pro’s failed tasks under video guidance, while solved-task medians remain comparable across settings at about 83–91 steps.The modality study uses a 100-task subset with two representative models; its absolute performance numbers are not directly comparable to the full benchmark.Source: §5.4.2 · p. 16, Table 6 · p. 17, Table 5 · p. 17

6 Conclusion … B.1 Workflow Stage Omission

Workflow-GYM evaluates long-horizon GUI work in professional domains, where agents can execute low-level actions but still struggle to complete workflows coherently. Its construction relies on domain experts, and a case study illustrates how omitting a required workflow stage can undermine the result.

6 Conclusion

  • Workflow-GYM is introduced as the first benchmark designed to rigorously evaluate LM-based GUI agents on real-world, long-horizon professional tasks across diverse domains and specialized software.Source: §6 · p. 17
  • Long-horizon professional workflows remain highly challenging because they demand complex decisions, precise execution, and coherent integration across extended interactions.The authors present Workflow-GYM as a testbed for advancing more capable, reliable, and practically useful computer-use systems.Source: §6 · p. 17
  • Table 7 contrasts success and failure on an identical pivotal step, differing only in whether the model can consult demonstrated video frames when choosing a control or command.Source: Table 7 · p. 18

A Annotator and Domain Expert Background

  • Workflow-GYM uses domain experts throughout its construction and validation lifecycle, from workflow proposal and task specification through environment verification, instruction refinement, and evaluation validation.The task construction requires domain expertise, familiarity with professional software ecosystems, and practical operational experience.Source: A Annotator and Domain Expert Background · p. 22
  • The 71-member expert team spans all benchmark task domains and contributes workflows based on authentic professional practices and real software usage rather than synthetic or publicly templated tasks.Table 8 summarizes the expert team's composition.Source: A Annotator and Domain Expert Background · p. 22, Table 8 · p. 22

B.1 Workflow Stage Omission

  • In a stage-omission failure, the agent skips layout establishment and creates UI elements directly, leaving all requested buttons present but violating the intended centered-menu layout.Source: §B.1 · p. 22

Workflow Stage Omission

The agent correctly establishes the named project and Control root, then omits the required layout-container stage and continues without detecting the deviation, leaving the final UI noncompliant.

Workflow Stage Omission

  • The model skips the required layout stage, creating buttons directly under Control instead of first establishing a VBoxContainer for the layout constraints.The reference workflow inserts the centered container before adding buttons.Source: §B.1 · p. 23
  • The project named game and Control root are correctly established as a shared prefix before the trajectory diverges.The model's early setup remains aligned with the ground-truth workflow.Source: §B.1 · p. 23
  • The three buttons become sibling children of Control, so they inherit the layout failure rather than the intended container structure.All three buttons exist, but their parent-child structure is wrong.Source: §B.1 · p. 24
  • Changing viewport dimensions cannot restore the missing layout hierarchy, and saving without visual verification leaves only one overlapping button visible.The model treats node existence as completion and does not check whether all three buttons are simultaneously visible.Source: §B.1 · p. 24, Figure 10 · p. 24

B.2 Objective Drift

The agent begins the video-editing workflow productively but shifts from the overall objective to configuring one fragment’s adjustment layer. It then exports without checking the remaining requirements, producing a video that fails several specified constraints.

B.2 Objective Drift

  • After making initial edits, the agent focuses on adjustment controls and layer configuration, treating this local GUI problem as the objective rather than advancing the full workflow.The shift follows difficulty configuring adjustment settings for one fragment; the agent’s actions remain locally reasonable but no longer support full-workflow completion.Source: §B.2 · pp. 24–25
  • The agent exports without verifying remaining editing requirements, and the output fails the specified duration, fragment-level settings, and export constraints.The export follows the prolonged focus on adjustment controls and layer configuration.Source: §B.2 · pp. 24–25

C Representative tasks from different domains in Workflow-GYM · C.1 Geography and Earth Science

The representative geography task tests an end-to-end QGIS workflow, from analyzing a shapefile to measuring a path and exporting results.

C.1 Geography and Earth Science

  • In QGIS, the agent loads county_dem_slope.shp, computes elevation statistics and ID-grouped mean coordinates, selects two specified points, builds a path, measures its length, and exports text, GeoJSON, and CSV.The task combines geospatial analysis, feature selection, path construction, measurement, and file export.Source: §C.1 · p. 25

C.2 Data Analysis … C.5 Engineering and Design

The appendix illustrates four professional GUI workflows spanning data analysis, finance, multimedia editing, and engineering design, each with specified operations and outputs.

C.2 Data Analysis

  • In Data Analysis, the agent preprocesses iris.arff in Weka, configures J48 and RandomForest, runs 10-fold cross-validation, and reports classification metrics.Preprocessing includes standardization and filtering the first four attributes with specified RemoveWithValues settings; save the result as iris_preprocessed.arff.Source: §C.2 · p. 25
  • The Data Analysis workflow requires saving iris_preprocessed.arff and reporting final classification metrics after cross-validation.The dataset is processed using specified RemoveWithValues settings before the classifiers are configured.Source: §C.2 · p. 25

C.3 Finance and Management

  • In Finance and Management, the agent creates mmex_case_account.mmb in Money Manager Ex, sets USD, and opens a User_1111 cash account with 5000.00.The workflow also requires six January 2025 expense transactions with specified dates, amounts, and categories.Source: §C.3 · p. 25
  • The Finance and Management task requires entering six January 2025 expenses and obtaining their total from the Income vs Expenses report.The report is generated from the specified Money Manager Ex database and account.Source: §C.3 · p. 25

C.4 Multimedia and Creative

  • In Multimedia and Creative, the agent edits 8.mp4 in CapCut with three timed text overlays, four timed atmosphere effects, and two transitions.The transitions must be placed at required split points; background music comes from 26.mp3 and is trimmed at a specified timestamp.Source: §C.4 · p. 25
  • The Multimedia and Creative workflow exports the finished edit as case7.mp4 and checks its duration, frame width, and frame height.The video includes background music from 26.mp3 trimmed at the specified timestamp.Source: §C.4 · p. 25

C.5 Engineering and Design

  • In Engineering and Design, the agent uses FreeCAD Part Design to make a stepped solid with a 60 mm by 40 mm base extruded to 16 mm.A centered 30 mm by 20 mm top step is extruded to 6 mm.Source: §C.5 · p. 25
  • The Engineering and Design task requires reading the final solid’s volume and surface area from the FreeCAD Python console.The measurements are for the stepped model with a centered 30 mm by 20 mm top step extruded to 6 mm.Source: §C.5 · p. 25

C.6 Scientific Computing

The examples illustrate complex professional GUI workflows and show how local UI problem-solving can displace the end-to-end objective, leaving the final deliverable inconsistent with requirements.

C.6 Scientific Computing

  • A KNIME engineering task asks the agent to build a three-row categorized table, create a bar chart and scatter plot, and arrange both in a dashboard component.The table specifies values 10/20/30 and 5/15/25, category colors red/blue/green, and a two-column layout.Source: §C.6 · p. 25
  • The CapCut agent’s local search for second-clip adjustment controls displaced the full editing objective, and its exported file failed timing, fragment-setting, and property checks.The task required four precisely segmented clips with fragment-specific settings and a verified CASE25.mp4 export.Source: §C.6 · p. 26
  • The CapCut trajectory began with the correct video-editing objective and made partial segmentation progress before the model became centered on the second fragment’s adjustment controls.It imported 11.mp4, began building the required timeline, and created multiple segments before the later drift.Source: §C.6 · p. 26

D List of all professional software in Workflow-GYM

Workflow-GYM includes professional GUI tasks across finance, multimedia creation, engineering design, and scientific computing. The examples require users to complete multi-step workflows in specialized software, from data entry and modeling to chart configuration.

Finance & Management

  • The Finance & Management task uses Money Manager Ex to create a USD database and $5,000 cash account, enter six January 2025 expenses, and verify the Income vs Expenses report.The transactions are entered as withdrawals with dates, amounts, payees, and categories including Food, Transfer, Gifts, and Education.Source: D List of all professional software in… · p. 29

CapCut Short Video Editing

  • The Multimedia & Creation task imports a video into CapCut, splits it at two specified timestamps, and applies Before/After Comparison II and Slide transitions.The specified split points are 00:00:11:03 and 00:00:21:00.Source: D List of all professional software in… · p. 29

FreeCAD Part Design

KNIME Dashboard Component

  • The Scientific Computing task uses KNIME to create a dataset, generate a bar chart of summed Val1 by Cat, and configure a scatter plot relating Val1 to Val2.The dataset has Cat, Val1, and Val2 columns with three rows; the charts are placed in a two-column component layout.Source: D List of all professional software in… · p. 30

Related papers

Loading 2606.11042v4…