Source-linked AI summary
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Liya Zhu, Jingzhe Ding, Jian Zhang, Jianbo Xue, Shihao Liang, Ge Zhang, Yi Zhu, Duju Zeng, Xiang Gao, Qingshui Gu, Mailun Gao, Huimin Che, Yan Zhao, Peiheng Zhou, Haojun Wang, Chaobo Xian, Lili Le, Chi Wu, Yiwei Liu, Shengda Long, Jiale Yang, Fangzhi Xu, Sijin Wu, Haodong Duan, Chao He, Zhaojian Li, Minchao Wang, Huan Zhou, Jiani Hou, Chuqian Yu, Weiran Shi, Hongwan Gao, Jiamin Chen, Guanhong Chen, Tingqin Luo, Kaiyuan Zhang, Zhixin Yao, Qing Hua, Yuhao Jiang, Jin Chen, Pu Chen, Zhenyu Hu, Xingyu Li, Zhengxuan Jiang, Meng Cao, Tianfeng Long, Haozhe Wang, Mingzhang Wang, Yichen Zhang, Yiming Dai, Chenchen Zhang, Jiaying Wang, Xinying Liu, Xingzu Liu, Lingling Zhang, Xinjie Chen, Yujia Qin, Wangchunshu Zhou, Zhiyong Wu, Yang Liu, Jiaheng Liu, Lei Zhang, Shen Yan, Wenhao Huang, Zaiyuan Wang, Xiaolong Chang
TL;DR
Existing GUI benchmarks rarely test long-horizon, domain-specific professional workflows. Workflow-GYM introduces such a benchmark, and experiments show that even the strongest model achieves only a 30% success rate, with workflows remaining highly challenging.
Problem
Existing GUI benchmarks rarely evaluate short-horizon, general-purpose tasks involving specialized professional software, leaving realistic high-value workflows insufficiently assessed.
Method
Workflow-GYM evaluates end-to-end domain-specific workflows in configured professional software environments from natural-language objectives, using final states or artifacts for automatic verification.
Results
30% success rate is achieved by the strongest model, while agents exhibit workflow stage omission, error propagation, objective drift, and insufficient software-specific knowledge.
Takeaways & Limitations
Workflow-GYM shows that long-horizon professional GUI workflows remain highly challenging because they demand complex decision-making, precise execution, and coherent integration over extended trajectories.
Takeaways & Limitations
Current GUI agents use snapshot-based observation-action cycles, leaving intermediate execution states inaccessible for real-time monitoring and correction.
Abstract
from arXiv · showhide
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple applications, and short-horizon tasks, leaving it largely unknown whether modern agents can follow user instructions to autonomously operate domain-specific professional software and accomplish economically valuable work in an end-to-end manner. To bridge this gap, we introduce Workflow-GYM, a benchmark for long-horizon GUI tasks centered on professional domains and specialized software environments. Through extensive experiments on state-of-the-art models, we find that even the strongest models achieve only slightly above 30% success rates, highlighting that professional long-horizon GUI workflows remain highly challenging for current GUI agents. Further analysis reveals that current agents struggle to maintain long-horizon workflow consistency, frequently exhibiting workflow stage omission, error propagation, objective drift, and insufficient understanding of professional software environments. Our findings provide important insights into the limitations of current agent systems and suggest key directions for the next generation of GUI-agent research.
1 Introduction
Workflow-GYM addresses the lack of realistic benchmarks for evaluating GUI agents on long-horizon, domain-specific professional workflows. Experiments show that current systems remain highly challenged, with the strongest model achieving only 30% success and exhibiting recurring workflow and software-understanding failures.
- Conceptualization: Workflow-GYM defines workflows as structured, goal-directed GUI-action sequences that transform an initial state into a desired outcome across diverse professional domains.This abstraction targets economically meaningful computer-use processes performed by domain experts.
- Motivation: Existing GUI-agent benchmarks mainly evaluate short-horizon tasks in general-purpose applications, rarely requiring specialized professional software or domain-specific expertise.This limits their ability to reflect realistic, high-value workflows.
- Benchmark: Workflow-GYM evaluates end-to-end, domain-specific workflows in fully configured environments using only natural-language task objectives.Tasks require agents to operate pre-installed professional software without intermediate guidance or additional input hints.
- Results: 30% is the strongest model’s success rate, highlighting a substantial gap between current capabilities and real-world task requirements.The result comes from extensive experiments with multiple state-of-the-art models under different agent frameworks.
- Failure Analysis: Recurring failures include workflow stage omission, error propagation, objective drift, insufficient software-specific knowledge, and repetitive action looping.The evaluation systematically analyzes these model-level limitations and long-horizon failures.
2 Related Works
Existing GUI benchmarks are organized by interaction device and task scenario, spanning general-purpose and domain-specific settings. Recent work increasingly evaluates GUI agents on end-to-end interaction tasks, but the supplied passages emphasize the diversity of benchmark environments and applications.
- Benchmark Taxonomy: GUI benchmarks vary by interaction device, including PC-based and mobile environments, and by task scenario, including general-purpose and domain-specific applications.The scenario distinction separates everyday UI operations from specialized software requiring domain knowledge.
- Representative Benchmarks: AmbiBench evaluates everyday mobile activities, whereas GUI-360 and Science-Board target productivity software and scientific workflows, respectively.These examples illustrate the contrast between general-purpose interaction scenarios and domain-specific or software-specific settings.
- GUI Agents: End-to-end GUI interaction tasks more directly measure practical interface operation than GUI-related question answering benchmarks.The passage frames end-to-end interaction as a better reflection of accomplishing objectives through real graphical user interfaces.
- GUI Agents: The advancement of LLMs and VLLMs has driven GUI agents that interact with graphical environments to accomplish tasks.Mobile-Agent-v2 is presented as studying mobile GUI interaction through multi-agent collaboration.
3 Workflow-GYM
Workflow-GYM constructs long-horizon GUI benchmarks from authentic professional workflows through expert task selection, controlled environments, executable task specifications, and multi-stage validation. The resulting benchmark contains 338 tasks across 6 top-level domains and 23 fine-grained subdomains.
- Workflow Selection: Workflow-GYM tasks are selected from experts’ real-world practices and must require professional software, domain knowledge, at least 30 atomic actions, and objectively verifiable outcomes.These criteria are intended to prevent retrieval-based solutions while ensuring realistic, complex, and reproducibly evaluable workflows.
- Workflow Selection: Over 1,000 candidate tasks form the initial pool for subsequent dataset-construction stages.
- Environment Setup: The benchmark separates reusable professional-software base environments from lightweight task-specific instantiation, enabling scalable, controlled, repeatable, and comparable evaluation.Base environments are packaged virtual-machine images with pre-installed software, while task artifacts are injected at runtime into a unified interaction-ready state.
- Task Construction: Each task combines a self-contained instruction, explicit artifact or GUI-state outcomes with binary success criteria, and an atomic step-by-step expert procedure.The procedure is designed so that strictly following it achieves the defined outcome without revealing the final answer or trivial hints.
- Task Validation: Only tasks passing environment verification, instruction-level review, and end-to-end agent-based refinement are retained in the final benchmark.Failures are iteratively revised until all validation criteria are satisfied.
- Final Benchmark: 338 tasks span 6 top-level domains and 23 fine-grained subdomains, providing broad coverage of professional computer-use tasks across diverse application areas.
4 Experiments
Workflow-GYM evaluates frontier models on end-to-end GUI workflows without intermediate guidance and finds that professional long-horizon tasks remain substantially beyond current agents’ capabilities. Performance declines with workflow length, varies across domains, and depends strongly on model–framework coupling.
- Evaluation Setup: Models receive one natural-language instruction and must complete each workflow through GUI interactions in an isolated environment without intermediate guidance.Seed-1.8 serves as the judge for non-rule-based scoring using predefined, validated rubrics.
- Overall Performance: 30.67% average pass rate and 41.42% pass@3 rate are achieved by the best-performing models, Gemini-3.1-pro and Kimi-k2.6, respectively.The majority of models have pass@3 rates below 30%, while SOTA models exceed 70% on OS-World.
- Long-Horizon Difficulty: All models show clear performance degradation as task difficulty and step number increase.Interdependent decisions, error propagation across stages, and complex low-level interactions such as dragging and multi-step manipulation amplify failures in longer workflows.
- Domain Performance: Performance varies substantially across workflow domains, with Kimi-k2.6 strongest on Data Analysis and Multimedia & Creative tasks and Gemini-3.1-pro strongest on several scientific and technical domains.Agents generally perform better on structured GUIs, such as data analysis and finance, than on visually open-ended multimedia environments.
- Agentic Framework: GUI performance exhibits strong model–framework coupling, requiring tailored agent-loop configurations, tool-group designs, and pixel-space grounding mechanisms for different models.This coupling differs from coding scenarios where many models perform strongly under a unified general-purpose framework.
5 Analysis and Discussions
Workflow-GYM analysis identifies workflow completion as a primary bottleneck, with failures driven by incompletion, error propagation, stage omission, objective drift, software knowledge deficiencies, and repetitive looping. It also highlights limitations of snapshot-based GUI-agent frameworks and shows that video tutorials improve performance for both evaluated models.
- Failure Categories: Workflow incompletion means failing to reach the required final state, whereas final-state-but-incorrect failures reach it but violate task requirements.Examples include missing output artifacts, premature termination, parameter mismatches, missing components, and incorrect configurations.
- Workflow Completion: Pearson r = −0.97 indicates workflow incompletion decreases consistently as model capability improves.Higher-performing models more often reach the final stage, while weaker models predominantly fail before completing the workflow.
- Failure Modes: Error propagation, workflow stage omission, and objective drift can make later actions locally plausible while the overall trajectory becomes unrecoverable.These failures arise when incorrect earlier states, skipped mandatory stages, or intermediate interface states replace the intended final objective.
- Failure Modes: Over 100 looping cases were identified, with Geography & Environment contributing 27.20% of looping cases and showing 2.48× enrichment relative to task representation.Looping is identified when an identical action is issued for ten consecutive interaction steps.
- Framework Limitations: Snapshot-based GUI agents observe only pre-action and post-action states, limiting continuous visual monitoring and dynamic adjustment during complex manipulations.The analysis contrasts this paradigm with a continuous perception-action loop that uses intermediate visual feedback.
- Video Tutorials: 31/100 for Seed and 35/100 for Gemini 3.1 Pro were achieved with the video tutorial, improving Seed by +3 tasks from the purely textual procedure.The tutorial helped avoid inefficient or incorrect interaction paths in the described single-step cases.
6 Conclusion
Workflow-GYM introduces a benchmark for evaluating GUI agents on long-horizon professional computer-use tasks across diverse domains and specialized software environments. Results show that these workflows remain highly challenging because they require complex decision-making, precise execution, and coherent integration over extended interactions.
- Workflow-GYM is designed to rigorously evaluate LM-based GUI agents on real-world, long-horizon professional computer-use tasks across diverse domains and specialized software environments.
- Long-horizon professional workflows remain highly challenging despite agents’ ability to interact with graphical interfaces and execute low-level actions.The challenge arises from the demands of complex decision-making, precise execution, and coherent integration over extended interaction trajectories.
7 Contributions … B.1 Workflow Stage Omission
Workflow-GYM is constructed and validated through an expert-driven process involving 71 domain experts and authentic professional workflows. A documented failure case shows that omitting a required workflow stage can make an otherwise complete interface violate its intended layout.
- 7 Contributions: The project leads are Liya Zhu, Jingzhe Ding, Jian Zhang, Jianbo Xue, Shihao Liang, and Ge Zhang.
- 7 Contributions: The broader contributor group includes 32 additional contributors spanning the listed project roles.The passage names contributors from Chao He through Xinjie Chen.
- 7 Contributions: The sponsor committee consists of Yujia Qin, Wangchunshu Zhou, Zhiyong Wu, Yang Liu, Jiaheng Liu, Lei Zhang, and Shen Yan.
- A Annotator and Domain Expert Background: Workflow-GYM uses an expert-driven construction pipeline because realistic professional workflows require domain expertise, software-ecosystem familiarity, and practical operational experience.Experts participate in workflow proposal, task specification, environment verification, instruction refinement, and evaluation validation.
- A Annotator and Domain Expert Background: 71 experts cover all benchmark task domains and contribute workflows derived from authentic professional practices and real software usage scenarios.
- B.1 Workflow Stage Omission: Figure 10 shows that skipping layout establishment before creating UI elements produces a final interface that violates the intended centered-menu layout.The interface still contains all required buttons, but the required layout is not preserved.
Workflow Stage Omission
The model correctly creates the Godot project and Control root but omits the required layout-container stage before adding buttons. This structural omission propagates through later steps, and the model finishes without verifying that the buttons are simultaneously visible.
- Critical Structural Omission: The model skips the required centered VBoxContainer layout stage and directly creates the Button nodes.The reference workflow inserts and configures the container before adding buttons.
- Correct Initial Progress: The early setup remains aligned: the model correctly names the project game and creates a Control root scene.These steps form the shared prefix before the layout-stage omission.
- Critical Structural Omission: The model creates begin, gameset, and exit as sibling Button nodes under Control, producing the wrong parent-child structure.All three buttons exist, but they inherit the resulting layout failure.
- Diverged Trajectory: Later viewport width and height changes cannot restore the intended UI hierarchy because the layout-management stage was never established.The downstream fix addresses a later-stage setting rather than the missing structural stage.
- Diverged Trajectory: The model saves and reports completion without checking whether all three buttons are simultaneously visible, leaving only one overlapping button on the final canvas.The missed verification prevents recovery from the earlier deviation.
B.2 Objective Drift
The agent initially advances the video-editing workflow but shifts its focus to configuring adjustment controls after encountering difficulties. It then exports without verifying the remaining requirements, producing an output that fails the specified constraints.
- Objective Drift: The agent initially segments the timeline and applies several required edits, making meaningful progress toward the intended video-editing task.
- Objective Drift: After difficulty configuring one fragment’s adjustment settings, the agent shifts from the overall editing objective to a local GUI subproblem.It spends much of the trajectory searching for adjustment controls and refining the adjustment-layer configuration.
- Objective Drift: The agent exports the video without verifying the remaining editing requirements, yielding an output that fails the specified duration, fragment-level settings, and export constraints.Its locally reasonable actions no longer advance completion of the full workflow.
C Representative tasks from different domains in Workflow-GYM … C.5 Engineering and Design
Workflow-GYM includes representative long-horizon GUI tasks spanning geography, data analysis, finance, multimedia, and engineering, each requiring coordinated operations in specialized software. These workflows culminate in exported artifacts, trained-model metrics, reports, media validation, or physical-model measurements.
- C.1 Geography and Earth Science: Geography tasks require QGIS agents to process county_dem_slope.shp, derive grouped statistics and a path, and export text, GeoJSON, and CSV results.The workflow includes elevation statistics, mean coordinates grouped by ID, point selection, path construction, and path-length calculation.
- C Representative tasks from different domains in Workflow-GYM: Across these domains, representative tasks combine multi-stage GUI manipulation with domain-specific outputs that must be saved, reported, exported, or measured.The examples span geospatial files, processed datasets and classifier metrics, financial reports, validated video properties, and FreeCAD solid measurements.
- C.2 Data Analysis: Data-analysis tasks require Weka agents to preprocess iris.arff, configure J48 and RandomForest classifiers, run 10-fold cross-validation, and report classification metrics.Preprocessing includes standardization and filtering the first four attributes with specified RemoveWithValues settings, followed by saving iris_preprocessed.arff.
- C.3 Finance and Management: Finance tasks require Money Manager Ex agents to create a USD account, record six January 2025 expenses, and obtain the total January expense from the Income vs Expenses report.The account User_1111 starts with an initial balance of 5000.00 in mmex_case_account.mmb.
- C.4 Multimedia and Creative: Multimedia tasks require CapCut agents to assemble timed overlays, atmosphere effects, transitions, and trimmed background music, then export and validate the finished video.The workflow edits 8.mp4 with three text overlays, four atmosphere effects, two transitions, and music from 26.mp3, exporting case7.mp4 and checking duration, frame width, and frame height.
C.6 Scientific Computing
The section illustrates professional GUI workflows through a KNIME dashboard-construction task and a CapCut video-editing task. The CapCut case shows objective drift from end-to-end delivery toward local adjustment-layer controls, producing an export that fails the required checks.
- Professional workflow examples: The KNIME task requires manually building a three-row table, creating two visualizations, and packaging them as My Dashboard in a two-column layout.The table uses categories A, B, and C; integer values 10/20/30 and 5/15/25; and red/blue/green category colors.
- Professional workflow examples: The CapCut task requires segmenting 11.mp4 into four clips, applying fragment-specific edits, and exporting CASE25.mp4 with duration and resolution checks.The required operations include speed, scale, and visual-parameter settings for each fragment.
- Objective drift: The model then shifts from completing the full edit and export checks to searching adjustment controls and optimizing a local layer for the second fragment.It searches Adjust, Smart Color, LUT, and related panels, creates a custom adjustment layer, and edits visible sliders while other requirements remain unresolved.
- Outcome: The resulting export fails the judge because its timing, fragment settings, and properties do not satisfy the requirements.The target output is CASE25.mp4 with 1922 x 1080 properties, but the exported file remains inconsistent with the specified checks.
- Initial progress: The model initially imports 11.mp4, builds timeline segments, and begins fragment-specific edits while remaining aligned with the full delivery goal.Early progress includes splitting the video and applying speed and scale settings.
D List of all professional software in Workflow-GYM
Workflow-GYM covers long-horizon professional workflows across GIS and Earth Science, Finance and Management, Multimedia and Creation, Engineering and Design, and Scientific Computing. The tasks require end-to-end operations in QGIS, Money Manager Ex, CapCut, FreeCAD, and KNIME.
- GIS & Earth Science: QGIS tasks in GIS and Earth Science load county_dem_slope.shp, compute statistics, coordinates, paths, and distances, then save text, GeoJSON, and CSV outputs.The workflow produces ziduanjibentongji.txt, pingjunzuobiao.geojson, lujing.geojson, and juli.csv.
- Finance & Management: Money Manager Ex tasks in Finance and Management create a USD database, initialize a cash account with 5000.00, enter six January 2025 expenses, and verify the report total.The database is named mmex_case_account.mmb, and the workflow verifies Expenses in the Income vs Expenses report.
- Multimedia & Creation: CapCut tasks in Multimedia and Creation import video, place it on the timeline, split it at two timestamps, and apply specified transitions.The timestamps are 00:00:11:03 and 00:00:21:00, with Before/After Comparison II and Slide transitions.
- Engineering & Design: FreeCAD tasks in Engineering and Design build a parametric two-pad solid and read final Shape.Volume and Shape.Area values from the Python console.The base pad is 60 mm x 40 mm x 16 mm, and the raised step pad is 30 mm x 20 mm x 6 mm.
- Scientific Computing: KNIME tasks in Scientific Computing create and execute a data table, generate a bar chart aggregating Val1 by Cat, and configure a scatter plot relating Val1 to Val2.The dashboard combines Bar Chart and Scatter Plot components in a two-column layout.