Source-linked AI summary
OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations
Yixiong Xiao, Lang An, Hucheng Yang, Pinxue Ma, Yongquan Chen, Jingjia Cao, Yusai Zhao, Ting Wang, Ting Liu, Siqi Bao, Jingbo Zhou, Hua Wu
TL;DR
Professional GUI agents still struggle with domain-specific SOPs that depend on implicit expertise, software conventions, and verification requirements. OmegaUse-SOP addresses this by iteratively converting human demonstrations into semantically grounded, configurable, and executable skills. In PVsyst case studies, it improved task completion across evaluated models, while ablation results indicate that semantic reasoning was important.
Problem
General computer-use benchmarks do not fully capture professional SOP requirements such as domain knowledge, software conventions, configurable parameters, and task-level verification.
Method
OmegaUse-SOP uses Observe, Reason, Configure, and Execute to record demonstrations, abstract semantic steps, add domain guidance and task parameters, and execute verified procedures.
Results
OmegaUse-SOP improved task completion across open-weight and proprietary vision-language models on photovoltaic simulation workflows in PVsyst 7.2, while removing Reason reduced Qwen3-VL pass rate from 5/5 to 2/5.
Takeaways & Limitations
The system provides a practical approach for capturing and reusing professional procedural knowledge in GUI agents and suggests improved reliability on professional PVsyst workflows.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly evolving from conversational assistants into agents capable of operating external digital environments. Graphical user interface (GUI) agents play an important role in this transition, as many real-world workflows remain accessible only through user-facing software interfaces. However, despite recent progress on general computer-use benchmarks, domain-specific professional standard operating procedures (SOPs) remain challenging for GUI agents because they often involve implicit domain knowledge, software-specific conventions, and task-level verification requirements. We introduce OmegaUse-SOP, a human-in-the-loop SOP Engineering system for transforming human demonstrations of professional computer use into reusable SOP skills for GUI agents. Analogous to prompt engineering, SOP Engineering iteratively refines demonstrations, execution rules, and domain knowledge to convert professional SOPs into reusable GUI-agent skills. OmegaUse-SOP consists of four modules: Observe, Reason, Configure, and Execute. Together, these modules record expert operations as multimodal GUI traces, abstract low-level events into semantic step-level instructions, incorporate domain rules and task-specific parameters, and execute the resulting skills in live GUI environments through step-wise grounding, action generation, and verification. To demonstrate its effectiveness, we collaborate with a power-sector client and test OmegaUse-SOP on photovoltaic simulation workflows in PVsyst 7.2. The results suggest that OmegaUse-SOP can improve GUI-agent reliability on professional SOP tasks, highlighting a practical path toward deploying GUI agents in domain-specific professional software environments.
1 Introduction
Professional GUI workflows remain difficult for agents because they require domain-specific SOPs, software conventions, implicit expertise, configurable parameters, and verification. OmegaUse-SOP addresses this gap by turning human demonstrations into reusable, semantically enriched skills and evaluates them on PVsyst workflows.
- Motivation: Professional workflows require domain rules, software-specific conventions, implicit knowledge, configurable parameters, and task-level verification beyond general GUI capabilities.Benchmarks report difficulties with ambiguity, branching, error handling, workflow omissions, error propagation, objective drift, and insufficient software-specific knowledge.
- Contribution: OmegaUse-SOP is a human-in-the-loop system that transforms human demonstrations of professional computer use into reusable SOP skills for GUI agents.Its SOP Engineering methodology iteratively refines demonstrations, execution rules, domain knowledge, and task-specific parameters.
- Contribution: The system comprises Observe, Reason, Configure, and Execute modules for recording traces, abstracting semantic instructions, incorporating context, and executing verified procedures.Together, the modules preserve demonstrated procedures while enriching them with semantic targets, configurable parameters, domain guidance, and step-wise verification.
- Evaluation: OmegaUse-SOP is evaluated with open-weight and proprietary vision-language models on five photovoltaic simulation tasks in PVsyst 7.2.The tasks cover meteorological data import, plane orientation, grid-connected system settings, detailed losses, and simulation execution.
2 System Design
OmegaUse-SOP converts professional demonstrations into editable, parameterized, semantically grounded procedures through four iterative modules. Execution progressively retrieves relevant step information and verifies outcomes while retaining human intervention for controlled debugging.
- 2.1 Observe: Observe records pre-action screenshots, visual targets, mouse and keyboard actions, timestamps, and event order as a structured multimodal SOP trace.It captures the interface state immediately before each human event to preserve the observed see-act sequence.
- 2.2 Reason: Reason transforms low-level events into semantic step instructions describing the operated object, visual context, and procedural role.For coordinate actions, it uses pre- and post-action screenshots to identify targets instead of planning alternative actions or replaying pixels blindly.
- 2.3 Configure: Configure adds editable domain SOP guidance and task-specific parameters so procedures can encode professional rules and adapt changing values across task instances.Guidance can specify checks, replacements, field clearing, and post-modification verification, while parameters mark demonstrated values as variables.
- 2.4 Execute: Execute applies the configured SOP in a live desktop environment using trace evidence, semantic understanding, domain knowledge, and task parameters.It progressively retrieves only information relevant to the current step to reduce context overhead, while verification supports controlled execution.
- 2.5 System integration: The complete system turns human demonstrations into visually grounded, editable, and parameterized procedures without replacing professional judgment with a black-box policy.The resulting intermediate layer supports capturing, inspecting, adapting, and enacting professional practice in live software environments.
3 Experiment
The experiment evaluates OmegaUse-SOP on five representative PVsyst 7.2 tasks using three vision-language models, comparing direct execution with SOP-guided execution. SOP guidance improves task completion across models, while removing semantic reasoning substantially reduces success.
- Experimental Setup: The study evaluates five client-derived PVsyst workflows, including meteorological data importation, plane orientation, grid-connected system, detailed losses, and production simulation.The experiment uses real-world power-system design workflows from the collaborating client.
- Main Results: 5/5 task completion was achieved by all three models with OmegaUse-SOP, compared with lower baseline completion for every model.This result indicates more reliable execution on the evaluated professional PVsyst workflows.
- Ablation Study: Removing the Reason module reduced Qwen3-VL’s pass rate from 5/5 to 2/5.Without Reason, the agent completed only plane orientation setting and simulation execution, failing the other three tasks.
4 Conclusion
The conclusion presents OmegaUse-SOP as a human-in-the-loop system for capturing professional procedural knowledge and improving GUI-agent reliability. Its case study in PVsyst 7.2 shows benefits across open-weight and proprietary vision-language models and highlights semantic reasoning over demonstrations.
- Conclusion: OmegaUse-SOP records expert GUI operations, abstracts them into semantic step-level instructions, incorporates domain guidance and task-specific parameters, and executes the resulting skills.The system is evaluated on photovoltaic simulation workflows derived from real-world client tasks.
- Conclusion: The case study shows improved task completion across both open-weight and proprietary vision-language models.The evaluation uses professional photovoltaic simulation workflows in PVsyst 7.2.
- Conclusion: Semantic reasoning over demonstrations is important for converting raw interaction traces into reusable SOP representations.The conclusion connects semantic abstraction with reuse in professional GUI-agent execution.
- Conclusion: OmegaUse-SOP provides a practical approach to capturing professional procedural knowledge and improving GUI-agent reliability in domain-specific professional software environments.The supported scope is professional GUI workflows represented by the PVsyst 7.2 case study.
A PVsyst Task Screenshots
Table 4 presents representative PVsyst panels associated with the five GUI tasks used in the main experiment.
- A PVsyst Task Screenshots: Table 4 associates representative PVsyst panels with the five GUI tasks evaluated in the main experiment.The caption identifies the panels as representative examples for the task set.
B Illustrative Examples of Domain SOP Guidance
The examples show how domain experts add targeted guidance after observing GUI-agent errors in PVsyst. The guidance addresses spinner direction and the priority of explicitly user-specified parameter values.
- Illustrative Examples of Domain SOP Guidance: Figures 3 and 4 show PVsyst interfaces linked to two pieces of domain SOP guidance added after model errors during trials.The examples connect observed execution failures with subsequent expert guidance.
- Operating a Spinner Control: For the number-of-modules-in-series spinner, the SOP instructs agents to use the down arrow to decrease and the up arrow to increase the value.The guidance targets an observed directional-selection error.
- Prioritizing User-Specified Parameter Values: When users specify parameter values such as a 35° plane tilt, the SOP prioritizes those values over operation instructions or historical inputs.The guidance addresses cases where the model retained conflicting prior values.