SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

Zeyi Sun; Ziyu Liu; Yuhang Zang; Yuhang Cao; Xiaoyi Dong; Tong Wu; Dahua Lin; Jiaqi Wang

Source-linked AI summary

SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience

Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, Jiaqi Wang

arXiv:2508.04700v2Aug 6, 2025cs.AIcs.CLcs.CVcs.LGcs.MAcs.MM

TL;DR

Current computer-use agents rely heavily on human-curated data and struggle with unfamiliar software, motivating autonomous adaptation. SEAgent enables experiential self-evolution through state-based assessment, curriculum generation, and specialist-to-generalist training, with the specialist-to-generalist strategy reaching 34.5% success rate and outperforming specialist and generalist reinforcement-learning baselines.

Problem
Current computer-use agents primarily depend on costly human-curated datasets, while direct generalist training underperforms specialized agents in multi-software environments.
Method
SEAgent combines autonomous exploration and experiential learning with a World State Model, Curriculum Generator, reinforcement learning from positive and negative actions, and specialist-to-generalist training.
Results
34.5% success rate from specialist-to-generalist training outperforms specialist RL at 32.2% and generalist RL at 30.6%, while the World State Model improves reward-model precision by +7.5% over baseline.
Takeaways & Limitations
SEAgent provides a route toward more versatile computer-use agents that learn software-specific skills autonomously and transfer experiential insights into a strong multi-software generalist.
Takeaways & Limitations
The system relies on GUI-Judge rather than environment reward signals and has been evaluated on tasks taking fewer than 20 human-expert steps, leaving hours-long workflows unresolved.

Abstract

from arXiv · show

Repurposing large vision-language models (LVLMs) as computer use agents (CUAs) has led to substantial breakthroughs, primarily driven by human-labeled data. However, these models often struggle with novel and specialized software, particularly in scenarios lacking human annotations. To address this challenge, we propose SEAgent, an agentic self-evolving framework enabling CUAs to autonomously evolve through interactions with unfamiliar software. Specifically, SEAgent empowers computer-use agents to autonomously master novel software environments via experiential learning, where agents explore new software, learn through iterative trial-and-error, and progressively tackle auto-generated tasks organized from simple to complex. To achieve this goal, we design a World State Model for step-wise trajectory assessment, along with a Curriculum Generator that generates increasingly diverse and challenging tasks. The agent's policy is updated through experiential learning, comprised of adversarial imitation of failure actions and Group Relative Policy Optimization (GRPO) on successful ones. Furthermore, we introduce a specialist-to-generalist training strategy that integrates individual experiential insights from specialist agents, facilitating the development of a stronger generalist CUA capable of continuous autonomous evolution. This unified agent ultimately achieves performance surpassing ensembles of individual specialist agents on their specialized software. We validate the effectiveness of SEAgent across five novel software environments within OS-World. Our approach achieves a significant improvement of 23.2% in success rate, from 11.3% to 34.5%, over a competitive open-source CUA, i.e., UI-TARS.

1 Introduction

SEAgent addresses the dependence of computer-use agents on human-curated data by enabling autonomous exploration and experiential learning in unfamiliar software. Its World State Model, Curriculum Generator, and specialist-to-generalist strategy support self-evolution, improving success from 11.3% to 34.5% over UI-TARS.

Current computer-use agents primarily depend on costly human-curated demonstrations and tutorials, despite continuously changing software often lacking annotations.
SEAgent enables CUAs to autonomously explore unfamiliar software and learn from experience without human supervision.
The framework combines a World State Model for step-wise trajectory assessment with a Curriculum Generator that produces increasingly challenging tasks.
+7.5% precision improvement over a baseline on AgentRewardBench enables the World State Model to provide higher-quality step-level reward signals.
SEAgent’s specialist-to-generalist strategy integrates software-specific learning into a generalist agent that surpasses individual specialists on their respective applications.
23.2% success-rate improvement, from 11.3% to 34.5%, was achieved over UI-TARS across five professional software applications from OSWorld.

2 Related Work

Prior computer-use research has established strong in-domain performance using structured or multimodal inputs, but training remains heavily reliant on human-curated procedures. Recent reinforcement-learning work introduces scalable optimization with verifiable rewards.

Computer-use agents process either structured text or multimodal screenshots with textual conditions, and have shown strong performance on in-domain benchmarks.
Supervised fine-tuning remains dependent on high-quality human-curated procedures, while recent work applies GRPO with verifiable rewards to improve reasoning capabilities.

3 Methods

SEAgent enables computer-use agents to explore unfamiliar software through a self-evolving curriculum, step-level state assessment, and experiential policy updates. It supports specialist and generalist training, with a specialist-to-generalist strategy addressing the weakness of direct generalist training.

3.1 Autonomous Exploration with Self-evolving Curriculum: SEAgent combines an Actor Model, World State Model, and Curriculum Generator to support autonomous exploration and reinforcement learning from experience.The Actor acts, the World State Model describes and evaluates states, and the Curriculum Generator creates tasks and maintains software knowledge.
3.2 Reinforcement Learning from Experience: The Actor Model is updated through reinforcement learning using step-level judgments and action-specific verifiable reward functions during successive curriculum phases.The resulting policy becomes the actor for the next phase of autonomous exploration.
3.1 Autonomous Exploration with Self-evolving Curriculum: The Curriculum Generator begins from GUI descriptions and initial tasks, then iteratively updates task instructions and guidebook memory from execution judgments and state changes.The guidebook becomes more comprehensive while subsequent task sets are tailored to the Actor Model's current capabilities.
3.1 Autonomous Exploration with Self-evolving Curriculum: The World State Model evaluates each trajectory step and captions GUI state changes, while the Curriculum Generator uses this feedback to expand the software guidebook and generate harder tasks.The curriculum proceeds through sequential phases in which judgments and state descriptions update the guidebook and inform increasingly challenging task sets.
3.2 Reinforcement Learning from Experience: SEAgent combines adversarial imitation of failure actions with GRPO-based learning from correct actions in a weighted objective L(π(θ)) = LGRPO + γLAI.The training loss uses γ = 0.2, and the failure-action objective encourages divergence from behaviors associated with failure.
3.3 From Specialist to Generalist: Direct generalist training is suboptimal compared with specialists, motivating a specialist-to-generalist strategy for learning across multiple software environments.The strategy is designed to integrate specialist experience rather than train a generalist directly across all environments.

4 Experiments

The experiments evaluate SEAgent’s reward modeling and self-evolution across OSWorld software environments, including curriculum learning, specialist training, generalization, and ablations.

4.1 Benchmark of Reward Model for computer use agent.: World State Model uses full-process screenshots and step-level analysis to improve trajectory judgment, narrowing the gap with GPT-4o.It generalizes from Chrome training data to other OSWorld software and AgentRewardBench.
4.1 Benchmark of Reward Model for computer use agent.: Additional historical screenshots improve GUI-Judge’s Average Precision, following a trend similar to GPT-4o relative to its base model.Figure 3 compares Average Precision as the number of input middle states increases.
4.2 Self evolution of GUI Agents: The Curriculum Generator progressively upgrades task instructions without human intervention according to the Actor Model’s evolving capabilities.Tasks advance from simple to complex across training phases and software environments.
4.2 Self evolution of GUI Agents: SEAgent specialists achieve strong performance across five software applications, with gains attributed to step-level rewards and progressively harder curricula.Performance gains saturate beyond three training phases.
4.2 Self evolution of GUI Agents: The specialist-to-generalist procedure produces a generalist that surpasses the individual specialist ensemble.The procedure distills 3.5K successful specialist trajectories before further refinement on multiple software applications.
4.2 Self evolution of GUI Agents: World State Model reward signals, reinforcement fine-tuning, and adversarial imitation each contribute to specialist-training performance on VSCode.The ablation emphasizes learning from both successful behaviors and failure-inducing actions.

5 Conclusion

SEAgent enables computer-use agents to learn from their own experience, specializing in individual software and generalizing across multiple environments. Its scope remains bounded by reward and workflow-complexity limitations.

SEAgent uses a World State Model for step-level rewards and reinforcement learning from trial-and-error experience.The framework supports autonomous exploration of specific software environments.
The resulting agents can evolve into specialists for individual software platforms or a generalist operating across multiple software environments.
The system is bounded by GUI-Judge reward signals rather than direct environmental rewards, while sparse rewards remain challenging in complex environments.
Current evaluations use tasks that human experts can complete in fewer than 20 steps, leaving hours-long workflows as future work.

A.2 Fine-Tuning Dataset and Process

The World State Model is trained on annotated trajectory and visual-difference data, then converts trajectory judgments into action-level reinforcement-learning signals while a curriculum generator creates increasingly difficult tasks.

A specialized dataset was constructed to fine-tune the World State Model for trajectory judgment and state-change description.
Trajectory sampling used UI-TARS and Gemini-2.5-Pro on 43 feasible Google Chrome tasks from OSWorld, with GPT-4o generating judgments and screenshot captions.
Fine-tuning used 860 annotated trajectories and a 1,000-sample change-description dataset with LoRA-based parameter-efficient training.
The World State Model labels trajectory actions as positive or failure actions based on correctness, redundancy, and the first error step.
These dynamic action labels provide rewards for the reinforcement-learning pipeline.
The Curriculum Generator produces tasks of increasing difficulty and diversity to guide systematic software exploration.

B.1 Task Generation Mechanism

The task-generation mechanism uses completed-task evaluations and observed state changes to enrich a software guidebook, which then drives progressively broader and harder task generation.

The Curriculum Generator analyzes completed trajectories to extract task evaluations and observed state changes.
It integrates extracted knowledge, such as rectangle properties, into the software guidebook for generating new tasks.
In the LibreOffice Impress example, the World State Model parses the initial interface and evaluates newly executed tasks and revealed properties.

C.2 Comparative Analysis of Instruction Generation Strategies.

The comparison shows a trade-off between efficient exploitation of known functionality and systematic exploration of unfamiliar software. The guidebook-based strategy is more robust for novel applications, while the two strategies are complementary.

Comparative setup: The Curriculum Generator was compared with NNetNav and WebRL instruction-generation strategies.
Comparative setup: The comparison used Qwen2.5-72B and Gemini-2.5-Pro, trained RL agents with GRPO, and evaluated them on two software applications.
Results: Table 4 reports success rates for different task-generation strategies across two software applications.
Results: NNetNav’s reverse instruction generation is efficient on in-domain VSCode but produces similar tasks that limit functional exploration, especially in OOD Celestia.
Results: The guidebook-based method has lower initial data-generation efficiency but systematically explores software and performs better on challenging Celestia.
Conclusion: Reverse instruction generation efficiently exploits known functionality, whereas guidebook-based generation explores new functionality and builds broader software understanding.
Additional evaluation: On OSWorld, UI-TARS-1.5 showed moderate gains, which the authors hypothesize reflect the base model’s familiarity with that benchmark.

E Sensitivity Analysis on Key Hyperparameters

The sensitivity analysis examines how task-generation breadth and guidebook updates affect SEAgent performance on VSCode. More diverse tasks help until performance plateaus, while excessive change descriptions degrade task generation.

Experimental Setup: The sensitivity analysis uses temperature t = 0 and evaluates the number of generated tasks and change descriptions on VSCode.The metric reported in Table 6 is Success Rate (%).
Number of Generated Tasks: Performance improves as more diverse tasks are generated, eventually plateauing around 100 tasks.The number of generated tasks controls exploration breadth in each learning cycle.
Number of Change Descriptions: 50–100 change descriptions are essential for learning new UI functionalities and generating meaningful unexplored tasks.The descriptions provide information for updating the software guidebook.
Number of Change Descriptions: Providing 200 change descriptions creates an overly long LLM context, degrading task-generation quality and final performance.This result indicates a trade-off between supplying new software knowledge and maintaining manageable context length.

F Ablation on the Loss Balance Factor.

The loss-balance ablation studies γ, which controls the relative contribution of adversarial imitation and GRPO, using VScode Success Rate on OSWorld.

Loss Balance: γ balances adversarial imitation, which learns from errors, against GRPO, which learns to achieve success.The total loss combines the two objectives through this balance factor.
Main Setting: The main experiments set γ = 0.2 based on the ablation results.The reward computation for GUI-agent action types is presented separately in Table 8.
Evaluation: The ablation evaluates VScode Success Rate on OSWorld under different γ values.Table 7 reports the effect of the loss-balance factor.

H Data Statistics during Iterative Reinforcement Learning.

The section documents the data and evaluation infrastructure used during iterative reinforcement learning, including phase-wise episodes, trajectory judges, task-buffer updates, and self-generated software manuals.

Episode Statistics: Table 9 reports Success/Failure episode counts across four self-evolution phases for different software tools.Each episode contains an average of 8.8 multi-turn conversations.
Trajectory Evaluation: OSWorld trajectories are evaluated with GPT-4o step-level judges, and sampled Chrome trajectories provide training data for GUI-Judge.The same judging template is used during GUI-Judge training and autonomous exploration.
Curriculum Data: The task-buffer update uses prior software manuals, actor performance, and exploration captions to generate new curriculum tasks and updated software documents.The generated tasks are used by the actor in the next reinforcement-learning phase.
Self-Documentation: Self-exploration produces usage manuals for GIMP, LibreOffice Impress, LibreOffice Writer, and VScode.These manuals document software knowledge accumulated by the navigator.
Web Evaluation: The web-agent judge prompt uses task goals, current URLs, actions, thought processes, and sequential key-frame screenshots as evaluation inputs.The prompt is designed for judging web-navigation trajectories.
Broader Impacts: The broader-impact discussion identifies misuse risks from autonomous software operation, including unauthorized automation, spamming, and surveillance.It also notes possible software-specific bias amplification and proposes controlled release, behavior filters, and World State Model safeguards.

L SEAgent Self-Evolution Algorithm

SEAgent repeatedly explores unfamiliar software, evaluates action-level outcomes, updates its policy from successful and failed actions, and expands its curriculum with increasingly challenging tasks. The loop produces a specialized policy after multiple self-evolution phases and supports automatically documented software knowledge.

Algorithm Overview: The algorithm comprises task initialization, autonomous exploration and effect evaluation, reinforcement fine-tuning, and task update.These four stages define the core self-evolution training loop.
Task Initialization: SEAgent begins by captioning the initial GUI state and generating executable tasks plus an editable software guidebook.The World State Model extracts structural semantics such as menu bars and buttons for initialization.
Autonomous Exploration and Effect Evaluation: During exploration, the World State Model labels each action correct or incorrect and generates captions describing GUI state changes.These signals provide supervision for policy learning and later task generation.
Policy Update via Reinforcement Fine-Tuning: SEAgent reinforces positive actions with GRPO and suppresses failure-prone behaviors through Adversarial Imitation.The trajectory data are split into positive and negative action steps before the two losses are applied.
Task Update: The Curriculum Generator uses trajectory feedback and GUI state transitions to create more diverse and challenging tasks for subsequent exploration phases.The updated tasks and guidebook incorporate newly acquired software knowledge and performance feedback.
Self-Evolution Outcome: Repeated curriculum phases yield a specialized policy capable of mastering complex operations in the target software environment.Algorithm 1 identifies the resulting policy as πP after P self-evolution stages.
Self-Documentation: The self-evolution system produces automatically documented usage manuals during exploration across multiple software applications.Examples are shown for VScode, GIMP, LibreOffice Impress, and LibreOffice Writer.