Source-linked AI summary
From Interaction Traces to Persistent Skills: Online Evolution for Computer-Use Agents
Longtao Hu, Xiao Liang, Linchao Zhu
TL;DR
Computer-use agents rarely retain procedural experience systematically, and the incremental value of evolving skill libraries remains insufficiently characterized. The paper introduces a persistent, versioned library updated from trajectories and evaluator feedback under frozen snapshots, then compares it with a matched empty-library control. Full scores higher after warm-up in every observed domain, while domain variation and provenance analyses show that benefits are conditional and revision does not guarantee recovery.
Problem
Computer-use agents do not systematically retain and reuse procedures, while the incremental value and longitudinal dynamics of evolving skill libraries remain insufficiently characterized.
Method
The framework converts interaction trajectories and evaluator feedback into a persistent, versioned skill library updated between frozen-snapshot iterations without changing model parameters.
Results
Full achieves a higher post-warm-up mean evaluator score than the matched empty-library control in all four observed domains, with differences of 5.7–18.6 percentage points.
Takeaways & Limitations
Evolving libraries can add measurable value to a fixed computer-use stack, but benefits vary by domain and repeated accepted edits do not necessarily recover failed tasks.
Takeaways & Limitations
Each condition–domain pair has one independently executed run, warm-up imbalance is a confound, repeated task sets test within-set adaptation, and detailed provenance covers only GIMP.
Abstract
from arXiv · showhide
Computer-use agents can execute increasingly complex tasks in graphical interfaces, but their interaction experience is typically transient: procedural knowledge acquired from one rollout is not systematically retained, refined, and reused in later tasks. Existing skill libraries provide external procedural knowledge, yet their incremental value over the same agent operating without skills, as well as their longitudinal dynamics under repeated interaction, remain insufficiently characterized. We present an online skill-evolution framework that converts interaction trajectories and evaluator feedback into a persistent, versioned library of reusable procedures. Each iteration executes against a frozen library snapshot, and evidence-guided skill updates become available in subsequent iterations without changing model parameters. We compare the full evolving-library system with a configuration-matched empty-library control across four OSWorld application domains under the same fixed action-generation and GUI-grounding stack, task sets, and iteration horizons. Following a five-iteration empty-library warm-up, Full attains a higher post-warm-up mean evaluator score in all four observed domain runs, with mean differences ranging from 5.7 to 18.6 percentage points and domain-dependent temporal stability. In GIMP, provenance-aware analysis reveals retrieval across task-of-origin boundaries and revision churn, where repeated accepted edits fail to recover the originating task. These findings characterize evolving skill libraries as auditable, shared procedural memory that can improve a fixed computer-use stack, while showing that their benefits are conditional and repeated revision does not guarantee recovery. Code is released at https://github.com/LongtaoHu/Skill-Evo4GUI.
1 Introduction
The paper addresses the transient nature of computer-use experience by introducing an evolving external skill library and evaluating whether it adds value over an empty-library control.
- Motivation: Computer-use agents do not systematically retain successful procedures, failure evidence, or application-specific knowledge after a rollout.This can lead agents to rediscover procedures or repeat failures across interactions.
- Approach: The framework converts trajectories and evaluator feedback into a persistent, versioned library while keeping action-generation and GUI-grounding models fixed.Execution uses frozen library snapshots, and accepted updates become available only in subsequent iterations.
- Results: Full achieves higher post-warm-up mean evaluator scores than the empty-library control in all four observed domains, with differences of 5.7–18.6 percentage points.The comparison uses the same fixed stack, task sets, and iteration horizons.
- Results: The performance difference varies in magnitude and temporal stability across domains, so evolving-library benefits are not uniform.The longitudinal comparison characterizes the integrated system as useful but conditional.
- Provenance: Provenance analysis in GIMP finds retrieval across task-of-origin boundaries and revision churn in which repeated accepted edits fail to recover the originating task.These analyses examine how skills are reused and revised over time.
2 Related Work
Prior work develops external memories, reusable procedures, evolving skill libraries, and evaluations of skill use, motivating longitudinal study of persistent procedural knowledge in grounded agents.
- Interactive environments and execution foundations: Interactive-environment benchmarks and computer-use systems provide reproducible settings for evaluating long-horizon GUI interaction.WebArena targets functional websites, while OSWorld targets open-ended tasks in desktop applications.
- Reusable procedural knowledge: External-memory systems retain feedback, distilled insights, executable code skills, or reusable routines beyond raw interaction trajectories.Examples include verbal reflections, experience distillation, executable skills, and workflow memory.
- Self-improving skill libraries: Self-improving libraries make skill creation, editing, retention, or co-evolution explicit learning processes.These systems differ in whether updates rely on reinforcement learning, failure analysis, validation, curation, or linked experience.
- Evaluating skill use: Skill evaluations distinguish possessing a skill from retrieving, following, and respecting the boundaries of a relevant procedure.This motivates recording retrieval coverage and downstream outcomes during persistent-library evolution.
3 Method
The method separates frozen-snapshot execution from evidence-guided library evolution, using structured trajectory facts, provenance telemetry, and validated versioned mutations while keeping the runtime stack fixed.
- Iterative setting: Each iteration sweeps a fixed task set under one read-only skill snapshot shared by all tasks.Full and Empty use the same task order, while only Full can accumulate persistent skills.
- Iterative setting: The first five observations occur under empty snapshots, and accepted Full mutations after iteration 4 form the first non-empty execution snapshot.Updates are withheld from execution until the next iteration.
- Controls and scope: Empty preserves the runtime skill interface but clamps the library to ∅, while online evolution excludes within-rollout adaptation and parameter training.The two conditions are separate longitudinal runs, so warm-up trajectories can differ.
- Runtime execution: The fixed Executor selects GUI tool calls and the fixed Grounding model maps coordinate descriptions and screenshots to pixel actions.Skills provide procedural guidance but do not execute GUI actions directly.
- Runtime execution: Progressive retrieval exposes only skill names and descriptions initially, with complete procedures retrieved on demand through get_skill.This keeps the entire library out of the Executor context.
- Trace abstraction: The Extractor converts trajectories, screenshots, evaluator scores, postcondition checks, and telemetry into structured facts for cross-iteration diagnosis.Recorded evaluator scores override model restatements, and Empty runs extraction without evolving the library.
- Provenance: Provenance records origin tasks, contributing edits, consumer tasks, delivered revisions, and observational downstream outcomes for each retrieval.These records support reuse and revision tracing but not causal attribution of execution outcomes.
- Skill evolution: The Proposer selects create, edit, delete, no-op, or unresolved actions from up to five iterations of task facts and live-library metadata.The Coordinator validates proposals before mutations, and accepted changes are committed as versioned skill documents.
4 Experiments
The experiments compare Full evolving-library and matched Empty-Library conditions across four OSWorld application domains using fixed task sets, horizons, and Executor–Grounding components. Full shows higher post-warm-up mean scores in every observed domain, but the advantage varies over time and cannot be isolated causally in these runs.
- Experimental setup: The study uses four OSWorld domains with fixed task sets, repeated task order, matched iteration horizons, and one independent longitudinal run per condition–domain pair.The evaluated domains are GIMP, VLC, LibreOffice Writer, and Thunderbird.
- Comparison protocol: Full and Empty use the same Executor–Grounding stack, prompts, catalog interface, get_skill tool, task order, and iteration horizons; only Full evolves its skill library.Full updates its library, while Empty keeps the catalog empty and disables proposal and skill-building components.
- Longitudinal performance: During t ≥5, Full scores higher than Empty Control in all four domains, with mean differences ranging from 5.7 to 18.6 percentage points.VLC has the largest and relatively stable margin; Writer and Thunderbird are more variable, while GIMP has the smallest margin and repeated crossings.
- Longitudinal performance: The trajectories show domain-dependent, nonmonotonic differences rather than uniform improvement at every iteration.The longitudinal curves therefore vary in both magnitude and temporal stability across domains.
- Interpretation limits: Pre-period differences already separate the independently executed conditions, limiting attribution of post-warm-up gains to skill availability.Writer begins with a +9.1-point difference, while GIMP and Thunderbird begin below Empty; the study is descriptive rather than a significance test or isolated causal estimate.
- Skill retrieval and revision: In GIMP, 82.4% of valid rollouts invoke get_skill, and 43.3% of recorded skill calls retrieve skills originating from another task.These measures indicate frequent retrieval and cross-task library use, but retrieval telemetry does not establish that procedures influenced subsequent actions.
- Skill retrieval and revision: Repeated accepted edits to gimp-add-alpha-channel yield success in only 2 of 35 iterations, illustrating revision churn without recovery of the originating task.The pattern shows active but ineffective evolution and is consistent with a bottleneck in the fixed Executor–Grounding stack, without identifying the failing component.
5 Discussion and Limitations
The evolving-library system adds measurable but domain-dependent value to a fixed computer-use stack, while warm-up imbalance, limited provenance, and serial evolution constrain causal and broader interpretation.
- Findings: Full outperforms the empty-library control post-warm-up in all observed domains, but magnitude and temporal stability vary by domain.Writer already shows a substantial positive offset during empty-library warm-up, limiting causal interpretation of raw post-warm-up differences.
- Findings: Cross-origin retrieval shows stored procedures are consulted beyond their tasks of origin, but retrieval does not establish that the Executor follows or benefits from them.The evidence supports shared procedural memory, not successful execution or utility from every retrieved skill.
- Findings: Repeated accepted edits can produce revision churn without recovering the originating task, consistent with an unresolved execution or localization bottleneck.The limiting component could be skill selection, action generation, or grounding; the case does not identify which.
- Limitations: One independently executed run per condition-domain pair leaves stochastic variation unestimated, while warm-up imbalance remains a confound.These constraints limit broader causal interpretation of the longitudinal comparison.
- Limitations: Repeated fixed task sets measure within-set adaptation rather than unseen-task transfer, and detailed provenance currently covers only GIMP.External OSWorld scores are contextual, and cross-origin retrieval remains observational.
- Limitations: Serial evolution is order-dependent and lacks automatic consolidation or rollback.The paper identifies replicated runs, held-out tasks, matched component ablations, and direct adherence measurement as priorities.
- Future work: Future work should test native action generation without an additional Grounding module and learn mutation policies for downstream utility.Reinforcement-learning-based optimization is proposed because plausible accepted mutations need not improve future utility.
6 Conclusion
The paper presents evolving skill libraries as persistent, auditable adaptation layers for computer-use agents. Across four OSWorld domains they improve post-warm-up scores over a matched empty condition, but benefits vary and repeated revision does not guarantee task recovery.
- Contribution: The framework converts interaction trajectories and evaluator feedback into persistent, versioned procedural memory without changing model parameters.Iteration-frozen snapshots separate execution from evidence-guided skill creation and revision, while provenance records make the lifecycle auditable.
- Results: Full records higher post-warm-up mean evaluator scores than the matched Empty condition in every observed run across four OSWorld application domains.The magnitude and temporal pattern vary, and warm-up offsets limit causal interpretation.
- Implications: Cross-origin retrieval shows skills are consulted beyond their tasks of origin, while revision churn shows accepted edits can fail to recover the originating task.Together, these findings support shared, auditable adaptation layers whose benefits remain domain-dependent rather than guaranteed.