Reflect, Revise, Reuse: Training-Free Skill Evolution for GUI Agents

1 Introduction

• GUI execution is non-stationary, while many skill designs are not tailored to GUI dynamics and remain static after deployment.Pop-ups, delayed loads, and relocated widgets can invalidate plans fixed before execution.Source: §1 · p. 1
• EvoSkill-GUI treats failures as actionable feedback for revising deployed skills, rather than discarding them or reserving them for future training.Grounding errors, unexpected pop-ups, and redundant steps can reveal which skill components need correction.Source: §1 · p. 2
• Its reflect-revise-reuse loop combines instant rollout revisions with isolated critique and tool-restricted edits by the same backbone model.The critic diagnoses failed trajectories under strict information isolation, and the executor edits specific skill files.Source: §1 · p. 2
• Ablations confirm the importance of structured packages, instant revision, and reflection-driven edits; analyses indicate evolved libraries benefit related tasks.The libraries can continue to help related tasks rather than being rebuilt from scratch.Source: §1 · p. 3
• Each skill package separates retrieval metadata, executable plans, backup localization, recovery rules, accessibility utilities, and failure examples.The structured files support reuse and updating at inference time.Source: §1 · p. 3
• Up to +16.2%, +6.0%, and +10.5% gains on MobileWorld, AndroidWorld, and OSWorld, respectively, improve multiple base models without training.The framework is evaluated across three GUI benchmarks spanning mobile and desktop environments.Source: §1 · pp. 2–3

2.1 Agent Skills

• Existing agent-skill work spans structured artifacts, executable guidance, multimodal evidence, continual learning, memory, and reinforcement-learning-based skill learning; structured memory systems focus on memory construction rather than skill revision.The section frames inference-time revision of deployed skills as largely unexplored.Source: §2.1 · p. 3

2.2 Self-evolving Agents

• Experience-driven agents commonly update memory, policies, or model weights, whereas EvoSkill-GUI revises deployed structured GUI skills from failure feedback without additional training.This distinguishes skill-level revision from other forms of agent self-improvement.Source: §2.2 · p. 3

3.1 Problem Formulation

• EvoSkill-GUI optimizes reusable skill packages for expected returns across related tasks in partially observable GUI environments.A task is modeled with screen states, GUI actions, transitions, observations, and a terminal completion reward.Source: §3.1 · pp. 3–4

3.2 Skill Package Representation

• The package separates retrieval metadata, accessibility utilities, executable plans, backup localization, recovery rules, and failure cases for targeted revision.Grounding failures can update localization strategies, missing contingencies recovery rules, and procedural errors the plan.Source: §3.2 · p. 4

3.3 Reflect-Revise Loop

• During rollout, instant edits correct local mismatches; after failure, an isolated critic diagnoses trajectory-level errors and the executor revises specific skill files.The critic shares the executor’s model parameters but works in a separate session with restricted information; failed rollouts can also be recorded as failure cases.Source: §3.3 · p. 5

3.4 Skill Retrieval and Reuse

• Metadata-based retrieval ranks skills for a new instruction, reusing a package above the threshold or creating a new one when no candidate qualifies.A newly created package enters the library after successful execution or revision through the self-evolution loop.Source: §3.4 · p. 5

4.1 Experimental Setup

• Evaluation spans AndroidWorld, long-horizon cross-app MobileWorld workflows, and OSWorld, using general-purpose and GUI-specialized models with up to 50 interaction steps per task.After a failed attempt, agents may revise skills for at most two rounds.Source: §4.1 · pp. 5–6

4.2 Main Results

• On GUI-only MobileWorld, EvoSkill-GUI improves every evaluated model: Claude-Sonnet-4.6 rises from 57.1% to 67.6%, and Qwen3.6-Plus from 53.3% to 69.5%.The figure compares performance against representative reference baselines.Source: Figure 3 · p. 6, §4.2 · p. 6
• On AndroidWorld seed 42, success rises by +6.0%; on OSWorld, overall success reaches 54.8% (+8.1) for GUI-Owl-1.5-8B and 34.3% (+10.5) for Qwen3-VL-8B-Instruct.The AndroidWorld results also report a +2.6% gain under seed 30.Source: §4.2 · p. 6, Table 2 · p. 7, Table 1 · p. 7

4.3 Ablation Study

• On MobileWorld, structured packages improve success from 66.67% to 69.52% (+2.85), while removing instant revision lowers it from 69.52% to 62.86% (−6.66).Structured files separate planning, localization, and recovery so revisions can target a relevant component; instant revision addresses mismatches before they cascade.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
• Removing information isolation lowers MobileWorld success from 69.52% to 60.95% (−8.57); accessibility-tree inputs improve three evaluated models by +3.80, +1.90, and +2.85.The accessibility gains are reported for Qwen3.6-Plus, Qwen3.6-35B-A3B, and MAI-UI-8B, respectively.Source: Table 4 · p. 7, §4.3 · pp. 7–8
• Metadata retrieval scores 0.88 versus 0.41 for full-text retrieval and recovers the correct skill in all 12 reuse cases; retrieval threshold 0.6 outperforms both 0.4 and 0.8.At thresholds 0.4 and 0.8, success is 55.2% and 66.7%, respectively; at 0.6 it is 69.5%.Source: §4.3 · pp. 7–9, Table 5 · p. 8, Table 6 · p. 9

4.4 Analysis

• On MobileWorld, three rounds raise Qwen3.6-Plus success by 12.4 points; two additional rounds add 0.9 point, while three-round evolution beats pass@3 with fewer tokens.Three-round evolution achieves 69.5% using 94.75M tokens, versus 62.8% using 103.00M tokens for three independent rollouts.Source: Figure 6 · p. 9, §4.4 · p. 9
• On related AndroidWorld tasks, Phase 2 success rises from 55.2% to 61.2% with all tasks reusing the evolved library; revisions recover 27 of 89 initially failed executions.Plan-level errors are most repairable (16/23); grounding errors are partially repaired (6/12), while missing contingencies are harder to fix (1/4).Source: §4.4 · pp. 9–10, Table 7 · p. 9, Table 9 · p. 10

• EvoSkill-GUI turns deployed GUI skills from static artifacts into procedural knowledge that can be revised from execution feedback.Source: §5 · p. 10
• A structured multi-file skill package and reflect-revise-reuse loop support skill construction, retrieval, execution, and revision at inference time.Source: §5 · p. 10
• Across MobileWorld, AndroidWorld, and OSWorld, EvoSkill-GUI consistently improves multiple base models without training.Source: §5 · p. 10

• Unreliable backbone perception or trajectory diagnosis may produce edits that miss failure causes or regress verified skill packages.Source: Limitations · p. 11
• The framework has no formal verifier to veto harmful edits, and evaluation covers three representative GUI benchmarks rather than the fuller complexity of real deployment.Source: Limitations · p. 11
• AndroidWorld reuse is measured on parameterized variants of the same task family, so transfer to genuinely unseen application categories may be weaker.Source: Limitations · p. 11
• Each failed rollout adds one critic call and one revision call, an overhead that should be amortized in latency-critical deployment.Source: Limitations · p. 11

• The experiments use benchmark GUI task execution and do not collect personal user data.Source: Ethics Statement · p. 11
• Real-application deployment may affect accounts, files, settings, or third-party services, motivating authorization, sensitive-action confirmation, sandboxing, and audit logs.Source: Ethics Statement · p. 11
• The authors identify skill review, versioning, human verification, and fail-safe rules as next steps for responsible adoption.Source: Ethics Statement · p. 11
• The authors state that the research ideas, framework design, protocols, analysis, and conclusions are their own.Source: Ethics Statement · p. 11

• API-based models use official provider endpoints with deterministic decoding; local open-weight models use vLLM on one RTX PRO 6000 GPU 96GB server.Source: §A.1 · p. 14
• Each rollout allows up to 50 interaction steps with a 3-second per-step wait, while self-evolution permits at most three execution iterations.Source: §A.1 · p. 14
• MobileWorld runs use 20 Docker containers in parallel, a 20-second launch interval, and up to 50 interaction rounds per task.Source: §A.1 · p. 14
• The experiments follow benchmark-default settings; MobileWorld uses model-dependent image resizing and normalized action coordinates on a 0–1000 scale.Source: §A.1 · p. 14
• Twelve Google-proxy tasks are excluded from MobileWorld evaluation for stability.Source: §A.1 · p. 14

A.2 Held-out skill transfer and retrieval quality

• 73.3% (22/30) of held-out tasks are solved by EvoSkill-GUI, versus 40.0% (12/30) for pass@3 without skills.The evaluation uses a library built from 87 source tasks and 30 held-out tasks.Source: Table 10 · p. 14, §A.2 · p. 14
• Five of EvoSkill-GUI’s ten additional held-out successes come from tasks retrieving an existing skill, which solve 12/16 tasks versus 7/16 for pass@3.The other five additional successes come from tasks where no skill is retrieved and a new package is generated.Source: §A.2 · p. 14
• Among 16 held-out tasks triggering retrieval, 15 retrieve an appropriate skill (93.8%), while one retrieves a mismatched skill (6.3%).The mismatch is MattermostVisualInstructionResponseTask retrieving an unrelated skill; mismatches remain a potential source of negative transfer.Source: §A.2 · p. 14

A.3 Same-backbone self-diagnosis

• The same-backbone design avoids external-critic supervision and additional model deployment, though critic calls still incur inference costs.An external critic would confound self-evolution with supervision from a more capable model.Source: §A.3 · pp. 14–15
• The critic operates in a separate session and receives only the instruction and execution trajectory, not the skill package, executor CoT, or ground truth.It must diagnose failures from observable trajectory evidence rather than inherit or rationalize the executor’s plan.Source: §A.3 · p. 15
• 80.4% (115 of 143) of manually annotated critic judgments are correct across 56 failed tasks.The audit records 28 incorrect judgments (19.6%).Source: Table 11 · p. 15, §A.3 · p. 15
• In at least four cases, critic overoptimism misjudges a failed task as successful and terminates evolution prematurely.The same-backbone critic therefore provides useful, but not infallible, diagnostic signals under information isolation.Source: §A.3 · p. 15

A.4 Analysis of negative OSWorld results

• Only 3 of 369 evaluated tasks across two domain/model entries show negative changes, too few for reliable general conclusions; small-domain differences warrant caution.For GUI-Owl-1.5-8B on OS, two reruns both achieve 79.2%, while the submitted negative entry reflects one additional failure versus baseline.Source: §A.4 · p. 15
• EvoSkill-GUI is most beneficial for long workflows with stable interface structure; imperfect skill guidance may add steps or encourage incorrect procedures on direct tasks.This mismatch plausibly explains occasional negative transfer on short, precise tasks.Source: §A.4 · p. 15

A.5 Token cost under repeated sampling.

• 94.75M total tokens (0.810M per task) are within the pass@3 baseline budget of 103.00M total (0.880M per task) on MobileWorld.The revised accounting excludes double-counted intermediate cumulative snapshots.Source: §A.5 · p. 15, Table 13 · p. 16
• Accessibility-tree inputs leave token usage largely comparable but consistently increase execution time through additional UI-information collection and processing.The reported benefit is more reliable grounding signals for GUI interaction.Source: Table 14 · p. 16, §A.5 · p. 16

A.6.1 Case 1: Verification Checkpoint Insertion

• In SendInterviewEmailTask, a missing verification checkpoint led to edits requiring the agent to open and verify the resume, attach it, and confirm the attachment before sending.The second rollout succeeds after the critic identifies the procedural failure; the skill shifts from locate-only to locate–verify–attach.Source: §A.6.1 · p. 16, Figure 7 · p. 17

A.6.2 Case 2: Completeness Enforcement for List Traversal

• In CheckCartPriceTask, incomplete traversal of a 20-item cart led first to systematic scanning rules and then to a failure memory; the third rollout reports 13186 in 11 steps.The earlier answer was 2657, based on a partial scan.Source: §A.6.2 · p. 17

A.6.3 Summary

• The case studies show critic diagnoses producing targeted package-file updates, with failure memory added when a plan revision alone does not prevent a repeated mistake.The visualized paths summarize shifts to locate–verify–attach and to systematic traversal with failure memory.Source: §A.6.3 · p. 18, Case 1: SendInterviewEmailTask · p. 19, Case 2: CheckCartPriceTask · p. 20

A.7 Algorithm Pseudocode

• The pseudocode retrieves or initializes a skill, runs an evolution loop with rollout and isolated reflection, revises the package, and registers updated skills.It also includes appending failure cases to the package.Source: §A.7 · p. 21