Source-linked AI summary
ContractSkill: Repairable Contract-Based Skills for Multimodal Web Agents
Zijian Lu, Yiping Zuo, Yupeng Nie, Xin He, Weibei Fan, Lianyong Qi, Shi Jin
TL;DR
Self-generated web skills are unstable because their implicit structure prevents reliable checking and local repair. ContractSkill converts drafts into explicit executable artifacts, uses deterministic verification and fault localization for minimal patching, and finds consistent gains across VisualWebArena and MiniWoB while supporting within-family cross-model reuse. Its evidence remains bounded to benchmark-family portability and a browser-focused artifact schema.
Problem
Self-generated web skills can be unstable or underperform direct acting because implicit drafts omit executable structure needed for checking and local repair.
Method
ContractSkill converts a draft into an explicit executable artifact, verifies it deterministically, localizes faults, and applies minimal patches to the implicated skill component.
Results
ContractSkill consistently outperforms self-generated skills across VisualWebArena and MiniWoB, while repaired artifacts remain reusable across GLM-4.6V and Qwen3.5-Plus.
Takeaways & Limitations
The results support treating web-agent skills as external procedural objects that can be audited, repaired, accumulated, and shared across models.
Takeaways & Limitations
The scope is limited to single browser-interaction skill artifacts, and transfer demonstrates portability within the same benchmark family rather than full-benchmark generalization.
Abstract
from arXiv · showhide
Self-generated skills for web agents are often unstable and can even hurt performance relative to direct acting. We argue that the key bottleneck is not only skill generation quality, but the fact that web skills remain implicit and therefore cannot be checked or locally repaired. To address this, we present ContractSkill, a framework that converts a draft skill into an executable artifact with explicit procedural structure, enabling deterministic verifica tion, fault localization, and minimal local repair. This turns skill refinement from full rewriting into localized editing of a single skill artifact. Experiments on VisualWebArena show that Contract Skill is effective in realistic web environments, while MiniWoB provides a controlled test of the mechanism behind the gain. Under matched transfer layers, repaired artifacts also remain reusable after removing the source model from the loop, providing evi dence of portability within the same benchmark family rather than full-benchmark generalization. These results suggest that the central challenge is not merely generating skills, but mak ing them explicit, executable, and repairable. Code is available at https://github.com/underfitting-lu/contractskill.git.
1 Introduction
ContractSkill argues that self-generated web skills fail because they remain implicit, unstable objects rather than executable artifacts. It converts each draft into a verifiable skill that can be diagnosed, locally repaired, and reused across models.
- Motivation: Self-generated skills can fail because loose textual drafts omit preconditions, step boundaries, success evidence, and recovery policies.When execution fails, existing practice often rewrites the whole skill or abandons it.
- Positioning: The framework targets the single skill artifact itself, distinguishing its repair objective from methods that update trajectories, memories, transient plans, or larger workflow collections.Its artifact includes explicit procedural structure that can persist beyond the current episode.
- Approach: ContractSkill compiles a model-generated draft into an explicit contract and repairs only the implicated selector, condition, recovery rule, or action argument.The artifact is improved through deterministic verification and fault localization rather than whole-skill regeneration.
- Evaluation: ContractSkill improves realistic web execution on VisualWebArena and shows controlled gains in MiniWoB attributable to verification, fault localization, and minimal repair.The evidence combines realistic benchmark evaluation with a controlled mechanism test.
- Evaluation: Repaired artifacts remain reusable across source and target models under matched transfer layers, but this is benchmark-specific portability rather than full-benchmark generalization.The limitation is stated explicitly for the transfer evidence.
2 Related Work
Prior work improves web-agent behavior through online reflection or reusable procedural knowledge, but ContractSkill focuses on making one model-written skill executable, diagnosable, minimally repairable, and reusable.
- Benchmark Context: Web-agent benchmarks increasingly evaluate longer-horizon, realistic, and visually grounded workflows, exposing failure modes beyond basic browser interaction.The benchmark landscape spans controlled settings such as MiniWoB and more realistic environments such as WebArena and VisualWebArena.
- Online Refinement: Reflection and self-repair methods revise trajectories, reasoning traces, memories, or policies after failure, usually updating current-episode behavior or internal state.Their updated object differs from a persistent skill artifact.
- Procedural Knowledge: Reusable-skill research externalizes procedural knowledge through workflows, executable skills, APIs, hierarchical memories, or retrieved GUI guidance.These approaches treat skills as explicit reusable knowledge rather than transient prompting context.
- Positioning: ContractSkill narrows the target to a single model-written multimodal web skill that must become executable, diagnosable, minimally repairable, and reusable across models.This distinguishes the framework from workflow collections, APIs, and broader memory structures.
- Repair Lens: Program-repair methods provide the methodological lens by combining failure localization, constrained edits, and verifier-based patch acceptance.These principles motivate ContractSkill’s repair process.
3 Problem Definition
The problem is to turn a weak textual web-skill draft into a structured artifact that can execute against multimodal observations and be checked, diagnosed, repaired, and reused. ContractSkill formalizes this artifact and searches for local repairs that balance success, execution cost, and edit size.
- Environment: The agent receives a natural-language task and multimodal observations combining screenshots with structured page summaries, then emits browser actions.The structured summary may include DOM, accessibility, or textual page-state information.
- Environment: A deterministic verifier judges success through reproducible final and intermediate page-state signals such as URLs, DOM matches, text presence, form values, and transitions.This makes failure analysis programmable rather than dependent on free-form language judgments.
- Draft Skill: The initial draft skill is a weak procedural object because it may omit hidden-state assumptions, underspecify selectors, conflate goals with actions, or omit completion evidence.The draft is treated as a starting point rather than the final reusable skill.
- Contracted Skill Artifact: The contracted artifact exposes explicit goals, preconditions, ordered steps, postconditions, recovery rules, termination checks, selectors, actions, arguments, and post-assertions.This step-level structure gives the verifier hooks for execution and diagnosis.
- Repair Objective: Local repair searches a neighborhood of candidate artifacts while maximizing verified success and penalizing execution cost and unnecessary edits.The objective preserves as much original procedural content as possible while improving the artifact.
4 ContractSkill
ContractSkill turns each draft skill into an explicit executable artifact with deterministic verification, step-level diagnosis, and minimal local repair. Its repair loop preserves unaffected portions while validating only patches that improve the current score.
- Contract representation: Each skill artifact specifies a name, goal, preconditions, steps, postconditions, recovery policy, and termination condition.Every step must include a selector, action, arguments, and postcondition assertion.
- Deterministic verification: The verifier checks both final task success and intermediate step validity using deterministic page-state evidence.Checks include URL matching, DOM retrieval, textual assertions, form values, and state-change predicates.
- Fault localization: Failure diagnosis identifies the failed step, structured error code, and local trace evidence before repair selection.Different diagnoses map to different repair actions, such as correcting a selector or inserting a missing precondition.
- Minimal patch repair: ContractSkill uses five local patch operators: selector replacement, precondition insertion, postcondition insertion, recovery insertion, and argument correction.Each operator targets the artifact fragment associated with the diagnosed failure.
- Minimal patch repair: The repair neighborhood is restricted to small edits, preserving the rest of the artifact instead of rewriting the full skill.This local design is intended to avoid instability from full-text rewrites.
- Repair loop: The execute-diagnose-patch-validate loop accepts a candidate only when it improves the current verifier score, typically within a one-to-three-patch budget.Candidates are validated under the same verifier used to diagnose failure.
5 Experimental Setup
The evaluation tests ContractSkill on realistic and controlled web benchmarks, using fixed interfaces, budgets, and deterministic verification. It also examines whether repaired artifacts remain usable by another model without regenerating the skill.
- Experimental questions: The experiments ask whether ContractSkill improves execution, whether local verifier-guided repair explains the gain, and whether repaired artifacts transfer across models.Efficiency is evaluated as a success-cost trade-off.
- Benchmarks: VisualWebArena provides the realistic multimodal setting, while MiniWoB provides a controlled benchmark with simpler task structure.The study uses VWA main_100 and MiniWoB main_20 with 10 instances per task, evaluated over three runs.
- Transfer design: Transfer uses benchmark-specific matched layers covering shared source-model successes and tasks solved by only one source model.These layers test reuse beyond the overlap of easy cases within each benchmark family.
- Agent configurations: In self-use, one model generates, repairs, and executes the artifact; in transfer, a target model executes an artifact repaired by a different source model.The transfer setup removes skill regeneration by the target model.
- Baselines: The main baselines are No-Skill direct acting and Self-Generated Skill without verifier-guided repair.The latter isolates skill writing from explicit verification and repair.
- Metrics: Success rate is the primary metric, supplemented by average steps and average total tokens when available.Transfer reports absolute success and point improvement against the target model’s self-generated-skill baseline.
- Controls: All methods share the same observation interface, action space, deterministic verifier, execution budget, and benchmark-specific repair limit.Selectors are re-resolved on the latest page snapshot after each action.
6 Main Results
ContractSkill improves success on VisualWebArena and MiniWoB relative to direct acting and self-generated skills, with especially clear gains in the controlled MiniWoB setting. Repaired artifacts also remain reusable under matched transfer layers after the source model is removed from the loop.
- VisualWebArena: On VisualWebArena, ContractSkill reaches 28.1% success with GLM-4.6V versus 12.5% for No-Skill and 9.4% for Self-Generated Skill.Its average steps fall to 3.62 from 12.59 for No-Skill, while tokens fall to 46.45k from 52.7k.
- VisualWebArena: On VisualWebArena, ContractSkill reaches 37.5% success with Qwen3.5-Plus versus 28.1% for No-Skill and 10.9% for Self-Generated Skill.Average token usage is 40.1k versus 42.6k for No-Skill.
- MiniWoB: On MiniWoB, ContractSkill reaches 77.5% success for GLM and 81.0% for Qwen, improving over Self-Generated Skill by 11.0 and 20.5 points.The gains occur without matching the higher average-step cost of No-Skill.
- MiniWoB: MiniWoB results support verifier-guided local repair as the source of improvement rather than rewriting alone or additional interaction budget.ContractSkill uses 1.93 average steps and 5.16k tokens on GLM, compared with 4.11 steps for No-Skill.
- Transfer: Under matched transfer layers, VWA success rises from 32.6% to 80.4% and MiniWoB success from 83.5% to 96.2% against target-model self-generated skills.These results concern artifact reuse after removing the source model, not unrestricted cross-benchmark generalization.
7 Analysis and Ablation
MiniWoB ablations show that ContractSkill’s gains come from verifier-guided fault localization and constrained local repair, not rewriting alone. The improvement is concentrated on templates requiring reliable continuation, while some templates remain unsolved or favor no-skill execution.
- Failure Localization and Minimal Repair: 62.0% on GLM and 60.5% on Qwen for Text-Only Rewrite trail full ContractSkill at 77.5% and 81.0%, respectively.The comparison indicates that rewriting after failure is insufficient without a contracted artifact, verifier feedback, and a localized repair target.
- Failure Localization and Minimal Repair: A representative VWA repair inserts only the missing click and answer steps after exploration, preserving the rest of the strategy.The verifier identifies a missing continuation, enabling a targeted patch rather than a full rewrite.
- Failure Localization and Minimal Repair: 65.0% on GLM and 70.0% on Qwen after removing failure localization, while unconstrained editing reaches 68.5% and 70.5%.On Qwen, unconstrained repair also raises average token usage from 7.9k to 11.3k without recovering the lost success.
- Template-Level MiniWoB Patterns: ContractSkill’s MiniWoB advantage concentrates on login templates and mw_m3_click_menu_2, which require reliable continuation after partial progress.Many easy templates are saturated across methods, so aggregate gains do not reflect uniform improvement.
- Template-Level MiniWoB Patterns: Some templates remain unsolved for every method, while no-skill execution outperforms skill variants on mw_m4_click_collapsible.Structured repair helps when a reusable procedural skeleton exists, but it is not a universal substitute for exploration.
- Failure Localization and Minimal Repair: Across VWA models, ContractSkill shifts residual failures from broad late-stage profiles toward earlier, more localizable selector or assertion errors.The analysis reports fewer step-6+ and loop-timeout-heavy failure mixes relative to No-Skill.
8 Discussion and Limitations
ContractSkill frames web skills as explicit procedural objects that can be checked, repaired, and reused rather than rewritten from scratch. Its current scope remains limited to single browser-oriented artifacts, within-family cross-model portability, and settings where deterministic verification is feasible.
- Discussion: ContractSkill treats failures as structured repair problems over explicit artifacts rather than prompts requiring whole-skill rewriting.The framework externalizes skills so they can be checked, repaired, and reused.
- Limitations: The study focuses on single multimodal web skill artifacts, not full lifelong skill libraries.The stated scope excludes broader lifelong skill-bank analysis.
- Limitations: The artifact schema is tailored to browser interaction and does not yet cover desktop environments such as OSWorld or Agent S.
- Limitations: Cross-model reuse is evaluated within the same benchmark family, making portability meaningful but limited rather than full-benchmark generalization.
- Limitations: Deterministic verification is strongest when success can be checked programmatically through URL, DOM state, or form values.Visually ambiguous or highly dynamic web pages remain harder to verify reliably.
9 Conclusion
ContractSkill converts draft textual skills into contracted executable artifacts using deterministic verification, step-level fault localization, and minimal patch repair. Across VisualWebArena and MiniWoB, it outperforms self-generated skills, while repaired artifacts remain reusable across GLM-4.6V and Qwen3.5-Plus.
- Conclusion: ContractSkill turns draft textual skills into contracted executable artifacts with deterministic verification, step-level fault localization, and minimal patch repair.
- Conclusion: Across VisualWebArena and MiniWoB, ContractSkill consistently outperforms self-generated skills.
- Conclusion: Repaired artifacts remain reusable across GLM-4.6V and Qwen3.5-Plus.
- Conclusion: The results support treating web-agent skills as external procedural objects that can be audited, repaired, accumulated, and shared across models.