PaperlayerView exact revision on arXiv

Source-linked AI summary

Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents

Suji Kim, Kangsan Kim, Sung Ju Hwang

arXiv:2605.28775v1cs.LGcs.AIcs.CL

TL;DR

Small computer-use agents remain weaker in domain-specific environments, and broad domain training alone offers limited gains. LearnWeak targets student weaknesses in data generation and training, achieving average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B across eight OSWorld domains.

  • Problem

    Small computer-use agents remain weaker than closed agents in domain-specific environments, while learning workflows across diverse software domains is challenging for small models.

    Source: §1 · p. 1

  • Method

    LearnWeak iteratively uses teacher-student comparisons to generate tasks targeting student weaknesses, then applies error-aware preference optimization that distinguishes planning from execution failures.

    Source: §1 · p. 2

  • Results

    11.6 and 11.1 percentage-point gains for EvoCUA-8B and OpenCUA-7B, respectively, averaged across all eight OSWorld domains.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The effectiveness of additional generation rounds depends on whether new tasks align with students’ unresolved weaknesses, rather than data volume alone.

    Source: §5.1 · p. 9

    LearnWeak assumes a reasonably reliable teacher in the target domain and a student that already has general computer-use skills.

    Source: Limitation · p. 10

Abstract

from arXiv · show

Computer-use agents (CUAs) have recently made substantial progress, but deploying a separate large expert for each software domain remains expensive. Small open computer-use agents are more practical specialization targets, but they remain substantially weaker and exhibit uneven domain-specific failures. A straightforward remedy is to synthesize large-scale training data for the target domain, yet we find that this naive approach yields only marginal improvements. Building on this observation, we introduce LearnWeak, an annotation-free specialization framework for small computer-use agents that uses a stronger reference agent to identify the student's weaknesses in the target domain, synthesize targeted tasks, and construct supervision automatically. LearnWeak further introduces an error-aware specialization objective that disentangles planning and execution errors, enabling more behaviorally precise updates than broad uniform supervision. On OSWorld, LearnWeak achieves average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively, across eight domains. We also validate that our student-aware dataset generation and training approaches outperform existing autonomous trajectory generation and training baselines. Our work highlights the importance of student awareness in both data synthesis and agent training, pointing toward a more principled and efficient path for specializing small computer-use agents in diverse domains.

1 Introduction

LEARNWEAK specializes small computer-use agents by identifying student weaknesses for targeted data generation and training, with error-aware objectives that distinguish planning from execution failures.

Motivation

  • Domain specialization focuses training on domain-specific interaction patterns because small agents struggle to learn diverse workflows simultaneously and heterogeneous training can cause forgetting.Small open agents offer faster, more cost-efficient inference and can serve edge devices and privacy-sensitive enterprises, but a performance gap with closed models persists in domain-specific software.Source: §1 · p. 1
  • Because long-horizon human trajectories are costly and fixed generation strategies ignore student deficiencies, synthesized queries should target weaknesses and missing domain knowledge.The introduction identifies dataset generation and agent training as the two stages of domain specialization.Source: §1 · p. 2
  • Naive fine-tuning can distort small agents’ reasoning and recovery patterns, while failures within one agent may arise from either planning or execution.Execution errors include inaccurate coordinates.Source: §1 · p. 2

Contributions

  • LEARNWEAK uses repeated teacher-student comparison, weakness analysis, and query synthesis to build a compact, annotation-free dataset from a small set of seed queries.The pipeline targets student deficiencies during dataset generation.Source: §1 · p. 2
  • Its error-aware preference optimization adapts training objectives to task-specific weaknesses by distinguishing planning failures from execution failures.The introduction frames this as a way to tailor training rather than applying broad uniform supervision.Source: §1 · p. 2

Evaluation

  • 11.6 and 11.1 percentage points are the average performance gains for EvoCUA-8B and OpenCUA-7B, respectively, across eight OSWorld domains.Specialized agents also surpass the teacher on several domains; under matched budgets, the data-generation pipeline achieves the strongest gains among autonomous-generation baselines, and error-aware preference optimization outperforms SFT and standard DPO variants.Source: §1 · p. 2

2 Preliminaries

The preliminaries model computer-use agents as history-conditioned policies and define domain specialization as annotation-free data generation followed by student training for target-domain success.

2.1 Computer-Use Agent

  • CUAs perceive screens and issue actions, are modeled as POMDPs, and condition their policies on the current observation and full interaction history [48] [42] [6].At each step, the policy produces internal reasoning, an action description, and executable tool actions.Source: §2.1 · pp. 2–3

2.2 Problem Formulation

  • Domain specialization finetunes a broadly pretrained student for a target domain, given a stronger teacher, K human-provided seed queries, and an executable environment with an automatic verifier.No further human annotation is assumed.Source: §2.2 · p. 3
  • The formulation couples autonomous generation of domain-specific training data from expanded seed queries and teacher trajectories with training a domain-specialized student on that data.Data generation produces training samples without human annotation.Source: §2.2 · p. 3
  • The overall objective is to maximize expected task success over the target-domain evaluation task distribution.Success is evaluated on the trajectory induced by the policy for a task query.Source: §2.2 · p. 3

3 Method

LEARNWEAK specializes small computer-use agents through annotation-free, student-aware data generation followed by teacher-guided training. Its pipeline discovers domain-specific failures, turns them into targeted tasks and step-wise supervision, and applies selective updates to correct behavior.

3.1 Weakness-Aware Data Generation (LEARNWEAK-GEN)

  • Teacher-student execution and verification identify tasks where the teacher succeeds but the student fails, isolating student-specific weaknesses for targeted generation.Verifier diagnostics are summarized into recurring failure modes, such as incorrect operation selection, inaccurate element localization, or invalid action arguments.Source: §3.1 · p. 4
  • Screenshot-grounded query synthesis combines weakness-focused tasks with exploration of unexplored functions, then iteratively shifts generation toward unresolved weaknesses while maintaining task diversity.Representative screenshots are selected for diversity and semantic informativeness to ground queries in realistic states and reduce infeasible tasks.Source: §3.1 · p. 4

3.2 Agent Training for Domain Specialization (LEARNWEAK-DPO)

  • Teacher-replay preference construction provides annotation-free, step-level supervision by comparing student responses with verified teacher actions and retaining only steps with differing tool executions.Matching teacher and student executions are filtered out, avoiding retraining steps that are already correct.Source: §3.2 · p. 5
  • Error-aware DPO distinguishes planning and execution failures and restricts preference updates to behaviorally relevant response spans rather than uniformly relearning full action sequences.The objective raises the relative likelihood of teacher actions over replayed student actions within the selected span.Source: §3.2 · p. 5
  • Domain-specific LoRA adapters specialize a shared frozen student, allowing the appropriate domain update to be activated at deployment.This modular design localizes domain knowledge in adapters while retaining the shared base policy.Source: §3.2 · pp. 5–6

4 Experiments

On OSWorld, LEARNWEAK improves both small CUA backbones across eight domains, while comparisons show that targeting student weaknesses is more effective than generic data reuse, generation, or filtering. The experiments evaluate domain-specific specialization and data-construction baselines under controlled settings.

4.1 Experimental Setup

  • Experiments use OSWorld’s eight domains, specialize each domain independently, and average success over three trials with a 50-step inference limit.EvoCUA-8B and OpenCUA-7B are students, EvoCUA-32B is the data-construction teacher, and subsequent analyses default to EvoCUA-8B.Source: §4.1 · p. 6

4.2 Domain Specialization Results

  • LEARNWEAK raises average OSWorld success from 50.69 to 62.24 for EvoCUA-8B and from 37.65 to 48.72 for OpenCUA-7B, gains of 11.6 and 11.1 percentage points, respectively.Improvements span office software, system utilities, visual editing, and coding workflows.Source: §4.2 · p. 7
  • Specialized EvoCUA-8B outperforms its 32B teacher on Gimp, Thunderbird, and VSCode, showing that corrective supervision can address student failures beyond imitation.The student uses teacher-conditioned training data to correct domain-specific weaknesses and surpass the teacher in selected domains.Source: §4.2 · p. 7
  • The strongest specialization gains vary by student: EvoCUA-8B improves most on VSCode, Gimp, Calc, and Impress, while OpenCUA-7B improves most on OS, VLC, Thunderbird, and VSCode.The passage attributes this variability to differences in how each student adapts to domain interaction patterns, rather than domain difficulty alone.Source: §4.2 · p. 7

4.3 Comparison with Dataset Construction Baselines

  • Existing AgentNet trajectories yield limited gains, Trajectory Boosting degrades performance, and zero-annotation generators perform comparably to AgentNet retraining.The comparisons use a matched training budget; the reported weakness-agnostic generators include AgentSynth, OS-Genesis, and ZeroGUI.Source: §4.3 · p. 7, Table 2 · p. 7
  • LEARNWEAK-GEN achieves the best average performance, outperforming WebSTAR by 5.58 percentage points under a comparison that differs only in the filtering stage.WebSTAR applies generic step-quality filtering, whereas LEARNWEAK retains trajectories aligned with the student’s identified weaknesses; Table 2 reports data-construction baseline results under a matched budget.Source: §4.3 · p. 7, Table 2 · p. 7

5 Analysis

The analysis finds that student-specific weakness reports make generation more effective, while specialization training benefits from preference learning and selective updates across both planning and execution errors. Teacher capability and iteration count also matter: stronger teachers perform similarly, and additional generation rounds can eventually reduce performance.

5.1 Data Generation Pipeline Analysis

  • Both OpenCUA-7B and EvoCUA-8B achieve their highest performance with datasets generated from their own failure cases, while cross-student datasets yield lower gains.The passage attributes this to differences in student failure cases and weakness types, supporting target-specific weakness reports for data generation.Source: §5.1 · p. 8
  • One-shot generation raises average score from 40.69 to 48.82, but iteration without weakness-report conditioning adds no improvement; the full guided pipeline performs best on average and in three of four domains.The reported results indicate that iterative expansion is most effective when guided by the student's observed failure patterns rather than domain exploration alone.Source: §5.1 · p. 8
  • Claude Haiku 4.6 yields smaller specialization gains than EvoCUA-32B and Kimi K2.5, whose specialized student performances are very similar despite a large gap in standalone success rates.The passage suggests teacher strength helps provide reliable successful trajectories to identify weaknesses, while further gains depend on targeting weaknesses actionable for the student.Source: §5.1 · p. 8
  • Performance in Calc and VSCode rises during early generation iterations, peaks at an intermediate stage, then declines with additional rounds.The reported pattern suggests that more data alone does not guarantee improvement; additional tasks need to remain aligned with the student's unresolved weaknesses.Source: §5.1 · pp. 8–9, Figure 3 · p. 8

5.2 Training Objective Analysis

  • LEARNWEAK-DPO outperforms standard DPO by 9.62 points on average, while effective specialization requires preference learning and selective updates to both planning- and execution-level tokens.Error-aware masking improves SFT, but masking only planning-level or only execution-level tokens is insufficient; standard SFT and DPO improve only modestly over the zero-shot student.Source: §5.2 · p. 9

6 Related Work

Related work establishes domain-specific weaknesses in computer-use agents and advances annotation-free trajectory synthesis and failure-aware training; LEARNWEAK distinguishes itself by targeting data and supervision to student failures.

Computer-Use Agents (CUAs)

  • Execution benchmarks expose domain gaps, especially in productivity software requiring application-specific interaction knowledge, motivating small specialized CUAs that reduce serving cost and latency.Software-specific adaptation efforts include SEAgent [32] and Fara-7B [3].Source: §6 · p. 9

Automated Trajectory Generation

  • PC-Agent-E [9] expands a small set of human trajectories, while AgentSynth [38], OS-Genesis [30], ZeroGUI [43], AgentTrek [41], and Watch-and-Learn [28] pursue zero-annotation synthesis for volume or diversity rather than failure-conditioned generation.LEARNWEAK identifies capability gaps from failed executions and synthesizes tasks conditioned on those weaknesses, making samples more informative for domain specialization.Source: §6 · p. 9

Agent Training

  • LEARNWEAK forms targeted DPO pairs from teacher trajectories and same-context student rollouts, then selects only divergent action spans for focused supervision.This builds on learner-state correction approaches such as DAgger [27] and On-Policy Expert Corrections [15], and failure-based preference methods such as ETO and DPO [26].Source: §6 · p. 9

7 Conclusion

LEARNWEAK specializes small computer-use agents by targeting each student’s demonstrated weaknesses rather than providing broad domain coverage. Its teacher-student data-generation and error-aware preference-training stages improve agents across software domains, while the modular adapter design suggests a path to multi-application deployment.

7 Conclusion

  • LEARNWEAK combines iterative weakness-aware dataset construction through teacher-student comparison and screenshot-grounded query synthesis with step-level preference supervision and error-aware masking.The preference cases are those where the teacher succeeds but the student fails.Source: §7 · p. 10
  • The targeted specialization strategy substantially improves small computer-use agents across diverse software domains and outperforms alternative data-construction methods.The conclusion attributes efficient specialization to identifying and repairing student-specific weaknesses rather than simply scaling synthetic data.Source: §7 · p. 10
  • Automated domain specialization can narrow the gap between small open computer-use agents and much larger agents without human trajectory annotation.The conclusion presents this as making small specialized agents a more practical deployment path for real-world software environments.Source: §7 · p. 10
  • The modular LoRA-based design can support multi-application deployment by maintaining per-domain adapters and activating the adapter matched to the target application at inference.A systematic empirical study of multi-adapter routing across many domains remains future work.Source: §7 · p. 10

Limitation · Appendix Overview · A Algorithmic Details

LearnWeak depends on reliable teacher guidance and a student that already has general computer-use skills, so it may not address foundational GUI weaknesses. The appendix lists supplementary algorithmic, implementation, experimental, prompt, and qualitative materials.

Limitation

  • The framework assumes a reasonably reliable teacher; unstable or biased guidance can pass errors into its supervision.The paper notes this reflects the broader dependence of teacher-guided offline learning on supervision quality.Source: Limitation · p. 10
  • The student must already have general computer-use skills, so specialization may not help models lacking foundational GUI capabilities.The assumed skills include visual grounding, action generation, and error recovery; the framework targets domain knowledge.Source: Limitation · p. 10

Appendix Overview

  • The appendix provides supplementary material for the main paper.Source: Appendix Overview · p. 14
  • The appendix lists algorithmic details for the data-generation pipeline and implementation details.Source: Appendix Overview · p. 14
  • Additional appendix experiments and analyses cover generated-data statistics, failure-focused trajectory selection, and other specialization results.Source: Appendix Overview · p. 14
  • The appendix includes prompt templates and qualitative results on weakness reports, synthetic queries, and case studies.Source: Appendix Overview · p. 14

A.1 Data Generation Pipeline … B.2 LEARNWEAK

LEARNWEAK specializes small computer-use agents by iteratively finding student failures, generating domain-specific tasks, and constructing targeted supervision. Its implementation uses benchmark-disjoint configurations and separates planning from execution errors during preference-data construction.

A.1 Data Generation Pipeline

  • LEARNWEAK-GEN runs teacher and student trajectories on seed tasks, evaluates them, and summarizes student weaknesses into an aggregated failure set.The pipeline takes a target domain, teacher and fixed student policies, a verifier, task generator, screenshot selector, and domain metadata; it returns the failure set and trajectory collection.Source: §A.1 · p. 14
  • Using representative screenshots and domain metadata, the generator creates both weakness-conditioned and unconstrained tasks for subsequent iterations.The algorithm selects screenshots, generates queries in both modes, and updates the seed task set across iterations.Source: §A.1 · p. 14

B.1 Shared Setup

  • Training configurations are disjoint from OSWorld evaluation setups to prevent benchmark-specific files, assets, screenshots, traces, and trial-and-error patterns from leaking into training.Configurations are structurally similar to evaluation setups but use different files and assets.Source: §B.1 · pp. 14–15
  • Each target domain uses six environment configurations and ten manually authored seed queries, a process taking less than two hours of human effort.The configurations support launching the target software and, where applicable, downloading or loading needed files.Source: §B.1 · p. 15
  • Evaluation runs each model-domain pair three times and reports mean success rate, excluding Chrome for weaker reproducibility and less stable evaluation behavior.Experiments use OSWorld's local Docker provider environment.Source: §B.1 · p. 15

B.2 LEARNWEAK

  • Main data generation uses EvoCUA-8B as student, EvoCUA-32B as teacher, GPT-5-mini for supporting tasks, and five iterations per domain.Each domain starts with ten seed queries on six benchmark-disjoint configurations; after CLIP diversity filtering, GPT-5-mini ranks screenshots and the top ten are kept. Across four post-seeding rounds, generation yields up to 144 candidate queries per domain.Source: §B.2 · p. 15
  • DPO preference data maps parameter mismatches to execution errors and action-type or tool-count mismatches to planning errors, discarding exact and wait-only matches.Coordinate selections within 20 pixels count as equivalent; training uses DPO with β = 0.1.Source: §B.2 · p. 15

B.3 Data-Construction Baselines · C Additional Experimental Results and Analysis

The section establishes a matched-budget comparison of data-construction baselines for specializing EvoCUA-8B, then specifies how each baseline’s data was generated or filtered.

B.3 Data-Construction Baselines

  • All baselines use benchmark-disjoint training configurations for EvoCUA-8B, GPT-5-mini for verification and query generation, and matched per-domain budgets.The method’s budgets are 22 trajectories for Calc, 52 for Impress, 32 for VLC, and 38 for VS Code.Source: §B.3 · p. 15
  • Trajectory Boosting [9] replaces human-annotated inputs with 10 EvoCUA-32B teacher trajectories and generates eight candidate actions per state.The candidates are generated at the step level from teacher trajectories on seed queries.Source: §B.3 · p. 16
  • OS-Genesis [30] follows its four-stage pipeline, using EvoCUA-32B for exploration and recollection before filtering and jointly training on planning and action data.It creates a 2× exploration buffer, keeps trajectories scoring at least 3 under the TRM-style evaluator, then retains top-scoring examples within the matched budget.Source: §B.3 · p. 16
  • AgentSynth [38] uses its multi-subtask pipeline with EvoCUA-32B as executor, then retains examples under the same per-domain budget as the proposed method.Each chain has six subtasks, with up to 10 steps per execution; the pipeline yields 84 raw Calc examples and 126 for each other domain before budget matching.Source: §B.3 · p. 16
  • ZeroGUI [43] is evaluated using generated-task training only, excluding test-time training, with reward-based training on budget-matched sampled trajectories.It generates 200 candidate tasks per domain from 10 instructions per round over 20 rounds, then randomly samples trajectories to match the method’s budget.Source: §B.3 · p. 16
  • WebSTAR [10] filters generated-dataset teacher trajectories using step-level thoughts and retains steps graded above 5 on its 0–10 scale.Because the generation pipeline already filters trajectories, that component is excluded from the LEARNWEAK-GEN setting when applying WebSTAR.Source: §B.3 · p. 16

C.1 Statistics of Generated Data … E Qualitative Results

The appendix shows that generated supervision reflects domain-specific student errors, that selecting teacher-success/student-fail tasks is most effective, and that the framework can be adapted—with caveats—to a different output format. The supplied prompt templates emphasize novel, concise, realistic tasks grounded in workspace constraints.

C.1 Statistics of Generated Data

  • Generated supervision varies across domains and student backbones, with some domains dominated by planning discrepancies and others mixing planning and execution errors.The domain-level statistics include teacher-pass and student-fail trajectories and planning-versus-execution error breakdowns.Source: Figure 4 · p. 16, §C.1 · pp. 16–17

C.3 Adapting Specialization to Different Output Format (UI-TARS-1.5-7B)

  • UI-TARS-1.5-7B improves on all four evaluated domains, with largest gains on OS and VSCode, but the modified masking result is preliminary rather than like-for-like validation.The adaptation supervises reasoning and tool-execution tokens for planning errors, but only tool-execution tokens for execution errors; it may be affected by teacher-student thought-style differences.Source: Table 8 · p. 17, §C.3 · p. 17

D Prompt Templates

  • The query-generation prompt uses screenshots to propose realistic new workflows, while avoiding prior instructions and following the workspace/path contract.It also supplies prior instructions, the target configuration, and optional file, folder, or code context as inputs.Source: D Prompt Templates · p. 21
  • Generated queries must be concise end-user instructions, contain at most two simple subtasks, differ from one another, and return in strict JSON with specified fields.Each instruction must be under 15 words, and exactly the requested number of query objects must be generated.Source: D Prompt Templates · p. 21

E.1 Weakness Report and Synthetic Query Results · E.2 Case Study

The appendix links LibreOffice Calc weakness reports to synthetic queries derived from failure categories, then uses OSWorld case studies to illustrate how specialization changes agents’ local decisions.

E.1 Weakness Report and Synthetic Query Results

  • The LibreOffice Calc examples connect weakness reporting to synthetic-query generation by presenting report excerpts, queries derived from identified failure categories, and analysis of their linkage.The section presents this report-to-query sequence for each example.Source: §E.1 · p. 22, Figure 10 · p. 22, Figure 11 · p. 23
  • Three examples illustrate the weakness-report-to-query linkage in LibreOffice Calc.Each example presents a formatted weakness-report excerpt followed by representative synthetic queries and brief linkage analysis.Source: §E.1 · p. 22, Figure 10 · p. 22, Figure 11 · p. 23

E.2 Case Study

  • The case studies compare OSWorld trajectories before and after specialization in LibreOffice Calc and LibreOffice Impress.The examples include two Calc cases and two Impress cases.Source: §E.2 · p. 25, Figure 13 · p. 25, Figure 14 · p. 26
  • Each case-study step pairs an agent observation shown as a screenshot with the corresponding model response and a concise action summary.The examples expose the step-by-step trajectory elements used in the comparison.Source: §E.2 · p. 25
  • The case studies omit the model response’s reasoning portion for brevity.The displayed responses therefore exclude their reasoning text.Source: §E.2 · p. 25
  • The examples illustrate changes in the model’s local decision-making behavior, not only differences in final task outcomes.The section presents these trajectories as evidence of how specialization alters behavior during task execution.Source: §E.2 · p. 25

Related papers

Loading 2605.28775v1…