Source-linked AI summary
Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents
Suji Kim, Kangsan Kim, Sung Ju Hwang
TL;DR
Small open computer-use agents are cheaper and more deployable than large domain-specific experts but remain weaker and inconsistent across software domains. LearnWeak uses a stronger teacher to identify student weaknesses, synthesize targeted tasks, and provide error-aware supervision without annotation. Across eight OSWorld domains, it improves average performance by 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively.
Problem
Small computer-use agents remain substantially weaker and exhibit uneven failures in domain-specific software, while naive large-scale target-domain data synthesis yields only marginal improvements.
Method
LearnWeak iteratively compares teacher and student behavior to identify weaknesses, synthesizes targeted domain tasks, and trains the student with error-aware preference optimization.
Results
11.6 and 11.1 percentage points are the average gains over EvoCUA-8B and OpenCUA-7B, respectively, across eight OSWorld domains.
Takeaways & Limitations
Student-aware data generation and training outperform autonomous trajectory-generation and alternative offline-training baselines for small-agent domain specialization.
Takeaways & Limitations
LearnWeak assumes a reasonably reliable teacher and a student that already has general computer-use skills such as visual grounding, action generation, and error recovery.
Abstract
from arXiv · showhide
Computer-use agents (CUAs) have recently made substantial progress, but deploying a separate large expert for each software domain remains expensive. Small open computer-use agents are more practical specialization targets, but they remain substantially weaker and exhibit uneven domain-specific failures. A straightforward remedy is to synthesize large-scale training data for the target domain, yet we find that this naive approach yields only marginal improvements. Building on this observation, we introduce LearnWeak, an annotation-free specialization framework for small computer-use agents that uses a stronger reference agent to identify the student's weaknesses in the target domain, synthesize targeted tasks, and construct supervision automatically. LearnWeak further introduces an error-aware specialization objective that disentangles planning and execution errors, enabling more behaviorally precise updates than broad uniform supervision. On OSWorld, LearnWeak achieves average gains of 11.6 and 11.1 percentage points over EvoCUA-8B and OpenCUA-7B, respectively, across eight domains. We also validate that our student-aware dataset generation and training approaches outperform existing autonomous trajectory generation and training baselines. Our work highlights the importance of student awareness in both data synthesis and agent training, pointing toward a more principled and efficient path for specializing small computer-use agents in diverse domains.
1 Introduction
Small computer-use agents are practical to deploy but remain weaker and uneven across software domains. LearnWeak addresses this gap by generating and training on tasks targeted to each student’s weaknesses, improving specialization without human annotation.
- Motivation: Small CUAs offer faster, cheaper, edge-device, and privacy-compatible deployment, but retain a performance gap in unfamiliar domain-specific software environments.Their limited capacity across heterogeneous workflows also makes broad training vulnerable to catastrophic forgetting and degraded domain performance.
- Motivation: Existing autonomous data-generation strategies overlook student deficiencies, making targeted task selection essential for efficient domain specialization.Generated queries should expose missing domain knowledge and unresolved weaknesses rather than merely increase data volume.
- LearnWeak: LearnWeak combines annotation-free teacher-student comparison, weakness analysis, and query synthesis to build compact, targeted domain datasets from a small seed set.Its training stage uses error-aware preference optimization that distinguishes planning failures from execution failures.
- Results: 11.6 and 11.1 percentage points are the average gains over EvoCUA-8B and OpenCUA-7B, respectively, across eight OSWorld domains.The study also reports stronger autonomous data-generation and training performance than matched baselines, including SFT and standard DPO variants.
2 Preliminaries
The paper models computer-use agents as history-conditioned policies operating in partially observable software environments. It formulates specialization as annotation-free dataset generation followed by domain-specific student training, optimizing target-domain task success.
- 2.1 Computer-Use Agent: A computer-use agent perceives screens and issues actions in an interactive software environment, which is modeled as a partially observable decision process.Conditioning on the full interaction history addresses the fact that the current screen does not reveal the complete environment state.
- 2.1 Computer-Use Agent: At each step, the policy receives a task query, current observation, and interaction history, then produces reasoning, an action description, and executable tool actions.The executable action includes a function type and parameters such as left_click(x,y) or type(text).
- 2.2 Problem Formulation: Domain specialization fine-tunes a broadly capable student policy for a target software domain using a stronger teacher, seed queries, and an automatic verifier.The formulation assumes no further human annotation.
- 2.2 Problem Formulation: The problem has two coupled stages: autonomous generation of a domain-specific training dataset and training of a domain-specialized student from that dataset.DataGen expands seed queries and collects teacher trajectories without human annotation.
- 2.2 Problem Formulation: The overall objective is to maximize expected task success on the target-domain evaluation distribution.Success is evaluated on trajectories induced by the specialized policy for target-domain queries.
3 Method
LEARNWEAK specializes small CUAs through student-aware data generation and error-aware training. It discovers failures through teacher-student comparison, synthesizes targeted tasks, and applies selective preference supervision to behaviorally relevant actions.
- Weakness-Aware Data Generation: LEARNWEAK iteratively compares teacher and student trajectories, verifies outcomes, analyzes weaknesses, and synthesizes new domain-specific queries without human annotation.The pipeline starts from a small seed set and repeats weakness discovery with screenshot-guided query generation before final filtering.
- Weakness Discovery: Teacher-successful and student-failed tasks identify student deficiencies while reducing the likelihood that failures arise from infeasible tasks or invalid environment states.The resulting weakness reports capture recurring issues such as incorrect operation selection, inaccurate localization, and invalid action arguments.
- Weakness-Aware Data Generation: Weakness-focused synthesis targets reported deficiencies, while exploration-focused synthesis uses representative screenshots to cover unexplored domain functionality.Combining both strategies balances student-aware targeting with diversity in query objectives.
- Teacher-Replay Preference Construction: Teacher-replay preference construction creates step-level supervision only where teacher and replayed-student tool executions differ, avoiding correction of already-correct steps.The teacher trajectory supplies a verified successful context, while the replayed student response identifies the behavior requiring correction.
- Error-Aware Preference Optimization: Error-aware preference optimization increases teacher-action likelihood over replayed student actions while restricting updates to behaviorally relevant spans.The method distinguishes planning-level and execution errors and uses masked action scores within DPO training.
- Domain Scalability: Domain-specific LoRA adapters attach specialized knowledge to a shared frozen student, allowing the appropriate domain module to be activated at deployment.This modular design localizes domain updates while retaining a shared base policy across domains.
4 Experiments
Experiments evaluate LEARNWEAK on OSWorld using two small CUA backbones and compare its domain specialization, data construction, and training objectives with established baselines. Across domains and application types, weakness-focused specialization produces consistent gains, while targeted generation and error-aware optimization outperform weaker alternatives.
- Experimental Setup: LEARNWEAK is evaluated on eight OSWorld domains using EvoCUA-8B and OpenCUA-7B as student models, with EvoCUA-32B as the default teacher.The process is performed independently for each domain.
- Domain Specialization Results: 11.6 and 11.1 percentage points are the average gains for EvoCUA-8B and OpenCUA-7B, respectively, after specialization.Average scores increase from 50.69 to 62.24 for EvoCUA-8B and from 37.65 to 48.72 for OpenCUA-7B.
- Domain Specialization Results: Improvements are consistent across all eight domains and span office software, system utilities, visual editing, and coding-oriented workflows.The gains are not confined to a single application type.
- Domain Specialization Results: Weakness-focused corrective supervision enables specialized EvoCUA-8B to outperform the 32B teacher on Gimp, Thunderbird, and VSCode.The result indicates that student-conditioned corrections can address domain-specific failures beyond simple imitation.
- Comparison with Dataset Construction Baselines: Under matched budgets, weakness-agnostic generation baselines provide limited or comparable gains, whereas LEARNWEAK achieves the best average performance and outperforms WebSTAR by 5.58 percentage points.Its advantage is most evident in domains other than VSCode, where methods perform comparably.
5 Analysis
Analysis shows that specialization works best when generated supervision reflects the target student’s own failures and when generation is iteratively aligned with unresolved weaknesses. Training also benefits from jointly targeting planning and execution errors, while additional generation rounds can eventually reduce performance.
- Weakness-awareness: Each student achieves the highest performance when trained on data generated from its own failure cases rather than another student’s weakness report.The analysis attributes this pattern to differences in failure cases and weakness types across student models.
- Pipeline Components: One-shot generation improves the average score from 40.69 to 48.82 relative to the zero-shot student, but iterative generation without weakness conditioning is insufficient.The ablation isolates iterative generation and weakness-report conditioning as separate pipeline components.
- Teacher Choice: Teacher strength matters up to a point: Claude Haiku 4.6 yields smaller gains, while EvoCUA-32B and Kimi K2.5 produce very similar specialized-student performance.The stronger teachers’ similar student outcomes occur despite a large gap in their standalone success rates.
- Number of Generation Iterations: Generation-round effects are non-monotonic: performance rises early, peaks at an intermediate stage, and decreases with additional rounds in Calc and VSCode.The marginal value of more data depends on whether newly generated tasks remain aligned with unresolved student weaknesses.
- Training Objective Analysis: LEARNWEAK-DPO outperforms standard DPO by 9.62 points on average, and selective supervision over only planning or only execution tokens is insufficient.The best results require preference learning together with updates spanning both error types.
6 Related Work
Related work addresses automated trajectory generation and training for computer-use agents, while LearnWeak targets student-specific weaknesses in both processes.
- Automated Trajectory Generation: Automated trajectory generation reduces reliance on costly human-annotated GUI trajectories through model-based expansion, subtask composition, and environment exploration.These approaches include PC-Agent-E, AgentSynth, and OS-Genesis, alongside other zero-annotation pipelines.
- Agent Training: Supervised imitation can accumulate errors under covariate shift, motivating expert corrections and preference-based training for interactive agents.DAgger and On-Policy Expert Corrections address learner-induced states, while ETO and DPO use successful and failed trajectories as preference signals.
- Agent Training: LearnWeak constructs targeted preference pairs from teacher successes and student rollouts, then trains only on spans where the student diverges.Error-aware span selection makes supervision more focused than imitating complete trajectories.
7 Conclusion
The paper concludes that efficient specialization of small computer-use agents depends on identifying and repairing student-specific weaknesses rather than merely scaling synthetic data. Its fully automated approach improves agents across software domains and can narrow their gap with larger agents without human trajectory annotation.
- Conclusion: LEARNWEAK-GEN builds weakness-aware domain datasets through iterative teacher-student comparison and screenshot-grounded query synthesis.LEARNWEAK-DPO converts teacher-successful and student-failed cases into step-level preference supervision with error-aware masking.
- Conclusion: Targeted specialization substantially improves small computer-use agents across diverse software domains and outperforms alternative data-construction methods.
- Conclusion: Automated domain specialization can narrow the gap between small open computer-use agents and much larger agents without human trajectory annotation.
Limitation
The study depends on reliable teacher guidance and on students already possessing general computer-use capabilities. It therefore does not establish effective improvement for models lacking foundational GUI skills or whose failures are not domain-specific.
- Limitation: LEARNWEAK assumes a teacher that provides reasonably reliable guidance within the target domain.An unstable or systematically biased teacher can pass its errors into the resulting supervision.
- Limitation: The framework assumes students already have general computer-use skills, including visual grounding, action generation, and error recovery.
- Limitation: The method does not guarantee effective improvement for general-purpose models without computer-use training or students lacking foundational GUI capabilities.It focuses on domain knowledge rather than foundational computer-use abilities.
A.1 Data Generation Pipeline
The data-generation pipeline iteratively compares teacher and student trajectories, verifies their outcomes, summarizes student weaknesses, and synthesizes targeted queries. It uses benchmark-disjoint environments and teacher-student rollouts to build training data, while downstream processing separates planning and execution mismatches for preference training.
- A.1 Data Generation Pipeline: LEARNWEAK-GEN requires a target domain, teacher and student policies, verifier, task generator, screenshot selector, domain metadata, and iteration count.
- A.1 Data Generation Pipeline: Each iteration initializes seed tasks, runs teacher and student trajectories, evaluates both, and summarizes the student’s weaknesses.
- A.1 Data Generation Pipeline: The pipeline uses weakness-conditioned and unconstrained query synthesis before aggregating failure tasks and teacher-student trajectories.
- A.1 Data Generation Pipeline: Training configurations are disjoint from OSWorld evaluation configurations to avoid contamination from benchmark-specific assets.The configurations remain structurally similar to OSWorld while using different files and assets.
- A.1 Data Generation Pipeline: Each domain uses six configurations and ten seed queries, with manually authored seeds requiring less than two hours of human effort.The configurations and seed queries are intended to support reproducibility.
- A.1 Data Generation Pipeline: The evaluation excludes Chrome because of weaker reproducibility and less stable evaluation behavior, and reports mean success rates over three runs.
- A.1 Data Generation Pipeline: Main data-generation runs use EvoCUA-8B as student, EvoCUA-32B as teacher, GPT-5-mini for auxiliary functions, and five iterations per domain.Screenshot pools are reduced with CLIP-based diversity filtering before selecting the top screenshots.
- A.1 Data Generation Pipeline: Teacher and replayed student tool calls are compared after removing waits, mapping parameter mismatches to execution errors and action-type or tool-count mismatches to planning errors.
B.3 Data-Construction Baselines
The study reimplements several autonomous data-construction baselines under matched, benchmark-disjoint domain-specialization settings for EvoCUA-8B. The baselines differ in how they generate, filter, or augment trajectories before training.
- Experimental setup: All baseline methods use benchmark-disjoint training configurations and matched per-domain trajectory budgets against LEARNWEAK.The target is EvoCUA-8B, with GPT-5-mini used for verification and query generation across methods.
- Trajectory generation: Trajectory Boosting replaces human-annotated source trajectories with 10 EvoCUA-32B teacher trajectories and generates ×8 candidate actions per state.Candidate actions are generated at the step level from the seed-query trajectories.
- Trajectory generation: OS-Genesis explores each domain, synthesizes instructions from exploration trajectories, recollects clean trajectories, and applies TRM-based filtering.Exploration and recollection use EvoCUA-32B, with a 2× exploration buffer relative to the final target count.
- Trajectory generation: AgentSynth uses six-subtask chains across seven configurations per domain, producing 84 raw Calc examples and 126 raw examples for the other three domains before budget matching.Each subtask execution is limited to 10 steps.
- Trajectory generation: ZeroGUI generates 200 candidate tasks per domain over 20 rounds, then samples trajectories to match LEARNWEAK’s per-domain budget and applies reward-based training.The comparison includes only ZeroGUI’s generated-task training stage, excluding test-time training.
- Trajectory filtering: WebSTAR filters teacher-generated trajectories by augmenting each step with a generated thought, grading it from 0–10, and retaining steps above its threshold.Its filtering component is applied to the generated dataset because LEARNWEAK-GEN already includes filtering.
C.1 Statistics of Generated Data
Generated specialization data contains domain- and backbone-specific mixtures of planning and execution errors rather than a uniform failure profile. Additional evaluations test student-aware task selection and adaptation to a student with a different output format.
- Generated-data statistics: Generated datasets report teacher-pass and student-fail trajectories together with planning- and execution-error breakdowns across domains and student backbones.The statistics characterize the supervision used for specialization.
- Generated-data statistics: Different software domains expose different student-failure mixtures, with some dominated by planning discrepancies and others containing a more balanced planning–execution profile.The generated data therefore reflects domain-specific correction needs rather than a uniform error profile.
- Task selection: The task-selection ablation compares all generated trajectories, teacher-pass trajectories, and teacher-pass/student-fail trajectories.It tests whether gains come from generic quality filtering or concentrating supervision on unresolved student failures.
- Cross-model evaluation: LEARNWEAK is also evaluated on UI-TARS-1.5-7B using a modified masking rule because the model lacks a separate action-description component.The original rule assumes the structured rt–st–et action decomposition.
- Cross-model evaluation: The modified training rule improves UI-TARS-1.5-7B on all four evaluated domains, with the largest gains on OS and VSCode.The gains are smaller than those observed for EvoCUA and OpenCUA because the original masking design is tailored to explicit rt–st–et outputs.
- Prompted generation: The generation prompts separately support teacher–student weakness analysis, screenshot ranking, and query generation with or without a weakness report.The prompts also constrain synthetic tasks to realistic workflows and the workspace/path contract.
E.1 Weakness Report and Synthetic Query Results
The case studies connect weakness reports to synthetic queries and show failures involving aggregation, print-area and freeze settings, partial-range sorting, spreadsheet calculations, and presentation formatting. Before–after OSWorld trajectories illustrate changes in local decision-making in LibreOffice Calc and Impress.
- Weakness-to-query linkage: Weakness-report examples are followed by synthetic queries derived from the identified failure categories, making the report-to-query linkage explicit.The examples are drawn from the libreoffice_calc domain.
- LibreOffice Calc: A Calc failure omitted department aggregation and summary-table construction, including unique-department extraction, AVERAGE formulas, or a pivot table.The student repeatedly selected cells and columns without completing the required summary.
- LibreOffice Calc: A Calc failure set the freeze line below row 2 and omitted setting the selected used range as the print area.The task required freezing only the header row and assigning the used range as the print area.
- LibreOffice Calc: A Calc sorting failure selected only Population and Area columns, omitting State and risking misalignment because whole rows were not moved together.The requested operation was descending sorting by Population in the Population_of_India sheet.
- Before–after case studies: The case studies compare OSWorld trajectories before and after specialization in LibreOffice Calc and Impress, showing changes in local decision-making behavior.Each step pairs an observation screenshot with the model response and a concise action summary.
- Synthetic-query examples: Additional case-study instructions cover ascending sorting, gross-profit calculation with a derived Year_Profit column, and setting all text-box font colors on slide 5 to yellow.These tasks span spreadsheet computation, cross-sheet transformation, and presentation formatting.