PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents

Jingxu Xie, Dylan Xu, Xuandong Zhao, Dawn Song

arXiv:2506.14205v2cs.CL

TL;DR

Computer-use benchmarks are costly to annotate and remain limited in diversity, while synthetic pipelines struggle with reliable long-horizon generation. AgentSynth composes verified simple subtasks into challenging tasks, generating over 6,000 tasks at $0.60 per trajectory; success falls from 18% at level 1 to 4% at levels 5 and 6. The benchmark exposes severe weaknesses in current agents, particularly their difficulty recovering from errors.

  • Problem

    Computer-use datasets rely heavily on costly human demonstrations, while synthetic pipelines face unreliable complex-task generation and limited diversity.

    Source: §1 · p. 1

  • Method

    AgentSynth generates and verifies simple sequential subtasks, collects their trajectories, and summarizes them into diverse long-horizon tasks with controllable difficulty.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Success drops from 18% at difficulty level 1 to 4% at levels 5 and 6, while AgentSynth generates over 6,000 tasks at an average cost of $0.60 per trajectory.

    Source: §1 · p. 2, §5.2 · p. 8

  • Takeaways & Limitations

    The benchmark reveals that current agents remain far from generalizable competence on realistic multi-step computer tasks.

    Source: §5.2 · p. 8

    Agents commonly fail to recover after errors, repeating failed behavior instead of exploring alternatives or self-correcting.

    Source: §5.3 · p. 9

Abstract

from arXiv · show

We introduce AgentSynth, a scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer-use agents. Leveraging information asymmetry, AgentSynth constructs subtasks that are simple during generation but significantly more challenging when composed into long-horizon tasks, enabling the creation of over 6,000 diverse and realistic tasks. A key strength of AgentSynth is its ability to precisely modulate task complexity by varying the number of subtasks. Empirical evaluations show that state-of-the-art LLM agents suffer a steep performance drop, from 18% success at difficulty level 1 to just 4% at level 6, highlighting the benchmark's difficulty and discriminative power. Moreover, our pipeline achieves a low average cost of $0.60 per trajectory, orders of magnitude cheaper than human annotations. Our code and data are available at https://github.com/sunblaze-ucb/AgentSynth

1 INTRODUCTION

AgentSynth addresses the cost, scalability, reliability, and diversity challenges of constructing realistic computer-use tasks by composing simple subtasks into difficult long-horizon tasks. Its benchmark provides controllable difficulty and reveals steep performance degradation in current agents.

  • Human-in-the-loop computer-use datasets are labor-intensive, expensive, and difficult to scale across real-world computing scenarios.Source: §1 · p. 1
  • Existing synthetic pipelines struggle with reliable complex-task trajectories and diverse generation, especially for visually grounded, long-horizon tasks.Source: §1 · p. 1
  • AgentSynth exploits information asymmetry by generating simple subtasks forward, collecting their trajectories, and composing them into harder long-horizon tasks.Source: §1 · pp. 1–2
  • Over 6,000 diverse tasks are generated at an average cost of $0.60 per trajectory, with task diversity arising from different subtask chains.Source: §1 · p. 2
  • The benchmark chains LLM-generated subtasks, controls difficulty through composition, and shows that state-of-the-art agents struggle substantially.Source: §1 · p. 2

2 RELATED WORK

Prior agent datasets and benchmarks remain constrained by manual annotation, limited diversity, or simplified interaction settings. AgentSynth builds on synthetic generation and realistic operating-system environments to target broader, more complex computer-use tasks.

  • Computer-use datasets and benchmarks predominantly rely on costly human annotation, limiting scalability and diversity.Source: §2 · p. 2
  • Synthetic-data research has automated dataset generation, but many pipelines focus on static text benchmarks or define tasks differently from AgentSynth’s subtask composition.Source: §2 · p. 2
  • Existing agent-task generation work is mainly web-based and typically lacks complex multi-step reasoning or extensive tool use.Source: §2 · p. 2
  • Agent environments have progressed from simplified web tasks toward realistic interfaces, but OSWorld provides broader access to authentic computer tools for synthetic generation.Source: §2 · pp. 2–3
  • AgentSynth’s Figure 1 summarizes a pipeline that verifies subtasks and composes them into tasks at different difficulty levels.Source: Figure 1 · p. 3

3 SCALABLE AGENT TASKS AND TRAJECTORIES GENERATION

AgentSynth uses six LLM-based agents to generate, execute, verify, revise, extend, and summarize subtasks in realistic computer environments. The resulting compositions preserve reliable trajectories while producing controllable long-horizon difficulty.

3 SCALABLE AGENT TASKS AND TRAJECTORIES GENERATION

  • Six LLM-based agents propose, execute, verify, revise, extend, and summarize subtasks into coherent long-horizon tasks.Source: §3 · p. 3

Environment

  • The pipeline runs in OSWorld, where agents act on full-screen screenshots using clicks, keystrokes, text input, and scrolling, and is also applied to InSTA.Source: §3 · pp. 3–4

Task Proposer

  • The initial proposer uses a persona and desktop screenshot to generate clear, specific tasks completed in a few atomic actions while excluding unsafe or credential-related actions.Source: §3 · p. 4

Scope

  • Task-generation agents currently rely on GPT-4.1, leaving the effects of other generator architectures on realism, complexity, and calibration open.Source: §3 · p. 4

Task Executor

  • GPT-4.1 plans actions from screenshots and history, while computer-use-preview grounds them in precise executable coordinates; executions are limited to 10 steps.Source: §3 · p. 4

Task Verifier

  • A verifier evaluates full screenshot sequences using key requirements and selected screenshots, producing success and completion estimates for revision when needed.Source: §3 · p. 4

Task Reviser and Follow-up Task Proposer

  • The reviser aligns task descriptions with completed behavior, while the follow-up proposer generates simple subtasks that build on the prior state and can simplify failed attempts.Source: §3 · p. 4

Task Summarizer

  • The summarizer abstracts away step details while preserving the objective, and increasing the number of summarized subtasks systematically increases task difficulty.Source: §3 · pp. 4–5

4 DATASET ANALYSIS

AgentSynth produces realistic, diverse, long-horizon tasks with reliable verification and quality metrics above 85%. Its difficulty scales through longer horizons, memory spans, application use, and switching, while generation costs $0.6 per trajectory.

4.1 QUALITY

  • All quality metrics exceed 85%, supporting the consistency, realism, and reliability of AgentSynth tasks and trajectories.Human evaluation covered feasibility, subtask coherence, persona relevance, and verifier accuracy; inter-rater agreement was 0.74.Source: §4.1 · p. 5

4.2 VERIFIER CALIBRATION

  • Verifier accuracy remains high across difficulty levels, with only a mild decline as tasks become harder.The verifier’s completion scores also track human-judged completion monotonically, providing a meaningful partial-credit signal.Source: §4.2 · p. 5, Figure 2 · p. 5
  • The verifier accepts only 12% of near-miss variants but 94% of benign variants, indicating low false positives and robustness to superficial UI changes.Near-miss variants subtly violate the goal, whereas benign variants preserve the goal while changing irrelevant interface details.Source: §4.2 · pp. 5–6, Table 2 · p. 6

4.3 COMPARISON TO OTHER DATASETS AND BENCHMARKS

  • Difficulty level 6 typically requires 40–60 steps, while tasks span domains including office productivity, retrieval, entertainment, coding, and research.Over 60% of trajectories use at least two applications, and more than 40% use at least three.Source: §4.3 · p. 6, Figure 3 · p. 6
  • From difficulty level 1 to 6, average horizon rises from 5 to 45 steps, memory span from 2 to 18, applications from 1.2 to 3.3, and app switches from 0.5 to 4.3.Higher difficulty therefore combines longer horizons with more memory and cross-application coordination demands.Source: §4.3 · pp. 6–7

4.4 COST ANALYSIS

  • AgentSynth costs $0.6 per trajectory with five follow-up subtasks, compared with $0.55 for AgentTrek, $0.28 for Explorer, and $0.27 for InSTA.The paper also reports that this is much cheaper than human annotation for complex tasks with long trajectories.Source: §4.4 · p. 7, Table 3 · p. 7

5 RESULTS AND DISCUSSION

AgentSynth evaluates generalist computer-use agents on increasingly difficult OSWorld tasks and finds sharply declining success, recurring interaction failures, and robust difficulty trends across scaffolds, generation strategies, and base models.

5.2 RESULTS

  • 18% success for o4-mini at level 1 falls to 4% at levels 5 and 6, while GPT-4.1 fails beyond level 3; humans reach 70% at level 6.All four evaluated language models decline steeply as difficulty increases.Source: §5.2 · p. 8
  • At levels 4 and 6, only o4-mini and Claude-3.7 achieve nonzero scores, at approximately 4%.The results indicate that current models struggle with long-term dependencies, state maintenance, and visual grounding over extended sequences.Source: §5.2 · p. 8

5.3 COMMON AGENT FAILURE MODES

  • Agents commonly fail through inaccurate clicks, weak screenshot and state tracking, and repeated actions without adaptation or recovery.These failures include misclicks, misidentified overlays, lost task state, and repetition after errors.Source: §5.3 · pp. 8–9

5.4 AGENT SCAFFOLDING EVALUATION

  • Agent S2.5 improves over bare agents similarly to Agent S3 and shows the same decline with increasing difficulty.This suggests the qualitative performance trends are robust across the evaluated scaffold designs.Source: §5.4 · p. 9, Figure 5 · p. 9

5.5 EFFECT OF INFORMATION ASYMMETRY

  • Direct-instruction generation success drops from 64% on Easy tasks to 11% on Hard tasks, whereas retained tasks maintain relatively high, flat evaluation success.The direct baseline targets approximately 5–10, 10–20, and 20–30 actions and retains only tasks with complete ground-truth trajectories.Source: §5.5 · p. 9, Table 5 · p. 10
  • AgentSynth keeps generation success high and stable while evaluation success falls from 62% at level 1 to 14% at level 6.This separates reliable trajectory collection from downstream task difficulty as more subtasks are summarized into composite instructions.Source: §5.5 · p. 10, Table 6 · p. 10

5.6 EFFECT OF BASE GENERATOR

  • Across GPT-4.1, GPT-5-mini, GPT-5, and GPT-5.1 generators, success decreases with difficulty and the stronger GPT-5.1 + S3 evaluator consistently outperforms GPT-4.1 + S3.Using GPT-5 or GPT-5.1 for synthesis produces slightly lower success rates, suggesting somewhat harder or more diverse tasks.Source: §5.6 · p. 10

6 CONCLUSION

AgentSynth provides a scalable pipeline for synthesizing diverse computer-use tasks and trajectories by decomposing complex tasks into solvable subtasks. Its benchmark enables fine-grained difficulty control and reveals sharply declining success as task complexity increases.

  • AgentSynth decomposes complex computer-use tasks into solvable subtasks, enabling fine-grained control over difficulty and long-horizon planning.Source: §6 · p. 10

ETHICS STATEMENT

The pipeline is intended for academic research on GUI-based computer-use agents and includes safeguards to reduce privacy, security, and service-impact risks. The authors also acknowledge possible misuse for harmful tasks such as cyberattacks.

  • The pipeline avoids credentials, personal information, backend-changing forms, malware-prone sites, and excessive external requests through prompting, secure-site selection, and action delays.Source: ETHICS STATEMENT · p. 11
  • The authors acknowledge that AgentSynth could improve LLM abilities for harmful tasks such as cyberattacks.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

The paper documents the pipeline, prompts, environment setup, evaluation protocol, cost accounting, and planned code and dataset release. It also states that LLMs were used only for editorial assistance rather than technical research contributions.

REPRODUCIBILITY STATEMENT

  • The authors document pipeline roles, OSWorld setup, action execution, verification, prompts, evaluation, dataset construction, and cost accounting to facilitate reproduction.Source: REPRODUCIBILITY STATEMENT · p. 11

A LLMS USAGE STATEMENT

  • LLMs assisted with grammar, wording, and minor style edits but did not contribute to research design, implementation, data processing, experiments, analysis, or results.Source: A LLMS USAGE STATEMENT · p. 14

B PROMPT DETAILS

C ADDITIONAL DATASET STATISTICS AND EVALUATION

The additional statistics characterize AgentSynth tasks across difficulty levels and include figures for dataset properties and model performance. Higher difficulty corresponds to longer, more coordinated, and more memory-intensive trajectories, while fine-grained perception remains relatively stable.

D EXAMPLE TASKS

AgentSynth generates realistic long-horizon tasks by composing simple subtasks into increasingly complex scenarios, with examples spanning calendar, document, spreadsheet, coding, and reporting workflows.

D EXAMPLE TASKS

  • Across examples, each follow-up task is simple and logically follows the prior state before the summarizer composes a coherent high-level task.This construction pattern is illustrated by the calendar sequence, where five follow-ups are generated and completed after the initial search.Source: D EXAMPLE TASKS · p. 21
  • Difficulty level n summarizes the first n subtasks, so adding subtasks produces progressively more complex and realistic evaluation scenarios.The composed tasks introduce additional actions, tools, and planning steps while preserving logical continuity between subtasks.Source: D EXAMPLE TASKS · pp. 21–22
  • Calendar examples progress from searching an academic calendar to finding a commencement date, navigating the Calendar app, creating an event, and setting a reminder.The six levels culminate in a titled event with a note and a one-day-before notification.Source: D EXAMPLE TASKS · p. 21
  • Environmental-investment tasks combine article research, source citation, sustainable-investment recommendations, text editing, and spreadsheet pivot-table preparation.The most complete version connects browser research and document creation with LibreOffice Calc data entry and pivot-table initiation.Source: D EXAMPLE TASKS · pp. 22–23
  • Advanced-programming tasks incrementally extend a VS Code script from environment setup and a print statement to functions, personalized output, and user input.The sequence repeatedly runs the script to verify each modification and introduces greet() and personal_greet as the task becomes more demanding.Source: D EXAMPLE TASKS · pp. 23–24
  • Whistleblower-reporting tasks combine SecureDrop research, anonymous-tip drafting, invoice creation, evidence citation, and PDF export.The final workflow saves both the text report and invoice PDF in Documents for anonymous submission.Source: D EXAMPLE TASKS · p. 25

E COST ANALYSIS

AgentSynth’s automated trajectory generation costs about $0.60 per trajectory, substantially below estimated human-curation costs for comparison datasets under stated labor-rate assumptions.

E COST ANALYSIS

  • Human-curation estimates use a labor-rate range of $2–$25 per hour because reported dataset construction costs and regional wages vary.The analysis uses $25 per hour as a graduate-student baseline while also accounting for rates as low as $2 per hour.Source: E COST ANALYSIS · p. 25
  • τ-bench costs roughly $4–$50 per task from reported inspection effort, but this is a lower-bound estimate excluding undisclosed database and API design costs.The estimate assumes more than 40 inspected trials per task at approximately three minutes per trial.Source: E COST ANALYSIS · p. 25
  • OSWorld costs approximately $8.8–$110 per task, averaging 4.4 hours of labor per task across 412 tasks.The dataset was created by nine computer-science students over approximately 1,800 person-hours.Source: E COST ANALYSIS · pp. 25–26
  • TheAgentCompany costs approximately $34–$425 per task, based on 17.1 hours of labor per task.This estimate derives from 3,000 person-hours used to create 175 tasks.Source: E COST ANALYSIS · p. 26
  • $0.60 per trajectory is AgentSynth’s average cost for a typical 50-step trajectory spanning six subtasks.Each execution step costs roughly $0.011, and generating six tasks from the six subtasks reduces the effective cost to $0.1 per task.Source: E COST ANALYSIS · p. 26

F APPLYING AGENTSYNTH PIPELINE FOR WEB AGENT

AgentSynth transfers its compositional task-generation approach to InSTA, where increasing the number of subtasks produces longer, harder web-agent tasks and exposes substantial performance degradation.

F APPLYING AGENTSYNTH PIPELINE FOR WEB AGENT

  • The InSTA evaluation covers 194 tasks in 61 task sequences using text-outlined webpages with numerically identified interactable elements and browser actions.Generation uses a Playwright server in Docker; five task sequences take 60–90 minutes with two workers on an E2 VM.Source: F APPLYING AGENTSYNTH PIPELINE FOR WEB… · p. 26
  • 12.9% and 6.45% are the summarized-task success rates for gpt-4.1 and o4-mini, down from 40% and 12% on InSTA’s baseline tasks.The comparison indicates that composed tasks are substantially more difficult than existing InSTA tasks.Source: F APPLYING AGENTSYNTH PIPELINE FOR WEB… · p. 26, Figure F.1 · p. 26
  • Increasing the number of included subtasks makes InSTA task descriptions longer, with Figure F.2 documenting growth in average token count.The paper connects longer compositions with more elaborate task descriptions and execution requirements.Source: F APPLYING AGENTSYNTH PIPELINE FOR WEB… · p. 26, Figure F.2 · p. 27
  • The appendix provides dedicated prompts for the InSTA task proposer and follow-up task proposer.These materials document the prompting components used for web-agent task generation.Source: Table F.1 · p. 27, Table F.2 · p. 28

G OSWORLD ACTION SPACE

  • Table G.1 specifies the computer-use agent’s action space and reports each action type’s relative frequency in the full AgentSynth dataset.The table concerns the broader dataset rather than the InSTA-specific evaluation results.Source: Table G.1 · p. 29

Related papers

Loading 2506.14205v2…