OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

Trajectory Data

• Ideal GUI trajectories combine high-level instructions, low-level instructions, actions, and visual or textual states for end-to-end agent training.These components support progression from action automation toward full-process autonomy.Source: §1 · pp. 1–2

Motivation

• Existing GUI trajectory collection is costly for humans and limited by predefined tasks, restricted diversity, and inconsistent intermediate-step quality.Human collection requires full-trajectory annotation and manual task design, while model-based synthesis can produce incomplete or incoherent trajectories.Source: §1 · p. 2

Approach

• OS-Genesis explores GUI environments through interactions, converts observed state transitions into low-level instructions, and derives broader high-level tasks for trajectory collection.The pipeline avoids human supervision and predefined tasks while linking abstract objectives to dynamic GUI behavior.Source: §1 · p. 2

Results

• 17.41% versus 9.82%: OS-Genesis nearly doubles AndroidWorld performance over task-driven methods.Experiments on AndroidWorld and WebArena demonstrate effectiveness on challenging online benchmarks.Source: §1 · p. 2

Agents for Digital Automation

• Language-agent research spans prompting, collaboration, tool use, self-improvement, and model integration, alongside fine-tuning for perception, action generation, and multi-environment operation.These efforts target digital automation across web, desktop, and mobile platforms.Source: §2 · p. 2

Data for Building Computer Agents

• GUI datasets cover mobile, web, and desktop applications, but effective agent training requires trajectories pairing states, hierarchical instructions, and actions.Existing resources include Rico, Mini-Wob, and later platform-specific datasets.Source: §2 · p. 3
• Current trajectory resources often omit essential components, depend on costly manual curation, or target a single GUI platform.These limitations restrict broader applicability.Source: §2 · p. 3

3.1 Interaction-Driven Functional Discovery

• Interaction-driven discovery systematically traverses mobile and web interfaces with actions such as CLICK, TYPE, and SCROLL, collecting pre-action state, action, and post-action state triplets.The exploration is rule-based except for generating contextually appropriate input-field contents.Source: §3.1 · pp. 3–4

3.2 Reverse Task Synthesis

• Reverse task synthesis maps each observed transition to an atomic executable low-level instruction, then associates it with a broader goal-oriented high-level task.For example, opening a dropdown can become a step toward configuring application settings.Source: §3.2 · p. 4
• The synthesis process generates semantically rich, environment-aligned high-level instructions without human intervention, which are then executed to produce complete trajectories.The resulting instruction set is denoted T and the generated trajectories G.Source: §3.2 · p. 4

3.3 Trajectory Reward Model

• The Trajectory Reward Model scores trajectories from 1 to 5 using completion and coherence, rather than discarding every incomplete trajectory.It considers the high-level instruction, action history or low-level instructions, and the last three states.Source: §3.3 · p. 5
• Reward-based sampling uses trajectory rewards to select training data, preserving diverse exploratory trajectories while emphasizing higher-quality examples.Algorithm 1 samples trajectories according to normalized reward-based probabilities.Source: §3.3 · p. 5

4.1 Experimental Settings

• Experiments use AndroidControl, AndroidWorld, and WebArena with InternVL2 and Qwen2-VL backbones, comparing zero-shot, task-driven, and self-instruction baselines.Training uses trajectory-based supervised fine-tuning with separate planning and action objectives.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.2 Baseline Construction and Training

• OS-Genesis and task-driven baselines use 1K trajectories, while self-instruction uses 1.5K trajectories for training.The comparison therefore includes a larger-data self-instruction baseline.Source: §4.2 · p. 6

4.3 Main Results

• On AndroidWorld, training with OS-Genesis often achieves performance improvements double those of task-driven baselines, narrowing the gap between open-source agents and the GPT-4o-based M3A agent.Even with 1.5 × more data, the self-instruct baseline fails to match OS-Genesis trajectory quality in online settings.Source: §4.3 · p. 6
• On AndroidControl, OS-Genesis consistently improves action and planning abilities across various backbones, outperforms GPT-4o especially in the low-level setting, and excels in the high-level setting.Only 20 of 833 apps covered by AndroidControl were directly encountered during data synthesis, making the evaluation a test of out-of-distribution performance.Source: §4.3 · pp. 6–7
• On WebArena, OS-Genesis training improves performance across backbones and enables substantial gains for models weak under zero-shot evaluation.Its advantage over task-driven baselines is especially apparent in websites with many interactive elements.Source: §4.3 · p. 7

5.1 How Diverse is Our Synthesized Data?

• OS-Genesis achieves the greatest instruction and trajectory diversity among synthetic methods across mobile and web environments.Human data has high instruction diversity but lower trajectory diversity, whereas OS-Genesis is high on both dimensions.Source: §5.1 · pp. 7–8

5.2 How TRM Impacts Performance?

• Trajectory Reward Modeling improves high-level performance, while labeler filtering provides slight high-level gains but reduces low-level performance.Low-level performance improves consistently across reward strategies because individual OS-Genesis steps remain useful.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 How Scaling Trajectory Data Improves Agentic Ability?

• Performance generally rises as the number of OS-Genesis trajectories increases, but saturation appears at larger scales.The authors attribute saturation to VLM capacity and limits in exploration and trajectory materialization.Source: §5.3 · p. 8, Figure 6 · p. 8

5.4 How Far are We from Human Data?

• Human-written high-level instructions underperform OS-Genesis instructions when used to construct training trajectories.The paper attributes this to task-environment misalignment and errors in interpreting annotator intentions.Source: §5.4 · p. 9, Figure 7 · p. 9
• OS-Genesis trajectories narrow the gap to human demonstrations, retaining more than 80% of the human-data average success rate.The effect is especially evident for high-level tasks, where OS-Genesis agents plan and solve problems more like humans.Source: §5.4 · p. 9, Figure 8 · p. 9

• OS-Genesis provides a promising interaction-driven direction for generating high-quality trajectory data for GUI agent training.The pipeline targets diversified computer-control agents and addresses bottlenecks in constructing meaningful and diverse GUI tasks.Source: §6 · p. 9

Proprietary Models

• GPT-4o is used for exploration and reward modeling because existing open-source VLMs cannot reliably follow instructions and proactively explore online environments.The authors expect more capable action models to replace these proprietary components in the future.Source: Limitations · p. 10

Data usage

• The agents use both textual and visual representations, while using either modality alone and partial full-trajectory data remain future work.The multimodal choice is intended to support planning and action in semantically rich environments and maintain evaluation consistency.Source: Limitations · p. 10

Model-based Trajectory Construction

• The number of successfully constructed trajectories is constrained by the capabilities of the task-executing model.More advanced VLMs with GUI capability are expected to address this limitation.Source: Limitations · p. 10

• Virtual environments limit concern about computer agents affecting normal system functioning.Source: Broader Impacts · p. 10

AndroidControl

• AndroidControl evaluates real-world mobile control using 7,708 tasks across 1,412 human-collected trajectories and reports success rate and action-type accuracy.The setup adds screenshots’ accessibility trees and historical actions, with coordinates counted correct within 14% of screen width from ground truth.Source: A Details of Benchmarks · p. 14

AndroidWorld

• AndroidWorld evaluates autonomous agents on 112 usable tasks across 20 real-world Android apps, using randomized inputs and system-state inspections for success.Tasks remaining incomplete after NaN retesting are marked false for fair comparisons.Source: A Details of Benchmarks · p. 14

WebArena

• WebArena evaluates autonomous web agents on 812 navigation tasks derived from 241 templates using functional-correctness success rates.The setup uses the default click-and-input action space with screenshots and accessibility trees as observations.Source: A Details of Benchmarks · p. 14

Experimental Details

• AndroidWorld adds terminate and keyboard_enter actions to meet evaluation requirements.Source: B Experimental Details · p. 14
• Mobile and web action spaces are documented separately in Tables 3 and 4.Source: Table 3 · p. 14, Table 4 · p. 14
• AndroidWorld and AndroidControl evaluations use dedicated prompts listed in Prompts 16 and 17.Source: B Experimental Details · p. 14

C Reverse Task Synthesis Details

• Reverse task synthesis gives GPT-4o the current action, before-and-after screenshots, and a highlighted interacted element to infer high-level tasks from observed changes.Source: C Reverse Task Synthesis Details · pp. 14–15

D Model and Training Details

• Training configurations include dynamic image processing, filtered accessibility trees, and benchmark-consistent data formats for GUI agents.InternVL2 uses up to 24 image tiles, Qwen2-VL uses image_resolution 1024, and accessibility trees retain visible-element position or index information.Source: D Model and Training Details · p. 15

E.1 Task-Driven

• The task-driven baseline prompts GPT-4o to infer high-level instructions from initial interfaces and task examples.Source: §E.1 · p. 15

E.2 Task-Driven w. Self Instruct

• The self-instruct baseline samples three task-driven demonstrations as in-context examples, while matching or exceeding OS-Genesis trajectory counts for fair comparison.Source: §E.2 · p. 15

F Details of Trajectory Reward Model

• The Trajectory Reward Model scores trajectories from 1 to 5 using completion and coherence, based on the instruction, action history, and final three screenshots.The same TRM is applied to mobile and web tasks.Source: F Details of Trajectory Reward Model · p. 15

F.1 Model Scoring Alignment

• OS-Genesis remains effective when Qwen2.5-VL-72B replaces GPT-4o as the trajectory-reward-model backbone.Source: §F.1 · p. 15, Table 5 · p. 16

F.2 Human Scoring Alignment

• TRM rewards align well with human evaluations on sampled web and mobile trajectories.The comparison uses human labels collected under the same guidelines as the GPT-4o-based TRM.Source: §F.2 · p. 16, Table 6 · p. 16

Instruction Diversity

• OS-Genesis generates more diverse instructions through its exploration-driven method.This diversity is demonstrated by visualizing instruction embeddings.Source: G Details about Diversity Analysis · p. 16

Instruction Length

• 18.01 words for mobile and 19.68 words for web, making OS-Genesis instructions longer and more context-rich than baseline instructions.For mobile tasks, human instructions average 18.71 words; for web tasks, Task-Driven and Self-Instruction average 11.79 and 8.45 words, respectively.Source: G Details about Diversity Analysis · p. 16

WebArena Evaluation Prompt

• The WebArena evaluation prompt instructs agents to issue one valid browser action at a time from accessibility-tree observations, URLs, screenshots, and task objectives.It defines page, tab, navigation, and completion actions, requires a fixed answer phrase and action format, and supports step-by-step reasoning.Source: G Details about Diversity Analysis · p. 24

Trajectory Reward Model

• The trajectory reward model evaluates high-level instructions, stepwise reasoning and actions, and GUI screenshots from the trajectory.For trajectories with at least three states, it includes screenshots of the last state; otherwise, it includes all states.Source: G Details about Diversity Analysis · p. 25
• The reward model scores trajectory coherence by checking logical progress, action specificity, and the absence of redundant or unnecessary actions.It also evaluates whether the instructed task is completed, all necessary interactions occur, and errors are handled appropriately.Source: G Details about Diversity Analysis · p. 25
• Scores range from 1 to 5, with 5 denoting perfect completion through clear, multi-action execution without noticeable redundancies.Lower scores represent partial completion, inefficient or goal-divergent behavior, loops, deadlocks, inaccessible tasks, or no meaningful initial actions; complex attempts may receive upward adjustment when they contribute meaningfully.Source: G Details about Diversity Analysis · p. 25