Source-linked AI summary
ANCHOR: Branch-Point Data Generation for GUI Agents
Jinbiao Wei, Yilun Zhao, Kangqi Ni, Arman Cohan
TL;DR
GUI agents need scalable, high-quality desktop interaction data, but human demonstrations are costly and existing synthesis can lack diversity or drift from task goals. ANCHOR expands verified seed trajectories at meaningful UI state changes, executes grounded task variants, and applies verification and step-level filtering. Across OSWorld and WindowsAgentArena, fine-tuning on the expanded corpus consistently outperforms representative baselines across operating systems and applications.
Problem
High-quality desktop GUI trajectories are expensive to collect, difficult to scale, and existing synthesis pipelines can produce limited diversity or noisy, goal-drifting data.
Method
ANCHOR expands verified seed trajectories by identifying meaningful UI branch points, proposing state-grounded task variants, executing them, and applying verification and step-level quality controls.
Results
Fine-tuning on ANCHOR-generated trajectories consistently improves performance across OSWorld and WindowsAgentArena, operating systems, applications, and evaluated backbones, outperforming representative synthesis baselines.
Takeaways & Limitations
State-grounded branching provides a practical route to scalable, reliable long-horizon desktop GUI supervision without requiring flawless exploration.
Takeaways & Limitations
Experiments focus on desktop workflows, while extension to mobile and web settings remains future work.
Abstract
from arXiv · showhide
End-to-end GUI agents for real desktop environments require large amounts of high-quality interaction data, yet collecting human demonstrations is expensive and existing synthetic pipelines often suffer from limited task diversity or noisy, goal-drifting trajectories. We present a trajectory expansion framework Anchor that bootstraps scalable desktop supervision from a small set of verified seed demonstrations. Starting from each seed, we identify branch points that correspond to meaningful state changes and propose new, state-grounded task variants conditioned on the current GUI context. An executing agent then follows the proposed instructions to generate new trajectories, while a verifier enforces task completion via state-aware checks and trajectory-level consistency. To improve supervision quality, we further apply task-conditioned step-level filtering to remove ungrounded actions and denoise post-branch segments to maintain coherent intent. Experiments on standard desktop benchmarks, OSWorld and WindowsAgentArena, show that models fine-tuned on our expanded corpus achieve consistent improvements over zero-shot agents and representative synthesis baselines, and generalize across applications and operating systems.
1 Introduction
ANCHOR addresses the scarcity and cost of high-quality desktop GUI trajectories by expanding verified seed demonstrations into diverse, quality-controlled supervision. Experiments show consistent gains over synthesis baselines across desktop benchmarks and model backbones.
- High-quality GUI training data is labor-intensive to collect and difficult to scale across applications, configurations, and task goals.
- ANCHOR expands a small set of verified desktop trajectories by branching at UI state changes and synthesizing new task variants during execution.The pipeline summarizes rollouts and verifies task completion.
- The resulting pipeline targets high-diversity, high-fidelity trajectories while reducing dependence on perfect model behavior.
- Fine-tuning different models on the expanded dataset consistently outperforms competitive baselines on challenging desktop benchmarks.
- Systematic ablations and diagnostic analyses identify contributors to performance gains and remaining failure modes.
2 Related Work
Prior GUI trajectory synthesis methods use tutorials, explicit task specifications, or unconstrained interaction, each with limitations in coverage, executor dependence, or trajectory quality. ANCHOR instead expands verified demonstrations around UI-defined branch points.
- Tutorial-based methods parse web resources into tasks and instructions, but their coverage depends on tutorial availability and diversity.
- Task-driven synthesis couples dataset quality to executor competence, while task diversity is bounded by the proposal model’s UI knowledge.
- ANCHOR contrasts with these paradigms by expanding around verified seed demonstrations and branching at UI-defined decision points.
- Interaction-driven synthesis improves state coverage through free-form exploration but can produce unconstrained trajectories.
3 GUI Agent Trajectory Expansion
ANCHOR generates desktop GUI supervision by expanding high-quality seed trajectories at meaningful UI branch points, then executing, summarizing, verifying, and filtering the resulting rollouts. Its controls target diversity, task grounding, and coherent long-horizon behavior.
- Pipeline Design Principles: Alternative goal-directed branches at UI-defined decision points provide systematic diversity without unconstrained exploration.
- Pipeline Design Principles: Concrete branch-point objectives reduce aimless navigation and low-level click sequences, while localized sub-tasks reduce dependence on model perfection.
- Pipeline Design Principles: The pipeline starts from a small set of high-quality seed demonstrations in executable GUI environments.A gold trajectory is a state-action sequence that successfully completes its task.
- Seed Trajectory Collection: Human annotators validate candidate seed runs and retain the shortest successful trajectory as the branching root.
- Branch-Point Identification: A branch point stops the original task at a state where substantial UI changes or newly visible content expose additional affordances.
- Branch Task Proposal: New tasks are proposed from the trajectory prefix together with a summary of progress, UI changes, and the current GUI state.
- Task Execution: During execution, task descriptions may be refined into feasible, semantically close variants when actions drift or the environment is incompatible.
- Summarization and Verification: A summarizer abstracts low-level operations into a user-level task description, and a verifier retains only completed trajectories judged successful.
4 Experiment Setup
The experiments evaluate ANCHOR-generated supervision on OSWorld and WindowsAgentArena using multiple baselines and vision-language backbones. Training uses trajectory-based supervised fine-tuning with screenshot history and interaction context.
- Evaluation Benchmarks: Evaluation uses OSWorld and WindowsAgentArena, with tasks judged by execution-based or deterministic final-state success checks.The evaluation excludes tasks used as branching seeds and removes tasks judged incompletable by human inspection.
- Baselines: The comparison includes zero-shot agents and a task-driven synthesis pipeline that generates task instructions and executions.These baselines assess robustness against both no supervised fine-tuning and representative goal-conditioned synthesis.
- Training Setup: Trajectory-based supervised fine-tuning conditions each step on the current screenshot, two preceding screenshots, and interaction history.The model operates in a tool-call GUI action space.
- Training Setup: Supervision combines prediction of a step-level reasoning state and next tool-call action with next-action-only prediction.The two supervision components are combined in the final training objective.
- Training Setup: The study fully fine-tunes GLM-4.1V-9B-Base, Qwen2.5-VL-7B-Instruct, and Qwen3-VL-8B-Instruct.The backbones span general-purpose and instruction-tuned multimodal models with different UI-understanding and grounding capabilities.
5 Experiment Results
ANCHOR-generated trajectories improve desktop-agent performance across OSWorld and WindowsAgentArena, backbones, applications, and operating systems. Scaling and quality-control analyses further show benefits from more in-domain or cross-domain supervision and from filtering noisy steps.
- Main Results: ANCHOR fine-tuning achieves the strongest overall OSWorld performance across all three evaluated backbones.For Qwen2.5-VL-7B, it reaches 7.94 overall versus 5.61 for task-driven synthesis and 4.67 for AgentNet human demonstrations.
- Main Results: On WindowsAgentArena, branching-generated supervision consistently improves over zero-shot and task-driven synthesis across all backbones.Scores reach 16.30 for GLM-4.1V-9B, 15.22 for Qwen2.5-VL-7B, and 30.76 for Qwen3-VL-8B.
- Qualitative Analysis: ANCHOR grounds complex multistep instructions in the correct GUI path, unlike task-driven and human-data models in the auto-save example.Branching from visually rich hub states yields descendant trajectories covering multiple settings panels and controls.
- Data Scaling Analysis: Increasing in-domain data generally improves OSWorld success, with Qwen2.5-VL-7B rising from 0.93 at 0.3K trajectories to 7.94 at 1K.Qwen3-VL-8B shows a small-scale dip before reaching 20.56 at 1K, attributed to low-scale format mismatch.
- Data Scaling Analysis: Adding 600 WindowsAgentArena trajectories to 1K Ubuntu trajectories raises Qwen2.5-VL-7B OSWorld success from 7.94 to 9.95.The larger-scale cross-domain advantage is associated with additional UI diversity and more robust grounding and recovery behaviors.
- Data Quality Controls: Removing step-level filtering and post-branch denoising consistently degrades performance across models.The ablation supports retaining task-consistent steps while excluding noisy or low-signal operations.
6 Conclusion
ANCHOR is a trajectory expansion framework for scalable desktop GUI data generation. By branching from verified demonstrations at meaningful state changes, it produces grounded trajectories that improve benchmark performance and generalization.
- ANCHOR expands desktop GUI data by branching from verified seed demonstrations at meaningful state-change points.The framework targets scalable trajectory generation while preserving task grounding.
- Task-conditioned filtering and post-branch denoising support diverse, executable trajectories with coherent intent.These quality-control stages reduce reliance on flawless rollout policies.
- Fine-tuning on ANCHOR-generated data consistently outperforms baselines across two desktop benchmarks and multiple VLM backbones.The reported gains extend across operating systems and applications.
- The framework offers a practical path to scaling high-quality GUI supervision without exhaustive human demonstrations or flawless exploration.This conclusion is stated within the scope of desktop GUI data generation.
Limitations
The experiments focus on desktop workflows and two desktop computer-use benchmarks. The pipeline is not platform-specific, but extension to mobile and web settings remains future work.
- The evaluation is limited to desktop workflows on OSWorld and WindowsAgentArena.These benchmarks align with the authors’ desktop environment setup.
- The branch-point expansion pipeline is not tied to a specific platform.The authors suggest extending evaluation to mobile and web settings with different UI primitives and accessibility signals.
- Future work includes studying the pipeline in mobile and web environments.Such settings may involve different UI primitives and accessibility signals.
A Details of Benchmarks
The benchmarks evaluate GUI agents through execution-based checks on controlled virtual machines. OSWorld primarily covers Ubuntu, while WindowsAgentArena focuses exclusively on Windows 11.
- OSWorld: OSWorld wraps full desktop operating systems, primarily Ubuntu, in controlled virtual-machine environments.Tasks include natural-language goals, reproducible initial snapshots, and execution scripts that inspect final state.
- OSWorld: OSWorld evaluates success only when its execution checker returns a positive signal.Agents interact through screenshots or optional accessibility metadata and GUI-level actions.
- WindowsAgentArena: WindowsAgentArena extends the OSWorld framework to an execution-based benchmark focused exclusively on Windows 11.Tasks run on Windows 11 virtual machines packaged inside Docker containers.
- WindowsAgentArena: WindowsAgentArena uses deterministic Python evaluators that inspect final state and return binary success flags.Its observation and action spaces mirror real Windows usage.
- Evaluation protocol: Evaluation uses official deterministic scripts over task sets disjoint from training and data-generation tasks.This protocol is stated for the WindowsAgentArena experiments.
B Experimental Details
The experiments represent GUI trajectories as single-function computer-use calls and document the action space and generation prompts. Evaluation excludes benchmark tasks used as branching seeds.
- Action representation: Each trajectory step predicts exactly one computer_use function call with an action field and optional arguments.Arguments may include coordinates, text, and keys.
- Action space: The complete GUI action space used in training is listed in Table 6.The table is identified as the action-space specification for the GUI setting.
- Prompt design: Prompts cover branch-state identification, task generation, execution, summarization, verification, denoising, and filtering.The corresponding prompts are presented across Figures 4–8.
- Data split: Evaluation excludes benchmark tasks whose IDs were used as branching seeds during data generation.Figures 9 and 10 list the selected seed task IDs for OSWorld and WindowsAgentArena.
C Qualitative Analysis Example
The passage only indicates that the qualitative comparison is shown in Figure 11.
- Figure 11 contains the referenced qualitative comparison.
D Training Details
Training uses equal trajectory budgets across methods and comparable trajectory horizons, with Anchor supervision built around branch-point identification and state-conditioned task generation. Models are fully fine-tuned using a standardized distributed PyTorch setup and fixed visual context.
- Fair-budget training: Each method uses 1,000 trajectories on OSWorld and 600 trajectories on WindowsAgentArena, with comparable horizons of about 18 steps on average.
- Branch-point identification: Branch-point identification selects intermediate states where new tasks can begin, prioritizing visually rich, decision-heavy states that unlock multiple possible continuations.The prompt recommends 3–5 high-quality branch points and avoids states where the next action is essentially forced.
- Optimization and infrastructure: Training uses fully fine-tuned models with FP16 mixed precision, AdamW optimization, a linear learning-rate schedule, gradient clipping, and one epoch over the shuffled dataset.Distributed execution uses four NVIDIA H200 GPUs, gradient accumulation to a global batch size of 16, and optional memory-efficiency features including DeepSpeed ZeRO-3.
- Input format: Supervised examples predict the next step from the current GUI state, interaction context, the current screenshot, and two preceding screenshots.Screenshots use the original 1920 × 1080 resolution, and the vision encoder uses patch size 16.
E Annotator Information
The pipeline uses human annotators for seed validation and auditing, while prompts automate progress summarization, task generation, reasoning filtering, and task-success verification. Generated tasks are constrained to be feasible, distinct, and verifiable from the current GUI state.
- Human annotation: Human annotators validate candidate seed trajectories and audit 100 automatically verified synthetic trajectories without access to verifier predictions.They inspect task instructions together with screenshots and actions, then judge whether the final state satisfies the task.
- Progress summarization: Progress summaries describe completed GUI actions chronologically, focusing on navigation, opened panes, and edited content rather than abstract plans.
- Branch-task generation: Generated follow-up tasks must be specific, feasible, verifiable, different from the original, non-authenticated, and completable within 5–15 steps.The generator also avoids redundancy with previously proposed tasks and returns mutually different tasks.
- Agent supervision: Planning-oriented supervision predicts reasoning and a tool call, whereas action-oriented supervision conditions on reasoning and predicts only the next tool call.
- Reasoning filtering: Candidate reasoning is generated as multiple plausible single-step actions and filtered by comparing each candidate with the recorded action and before/after screenshots.
- Task verification: Task verification evaluates whether the instruction is completed using the full action trajectory and chronological screenshots, returning a binary success judgment with a brief explanation.