PaperlayerView exact revision on arXiv

Source-linked AI summary

TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution

Deyang Jiang, Jing Huang, Xuanle Zhao, Lei Chen, Liming Zheng, Fanfan Liu, Haibo Qiu, Peng Shi, Zhixiong Zeng

arXiv:2602.09662v1cs.CV

TL;DR

Scaling GUI planning is difficult because existing work emphasizes grounding, while diverse long-horizon trajectory synthesis remains costly and constrained by scarce open-source data. TreeCUA uses tree-structured verifiable evolution with multi-agent exploration, quality improvement, and branch-based preference learning. The resulting TreeCUA and TreeCUA-DPO models reportedly surpass existing models on OSWorld and OOD tasks, demonstrating strong generalization.

  • Problem

    Scaling diverse GUI planning trajectories remains challenging because prior work focuses mainly on grounding, while trajectory synthesis requires costly human verification and open-source data is scarce.

  • Method

    TreeCUA combines tree-structured exploration with multi-agent verification, summarization, evaluation, world-knowledge initialization, and adaptive exploration, while TreeCUA-DPO learns from adjacent branch preferences.

  • Results

    TreeCUA-7B and TreeCUA-DPO-7B significantly surpass existing models on OSWorld and OOD tasks, demonstrating strong performance and generalization.

  • Takeaways & Limitations

    Tree-structured verifiable evolution supports scalable GUI trajectory synthesis, and branching nodes provide preference data for training GUI planning models without additional data-collection costs.

  • Takeaways & Limitations

    Experiments require ephemeral, sandboxed virtual environments reset after each session to contain destructive or erroneous agent operations.

Abstract

from arXiv · show

Effectively scaling GUI automation is essential for computer-use agents (CUAs); however, existing work primarily focuses on scaling GUI grounding rather than the more crucial GUI planning, which requires more sophisticated data collection. In reality, the exploration process of a CUA across apps/desktops/web pages typically follows a tree structure, with earlier functional entry points often being explored more frequently. Thus, organizing large-scale trajectories into tree structures can reduce data cost and streamline the data scaling of GUI planning. In this work, we propose TreeCUA to efficiently scale GUI automation with tree-structured verifiable evolution. We propose a multi-agent collaborative framework to explore the environment, verify actions, summarize trajectories, and evaluate quality to generate high-quality and scalable GUI trajectories. To improve efficiency, we devise a novel tree-based topology to store and replay duplicate exploration nodes, and design an adaptive exploration algorithm to balance the depth (\emph{i.e.}, trajectory difficulty) and breadth (\emph{i.e.}, trajectory diversity). Moreover, we develop world knowledge guidance and global memory backtracking to avoid low-quality generation. Finally, we naturally extend and propose the TreeCUA-DPO method from abundant tree node information, improving GUI planning capability by referring to the branch information of adjacent trajectories. Experimental results show that TreeCUA and TreeCUA-DPO offer significant improvements, and out-of-domain (OOD) studies further demonstrate strong generalization. All trajectory node information and code will be available at https://github.com/UITron-hub/TreeCUA.

1. Introduction

TreeCUA targets the costly, underexplored problem of scaling diverse GUI planning trajectories rather than only GUI grounding. It organizes exploration as a reusable tree and combines multi-agent verification, adaptive exploration, and branch-based preference learning, with reported gains on in-domain and OOD tasks.

  • Research gap: Scaling long-horizon GUI planning remains difficult because it requires costly temporal reasoning and dynamic interaction data.Prior scaling efforts primarily addressed static GUI grounding, while trajectory collection often depends on human demonstrations or annotations.
  • Research gap: Repeated exploration of early functional entry points creates step redundancy, while distillation-model bias can reduce trajectory diversity.These challenges become more pronounced as trajectory scale increases and may underrepresent long-tail behaviors.
  • TreeCUA: TreeCUA organizes trajectories into a tree and uses a multi-agent framework to explore environments, verify actions, summarize trajectories, and evaluate quality.Its tree topology stores and reuses intermediate exploration nodes, while adaptive exploration balances trajectory difficulty and diversity.
  • TreeCUA-DPO: TreeCUA-DPO uses branching information from adjacent trajectories as preference pairs to enhance GUI planning capabilities.The method extends TreeCUA by leveraging rich tree-structured data for preference-based training.
  • Results: Experimental results report state-of-the-art performance on in-domain OSWorld and OOD tasks, surpassing existing open-source trajectories.The reported improvements include task success rates in online environments and strong generalization capabilities.

2. Related Works

Prior CUA research progressed from metadata-based interaction toward visual perception, but scalable planning-trajectory synthesis remains constrained by scarce open-source data and costly human verification. TreeCUA addresses this data-synthesis gap through tree-structured exploration and post-hoc quality improvement.

  • CUA paradigms: CUA research shifted from metadata-based methods toward holistic visual perception because structured metadata can be brittle in dynamic or poorly tagged environments.Earlier systems commonly relied on DOM trees and symbolic commands, whereas later approaches incorporated visual perception.
  • CUA capabilities: GUI automation requires both grounding and planning, but previous data-collection efforts concentrated mainly on GUI grounding.Grounding identifies visual elements from screenshots, while planning determines actions needed to complete tasks.
  • Data synthesis gap: Synthetic CUA datasets remain small, open-source trajectories are scarce, and costly human verification limits scalable synthesis of diverse planning trajectories.This combination is identified as a critical open challenge for the field.

3. Tree-Structured Data Synthesis Pipeline

Tree-Structured Verifiable Evolution models GUI exploration as a tree and coordinates specialized agents for exploration, verification, summarization, filtering, and reasoning improvement. World-knowledge initialization, adaptive topology, global memory, and deterministic replay support diverse, efficient, and higher-quality trajectory synthesis.

  • Tree-Structured Verifiable Evolution: The framework coordinates five specialized agents across online tree exploration and post-hoc quality evaluation and improvement.The exploration manifold represents states as nodes and transition actions as edges.
  • Initialization with World Knowledge: World-Knowledge Initialization organizes documentation into hierarchical task categories and initializes applications with context-specific assets for semantically rich exploration.The environment is formulated as s0 = Φ(wc, P), where wc selects a category and P supplies pre-configured assets.
  • Online Exploration: Each exploration node stores an executable action, immediate and long-term goals, an expected observation, and a rationale, while the policy conditions generation on visual input, knowledge, history, and global prefix memory.The exploration agent samples K candidate actions and retains textual history for context efficiency.
  • Adaptive Tree Topology: Adaptive topology reduces the branching-factor bound with depth, prioritizing early diversity while concentrating computation later through prefix sharing.The strategy partitions trajectories into discovery, development, and convergence phases.
  • Step Verification: Step verification classifies transition outcomes and filters invalid branches while providing immediate feedback for on-the-fly error recovery.The verifier checks whether interface state changes match expected outcomes.
  • Global Memory and Replay: Global prefix memory imposes novelty constraints during shallow exploration, while deterministic node replay revisits targets by resetting to s0 and re-executing recorded actions.These mechanisms reduce redundancy across trees without requiring arbitrary operating-system snapshots.
  • Quality Evaluation and Improvement: Hierarchical summarization extracts trajectory-level instructions and coherent intent-driven sub-trajectories, followed by quality scoring and hindsight reasoning synthesis.Quality evaluation considers task utility, step efficiency, consistency, and coherence; synthesized reasoning covers observation, progress, planning, and impact.

4. Synthesized Data Statistics and Analysis

Tree-structured synthesis produces a large filtered dataset while balancing diversity against computational cost. Analyses show benefits from node reuse, world knowledge, and global history for exploration efficiency, task diversity, and reduced inter-tree redundancy.

  • Dataset Overview: 50k high-quality trajectories are retained from an initial pool of 100k generated trajectories.The filtered trajectories are further decomposed into high-quality sub-trajectories and step-level samples.
  • Dataset Overview: 101k high-quality sub-trajectories and 708k step-level training samples are compiled after redundant-stage pruning and single-step validation.The final dataset aggregates explored tree nodes and validates individual execution outcomes.
  • Tree Depth and Efficiency Analysis: Branching depth mediates diversity and cost: shallow branching increases diversity but overhead, whereas deeper branching improves prefix reuse but restricts diversity.The framework targets a balance between trajectory coverage and computational efficiency.
  • Tree Depth and Efficiency Analysis: Trajectories predominantly branch around depth 10, and tree-structured exploration leverages node reuse to reduce average inference overhead relative to sequential baselines.Sequential methods accumulate total cost linearly and maintain constant average inference overhead.
  • Comparison of Synthesis Methods: Table 2 characterizes the method as fully automated, tree-structured exploration with amortized costs and dual-level verification.The supplied comparison caption states that this outperforms baselines.
  • World Knowledge and Diversity: With world knowledge, VS Code exploration discovers 535 unique semantic tasks versus 344 without world knowledge under the stated TF-IDF threshold.The with-knowledge condition also shows a significantly higher median Type-Token Ratio across repeated samples.
  • Inter-Tree Redundancy: Global history reduces average inter-tree action redundancy from 0.17 without history to 0.08 with history.Redundancy is measured using pairwise Jaccard similarity after matching action type, grid location, and input text.

5. TreeCUA Training Recipe

TreeCUA’s training recipe combines two-stage SFT with TreeCUA-DPO, using tree-structured exploration to provide verified trajectories and preference pairs for GUI planning.

  • Training recipe: The recipe trains foundational perception and planning before cognitive alignment, followed by TreeCUA-DPO to optimize planning.The method therefore orders capability acquisition, intent alignment, and preference optimization as successive training components.
  • Two-stage SFT: Two-stage SFT first builds foundational exploration, then aligns the model with realistic human intent using cognitive intent data.Stage 1 uses filtered step-level data and summarized tasks; Stage 2 refines tasks with human-expert examples and resampled trajectories.
  • TreeCUA-DPO: TreeCUA-DPO uses branching nodes to contrast successful trajectories and construct preference pairs without additional data-generation cost.The framework discards pairs whose positive action fails verification and samples uniformly across trajectory depths with a per-node cap.
  • TreeCUA-DPO: When multiple actions are physically valid, Tree-DPO uses their differing outcomes and goals to align predictions with semantic intent rather than simple action validity.This preference construction separates interface affordance from the user’s intended goal.

6. Experiments

Experiments evaluate TreeCUA against open-source baselines on OSWorld-Verified and constructed OOD tasks, finding stronger performance and benefits from both training stages and tree-structured data.

  • Experimental setup: The evaluation compares against open-source models and trajectory-synthesis pipelines using OSWorld-Verified as the primary benchmark.Official leaderboard scores are used for listed models, while original-paper results are used for unlisted models.
  • OSWorld-Verified: TreeCUA-7B outperforms similarly scaled ScaleCUA-7B and OpenCUA-7B on OSWorld-Verified, while TreeCUA-DPO-7B performs better than TreeCUA-7B.The authors report especially strong DPO benefits on logic-intensive and sequential tasks such as TB, Code, and Chrome.
  • OOD evaluation: TreeCUA significantly increases OOD performance compared with the Qwen2.5-VL-7B baseline on the constructed benchmark.The OOD benchmark covers six applications and uses 20 tasks per application.
  • Ablation study: Removing either SFT stage causes marked performance drops across ID and OOD evaluations.The authors report that excluding cognitive intent data is especially detrimental because it aligns the agent with human decision-making patterns.
  • Data comparison: Using the same Qwen2.5-VL-7B backbone, TreeCUA’s synthetic data consistently outperforms OpenCUA and ScaleCUA data on both ID and OOD benchmarks.These comparisons are summarized in Table 6 as evaluation on in-domain OSWorld and out-of-domain tests.

7. Conclusion

TreeCUA scales GUI automation through tree-structured verifiable evolution and extends the resulting branching information into TreeCUA-DPO, with reported gains on OSWorld and OOD tasks.

  • Conclusion: TreeCUA uses a multi-agent framework and tree-structured verifiable evolution to scale GUI trajectory synthesis efficiently.The approach combines world knowledge initialization, online asynchronous exploration, and diversified post-processing to improve data diversity and quality.
  • Conclusion: TreeCUA-DPO derives preference data from branching nodes without additional costs and uses it to train the planning model.The conclusion presents branching nodes as a source of preference supervision for the DPO extension.
  • Conclusion: TreeCUA-7B and TreeCUA-DPO-7B significantly surpass existing models on OSWorld and OOD tasks, demonstrating strong generalization.This is the paper’s reported overall experimental conclusion.

Impact Statement

The experiments isolate agent interactions in ephemeral sandboxed virtual environments that reset after each session to contain destructive or erroneous operations.

  • Safety isolation: Ephemeral sandboxed virtual environments confine destructive or erroneous agent operations to each simulation session.Environments are reset after every session to mitigate risks from irreversible OS commands.

A. Implementation Details of Scalable Execution

TreeCUA restores and validates GUI states through deterministic replay, then expands the exploration tree with concurrent workers and hybrid traversal. Its reasoning-quality evaluation uses ROSCOE-inspired dimensions and reports stronger results than Claude.

  • Deterministic Replay: Replay resets the environment to s0 and sequentially executes the stored action history to reconstruct a target state.This mechanism addresses operating systems that lack emulator-style state snapshotting.
  • Visual Consistency: An RMS screenshot check validates semantic equivalence between the original and replayed states despite visual noise.The check uses ϵ = 5.0 for 0–255 pixel values; failed branches are marked CORRUPTED and pruned.
  • Concurrent Exploration: A multi-worker framework samples unexplored nodes, restores their states, and resumes tree exploration asynchronously.Workers retain one child for local extension while dispatching K −1 siblings to a global pool.
  • Reasoning Evaluation: TreeCUA significantly surpasses Claude across all ROSCOE-inspired metrics in the zero-shot OOD reasoning evaluation.The reported comparison concerns reasoning quality on the offline AndroidControl dataset.

C. Experimental Setup

The experiments use Claude-4.5-Sonnet for exploration, GPT-4o-mini for auxiliary agents, and Qwen2.5-VL-7B as the base model. They also compare SFT and DPO variants across application domains.

  • Data Synthesis Setup: Claude-4.5-Sonnet serves as the exploration backbone, while GPT-4o-mini handles verification, summarization, and evaluation.This division balances exploration quality with auxiliary-agent cost and efficiency.
  • Training Setup: Qwen2.5-VL-7B is the base model, trained with a three-stage pipeline and global batch size 32.The setup identifies the base model and overall training configuration.
  • Preference Alignment Analysis: Table 7 compares TreeCUA-7B Stage 3 with TreeCUA-DPO-7B Stage 4 across applications, sorted by net performance gain.The comparison isolates the performance shift associated with the preference-alignment stage.

D.1. Theoretical Attribution: Action Space and Topology

DPO gains vary substantially across applications and are attributed to how application action spaces and workflow topologies align with the TreeCUA exploration tree. The paper contrasts discrete hierarchical tasks with continuous or in-situ interactions using synthesized trajectory examples.

  • Performance Variation: +20.0% in Thunderbird and -10.6% in Impress mark the observed range of DPO gains across applications.The paper attributes this divergence to structural alignment between the exploration topology and application interaction logic.
  • Action Space: Discrete Semantic Actions provide sharper preference signals because correct and incorrect branches differ in a binary, readily distinguishable way.The strongest gains occur in environments such as Thunderbird and Code, whereas Impress requires Continuous Visual Precision.
  • Topology: DPO is most effective when workflows follow a Hierarchical Navigation Topology that resembles the exploration tree.Chrome and Thunderbird traverse distinct states, unlike shallow, wide In-Situ Mutation workflows in Impress and Calc.
  • Trajectory Examples: Figures 6 and 7 visualize a synthesized task trajectory together with its task instruction, reasoning, goals, actions, and verification results.The example includes a Chrome workflow involving history, startup, language, appearance, and default-browser settings.
Loading 2602.09662v1…