PaperlayerView exact revision on arXiv

Source-linked AI summary

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Zihan Ding, Longxu Dou, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Zihang Jiang, Lei Ke, Mengcheng Lan, Weixian Lei, Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, Yingchen Yu, Chenyu Zhang, Yuhao Zheng

arXiv:2608.15930v1cs.AIcs.CV

TL;DR

GUI agents face training-data and interaction gaps that limit reliable execution of complex, user-specific workflows. UI-Mate combines environment-grounded training with demonstration-guided workflows, and reports the strongest open-weight results in its comparison alongside improved OSWorkerBench strict success over its base model.

  • Problem

    GUI agents face training and interaction bottlenecks: scalable learning needs executable environments, verifiers, and broad capability coverage, while instructions often omit user-specific procedures.

    Source: §1 · p. 4

  • Method

    UI-Mate combines a closed-loop pipeline for constructing tasks and environments and producing SFT and online RL supervision with in-context demonstrations converted into adaptive subtask workflows.

    Source: §1 · p. 5

  • Results

    UI-Mate-27B reports the strongest open-weight results in its comparison and improves OSWorkerBench strict success over Qwen3.6-27B by 17.7 percentage points.

    Source: §1 · p. 5

  • Takeaways & Limitations

    Demonstrations substantially improve long-horizon execution reliability in the evaluated setting.

    Source: §1 · p. 5

    The 45-task variant-demo setting lacks a systematic aggregate result, and preliminary pilot performance was not yet stable enough for a main benchmark claim.

    Source: §10 · p. 37

Abstract

from arXiv · show

Foundation GUI agents can automate complex digital tasks, but deployment is hindered by scarce and biased training data, ambiguous prompts, and unreliable execution. Routine workflows rely on user-specific tools and tacit conventions, so unstated instructions can produce arbitrary variations across runs. We present UI-Mate, a foundation GUI agent that integrates an environment-grounded training stack with in-context demonstration learning. UI-Mate makes three contributions: A Scalable Environment-Grounded Training Stack: A closed-loop data engine automates task generation, environment construction, rollout, filtering, capability balancing, SFT, and online RL across massively parallel environments via unified task-verifier bundles. In-Context Demonstration Learning: A mechanism that transforms multimodal demonstrations into flexible subtask-level workflows, follows relevant demonstrated steps, and re-plans from the live interface. OSWorkerBench Benchmark and Insights: A benchmark of 100 long-horizon office tasks across 41 applications that supports instruction-only and demonstration-guided evaluation. Its demonstration resources separate a 33-task self-demo setting, built from successful strong-agent rollouts of the same targets, from a 45-task variant-demo setting, built from human recordings of related but non-identical tasks. Experiments show that UI-Mate-27B sets a new open-weight state of the art on general computer-use benchmarks, scoring 77.0% on OSWorld-Verified and 66.2% on WindowsAgentArena. On OSWorkerBench, it reaches 41.0% strict success and 76.9% progress, outperforming its Qwen3.6-27B base by 17.7 and 24.5 points. On the 33-task self-demo subset, one demonstration raises strict success from 17.2% to 35.4% and progress from 67.9% to 81.1%, substantially improving long-horizon reliability. Project page: https://ui-mate.github.io.

1 Introduction

UI-Mate addresses the data and interaction bottlenecks in reliable GUI automation with environment-grounded training and adaptive learning from multimodal demonstrations. Its evaluation reports strong open-weight benchmark results and substantial gains from demonstrations on long-horizon tasks.

1 Introduction

  • GUI learning is limited by sparse coverage of long-horizon, cross-app, and error-recovery workflows, while implicit user conventions can make identical requests resolve inconsistently across runs.Useful trajectories require instantiable initial states, executable actions, and verifiable outcomes; users often leave tools, file organization, templates, naming conventions, or output formats unstated.Source: §1 · p. 4
  • UI-Mate’s closed-loop pipeline constructs tasks and executable environments, collects and filters rollouts, diagnoses and rebalances capability gaps, then supports SFT and online agentic RL.Verified trajectories and task–verifier bundles connect environment construction, rollout, filtering, and training.Source: §1 · pp. 4–5
  • DemoCUA converts human or agent recordings into adaptive subtask workflows, follows relevant demonstrated steps, supplies omitted low-level actions, and replans when the live interface diverges.The live screenshot remains authoritative, so execution preserves demonstrated procedural intent without rigidly replaying an action sequence.Source: §1 · pp. 4–5
  • OSWorkerBench covers 100 long-horizon office tasks across 41 normalized applications and 10 job families, with instruction-only and demonstration-guided evaluation.Its 33-task self-demo set pairs targets with successful strong-agent rollouts of the same tasks; its 45-task variant-demo set pairs targets with human recordings of related, non-identical tasks. Paired comparisons keep the target instruction, initialized environment, interaction budget, and executable verifier fixed, changing only demonstration availability.Source: §1 · pp. 4–5
  • UI-Mate-27B reaches 77.0% on OSWorld-Verified and 66.2% on Windows Agent Arena, and scores 41.0% strict success and 76.9% progress on OSWorkerBench.On OSWorkerBench, it improves over its Qwen3.6-27B base model by 17.7 and 24.5 percentage points, respectively, and establishes the strongest open-weight results among compared systems.Source: §1 · p. 5
  • With one demonstration, OSWorkerBench self-demo strict success rises from 17.2% to 35.4%, while progress rises from 67.9% to 81.1%.On a 30-task OSWorld-Verified subset, one demonstration raises average task score from 40.3% to 65.8%.Source: §1 · p. 5

2 Overview

UI-Mate frames computer use as interaction with a live environment, scored by an executable verifier, and extends it with demonstration workflows that guide—but do not dictate—actions. Its system connects environment-grounded data and training, DemoCUA, OSWorkerBench, and a shared deployment and evaluation harness.

2.1 Task Formulation

  • Each computer-use task pairs a natural-language instruction with an environment; the agent acts from screenshots in decision turns, using a bounded window of recent interaction history.A response may execute consecutive actions without a new observation until they finish.Source: §2.1 · p. 5
  • An executable verifier inspects the environment’s final state and returns a binary success outcome, which the agent’s objective maximizes.Source: §2.1 · p. 6
  • Demonstrations become ordered subtasks with goals, completion criteria, and visually grounded action descriptions; the agent sees the relevant workflow and advances when the screen satisfies a criterion.Action descriptions are advisory, not executable coordinates, and the live observation can override the demonstration when steps are missing, misaligned, or refer to absent elements.Source: §2.1 · p. 6

2.2 UI-Mate System Overview

  • UI-Mate links environment-grounded data collection with supervised fine-tuning and reinforcement learning, using verified trajectories and executable task–verifier bundles to train general computer-use policies.Rollout outcomes and capability diagnostics feed back into task and environment construction to target weak or missing capabilities.Source: §2.2 · pp. 6–7
  • DemoCUA converts human recordings or successful agent rollouts into structured workflows, then uses the live screenshot to follow, skip, or revise demonstrated steps.Its training covers demonstrations that agree with, diverge from, or are unrelated to the current interface state.Source: §2.2 · p. 7
  • OSWorkerBench covers 100 long-horizon office tasks across 41 applications and separates 33 same-task self-demos from 45 human-recorded variant-demos for related, non-identical tasks.Paired evaluations hold the instruction, environment, interaction budget, and verifier constant; only variant-demos measure transfer across task variants, and the benchmark reports strict success and partial progress.Source: §2.2 · p. 7
  • A shared harness manages the observation–reasoning–action loop, model invocation, execution, and trajectory recording across local desktops, virtual machines, and sandbox environments.UI-Mate App supports execution, demonstration recording, and user oversight, while evaluation uses the same interaction format with executable verifiers.Source: §2.2 · p. 7

3 Data Pipeline

UI-Mate builds a closed-loop data pipeline that combines realistic task and environment generation, scalable and filtered rollouts, capability-based diagnosis, and human trajectories. For RL, it constructs executable task bundles and independently refines their verifiers so added scale expands task coverage and difficulty while preserving executability and reward reliability.

3.1 Task Instructions

  • Four complementary instruction sources balance realistic everyday computer use with systematic capability coverage.They include open-source datasets, subtasks derived from failed or stalled rollouts, tasks grounded in real working materials, and capability trees built from application specifications.Source: §3.1 · p. 7

3.2 Environment Construction

  • The pipeline creates executable environments from instructions, randomizes interface configurations without breaking task feasibility, and grounds tasks in real files when available.It retrieves and copies indexed documents, presentations, spreadsheets, images, videos, and audio, synthesizing resources when none exist; tasks already grounded in real files or websites reuse those resources directly.Source: §3.2 · p. 8

3.3.1 Rollout Infrastructure

  • A unified rollout system runs isolated, parallel tasks across Ubuntu, Windows, and macOS, while two-stage filtering rejects invalid setups and retains only trajectories evidencing every deliverable.Filtering checks task and environment validity before assessing behavior, then tracks verifiable deliverables across observations and actions to catch partial failures that a plausible final state could hide.Source: §3.3.1 · p. 8, §3.3.2 · p. 9

3.4 Capability Tree and Data Diagnosis

  • A shared capability taxonomy maps tasks and outcomes to coverage gaps, organizes capabilities hierarchically, and guides rebalancing of generation and sampling.Capabilities are initialized from application documentation and instructions, with a separate cross-application domain; tasks route through application, coarse capability, and fine-grained operation levels. Rebalancing uses target coverage, data density, rollout success, and rejection rates, treating task length separately to expose long-horizon gaps.Source: §3.4 · p. 9, §3.4.1 · p. 9, §3.4.2 · p. 9

3.5 Human Annotation

  • Human-annotated workflows complement agent rollouts with natural interaction patterns and long-tail behavior, then undergo validation, observation repair, and rollout-aligned relabeling.Repair addresses target leakage and incomplete rendering; a teacher adds policy-style reasoning and action descriptions while preserving human-annotated action parameters and rejecting mismatches or privileged leakage.Source: §3.5 · p. 10

3.6.1 Verifiable Task Generation

  • RL tasks bind instructions and environments to fixed initial states, reference completions, and executable verifiers, while jointly scaling capability coverage, interaction contexts, and difficulty.A generator constructs configurations and reference states, while a verifier judges the instruction against the resulting state through a common interface that exposes application-specific success state. Generation-time checks reject inconsistent bundles, but independent refinement is still required.Source: §3.6 · p. 10, §3.6.1 · pp. 10–11

3.6.2 Evaluator Refinement

  • Independent hard-negative and alternative-positive probes, followed by bounded rollout-informed repair, test reward alignment and gate promotion to the RL corpus.Hard negatives test false acceptance and alternative positives test false rejection; repairs to configurations, reference states, evaluators, or ambiguous instructions must pass the invariant and probes again. This coupled design aims to add capabilities, contexts, and difficulty without introducing paraphrases or noisy rewards.Source: §3.6.2 · p. 11

4 Training UI-Mate for General Computer Use

UI-Mate is trained with supervised fine-tuning on filtered offline trajectories, followed by agentic reinforcement learning in verifiable environments. Its RL pipeline combines adaptive task sampling, grouped online rollouts, verifier-based credit assignment, and asynchronous GRPO updates.

4.1 Training Recipe

  • SFT teaches executable GUI responses, screen-coordinate grounding, and screenshot- and history-conditioned action selection; RL targets task completion and improves long-horizon planning, state tracking, and recovery.SFT uses filtered offline trajectories, while RL optimizes through online interaction rather than action imitation.Source: §4.1 · p. 11

4.2 Supervised Fine-Tuning

  • SFT retains trajectories that pass step-level verification and balances coverage across applications, capability levels, and task lengths, including atomic, single-app, and cross-app tasks [1].Each decision-turn example conditions on the instruction, interaction history, and screenshot, and predicts the response tokens; the context itself carries no loss.Source: §4.2 · p. 12

4.3.1 Online GUI RL Preliminaries

  • GRPO compares binary verifier outcomes across multiple independent rollouts of the same task and policy snapshot, retaining successes and failures but excluding infrastructure-corrupted runs and groups with uniform outcomes [13] [15].Rollouts end when the model emits a terminal action or reaches the interaction budget.Source: §4.3.1 · pp. 12–13

4.3.2 Trajectory-to-Token Credit Assignment

  • Decision-turn centering corrects the extra weight longer trajectories receive, while process credit optionally reweights informative steps without changing task-level rewards.PCM uses milestone annotations to distinguish completed, failed, unattempted, and branch-skipped subgoals; it is not applied when a task group has no verified success. Token-level advantage normalization also addresses response-length bias.Source: §4.3.2 · pp. 13–14

4.3.3 Asynchronous Group-Relative Optimization

  • Asynchronous GRPO starts updates once enough complete trajectories from the same task group are buffered; IcePop and SeqClip filter stale tokens, while PPO clipping bounds retained-token contributions [23].Rollouts are never truncated, and trajectories compared within a group share a task configuration and policy snapshot.Source: §4.3.3 · p. 14

4.3.4 Adaptive Curriculum Sampling

  • Adaptive sampling preserves broad corpus coverage while directing part of the rollout budget to weak application domains identified from reliable recent results [24] [25] [26] [27] [28].It samples disjoint tasks, favors partially learned tasks with mixed outcomes, and returns unused adaptive budget to base sampling when estimates or eligible tasks are insufficient.Source: §4.3.4 · pp. 14–15

5 DemoCUA: Learning from In-Context Demonstrations

DemoCUA converts recorded executions into subtask workflows, trains agents to use them without blindly copying, and provides fuller guidance at inference while relying on live screenshots.

5.1 Demonstration Representation

  • Demonstrations are normalized, annotated, and segmented into subtasks with goals and checkable completion criteria, then used as guides rather than scripts to replay.Before each model call, the agent receives summaries of completed, current, and upcoming subtasks, but must use the live screenshot to adapt when the interface changes and mark each subtask complete before proceeding.Source: §5.1 · p. 15

5.2 Training with Demonstration

  • DemoCUA generates demonstrations through rollout, hybrid scoring, and filter-and-repair, while training workflows include alignment, mismatch, irrelevance, and omitted intermediate actions to prevent shortcut copying.The rollout stage uses demo-guided prompting on perturbed AgentNet data [1]; a VLM judge assesses semantic completion, rule-based metrics check format validity, and fixable errors are repaired. Full trajectories remain supervision targets, requiring the model to infer omitted actions from screenshots.Source: §5.2 · pp. 15–16

5.3 Inference with Demonstration

  • At inference, the agent receives the complete subtask action sequence, while long-horizon context is managed by summarizing older steps and retaining recent screenshots.Inference omits LLM-based key-action extraction, removing a model call and extraction errors. Because workflows appear at the context beginning, subtask updates invalidate the shared prefix and prevent efficient KV-cache reuse; future work would move workflows to the end.Source: §5.3 · pp. 16, 18

6 OSWorkerBench: Realistic Cross-Application Office Workflows with Multimodal Demonstrations

OSWorkerBench is a 100-task benchmark spanning 41 applications, designed to evaluate realistic long-horizon office workflows with either instructions alone or one multimodal demonstration. It includes distinct same-task and related-task demonstration settings, capability-focused task subsets, and outcome-based evaluation.

6.2 Multimodal Demonstration Guidance

  • The 33-task self-demo setting uses successful strong-agent rollouts of the same targets, while the 45-task variant-demo setting pairs targets with human recordings of related but non-identical tasks.The settings serve different purposes: self-demo evaluates same-task execution guidance, while variant-demo is designed to test procedural transfer; its systematic evaluation is left to future work.Source: §6 · pp. 18–19, §6.2 · p. 20

6.1 Benchmark Design and Construction Pipeline

  • OSWorkerBench synthesizes tasks from verified application capabilities and requires explicit cross-application dependencies, with difficulty jointly varying breadth, depth, and reasoning.Tasks are realized as natural-language instructions, deterministic initial states, and executable evaluators; CUA-Gym provides isolated, resettable sessions for reproducible execution [8].Source: §6.1 · pp. 19–20
  • Evaluators decompose functional outcomes into weighted checkpoints, and controlled terminal-state tests plus human and agent review validate scores before deployment.Checkpoints detect missing, incorrect, or extra outputs, enforce prerequisites, and prioritize primary outcomes; validation includes golden, partial, and negative cases, with mismatches triggering revisions and reruns.Source: §6.1 · p. 20

6.2 Multimodal Demonstration Guidance

  • Guided and unguided runs use identical instructions, initialized environments, interaction budgets, and evaluators, differing only in whether one demonstration is provided.The 45 variant-demo targets are deliberately challenging and long-horizon: Kimi-2.6 averages more than 100 observation-to-action decision turns per task under instruction-only evaluation.Source: §6 · p. 18, §6.2 · p. 21

6.3 Task Coverage and Characteristics

  • 99 of 100 tasks require at least two applications, and the mean is 3.26 applications per task, with a maximum of 7.The 49-task Multi-App subset is narrower: it requires substantive dynamic information transfer across at least three logical applications. Separately, 67 tasks are tagged Long-Memory for temporal dependencies; the tags are assigned independently and may overlap.Source: §6 · p. 18, §6.3 · p. 22
  • Slack appears in 65 tasks, while the most common application pairs, Google Sheets–Slack and Gmail–Slack, each occur in 23 tasks.The 41 applications form a hub-and-spoke pattern; evaluators range from 1–13 checkpoints, with a mean of 4.86 and median of 5.Source: §6.1 · p. 20, Figure 9 · p. 21, §6.3 · p. 22

6.4 Executable Evaluation

  • Kimi-2.6 trajectories have a median of 68 observation–decision turns and a mean of 88.3; 38 of 100 reach at least 100 turns.The benchmark also spans 10 job families, and evaluators measure functional outcomes rather than matching a reference trajectory; 88 use application-backend state and 12 use task-specific checks for other deliverables or conditional workflows.Source: §6 · p. 18, §6.3 · p. 22, §6.4 · p. 22
  • Strict success requires all final-state conditions, while partial progress uses a checkpoint-weighted score; capability subsets report strict success only.The Long-Memory and Multi-App labels may cover only some checkpoints, making partial scores ambiguous for those subsets.Source: §6.4 · p. 22

7 Evaluations

The evaluations assess UI-Mate across GUI benchmarks and long-horizon tasks, including how demonstrations affect execution reliability and efficiency. They also examine workflow length, partial progress and late-stage omissions, and how data realism, reward validity, and training choices shape performance.

7.1.1 Benchmarks; 7.1.2 Baselines; 7.3 DemoCUA Results

  • UI-Mate is evaluated on OSWorld-Verified and WindowsAgentArena, and on 100 OSWorkerBench tasks instruction-only; quantitative self-demo results use 33 same-task rollouts, while aggregate 45-task variant-demo results are deferred.Comparisons use identical initialized environments, task instructions, interaction budgets, and evaluators; baselines include general-purpose multimodal models and specialized computer-use agents.Source: §7.1.1 · p. 23, §7.1.2 · pp. 23–24

7.2.3 OSWorkerBench

  • 41.00% overall binary success and 76.86% progress on OSWorkerBench exceed Qwen3.6-27B by 17.67 and 24.51 percentage points, respectively.UI-Mate-27B slightly exceeds Kimi-K2.6 on both overall completion (41.00% vs. 40.67%) and progress (76.86% vs. 72.42%).Source: Table 2 · p. 24, §7.2.3 · pp. 25–26

7.2.1 OSWorld-Verified

  • 77.0% average score on OSWorld-Verified puts UI-Mate-27B above Kimi-K2.6 and Qwen3.7-Plus, while 9B reaches 66.2% and exceeds EvoCUA-32B.The 27B model's OS score matches the 9B model's 91.7%; its gains primarily occur on Office, Daily, Professional, and Workflow tasks, while Professional remains weaker than Kimi-K2.6.Source: §7.2.1 · p. 25

7.2.4 Decision-Turn Analysis on OSWorkerBench

  • 71 median decision turns for UI-Mate-27B and 68 for Kimi-K2.6 show that OSWorkerBench requires sustained interaction, while GPT's shorter trajectories partly reflect action batching.Forty UI-Mate trajectories reach at least 100 turns; some observed lengths are lower bounds because of turn limits or technical failures.Source: Figure 10 · p. 26, §7.2.4 · p. 27

7.2.2 WindowsAgentArena

  • 66.2% task success on WindowsAgentArena makes UI-Mate-27B the strongest open-weight system, ahead of Kimi-K2.6 at 63.3%.It also exceeds specialized agents and trails GPT-5.5 by 4.2 percentage points.Source: §7.2.2 · p. 25, Table 3 · p. 27

7.3.4 Results

  • 13.29 percentage points: on the 33-task OSWorkerBench self-demo subset, average normalized task score rises from 67.85% to 81.14%.Scores improve on 28 tasks, and perfect-score tasks increase from one to five; longer average trajectories (173.3 to 216.0 steps) reflect execution of additional branches and repeated operations.Source: Table 5 · p. 29, §7.3.4 · pp. 29–30
  • 25.48 percentage points: demonstrations raise the OSWorld-Subset average score from 40.27% to 65.75%, while GameDev's average rises from 76.76% to 81.15%.On OSWorld, four previously zero-score tasks are solved perfectly in all demo-conditioned runs, but four tasks decline; on GameDev, gains are largest on tasks requiring long, structured fine-grained operations.Source: §7.3.4 · pp. 29–30, Table 4 · p. 29

7.3.5 Case Study

  • A GameDev case shows how demonstrations can prevent an apparently correct action in the wrong context: the demo-conditioned agent modified the reusable character template and passed artifact evaluation.Without the demonstration, the agent changed a character instance in the game level and failed the player-bullet-reference check.Source: §7.3.5 · p. 30
  • 99.5% versus 24.5%: on the DS-2019 application, the demo-conditioned run passed all JSON checks and uploaded the correct certificate, while the no-demo run submitted incorrect values and insufficient evidence.The demonstration guided PDF viewport control, value cross-checking, and selection of the $18,000 certificate over an insufficient $12,000 certificate.Source: §7.3.5 · p. 30

7.4.1 Data

  • Real retrieved office resources are roughly 6× larger than synthesized ones, with 58.4 versus 38.5 average steps; capability-tree rebalancing expanded coverage and raised Multi-App performance 15.5 pp.The rebalancing gains were concentrated in Multi-App workflows and previously underrepresented capabilities; the evidence is correlational rather than a controlled isolation of capability tagging.Source: §7.4.1 · pp. 30–31
  • Roughly 18% of evaluators that passed the execution-invariant filter remained misaligned, making reward validity—not artifact checkability—the binding constraint for synthesizing verifiable tasks at scale.Leading failure modes were over-strict matching (40%), semantically vacuous assertions (23%), and checks on the wrong target (19%); flagged tasks enter repair rather than automatic discard.Source: §7.4.1 · p. 31

7.4.2 Model Training

  • Evaluation-time historical reasoning adds +3.43 pp for SFT and +2.27 pp for RL trained without it; adding it during SFT yields approximately +2.85 pp on long-horizon cross-application tasks.Gains are negligible on shorter, simpler tasks; qualitative analysis links them to stronger cross-application state tracking, constraint adherence, source grounding, and recovery from execution errors.Source: §7.4.2 · pp. 31–32
  • Incorporating historical reasoning into RL training accelerates policy-entropy reduction and exhibits entropy-collapse signatures, restricting exploration and lowering evaluation performance.The findings motivate integrating historical reasoning in a way that preserves policy diversity while retaining its long-horizon state-tracking benefits.Source: §7.4.2 · p. 32
  • Adaptive curriculum sampling and process credit do not consistently improve final success, but often reach comparable performance in fewer than half as many optimization updates.Curriculum sampling prioritizes weak domains, while PCM concentrates verifier-derived credit on decisions most relevant to success or failure; further gains remain constrained by data coverage and quality.Source: §7.4.2 · p. 32
  • The 9B model benefits from explicit reasoning in SFT, carefully validated evaluators in RL, and repeated exposure to the same tasks across training stages.The 27B model remains effective with a mixture of trajectories with and without reasoning; the 9B model is more vulnerable to incomplete success criteria and permissive reward proxies.Source: §7.4.2 · p. 33

7.4.3 DemoCUA

  • Demonstrations shorten GameDev trajectories from 303.6 to 253.1 steps (16.6%) while raising average score from 76.76% to 81.15%.Savings concentrate on five tasks exceeding 400 no-demo steps, where averages fall from 499.8 to 401.0; longer tasks tend to score lower, and human demonstration length correlates with model steps at +0.83.Source: §7.4.3 · p. 33, Table 6 · p. 33

8 UI-Mate App

UI-Mate App turns a configured VLM into a desktop agent that operates applications directly, records demonstrations, and supports configurable serving. It combines general computer-use runs with workflows that users can demonstrate and reuse.

8.1 Overview

  • The app separates desktop control from model choice: a model-free client sends OpenAI-format requests, while a harness selects actions and platform bridges execute them.Its frontend visualizes runs and records demonstrations, the backend validates and configures requests, and JSON-line events support live updates and mid-run commands; bridges include a native macOS helper and Linux pyautogui, with Windows support in progress.Source: §8.1 · p. 34

8.2 General CUA

  • General runs execute natural-language goals through a observe–act–settle cycle, with controls to pause, resume, stop, or message the agent and export saved trajectories.Each run is saved incrementally as JSONL containing screenshots, model inputs, and actions; the platform bridge maps image-grid coordinates to desktop coordinates and uses macOS Accessibility when available, otherwise simulating clicks.Source: §8.2 · p. 34
  • 3030 ms is the median step time, including 2108 ms for model calls and 910 ms elsewhere; halving non-model costs reduces it by only about 15%.Prefill and decoding account for roughly 76% and 15% of the model call, respectively; the app exposes controls targeting those costs.Source: §8.2 · pp. 34–35, Figure 14 · p. 35

8.3 Demo CUA

  • A user can record a workflow once, convert it into editable named subtasks, and attach it to a task so the agent follows recorded steps while tracking progress.A native macOS recorder captures video and raw input; Demo2Workflow describes actions from frame pairs and recorded facts, while manual edits remain separate from model output.Source: §8.3 · p. 35

8.4 Deployment

  • Users configure an OpenAI-compatible endpoint by base URL and model name, and the signed, notarized macOS installer requests Screen Recording and Accessibility permissions.One installation can switch among serving arrangements without rebuilding.Source: §8.4 · pp. 35–36
  • On-device changes reduce typical step time from about 45 s to 10–20 s.The setup uses a roughly 7 GB six-bit 9B model, 720p captures instead of 1080p, and prompt-prefix and cached-image-feature reuse; the paper also reports that quantization and speculative decoding reduce a step from 3–5 s to 2–3 s on the same machine without application changes.Source: §8.4 · pp. 35–36, Table 7 · p. 36

9 Related Work

UI-Mate is positioned as advancing both environment-grounded training and demonstration-guided execution, addressing limitations in instruction-only protocols and workflow replay. OSWorkerBench provides a controlled way to evaluate instruction-only and demonstration-guided completion of long-horizon office tasks.

Computer Use Agents

  • UI-Mate’s data engine budgets generation by a hierarchical capability tree and promotes audited task–verifier bundles to RL, while its agent selectively follows or overrides demonstrated steps using the live screen.Its demonstrations are distilled into subtask-level workflows to address prompt ambiguity and execution unreliability that instruction-only training alone does not directly address.Source: §9 · p. 36
  • Computer-use agents have shifted from modular vision–language-model scaffolds [39] [40] [41] to native end-to-end models [42] [43] [45] [46] [47], with training increasingly incorporating environment interaction [25] [27] [51].The related work names UI-TARS, OpenCUA [1], ScaleCUA [2], Mobile-Agent-v3 [48], UI-Venus-1.5 [49], and MAI-UI/Qwen-UI-Agent [5] [50] among native agents.Source: §9 · p. 36

Computer Use Benchmarks

  • GUI-agent evaluation spans static grounding, offline web navigation, and interactive verified environments for web, mobile, and desktop, with OSWorld [10] and WindowsAgentArena [11] as desktop anchors.Recent suites extend realism and horizon: OSWorld 2.0 [58] evaluates hour-scale workflows, WeaveBench [59] audits interleaved GUI–CLI execution, and ScienceBoard [60] targets scientific workflows.Source: §9 · p. 36
  • OSWorkerBench covers 100 long-horizon office tasks across 41 applications and separates same-task strong-agent self-demos from human-recorded related-task variant-demos for controlled comparison with instruction-only execution.It includes requirement-level Long-Memory and Multi-App subsets and validated checkpoint evaluators; conditions hold the target instruction, environment, budget, and verifier constant.Source: §9 · p. 36

Demonstration-Guided Computer Use

  • Prior approaches range from brittle fixed-sequence RPA to reusable skills in CUA-Skill [64] and MMSkills, while ShowUI-Aloha [9] guides execution with traces that mainly transfer across instances sharing workflow logic.ShowUI-Aloha converts a human screen recording into a semantic action trace for planning and execution.Source: §9 · pp. 36–37
  • In contrast to workflow replay, UI-Mate distills human recordings or agent rollouts into subtask-level procedural intent, selectively following, skipping, or adapting steps grounded in the live interface.This enables demonstration-guided procedural generalization.Source: §9 · p. 37

10 Discussion and Future Work

The discussion proposes verifier-grounded progress credit to improve optimization efficiency, while emphasizing that existing DemoCUA results establish same-task execution guidance—not transfer across task variants. Variant-demo transfer remains unstable, motivating work on extracting transferable structure, scaling demonstration acquisition, and retrieval from the live state.

Verifier-Grounded Progress Credit

  • Verifier-grounded progress credit could reward milestone completion and recovery, penalize regression, and retain signal in homogeneous-outcome groups while preserving executable outcomes as authoritative.Deterministic checks would score verifiable milestones, learned judges would handle uncertain states, and neutral spans could remain as context while being masked from actor loss.Source: §10 · p. 37

Beyond Self-Demo: the Variant-Demo Setting

  • All reported DemoCUA quantitative results use same-task demonstrations and evaluation targets, so they establish execution guidance rather than procedural transfer across variants.On OSWorker-Subset, the self-demo setting pairs 33 targets with successful same-task rollouts from a stronger GUI agent.Source: §10 · p. 37
  • A ten-target pilot found variant-demo performance net positive when demonstrated segments were replicated to match target entities, but results were too unstable for a main benchmark claim.The 45-task collection pairs targets with human demonstrations from related but non-identical source tasks, and no systematic aggregate result is reported. Future directions include learning transferable structure and its limits, acquiring demonstrations from books, documentation and instructional videos, and retrieving demonstrations against the live state.Source: §10 · p. 37

11 Author Contributions · Appendix · A Source of Task Instructions

The appendix acknowledges contributors and describes four complementary sources of task instructions: converted open-source tasks, rollout-derived atomic capabilities, real-content workflows, and capability-tree-driven generation.

11 Author Contributions

  • The authors thank contributors, including those not listed in the paper, and state that authors within each role are listed alphabetically by last name.Source: §11 · p. 37

Core Contributors

  • Core contributors listed are Hanxuan Li, Honglin Li, Xiyun Li, Zaitang Li, Leowei Liang, Qi Gao, Xiangwu Guo, Shengchao Hu, Zilong Huang, Xin Luo, Haozhe Ma, Jiayi Mao, Zhoujie Pan, Can Qin, Mengcheng Lan, Weixian Lei, Tianyuan Qu, Weiqi Wang, Wenkai Wang, Yonglin Wang, Yuxin Wang, Chenxu Wu, and Yingchen Yu.Source: Core Contributors · p. 37

A Source of Task Instructions

  • Instructions adapted from AgentNet [1] and ScaleCUA [12] are cleaned and converted for target platforms, excluding unavailable applications, ambiguous targets, and unverifiable outcomes.Application references may be replaced with platform equivalents while preserving operational intent.Source: A Source of Task Instructions · p. 42
  • Failed or stalled rollouts are decomposed into targeted atomic subtasks that evaluate difficult operations, expand operating-system interaction coverage, and isolate failure modes.This approach follows [6].Source: A Source of Task Instructions · p. 42
  • Tasks grounded in real documents, spreadsheets, presentations, and websites use concrete content and realistic cross-application workflows to support more authentic, evaluable long-horizon tasks.The real content comes from InfiniteWeb [7], and operational patterns from application manuals guide sequences such as transferring website information into a spreadsheet and updating a document or presentation.Source: A Source of Task Instructions · p. 42
  • Capability trees built from application specifications identify underrepresented fine-grained operations and generate targeted instructions to fill remaining capability gaps.Source: A Source of Task Instructions · p. 42

B Details of DemoCUA Data Generation

DemoCUA evaluates rollouts with a hybrid score covering task completion, subtask completion, and demo-following adherence. Its Filter & Repair phase recovers training data from fixable errors using deterministic rules and LLM-generated boundary reports.

Score

  • Rollouts are scored on Whole-Task Completion (S1), Per-Subtask Completion (S2), and Demo-following adherence (S3–S4), combining VLM judgments with rule-based metrics.S1 uses a Skeptical-Auditor VLM prompt over a visual chain; S2 evaluates subtasks from boundary frames and self-reports; S3–S4 measure completion rate and execution-order alignment via LCS.Source: B Details of DemoCUA Data Generation · p. 43, Table 8 · p. 43

Filter & Repair

  • Filter & Repair preserves trajectories with fixable errors by applying deterministic normalization rules and using an LLM to synthesize missing subtask-boundary reports.Deterministic repairs collapse duplicate reports, close unclosed trajectories, re-derive subtask IDs, and strip leading WAIT or empty actions.Source: B Details of DemoCUA Data Generation · p. 43, Table 9 · p. 43

C Details of GameDev Tasks

The section presents curated GameDev task case studies and describes how their artifact-check rubrics are weighted and evaluated.

C Details of GameDev Tasks

  • Table 10 presents ten curated task case studies with verbatim instructions and equal-weight artifact-check rubrics.The rubric entries enumerate the artifact checks for each case.Source: Table 10 · p. 44
  • Table 11 presents ten curated task case studies that are independently evaluated, with equal weight assigned to each case’s listed rubric checks.Source: Table 11 · p. 45

D GameDev Performance of Kimi-K2.6

Kimi K2.6 already performs strongly on GameDev without demonstrations, while demonstrations raise its average score and improve most tasks not already at ceiling. Demonstration-led GUI workflows take more steps but achieve higher scores, suggesting they help satisfy fine-grained requirements.

D GameDev Performance of Kimi-K2.6

  • Demonstrations raise Kimi K2.6’s mean score from 83.46 to 88.46, a gain of 5.00 points, over five runs.Without demonstrations, it fully solves three tasks.Source: D GameDev Performance of Kimi-K2.6 · p. 46, Table 12 · p. 46
  • Demonstrations improve six of seven tasks below ceiling, with gains of 30.00 points on QGIS, 13.68 on godot-04, and 9.41 on godot-05.The paper interprets these gains as better coverage of fine-grained artifact requirements.Source: D GameDev Performance of Kimi-K2.6 · p. 46
  • With demonstrations primarily following GUI workflows, Kimi averages 257.5 steps but achieves higher scores; without them, CLI bypasses lengthy GUI sequences and reduces steps.Tasks requiring longer trajectories are generally associated with lower scores.Source: D GameDev Performance of Kimi-K2.6 · p. 46, Table 13 · p. 46

E Details of the OSWorld-Subset under DemoCUA setting · F Details of the OSWorker-Subset under DemoCUA setting

On the 30-task OSWorld subset, self-demonstrations substantially improve average performance and shorten model trajectories. On the 33-task OSWorkerBench self-demo subset, demonstrations improve task scores while sometimes prompting longer, more complete workflows.

E Details of the OSWorld-Subset under DemoCUA setting

F Details of the OSWorker-Subset under DemoCUA setting

  • On the 33-task OSWorkerBench self-demo subset, demonstrations improve average task score by 13.29 percentage points but increase trajectory length.The longer trajectories reflect greater task completeness on repetitive or branching tasks, where instruction-only runs may prematurely declare completion.Source: Table 16 · p. 50
  • The OSWorkerBench self-demo condition supplies a successful stronger-agent rollout of the same target task, using screenshots and action types without concrete pixel coordinates.No-Demo and Demo scores and steps are averaged over three runs under otherwise identical conditions.Source: Table 16 · p. 50
  • For OSWorkerBench Task 18, a demonstration guides the model to process both qualifying Hot leads, raising average score from 52.97% to 97.92%.Without a demonstration, it processed only one of two Hot leads; with one, it repeated the reference workflow, including Salesforce updates, follow-up tasks, and calendar events, while average steps rose from 106.7 to 165.7.Source: F Details of the OSWorker-Subset under… · p. 50

Related papers

Loading 2608.15930v1…