PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effects of Data Scale on UI Control Agents

Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, Oriana Riva

arXiv:2406.03679v6cs.AIcs.LG

TL;DR

UI control agents still perform relatively poorly without fine-tuning, raising questions about whether collecting demonstrations and fine-tuning alone can support robust real-world control. The paper introduces AndroidControl and evaluates scaling across low- and high-level tasks in and out of domain. Fine-tuning scales favorably in-domain, but out-of-domain high-level performance requires substantially more data and may need additional approaches.

  • Problem

    The paper asks whether fine-tuning alone can provide robust UI control as data grows, especially across low- versus high-level tasks and in- versus out-of-domain settings.

  • Method

    The authors collect AndroidControl and use it to evaluate fine-tuned models across task complexity and domain settings while comparing zero-shot and few-shot baselines.

  • Results

    Fine-tuning scales favorably in-domain, but out-of-domain high-level tasks require one or two orders of magnitude more data for robust performance.

  • Takeaways & Limitations

    Fine-tuning may be a viable, though potentially expensive, route to high in-domain performance, whereas additional approaches may benefit robust out-of-domain high-level control.

  • Takeaways & Limitations

    The study fine-tunes only PaLM-2S, so absolute performance values and the consistency of relative findings across model families remain uncertain.

Abstract

from arXiv · show

Autonomous agents that control computer interfaces to accomplish human tasks are emerging. Leveraging LLMs to power such agents has been of special interest, but unless fine-tuned on human-collected task demonstrations, performance is still relatively low. In this work we study whether fine-tuning alone is a viable approach for building real-world computer control agents. In particularly, we investigate how performance measured on both high and low-level tasks in domain and out of domain scales as more training data is collected. To this end we collect and release a new dataset, AndroidControl, consisting of 15,283 demonstrations of everyday tasks with Android apps. Compared to existing datasets, each AndroidControl task instance includes both high and low-level human-generated instructions, allowing us to explore the level of task complexity an agent can handle. Moreover, AndroidControl is the most diverse computer control dataset to date, including 14,548 unique tasks over 833 Android apps, thus allowing us to conduct in-depth analysis of the model performance in and out of the domain of the training data. Using the dataset, we find that when tested in domain fine-tuned models outperform zero and few-shot baselines and scale in such a way that robust performance might feasibly be obtained simply by collecting more data. Out of domain, performance scales significantly more slowly and suggests that in particular for high-level tasks, fine-tuning on more data alone may be insufficient for achieving robust out-of-domain performance.

1 Introduction

UI control agents use LLMs to perceive device states and execute interface actions, but pretrained models achieve relatively low real-world success without fine-tuning. This work introduces AndroidControl to measure how fine-tuning scales across task complexity and domain boundaries.

  • Motivation: 12%–46%: pretrained LLM agents report relatively low real-world success rates, compared with up to 80% for fine-tuned agents on similar website tasks.The reported rates span desktop and mobile applications, while the 80% result concerns websites and tasks similar to training data.
  • Motivation: Fine-tuning data is time-consuming and expensive to collect, motivating measurements of performance scaling in-domain and out-of-domain.The study examines unseen tasks and applications to assess whether fine-tuning alone can support real-world deployment.
  • Task complexity: High-level tasks require decomposing goals into atomic actions, whereas low-level actions such as clicking and typing are more constrained.This distinction motivates evaluating whether fine-tuning transfers across different levels of task complexity.
  • Dataset: AndroidControl contains 15,283 human demonstrations, 14,548 unique tasks, and 833 Android apps, with both high- and low-level instructions for every task.Its size and diversity enable multiple in-domain and out-of-domain test splits.
  • Findings: Fine-tuning scales favorably in-domain, but out-of-domain high-level tasks require one or two orders of magnitude more data for robust performance.The paper compares fine-tuning with zero-shot and few-shot baselines across low- and high-level tasks and domain settings.

2 Related work

Prior UI control work spans behavioral cloning, reinforcement learning, prompting, and fine-tuning, using varied task descriptions and UI-state representations. AndroidControl extends this landscape with human-generated high- and low-level instructions and broad Android-app diversity while testing fine-tuning’s domain generalization.

  • Datasets: Existing UI control datasets vary in platform, size, task diversity, episode length, UI-state representation, and whether instructions are high- or low-level.The comparison distinguishes screenshots from UI trees and single-step grounding tasks from multi-step tasks.
  • Datasets: AndroidControl uniquely provides both human-generated high- and low-level instructions for every task, enabling richer supervision and evaluation across task complexity.Other datasets with both annotation types use synthetic low-level instructions or restrict them to clicks.
  • Evaluation environments: Interactive testing environments provide reward signals for online evaluation but generally cover no more than 20 applications or websites, except simplified synthetic MiniWob.This contrasts with the broader application coverage of demonstration datasets.
  • UI control agents: Current UI agents use pretrained LLMs or multimodal models with zero-shot, few-shot, end-to-end fine-tuning, or capability-specific fine-tuning.Examples include SeeAct for webpage element selection, WebGPT for browser use, WebAgent for web control, and Synapse for trajectory-based prompting.
  • Domain generalization: Scaling data and models can improve domain generalization, while fine-tuning downstream models has also been associated with reduced robustness to distribution shifts.This work empirically studies how fine-tuning data size affects in-domain and out-of-domain performance.

3 The ANDROIDCONTROL dataset

ANDROIDCONTROL is designed to study how fine-tuning scales across task complexity and domain shifts. It combines diverse Android demonstrations with high- and low-level instructions, structured collection procedures, and multiple evaluation splits.

  • ANDROIDCONTROL targets both fine-tuning data scaling and the task complexity that fine-tuned UI control models can handle.
  • Data collection: Crowdworkers instantiated generic feature descriptions into tasks across apps from 40 categories.
  • Data collection: The dataset spans 833 Android apps, including popular, high-trend, less-popular, and regional applications.Annotators could choose any app, increasing variation in interfaces and accessibility-tree quality.
  • Data collection: Annotators provided both high-level task descriptions and low-level action instructions for each demonstration.The high-level description specifies the overall goal, while the low-level instructions accompany interaction traces.
  • Dataset statistics: Tasks contain 1–13 steps between the 5th and 95th percentiles, while high-level instructions contain 8–34 words and low-level instructions 3–14 words.
  • Dataset splits: The evaluation uses in-domain, app-unseen, task-unseen, and category-unseen test splits to measure domain and out-of-domain scaling.The splits are constructed alongside train and validation data, and some out-of-domain splits may overlap.

4 Experiments and results

Experiments compare zero-shot, few-shot, and LoRA-tuned agents across task complexity, data scale, and domain transfer. Fine-tuning performs strongly in-domain, while out-of-domain performance improves more slowly, especially for high-level tasks.

  • Agent implementation: The Android agent observes accessibility-tree screen representations, maintains previous-action history, and predicts typed actions with required arguments or termination.Inputs include current-screen UI elements and self-contained descriptions of prior actions; outputs can target elements, enter text, open apps, or scroll.
  • Experimental setup: The experiments vary training or prompt examples across 5, 10, 100, 1k, 10k, and all 13,604 episodes, while evaluating zero-shot and few-shot baselines.Performance is measured primarily with step-wise accuracy, counting correct actions and arguments, including equivalent actions under a relaxed metric.
  • In-domain performance: In-domain, the best fine-tuned model reaches 71.5% on high-level instructions and 86.6% on low-level instructions.Low-level instructions surpass non-fine-tuned models earlier, whereas high-level instructions require more training data, including 1k episodes.
  • Baseline comparison: The strongest zero-shot baselines reach 56.7% for low-level instructions and 42.1% for high-level instructions, while few-shot performance is mostly inferior to zero-shot.These results vary with prompts, base-model strength, and the inclusion of high-level reasoning in M3A.
  • Effect of scale in-domain: In-domain step accuracy follows near-linear trends with the log of training data, with R2 coefficients above 0.95.The analysis extrapolates 500K episodes for 95% low-level step accuracy and 1M for 95% high-level step accuracy; five-step high-level completion at 95% is projected to require 2M episodes.
  • Effect of scale out-of-domain: Out-of-domain step accuracy grows more slowly, with 95% projected at 10M episodes for low-level and 60M for high-level instructions.For five-step high-level tasks, 150M episodes are projected for 95% episode completion, suggesting fine-tuning alone may be insufficient for robust out-of-domain performance.

5 Limitations

The paper identifies limitations concerning model coverage, offline evaluation, dataset scope, and inference-cost analysis.

  • Model coverage: Only PaLM-2S is fine-tuned, so absolute performance may differ across model families even if relative findings are expected to remain consistent.The limitation concerns generalization of the reported scaling results beyond the single fine-tuned model.
  • Evaluation: Offline evaluation does not reward alternative task routes or corrective actions.This can constrain how agent performance is represented relative to interactive execution.
  • Dataset scope: AndroidControl’s app categories cover important use cases but incompletely represent all tasks users may ask agents to perform.The dataset therefore has a defined application and task-scope boundary.
  • Inference costs: Inference costs are not studied, although fine-tuned PaLM-2S prediction is cheaper than prediction with larger models such as PaLM-2L or GPT-4.The paper explicitly leaves cost analysis outside its focus.

6 Conclusion

The conclusion presents AndroidControl as a dataset for studying scaling across task complexity and domain. Fine-tuning appears viable for high in-domain performance but scales much less favorably out of domain, especially for high-level tasks, while deployment also carries societal and security risks.

  • Contribution: AndroidControl is a large, diverse dataset structured to evaluate low- and high-level tasks in and out of domain as training data scales.The conclusion frames the dataset as the basis for scaling analysis of LoRA fine-tuned models.
  • In-domain scaling: In-domain, the analysis projects 1M episodes for 95% low-level accuracy and 2M episodes for 95% completion of five-step high-level tasks.These projections suggest fine-tuning may achieve high in-domain performance, though potentially at substantial cost.
  • Out-of-domain scaling: Out of domain, projections rise to 10M episodes for low-level performance and 150M episodes for five-step high-level completion.This one- to two-orders-of-magnitude increase suggests fine-tuning may not scale well out of domain or suffice for robust high-level performance.
  • Ethical considerations: UI control agents could expand access for visually impaired users and automate everyday tasks, but deployment raises societal, security, and privacy risks.Potential harms include leaking private information, unwanted side effects, overriding anti-fraud mechanisms, and manipulating applications.
  • Experimental setting: The experiments use PaLM 2 through Google’s Vertex AI PaLM API under Google’s prohibited-use policy.This identifies the model-access and policy setting for the reported experiments.

B.1 Data collection

ANDROIDCONTROL combines human-recorded Android interactions with structured episode metadata and accessibility-tree information. Its analysis also examines how UI-element density varies across apps and test splits.

  • Dataset collection and release: 15,283 demonstrations were collected from human interactions with Android apps and released publicly.The dataset is stored as TFRecord files with episode identifiers, goals, screenshots, accessibility trees, dimensions, and actions.
  • UI representation: Each accessibility-tree node represents a UI element described through type, textual attributes, location, size, status, and interaction properties.These metadata fields include class name, text-related attributes, screen bounds, boolean status flags, and capabilities such as clickability or scrollability.
  • Test-split composition: The IDD split has comparable UI-element density to training, whereas the three out-of-domain splits have significantly fewer elements per screen.This difference is attributed to the long-tailed distribution of UI elements across apps and the relatively small number of apps in each split.
  • Test-split composition: After removing the top 5% of apps by UI-element count, the remaining apps average 180.9 elements per screen.The dataset-wide distribution has a long tail driven by a few apps with very many UI elements.

C UI control agent implementation

The agent represents the current Android screen as structured UI-element information and predicts actions in JSON. It also uses action history to condition decisions across multi-step tasks.

  • Screen representation: The screen representation lists each UI element’s text, position, and status derived from the Android accessibility tree.The current screen and application are part of the agent input.
  • Action prediction: The agent predicts actions using predefined JSON templates, including coordinates for element-targeting actions such as clicks.For a click, the target is represented by the center coordinates of the selected element.
  • Action prediction: Input text is combined with its preceding focus click, and the corresponding low-level instructions are concatenated.This modification reduces the need to represent the focus-setting click as a separate predicted action.
  • Action history: The agent receives the current screen description together with a self-contained history of previous actions and their outcomes.History descriptions avoid external references to earlier screen representations.
  • Screen representation: For screen descriptions, element position and shape are encoded by center and size, while semantic meaning is carried by text and state fields.A switch without textual attributes receives the label “Switch” from its class name and uses checked status to represent state.

D.1 Data processing and training details

The experiments transform ANDROIDCONTROL into high- and low-level SeqIO tasks, define prompts and action formats, and use exact or device-consistent matching for step accuracy.

  • Data processing: Two SeqIO tasks, HL and LL, are generated for high-level and low-level instructions.Processing inserts terminal actions, recovers target identifiers where possible, and drops episodes only when all steps are discarded.
  • Data processing: A terminate action is artificially appended to every episode so the agent can predict successful or infeasible task completion.The action is absent from the original dataset and is added for training and evaluation.
  • Prompting: Each prompt includes the instruction, action space, expected output format, current screen and application, and previous-action history.The ER prompt specifically supplies previous actions, active app, screen dimensions, and a structured screen description.
  • Prompting: The ER prompt uses a simple UI-element screen description and is applied to few-shot and fine-tuning experiments.Unlike the other prompts described, ER does not encourage the LLM to reason explicitly.
  • Evaluation: Step accuracy accepts exact action matches or device-consistent equivalents, including coordinates inside the target element’s bounding box.Navigate-back and open-app also receive specified equivalences to corresponding Android interactions.

E.1 Confusion matrices for action predictions

Action-type prediction is strongest for explicit, common actions and weaker for rare or implicit actions. Low-level instructions generally make both action and argument prediction easier than high-level instructions.

  • Action-type prediction: Click and open_app actions are predicted with accuracy above 88%.The result comes from the LoRA-tuned PaLM 2S model evaluated across all four test splits.
  • Instruction level: High-level instructions reduce action-type accuracy because the model must decompose goals into lower-level actions.Terminate, navigate_back, and wait are especially challenging because they are often implicit in user instructions.
  • Action-argument prediction: Long_press has the lowest argument accuracy, likely because it is scarce in the dataset and uncommon in mobile apps.App-name detection and scroll-direction prediction perform well, while explicit low-level commands make argument inference easier.
  • Training configuration: Mixed HL-and-LL training is equal or better than training separate models for each instruction level.The authors associate this result with increased opportunity for transfer learning, while noting that conflicting tasks could undermine the benefit.
  • Episode length: Fine-tuned PaLM 2S falls from 21.3% to 7.6% episode accuracy when task length increases from five to six steps.Step accuracy remains relatively flat with episode length, but fully successful episodes become less likely as tasks lengthen.

E.5 Step-accuracy performance vs. application types

Performance differs substantially between first-party and third-party apps, especially for high-level instructions. The comparison reports step and episode accuracy across episode lengths and evaluates fine-tuned and zero-shot models.

  • Evaluation setup: The figure compares step accuracy and episode accuracy against episode length on the full test split.It includes PaLM-2S-FT and PaLM-2L-ZS using the ER prompt.
  • High-level instructions: 82.5% step accuracy on first-party apps falls to 58.7% on third-party apps for LT-all on high-level instructions.This is the largest highlighted example of the application-type gap for the fine-tuned models.
  • Low-level instructions: The first-party/third-party gap is smaller for low-level instructions, and few-shot performance is sometimes higher on third-party apps.The passage links the stronger high-level gap to the importance of accurate screen representations for tasks requiring stronger reasoning.
  • Evaluation setup: Table 13 breaks down step accuracy across Google first-party and third-party apps for zero-shot, few-shot, and LoRA-tuned models.Tests use Random-500; LoRA-tuned models use rank=4 except LT-all-r64.
Loading 2406.03679v6…