DigiData: Training and Evaluating General-Purpose Mobile Control Agents

• Existing datasets lack the depth, diversity, or scale needed for general-purpose agents, while benchmarks often rely on inaccurate single-step matching or narrow sets of easily verifiable goals.Source: §1 · p. 1
• DigiData is a large-scale, high-quality, diverse, multi-modal dataset whose goals come from comprehensive app-feature enumeration, enabling deep exploration and greater goal diversity.Source: §1 · pp. 1–2, Figure 1 · p. 2
• DigiData-Bench curates diverse goals and supports precise human-assisted dynamic evaluation and AI-assisted dynamic evaluation with LLM judges.Source: §1 · p. 2
• Training with DigiData improves agents on DigiData-Bench and existing benchmarks, while the paper studies data and parameter scaling and evaluates LLM judges for measuring success.Source: §1 · p. 2

Mobile Control Datasets

• Mobile-control datasets pair natural-language goals with human demonstrations and observations, while Android in the Wild and AndroidControl are the only large-scale datasets described.Earlier datasets include MoTIF, PixelHelp, UGIF, and AMEX; observation formats include screenshots, UI trees, and LLM chain of thought.Source: §2 · p. 2
• AndroidWorld automates success-rate evaluation in restricted settings, while concurrent work proposes text-matching-based verification for mobile-agent benchmarking.AndroidWorld builds on existing Android emulator interfaces.Source: §2 · p. 2

Computer Use

• Computer-use benchmarks cover web browsers and operating systems, including MiniWob, WebVoyager, WebArena, VisualWebArena, Mind2Web, and OSWorld.The paper positions mobile control as part of the broader field of agents that autonomously operate computers.Source: §2 · p. 2
• Computer-use and mobile-control methods improve agents through prompting, supervised fine-tuning, and reinforcement learning.The paper notes that mobile control has distinct challenges and draws lessons from other forms of computer use.Source: §2 · pp. 2–3

3.1 Data Collection

• For each app, DigiData’s collection pipeline curates goals through feature exploration, records demonstrations on physical or emulated Android devices, and verifies trajectories with AI and human review.Annotators explore screens and menus, and some goals form sequences to cover deeper features while limiting individual goal complexity. The LLM judge has a low false-accept rate; trajectories it marks unsuccessful go to human review, and 5.3% of raw trajectories are filtered out.Source: §3.1 · pp. 3–4

3.2 Dataset Features

• DigiData contains 8,275 goals across 26 apps and 152,000 trajectories; goal success was 84% for sampled AitW trajectories, versus 94.6% for DigiData before verification and 100% after.The AitW comparison used 20,000 randomly sampled trajectories assessed by human annotators.Source: §3.2 · p. 4
• DigiData adds UI trees and LLM-generated Chain-of-Thought annotations to screenshots, including screen descriptions, action descriptions, rationales, and expected UI changes.The annotations are generated with Llama 4 from parsed UI trees and screenshots; the paper describes these modalities at DigiData’s scale with permissive licensing as unavailable in public datasets.Source: §3.2 · p. 4
• Human demonstrators need approximately 9.2 steps per DigiData goal, roughly 50% more than existing datasets; DigiData averages 318 goals per app versus AitW’s 85.The paper attributes the greater goal complexity to curation that targets deep app features.Source: §3.2 · p. 4
• DigiData’s goal instructions show higher pairwise cosine-distance diversity than AitW and AndroidControl, while its goals are balanced across apps.Figure 3b compares the distributions of pairwise distances; higher distances indicate greater vector diversity.Source: §3.2 · p. 5, Figure 3 · p. 5

4 The DigiData-Bench Benchmark

• DigiData-Bench covers 309 goals across 37 Android apps and 8 categories, including seen, familiar, and novel apps to assess generalization.Its goals span varied use cases and were selected to avoid direct overlap with DigiData’s goals.Source: §4 · pp. 5–6

4.1 Human-assisted Dynamic Evaluation

• Human-assisted dynamic evaluation uses goal-specific instructions to initialize app state, monitor agent actions, and judge whether each trajectory succeeds.Operators stop unsafe actions; agents were stopped in 0.5% of trajectories on average. The protocol terminates trajectories after an agent’s termination action, an unsafe or invalid action, or enough steps.Source: §4.1 · p. 6, Figure 4 · p. 6

4.2 AI-assisted Dynamic Evaluation

• DigiData-Bench-Auto automates inference and evaluation on a tethered Android device using LLM judges to classify trajectory success from screenshots and UI trees.It selects roughly half the benchmark’s goals, excluding those needing manual initialization, login, or location information, or posing dangerous side-effect risks; apps are automatically closed, reset, and reopened before evaluation.Source: §4.2 · pp. 6–7

4.3 Offline Evaluation

• Offline step-accuracy measures per-screenshot action matching against human demonstrations, providing a proxy metric that complements dynamic success evaluation.The test set uses the DigiData demonstration protocol and the same goals as DigiData-Bench.Source: §4.3 · p. 7

5.1 Evaluation Setup

• The evaluation fine-tunes PLM with DigiData, AitW, AndroidControl, and Cauldron, and compares against zero-shot GPT4o and QWen2.5VL baselines.Cauldron is included to help prevent catastrophic forgetting of general visual understanding.Source: §5.1 · pp. 7–8

5.2 Results

• 47% task success rate is achieved by the model trained with chain-of-thought data, which outperforms the public-data baseline and GPT4o and Qwen2.5VL on DigiData-Bench.Ours 8B COT has the highest DigiData-Bench success rate under both human evaluation and the LLM judge.Source: Table 2 · p. 8, §5.2 · p. 8
• Human evaluation has a standard deviation of ±0.4% across three runs, while human experts achieve 90.1% task success, leaving a substantial agent-human gap.Source: §5.2 · p. 8
• Step accuracy does not reliably rank agents by human-evaluated performance, and the 3B and 8B models have similar step accuracy but significantly different DigiData-Bench success rates.Task-success rate is needed to measure agent capability accurately.Source: §5.2 · p. 8, Table 2 · p. 8, Figure 5 · p. 9
• More DigiData training data improves performance on seen and familiar apps but does not significantly improve performance on novel apps.The authors identify limited generalization as an SFT limitation and suggest reinforcement learning as a future direction.Source: Figure 5 · p. 9, §5.2 · p. 9
• LLM judges correlate with human agent rankings and outperform step accuracy; fine-tuned Llama 4 approaches GPT4o judge performance and slightly exceeds it in true negative rate.True negative rate matters for flagging unfit trajectories during DigiData data collection.Source: Table 3 · p. 9, §5.2 · p. 9

6 Conclusion

• DigiData is a large-scale, high-quality, diverse dataset built by comprehensively exploring mobile app functionality, alongside DigiData-Bench for agent evaluation.The benchmark uses human- and AI-assisted evaluation protocols.Source: §6 · p. 10
• Agents trained with DigiData achieve strong performance on DigiData-Bench according to both accuracy and success metrics.The paper evaluates agents trained on the dataset using the benchmark.Source: §6 · p. 10
• Existing and fine-tuned LLM judges show promising performance with potential to further automate mobile control agent evaluation.The paper evaluates LLM judges of mobile control agents.Source: §6 · p. 10
• The authors provide training and test data, detailed setup and execution instructions, and code for running dynamic evaluations to support reproducibility.They state that the work is fully reproducible.Source: §6 · p. 10

Appendix

• The model is trained using the PLM (Cho et al., 2025) code base, with training recipes and prompts provided in the appendix.Source: §6 · p. 10

6 Conclusion

• DigiData covers 26 mobile apps, and its generalizability to computer-based interfaces remains unexplored; human- or AI-assisted evaluation may also introduce biases or limitations.The paper notes that the dataset may not match the diversity of real-world applications.Source: §6 · p. 10
• Future work should expand app coverage, explore transfer learning to novel apps, and adapt the methods to computer-based interfaces.Source: §6 · p. 10

A Trajectory Visualization

• Figure 6 shows tapping and swiping at each step across two DigiData-Bench goals, illustrating the model’s completion of complex, multi-step tasks.The authors report that the model trained on DigiData can navigate a mobile device and complete tasks requiring multiple steps.Source: Figure 6 · p. 13, A Trajectory Visualization · p. 13

Diversity of Apps and Goals

• DigiData-Bench covers eight app categories: Communications, Food, Management, Media, Navigation, Restricted Search, Shopping, and Misfit.Figure 7 presents detailed goal counts per app and the app-category mapping.Source: B Benchmark details · p. 13, Figure 7 · p. 14

Step accuracy computation

• Step accuracy marks an action correct when its type matches the target and its parameters fall within a specified threshold; for taps, equivalence uses screen distance.At each step, the agent receives the goal and a demonstration screenshot and predicts an action.Source: B Benchmark details · p. 13

Goal similarity categories

• Goals are Similar when the same app contains an almost identically executed goal, including template variants or rephrasings; Unseen goals have no similar in-app goal and test generalization.Figure 8(b) presents the goal similarity distribution.Source: B Benchmark details · p. 13, Figure 8 · p. 14

Goal intuitiveness categories

• Intuitive goals are self-explanatory to an average user; non-intuitive goals may require in-app exploration or external resources, and intuitiveness does not determine path length.Figure 8(c) presents the goal intuitiveness distribution.Source: B Benchmark details · pp. 13–14, Figure 8 · p. 14

C Limitations of Step Accuracy Evaluation

• Step accuracy typically treats only one path as correct, potentially overlooking valid alternative routes to the same outcome.Trajectory-level task success offers a more comprehensive assessment by accounting for different routes that complete the task.Source: C Limitations of Step Accuracy… · p. 14
• Figure 9 shows direct search, menu scrolling, and an alphabetic index all achieving the goal of changing a shipping location to the Dominican Republic.Source: C Limitations of Step Accuracy… · pp. 14–15
• Evaluating trajectory-level task success captures route diversity that step accuracy can miss.Source: C Limitations of Step Accuracy… · p. 14

• Task complexity was proxied by the average number of steps users needed to complete each task, despite step count’s imperfect correlation with complexity.Source: D Preliminary Insights into the… · p. 15
• Figure 10 plots each model’s task success rate by step range and shows the total number of tasks in each range as a histogram.Source: D Preliminary Insights into the… · p. 15
• Task success rates tend to decrease as step counts increase, suggesting model performance diminishes with rising task complexity.Source: D Preliminary Insights into the… · p. 15

E.1 Supervised Finetuning

• The supervised-finetuning mixture combines Android-in-the-wild, Android Control, DigiData, and Cauldron, with incomplete trajectories and Cauldron QA pairs containing interleaving images removed.Android-in-the-wild and Android Control prompts include three-step action histories; DigiData uses a similar history.Source: §E.1 · p. 15
• Training uses 128 NVIDIA A100s, 60k iterations, a 2e-5 learning rate, and whole-model fine-tuning; the best step-accuracy model is selected for human evaluation.Evaluation uses 32 NVIDIA A100s and greedy decoding; each image has at most four 448x448 tiles.Source: §E.1 · p. 15

E.2 Baselines.

• GPT4o follows Android World Rawles et al. (2024a) and SeeAct Zheng et al. (2024), adding simplified XML for coordinate errors; QWen2.5VL 7B tools were adapted to match the action space.Both foundation models receive task-specific prompting for device-control tasks.Source: §E.2 · pp. 15–16

E.3 LLM Judge

• The LLM-judge test set covers 327 goals and includes 229 successful and 327 failed trajectories for human evaluation.One successful and one failed agent trajectory were randomly selected per goal when available; some challenging goals had no successful model-generated trajectory.Source: §E.3 · p. 16, F Evaluating LLM judges of device… · p. 16

F Evaluating LLM judges of device control

• The modular judge first summarizes each action and its before-and-after screenshots and XML, then reasons over the step descriptions and trajectory endpoints; fine-tuning used 7,057 failed and 2,034 successful trajectories.The fine-tuned judge learns to produce reasoning and binary human judgments from the goal and step summaries.Source: F Evaluating LLM judges of device… · p. 16
• LLM judges achieve promising but imperfect trajectory-judging performance; GPT4o generally performs strongest, while fine-tuning open-weight models on a few thousand samples can reach similar precision.The evaluation uses Accuracy, Precision, Recall, Negative Predictive Value (NPV), and True Negative Rate (TNR).Source: F Evaluating LLM judges of device… · p. 16

G Dataset Format

• Table 5 describes the dataset format, and Table 6 describes the action space.Source: G Dataset Format · p. 16

H License of Assets

• PerceptionLM uses the FAIR Noncommercial Research License; Llama 3 and Llama 4 use their respective community licenses, and GPT4o is accessed through Azure under its terms of use.Source: H License of Assets · pp. 16–17
• Android-in-the-wild and Android Control use Apache 2.0, while Cauldron uses CC-BY-4.0.Source: H License of Assets · p. 17

I Prompts

• The default prompt asks the agent to use the conversational goal and current screen to return one action, selected from tap, swipe, type, navigation, or status options.Source: I Prompts · p. 18
• The CoT prompt structures each step as an observation, thought, brief action description, expected UI change, and precise action, using the last three step histories.Source: I Prompts · p. 18
• The step-summary prompt uses paired screenshots, the action, and supplementary UI-element lists to produce concise, goal-relevant summaries of each screen and their changes for the judge.Source: I Prompts · p. 19
• The judge evaluates whether the goal was achieved and remains achieved from state transitions, requiring a final ending action and a reasoned Yes/No judgment.The prompt asks the judge to assess conditions for the goal using relevant transitions rather than treating the ending action alone as proof of completion.Source: I Prompts · p. 20
• The judge’s criteria require full goal completion with specified parameters and comparable save actions, while allowing maintained success reached earlier, default-enabled states, or externally blocked efforts.Source: I Prompts · p. 20
• The GPT4o baseline prompt guides stepwise screen and history analysis, while the QWen2.5VL baseline uses a mobile-control function-call interface with structured actions.Source: I Prompts · pp. 24–25
• The prompt treats extracted UI-element lists as incomplete and directs the model to prioritize screenshot images when available.It also asks for key elements and their locations in each screenshot, without using the labels “screen A” or “screen B.”Source: I Prompts · p. 26
• The action description should be one short sentence, inferring taps or swipe direction from the action, visual cues, and screenshot context.The response should not mention the green-circle or green-arrow annotations.Source: I Prompts · p. 26
• The prompt asks for goal-related changes between the current and next screenshots, including subtle altered or new elements, and a concise rationale for the action.The rationale should explain how the action helps achieve the goal.Source: I Prompts · p. 26
• The answer must use five labeled sections—screen A, screen B, action description, changes, and reason—with each section no more than 100 words.Prompt 8 is identified as the prompt for generating chain-of-thought data.Source: I Prompts · p. 26

J Data Verification

• Over 100 contractors in two cohorts trained on alternative datasets, a DigiData golden set, and a rubric-based qualification test; two verifiers reviewed each trajectory, requiring agreement for goal success.Verifiers also checked whether generated goals were valid.Source: J Data Verification · p. 27

K Goal Generation Protocol

• Research assistants systematically explored app pages and interactive elements to create goals covering app features, phrased as natural user intentions with varied language rather than mechanical templates.The protocol treats goal quality as crucial because the goal distribution shapes the dataset distribution.Source: K Goal Generation Protocol · p. 27
• Assistants created functionally distinct goal templates with placeholders, then instantiated them with specific arguments to produce concrete goals.For example, templates could specify an article to search for or a language to change to.Source: K Goal Generation Protocol · p. 27
• Generated goals passed through researcher feedback, annotator checks for clarity and feasibility, and verification judges’ assessments of whether each goal was meaningful and achievable.Problematic or invalid goals were revised, excluded, or removed from the final dataset.Source: K Goal Generation Protocol · pp. 27–28

L Runbooks

• Annotators repeatedly actuated a selected model and confirmed non-dangerous actions after the screen settled, stopping when it completed the goal, looped, reached 25 steps, or proposed dangerous or inaccurate actions.The runbook required uploading the trajectory and completing a survey after the stopping condition.Source: L Runbooks · p. 29
• The evaluation survey classified failures as missing final step, impossible, overshot, dangerous, looping, max steps, or inaccurate.A loop meant repeating the same action or action set at least three times in a row.Source: L Runbooks · p. 29
• A dangerous action was one that wrote to app state, such as posting a message, making a purchase, or changing an account; preceding steps that did not change state were not dangerous.For a review task, the runbook considered pressing the post button—not composing the review—the dangerous action.Source: L Runbooks · p. 30
• The data-collection runbook specified submissions for direct or on-screen information retrieval and browse or side-effect tasks, and prohibited personal information, sign-ins, or access to personal accounts.Participants were instructed to use provided test information and submit failure when a task could not be completed.Source: L Runbooks · p. 31