PaperlayerView exact revision on arXiv

Source-linked AI summary

iOSWorld: A Benchmark for Personally Intelligent Phone Agents

Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, Ruslan Salakhutdinov

arXiv:2606.09764v1cs.LGcs.CL

TL;DR

Existing phone-agent benchmarks do not test reasoning over persistent, cross-app personal data. iOSWorld introduces a native iOS benchmark for this capability and finds that the best vision+XML configuration reaches 52% overall but only 37% on multi-app tasks.

  • Problem

    Existing phone-agent benchmarks lack persistent user identities, cross-app continuity, and evaluation of reasoning over personal data.

  • Method

    iOSWorld builds a native iOS simulator benchmark with one user identity, connected data across 26 apps, and 133 tasks spanning single-app, multi-app, and personalization categories.

  • Results

    52% overall is achieved by the best vision+XML configuration, with 93% on single-app, 54% on memory, and 37% on multi-app tasks.

  • Takeaways & Limitations

    Multi-app tasks remain the hardest category, while privileged vision+XML access benefits stronger frontier models but not smaller models to the same extent.

  • Takeaways & Limitations

    iOSWorld is a controlled simulator benchmark, so its results should not be interpreted as readiness for deployment on real devices with real user data.

Abstract

from arXiv · show

A useful phone agent needs to be personally intelligent. It should reason over a user's identity, history, and preferences as they exist on the device, not just follow isolated instructions in an impersonal sandbox. Existing mobile agent benchmarks lack this kind of personalization. We introduce iOSWorld, the first interactive native iOS simulator benchmark built around a persistent user identity spanning 26 newly built iOS apps. These apps contain connected data such as transactions, messages, travel records, social relationships, and financial activity. iOSWorld includes 133 tasks across three increasingly difficult categories. Single-app tasks (27) test one app, multi-app tasks (60) span 2 to 8 apps, and memory and personalization tasks (46) require agents to infer patterns from personal data. We evaluate frontier and open-source computer-use models in both vision-only and privileged vision+XML settings. The best configuration reaches 52\% overall but only 37\% on multi-app tasks. Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input. We release iOSWorld as an open-source benchmark with all apps, seeded data, tasks, rubrics, and evaluation code.

1 Introduction

iOSWorld addresses the gap in phone-agent benchmarks by testing personally intelligent assistance over persistent identity, history, preferences, and connected data on native iOS. It provides 26 interconnected apps and 133 tasks, with evaluation spanning task complexity, personalization, model families, and interface access.

  • Motivation: Useful phone agents must reason over a user’s identity, history, and preferences across accumulated transactions, messages, social connections, and financial records.Existing benchmarks treat tasks as isolated and ignore this persistent personal context.
  • Benchmark design: iOSWorld is the first interactive native iOS simulator benchmark with one user identity spanning 26 purpose-built applications containing connected personal data.The shared persona is Jordan Avery, with cross-app links such as contacts, orders, bank charges, receipts, flights, hotels, and confirmation emails.
  • Task coverage: 133 tasks cover 27 single-app, 60 multi-app, and 46 memory and personalization tasks.These categories increase the scope from one application to cross-application workflows and inference over personal data.
  • Evaluation: 52% overall is achieved by the best configuration, including 82% single-app, 54% memory, and 37% multi-app performance.Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input.

2 Related Work

Prior GUI-agent benchmarks have focused mainly on web, desktop, and Android environments. iOSWorld addresses the remaining gap in dynamic iOS evaluation and personalized reasoning over user data distributed across apps.

  • Web and Desktop GUI Agents: Web and desktop benchmarks have evaluated GUI agents in emulated tasks, real websites, and live OS or desktop conditions.Examples include MiniWoB, WebShop, WebArena, Mind2Web, WebVoyager, and OSWorld.
  • Android Mobile Agents: Android benchmarks support reinforcement learning, human-demonstration evaluation, dynamic simulator tasks, long-horizon interaction, and tool use.The cited systems include AndroidEnv, Android-in-the-Wild, AndroidWorld, and MobileWorld.
  • Personalization Gap: No prior mobile benchmark seeds applications with a user identity or evaluates reasoning over extensive personal data distributed across apps.This gap concerns personalized, cross-application reasoning rather than isolated mobile interaction.
  • iOS Evaluation Gap: Dynamic iOS evaluation remains underrepresented because iOS differs from Android in its UI framework, navigation patterns, and accessibility infrastructure.These platform differences motivate a benchmark designed specifically for native iOS interaction.

3 iOSWorld

iOSWorld is a partially observable, interactive iOS simulator benchmark centered on a persistent user identity across 26 connected apps. It evaluates agents on 133 tasks spanning single-app interaction, multi-app information transfer, and memory-based personalization under vision-only and vision+XML observation settings.

  • 3 iOSWorld: iOSWorld models agent interaction as a POMDP with simulator states, actions, observations, and deterministic state transitions.At each step, the agent receives a partial observation and produces an action that transitions the simulator to the next state.
  • 3 iOSWorld: The benchmark evaluates vision-only screenshots against vision+XML observations that add cleaned accessibility data for visible interactive elements.Vision-only agents estimate UI coordinates from pixels, whereas XML exposes element types, labels, values, normalized coordinates, and identifiers.
  • 3 iOSWorld: The evaluation uses GPT-5.4-Mini to judge complete trajectories with binary pass/fail outcomes, with human validation reaching κ=0.77 at task level and 89% accuracy.The judge reviews screenshots, actions, and the final answer; a per-step variant was more lenient without improving discrimination.
  • 3 iOSWorld: All 26 applications share Jordan Avery’s identity and connected personal data, including contacts, purchases, messages, receipts, travel records, and workplace relationships.Cross-app links include QuickBite orders generating MyBank charges and Mail receipts, while contacts recur across messaging, payments, email, networking, and collaboration apps.
  • 3 iOSWorld: 133 tasks are divided into 27 single-app tasks, 60 multi-app tasks spanning two to eight applications, and memory tasks requiring exploration of multiple personal data sources.Memory tasks involve 4.4 apps per task on average; the final set was reduced from an initial pool of 175 tasks.

4 Experiments

Experiments evaluate frontier and open-weights computer-use models under vision-only and privileged vision+XML settings. Privileged XML substantially helps stronger models but can hurt smaller ones, while multi-app and memory tasks remain constrained by interface friction and step budgets.

  • Models: The evaluation covers five frontier models—Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, GPT-5.4 Mini, and Gemini 3 Flash—and open-weights Qwen3.5 35B-A3B.The frontier models use dedicated computer-use APIs, while Qwen3.5 is served via vLLM with the official mobile-agent cookbook.
  • Vision and XML: +25.6 pp lifts Opus from 26% to 52% overall with vision+XML, compared with +18.0 pp for Sonnet and +19.5 pp for GPT-5.4.Privileged vision+XML access helps the stronger frontier models.
  • Task performance: 93% is Sonnet’s vision+XML result on single-app tasks, while Opus leads memory at 54% and multi-app at 37%.Multi-app tasks remain the hardest category.
  • Vision and XML: XML helps by exposing small or off-screen labels and reducing ordinary iOS navigation friction; ∼70% of Opus’s 26 vision-only failures rescued by XML involve home-screen or app-switching issues.The benchmark treats XML as privileged access rather than merely better text input.
  • Vision and XML: 26% to 16% is GPT-5.4 Mini’s overall change with XML, while Qwen3.5 falls from 7% to 0% on multi-app tasks.GPT-5.4 Mini has 22 of 35 vision-only successes become failures under XML, and added context is estimated at ∼3,100 tokens per step.
  • Failure analysis: 51% of frontier vision+XML failures exhaust the 50-step budget, followed by 26% that give up and 23% that stop prematurely.Budget exhaustion is most common on multi-app tasks at 55% and memory tasks at 52%, whereas premature stopping is most common on single-app tasks at 48%.
  • Scaling and validation: 17% to 54% is Opus’s memory-task improvement from step 30 to step 50, while the trajectory judge reaches κ=0.77 task-level accuracy against human annotators.Single-app tasks saturate by step 20, multi-app tasks improve through step 40, and the judge’s rubric-criteria agreement is κ=0.69 with Pearson r=0.85.

5 Conclusion

iOSWorld benchmarks personally intelligent phone agents across 26 native iOS apps with persistent user identity, revealing strong single-app performance but substantially weaker multi-app and memory-task results. The conclusion identifies recovery, grounding, and history-aware planning as key areas for improvement.

  • Benchmark scope: iOSWorld spans 26 native iOS apps through a persistent user identity, benchmarking single-app, multi-app, and memory tasks.The benchmark evaluates agents in an interactive native iOS environment rather than an impersonal sandbox.
  • Performance: 93% is reached on single-app tasks by the best vision+XML configuration, versus 37% on multi-app and 54% on memory tasks.The results show a marked performance gap as tasks require broader cross-app reasoning or memory-based personalization.
  • Performance: 11% overall is achieved by Qwen3.5 35B-A3B, trailing the benchmark’s best configuration.This result is reported for the smaller open-source model evaluated in iOSWorld.
  • Failure modes and future work: 51% of frontier-model failures exhaust the 50-step budget, indicating that many agents run out of room before completing tasks.The conclusion points to stronger loop recovery, better action and visual grounding, and planning aware of the user’s history as needed improvements.

Ethics Statement · A LLM Disclosure

iOSWorld uses entirely synthetic, deterministic data and a fictional persona to study personalization without real-user privacy risks, while acknowledging risks, access limitations, and deployment boundaries. The authors disclose human-directed use of language models for prose, figures, and quantitative analysis.

  • Ethics Statement: All benchmark data is synthetic, uses deterministic seeds, and represents the fictional Jordan Avery persona without real users, accounts, services, or external databases.This design supports personalization and memory research while avoiding privacy risks from real user data.
  • Ethics Statement: Autonomous phone agents pose dual-use risks, including surveillance, unauthorized transactions, social engineering, data exfiltration, and harmful errors.Examples include messaging the wrong contact, unintended purchases, and leaking personal information across apps.
  • Ethics Statement: macOS and Xcode requirements limit iOSWorld reproducibility to researchers with Apple hardware, although released code, seed data, evaluation scripts, and AWS Mac runners broaden access.The AWS deployment uses EC2-managed Mac instances and returns the same evaluation bundle.
  • Ethics Statement: The release uses one fictional persona for verifiable personalization ground truth, while releasing the seeding pipeline, schema, task generator, and rubric framework for future comparable personas.Multi-persona evaluation remains future work.
  • Ethics Statement: Phone agents may improve accessibility for users with visual, motor, or cognitive impairments, but iOSWorld results do not establish readiness for real devices or user data.The benchmark measures progress in a controlled simulator.
  • A LLM Disclosure: Human authors made all drafting and structural decisions, while Claude Code polished prose, checked grammar, verified consistency, and generated figures from human-provided sketches under human direction.The disclosure describes human-in-the-loop review and author-controlled figure layout and content.
  • A LLM Disclosure: A multimodal LLM coding agent was used for high-level quantitative analysis with human-provided direction.The passage identifies quantitative analysis as one stage involving language models.

B Application Details and Dataset Statistics

The benchmark spans 26 applications, with task references concentrated in QuickChat, Notes, CloudDocs, and Mail. Under Opus 4.6 vision+XML, per-app pass rates range from 12% on Cinephile to 65% on CalTrack, with QuickChat remaining challenging despite its frequent use.

  • Application coverage: 26 applications are included, with QuickChat (44 task references), Notes (41), CloudDocs (35), and Mail (29) most frequently involved.These four apps account for the highest task-reference counts in Table 4.
  • Per-app difficulty: 12% is Opus 4.6’s pass rate on Cinephile, the hardest app, across 8 references.The evaluation uses the vision+XML configuration.
  • Per-app difficulty: 65% is Opus 4.6’s pass rate on CalTrack, the easiest app, across 17 references.This is the highest reported per-app pass rate in the supplied passage.
  • Per-app difficulty: 20% is QuickChat’s pass rate across 44 references, reflecting difficulty with precise thread navigation across many tasks.QuickChat is both the most frequently referenced app and one of the more challenging apps.
  • Per-app difficulty: 59% is the pass rate for both Mail and MyBank, across 29 and 22 references respectively.They are among the strongest-performing applications in the per-app results.

C Rubric-Based Evaluation Details

iOSWorld evaluates tasks with independently verifiable rubric criteria, revealing partial progress beyond binary success. The benchmark therefore uses a trajectory-level judge because per-step evaluation is more lenient and less aligned with human annotations.

  • Rubric construction: 1,123 rubric items decompose 133 tasks into independently verifiable criteria, averaging 8.4 items per task.Tasks contain 4–13 rubric items; multi-app tasks average 9.4, reflecting their intermediate coordination steps.
  • Rubric results: Under vision+XML, average rubric scores range from 29% for Qwen3.5 to 81% for Opus, exposing substantial partial progress on failed tasks.Frontier agents satisfy a majority of criteria even when they fail the overall task.
  • Rubric results: The rubric perfect rate tracks binary pass rate within 0.8–2.3 percentage points, supporting consistency between holistic and per-criterion evaluation.A perfect rubric score requires satisfying every criterion.
  • Judge selection: Per-step evaluation raises average rubric scores to 73–87% but is less human-aligned than trajectory judging, with κ=0.51–0.61 versus 0.77.Its mean rubric score is 0.83 versus 0.70 for humans, and it produces 188 versus 79 false-positive criteria.

D Failure Analysis Details

The failure analysis assigns every failed trajectory to one of three mutually exclusive modes based on step count and final rubric score. Examples illustrate budget exhaustion, premature stopping, and giving up, while Qwen3.5 failures frequently involve stuck-action loops.

  • Methodology: Every failed trajectory is classified as a premature stop, gave up, or budget exhausted using its step count and final rubric score.Runs stopping before 50 steps with rubric score ≥0.67 are premature stops; other early terminations are gave up, while runs using all 50 steps are budget exhausted.
  • Examples: GPT-5.4 prematurely stops after 8 steps at the final “Request” button despite correctly reporting fare and ETA, scoring 0.80.The booking is not confirmed.
  • Open-source baseline (Qwen3.5 35B-A3B): Qwen3.5 35B-A3B has a 10.5% pass rate, with 60/119 vision+XML failures flagged as stuck-action loops.The ≥3-consecutive-identical-actions heuristic is described as reliable because Qwen’s loops are extreme.

E MCP Tools Ablation … I Example Trajectories

The paper analyzes structured MCP tools, additional result patterns, benchmark construction, evaluation prompts, and representative trajectories. Structured tools improve the open-source baseline, while examples expose both successful completion and budget-limited failures.

  • E MCP Tools Ablation: A DineSpot reservation takes 17 steps with structured MCP tools and scores 1.0, versus 50 steps and 0.25 with cookbook mobile use.The structured interface reaches a confirmed Harborline Seafood booking, while cookbook use gets stuck on the filter sheet.
  • E MCP Tools Ablation: Structured tools raise Qwen3.5’s pass rate from 12.8% to 24.8% and mean rubric score from 0.33 to 0.683, though it trails frontier models.The ablation holds the model, task set, judge, and 50-step budget fixed while varying only the action interface.
  • F Additional Results Figures: Opus gains +25.6 pp, Sonnet +18.0 pp, GPT-5.4 +19.5 pp, and Gemini +0.8 pp from vision+XML, while smaller models do not benefit.The comparison is vision-only versus vision+XML accuracy per model.
  • F Additional Results Figures: Single-app tasks saturate by step 20, multi-app tasks scale through step 40, and memory tasks show varied scaling, with Opus climbing steeply past step 30.The figure reports cumulative pass rate versus step budget overall and by task category.
  • G App and Task Construction: Apps use SwiftUI, deterministic seeded data, accessibility identifiers, specified workflows and data models, iterative refinement, and manual developer verification.Claude Code created or adapted the apps from structured prompts specifying constraints and seed quantities.
  • G App and Task Construction: Tasks were generated from app source code and seed data, normalized and rewritten in first-person voice, converted into rubrics, then executed by human annotators.The three-stage pipeline combines Claude Code generation, Python processing, and human simulator execution.
  • H Prompts: The system prompt specifies iOS touch interactions, app discovery, navigation, typing, scrolling, task completion, and optional accessibility-tree use in vision+XML mode.The accessibility tree provides element labels, values, identifiers, and coordinates, while screenshots take precedence when the two disagree.
  • I Example Trajectories: Example trajectories include a 13-step successful Notes task, a 26-step successful DineSpot-to-TeamChat task, and budget-limited failures scoring 0.56 and 0.50.The failed SkyTrip workflow completed 3/4 subtasks before messaging, while the memory workflow found birthday information and bought a gift before the dinner reservation.

J Human Agreement

Human validation shows that the trajectory-level judge substantially agrees with annotators, with balanced criterion-level errors and strong per-annotator consistency. Robustness is weaker for some alternate judges and semantic criteria, while agreement generalizes moderately to substantially beyond Opus trajectories.

  • Overall agreement: 89% task-level judgments agree with humans (κ=0.77, F1=0.86), while rubric-criterion accuracy reaches 86% (κ=0.69, F1=0.90).The lower rubric-level κ reflects Cohen’s κ sensitivity to marginal distributions when 67% of criteria are satisfied.
  • Overall agreement: 148 criterion-level disagreements split into 79 false positives and 69 false negatives, indicating no strong directional bias.False positives indicate the judge was too lenient, whereas false negatives indicate it was too strict.
  • Per-annotator agreement: Task-level κ ranges from 0.64 to 0.92 across annotators, while rubric-level κ ranges from 0.67 to 0.72.All annotators show substantial agreement, and rubric judgments are more tightly clustered than holistic task judgments.
  • Cross-judge robustness: The larger GPT-5.4 is the weakest alternate judge at κ=0.51, with 1 false positive and 27 false negatives against humans.Pairwise agreement is [0.74, 0.90] for other judges but [0.53, 0.74] for GPT-5.4.
  • Error patterns: 0% error occurs on observable taps and swipes, whereas semantic and report criteria have 13 to 16% error.At the task level, the judge over-accepts single-app tasks, over-rejects multi-app tasks, and is balanced on memory tasks.
  • Generalization beyond Opus: Agreement on additional Gemini 3 Flash and GPT-5.4 Mini trajectories is κ=0.49 and 0.60, respectively.The lower Gemini κ partly reflects the smaller sample and absence of the original multi-annotator calibration.

K iOS-Specific Interaction Patterns

The section quantifies two iOS-specific interaction problems: coordinate-grounding retries and under-use of edge-swipe back navigation. Vision-only traces show substantial model variation in missed-target proxies, while frontier models rarely use the iOS back gesture.

  • Coordinate-grounding miss rate: 12.3% is GPT-5.4’s vision-only coordinate-grounding miss rate, compared with 10.1% for Opus, 10.2% for Sonnet, 10.3% for GPT-5.4 Mini, and 5.3% for Gemini.The proxy counts taps within 60 px on consecutive steps, computed across all 133 vision-only trajectories per model.
  • Edge-swipe back-navigation under-use: 1.1% of 12,255 frontier-model swipes are left-edge rightward swipes, the iOS gesture required for back navigation without a hardware back button.GPT-5.4 reaches 2.7% and GPT-5.4 Mini 2.1% in vision-only mode, while Claude and Gemini reach at most 1.6% in vision-only and 1.2% in vision+XML.
  • Coordinate-grounding miss rate: ≈0% is the coordinate-grounding miss rate across models when taps use XCUITest accessibility IDs.This comparison is reported in Table 17 against vision-only coordinate-grounding misses.
Loading 2606.09764v1…