PaperlayerView exact revision on arXiv

Source-linked AI summary

iPrefBench: A User-Centric Benchmark for Personalized Mobile Use Agents

Yuting Ning, Ruijia Cheng, Sushobhan Ghosh, Peitong Duan, HungShi Lin, Abhishek Sundararajan, Shreyash Pandey, Huan Sun, Jeffrey Nichols, Eldon Schoop

arXiv:2610.34735v1cs.CL

TL;DR

Real user requests often omit information that agents must either infer from activity history or obtain by asking, while current evaluations do not always distinguish those sources. iPrefBench evaluates both capabilities in underspecified mobile tasks; agents show a gap between execution and personalization and struggle to ask when necessary.

  • Problem

    Underspecified requests require agents to distinguish details inferable from user history from details that only the user can provide, a distinction many existing evaluations do not capture.

    Source: §1 · pp. 1–2

  • Method

    iPrefBench evaluates 130 underspecified mobile tasks whose omitted details are designated history-inferable or ask-only, measuring personalized task completion and user-agent interaction.

    Source: §1 · p. 2, §5 · p. 9

  • Results

    91.6% action completion but 47.7% task success for Claude Opus 5; agents also struggle to request ask-only information.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The benchmark provides a testbed for evaluating whether agents know what to infer from history and when to ask the user.

    Source: §1 · p. 1, §5 · p. 9

Abstract

from arXiv · show

An agent that truly knows and acts on behalf of its user has been a long-standing promise of AI assistants. Computer-use agents bring this promise within reach by capably executing fully specified tasks across apps. However, real-world user requests are often underspecified. To act correctly for the particular user, an agent must infer some missing information from the user's activity history and ask for other information that only the user can provide, without making unsupported assumptions or burdening the user with unnecessary questions. We present iPrefBench, a benchmark for evaluating the infer and ask capabilities of personal mobile-use agents in an executable iOS environment with an interactive simulated user. Grounded in a coherent persona with 3,051 past activity events, iPrefBench comprises 130 underspecified user tasks across 26 apps. Each omitted user-dependent requirement is, by construction, either history-inferable or ask-only. iPrefBench evaluates personalized mobile agents on both whether agents complete tasks correctly for the user (i.e., personalized task completion) and whether they seek user input when necessary (i.e., user-agent interaction). Evaluation of 7 frontier agents reveals a substantial gap between execution and personalization. Agents achieve high action completion scores (up to 91.6%) but low full task success (up to 47.7%) because personalization lags behind. They especially struggle with ask-only information, failing to request enough necessary information. Our findings highlight a challenge for personalized agents in knowing what to infer and when to ask. We hope that iPrefBench can serve as a valuable testbed for building agents that know their user and work with them.

1 Introduction

Real user requests often omit details, requiring agents to infer some from user history and ask about others that history cannot resolve. iPrefBench evaluates both abilities, revealing that strong execution does not guarantee personalization and that agents often under-ask.

1 Introduction

  • Existing evaluations often omit the joint challenge: benchmarks provide fully specified tasks, user history without a user to ask, or simulated users without history.Some benchmarks provide both sources but do not distinguish which missing details are inferable versus clarification-only.Source: §1 · pp. 1–2
  • iPrefBench evaluates 130 underspecified tasks in an executable iOS environment, with omitted details designated as history-inferable or ask-only.Tasks also test instruction overrides, changing preferences, and unavailable preferred options.Source: §1 · p. 2
  • The benchmark assesses personalized task completion and user-agent interaction, including whether agents elicit ask-only information without redundant or irrelevant requests.Task rubrics distinguish action completion from user-dependent decisions; interaction is measured using ask-only recall and precision.Source: §1 · p. 2
  • Claude Opus 5 achieves 91.6% action completion but only 47.7% task success because personalization lags behind.Agents also struggle particularly with ask-only information.Source: §1 · p. 2
  • Ask-only recall is low across agents, peaking at 44.3%; explicit clarification raises questions and recall but lowers precision, exposing a tradeoff.Reported precision ranges from 60% to 77%.Source: §1 · p. 3

2 Related Work

Prior benchmarks address computer-use execution, preference inference, or clarification, but often lack the combination needed to assess whether an agent should infer or ask. iPrefBench explicitly distinguishes information sources.

2 Related Work

  • Computer-use benchmarks span desktop, mobile, and web environments but typically use fully specified tasks without prior user knowledge.MobileWorld adds underspecified tasks and an ask_user action, but has no user history.Source: §2 · p. 3
  • Personalized-history benchmarks study preference inference but do not support interactive clarification.iOSWorld lacks both history exposure and user interaction; VitaBench 2.0 and KnowU-Bench do not explicitly distinguish history-recoverable details from user-only information.Source: §2 · p. 3
  • Interactive clarification benchmarks generally lack user histories, so they cannot assess whether questions are necessary given what an agent already knows.Some evaluate missed and unnecessary questions, but without histories against which necessity can be judged.Source: §2 · p. 3
  • iPrefBench distinguishes history-inferable from ask-only information, enabling evaluation of personalized completion and necessity-aware user-agent interaction.The paper provides a detailed benchmark comparison in Table A.1.Source: §2 · p. 3

3 iPrefBench

iPrefBench separates omitted user-dependent details by whether activity history supports them or the user must supply them. Its environment, task design, and evaluations are built to test both personalized outcomes and appropriate clarification.

3.1 Benchmark Overview

  • Each unresolved detail is labeled inferable when consistent activity-history evidence supports its intended value, or ask-only when that value is unavailable from history.The distinction reflects stable preferences and habits versus specifics of the current situation.Source: §3.1 · p. 3

3.2 Environment

  • The executable iOS environment contains 26 mock apps with interconnected data for one persona and adds ask_user for messages to a simulated user.A chronological activity history is the agent’s only prior knowledge; the hidden profile grounds simulation and evaluation.Source: §3.2 · p. 4

3.3 Tasks

  • Tasks may require combining evidence across events or transferring preferences across apps; perturbations test instruction overrides, evolving preferences, and unavailable options.For unavailable preferred options, agents should surface the unavailability and follow the user’s alternative preference or stop if substitution is unacceptable.Source: §3.3 · p. 4
  • Task authoring combines candidate generation with two rounds of human review to verify resolution sources, feasibility, and rubric correctness.Ask-only values are chosen to differ from app defaults when possible, reducing the chance of success through silent defaults or guesses.Source: §3.3 · pp. 4–5

3.4 Evaluation

  • Personalized task completion uses task-specific action and personalization criteria, with success requiring every criterion to pass.An LLM judge scores criteria using trajectories, profile, hidden context, and rubrics; category scores are macro-averaged over tasks.Source: §3.4 · p. 5
  • User-agent interaction measures ask-only Recall and Precision, penalizing both failure to elicit required details and requests for inferable, irrelevant, or repeated information.The evaluation decomposes agent messages into individual information requests and matches them to ask-only requirements.Source: §3.4 · p. 5

3.5 Benchmark Statistics

  • The benchmark contains 130 tasks grounded in 3,051 activity events across 26 apps; 26.9% require cross-app preference transfer.Tasks average 2.09 action-completion and 2.87 personalization criteria, including 1.98 inferable and 0.89 ask-only criteria.Source: §3.5 · p. 5
  • Table 1 reports success rate, action completion, personalization by resolution source, interaction Recall and Precision, ask_user calls, and steps.All scores are percentages except the average counts of ask_user calls and actions.Source: Table 1 · p. 6

4 Experiments

Across seven frontier agents, mobile task execution exceeds personalization, with ask-only information the main difficulty. Error analysis and follow-up studies examine why agents fail, how clarification and history exposure affect performance, and the reliability of the evaluation.

4.2 Main Results

  • 77.5%–91.6% action-completion scores coexist with 26.2%–47.7% full task success because personalization lags behind.Claude Opus 5 completes 91.6% of required actions but only 69.4% of user-dependent decisions, succeeding on 47.7% of tasks.Source: §4.2 · p. 6
  • Ask-only personalization trails inferable information by up to 46 points, while agents elicit at most 44.3% of ask-only requirements and have 60.4%–77.1% interaction precision.Tasks contain 0.89 ask-only requirements on average, but agents make only 0.08–0.35 ask_user calls per task.Source: §4.2 · pp. 6–7
  • On instruction- and preference-perturbation subsets, personalization falls below overall averages; environment perturbations elicit 2.3× as many questions and improve ask-only personalization for every agent.Figure 2 plots subset-minus-overall scores for action completion and personalization, plus average ask_user actions per task.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Error Analysis

  • Personalization failures include silent guesses or defaults for ask-only details, and missed or wrong inferences for history-inferable details.The error analysis identifies two challenges: recognizing unresolved user-dependent choices and deciding whether to infer or ask once a choice is recognized.Source: §4.3 · p. 7, Figure 3 · p. 8

4.4 Additional Analysis and Discussion

  • A pre-execution clarification phase improves personalization and interaction recall but lowers precision, increasing coverage at the cost of unnecessary requests.Simply encouraging clarification does not help agents distinguish necessary requests from unnecessary ones.Source: §4.4 · p. 8
  • More user history improves inferable personalization and interaction precision, but ask-only personalization barely changes and ask-only recall can fall.For Qwen3.8-27B, the top-50 relevant events raise inferable personalization from 35% to 63%; recall falls from 33.6 to 21.2.Source: Figure 4 · p. 8, §4.4 · p. 9

4.5 Validation of User Simulator and LLM Judges

  • The user simulator’s responses were grounded in profile or task context and answered exactly what was asked in 96.6% of inspected pairs.The inspection covered question–answer pairs from four representative agents.Source: §4.5 · p. 9
  • LLM judges agreed with human labels on 95.7% of rubric criteria, 97.0% of task-level success, and 95.2% / 93.9% of recall / precision judgments.Agreement was κ = 0.89, 0.94, and 0.90 / 0.82, respectively; results were consistent with Gemini 3.7 Flash as an alternative judge.Source: §4.5 · p. 9

5 Conclusion

iPrefBench evaluates whether mobile agents infer user preferences from activity history and ask for clarification when needed. Across seven frontier agents, execution outpaces personalization, especially for ask-only information; encouraging clarification increases questions but also over-asking.

5 Conclusion

  • iPrefBench evaluates personalized agents on 130 underspecified mobile-use tasks, controlling whether omitted details are inferable or ask-only.It measures action completion, personalization, and whether agents involve users only when necessary.Source: §5 · p. 9

A Comparison with Existing Benchmarks

The comparison distinguishes iPrefBench by whether benchmarks expose inferable and ask-only information and evaluate whether questions are necessary. Its environment includes 26 mock apps and an added ask_user action for clarification.

A Comparison with Existing Benchmarks

B.1 Mock Applications

  • The environment contains 26 mock applications populated with interconnected data for a coherent user persona.The seeded data are verified and revised to provide evidence for user preferences.Source: §B.1 · p. 14, Table B.1 · p. 15

B.2 Action Space

  • Agents use standard mobile GUI actions plus ask_user, which forwards clarification messages to the simulated user and returns its response.The supplied passages describe ask_user as an extension to the standard action space.Source: §B.2 · p. 14

B.3 Per-Task Environment Control

The benchmark uses task-specific environment changes alongside a shared user persona, chronological history, hidden profile, and interactive simulator. The history supports inference, while ask-only requirements must be elicited from the simulator.

B.3 Per-Task Environment Control

  • Per-task overrides modify app data, append synthetic history events, or update the profile, then revert after the episode.Preference perturbations may combine all three override types to align history, app data, and simulator and evaluator preferences.Source: §B.3 · p. 15

C User Details

  • The agent receives a chronological activity history and can consult an interactive simulator grounded in a structured profile that is hidden from the agent.The profile is used as ground truth for simulation and evaluation.Source: C User Details · pp. 15–16

C.1 User Activity History

  • History events are natural-language descriptions ordered by timestamp, combining cross-app activity that can jointly evidence preferences and plans.The full history is the agent’s only prior user knowledge; inferable requirements are grounded in it, while ask-only requirements require ask_user.Source: §C.1 · p. 16

C.2 User Profile

  • The hidden profile has 92 entries across 13 dimensions, each linked to supporting history events and manually verified against them.It serves as an oracle for the simulator and evaluation, not as information directly exposed to the agent.Source: §C.2 · p. 16, Table C.1 · p. 17

C.3 User Simulator

  • The simulator uses the profile, task-specific ask-only values, and optional situational context, and answers only information explicitly requested.It is instructed not to volunteer additional information.Source: §C.3 · p. 17

C.4 Human Validation of User Simulator

Human review found that 96.6% of 89 question-answer pairs were valid on both grounding and scope. Grounding failures arose in tasks without ask-only requirements, while the sole scope failure involved unsolicited information.

C.4 Human Validation of User Simulator

  • Four representative agents produced 89 question-answer pairs that reviewers manually assessed for grounding and scope.Grounding checks covered support, contradiction, withholding, and judgeability; scope checks covered unsolicited or incomplete replies.Source: §C.4 · pp. 17–18
  • 96.6% of pairs (86 of 89) are valid on both axes.Both grounding failures occurred without ask-only requirements; the single scope failure was a reservation reply that added the time.Source: §C.4 · p. 18

D.1 Task Collection

The task-collection pipeline generates candidates, refines them with human review, and validates execution in the simulator. The final benchmark retains 130 tasks after each retained task undergoes both refinement and validation.

D.1 Task Collection

  • Two task-authoring skills generate underspecified personalized tasks and tasks involving instruction, preference, or environment perturbations.Together, the skills produce the initial candidate pool.Source: §D.1 · p. 18
  • Eight reviewers refine candidates for natural instructions and feasible operations, verify each omitted requirement’s resolution source, and clarify its ground truth.Inferable requirements need clear history evidence; ask-only values must not be recoverable from history and are recorded in hidden context.Source: §D.1 · p. 18
  • Two experienced validators execute refined tasks to check feasibility and rubric correctness, correcting remaining instruction, environment, and rubric issues.The validation stage checks expected outcomes in the iOS simulator.Source: §D.1 · p. 18
  • 130 tasks remain in the final benchmark, and each receives human refinement and execution-based validation.The pipeline began with 188 candidates.Source: §D.1 · p. 18

D.2 Benchmark statistics

iPrefBench statistics cover its user environment, task composition, and evaluation rubrics. The benchmark also tracks cross-app preference transfer and the distribution of tasks across applications.

D.2 Benchmark statistics

  • Cross-app preference transfer occurs when evidence for an inferable requirement comes from a different app than the one where it must be applied.For example, a dining preference from TasteRank may inform a DineSpot reservation.Source: §D.2 · p. 19
  • Table D.1 summarizes statistics on the user environment, task composition, and evaluation rubrics.Source: Table D.1 · p. 19
  • Figure D.1 counts tasks involving each app, grouped by functional category and sorted within categories; some tasks involve multiple apps.Source: §D.2 · p. 19

E Rubric and Interaction Judge Details

The appendix details how task rubrics distinguish required actions from user-dependent decisions and how LLM judges assess completion and clarification behavior. It also reports validation of the judges and explains which agents were included in the comparison.

E.1 Task-specific Rubrics

  • Rubrics separate required operations from user-dependent decisions, with each personalization criterion labeled inferable from history or ask-only.Source: §E.1 · p. 19

E.2 Rubric-Based Completion Judge

  • The completion judge evaluates each criterion independently using the task, trajectory, final answer, ground-truth profile, applicable hidden context, and simulator time; a criterion passes only with clear evidence.Action-completion and personalization scores are fractions of satisfied criteria macro-averaged across tasks, while full success requires every criterion to pass.Source: §E.2 · pp. 19, 21
  • Figure D.1 groups app-use task counts by category and sorts apps within categories; some tasks span multiple apps.Source: Figure D.1 · p. 20

E.3 Interaction Judge

  • The interaction judge decomposes questions into information pieces, semantically matches them to ask-only requirements, and treats repeats as separate requests; inferable, irrelevant, or unmatched requests do not count as matches.Recall and precision are micro-aggregated across tasks.Source: §E.3 · p. 21
  • Message quality covers specificity, answerability, and option provision; agents score near-perfectly overall, with option provision a modest challenge.Option provision applies when answers form a small closed set; agents sometimes omit visible options, shifting the burden of inspecting them to users.Source: §E.3 · p. 21, Table E.1 · p. 21

E.4 Validation of LLM-as-a-Judge

  • 95.7% criterion-level and 97.0% task-level agreement with human annotations support the completion judge; interaction recall agreement was 95.2%, and extra-request-count agreement was 93.9%.The corresponding Cohen’s κ values were 0.89, 0.94, 0.90, and 0.82.Source: §E.4 · p. 22
  • With Gemini 3.7 Flash as an alternative judge, agreement was 95.9% on rubric criteria, 96.4% on task success, 99.1% on recall requirements, and 98.6% on per-task extra-request counts.The authors report similar action-completion and personalization agreement and conclude that results remain consistent with the alternative judge.Source: §E.4 · p. 22

F.1 Evaluated Agents

  • Seven frontier agents were evaluated, while specialized GUI agents were excluded when context limits or malformed actions under full-history input prevented meaningful personalization comparisons.MAI-UI-8B, for example, had a 16.9% Action Completion score in that setting.Source: §F.1 · p. 22

F.2 Error Analysis

This section analyzes personalization-related failures, including how agents mishandle inferable preferences, ask-only details, and unavailable options. It also describes the prompts governing clarification and task-success evaluation, with examples of agents silently substituting or omitting unavailable options without asking the user.

F.2 Error Analysis

  • 89% of Sonnet 5’s and 84% of Qwen3.8-27B’s failed trajectories contained personalization errors: silent defaults, missed inferences, silent guesses, or wrong inferences.The two agents’ error distributions were close, suggesting shared behaviors among agents optimized for autonomous execution rather than a weakness of one model.Source: §F.2 · pp. 22–23

F.3 Additional Case Studies

  • When a preferred option is unavailable, agents may keep trying to obtain it, substitute without consulting the user, or skip it without considering alternatives.The expected behavior is to surface unavailability, then proceed with an acceptable alternative or stop if none is acceptable.Source: §F.3 · p. 23

G.1 User Simulator

  • The simulated user is instructed to answer from the profile and plausible knowledge, honestly acknowledge unknowns, and reveal request-specific details only when explicitly asked.The prompt also instructs the user not to volunteer unasked details or perform the task.Source: §G.1 · pp. 23–24

G.2 Personalized Mobile-Use Agent

  • The mobile-use agent is instructed to infer open details from the user’s activity log and ask rather than guess when a material choice cannot be resolved confidently.It operates in vision-only mode, acts through a constrained action schema, and should stop only when the task is complete.Source: §G.2 · pp. 24–25

G.3 Rubric-based Judge

  • The rubric judge evaluates each criterion independently against trajectory evidence and the provided profile or request specifics, treating the criterion’s wording as authoritative.For question-answering tasks, it also checks whether the agent’s final answer is correct.Source: §G.3 · p. 26

G.4 Interaction Judge

  • The interaction judge splits clarifying questions into separately answerable information pieces, counts repeated asks, and matches each piece to a required-to-ask slot or null.Choices among alternatives count as one piece for one decision.Source: §G.4 · pp. 26–27

G.5 Pre-execution Clarification

  • Before execution, the clarification prompt directs the assistant to resolve open user-dependent details from the activity log and ask only about material details it cannot confidently settle.It calls for a short question for each such detail and an empty list when none needs clarification.Source: §G.5 · p. 27
  • When the preferred color is unavailable, the agent orders a different color and does not disclose the substitution in its final answer.Figure F.6 illustrates silent substitution.Source: Figure F.6 · p. 32
  • When an item is unavailable, the agent skips it without asking the user and completes the task without substituting another item.Figure F.7 illustrates direct omission.Source: Figure F.7 · p. 33

Related papers

Loading 2610.34735v1…