PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing Personalization Failures in Mobile GUI Agents

Zhixin Lin, Dongliang Xu, Jungang Li, Shidong Pan, Jun Feng, Yanbin Sun, Jing Li, Liang Liu, Yue Yao

arXiv:2610.30573v1cs.AI

TL;DR

Personalized GUI agents must determine how user preferences change execution and when they apply, questions that overall task-success benchmarks reveal only partially. PrefDiag-Bench combines five preference types with stage-wise diagnostic probes and evaluates agents using explicit profiles and behavioral logs. Its evaluation finds substantial challenges on preference-conditioned tasks, especially Ranking and Process, and identifies evidence selection and trigger recognition as bottlenecks.

  • Problem

    Existing evaluations centered on task-level success or preference-sensitive progress provide limited information about how preferences should alter execution and when they should take effect.

    Source: §1 · p. 2

  • Method

    PrefDiag-Bench categorizes preferences into five execution-oriented types and pairs end-to-end tasks with stage-wise diagnostic probes, evaluating explicit profiles and behavioral memory logs.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Performance varies substantially across preference types, profiles do not consistently outperform logs, and evidence selection and trigger recognition are bottlenecks; evidence guidance improves task success by up to 7.06 percentage points.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Connecting capability-level assessments with end-to-end outcomes provides a basis for more informative evaluation and targeted interventions.

    Source: §5 · p. 10

Abstract

from arXiv · show

Unlike conventional GUI agents, personalized GUI agents require user preferences to guide sequential execution over changing GUI states. This raises two challenges: determining HOW each preference should alter execution strategy and WHEN it should take effect along the interaction trajectory. Existing benchmarks primarily evaluate overall task success, offering limited insight into these questions. We introduce PrefDiag-Bench with two components: (1) a categorization of preferences into five types, characterizing HOW they affect execution; and (2) a stage-wise diagnostic framework with 1,664 probes, tracing WHEN preferences should take effect and where failures occur. We evaluate eleven GUI agent systems, including end-to-end agents and agentic workflows, under two forms of user context: explicit user profiles and behavioral memory logs. Although existing agents achieve relatively high completion rates on non-personalized tasks, performance is substantially lower on preference-personalized tasks. Even the best-performing model, Seed-2.0-Pro, achieves only 47.17% on Process preferences under explicit user profiles. Agents struggle particularly with Ranking and Process preferences. Further analysis reveals two key capability bottlenecks: selecting relevant evidence from user history and recognizing when preferences should take effect during execution. Controlled interventions that explicitly provide relevant evidence improve end-to-end task success by up to 7.06%. PrefDiag-Bench offers a more systematic evaluation of GUI agents' personalization capabilities beyond overall task success. Code and data are available at https://github.com/Zhixin-L/PrefDiag-Bench.

1 INTRODUCTION

Personalized GUI agents must use user history to adapt execution, determining both how preferences change decisions and when they apply along the GUI trajectory. PrefDiag-Bench pairs preference types with stage-wise probes, and its evaluation identifies evidence selection and trigger recognition as key bottlenecks.

1 INTRODUCTION

  • Personalization requires agents to draw on users’ past behavior, habits, and preferences to make decisions that meet individual needs during execution.The same base task can impose different execution requirements under different user preferences.Source: §1 · pp. 1–2
  • Agents must determine how each preference changes execution and when it should take effect, beyond simply completing the task.Existing task-level success or preference-sensitive progress evaluations provide limited information about this process.Source: §1 · p. 2
  • PrefDiag-Bench classifies preferences into five execution-oriented types and links 318 end-to-end tasks with 1,664 diagnostic probes.It probes information processing, contextual reasoning, and action formulation, with explicit user profiles and behavioral memory logs as context forms.Source: §1 · pp. 2–3
  • Across 11 evaluated systems, performance varies by preference type, explicit profiles do not consistently outperform behavioral logs, and evidence selection and trigger recognition are key bottlenecks.Controlled evidence guidance improves end-to-end task success by up to 7.06 percentage points.Source: §1 · p. 3

2 RELATED WORK

Prior work studies GUI agents and personalization from complementary directions, but does not jointly examine preference effects on GUI execution and the capabilities that bottleneck preference application. PrefDiag-Bench adds an execution-oriented, stage-wise diagnostic perspective and compares two forms of user context.

2.1 MOBILE GUI AGENTS AND THEIR PERSONALIZED EVALUATION

  • Prior personalized GUI evaluations broaden the field through long-term user records, preference-sensitive progress, behavioral logs, and contextual reasoning, but do not jointly diagnose preference effects on GUI execution and capability bottlenecks.PrefDiag-Bench complements these directions with execution-oriented and stage-wise diagnosis.Source: §2.1 · p. 3

2.2 PERSONALIZATION USER CONTEXT AND MEMORY

  • Memory benchmarks assess tracking, management, retention, and use, while PrefDiag-Bench compares explicit profiles and behavioral logs and probes preference application from context to action.Correctly inferring or retrieving user information does not necessarily mean an agent can apply preferences correctly in a GUI task.Source: §2.2 · p. 3
  • Table 1 frames personalization reasoning as evaluating both how preferences alter execution requirements and when they take effect, with diagnosis pairing tasks and capability probes.EUP denotes explicit user profiles; BML denotes behavioral memory logs.Source: Table 1 · p. 4

3 BENCHMARK DESIGN

PrefDiag-Bench evaluates personalized mobile GUI agents at task and diagnostic levels, organizing preferences by the execution decision they change. Its paired tasks and probes connect preference-conditioned outcomes to specific capabilities and data-construction choices.

3 BENCHMARK DESIGN

  • The benchmark contains 318 end-to-end tasks—53 Basic and 265 preference-conditioned—and 1,664 diagnostic probes linked to the same underlying preference instances.The tasks span seven application domains and preference atoms cover nine semantic categories.Source: §3 · p. 4, §3.2 · p. 4, Figure 2 · p. 5

3.1 PREFERENCE TAXONOMY

  • The taxonomy classifies each preference by the part of a personalized task’s success criterion it directly changes: Outcome, Constraint, Ranking, Process, or Interaction.Process requires following a procedure, while Interaction governs whether, when, and how to communicate with the user; a final GUI state alone may not verify either.Source: §3.1 · p. 4

3.2 BENCHMARK OVERVIEW AND STATISTICS

  • A personalized instance combines an instruction, user context, initial environment state, and deterministic success checker; success requires the task goal and preference-specific requirements.The checker can use the final state and relevant action or interaction records.Source: §3.2 · p. 5
  • The 1,664 probes test preference induction, evidence selection, task relevance, trigger recognition, and semantic action selection across five diagnostic stages.They are associated with 260 preference atoms from 256 tasks with observed behavioral-log failures.Source: §3.2 · p. 5

3.3 DATA CONSTRUCTION AND ANNOTATION

  • Preference-conditioned variants and paired profiles and behavioral logs were reviewed while the preference, instruction, initial state, and success checker stayed fixed across context forms.Task-linked probes were derived from behavioral-log failures, preference requirements were separated into atoms, and human reviewers answered the multiple-choice probes.Source: §3.3 · p. 5

4 EXPERIMENTS

PrefDiag-Bench evaluates GUI agents’ ability to apply user preferences across multi-step tasks, using preference categories and stage-wise probes. Across systems, personalized execution is harder than Basic tasks, with recurring weaknesses in Ranking, Process, evidence selection, and trigger recognition.

4.1 EXPERIMENTAL SETUP

  • The protocol tests 11 systems on 53 Basic and 265 personalized tasks, pairing profiles and logs while holding instructions, initial states, and checkers fixed.It applies independent P1–P5 probes to the nine directly queried agents; the two workflows are excluded from those probes.Source: §4.1 · p. 6

4.2 END-TO-END RESULTS

  • All 11 systems score lower on preference-conditioned than Basic tasks; Seed-2.0-Pro reaches 62.64% with profiles and 65.66% with logs versus 81.13% on Basic.Claude Sonnet 4.6 leads on Basic tasks at 86.79%, but Seed-2.0-Pro leads under both personalized settings.Source: §4.2 · p. 7
  • Outcome leads most systems, while Ranking and Process recur as weakest; under logs Seed-2.0-Pro scores 83.02% on Outcome versus 45.28% on Ranking.The authors associate Ranking difficulty with additional candidate comparisons and Process difficulty with longer execution procedures.Source: §4.2 · p. 7
  • Profiles and logs have no consistent winner: Seed-2.0-Pro and T3A improve with logs, but M3A falls from 37.73% with profiles to 18.49% with logs.Profiles provide compact preference rules, while logs preserve behavioral evidence that agents must identify as relevant.Source: §4.2 · p. 7

4.4 JOINT ANALYSIS

  • P2 and P4 failures often coincide with task failures; Step-GUI-4B fails both task and probe on 82.1% of P2 pairs and 66.0% of P4 pairs.Figure 4 partitions paired outcomes into both passing, task-only passing, probe-only passing, and both failing.Source: Figure 4 · p. 8, §4.4 · p. 8

4.3 STAGE-WISE DIAGNOSTIC RESULTS

  • P2 Evidence Selection and P4 Trigger Recognition average 40.06% and 39.69%; Qwen3-VL-8B-Thinking leads open-source agents at 69.53% on P2 and 45.79% on P4.P1 and P5 exceed 98.9% on average, while P3 averages 88.80%.Source: §4.3 · p. 8

4.5 CONTROLLED EVIDENCE INTERVENTION

  • C1 labeled logs raise Seed-2.0-Pro TSR from 76.47% to 83.53%, versus 81.18% with C2 supporting-only logs.On the 85-task subset, Qwen3-VL-8B-Thinking reaches only 16.47% under C1, suggesting limited GUI execution can constrain gains from evidence guidance.Source: Table 3 · p. 9, §4.5 · p. 9

4.6 ROBUSTNESS TO MEMORY CONSTRUCTION METHODS

  • Across alternative memory methods, Outcome remains highest and Ranking and Process lowest; Mem0 scores 71.70% on Outcome versus 43.40% on Ranking, while Hindsight scores 79.25% on Outcome versus 37.74% on Process.The preference-type pattern persists despite changes in absolute scores and the ordering of Ranking and Process.Source: §4.6 · pp. 9–10

5 CONCLUSION

PrefDiag-Bench combines five preference types with stage-wise diagnostics to examine how preferences shape execution, when they should apply, and where capability gaps arise. Its evaluation shows that overall task success alone can obscure differences across preference types and application stages.

5 CONCLUSION

  • The benchmark pairs five preference types with stage-wise diagnosis to examine execution effects, timing, and capability gaps.It connects capability-level assessments with end-to-end outcomes as a basis for more informative evaluation and targeted interventions.Source: §5 · p. 10
  • Overall task success alone can obscure agents’ distinct strengths and weaknesses across preference types and stages of preference application.Source: §5 · p. 10
  • The authors hope the benchmark supports GUI agents that understand preferences and apply them reliably throughout task execution.Source: §5 · p. 10

AI USE STATEMENT

The authors used generative AI for synthetic-data generation, translation, and language editing, with manual review of AI-assisted content.

AI USE STATEMENT

  • Generative AI assisted synthetic-data generation, translation, and language editing and polishing.Source: AI USE STATEMENT · p. 10
  • The authors manually reviewed and revised AI-generated synthetic data before including it in the benchmark.Source: AI USE STATEMENT · p. 10
  • The authors state that they reviewed all other AI-assisted content and take responsibility for the final work.Source: AI USE STATEMENT · p. 10

A DATASET DETAILS AND EXAMPLES

The appendix describes the task catalog, paired user contexts, and examples of preference types P1–P5; task families are labeled PREFD 01–53.

A DATASET DETAILS AND EXAMPLES

A.1 TASK CATALOG AND EXAMPLES

The catalog describes 53 Basic task families and their preference-conditioned variants, with paired explicit profiles and behavioral logs for personalized tasks. Its examples show how variants add user preferences to otherwise shared task goals.

A.1 TASK CATALOG AND EXAMPLES

  • The catalog covers 53 Basic tasks, each with five preference-conditioned variants, yielding 318 end-to-end instances and 265 personalized tasks.Profiles and behavioral logs provide alternative contexts for the same personalized tasks.Source: §A.1 · p. 13, Table 4 · p. 13
  • The application column in Table 4 lists applications for each Basic task, not every application available in its environment.Source: Table 4 · p. 13
  • PREFD 02’s six instances retain a shared purchase goal while variants add an outcome, budget, store-priority, procedure, or confirmation preference.The Outcome variant retrieves usual-choice and recipient details from context, while the Basic instruction specifies the shoes and delivery details.Source: §A.1 · p. 15, Table 5 · p. 15
  • The Outcome example uses context to identify the user’s usual commuting sneakers, size, and delivery details.Source: §A.1 · p. 16
  • The Constraint example caps the shoe price at CNY 300 and requires explaining the situation before ordering if only costlier options are available.Source: §A.1 · p. 16
  • The Ranking example prioritizes an official flagship store over an ordinary store, considering the latter only if the official store is out of stock.Source: §A.1 · p. 16
  • The Process example requires comparing the same shoe on Taodian and JD before ordering from the cheaper platform; the Interaction example requires confirmation before ordering.Source: §A.1 · p. 16

A.2 USER-CONTEXT CONSTRUCTION AND ANNOTATION

The benchmark pairs reviewed explicit profiles with behavioral logs, annotates preference evidence and trigger conditions, and uses five independent probe types to examine personalization across a task trajectory.

A.2 USER-CONTEXT CONSTRUCTION AND ANNOTATION

  • Profiles are drafted from intended preferences and applicability conditions, revised by GPT-5.6 Sol, and manually checked before serving as the basis for behavioral-log synthesis.Logs then combine supporting records with plausible distractors and are revised and reviewed to ensure compatibility with the profile.Source: §A.2 · p. 16
  • Reference annotations link preference atoms to supporting log entries, applicability, trigger conditions, and expected behavior, with reviewers checking evidence against record content.Inconsistencies prompt source-record revisions and rechecks of affected context pairs and probes; automated checks validate identifiers, required fields, and evidence references.Source: §A.2 · pp. 16–17

A.2.1 PAIRED-CONTEXT EXAMPLE: PREFD 02 / INTERACTION

  • The interaction example pairs a confirmation-before-submission profile with logs where an ask–confirm–submit sequence and a correction support the preference, while other records are distractors.The instruction and initial task context are paired with the profile and logs; reference evidence labels are not part of ordinary behavioral-log input.Source: §A.2.1 · pp. 17–18

A.3 DIAGNOSTIC PROBE EXAMPLES

  • 1,664 diagnostic items are queried independently, with shared inputs supplied in full and reference answers kept separate from model inputs.The set includes 260 P1, 260 P2, 325 P3, 601 P4, and 218 P5 items.Source: §A.3 · p. 18

A.3.1 P1: LOG-TO-MEMORY INDUCTION

  • P1 tests whether logs support induction of a stable confirmation preference: search and filtering may continue, but a final decision opportunity precedes an order.The reference answer distinguishes this from asking after ordering, asking only at search initiation, or asking at every product or size change.Source: §A.3.1 · p. 18

A.3.2 P2: EVIDENCE SELECTION

  • P2 tests selection of the logs that directly support a known preference, requiring an exact match to the reference set.The reference set includes a correction against automatic ordering and a prior ask–confirm–submit behavior.Source: §A.3.2 · pp. 18–19

A.3.3 P3: RELEVANCE SELECTION

  • P3 tests whether a known preference is relevant to the whole current task, distinguishing task relevance from whether its trigger is active now.The reference answer selects the everyday-shoe confirmation preference for the shoe-purchase task.Source: §A.3.3 · p. 19

A.3.4 P4: TRIGGER RECOGNITION; A.3.5 P5: OPERATION MAPPING

  • P4 asks whether the final-confirmation trigger is active at the pending order screen, while P5 tests mapping that active preference to asking and waiting before submission.The P4 reference state follows product and delivery selection; P5’s correct strategy is to request explicit confirmation before deciding whether to submit.Source: §A.3.4 · p. 20, §A.3.5 · pp. 20–21

B EXPERIMENTAL DETAILS

The experiments use a shared Android evaluation environment for basic and personalized tasks, with common step limits and task checkers across agents and workflows.

B.1 ENVIRONMENT AND AGENT CONFIGURATION

  • The evaluation reuses KnowU-Bench’s Android environment and infrastructure with task-specific initial data and checkers, resetting agent history between episodes.The 53 Basic tasks have no added user context; each of 265 personalized tasks receives either a profile or behavioral logs, which remain available during execution.Source: §B.1 · p. 21
  • Episodes allow at most 65 environment steps with a one-second inter-step wait and may end earlier when an agent reports completion or inability to proceed.Open-source models run on eight NVIDIA RTX 4090 GPUs.Source: §B.1 · p. 21
  • Adapters retain native observation and action formats while mapping to a common interface, and M3A and T3A use the same contexts, step budget, and checkers as direct agents.The passage also specifies model-specific decoding settings and workflow limits, including up to 20 objective action summaries for M3A and T3A.Source: §B.1 · p. 21

B.2 SCORING AND TASK–PROBE ASSOCIATION

The section defines task success and diagnostic scoring, links probe outcomes to task execution, and describes an evidence-guidance intervention and isolated memory-module configurations.

B.2 SCORING AND TASK–PROBE ASSOCIATION

  • Binary task success is derived from the native checker score, and TSR is the percentage of valid tasks that succeed; personalized AVG macro-averages the five preference-type TSRs.Checkers inspect application state, database or callback records, and required action and interaction traces, not a separate natural-language reason.Source: §B.2 · p. 21
  • Diagnostic accuracy requires exact reference answers, with task-level success requiring all associated canonical questions for a probe type to be correct.Groups with missing, truncated, or unparseable responses are excluded and their coverage recorded.Source: §B.2 · p. 21
  • Diagnostic and execution outcomes are joined by model, task-family ID, and preference type, with pooled proportions computed from valid pair counts.Pairs are categorized as TT, TF, FT, or FF according to task success and diagnostic accuracy.Source: §B.2 · p. 21

B.3 EVIDENCE INTERVENTION

  • The evidence intervention compares unlabelled logs, the same logs with relevant/irrelevant evidence labels, and logs retaining only supporting entries.The fixed, shared subset contains 85 task–preference pairs from 43 families and is selected using pre-intervention records.Source: §B.3 · p. 22

B.4 MEMORY-MODULE CONFIGURATIONS

  • Mem0, LangMem, and Hindsight process frozen synthetic episodes in isolated stores for each task–preference pair, and their returned memory text replaces Seed-2.0-Pro’s behavioral-log context.The reference profile, preference atom, evidence labels, and checker are not supplied to the memory modules; no reference-based correction is applied to their output.Source: §B.4 · p. 22, Table 6 · p. 22

C PREFERENCE APPLICATION IN GUI TASKS

The examples show how personalization changes GUI choices while preserving task-checker success: a ranking preference changes store choice, and a budget constraint changes product selection.

C PREFERENCE APPLICATION IN GUI TASKS

  • The examples align corresponding Basic and Personalized decision stages, with the task instruction alone in Basic and behavioral memory logs added in Personalized.The figures retain original step numbers.Source: C PREFERENCE APPLICATION IN GUI TASKS · p. 22

C.1 CHOOSING AN OFFICIAL STORE (PREFD 03, RANKING)

  • The official-store preference changes selection from the ordinary listing at CNY 8,299 to the official listing at CNY 8,999, and both runs pass their checkers.The preference prioritizes official sources among products meeting the configuration and price requirements.Source: Figure 7 · p. 23, §C.1 · p. 23

C.2 STAYING WITHIN BUDGET (PREFD 02, CONSTRAINT)

  • The budget preference changes selection from a CNY 399 pair in initial results to a CNY 259 pair found by scrolling further, with both orders passing their checkers.Both selected pairs are white, size 42 shoes; the budget cap is CNY 300.Source: Figure 8 · p. 24, §C.2 · p. 24

Related papers

Loading 2610.30573v1…