Source-linked AI summary
KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation
Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, Wenqi Zhang, Xu Tan, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen
TL;DR
Existing mobile-agent benchmarks do not directly test interactive preference acquisition or calibrated intervention in live GUI environments. KnowU-Bench addresses this gap with an online, execution-grounded benchmark and finds that agents remain much weaker at personalized and proactive decision making than at explicit GUI execution.
Problem
Existing benchmarks mainly evaluate static preference recovery or fixed-context intent prediction, leaving interactive preference elicitation and proactive decisions about intervention, consent, or silence insufficiently tested.
Method
KnowU-Bench combines a reproducible Android emulator, hidden user profiles with exposed behavioral logs, interactive user simulation, and hybrid rule-based and LLM-based evaluation.
Results
Agents show a large gap between explicit-task execution and personalized or proactive decision making, with proactive failures dominated by intervention and passivity errors totaling 80.0%.
Takeaways & Limitations
The main bottlenecks are interactive preference acquisition for personalization and initiative calibration for proactivity, rather than GUI navigation alone.
Abstract
from arXiv · showhide
Personalized mobile agents that infer user preferences and calibrate proactive assistance hold great promise as everyday digital assistants, yet existing benchmarks fail to capture what this requires. Prior work evaluates preference recovery from static histories or intent prediction from fixed contexts. Neither tests whether an agent can elicit missing preferences through interaction, nor whether it can decide when to intervene, seek consent, or remain silent in a live GUI environment. We introduce KnowU-Bench, an online benchmark for personalized mobile agents built on a reproducible Android emulation environment, covering 42 general GUI tasks, 86 personalized tasks, and 64 proactive tasks. Unlike prior work that treats user preferences as static context, KnowU-Bench hides the user profile from the agent and exposes only behavioral logs, forcing genuine preference inference rather than context lookup. To support multi-turn preference elicitation, it instantiates an LLM-driven user simulator grounded in structured profiles, enabling realistic clarification dialogues and proactive consent handling. Beyond personalization, KnowU-Bench provides comprehensive evaluation of the complete proactive decision chain, including grounded GUI execution, consent negotiation, and post-rejection restraint, evaluated through a hybrid protocol combining rule-based verification with LLM-as-a-Judge scoring. Our experiments reveal a striking degradation: agents that excel at explicit task execution fall below 50% under vague instructions requiring user preference inference or intervention calibration, even for frontier models like Claude Sonnet 4.6. The core bottlenecks are not GUI navigation but preference acquisition and intervention calibration, exposing a fundamental gap between competent interface operation and trustworthy personal assistance.
1 Introduction
KnowU-Bench addresses gaps in mobile-agent evaluation by testing online preference acquisition and calibrated proactive assistance in reproducible GUI tasks. Its evaluation shows that capable explicit-task agents degrade substantially when success requires personalization or deciding whether to intervene.
- Motivation: Existing benchmarks largely evaluate offline preference recovery, trajectory matching, or intent similarity rather than live GUI task completion.Online benchmarks are more realistic but less reproducible.
- Motivation: Interactive preference acquisition is missing from prior evaluation, despite practical agents needing to obtain unspecified preferences through dialogue.Existing benchmarks generally infer intent from static behavioral logs.
- Motivation: Proactive evaluation remains incomplete because agents must decide whether to intervene, seek consent, or remain silent after rejection or when no routine applies.The missing capability is calibrated initiative across the full decision chain.
- Benchmark: KnowU-Bench couples a reproducible Android emulator with interactive user feedback and programmatic verification for personalized and proactive mobile-agent evaluation.Its design includes structured profiles, behavioral logs, and a full proactive service chain.
- Benchmark: 42 general, 86 personalized, and 64 proactive tasks expose a sharp difficulty shift from explicit GUI execution to preference-conditioned and proactive assistance.The benchmark evaluates 11 mainstream models and separates general execution from personalization and proactivity.
- Findings: About 30% average performance is lost on unspecified tasks requiring personalization or proactivity, while 93.8% of Claude Sonnet 4.6 personalized failures involve clarification or partial-preference errors.For the same model, 80.0% of proactive failures are intervention or passivity errors.
2 Related Work
Related benchmarks have progressed from offline action matching toward reproducible Android environments and ambiguous user interaction, but personalization is still commonly treated as static inference from behavioral history.
- Prior GUI benchmarks: Early benchmarks established offline action-matching protocols, while AndroidWorld added reproducible full-stack Android evaluation with programmatic rewards.These advances primarily measure execution ability.
- Prior GUI benchmarks: MobileWorld introduced agent-user interaction under ambiguous instructions, and MemGUI-Bench incorporated long-term memory into mobile evaluation.Despite these additions, tasks remain largely one-shot and explicitly specified.
- Personalization and proactivity: PersonalAlign and Me-Agent study ambiguous instructions by recovering intent from fixed historical preference signals, while FingerTip 20K mines long-term logs for proactive suggestions and personalized execution.These approaches differ from KnowU-Bench’s online, execution-grounded evaluation.
3 KnowU-Bench
KnowU-Bench combines a reproducible Android environment, hidden user profiles with exposed logs, interactive user simulation, and hybrid evaluation to test general, personalized, and proactive mobile tasks.
- Environment Setup: Mobile automation is formulated as a POMDP whose observations include instructions and interface views, actions are mobile UI operations, and rewards indicate task completion.The transition function maps environment states and actions to new states.
- Environment Setup: The benchmark runs tasks in a containerized rooted Android stack with a unified controller and fixed emulator snapshots for reproducible initialization and reset.The environment supports the full lifecycle from task initialization through evaluation.
- Environment Setup: KnowU-Bench covers 23 applications, including added shopping and food-delivery apps that support cross-platform preference following.The broader app ecosystem targets personalized commerce and daily-service decisions.
- User Agent: An LLM-driven user simulator uses a structured profile and timestamped interaction log, exposing the log to the GUI agent while keeping the profile hidden.This asymmetry requires preference inference from behavioral patterns rather than privileged profile access.
- Task Definition: Agents receive instructions, screenshots, environment feedback, and interaction history, and may obtain user responses through ask_user actions.This extends standard GUI-agent inputs with history-grounded context and online feedback.
- Task Definition: General tasks isolate GUI execution, personalized tasks require resolving user preferences through logs or dialogue, and proactive tasks require acting, asking for confirmation, or remaining silent.After confirmation, agents must condition execution on acceptance or adjust behavior after rejection.
- Hybrid Evaluation Strategy: Hybrid evaluation combines deterministic checks of verifiable states with rubric-conditioned LLM judging of semantic dimensions such as preference alignment and clarification quality.The mixing weight is 1 for deterministic tasks, 0 for purely semantic tasks, and proportional to preference-dependent requirements for personalized tasks.
4 Experiment
KnowU-Bench evaluates mobile agents across general, personalized, and proactive tasks, showing that performance declines sharply when agents must infer preferences, calibrate initiative, or use memory effectively. The main failures arise from preference acquisition and proactive calibration rather than GUI navigation.
- Experimental Setup: KnowU-Bench evaluates 11 models across general, personalized, and proactive tasks using success, efficiency, preference-alignment, interaction, and policy-aware metrics.The setup includes noisy full-history memory and reports Act, Silent, and Stop for proactive tasks.
- Difficulty Progression: 100.0% success on the easy general split contrasts with 44.2% for Claude Sonnet 4.6 on the hard personalized split, while open-source models remain below 12%.The contrast indicates that fully specified GUI execution is easier than user-specific reasoning.
- Role Dependence: Claude Sonnet 4.6 remains relatively stable across roles at 71.7%–79.4%, whereas Seed 2.0 Pro ranges from 48.5% on grandma to 71.3% on researcher.Grandma is the hardest role on average, and student produces the largest spread across models.
- Preference Acquisition Through Interaction: Claude Sonnet 4.6 reaches 44.2% success and 78.9% average score while asking 0.4 questions per task, showing that effective interaction depends on using feedback correctly.Asking more questions alone does not guarantee better downstream execution.
- Proactive Safety Analysis: Proactive performance involves a trade-off between initiative and restraint: Claude Sonnet 4.6 leads Act at 70.8%, while Qwen3.5-122B-A10B leads Stop at 83.3% but performs weakly on Act and Silent.Act, Silent, and Stop capture intervention, appropriate non-intervention, and stopping after rejection.
- Memory Implementation: Memory interfaces are model-dependent: Qwen3-VL-8B improves from 13.6% with full-log clean memory to 20.4% with RAG-log clean memory, while noisy retrieval reduces MAI-UI-8B to 9.3%.Selective retrieval can sharpen grounding for some models but discard useful context or destabilize fragile memory use for others.
- Judge and Simulator Sensitivity: The hybrid evaluator aligns more closely with human ratings than the pure rule-based variant, combining deterministic verification with semantic assessment of preference satisfaction.On 26 shared trajectories, it shows tighter clustering around the agreement diagonal and lower mean absolute error.
- Error Analysis: 66.7% of Claude Sonnet 4.6 personalized failures are Clarify errors and 60.0% of proactive failures are Intervention errors, making preference acquisition and initiative calibration the dominant bottlenecks.Partial failures account for 27.1% of personalized failures, while Passive, GUI, and Rejection account for 20.0%, 15.0%, and 5.0% of proactive failures.
5 Conclusion
KnowU-Bench reframes personalized mobile assistance as deciding what to do for which user and when, not merely executing explicit instructions. Experiments show substantial remaining gaps in personalized decision making, especially proactive initiative and restraint.
- KnowU-Bench combines reproducible Android emulation, structured profiles, user logs, interaction, and hybrid evaluation into an online execution-grounded benchmark.
- Current agents often navigate interfaces successfully but struggle to determine the appropriate action for a particular user and timing.
- Proactive routine scenarios widen the gap by requiring initiative calibration and restraint after rejection.
- The benchmark is positioned as both an evaluation benchmark and a research platform for personalized mobile intelligence.
- Future progress is framed around long-term memory access, ambiguity-resolution policies, and safer proactive decision boundaries.
B GUI Action Space
The supplied passages identify the KnowU-Bench action-space table but do not describe its actions or organization.
- Table 4 summarizes the GUI action space used by KnowU-Bench.
C.1 App List
KnowU-Bench extends mobile service environments with controlled shopping and food-delivery apps for preference-sensitive tasks. Its profiles use a shared, extensible top-level schema supporting task-specific access to habits and preferences.
- App List: Table 5 catalogs covered apps, functional roles, comparable commercial apps, and associated task counts.
- App List: The benchmark adds Taodian, jingdian, chilemei, and tuantuan as two shopping and two food-delivery applications.
- App List: These apps support controlled evaluation of platform choice, payment habits, delivery addresses, cuisine preferences, and app-specific ordering routines.
- App List: Shopping apps use editable local mock data and varied layouts, inventories, and styling to evaluate cross-platform shopping preferences.
- App List: Food-delivery apps use self-contained frontend pipelines with static shop, menu, rating, and address data while varying storefront content and appearance.
- App List: Critical completion events are instrumented for automated verification, while deployment uses a two-stage packaging pipeline after UniApp Android compatibility problems.
- User Profiles: Role profiles are YAML files synthesized with LLM assistance, curated from four archetypes, and exposed through a unified top-level interface.
- User Profiles: The profile schema includes identity, locations, digital_context, habits, preferences, decision_criteria, and social_graph, with weakly constrained nested fields.
D.2 User Logs
KnowU-Bench exposes reviewed behavioral logs while keeping the underlying profile hidden, requiring agents to infer preferences from free-text temporal, spatial, and behavioral traces. The setup also supports interactive strategy selection across execution, asking first, and monitoring.
- Log Construction: Each role and noise condition has a JSON log file, while the exposed history remains separate from the hidden profile.
- Log Construction: Log entries are flat event records organized around five fields summarized in Table 7.
- Log Construction: Noisy logs add roughly 25% distractor events, creating a controllable clean-versus-noise condition for personalization and memory experiments.
- Log Presentation: The default context hides label and category fields, presenting the agent with natural-language traces emphasizing temporal, spatial, and behavioral evidence.
- Agent Interaction: Task contexts combine user logs and system state with prompts that ask agents to choose interactive execution, autonomous execution, or monitoring only.
- Task Examples: Preference examples pair activity histories with ambiguous requests, while proactive examples omit explicit instructions and require calibrated initiative.
- User Simulator: The simulator grounds responses in role profiles, including dietary, app, lifestyle, contact, and decision-making preferences.
- User Profile Example: The example profile specifies locations, devices, routines, preferences, social priorities, and tradeoffs such as time over money.
F.1 General Task Successful Cases
General tasks test direct execution of explicit instructions, with successful examples showing agents completing contact calls and posting tasks.
- General tasks evaluate direct execution of explicit instructions.
- A representative success opens Contacts, selects Son (Qiang), and places the call.
- A Mastodon task asks the agent to post about finally beating a difficult game boss.
F.2 Personalized Task Successful Cases
Successful personalized and proactive cases require agents to infer preferences or detect risks, then complete the corresponding GUI action safely.
- A personalized posting task succeeds when the agent infers the user’s usual followers-only visibility preference.
- A proactive agent detects a suspicious SMS and blocks and reports the sender as spam.The example requires both correct intervention timing and reliable GUI follow-through.
- Personalized failures mainly involve preference inference or acquisition, whereas proactive failures involve intervention decisions or downstream execution.
F.4.1 Personalized Task Failure Cases
Personalized-task failures arise from incorrect preference grounding, insufficient clarification, partial preference satisfaction, and GUI execution errors.
- Personalized failures include preference grounding, clarification, execution, and partial preference satisfaction errors.
- Preference Misidentification: A Mastodon agent publishes publicly instead of applying the user’s usual followers-only visibility preference.
- Insufficient Clarification: When clothing preferences are underdetermined, the agent browses the cart instead of asking for clarification.
- Partial Preference Satisfaction: The agent removes expensive clothes from Taodian despite the user’s preference for shopping on jingdian.
- GUI Navigation Failure: A beverage-purchase agent identifies the personalized product and destination but mishandles the quantity semantics for a 24-drink case.
F.4.2 Proactive Task Failure Cases
Proactive-task failures span missed warranted interventions, unjustified actions, violations after rejection, and GUI execution breakdowns.
- Proactive failures include false passivity, unwarranted intervention, post-rejection violation, and GUI navigation errors.
- False Passivity: Despite a valid morning weather trigger, the agent remains inactive instead of initiating the routine.
- Unwarranted Intervention: Without a valid trigger, the agent hallucinates shopping intent and opens Taodian without permission.
- Post Rejection Violation: After explicit rejection, the agent overrides the role prior and user response, labels the sender spam, and blocks the number.
- GUI Navigation Failure: Correct timing and policy grounding do not guarantee execution: the gallery agent fails to delete the targeted screenshots.