PaperlayerView exact revision on arXiv

Source-linked AI summary

SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information

Junjie Ye, Zhuohui Sheng, Shaofan Liu, Yulun Zhu, Wenjie Fu, Dingwei Zhu, Ming Zhang, Yujiong Shen, Weichao Wang, Xin Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang, Pluto Zhou

arXiv:2608.10692v1cs.CLcs.AI

TL;DR

Mobile assistants must find and combine personal information scattered across apps, but existing benchmarks provide limited evidence about this setting. SPIEval evaluates this capability, and its results show that even the strongest evaluated model achieves only 57.3% accuracy.

  • Problem

    Existing benchmarks provide limited evidence about assistants locating personal information across multiple apps, since required details are often explicit, directly accessible, or confined to individual documents.

    Source: §1 · p. 2

  • Method

    SPIEval is a human-curated benchmark of 250 tasks across five cognitive capabilities, grounded in 4,335 records from 10 apps and supporting multi-turn interaction through 21 tools.

    Source: §1 · p. 2

  • Results

    GPT-5.5 (xhigh), the strongest evaluated model, achieves only 57.3% accuracy on SPIEval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Information localization emerges as the primary bottleneck for current LLM-based mobile assistants.

    Source: §1 · p. 2, §7 · p. 10

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduce SPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e., reasoning, disambiguation, integration, preference inference, and multi-intent decomposition). SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurate information localization, as LLMs often commit to plausible but incorrect information instead of continuing retrieval for verification. We also find that fewer than 2% of retrieval actions employ advanced search methods and observe substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research in this direction. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.

1 INTRODUCTION

Mobile assistants must resolve underspecified requests by locating and combining personal information scattered across apps. SPIEval benchmarks this challenge, and its evaluation finds substantial room for improvement among current LLMs.

  • Prior work evaluates app orchestration (Trivedi et al., 2024), app-operation safety (Lin et al., 2026), and personalized responses (Mok et al., 2025), but not cross-app information localization.Source: §1 · pp. 1–2
  • Existing benchmarks often provide parameters explicitly or through earlier tool calls (Trivedi et al., 2024; Xie et al., 2024; Ye et al., 2025; Zhang et al., 2025a), while personalization benchmarks generally search individual documents (Tan et al., 2025).Source: §1 · p. 2
  • SPIEval comprises 250 tasks across five cognitive capabilities, grounded in 4,335 records from 10 apps, with 21 tools supporting multi-turn interaction.The tools include 11 retrieval tools and 10 execution tools; the benchmark also features diverse scenarios, challenging tasks, controllable environments, and verifiable outcomes.Source: §1 · p. 2
  • GPT-5.5 (xhigh) achieves 57.3% accuracy, while the weakest model reaches 16.4%, revealing substantial room for improvement across nine evaluated LLMs.Source: §1 · p. 2
  • 79% of failures stem from inaccurate information localization, and fewer than 2% of retrieval actions use advanced search methods, with search efficiency varying substantially across models.LLMs often commit to plausible but incorrect information rather than continuing retrieval for verification.Source: §1 · p. 2

2 RELATED WORK

Prior mobile-assistant systems and benchmarks advance task automation and personalization, but existing evaluations do not test locating personal information scattered across apps; SPIEval targets that gap.

2 RELATED WORK

  • Mobile-assistant research progressed from commonsense-planned multi-step tasks to real-world end-to-end completion, cross-app multi-agent coordination, and self-evolving systems.Early work focused on action planning; later systems extended task completion to real-world environments and improved coordination and learning through interaction.Source: §2 · p. 2
  • SPIEval targets five cognitive capabilities with manually constructed, independently verified tasks spanning personal records across 10 apps.The framework associates records with a unified user profile and provides 10 execution tools and 11 retrieval tools.Source: Figure 2 · p. 3, §2 · p. 3
  • Apple Intelligence1, Doubao Mobile Assistant2, and Honor YOYO Agent3 bring intelligent task automation to hundreds of millions of users.As such systems become widely deployed, evaluating their ability to handle real-world personalized tasks becomes increasingly important.Source: §2 · p. 3
  • Existing mobile benchmarks cover tool-driven execution, dynamic conditions, safety, or personalization, but assume information is in instructions or simple document retrieval rather than scattered across apps.SPIEval evaluates assistants in this cross-app setting.Source: §2 · p. 3

3 SPIEVAL

SPIEval evaluates mobile assistants that must retrieve and reason over scattered personal information, then execute tool calls to fulfill underspecified instructions. It operationalizes five cognitive capabilities in a manually constructed, multi-app benchmark with reproducible, automatically verifiable outcomes.

3.2 COGNITIVE CAPABILITIES

  • The five capabilities are multi-hop reasoning, disambiguating evolving or conflicting records, integrating independent sources, inferring user preferences, and decomposing multi-intent instructions.Tasks may require resolving a person before retrieving a phone number, identifying a currently valid value, combining records across apps, inferring a habitual choice, or splitting a request into subtasks.Source: §3.2 · p. 4

3.3 BENCHMARK CONSTRUCTION

  • SPIEval models 10 apps with structured schemas and 21 tools, including 11 retrieval tools and 10 app-specific execution tools.Retrieval supports substring, regular-expression, and fuzzy matching, field targeting, case-sensitivity controls, and paginated results; execution tools specify required and optional parameters.Source: §3.3 · pp. 4–5
  • SPIEval contains 250 tasks grounded in 4,335 records across 10 apps, with 50 tasks for each of five cognitive capabilities.Tasks include natural-language instructions, task-specific records, reasoning annotations, and gold execution calls with parameters.Source: §3.3 · p. 5

3.4 DATASET ANALYSIS

  • Cross-validation, execution of gold answers, and deterministic local tools support reliable, reproducible evaluation against annotated parameters rather than LLM judgments.Tasks are accepted only when an independent researcher reaches the same answer through the intended retrieval process; 7.0% of parameters allow multiple valid values, and tool order does not matter for 76 multi-tool tasks.Source: §3.3 · p. 5, §3.4 · pp. 6–7
  • The benchmark spans 357 execution operations in 10 categories, covering all 50 capability-operation pairs at least five times each.Even the least frequent operation category appears 27 times.Source: Table 1 · p. 5, §3.4 · p. 6
  • Instructions average 34 characters while requiring 8.47 execution parameters on average; one 20-character instruction requires 45 parameters.Instructions explicitly provide none of the required parameters, leaving tool selection and parameter inference largely implicit.Source: §3.4 · p. 6, Figure 3 · p. 6
  • Each instruction requires an average of 17.3 task-relevant records spanning 6.2 apps, with the most demanding involving up to 39 records.This requires models to identify relevant records across apps, reconcile them, and integrate evidence.Source: §3.4 · p. 6, Figure 4 · p. 6

4 EXPERIMENTAL SETUP

The experimental setup evaluates nine representative LLMs using exact-match accuracy on final execution calls, with controlled reasoning effort and repeated runs. Retrieval is paginated and tasks are capped at 50 interaction turns.

Models

  • Nine representative state-of-the-art LLMs were evaluated.They were Claude Opus 4.8, DeepSeek-V4-Pro, Gemini 3.1 Pro, GLM-5.2 (Zhipu AI, 2026), GPT-5.5, Hy3, Kimi K2.6, Qwen3.7-Plus, and Seed-2.1-Pro (Bytedance Seed, 2026).Source: §4 · p. 7

Metrics

  • Binary accuracy counts an outcome as correct only when final execution tools and all parameter values exactly match an annotated gold answer.Intermediate steps are not evaluated because tool feedback may change a model’s behavior.Source: §4 · p. 7

Implementation Details

  • Each model was tested at its highest and lowest reasoning effort, with defaults otherwise, a 50-turn task limit, and three runs per setting.Retrieval returns at most five results per page, requiring models to request additional pages; results are reported as the mean and standard deviation across runs.Source: §4 · p. 7

5 MAIN RESULTS

SPIEval exposes major limits in mobile assistants’ ability to locate personal information: even the strongest evaluated model achieves only 57.3% accuracy. Results point to retrieval and information localization as key bottlenecks, alongside weaknesses in preference inference, multi-intent decomposition, and converting extra reasoning into effective retrieval.

5 MAIN RESULTS

  • GPT-5.5 (xhigh) achieves 57.3% accuracy, while Kimi K2.6 (none) achieves 16.4%, showing that even the strongest evaluated model leaves substantial room for improvement.The paper describes current LLMs as struggling with information localization and characterizes the weakest models as largely impractical for this task.Source: §5 · p. 8
  • 79% of failures stem from incorrect parameter values, and removing retrieval raises average accuracy from 35.5% to 66.8%, implicating target-record localization as a key bottleneck.Returning all matching records at once raises average accuracy only from 35.5% to 36.0%; incorrect tool selection is relatively uncommon, and 6% of failures involve unspecified mandatory parameters. The findings suggest that models may act on plausible but incorrect information rather than continue retrieval to verify it.Source: §5 · p. 8, Figure 6 · p. 8
  • Average accuracy is around 46% for reasoning, disambiguation, and integration, while preference inference and multi-intent decomposition perform about half as well.With relevant records provided, preference inference reaches 67.0%, a 44.1-point gain over standard conditions; multi-intent decomposition reaches only 45.7%, indicating continued difficulty decomposing requests and planning coordinated subtasks.Source: §5 · p. 8
  • Increasing reasoning effort yields an average gain of 13.8 points, ranging from 28.8 points for GPT-5.5 to 6.0 points for GLM-5.2.The paper links larger gains to using extra reasoning to plan retrieval, verify intermediate results, and reformulate unsuccessful queries; some models already retrieve efficiently or fail to turn extra deliberation into better decisions.Source: §5 · pp. 8–9

6 FURTHER ANALYSIS

Further analysis links failures to premature stopping rather than insufficient effort, finds retrieval dominated by simple substring queries, and shows that models differ in capability profiles and retrieval strategies.

6 FURTHER ANALYSIS

  • All evaluated models made fewer retrievals on failed tasks, by 1.0 for GLM-5.2 and 3.8 for Kimi K2.6.Models often stopped after finding a plausible record without retrieving further evidence to verify it or distinguish competing candidates.Source: §6 · p. 9
  • 98.5% of 126,279 retrieval calls were plain substring queries; regular expressions and fuzzy matching together were under 2%, and 9.5% restricted fields.Basic substring matching can miss records with noisy values, lexical variations, or partial matches.Source: §6 · p. 9
  • Gemini 3.1 Pro exceeded 73% on reasoning, disambiguation, and integration but scored 20% on preference inference; GPT-5.5 was strongest on preference inference and multi-intent decomposition.Gemini 3.1 Pro and Claude Opus 4.8 averaged 7.6 retrievals per task, versus 11.2 for GPT-5.5; the former achieved comparable performance with fewer retrievals, while GPT-5.5 had the highest overall accuracy through more comprehensive retrieval.Source: §6 · pp. 9–10

7 CONCLUSION

SPIEval benchmarks LLMs as mobile assistants that use scattered personal information, and its evaluation finds that current models struggle, especially with information localization.

7 CONCLUSION

  • SPIEval covers five cognitive capabilities and 4,335 personal records collected from 10 commonly used apps.It evaluates LLMs as mobile assistants completing tasks by leveraging scattered personal information.Source: §7 · p. 10
  • Nine representative LLMs struggle in this setting, with information localization emerging as the primary bottleneck.The evaluation also identifies systematic limitations in retrieval behavior, reasoning strategies, and capability utilization.Source: §7 · p. 10
  • Further analyses reveal systematic limitations in retrieval behavior, reasoning strategies, and capability utilization.These analyses provide a deeper understanding of current LLM-based mobile assistants.Source: §7 · p. 10

A USER PROFILE AND SYSTEM PROMPT

The section describes a system prompt that gives the model a unified user profile and directs it to retrieve information from phone records before acting. It also specifies retrieval and execution tools and search procedures.

A USER PROFILE AND SYSTEM PROMPT

  • A unified user profile is included in the system prompt to establish a consistent identity across all tasks.Table 3 presents the profile.Source: A USER PROFILE AND SYSTEM PROMPT · p. 15, Table 3 · p. 15
  • The prompt instructs the assistant to paginate results, try another keyword or app when a search fails, parallelize independent searches, and provide complete execution parameters.After completing operations, it instructs the assistant to reply with a completion message.Source: A USER PROFILE AND SYSTEM PROMPT · p. 15
  • The prompt directs the assistant to retrieve needed information from phone applications before invoking tools, separating retrieval tools from execution tools.Retrieval includes app-specific tools and one global tool; execution tools perform requested actions.Source: A USER PROFILE AND SYSTEM PROMPT · pp. 15–16

B APPLICATION SCHEMAS

SPIEval models 10 simulated apps with structured schemas abstracted from real-world counterparts, averaging 8.1 domain-specific fields per app. The schemas distinguish required from optional fields.

B APPLICATION SCHEMAS

C TOOL SCHEMAS

This section specifies the schemas for retrieval and execution tools across mobile apps, including global phone search, record creation or actions, transportation requests, and phone calls.

C TOOL SCHEMAS

  • 21 tools comprise 11 retrieval and 10 execution tools; each app has one of each, plus a global retrieval tool for cross-app access.Their specification documents are presented in Figures 10 and 11.Source: C TOOL SCHEMAS · p. 18
  • Retrieval schemas expose query and field parameters and support fixed (default), regex, or fuzzy matching, with case handling and pagination options.The accommodation search schema illustrates these options; the section also names app-specific retrieval tools such as contact, meeting, notes, schedule, and SMS search.Source: C TOOL SCHEMAS · p. 18
  • search_phone searches records across all apps by substring across fields, allows an optional field filter, returns five results per page, and supports pagination.Source: C TOOL SCHEMAS · p. 26
  • Creation schemas cover accommodation, alarms, contacts, meetings, and to-do items.Source: C TOOL SCHEMAS · pp. 27, 28, 29, 32
  • Additional schemas support creating schedules, sending text messages, and recording payments to others.Source: C TOOL SCHEMAS · pp. 33–34
  • Transport_Create requires date, time, start, end, passenger, telephone, and transportType inputs, with transportType limited to car, train, or airplane.Source: C TOOL SCHEMAS · pp. 35–36
  • Departure dates use YYYY-MM-DD, while times use HH:MM:SS or “now” for an immediate departure.Source: C TOOL SCHEMAS · p. 35
  • Phone_Call takes a phone number and supports voice or video calls, defaulting to voice.Source: C TOOL SCHEMAS · p. 36

D COMPARISON WITH EXISTING BENCHMARKS

This section compares SPIEval with existing benchmarks to illustrate its strengths, including five characteristics: diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes.

  • SPIEval is compared with existing benchmarks to illustrate its strengths, with results summarized in Table 5.Source: D COMPARISON WITH EXISTING BENCHMARKS · p. 38, Table 5 · p. 38
  • Table 5 reports task counts as question–answer instances for dialogue- and document-based benchmarks, while #Apps is not applicable to them.Source: Table 5 · p. 38
  • Table 5 uses checkmarks and crosses to indicate whether each benchmark-level property is present or absent.Source: Table 5 · p. 38
Loading 2608.10692v1…