Source-linked AI summary
SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information
Junjie Ye, Zhuohui Sheng, Shaofan Liu, Yulun Zhu, Wenjie Fu, Dingwei Zhu, Ming Zhang, Yujiong Shen, Weichao Wang, Xin Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang, Pluto Zhou
TL;DR
Mobile assistants must retrieve and combine personal information scattered across apps, but existing evidence does not adequately measure this capability. SPIEval introduces a human-curated benchmark spanning five cognitive capabilities, and its evaluation shows that current models remain limited, with information localization as the primary bottleneck.
Problem
Existing mobile-assistant benchmarks do not adequately evaluate locating and using personal information scattered across multiple apps.
Method
SPIEval is a human-curated benchmark of 250 tasks covering five cognitive capabilities, 4,335 records across 10 apps, and 21 retrieval and execution tools.
Results
GPT-5.5 (xhigh) achieves 57.3% accuracy, while information localization emerges as the primary bottleneck across current models.
Takeaways & Limitations
SPIEval exposes systematic limitations in retrieval behavior, reasoning strategies, and capability utilization among current LLM-based mobile assistants.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly deployed as mobile assistants, where a key challenge is leveraging personal information scattered across multiple applications (apps) to complete user instructions. However, due to the lack of dedicated benchmarks, their capabilities remain poorly understood. To address this gap, we introduce SPIEval, a human-curated benchmark grounded in five cognitive capabilities (i.e., reasoning, disambiguation, integration, preference inference, and multi-intent decomposition). SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools. Analysis shows that the benchmark exhibits diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes. We evaluate nine representative LLMs and find substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%. Further analysis reveals that 79% of failures stem from inaccurate information localization, as LLMs often commit to plausible but incorrect information instead of continuing retrieval for verification. We also find that fewer than 2% of retrieval actions employ advanced search methods and observe substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research in this direction. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.
1 INTRODUCTION
Existing mobile-assistant benchmarks often provide needed parameters directly or limit personalization to individual documents, leaving scattered cross-app information use under-evaluated. SPIEval addresses this gap with a human-curated benchmark and finds substantial performance limitations, especially in locating personal information.
- Benchmark gap: Existing benchmarks primarily test tool use when required parameters are explicit or directly returned, rather than locating information across multiple apps.Personalization benchmarks generally retrieve from individual documents instead of requiring cross-app information localization.
- SPIEval: SPIEval contains 250 tasks covering five cognitive capabilities, grounded in 4,335 records across 10 apps and supported by 21 tools.The benchmark includes 11 retrieval tools and 10 execution tools for multi-turn interaction.
- SPIEval: SPIEval is designed for diverse, challenging, scattered-information scenarios with controllable environments and verifiable outcomes.
- Evaluation findings: 79% of failures arise from inaccurate information localization, as models commit to plausible but incorrect information instead of continuing retrieval for verification.Fewer than 2% of retrieval actions use advanced search methods, and search efficiency varies substantially across models.
- Evaluation findings: 57.3% accuracy is achieved by GPT-5.5 (xhigh), while the weakest evaluated model achieves 16.4%.
2 RELATED WORK
Prior work spans mobile task execution, cross-app coordination, and benchmarks for evaluating app-centric assistants. SPIEval organizes scattered-information evaluation around task-specific cognitive capabilities and structured app records and tools.
- Mobile assistants: Prior mobile-assistant research includes multi-step task execution, real-world natural-language completion, and cross-app coordination through multi-agent architectures.
- Task construction: SPIEval tasks include instructions, task-specific app records, and gold execution answers.
- SPIEval framework: SPIEval’s framework combines commonly used apps, retrieval and execution tools, task-specific records, and a unified user profile.
- SPIEval framework: The benchmark evaluates five cognitive capabilities for handling scattered personal information.
- Task construction: Benchmark quality control manually constructs and independently verifies the instruction, records, reasoning process, and gold answer with at least two annotators.
3 SPIEVAL
SPIEVAL formulates mobile-assistant tasks around retrieving, reasoning over, and executing with personal information scattered across apps. It constructs and quality-controls a benchmark spanning five cognitive capabilities, diverse operations, concise instructions, and multi-app records.
- Task Formulation: SPIEVAL models underspecified instructions as tool-call trajectories that retrieve records, infer parameters, and terminate with execution calls.The task formulation uses retrieval and execution tools, with tool feedback interleaved throughout the trajectory.
- Cognitive Capabilities: Five capabilities organize the benchmark: reasoning, disambiguation, integration, preference inference, and multi-intent decomposition.Examples include resolving a manager’s identity, selecting a valid phone number, aggregating supplier information, inferring calling preferences, and splitting independent subtasks.
- Benchmark Construction: SPIEVAL combines a unified user profile with 10 structured app simulations whose partially populated records can be linked across sources.The profile supports natural references such as family relationships, while shared fields allow information about an entity to be assembled across apps.
- Benchmark Construction: 21 tools support multi-turn interaction through 11 retrieval tools and complementary execution tools, including substring, regular-expression, and fuzzy retrieval modes.Retrieval tools also support field targeting and case-sensitivity control.
- Dataset Properties: 357 execution operations cover all 50 capability-operation pairs across five capabilities and ten operation categories.Even the least frequent operation category appears 27 times, reducing underrepresentation across operations.
- Dataset Properties: Instructions average 34 characters while requiring 8.47 execution parameters, and task-relevant records average 17.3 records across 6.2 of 10 apps.One 20-character instruction requires 45 parameters, while the most demanding tasks involve up to 39 records.
4 EXPERIMENTAL SETUP
The experiments evaluate nine representative LLMs with exact-match final-call accuracy under controlled interaction settings. Models are tested across reasoning-effort levels, paginated retrieval, and alternative settings summarized in Figure 5.
- Models: Nine representative state-of-the-art LLMs are evaluated at their highest and lowest available reasoning-effort levels.Other hyperparameters remain at default values to maximize each model’s performance.
- Metrics: Binary accuracy requires every final execution tool and parameter value to exactly match an annotated gold answer.Intermediate steps are not scored because models may adapt their behavior based on tool feedback.
- Implementation Details: Each task allows at most 50 interaction turns, and retrieval results are paginated with no more than five records per page.The protocol therefore requires models to request additional pages when necessary while limiting latency and computation.
- Controlled Settings: Figure 5 compares accuracy under the standard paginated protocol with two controlled settings, averaged across model configurations.The caption identifies the three evaluation settings but does not state their comparative outcomes.
5 MAIN RESULTS
SPIEval remains difficult for current LLMs, with performance varying across capabilities, models, and reasoning effort. The main bottleneck is locating relevant personal information rather than merely retrieving more records.
- Overall performance: 57.3% accuracy is achieved by GPT-5.5 (xhigh), while Kimi K2.6 (none) achieves 16.4%, demonstrating substantial difficulty.Removing retrieval increases average accuracy from 35.5% to 66.8%, whereas returning all matching records improves it only from 35.5% to 36.0%.
- Error sources: 79% of failures arise from incorrect parameter values, identifying information localization as the primary bottleneck.The no-search setting provides task-relevant records directly, isolating query formulation and target-record identification as major sources of difficulty.
- Capability differences: Average accuracy is around 46% for reasoning, disambiguation, and integration, but about half as high for preference inference and multi-intent decomposition.Preference inference reaches 67.0% without search, while multi-intent decomposition reaches only 45.7% even when all relevant records are directly available.
- Reasoning effort: 13.8 points is the average gain from increasing reasoning effort, with improvements ranging from 28.8 points for GPT-5.5 to 6.0 points for GLM-5.2.Additional reasoning can support retrieval planning, intermediate-result verification, and query reformulation.
6 FURTHER ANALYSIS
Further analysis shows that models often stop retrieval prematurely and rely overwhelmingly on basic search. Successful systems differ in how they balance targeted retrieval, comprehensive information gathering, and downstream reasoning.
- Retrieval behavior: Failed tasks receive fewer retrievals than successful tasks for every evaluated model, with differences ranging from 1.0 to 3.8 retrievals.Models often stop after finding a plausible record instead of retrieving further to verify it or distinguish competing candidates.
- Search methods: 98.5% of 126,279 retrieval calls use plain substring queries, while regular expressions and fuzzy matching together account for less than 2%.Only 9.5% of retrievals restrict searches to specific fields, despite advanced methods being available through the tools.
- Model strategies: Gemini 3.1 Pro and Claude Opus 4.8 average 7.6 retrievals per task, compared with 11.2 for GPT-5.5, yet achieve comparable performance.This pattern suggests that some models use more targeted retrieval strategies, whereas GPT-5.5 relies on more comprehensive retrieval.
7 CONCLUSION
SPIEval evaluates LLM mobile assistants on tasks requiring scattered personal information across apps. Results show that current models struggle primarily with information localization, alongside limitations in retrieval behavior, reasoning strategies, and capability utilization.
- Benchmark and evaluation: SPIEval covers five cognitive capabilities and 4,335 personal records from 10 apps while evaluating nine representative LLMs.The benchmark targets mobile-assistant tasks that require leveraging scattered personal information.
- Main conclusion: Information localization emerges as the primary bottleneck, with additional limitations in retrieval behavior, reasoning strategies, and capability utilization.These findings provide a deeper understanding of current LLM-based mobile assistants.
A USER PROFILE AND SYSTEM PROMPT
The experiments provide a unified user profile and system prompt, then require assistants to retrieve information across phone applications before executing requests. The prompt includes separate retrieval and execution tools plus instructions for iterative search and complete execution parameters.
- User profile: A unified user profile establishes a consistent identity across all tasks and is provided in the system prompt.Table 3 presents the profile, while Figure 9 shows the complete prompt template.
- Retrieval tools: The system provides app-specific and global search tools for retrieving records across phone applications.The prompt describes retrieval tools as including app-specific tools and one global retrieval tool.
- Execution tools: The system also provides execution tools for carrying out the requested actions after retrieval.The prompt separates retrieval tools from execution tools used to fulfill user requests.
- Search protocol: The prompt instructs models to search sufficiently, paginate results, vary keywords or apps, and include complete parameters in the final execution call.It also permits parallel independent searches and specifies a completion response after all operations finish.
B APPLICATION SCHEMAS
SPIEval models 10 simulated apps with structured, domain-specific schemas that distinguish required from optional fields. These schemas provide the application-level data structure summarized in Table 4.
- 10 simulated apps are represented by structured schemas containing domain-specific fields.Each app averages 8.1 fields.
- Each schema distinguishes between required and optional fields.
- Table 4 provides the complete schemas for the 10 simulated apps in SPIEval.
C TOOL SCHEMAS
SPIEval provides 21 tools for retrieving and executing actions across simulated apps. Retrieval specifications support app-specific and global search with configurable fields, matching modes, case handling, and pagination, while execution tools encode structured actions.
- 21 tools comprise 11 retrieval tools and 10 execution tools.Each app has one app-specific retrieval tool and one execution tool; a global retrieval tool supports cross-app access.
- Retrieval tools: Retrieval specifications include app-specific search tools for contacts, meetings, accommodation, notes, schedules, SMS, transactions, and transport.
- Retrieval tools: Search tools can specify fields and use fixed, regex, or fuzzy matching, with optional case-insensitive search and pagination.
- Retrieval tools: The global search_phone tool searches records across all applications and returns five results per page with pagination.
- Execution tools: Execution tools define structured operations such as creating accommodation records, alarms, and contacts with typed parameters and required fields.
D COMPARISON WITH EXISTING BENCHMARKS
SPIEval is designed around five benchmark-level characteristics and is compared with existing benchmarks using a systematic property-based table. The comparison distinguishes task and app coverage conventions across benchmark types.
- Five characteristics define SPIEval: diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes.
- Table 5 compares SPIEval with existing benchmarks.
- Table 5 uses ✓ and ✗ to indicate whether systematic benchmark-level properties are present or absent.For dialogue- and document-based benchmarks, #Tasks counts question–answer instances and #Apps is not applicable.