Source-linked AI summary
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
Zhiheng Song, Jingshuai Zhang, Chuan Qin, Chao Wang, Chao Chen, Longfei Xu, Kaikui Liu, Xiangxiang Chu, Hengshu Zhu
TL;DR
MobilityBench addresses limited reproducibility and coverage in evaluating route-planning agents under diverse real-world mobility demands. It builds a large anonymized-query benchmark with deterministic API replay and multi-dimensional assessment, finding stronger performance on basic retrieval and route planning than on preference-constrained planning.
Problem
Real-world route-planning agents are difficult to evaluate systematically because mobility demands are diverse, instructions require nuanced reasoning, and live mapping services are non-deterministic.
Method
MobilityBench combines anonymized real mobility queries, structured ground truth, deterministic API replay, and evaluation of outcome validity, instruction understanding, planning, tool use, and efficiency.
Results
The evaluated agents perform competently on Basic Information Retrieval and Basic Route Planning but struggle considerably with Preference-Constrained Route Planning.
Takeaways & Limitations
MobilityBench provides a reproducible basis for comparing route-planning agents across diverse real-world mobility scenarios and highlights persistent difficulty with personalized routing.
Takeaways & Limitations
Episodes must be fully solvable from the initial query because agents are not permitted to ask users for clarification.
Abstract
from arXiv · showhide
Route-planning agents powered by large language models (LLMs) have emerged as a promising paradigm for supporting everyday human mobility through natural language interaction and tool-mediated decision making. However, systematic evaluation in real-world mobility settings is hindered by diverse routing demands, non-deterministic mapping services, and limited reproducibility. In this study, we introduce MobilityBench, a scalable benchmark for evaluating LLM-based route-planning agents in real-world mobility scenarios. MobilityBench is constructed from large-scale, anonymized real user queries collected from Amap and covers a broad spectrum of route-planning intents across multiple cities worldwide. To enable reproducible, end-to-end evaluation, we design a deterministic API-replay sandbox that eliminates environmental variance from live services. We further propose a multi-dimensional evaluation protocol centered on outcome validity, complemented by assessments of instruction understanding, planning, tool use, and efficiency. Using MobilityBench, we evaluate multiple LLM-based route-planning agents across diverse real-world mobility scenarios and provide an in-depth analysis of their behaviors and performance. Our findings reveal that current models perform competently on Basic information retrieval and Route Planning tasks, yet struggle considerably with Preference-Constrained Route Planning, underscoring significant room for improvement in personalized mobility applications. We publicly release the benchmark data, evaluation toolkit, and documentation at https://github.com/AMAP-ML/MobilityBench.
1 Introduction
MobilityBench addresses the difficulty of evaluating route-planning agents under diverse real-world mobility demands and non-deterministic mapping services. It combines broad real-user scenarios with deterministic replay and multi-dimensional evaluation.
- Tool-augmented LLM agents combine natural-language reasoning with executable actions through external APIs, supporting tasks such as route planning.
- Real-world mobility requests involve interacting preferences, waypoints, modality conditions, and time requirements that demand accurate interpretation, API use, and executable itineraries.
- Existing benchmarks emphasize high-level itinerary generation and abstract constraint reasoning, missing fine-grained reasoning over large-scale map environments and changing conditions.
- MobilityBench uses large-scale anonymized Amap queries spanning point-to-point, multi-waypoint, multimodal, preference-aware, and mobility-information tasks across over 350 cities worldwide.
- A deterministic API-replay sandbox caches routing and points-of-interest responses, removing live-service variance from end-to-end evaluation.
- The evaluation protocol centers on outcome validity and adds assessments of instruction understanding, planning, tool use, and efficiency.
2 Related Work
Route planning has evolved from shortest-path optimization toward preference-aware and language-mediated agent evaluation. Existing approaches and benchmarks remain limited in handling ambiguous natural-language mobility needs and realistic tool interactions.
- Classical shortest-path methods optimize distance or travel time on graph-based road networks but typically assume homogeneous objectives and well-defined costs.
- Preference-aware routing incorporates user interests and context, yet structured features and predefined preference spaces limit coverage of ambiguous or long-tail requirements.
- ToolBench evaluates sequential planning over real-world APIs, while τ-bench emphasizes interactive user-agent consistency across repeated trials.
3 MobilityBench
MobilityBench evaluates route-planning agents through realistic mobility episodes, structured ground truth, deterministic API replay, and multiple performance dimensions.
- Each benchmark episode represents a self-contained real-world mobility request that agents solve through tool augmentation.
- Structured ground truth records the minimal tool interactions and intermediate evidence needed to resolve each request, providing a stable evaluation reference.
- All tool interactions execute within a deterministic replay sandbox, enabling reproducible assessment across agent runs.
- The multi-dimensional protocol uses the structured ground truth to evaluate agent performance across complementary capabilities.
3.1 Benchmark Construction
MobilityBench organizes executable mobility requests into a taxonomy of scenarios, constructs tool-grounded references, and evaluates them against replayable API states. Its filtered corpus contains 100,000 checkable episodes across 22 countries and over 350 cities.
- Episode-centric Formulation: Each episode is a four-tuple containing an anonymized query, contextual information, a replayable API snapshot, and hidden structured ground truth.
- Episode-centric Formulation: Agents cannot request clarification, so every episode must be solvable from the initial user query alone.
- Query Collection: The benchmark uses anonymized AMap mobility queries, including voice-based expressions of real user intent from driving and walking contexts.
- Task Taxonomy: Expert-adjudicated intent classification yields 11 task scenarios grouped into four families: Basic Information Retrieval, Route-Dependent Information Retrieval, Basic Route Planning, and Preference-Constrained Route Planning.
- Ground-Truth Construction: Ground-truth construction defines minimal scenario-specific tool programs for slot extraction, location resolution, downstream API calls, and constraint verification.
- Dataset Statistics: 100,000 filtered episodes span 22 countries and over 350 cities, with 42.5% Basic Route Planning and 11.3% Preference-Constrained Route Planning tasks.
3.2 Evaluation Protocol
The evaluation protocol decomposes route-planning agents into measurable capabilities and evaluates both behavioral quality and computational efficiency. It uses fine-grained indicators to diagnose failures that end-to-end success rates can obscure.
- The protocol evaluates Instruction Understanding, Planning, Tool Use, Decision Making, and efficiency through multiple fine-grained indicators.These dimensions target distinct stages of route-planning behavior and computational overhead.
- Instruction Understanding: Instruction Understanding measures intent detection and exact extraction of spatial, temporal, and preference constraints from user queries.Intent labels are compared with ground truth using a similarity threshold, while extracted constraint sets must exactly match.
- Planning: Planning assesses whether agents decompose high-level routing goals into coherent sequences of atomic actions without omissions or redundancy.Task decomposition jointly considers step coverage and step correctness by matching predicted and ground-truth action sequences.
- Tool Use: Tool Use evaluates tool selection, schema compliance, and parameter filling at the interface between agents and the sandbox environment.Tool selection considers necessary-tool coverage and unnecessary-call redundancy, while schema compliance checks mandatory parameters and valid formats or ranges.
- Decision Making: Decision Making measures whether agents deliver complete executable outputs and whether those solutions are correct.Delivery Rate captures uninterrupted completion of the end-to-end pipeline, while Final Pass Rate evaluates the generated solution.
- Efficiency: Input Token measures cumulative processed context, characterizing computational overhead and practical deployability.The measure includes system prompts, task instructions, and historical observations and actions.
4 Experiments
The experiments use a stratified 100,000-episode benchmark and compare diverse LLM backbones within representative agent frameworks under unified evaluation settings. Model performance is summarized across instruction understanding, decision making, tool use, planning, and efficiency metrics.
- Models: The evaluated backbones include dense models, Mixture-of-Experts architectures, and reasoning-oriented models from open- and closed-source families.The suite spans Qwen and DeepSeek open-source models alongside closed-source backbones.
- Agent Frameworks: Agents are constructed with ReAct and Plan-and-Execute, two representative frameworks for mainstream agent reasoning pipelines.Alternative frameworks such as LLM Compiler, LATS, and Tree-of-Thought are not incorporated.
- Experimental Settings: A unified set of evaluation settings is applied across all LLM backbones and agent frameworks to support reproducibility and fair comparison.The settings standardize inputs, tool-use representations, and model configuration.
- Evaluation Metrics: Table 2 reports Instruction Understanding, Decision Making, planning, Tool Use, Delivery Rate, Final Pass Rate, Input Token, and Output Token metrics.The table defines abbreviations including ID, IE, DEC, TS, SC, DR, FPR, IT, and OT.
4.2 Experimental Results
Experiments compare models, execution frameworks, reasoning modes, and task families, revealing trade-offs between success, robustness, efficiency, and scenario difficulty.
- 83.53% Delivery Rate and 65.77% Final Pass Rate made Claude-Opus-4.5 the strongest Plan-and-Execute performer.
- 69.09% Final Pass Rate made Gemini-3-Pro-Preview the top ReAct performer.
- Framework Comparison: ReAct generally achieves higher Final Pass Rates than Plan-and-Execute because closed-loop feedback supports dynamic strategy adjustment, but increases API costs and inference time.Plan-and-Execute’s static pre-planning is less robust to dynamic mobile feedback.
- Scenario Study: Preference-Constrained Planning is the most error-prone task category, while Plan-and-Execute performs best there according to the scenario analysis.The framework’s advance strategy makes structured, logically ordered tasks more predictable and efficient.
- Scaling Effect: Scaling from 4B to 32B increases average success rate by 0.91%, while scaling Qwen’s MoE model from 30B-A3B to 235B-A22B adds 5.43%.Larger models also produce longer trajectories that explore more possible outcomes.
- Thinking vs. Non-thinking: 70.46% Final Pass Rate made DeepSeek-R1 a competitive reference, while enabling Thinking increased Qwen-30B-A3B’s rate by 5.98% but raised cost and latency.The additional token volume complicates real-time, production-grade deployment.
5 Conclusion
MobilityBench enables systematic, reproducible evaluation of route-planning agents under realistic mobility demands and supports comparisons across models and frameworks.
- MobilityBench uses anonymized real user queries and a deterministic API-replay sandbox to capture realistic mobility complexity reproducibly.
- Its evaluation protocol centers on outcome validity and also assesses instruction understanding, planning, tool use, and efficiency.
- The benchmark provides an extensible foundation for advancing route-planning-agent research and enabling fair, reproducible comparisons.
A.1 MobilityBench Task Scenarios
The benchmark’s taxonomy and examples are designed to expose the breadth of mobility-related language and task scenarios.
- Table S1 presents a detailed taxonomy with fine-grained scenario subtypes, definitions, and representative user queries.
- The representative examples reflect the diverse natural-language expressions users employ for mobility instructions.
A.2 Sandbox Tools
MobilityBench documents its sandbox tools and task-scenario coverage to support realistic, reproducible route-planning evaluation.
- Table S2 documents each sandbox tool’s name, input arguments, and output fields.
- The sandbox tools are sourced from the AMap Open Platform, with detailed parameter and response-field definitions available in official documentation.
- Table S1 provides concise definitions and representative user queries for MobilityBench task scenarios.