Source-linked AI summary
LifePlanner: Evaluating LLM Agents for Geo-spatial Planning with Social Media Data
Zhen Dong, Yuning Peng, Yutao Shi, Lei Zhong, Yongsen Mao, Yuan Liu, Haiping Wang
TL;DR
Existing geo-spatial planning benchmarks largely omit the noisy, open-ended social signals used in everyday planning. LifePlanner combines structured maps, large-scale local social-media evidence, MCP tools, and four task categories across three difficulty levels. Frontier models perform well on simple retrieval but degrade sharply on complex planning, with failures mainly involving evidence acquisition, tool use, and constraint integration.
Problem
Most geo-spatial planning benchmarks rely on clean structured data and tools while omitting open-ended social signals important to daily planning.
Method
LifePlanner augments map data with large-scale regional social-media evidence, MCP-based tools, and four planning categories organized into three difficulty levels.
Results
40.2% Pass Rate on L2 tasks demonstrates a sharp gap between simple retrieval and complex multi-constraint planning.
Takeaways & Limitations
Failures mainly arise from incomplete evidence acquisition, inefficient or imprecise tool use, and weak constraint integration rather than hallucination, model scaling, or longer reasoning.
Takeaways & Limitations
The benchmark uses a temporally concentrated social-media snapshot and synthetic, single-turn queries with constraints provided upfront.
Abstract
from arXiv · showhide
Geo-spatial planning, like trip design, is a realistic testbed for LLM agents because it requires grounded tool use, noisy evidence retrieval, and multi-constraint reasoning. Most benchmarks, however, only provide clean geospatial data and tools, missing the open-ended social signals that people use in daily planning. We introduce LifePlanner, a benchmark that enriches map data with large-scale local social media posts and provides access through an MCP toolset. LifePlanner provides an evaluation suite spanning four task categories and three difficulty levels. Experiments show frontier LLMs perform well on simple retrieval but degrade sharply on complex planning, with the Pass Rate dropping to 40.2%. Results show that failures mainly stem from incomplete evidence acquisition from such a large multimodal database, imprecise tool use, and weak constraint integration rather than model size or reasoning length, suggesting that future progress requires effective grounded planning instead of scaling alone.
1 Introduction
LifePlanner addresses gaps in geo-spatial planning benchmarks by combining noisy social-media evidence with structured maps, tool use, and multi-level evaluation. It tests agents across diverse planning tasks and finds sharp degradation as constrained reasoning becomes more complex.
- Geo-spatial planning evaluates long-context multimodal processing, grounded tool use, and constrained reasoning over spatial evidence.
- Existing benchmarks typically use clean geospatial databases and rarely incorporate open-ended social signals central to timely, locally grounded knowledge.
- LifePlanner augments geospatial information with large-scale regional social-media data and MCP tools for grounded, up-to-date spatial analysis.
- The benchmark spans Place Perception, Nearby Discovery, Routing, and Trip Design across L0 single-search, L1 multi-retrieval, and L2 constrained decision-making tasks.
- 40.2% Pass Rate on L2 tasks reflects failures in evidence acquisition, tool use, and constraint integration rather than model size or reasoning length.
2 Related Works
Prior geo-spatial benchmarks increasingly use structured data, retrieval, and dynamic tools but remain distant from the noisy social-media evidence used in everyday planning. LifePlanner closes this gap by combining cartographic platforms with socially driven preferences and high-noise information exploration.
- Prior work addresses geo-spatial reasoning through knowledge injection, multimodal inputs, retrieval-augmented generation, and dynamic tool invocation.
- LifePlanner expands environment-side information capacity by enabling agents to query maps and dynamically acquire real-time social-media insights.
- Existing benchmarks cover static retrieval, active data fetching, interactive preference inference, multi-constraint optimization, and real-world mobility planning.
- Unlike predominantly structured benchmarks, LifePlanner simulates noisy social-media exploration for preferences such as hidden gems and limited-time events.
3 LifePlanner
LifePlanner combines a social-media-enriched geospatial environment with a structured benchmark that evaluates tool use, evidence grounding, and increasingly complex planning. It measures both final plan quality and the intermediate processes used to produce plans across task categories and difficulty levels.
- Environment: LifePlanner combines an urban geospatial database with social-media evidence and MCP tools for grounded spatial analysis.The environment covers approximately 10 km^2, over 3,600 locations, and about 200,000 social-media notes and comments.
- Task Taxonomy: The benchmark covers Place Perception, Nearby Discovery, Routing, and Trip Design across progressively harder planning operations.L0 uses single-step lookup, L1 adds multi-step retrieval and computation, and L2 integrates implicit constraints, costs, and preferences; Trip Design begins at L1.
- Evaluation Protocol: Evaluation scores both final answers and tool-use trajectories using outcome metrics for attributes, place overlap, routing, order, semantic constraints, and overall task success.Pass Rate counts cases whose Semantic Constraint score exceeds α while the other calculable indicators equal 1.
- Evaluation Protocol: Process metrics assess whether agents acquire required information, use tools efficiently, and keep reasoning faithful to database evidence.Information Grounding measures coverage of required intermediate information, Tool Efficiency penalizes redundant calls, and Reasoning Purity evaluates faithfulness without hallucination or distortion.
4 Experiments
LifePlanner evaluates LLM agents across diverse geo-spatial planning tasks and process metrics, revealing sharp degradation as tasks become more complex. Failures reflect incomplete evidence acquisition, imprecise tool use, and weak integration of multiple constraints rather than hallucination alone.
- Experimental Setup: LifePlanner evaluates frontier and open-source LLMs under a shared agent scaffold across 11 tasks.Every model receives the same system prompt, MCP tools, output schema, and maximum budget.
- Difficulty: 89.4% to 40.2%: average Pass Rate drops from L0 to L2 while Token Usage rises from 24.6k to 251.1k.The increase reflects longer exploration for multi-hop evidence collection and integration of spatial, semantic, ordering, and implicit constraints.
- Task Differences: Over 90% Pass Rate remains achievable on L2 place-perception tasks, but strong models reach only about 25% on L2 trip design.Trip design requires simultaneous retrieval, constraint satisfaction, and correct tool use across intermediate steps.
- Model Comparisons: Model rankings are not monotonic with size: GPT-5.4 does not outperform Qwen3.5-27B or GLM-5.1.Within the Qwen3.5 family, however, Qwen3.5-27B substantially outperforms Qwen3.5-9B.
- Process Analysis: 86.4% Reasoning Purity contrasts with 64.6% Information Grounding and 44.6% Tool Efficiency on L2 tasks.The process analysis indicates that agents preserve retrieved facts relatively well but often fail to acquire sufficient evidence or formulate precise, nonredundant queries.
- Failure Analysis: Current agents often fail to integrate retrieved evidence with implicit or soft constraints when composing globally reasonable plans.Representative failures include repeated retrieval of irrelevant results and optimizing shortest distance while ignoring store closing-time order.
5 Conclusion
LifePlanner combines structured maps, large-scale social-media evidence, MCP tools, and multi-level planning tasks to evaluate geo-spatial agents. It exposes a sharp gap between simple retrieval and complex multi-constraint planning, with failures tied mainly to evidence acquisition, tool use, and constraint integration.
- LifePlanner combines structured map data, large-scale social-media evidence, MCP-based tools, and multi-level planning tasks.
- A sharp performance gap separates simple retrieval from complex multi-constraint planning.
- Failures mainly arise from incomplete evidence acquisition, inefficient tool usage, and weak constraint integration rather than hallucination.
- Model scaling and longer reasoning are insufficient without precise tool queries and constrained plan aggregation.
Limitations
LifePlanner has a temporally concentrated social-media snapshot and synthetic queries, limiting evaluation of temporal information conflicts and potentially making goals and constraints more complete than real-world requests.
- A single large-scale social-media collection creates a temporally concentrated snapshot.This limits evaluation of how agents handle information conflicts caused by temporal changes.
- Synthetic benchmark queries may state goals and constraints more completely than real-world requests.
A.1 Dataset Composition
The dataset section documents task composition, task-instance fields, field activation across task families and difficulty tiers, and the retrievable environment schema. It also distinguishes the exploration-oriented agent toolset from the more comprehensive data-generation toolset.
- Dataset Composition: Tables 4–6 describe task distributions, task-instance fields, and field triggers across task families and difficulty tiers.Figure 8 demonstrates a concrete task instance.
- Dataset Composition: The Standard Agent Toolset returns lightweight, granular initial results and requires drill-down calls for further insights.This is designed to simulate progressive human-like exploration through tools such as place_detail and note_detail.
- Dataset Composition: The Data Engine Agent Toolset returns comprehensive records upfront, including social memory, temporal attributes, and tags.It also provides specialized tools for random sampling and multi-stop permutations to support efficient dataset generation and complex constraints.
- Dataset Composition: The map and social-media database schemas list only attributes explicitly retrievable by agents through MCP tools.Internal routing geometries and social-media tokenized texts or vector embeddings are used for system operation but are not among the explicitly retrievable fields described.
B.1 Metric Applicability
The benchmark specifies which evaluation metrics apply to each task category and difficulty level, and it calibrates subjective scoring with human references. Three judge models with the closest overall agreement are selected for the final ensemble.
- Metric Applicability: Table 11 maps the applicability of each evaluation metric across task categories and difficulty levels.
- Metric Applicability: Human-calibrated screening selects challenging cases with the largest variance among preliminary scores for reference annotation.Flagship Claude, GPT, and Gemini models first score responses independently, after which human annotators assign reference scores using the same rubric and verification evidence.
- Metric Applicability: GPT-5.4, Claude Haiku 4.5, and GLM-5.1 show the closest agreement with human scores and form the final judge ensemble.Semantic Constraint Satisfaction and Reasoning Purity are averaged across these three judges.
B.3 Hyperparameters
The benchmark uses controlled evaluation settings, standardized agent scaffolding, and distinct toolsets for evaluated agents and data-engine-generation agents. Its data and evaluation components combine retrieved map and social-media fields with semantic, rule-based, and verification-based assessment.
- Hyperparameters: Evaluator and judge models use temperature 0, while the data engine agent uses temperature 0.7 to encourage generative diversity.
- Hyperparameters: All evaluated models share the same system prompt, MCP tool schemas, output requirements, and maximum budget of 30 tool invocations per task instance.
- Data construction: LLM-based data engines synthesize evaluation cases, candidate answers, and reference tool-use paths, with authors manually checking solvability, correctness, and answer uniqueness.
- Evaluation: Semantic Constraint Satisfaction and Reasoning Purity use LLM-as-a-judge models, combined with rule-based metrics and verification tools.
- Data and tools: The benchmark exposes retrievable fields from map and social-media databases, while its toolsets support lightweight evaluated-agent calls and denser data-engine retrieval.
Example for Task Instance
This task instance asks an agent to select socially grounded waypoints and plan a time-aware driving route from an office to Hongbo Gate. The example combines place identification, evidence references, opening-hour verification, and route optimization in a strict structured-output workflow.
- User request: The user must leave Wuhan Zhongke Kaiwu Technology Co., Ltd., stop at a short-legged-dog venue and a popular lakeside sunset spot, then reach Hongbo Gate by driving.
- Waypoint selection: Corgi & Shiba Inu Pet Cafe is selected as a waypoint because social media verifies its dog-related fit and 11:00–21:00 hours, with arrival around 11:06.
- Waypoint selection: Lingbo Gate Trestle is selected as the viewing waypoint because it is a popular spot where visitors can walk above the lake.
- Route output: The returned route orders the office, Corgi & Shiba Inu Pet Cafe, Lingbo Gate Trestle, and Hongbo Gate, using driving and time mode with total distance 8618 m.
- Agent constraints: The evaluated agent must ground reasoning in tool results, call tools before answering, cite social-media note_ids, preserve exact returned values, and submit one structured JSON object.
- Task category: The task is categorized as L2 mixed reasoning with dynamic waypoint selection, requiring tag or memory conditions followed by optimal route planning.