PaperlayerView exact revision on arXiv

Source-linked AI summary

UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang

arXiv:2608.27456v2cs.CV

TL;DR

Reliable urban action requires local evidence to remain useful as an agent moves and its view changes. UrbanGround evaluates MLLM agents in a real-scale Hong Kong environment across local question answering, navigation, and changing conditions. Agents show useful local skills, but these do not reliably compose into sustained goal-directed behavior.

  • Problem

    It remains unclear whether strong recognition from individual urban views stays coherent enough to guide reliable action as an agent moves and its evidence changes.

    Source: §1 · p. 2

  • Method

    UrbanGround is a georegistered Hong Kong replica for closed-loop MLLM evaluation, with tasks probing grounding, navigation, and adaptation through first-person or map observations and simulator actions.

    Source: §1 · p. 2, §3.1 · p. 4, §5 · p. 14

  • Results

    Current agents show useful local spatial skills, but these fail to compose reliably into sustained exploration, with errors accumulating without effective correction across longer routes and changing conditions.

    Source: RQ2 How does local grounding scale… · p. 3, RQ3 How do changes in the city affect… · p. 3, §5 · p. 15

  • Takeaways & Limitations

    City-scale agency depends on maintaining a spatial estimate beyond the current view and revising it when new evidence makes earlier plans infeasible.

    Source: §5 · p. 15

    Experiments take place in the simulator and do not involve real-world deployment.

    Source: ETHICS STATEMENT · p. 15

Abstract

from arXiv · show

Multimodal large language models (MLLMs) can interpret a street view, but reliable urban action depends on whether such local evidence remains useful after the agent starts to move. In this paper, we investigate how far current MLLM agents can turn local urban perception into reliable action in a real-scale city. We propose UrbanGround, an urban sandbox built from Hong Kong's territory-wide 3D geospatial data. It combines the city's geographic structure with continuous, collision-constrained control through a shared evaluation interface. Agents use first-person observations and an interactive map to select actions across tasks ranging from local question answering to long-horizon navigation. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an agent can gather and interpret local visual evidence to answer spatial questions. Then we ask whether these abilities support navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. MLLM agents usually show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far MLLM agents can explore reliably in open-ended urban environments.

1 INTRODUCTION

The paper asks whether local urban perception can support reliable action as agents move through a real-scale city, and introduces UrbanGround to study that transition. Its evaluation follows local grounding, longer-range navigation, and adaptation to city changes.

1 INTRODUCTION

  • UrbanGround turns Hong Kong’s complex geographic structure into an interactive environment for evaluating MLLM agents across local grounding, navigation, and changing-city conditions.The city’s elevated passages and steep terrain can make an apparently direct approach inaccessible, requiring agents to combine observations with map context.Source: §1 · pp. 2–3

RQ1 Can MLLM agents establish a usable local spatial grounding?

  • Local question answering shows useful spatial skills, but correct answers do not guarantee physically valid movement on the pedestrian network.Agents sometimes leave the network even after reaching the answer.Source: RQ1 Can MLLM agents establish a usable… · p. 3

RQ2 How does local grounding scale into goal-directed navigation?

  • Navigation performance collapses across only a few city blocks, as agents may identify the global direction yet fail to find a feasible local path or recover when blocked.Blocked agents often repeat the same action without recovery.Source: RQ2 How does local grounding scale… · p. 3

RQ3 How do changes in the city affect grounded perception and action?

  • Reduced visibility mainly weakens local question answering, while route changes often leave agents moving plausibly on the pedestrian network despite obsolete plans.The effect of reduced visibility on short-range navigation is less consistent across models.Source: RQ3 How do changes in the city affect… · p. 3

2 RELATED WORK

Prior work studies agent perception and action in games, physical environments, and urban settings, but each setting leaves aspects of sustained spatial agency unresolved. The paper frames the remaining gap around maintaining a spatial account as observation and movement unfold in a real city.

Agent Evaluation in Game Environments

  • Game benchmarks test visual decisions and longer trajectories, but game-defined spatial structure does not establish whether agents maintain a valid spatial frame beyond those environments.These benchmarks include controlled visual-spatial tasks and persistent worlds for extended exploration (Paglieri et al., 2025; Zheng et al., 2026; Hu et al., 2026; Zhang et al., 2025a; Cai et al., 2025; Liu et al., 2025a; Ju et al., 2026; Park et al., 2026; Tan et al., 2025b;a; Luo et al., 2026).Source: §2 · pp. 3–4

Agent Evaluation in Physical-World Environments

  • Physical-world benchmarks ground behavior in scene geometry, but bounded environments limit how far local observations must relate to a larger spatial frame.They cover indoor embodied question answering, navigation, and open-vocabulary target search (Majumdar et al., 2024; Yang et al., 2025; Qiao et al., 2025; Zhao et al., 2026; Su et al., 2026; Zhang et al., 2024; Cheng et al., 2024; Zhang et al., 2026a;b; Li et al., 2025; Lin et al., 2025).Source: §2 · p. 4

Urban Evaluation

  • Urban evaluations extend spatial scale, but recorded views lack continuous movement through the same geometry, while interactive simulators do not preserve a metropolis’s full georegistered structure.This leaves open whether local multimodal competence composes into spatial agency under consistent observation and movement (Yang et al., 2024; Schumann et al., 2024; Zhang et al., 2025b; Dalal et al., 2026; Wang et al., 2026; Liu et al., 2025b; Mei et al., 2026; Wu et al., 2025; Shang et al., 2024; Zhuang et al., 2025; Zhou et al., 2026).Source: §2 · p. 4

3 URBANGROUND: A FRAMEWORK FOR EVALUATING SPATIAL AGENCY

URBANGROUND is a Unity-based, physically constrained city framework for evaluating whether spatial evidence remains useful as MLLM agents move. Its shared geographic setting and interface support tasks that progress from local understanding to navigation and adaptation.

3 URBANGROUND: A FRAMEWORK FOR EVALUATING SPATIAL AGENCY

  • URBANGROUND turns territory-scale geospatial data into a physically constrained city for evaluating MLLM agents.The shared world and interface are fixed across evaluated models.Source: §3 · p. 4, §3.1 · p. 5

3.1 SPATIAL AGENCY IN CLOSED-LOOP URBAN INTERACTION

  • Spatial agency comprises grounding local relations, persisting them after views change, and adapting when conditions invalidate earlier assumptions.The evaluation links these capabilities to local understanding, navigation, and robustness under changing visible conditions.Source: §3.1 · p. 4
  • A closed loop updates task-relevant spatial state from observations, prior actions, and the goal, then uses that state to choose the next action.Executed actions update the physical or interface state and determine the next evidence available to the agent.Source: §3.1 · pp. 4–5

3.2 GEOSPATIAL LAYER

  • The geospatial layer combines a territory-wide textured city mesh with a georeferenced pedestrian network describing connected pedestrian spaces.The agent moves continuously rather than being restricted to predefined network edges.Source: §3.2 · pp. 5–6

3.3 SIMULATION LAYER

  • The simulation layer supports continuous motion constrained by city geometry, with configurable time, weather, and pedestrian activity.Pedestrian motion is recorded, while the agent remains free to move outside registered network edges.Source: §3.3 · p. 6

3.4 AGENT LAYER

  • Agents receive first-person RGB observations and can inspect a georeferenced map without a computed or highlighted route.Structured movement, look, jump, map, and termination actions connect the external MLLM to the Unity environment.Source: §3.4 · p. 7

3.5 SPATIAL AGENCY EVALUATION LADDER

  • The five-level ladder progresses from local understanding to explicit and implicit navigation, multi-task planning, and dynamic adaptation.Matched time and weather variations alter visual evidence at the same task locations.Source: §3.5 · pp. 7–8

3.6 HUMAN ANNOTATION

  • Annotators manually construct and verify each task instance, confirming it is completable within 150 steps and checking it again before inclusion.Tasks cover major Hong Kong regions with varied street patterns, terrain, and vertical pedestrian connections.Source: §3.6 · p. 8

4 EXPERIMENTS

Across local grounding, navigation, and dynamic-city tests, MLLMs show useful local abilities but do not reliably sustain goal-directed behavior as routes lengthen or conditions change. The experiments also find weak directional grounding, uneven evidence gathering, and limited adaptation to closures and pedestrians.

4.1 EXPERIMENTAL SETUP

  • The study evaluates contemporary MLLMs through a fixed interaction protocol on 810 manually verified Hong Kong instances, measuring answer accuracy, navigation success, and movement constraints.Episodes use first-person RGB observations, task instructions, and interaction history, with a 150-step limit.Source: §4.1 · pp. 8–9

4.2 RQ1: CAN MLLM AGENTS ESTABLISH A USABLE LOCAL SPATIAL GROUNDING?

  • Local visual recognition and short-range reasoning are useful, but active-exploration accuracy ranges from 46.3% to 82.5% and directional awareness remains unreliable.Movement sometimes leaves the registered pedestrian network, as agents favor shorter paths to evidence.Source: §4.2 · pp. 9–10

4.3 RQ2: HOW DOES LOCAL GROUNDING SCALE INTO GOAL-DIRECTED NAVIGATION?

  • Short routes with visible destinations can succeed, but navigation largely breaks down across several streets as agents repeat blocked actions or lose progress.Success generally declines across increasing distance bins in both ShortNav and InstructionNav; long-range episodes may still reduce goal distance, while multi-stop agents reach 6.8–20.7% of required destinations on average.Source: §4.3 · pp. 11–12, Figure 8 · p. 12
  • Navigation also degrades when destinations must be inferred from instructions, even at distances comparable to short-range tasks.Success is concentrated in shorter-distance bins, and the widening model differences suggest that closed-loop interaction exposes instability not revealed by local question answering.Source: §4.3 · p. 11, Figure 7 · p. 12

4.4 RQ3: HOW DO CHANGES IN THE CITY AFFECT GROUNDED PERCEPTION AND ACTION?

  • Dusk and night can weaken local question answering, while weather and time affect short-range navigation less consistently across models.GPT-5.6-Luna and Gemini-3.8-Flash are comparatively stable, whereas GPT-5.2 and Gemini-3.1-Pro vary more with lower visibility.Source: §4.4 · pp. 12–13, Table 3 · p. 14
  • Under route closures, closure compliance remains low despite high pedestrian-network adherence, and pedestrian-collision rates are high across models.The measures expose weak route revision and local motion adaptation in changed scenes.Source: §4.4 · pp. 13–14, Table 4 · p. 14
  • Across spatial scales, models recognize local evidence more reliably than they orient it, complete routes, or update plans after the city changes.Plausible local movements can continue after the grounded state no longer guides action toward the objective.Source: §4.4 · p. 14

5 CONCLUSION

The paper introduces UrbanGround to examine whether MLLMs can carry local spatial understanding into sustained action in a real-scale city. Results show useful atomic abilities, but local skills often fail to compose into extended exploration and adaptation.

5 CONCLUSION

  • UrbanGround is a georegistered replica of Hong Kong for studying closed-loop interaction under physical change.The study progresses from active question answering to longer and less explicit navigation, then to adaptation when the city changes.Source: §5 · pp. 14–15
  • Current agents show useful visual-recognition and short-range spatial skills, but errors accumulate during longer exploration without effective correction.Changes in visibility affect question answering, while closures and moving pedestrians expose weak online updating and constraint adherence.Source: §5 · p. 15
  • City-scale agency requires maintaining a spatial estimate after the view changes and revising it when earlier plans cease to be feasible.The authors hope UrbanGround will support research on more adaptable movement through complex cities.Source: §5 · p. 15

AI USE STATEMENT

The authors disclose using AI assistants to polish the manuscript and MLLM APIs in their experiments, with no other AI use.

AI USE STATEMENT

ETHICS STATEMENT

The study uses a Hong Kong urban simulator built from publicly available geospatial data, with human annotation limited to benchmark construction and verification. The authors foresee no ethical concerns because experiments remain within the simulator and do not involve real-world deployment.

ETHICS STATEMENT

  • Experiments take place in an urban simulator built from publicly available Hong Kong geospatial data.Source: ETHICS STATEMENT · p. 15
  • Human annotation is limited to constructing and verifying benchmark tasks.Source: ETHICS STATEMENT · p. 15
  • The authors foresee no ethical concerns in this paper because the experiments remain in simulation and do not involve real-world deployment.Source: ETHICS STATEMENT · p. 15

REPRODUCIBILITY STATEMENT

The statement directs readers to the paper’s setup, task-construction, prompt, and complete-results details. The urban-environment comparison defines the capabilities used to describe benchmark settings.

REPRODUCIBILITY STATEMENT

  • The setup and evaluation metrics are in Section 4, framework implementation in Section 3, task construction and human verification in Appendix B and Section 3.6, and prompts and complete results in Appendices F and C.These references form the reproducibility statement’s pointers to supporting material.Source: REPRODUCIBILITY STATEMENT · p. 15

A COMPARISON WITH URBAN ENVIRONMENTS

  • Table 5 compares urban environments and navigation benchmarks, marking documented support and absence in the described settings.The supplied caption does not provide the table’s individual benchmark entries.Source: Table 5 · p. 22, A COMPARISON WITH URBAN ENVIRONMENTS · p. 22
  • The comparison defines real-city geography, continuous motion, collision response, pedestrian networks, and locomotion-primitive control as distinct benchmark characteristics.A pedestrian network is a georeferenced 3D graph of real-world pedestrian paths and their connectivity across elevation levels.Source: A COMPARISON WITH URBAN ENVIRONMENTS · p. 22

B EXPERIMENTAL TASK DESIGN

The task design progresses from local spatial understanding to explicit-route navigation and then to exploration when instructions leave the destination unspecified. Instances are manually constructed, verified, and distributed across Hong Kong regions.

B EXPERIMENTAL TASK DESIGN

  • Each task instance is manually constructed, tested from its specified initial state with the provided tools and constraints, re-verified, and distributed across Hong Kong regions.The sequence moves from immediate surroundings to long-range objectives, uncertain destinations, temporal constraints, and environmental changes.Source: B EXPERIMENTAL TASK DESIGN · p. 22

B.1 LEVEL 1: LOCAL ENVIRONMENT UNDERSTANDING

  • Level 1 tests local scene recognition, orientation, and active evidence gathering, with exploration questions designed for an effective strategy within 60 seconds.Active exploration requires choosing informative viewpoints and integrating observations into one answer; orientation requires matching heading, scene layout, and referenced locations.Source: §B.1 · pp. 22–23

B.2 LEVEL 2: NAVIGATION UNDER EXPLICIT INSTRUCTIONS

  • Level 2 varies distance, route representation, and constraints while keeping the goal explicit, from visible short-range destinations to extended routes and sequential instructions.Long-range navigation requires map-based routing and sustained localization; constrained navigation adds restrictions on usable pedestrian areas.Source: §B.2 · p. 23

B.3 LEVEL 3: EXPLORATION UNDER IMPLICIT INSTRUCTIONS

  • Level 3 removes direct target specification, requiring the agent to find a place matching a category or infer a concrete destination from an intended outcome.Place-type search couples semantic search with spatial exploration.Source: §B.3 · p. 23

B.4 LEVEL 4: MULTI-TASK PLANNING

Level 4 tests coherent planning across multiple destinations, while Level 5 tests navigation after environmental changes during execution.

B.4 LEVEL 4: MULTI-TASK PLANNING

  • Level 4 shifts the challenge from reaching one endpoint to maintaining a plan across multiple visits.Time-window scheduling combines route choice with temporal feasibility, while multi-stop planning requires choosing and executing an order for destinations.Source: §B.4 · pp. 23–24

B.5 LEVEL 5: DYNAMIC ENVIRONMENT INTERACTION

  • Level 5 introduces changes during validated navigation instances without changing the original objective.For road closures, the agent receives a one-time notice and must revise a route that may no longer be executable.Source: §B.5 · p. 24
  • Navigation among pedestrians requires preserving progress toward the destination while adjusting motion to avoid contact with moving pedestrians.The pedestrians are introduced into validated long-range navigation instances.Source: §B.5 · p. 24

C COMPLETE EXPERIMENTAL RESULTS

The appendix provides complete task-level results as percentages, with separate tables for the five-level evaluation ladder and pedestrian-network adherence.

C COMPLETE EXPERIMENTAL RESULTS

  • The appendix tables retain every evaluated model and report task-level results as percentages.This describes the reporting scope, not the numerical outcomes.Source: C COMPLETE EXPERIMENTAL RESULTS · p. 24
  • Table 6 reports complete answer-accuracy and task-success results across the five-level evaluation ladder.The main text presents compact tables organized by the three research questions; the supplied caption gives no individual result values.Source: Table 6 · p. 24, C COMPLETE EXPERIMENTAL RESULTS · p. 24
  • Table 7 reports complete pedestrian-network-adherence results as percentages of executed action duration.The supplied passage does not provide individual values.Source: Table 7 · p. 25

D MATCHED RETROSPECTIVE CHECKPOINT ANALYSIS

The matched retrospective-checkpoint analysis tests whether endpoint regression contributes to GPT-5.5’s LongNav outcomes. Oracle rescoring recovers partial progress but does not improve arrival.

D MATCHED RETROSPECTIVE CHECKPOINT ANALYSIS

E LONGNAV FAILURE-TYPE ANALYSIS

The analysis divides GPT-5.5’s LongNav runs into four mutually exclusive trajectory categories. Unstable progress is the most prominent behavior, while stopping short and retaining progress until timeout point to other difficulties.

E LONGNAV FAILURE-TYPE ANALYSIS

  • Four mutually exclusive categories classify GPT-5.5’s LongNav runs by stopping, progress, and arrival outcomes.Substantial progress means at least a 20% reduction in horizontal goal distance; the final recorded checkpoint defines the endpoint.Source: E LONGNAV FAILURE-TYPE ANALYSIS · p. 25
  • Unstable progress is the most prominent LongNav failure behavior: the model approaches the goal and later regresses.The paper connects this pattern to weaknesses in maintaining global orientation, preserving spatial state, and recovering after a wrong turn.Source: E LONGNAV FAILURE-TYPE ANALYSIS · p. 25
  • Stopping outside the arrival radius suggests poorly calibrated localization or arrival verification, while progress retained at timeout suggests inefficient execution or unresolved access constraints.Source: E LONGNAV FAILURE-TYPE ANALYSIS · p. 25

F PROMPTS USED IN THE EXPERIMENTS

This appendix documents the prompts and interaction protocols that guide agents across local question-answering and navigation tasks, specifying task goals, available actions and information, constraints, response formats, and success criteria.

F PROMPTS USED IN THE EXPERIMENTS

F.1 SHARED AGENT INTERACTION PROTOCOL

  • The shared action space covers first-person movement and camera controls, map selection and navigation controls, and termination, with movement limited to two seconds per action.Source: §F.1 · pp. 26–27
  • Both task types use a visual ReAct loop and JSON actions, but question answering ends with a final multiple-choice response whereas navigation scores the current state at termination.Source: §F.1 · pp. 27–28

F.2 RQ1 / LEVEL 1: LOCAL ENVIRONMENT UNDERSTANDING

  • VR prompts direct landmark inspection, OU prompts preserve the original orientation reference, and AEQ prompts systematic local search for evidence that may not be initially visible.Source: §F.2 · pp. 29–30

F.3 RQ2 / LEVEL 2: NAVIGATION UNDER EXPLICIT INSTRUCTIONS

  • SGN asks agents to reach a visible nearby destination, while LGN requires deriving a route from map markers and first-person observations without computed guidance.Source: §F.3 · pp. 30–31
  • Instructional Navigation requires executing turn-by-turn directions by identifying referenced junctions, crossings, and landmarks from screenshots while relying on route memory.Source: §F.3 · p. 31
  • Constrained Navigation requires planning around closed road segments and never crossing a closure line, which immediately fails the task.Source: §F.3 · p. 31
  • Level 2 requires reaching a stated destination without crossing any listed road closures, with the map opened before departure to plan around them.The prompt says closures persist for the whole episode and provides no hidden coordinates, distance readout, or automatic crossing warning.Source: §F.3 · p. 32

F.4 RQ2 / LEVEL 3: EXPLORATION UNDER IMPLICIT INSTRUCTIONS

  • Place-Type Search asks agents to locate a nearby requested place, verify it using map evidence and street-level signs or storefronts, then stop at its entrance or signage.Success depends on whether the final position counts as arrival; hidden coordinates, distance readouts, and search results are unavailable.Source: §F.4 · p. 32
  • Implicit Intent Inference requires inferring a destination category from an everyday goal, finding a suitable nearby POI through visible signs and map evidence, and navigating there.The prompt recommends pedestrian infrastructure and remembering searched streets so exploration covers new nearby areas.Source: §F.4 · p. 33

F.5 RQ2 / LEVEL 4: MULTI-TASK PLANNING

  • Time-window Scheduling requires visiting appointments in order and before their deadlines; arriving late or visiting out of order fails, and agents receive current-time and schedule-status updates.Prompts direct agents to plan efficiently, avoid unnecessary detours, and use the time remaining to decide when to hurry.Source: §F.5 · pp. 33–34
  • Multi-stop Route Planning requires visiting several destinations in a self-chosen order, using the map to minimize backtracking; success reflects destinations reached and route efficiency.There is no time schedule or required visiting order, and the task ends when every destination is visited or turns run out.Source: §F.5 · p. 34

F.6 RQ3 / LEVEL 5: DYNAMIC ENVIRONMENT INTERACTION

  • Dynamic Road-closure Replanning introduces a closure after navigation begins; once notified, crossing its waypoint-connected line in either direction immediately fails, though earlier crossings do not.Agents are instructed to inspect the map and replan around the closure after receiving the notice.Source: §F.6 · p. 35
  • Navigation among Pedestrians uses the Long-range Goal Navigation prompt: agents must determine their own route to a far, likely unseen destination from first-person observations and a map that shows no route.They receive no updated coordinates, distance readout, or route guidance, and should prefer pedestrian infrastructure.Source: §F.6 · p. 36

Related papers

Loading 2608.27456v2…