Source-linked AI summary
360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents
Kenta Watanabe, Atsuyuki Miyai, Mizuki Takenawa, Kiyoharu Aizawa, Toshihiko Yamasaki
TL;DR
Existing urban-navigation benchmarks do not adequately capture the photorealism, complexity, and wide-area continuity of real city environments. 360CityArena addresses this gap with a photorealistic Akihabara benchmark and shows that state-of-the-art LMM-based agents remain far below humans across tasks.
Problem
Existing benchmarks lack sufficient photorealism, complexity, or continuous wide-area navigation for evaluating embodied agents in realistic urban spaces.
Method
360CityArena evaluates embodied agents on 175 tasks across seven types and three categories in an interconnected 360°-video reconstruction of Tokyo’s Akihabara district.
Results
State-of-the-art LMM-based agents perform far below human level across all benchmark tasks, with especially poor performance in map navigation, object counting, and relational spatial reasoning.
Takeaways & Limitations
360CityArena provides a challenging benchmark for realistic urban navigation and spatial reasoning, supporting further development of embodied agents applicable to real urban settings.
Takeaways & Limitations
The benchmark covers only one urban district, limiting urban diversity and conclusions about cross-regional generalization.
Abstract
from arXiv · showhide
We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment constructed from 360-degree videos. Existing outdoor benchmarks either lack sufficient photorealism or complexity, resulting in a considerable gap from real-world urban environments. 360CityArena is built on a realistic reconstruction of the Akihabara district in Tokyo, Japan, using 602 360-degree video segments covering 85 streets, and consists of 175 meticulously human-crafted tasks. It encompasses three task categories: Environment Understanding, Path Reasoning, and Spatial Reasoning, covering fundamental abilities required for urban exploration, such as localization, landmark search, path planning, and relational spatial reasoning, thereby enabling comprehensive evaluation in realistic urban scenes. Our evaluation using state-of-the-art LMM-based agents shows that even the strongest model, Gemini 2.5 Flash, performs far below human level (human: 77.3% vs. Gemini 2.5 Flash: 17.1%), revealing substantial challenges that remain in city-scale embodied navigation and reasoning. 360CityArena provides a necessary and challenging testbed for photorealistic urban-district navigation and spatial reasoning.
1 Introduction
360CityArena addresses the shortage of realistic, complex benchmarks for wide-area urban navigation by reconstructing Tokyo’s Akihabara district from interconnected 360° video trajectories. It evaluates embodied agents across environment understanding, path reasoning, and spatial reasoning, revealing substantial gaps from human performance.
- Motivation: Existing 3D simulators lack sufficient photorealism and complexity, while Google Street View environments lack dynamic elements and continuous, fully navigable spaces.These limitations motivate a benchmark designed for realistic wide-area navigation in real urban spaces.
- Benchmark proposal: 360CityArena reconstructs Tokyo’s Akihabara district as a photorealistic urban environment with 175 manually created tasks across 85 interconnected streets.The environment spans approximately 750 meters north–south and 650 meters east–west.
- Task design: The benchmark contains seven task types organized into Environment Understanding, Path Reasoning, and Spatial Reasoning for comprehensive urban exploration evaluation.These categories cover visual recognition and localization, route planning and decision-making, and geospatial reasoning through navigation.
- Evaluation and analysis: State-of-the-art LMM-based agents remain far below human performance across all tasks, with especially pronounced challenges in Map Navigation, Object Count, and Relational Spatial Reasoning.The study also analyzes input modality, self-location information, and model- and task-specific failure patterns.
2 Related Work
Prior embodied-navigation research spans robots, indoor and outdoor simulators, and several goal formulations, but existing evaluations do not jointly capture egocentric navigation and urban reasoning in realistic street environments. 360CityArena addresses this gap with a city-scale exploration benchmark built from realistic filmed trajectories, while retaining limitations from prerecorded video.
- Embodied Agent: Embodied-agent research spans real robots, indoor simulators, and outdoor simulators, with LMM advances strengthening multimodal perception and reasoning.Real robots offer greater realism, but data collection is difficult.
- Navigation Agents in Outdoor Environments: Outdoor navigation commonly specifies goals as point-goal coordinates, reference images, or semantic objects.These formulations emphasize reaching targets through visual or semantic cues.
- Navigation Agents in Outdoor Environments: VLN adds higher-level reasoning for linguistic ambiguity and landmark references beyond efficient target reaching.This distinguishes language-guided navigation from point-, image-, and object-goal tasks.
- Navigation Agents in Outdoor Environments: Existing aerial-VLN datasets and map-only evaluations do not jointly capture egocentric navigation and urban reasoning in realistic street environments.The benchmark is introduced to address this missing combination.
- Real-world Simulation Environments: Neural Radiance Fields and Gaussian Splatting have advanced 3D scene reconstruction, but most resulting environments remain static rather than fully interactive.Prior real-world simulation settings also include Google Street View–based environments and 3D simulators.
- Real-world Simulation Environments: 360CityArena supports continuous, smooth navigation within filmed trajectories, but trajectory boundaries create discontinuities and prerecorded video prevents physical interaction.These properties motivate interpreting the benchmark as realistic city-scale exploration without full environmental interactivity.
3 The 360CityArena
360CityArena is a realistic urban-district benchmark reconstructed from Akihabara and designed to evaluate embodied agents across 175 tasks in three major categories. Its pose-graph environment, human-constructed difficulty levels, and multiple evaluation protocols support assessment of perception, navigation, and spatial reasoning.
- Benchmark Overview: 360CityArena contains 175 tasks organized into three major categories and seven subcategories for realistic urban exploration evaluation.The benchmark is built on a Realistic Virtual World reconstructed from Akihabara, Japan.
- Virtual Environment: 602 360° video segments spanning 85 streets form a navigable Akihabara pose graph with 193 nodes and 305 edges.Agents traverse prerecorded 360° video trajectories in Unity rather than moving freely through arbitrary 3D positions.
- Task Categories: The benchmark evaluates Environment Understanding, Path Reasoning, and Spatial Reasoning, covering localization, landmark search, navigation, and relational reasoning.Spatial Reasoning includes Relational Spatial Reasoning and Object Count, while Environment Understanding includes localization and landmark-search tasks.
- Task Categories: Each of the seven subcategories contains 25 tasks, producing a consistent task allocation across the benchmark.The categories and subcategories are illustrated in Figure 2 and described separately in the benchmark section.
- Task Construction: Tasks are labeled Easy, Medium, or Hard using distance, instruction ambiguity, landmark visibility, and required exploration, with statistics increasing monotonically across levels.Map Navigation path lengths are 124/247/347 m, decision points are 3.3/6.4/9.0, Object Count values are 3.5/4.6/9.0, and VLN instruction steps are 4.4/5.9/7.4.
- Evaluation Protocol: Four evaluation protocols assess different aspects of task success, including exact textual or numerical matching and coordinate-based final-position accuracy.coordinate_match applies to Map Navigation, VLN, and Landmark Search with Image / Language, using Euclidean distance to the target.
4 Navigation Agents
The navigation-agent framework models environment interaction as a partially observable sequential decision process with deterministic action-conditioned transitions. Agents operate through seven discrete viewpoint, movement, reset, and answer actions, and experiments evaluate six LMM baselines spanning closed- and open-source models.
- Agent-Environment Formulation: Navigation is modeled as a partially observable sequential decision process with states, actions, observations, and deterministic transitions conditioned on actions.At each timestep, the agent occupies a state such as a position and viewing direction and receives a partial observation.
- Baseline Agents: Experiments evaluate six LMM baselines: GPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen2.5-VL-32B-Instruct, InternVL3.5-8B, and InternVL3.5-38B.GPT-5, Claude Sonnet 4.5, and Gemini 2.5 Flash are closed-source; the remaining models are open-source.
- Action Space: The base action space contains seven discrete actions for forward movement, viewpoint tilting, horizontal rotation, heading reset, and answer output.At branching points, movement actions are augmented with available traversable directions.
5 Experiment
Experiments show that current LMMs remain far below local-expert human performance on realistic urban navigation tasks, with accuracy generally declining as difficulty increases. Analyses further reveal task- and model-specific effects from visual inputs and distinct failure patterns in exploration, control, and perception.
- Human Performance: Humans reached around 90% accuracy on Map Navigation, Landmark Search with Image, Vision-Language Navigation, and Relational Spatial Reasoning.Human accuracy was lower on Localization, Landmark Search with Language, and Object Count, at 68%, 64%, and 45%, respectively.
- Input Effects: Landmark Search with Image generally outperforms Landmark Search with Language, including GPT-5 at 48.0 vs. 16.0 and Claude at 16.0 vs. 4.0.The passage attributes this pattern to concrete visual cues such as appearance, textures, and surrounding context, but reports no clear image-input gain for InternVL.
- Difficulty Effects: GPT-5’s accuracy in Environment Understanding drops from 28.0 →25.0 →18.5, and in Path Reasoning from 11.1 →0.0 →0.0 across Easy, Medium, and Hard settings.Model performance generally declines as task difficulty increases.
- Failure Analysis: GPT-5 shows minimal Action failures at 12% in Environment Understanding but dominant Explore failures at 55%, whereas Gemini 2.5 Flash and InternVL3.5-38B have around 40% Action failures.Perception failures surge in Spatial Reasoning, indicating distinct failure patterns across models.
6 Limitations and Future Work
360CityArena restricts agents to captured 360° trajectories, limiting free exploration and introducing possible boundary-transition discontinuities. Its focus on a single urban district also limits urban diversity and conclusions about cross-regional generalization, motivating future expansion.
- Restricted Exploration and Limited Interaction: Agents can explore only along captured trajectories rather than move freely to arbitrary locations.This restriction follows from constructing 360CityArena from pre-recorded 360° videos.
- Restricted Exploration and Limited Interaction: 11.3% of all actions were boundary-transition actions, which may introduce discontinuities absent from fully interactive 3D simulators or the real world.These actions were not overrepresented among Action or Ex...
- Limited Urban Diversity: The benchmark is restricted to a single urban district, limiting urban diversity and conclusions regarding cross-regional generalization.Cross-regional generalization is identified as an important direction for future work.
7 Conclusions
360CityArena evaluates embodied agents’ urban exploration in a photorealistic environment built from 360-degree videos. Results show that state-of-the-art LMM-based agents still struggle with understanding and reasoning in city environments, motivating benchmarks closer to real-world conditions.
- Contributions: 360CityArena is a benchmark for evaluating embodied agents’ urban exploration capabilities.It uses a photorealistic environment constructed from 360-degree videos.
- Findings: State-of-the-art LMM-based agents still face significant challenges in understanding and reasoning within city environments.
- Implications: A benchmark closer to real-world conditions could advance embodied agents applicable to real environments.
A Experiment Details · A.1 Task Settings
Each task defines a starting point and direction, with task-specific prompts, variables, and reference images. The settings evaluate complementary embodied-navigation abilities, including localization, landmark grounding, instruction following, and spatial reasoning.
- A.1 Task Settings: Each task specifies a starting point for navigation.
- A.1 Task Settings: Each task also specifies an initial direction.
- A.1 Task Settings: Task prompts are defined separately for each task type.
- A.1 Task Settings: The task settings specify variables for each task.
- A.1 Task Settings: Reference images are specified for each task.
- A.1 Task Settings: The tasks evaluate localization as one aspect of embodied navigation.
- A.1 Task Settings: The tasks evaluate landmark grounding and instruction following.
- A.1 Task Settings: The tasks also evaluate spatial reasoning.
A.2 Map Information · A.3 Stopping Conditions
The benchmark maps are produced by cropping the exploration area from OpenStreetMap. Experiments terminate under step-limit, stagnation, or away-from-goal conditions based on action counts, repeated actions, and consecutive movements.
- A.2 Map Information: Maps are created by cropping the exploration area from OpenStreetMap.
- A.3 Stopping Conditions: Experiments introduce predefined stopping conditions for the navigation process.
- A.3 Stopping Conditions: STEP_LIMIT terminates the process when the number of steps exceeds 50, with one action counted as one step.
- A.3 Stopping Conditions: STAGNATION terminates the process after the agent repeats the same action 20 times consecutively without changing position.
- A.3 Stopping Conditions: AWAY_FROM_GOAL terminates the process when the agent moves away from the goal location for five consecutive movements.
- A.3 Stopping Conditions: Viewpoint changes are excluded when determining whether the agent has moved away from the goal location.
A.4 Model Settings
The evaluation used model-specific temperature settings, a maximum output length of 8,096 tokens, and multimodal inputs combining visual, spatial, memory, and task-specific information.
- Model Settings: Claude Sonnet 4.5 and Qwen2.5-VL-32B-Instruct used temperature 1.0, whereas GPT-5 did not allow temperature configuration.The settings differed by model because GPT-5 lacked configurable temperature.
- Model Settings: The maximum output length was set to 8,096 tokens.This cap applied to the evaluated models’ outputs.
- Model Settings: Each model input included a viewpoint image, current-location map, Reflection Memory, and task-specific reference image.The Reflection Memory was updated at every step and included in the model input.
B Prompts for 360CityArena Experiments
The experiments use benchmark-specific prompts that combine ReACT navigation, visual and map observations, persistent reflection memory, and task-specific instructions. These prompts define action constraints, output formats, and procedures for localization, landmark search, and map navigation.
- B.1 System Prompt: The system prompt defines a ReACT loop in which the agent thinks, selects an action, and receives the resulting observation.Available actions include movement, camera rotation, direction alignment, and answering the task.
- B.1 System Prompt: Navigation actions are constrained by visible red arrows, including selecting UP to proceed straight when a large forward arrow appears.The agent cannot select LEFT, RIGHT, or UP, or move in an unavailable direction, without a visible red arrow.
- B.1 System Prompt: The agent receives first-person camera and optional top-down map images, which should be used together to understand location and plan routes.The map shows the current position and direction with a red arrow.
- B.2 Reflection Prompt: The reflection prompt maintains long-term consistency by updating text-based memory after each action for use in the next reasoning step.Agents are instructed to preserve prior reflections and record goals, strategies, visited locations, and city status.
- B.3 Task Prompts: Localization prompts require exploring the surroundings and selecting the single starting-position cell from a reference grid map.The final response must use ANSWER with discrete coordinates in the format "x:[grid_x] y:[grid_y]" and must be completed within 50 steps.
- B.3 Task Prompts: Landmark-search prompts ask agents to reach a specified landmark identified either by language or by a task-specific reference image, then confirm completion with ANSWER.Both variants state that the goal is not far from the starting point.
- B.3 Task Prompts: Map-navigation prompts provide a task-specific map and require navigating from the starting position to the goal destination using available actions.The blue marker denotes the start, the red marker denotes the goal, and ANSWER confirms arrival in the goal area.