PaperlayerView exact revision on arXiv

Source-linked AI summary

CityCube: Benchmarking Cross-view Spatial Reasoning on Vision-Language Models in Urban Environments

Haotian Xu, Yue Hu, Zhengqiu Zhu, Chen Gao, Ziyou Wang, Junreng Rao, Wenhao Lu, Weishi Li, Quanjun Yin, Yong Li

arXiv:2601.14339v1cs.CVcs.AI

TL;DR

Comprehensive evaluations of VLMs’ ability to perceive and understand multiple views in expansive urban spaces are lacking, despite this being a requirement for embodied agents. CityCube benchmarks cross-view urban spatial reasoning using urban views from real-world datasets and realistic simulators, with 5,022 human-verified multiple-choice questions across 59 tasks and five cognitive categories. The best-performing proprietary model reaches only 54.1% accuracy, 34.2% below human performance, while fine-tuned CityBot models outperform leading proprietary models.

  • Problem

    Comprehensive evaluations of VLMs’ ability to perceive and understand multiple views in expansive urban spaces are lacking, despite this being a requirement for embodied agents.

    Source: §1 · p. 1

  • Method

    CityCube benchmarks cross-view urban spatial reasoning using urban views from real-world datasets and realistic simulators, with 5,022 human-verified multiple-choice questions across 59 tasks and five cognitive categories.

    Source: §1 · p. 1, §6 · p. 8

  • Results

    The best-performing proprietary model reaches only 54.1% accuracy, 34.2% below human performance, while fine-tuned CityBot models outperform leading proprietary models.

    Source: §5.2 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    CityCube offers a benchmark and diagnostic tool for studying difficult urban cross-view reasoning, and its training set shows potential for improving spatial reasoning through fine-tuning.

    Source: §1 · p. 2, §6 · p. 8

    The study does not evaluate Sim-to-Real transfer or domain gaps in its simulated urban environment.

    Source: §7 · p. 9

Abstract

from arXiv · show

Cross-view spatial reasoning is essential for embodied AI, underpinning spatial understanding, mental simulation and planning in complex environments. Existing benchmarks primarily emphasize indoor or street settings, overlooking the unique challenges of open-ended urban spaces characterized by rich semantics, complex geometries, and view variations. To address this, we introduce CityCube, a systematic benchmark designed to probe cross-view reasoning capabilities of current VLMs in urban settings. CityCube integrates four viewpoint dynamics to mimic camera movements and spans a wide spectrum of perspectives from multiple platforms, e.g., vehicles, drones and satellites. For a comprehensive assessment, it features 5,022 meticulously annotated multi-view QA pairs categorized into five cognitive dimensions and three spatial relation expressions. A comprehensive evaluation of 33 VLMs reveals a significant performance disparity with humans: even large-scale models struggle to exceed 54.1% accuracy, remaining 34.2% below human performance. By contrast, small-scale fine-tuned VLMs achieve over 60.0% accuracy, highlighting the necessity of our benchmark. Further analyses indicate the task correlations and fundamental cognitive disparity between VLMs and human-like reasoning.

1 Introduction

CityCube addresses the limited evaluation of VLMs’ cross-view spatial reasoning in open urban environments, where rich semantics, complex geometry, and viewpoint variation pose challenges. It introduces a multi-view benchmark and reports substantial gaps between current VLMs and human reasoning.

Motivation

  • CityCube is motivated by a gap in comprehensive evaluations of VLMs’ capabilities in expansive urban environments.The introduction notes that embodied agents such as drones, mobile robots, and autonomous vehicles need to perceive and understand multiple views.Source: §1 · p. 1
  • Urban open spaces bring dense, repetitive objects that demand spatial grounding and disambiguation from VLMs.The introduction also identifies intricate 3D structures, road networks, and cross-scale viewpoint variation as challenges.Source: §1 · p. 1

Benchmark

  • CityCube evaluates 5,022 QA pairs across five spatial-intelligence dimensions, using multi-view urban scenes.The dimensions include Spatial Relations, Perspective Taking, Mental Reconstruction, World Knowledge, and Comprehensive Reasoning.Source: §1 · p. 2

Findings

  • The authors report that fine-tuning Qwen3-VL at multiple scales on CityCube training data shows potential to enhance spatial reasoning.They use parameter-efficient LoRA and produce CityBot-2B, 4B, and 8B.Source: §1 · p. 2

Dataset

  • The dataset contains 18.1K images from diverse urban perspectives, and rearranges imagery into four observation behaviors that mimic camera movements.The benchmark draws on multi-view imagery captured from heterogeneous platforms and spans first-person and aerial perspectives.Source: §1 · p. 2

Findings

  • Leading proprietary and open-source VLMs achieve below 54.1% accuracy on CityCube, and analyses find a disparity with human-like reasoning.The introduction also reports performance gaps across models and persistent discrepancies with human reasoning.Source: §1 · p. 2

2 Related Work

Related work covers spatial-intelligence benchmarks and urban visual question answering. It describes a gap in evaluating cross-view reasoning across urban perspectives and embodied contexts.

VLM Spatial Intelligence Benchmark

  • Existing spatial-intelligence benchmarks often overlook urban environments’ cross-view nature, while several urban embodied benchmarks focus on single or restricted perspectives.The cited examples include bird’s-eye-view settings.Source: §2 · p. 2

Urban Visual Question Answering

  • Urban VQA spans macro-level queries and analyses as well as micro-level agentic perception, but current benchmarks have limitations in evaluating multi-image understanding for embodied cross-view reasoning.The cited tasks include urban governance, socioeconomic analysis, object localization, and motion planning.Source: §2 · p. 3

VLM Spatial Intelligence Benchmark

  • The related-work discussion presents CityCube as integrating multi-source imagery and multi-centric perspectives with a broad set of task types and annotated reasoning processes.Table 1 compares CityCube with popular multi-view spatial-intelligence benchmarks.Source: §2 · p. 3, Table 1 · p. 3

3 CityCube Benchmark

CityCube organizes urban multi-view QA around viewpoint dynamics, camera perspectives, cognitive task categories, and spatial-reference expressions. Together, these dimensions define the visual inputs and QA structure for diagnosing cross-view spatial reasoning.

3.1 Overview

  • The benchmark draws on 18.1K observation points and curates about 5.0K QA pairs, with four dimensions jointly configuring multi-image QA instances.The first two dimensions determine visual perspective and content; the latter two are expressed in the QA text.Source: §3.1 · p. 3

3.2 Dimension 1: Viewpoint Dynamics

  • Four behavior primitives model viewpoint dynamics: rotation, orbit, ego-allocentric view, and dynamic translation across urban scales.They represent camera-orientation changes, movement around a region, alignment between first- and third-person views, and coarse-to-fine movement.Source: §3.2 · p. 3

3.3 Dimension 2: View Perspectives

  • CityCube includes ground-level panoramas, low-altitude oblique imagery, and bird’s-eye views, spanning vehicle, drone, and satellite-like perspectives.Figure 2 identifies camera orientations as the focus of Dimension 2.Source: §3.3 · pp. 3–4, Figure 2 · p. 4

3.4 Dimension 3: Benchmark Tasks

  • Five task categories probe mental reconstruction, perspective taking, spatial relations, comprehensive reasoning, and world knowledge.They cover imagined movement between views, cross-view grounding, object relations, multi-step inference, and urban commonsense.Source: §3.4 · p. 4

3.5 Dimension 4: Spatial Relation Expressions

  • Three binary spatial relations define QA reference frames: object-to-object, camera-to-object, and camera-to-camera.The benchmark targets integration across views, geometric consistency under motion, reference-frame alignment, and identity preservation across scale transitions.Source: §3.5 · pp. 4–5

4 Benchmark Curation

Benchmark curation combines image processing with QA generation. The authors assemble corresponding real and simulated views, generate questions with structured context, and refine the resulting QA pairs through model filtering and human review.

4 Benchmark Curation

  • Benchmark curation has two main parts: image processing and QA generation.The section describes construction of the image dataset and benchmark introduced in Section 3.Source: §4 · p. 5

4.1 Data Collection and Pre-processing

  • Real-world and simulated views are organized into multi-view sets, with preprocessing intended to ensure that views correspond to the same scene.Real-world images undergo filtering and matching; simulator samples include camera poses and motion trajectories. Sources include nuScenes, GeoText-1652, EmbodiedCity, and MatrixCity.Source: §4.1 · p. 5

4.2 Question-Answer Generation

  • QA generation uses Gemini-2.5 Pro with role-playing, task templates, and geometric-reference context, followed by a two-stage refinement pipeline.Blind filtering scores questions without visual input to remove trivial samples and balance difficulty; human annotators verify and proofread accepted QA pairs.Source: §4.2 · p. 5
  • Human verification checks QA-pair rationality and authenticity, with proofreading to resolve ambiguities, invalid options, or erroneous reasoning.Human-authored reasoning processes are added to accepted questions.Source: §4.2 · p. 6

5 Experiments

Across 33 VLMs, CityCube exposes a large gap between current models and human performance, while fine-tuning on the benchmark improves results. Analyses find correlated task performance, distinct model failure modes, and little alignment between human and VLM task difficulty.

5.1 Evaluation Setups

  • Evaluation averages task-level and overall multiple-choice accuracy, with a separate group of ten urban-science graduate students providing the human baseline.The evaluated models include 25 baseline VLMs in addition to the fine-tuned models; the human evaluators did not overlap with the verification group.Source: §5.1 · p. 6

5.2 Main Results

  • 54.1% accuracy is the best proprietary-model result, 9.2% above the strongest open-source model but 34.2% below human performance.Table 2 reports accuracy for 33 VLMs; the prose highlights the leading proprietary-versus-open-source and model-versus-human comparisons.Source: §5.2 · p. 6, Table 2 · p. 7
  • Fine-tuning consistently improves performance across model scales; human-annotation reasoning guidance adds 0.6% to 3.6%, and the fine-tuned 2B model surpasses strong proprietary baselines.The findings identify gains from CityCube fine-tuning and further gains for CityBot-4B and -8B with CoT.Source: §5.2 · p. 6
  • Reasoning-oriented VLMs show no consistent advantage, general-purpose spatial models perform sub-optimally, and models do better on semantic WK and CR than geometric PT, SR, and MR tasks.The authors hypothesize that generic reasoning supervision is insufficient without explicit modeling of view-dependent geometry and spatial transformations.Source: §5.2 · p. 6

5.3 Task Correlation Analysis

  • Across 59 tasks, 81.4% of high-correlation pairs span categories; MR–PT correlation is highest across dimensions (r = 0.536), while Height Estimation is weakly coupled to most tasks.Examples of dense cross-category task correlations include Behind View–Left-turn Prediction (r = 0.954) and Hierarchical Structure–Object Localization Sequence (r = 0.965).Source: §5.3 · pp. 6–7

5.4 The VLMs Error Analysis

  • Four recurring failure modes are missing small urban entities, egocentric position and orientation errors, mismatched street-to-aerial correspondences, and difficulty interpreting camera motion and scale changes.The errors affect landmark recognition, relative-position prediction, cross-view spatial relationships, scene reconstruction, and navigation-related reasoning.Source: §5.4 · pp. 7–8

5.5 Human-AI Difficulty Bias Analysis

  • Across 59 subtasks, human and VLM accuracy correlate weakly (r = 0.098, p = 0.4608; R2 = 0.010), with VLM performance ranging from 19.2% to 91.8%.The analysis finds that tasks difficult for VLMs do not necessarily challenge humans, and vice versa.Source: §5.5 · p. 8, Figure 4 · p. 8

6 Conclusion

CityCube is a human-verified benchmark for evaluating cross-view spatial intelligence in urban environments. Evaluation of 33 VLMs finds that spatial reasoning remains challenging, with CityBot models outperforming leading proprietary models while remaining below human-level cognition.

6 Conclusion

  • CityCube contains 5,022 human-annotated and verified multiple-choice questions spanning 59 tasks across five cognitive categories.The benchmark uses images from dynamic viewpoints and diverse orientations to evaluate cross-view spatial intelligence in urban environments.Source: §6 · p. 8
  • The authors propose CityCube as a foundation for future spatially grounded learning studies and as a diagnostic tool for developing VLMs with stronger urban spatial intelligence.The paper presents these as intended uses of the benchmark.Source: §6 · pp. 8–9

7 Limitations

The authors identify limitations in perspective-bias analysis, simulated-environment coverage, interpretability, and validation of spatial post-training strategies.

7 Limitations

  • CityCube does not explicitly isolate viewpoint asymmetry or reference-frame ambiguity, and its simulated urban setting leaves Sim-to-Real transfer and domain gaps unevaluated.The authors identify these as limits on the benchmark’s coverage and evaluation scope.Source: §7 · p. 9
  • The analysis focuses on task-level performance, leaving internal representations and multi-view fusion mechanisms unexamined and model failure modes less interpretable.Source: §7 · p. 9
  • The study hypothesizes that explicit spatial supervision matters but does not implement or validate dedicated post-training strategies such as spatial Chain-of-Thought.Source: §7 · p. 9

8 Ethics Statement

The appendix records the paper’s ethics statement and documents how its urban multi-view data are visualized, sourced, sampled, and paired. Its acquisition procedures combine real and simulated imagery with dataset-specific methods for retaining diverse, interpretable views.

8 Ethics Statement

  • The study reports using public data, programs, and pretrained models; annotation had informed consent, followed ACL ethics guidelines, and raised no stated privacy or public-safety concerns.Source: §8 · p. 9

A Appendix

  • The appendix covers data collection and processing, benchmark generation and refinement, reproduction materials, VLM error analyses, experiment setup, and further discussion of related research.Source: A Appendix · p. 12

B.1 Dataset Visualization

  • The dataset visualization summarizes spatial-reference frames and task frequencies by observation behavior; its approximately 4.5k training and 0.5k test questions use stratified sampling to preserve task proportions.Source: §B.1 · p. 12, Figure 6 · p. 13

B.2 Details of Image Acquisition

  • Image sources include real-world nuScenes and GeoText-1652 data and virtual EmbodiedCity and MatrixCity imagery, combining ground-level, aerial, and first-person perspectives.Source: §B.2 · p. 12

B.3.1 nuScenes

  • For nuScenes, temporal subsampling followed by appearance-based deduplication reduces redundant driving frames, while retaining structured multi-view samples with scene and camera metadata.Source: §B.3.1 · pp. 13–14

B.3.2 Geotext-1652

  • GeoText-1652 sequences are sampled into orbit views for fine viewpoint variation and spiral views for altitude and scale changes, with per-subset caps to control redundancy.Source: §B.3.2 · p. 14

B.3.3 Cross-view Geometric Pairing in MatrixCity

  • MatrixCity’s precise camera poses support aerial–street matching through geometric constraints on distance, height, viewing overlap, orientation, and projected viewing rays.Source: §B.3.3 · pp. 14–15

B.4 Details of Manually Collected Data

The appendix details manual data collection, role and task templates, question filtering, and human refinement. These procedures define varied urban viewpoint sequences and retain questions judged both visually grounded and answerable.

B.5.1 Role Templates

  • Four role templates frame reasoning across scales, between egocentric and allocentric views, across surrounding viewpoints, or through rotation around a fixed location.Source: §B.5.1 · pp. 16–17

B.5.2 Task Templates

  • Task templates are presented sequentially for each camera dynamic because the tasks are specific to their viewpoint modes.Source: §B.5.2 · p. 17

B.6.1 Blind Filter

  • The blind filter removes questions all six tested open-source VLMs answer without images and questions all fail, retaining items requiring visual evidence that remain solvable by some models.Source: §B.6.1 · pp. 17–18

B.6.2 Details of Human Refinement

  • Ten trained annotators work in separate refinement and verification groups; each QA pair receives review by both groups, with disagreements adjudicated by verification.Ambiguous questions are revised or removed, and accepted pairs are proofread and receive human-authored reasoning.Source: §B.6.2 · p. 18

C Experimental Details

C.1 Brief Introduction on Baselines

The evaluation includes proprietary and open-source multi-image models alongside random and human baselines. The appendix illustrates errors involving small-object perception, spatial orientation, and consistency across views.

C.1 Brief Introduction on Baselines

  • The evaluated baselines include proprietary and open-source models trained for multi-image input, as well as random and human comparisons.Source: §C.1 · pp. 18, 21

D Detailed Error Analysis

D.1 Small Object Perception Error

  • In the small-object case, Seed 1.6 misses a low-salience paved area and selects a more prominent but incorrect shelter.Source: §D.1 · p. 22

D.2 Spatial Reasoning Error

  • In the spatial-reasoning case, Gemini 2.5 Pro misorients views when inferring what lies behind an aerial camera position.Source: §D.2 · p. 22

D.3 Cross-view Consistency Error

  • In the cross-view consistency case, Qwen3-VL guesses a building’s aerial shape from its street view instead of grounding the answer in the aerial image.Source: §D.3 · p. 22

D.4 Dynamic Understanding Error

The dynamic-understanding example links a mistaken motion-direction interpretation to a reversed judgment about changing perspective. The accompanying implementation note specifies the training–test split and data-loading setup.

D.4 Dynamic Understanding Error

  • Gemini 2.5 Pro predicts buildings will appear closer rather than further apart, misreading the camera’s motion direction between views.Source: §D.4 · pp. 22–23

E Implementation Details

E.1 Training Details

The appendix specifies CityBot fine-tuning settings, describes volunteer evaluation, and positions CityCube as extending multi-view 3DQA into urban environments.

E.1 Training Details

  • CityBot models use supervised fine-tuning with LoRA, while the vision encoder and vision-language alignment module remain unfrozen for end-to-end adaptation.LoRA uses rank 8 and scaling factor 32; training also uses padding-free training and sample packing.Source: §E.1 · p. 23
  • Training runs for 5 epochs with Adam, a 1 × 10−4 initial learning rate, and a 0.05 warmup ratio.The setup also uses bfloat16, four-step gradient accumulation, FlashAttention, a 4096-token input limit, and images capped at 1.6M pixels.Source: §E.1 · p. 23, Table 3 · p. 23

E.2 Human Evaluation Details

  • Volunteers evaluated CityCube through an interactive web interface and were recruited independently from the question-refinement annotators.They received no additional spatial-knowledge training; Figure 21 demonstrates the multi-choice QA test system.Source: §E.2 · p. 24, Figure 21 · p. 25

F Further Discussion

Loading 2601.14339v1…