PaperlayerView exact revision on arXiv

Source-linked AI summary

CityNav: A Large-Scale Dataset for Real-World Aerial Navigation

Jungdae Lee, Taiki Miyanishi, Shuhei Kurita, Koya Sakamoto, Daichi Azuma, Yutaka Matsuo, Nakamasa Inoue

arXiv:2406.14240v3cs.CVcs.AI

TL;DR

Real-world aerial VLN lacks high-fidelity city datasets and faces difficulty integrating geographic and visual information. CityNav supplies a large real-world dataset and geographic semantic maps as an auxiliary navigation input; experiments report improved performance across three representative agents.

  • Problem

    Real-world city aerial VLN remains underexplored because high-fidelity 3D datasets are limited and geographic and visual information are difficult to integrate.

    Source: §1 · p. 2

  • Method

    CityNav provides 32,637 human demonstration trajectories across two real cities and introduces geographic semantic maps as an auxiliary modality for three aerial VLN agents.

    Source: §1 · p. 2

  • Results

    Geographic semantic maps significantly improve navigation performance for Seq2Seq, CMA, and AerialVLN models.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CityNav adds real-world landmark-to-destination spatial reasoning to aerial VLN and provides a benchmark for studying it.

    Source: §1 · p. 2, §3.4 · p. 5

    CityNav covers two real cities, but its geographic coverage remains limited compared with cities worldwide.

    Source: §6 · p. 8

Abstract

from arXiv · show

Vision-and-language navigation (VLN) aims to develop agents capable of navigating in realistic environments. While recent cross-modal training approaches have significantly improved navigation performance in both indoor and outdoor scenarios, aerial navigation over real-world cities remains underexplored primarily due to limited datasets and the difficulty of integrating visual and geographic information. To fill this gap, we introduce CityNav, the first large-scale real-world dataset for aerial VLN. Our dataset consists of 32,637 human demonstration trajectories, each paired with a natural language description, covering 4.65 km$^2$ across two real cities: Cambridge and Birmingham. In contrast to existing datasets composed of synthetic scenes such as AerialVLN, our dataset presents a unique challenge because agents must interpret spatial relationships between real-world landmarks and the navigation destination, making CityNav an essential benchmark for advancing aerial VLN. Furthermore, as an initial step toward addressing this challenge, we provide a methodology of creating geographic semantic maps that can be used as an auxiliary modality input during navigation. In our experiments, we compare performance of three representative aerial VLN agents (Seq2seq, CMA and AerialVLN models) and demonstrate that the semantic map representation significantly improves their navigation performance.

1. Introduction

Real-world aerial VLN remains underexplored because high-fidelity city datasets are scarce and integrating geographic with visual information is difficult. CityNav addresses this gap with a large real-world dataset and geographic semantic maps that significantly improve three representative agents’ navigation performance.

1. Introduction

  • Real-world city aerial VLN remains underexplored because high-fidelity 3D datasets and methods for integrating geographic and visual information are limited.Source: §1 · p. 2
  • CityNav provides 32,637 human trajectories across 4.65 km2 in Cambridge and Birmingham, creating a large-scale real-world 3D aerial VLN dataset.The dataset’s challenge includes interpreting spatial structure between real-world landmarks and the navigation destination.Source: §1 · p. 2
  • Geographic semantic maps significantly improve navigation performance for Seq2Seq, CMA, and AerialVLN agents.The maps provide geographic information, including landmark locations, as an auxiliary modality alongside visual information.Source: §1 · p. 2

2. Related Work

Prior VLN work spans ground-level, outdoor street, and aerial settings, with aerial datasets including virtual environments, satellite imagery, and synthetic 3D cities. CityNav extends aerial VLN to large-scale real-world environments with landmark-to-destination spatial reasoning.

2. Related Work

  • Ground-level VLN work includes indoor 3D tasks and outdoor street-view datasets such as TouchDown and Talk2Nav.TouchDown has 9,326 human demonstration trajectories, and Talk2Nav has 10,714 trajectories with verbal descriptions.Source: §2 · p. 2
  • CityFlight synchronizes its 3D flight simulation with OpenStreetMap, and human annotators navigate to specified goal objects using five actions.Source: Figure 2 · p. 3
  • Aerial VLN datasets progress from LANI’s small virtual environment to AVDN’s satellite imagery and AerialVLN’s synthetic 3D city, while CityNav extends the task to real-world cities.AerialVLN also proposed look-ahead guidance for a cross-modal attention model.Source: §2 · p. 3

3. CityNav Dataset

CityNav pairs real-world 3D city navigation with 32,637 human demonstrations across Cambridge and Birmingham. Its trajectories span varied distances and action patterns, and descriptions frequently combine spatial relations with visual details.

3. CityNav Dataset

  • CityNav provides 32,637 human demonstration trajectories across 4.65 km^2 in Cambridge and Birmingham, the largest number reported to date for aerial VLN.The dataset includes a diverse range of real-world 3D navigation scenarios.Source: §3 · p. 3, §3.4 · p. 5

3.1. CityFlight Environment

  • CityFlight uses real-world 3D scans synchronized with OpenStreetMap; UAVs navigate with six actions and may fly up to 200 meters.The environment supports real-world geographic context but excludes scenarios where goal coordinates are known.Source: §3.1 · pp. 3–4

3.2. Task Definition

  • The task is to reach a text-described goal object using visual and geographic observations, with descriptions specifying landmarks and spatial relationships.Starts are randomly selected within 500 meters of the goal, and success requires stopping within a 20-meter radius.Source: §3.2 · p. 4

3.3. Human Demonstration Trajectories

  • After qualification and re-collection, 171 annotators contributed 32,637 high-quality description-trajectory pairs.Trajectories failing the success criteria were excluded, including 7.2% of goals that still failed after re-collection.Source: §3.3 · p. 4

3.4. Dataset Statistics

  • Goal distances range from 50 to 500 meters, typically around 200 meters, while trajectories average 240 actions, compared with 49 for ADVN and 230 for AerialVLN.These distances and action counts represent varied navigation scenarios and comparable or greater trajectory lengths.Source: Figure 3 · p. 4, Figure 5 · p. 4, §3.4 · p. 5
  • Descriptions frequently include relative-position words and visual details such as goal colors and object parts.Figure 4 presents description lengths by word count and a word-cloud representation of word distribution.Source: Figure 4 · p. 4, §3.4 · p. 5
  • Human operators make extra ascending movements to view surrounding objects, while their overall action distribution remains close to the shortest path.Figure 6 compares action distributions with those for shortest paths.Source: Figure 6 · p. 4, §3.4 · p. 5
  • Human-annotated trajectories pass near landmarks more frequently than shortest paths.The analysis checks whether trajectories pass over landmark polygons named in their descriptions.Source: §3.4 · p. 5, Figure 7 · p. 5

4. Models

The paper evaluates three representative aerial VLN baselines and adds geographic semantic maps as an auxiliary input. The maps encode geographic and visual context, including landmarks and explored areas.

4. Models

  • Seq2Seq, CMA, and AerialVLN are selected as representative aerial VLN baselines.Seq2Seq uses recurrent action prediction; CMA adds cross-modal attention, and AerialVLN adds look-ahead guidance during training.Source: §4 · p. 5, §4.1 · p. 5

4.2. Geographic Semantic Map

  • GSM is introduced as an auxiliary modality to let the baselines use geographic information from 2D maps.Table 2 presents a navigation-performance comparison with GSM incorporated into each baseline; the supplied passage does not report the table’s values.Source: §4 · p. 5, Table 2 · p. 6
  • GSM combines agent pose with OpenStreetMap data and marks the current view, explored area, landmarks, potential goals, and surrounding objects.Binary masks aligned with the 2D map are encoded by a small convolutional network, and the resulting representation is appended to the models’ input sequence.Source: §4.2 · pp. 5–6

5. Experiments

Experiments on CityNav compare three aerial VLN models with and without geographic semantic maps, then examine performance across instruction and landmark conditions, training trajectories, and robustness settings. The results show benefits from semantic maps and human demonstrations, while exposing challenges in unseen environments and more difficult conditions.

5.1. Experimental settings

  • The train and val-seen subsets share 24 scenes, while val-unseen and test-unseen use distinct scenes to assess generalization.The splits contain 22,002 train descriptions, 2,498 val-seen descriptions, 2,826 val-unseen descriptions, and 5,311 test-unseen descriptions.Source: §5.1 · p. 6

5.2. Experimental Results

  • GSM significantly improves all three models; AerialVLN performs best, followed by CMA in navigation error on unseen environments.Human navigators achieve 87–90% success across all subsets, leaving a substantial gap from current VLN models.Source: §5.2 · pp. 6–7
  • Longer descriptions and higher landmark density are associated with higher success rates across models.Figure 9 compares short and long instructions, and Figure 10 compares few and many landmarks on test-unseen; the categories use average instruction length and landmark count.Source: §5.2 · p. 7, Figure 9 · p. 7, Figure 10 · p. 7
  • Models trained on human demonstrations outperform those trained on shortest paths, with a larger advantage as training data increases.The authors attribute the difference to human trajectories exposing relative positions of goals and other objects that shortest paths may miss.Source: §5.2 · p. 7, Table 3 · p. 7
  • Under ±100-meter position noise, the NE gap between human-demonstration and shortest-path training widens from 10.0 to 28.1.The authors attribute this robustness to the greater trajectory diversity in human demonstration data.Source: §5.2 · p. 7, Table 4 · p. 7
  • Every GSM component contributes to performance, and qualitative examples show geographic cues helping agents align movement with landmarks and locate targets.Removing landmark information causes a significant performance drop; Figure 11 presents qualitative examples.Source: Table 5 · p. 7, §5.2 · p. 8, Figure 11 · p. 8

6. Conclusion

The conclusion presents CityNav as a large real-world aerial navigation dataset and reports that semantic maps improve model performance. It also identifies limited city coverage and outlines directions for expanding the dataset and navigation setting.

6. Conclusion

  • CityNav covers 4.65 km2 across Cambridge and Birmingham and contains 32,637 human demonstration trajectories, making it the largest aerial VLN dataset to date.The paper also introduces three baseline models and a semantic map representation.Source: §6 · p. 8
  • Experiments show that the semantic map representation substantially improves model performance, and the paper evaluates robustness in a simulated flooding disaster scenario.The appendix provides additional details on data collection, dataset statistics, models, and experimental results.Source: §6 · p. 8, APPENDIX · p. 12
  • Coverage remains limited relative to cities worldwide; broader coverage would require reducing outdoor 3D-scan costs.Future directions include multi-agent collaboration and integrating aerial, ground-level, and indoor navigation, which the paper says involves differing 3D-scan granularity requirements.Source: §6 · p. 8

A. Data Collection Interface

The data collection website, built on Amazon Mechanical Turk, lets users operate an aerial agent in CityFlight to collect human demonstration trajectories.

B. Dataset Statistics

Dataset statistics describe human flight altitude and show that demonstration trajectories pass near landmarks more often than shortest-path trajectories. The described human strategy uses landmarks to narrow the search area before locating a goal object.

B. Dataset Statistics

  • Human-operated flights averaged 35.96 meters in 3D altitude and gradually descended toward their goals.Figure 13 summarizes mean altitude in 20-meter distance-to-goal intervals; Figure 14 shows a top-down view from 150 meters.Source: B. Dataset Statistics · p. 12, Figure 13 · p. 12, Figure 14 · p. 12
  • The described human strategy first uses named landmarks to narrow the search area, then searches around the landmark for the goal object.Source: B. Dataset Statistics · p. 12
  • Human demonstrations passed directly over landmarks in 36.3% of trajectories, compared with 24.6% for shortest paths.Within 20 meters of a landmark, the respective rates were 35.5% and 24.0%.Source: B. Dataset Statistics · p. 12
  • Within 40 meters of a landmark, 62.5% of human demonstrations and 51.9% of shortest paths passed nearby.Source: B. Dataset Statistics · p. 13

C.1. AerialVLN+GSM architecture

AerialVLN+GSM combines RGB, depth, geographic semantic map, and text inputs in a multimodal navigation architecture. Its GSM encodes the UAV’s view and explored area alongside landmarks, potential goals, and surrounding objects.

C.1. AerialVLN+GSM architecture

  • AerialVLN+GSM encodes RGB, depth, and GSM images with separate ResNet-50 and convolutional networks, while a Bi-LSTM encodes navigation descriptions.The RGB and GSM inputs are 224 × 224 pixels; depth is 256 × 256.Source: §C.1 · p. 13
  • A first GRU processes the visual embeddings, description and modality-specific visual attention refine them, and a second GRU aggregates the outputs.A learnable linear layer predicts the action from the final output embedding.Source: §C.1 · p. 13

C.2. Geographic Semantic Map

  • The GSM combines current field of view, explored area, landmarks, potential goals, and surrounding objects in a 224 × 224 map.GNSS coordinates determine the view and explored area; OpenStreetMap supplies landmarks, while an object detector identifies goals and surrounding objects.Source: §C.2 · p. 13

C.3. Training

Models were trained for five epochs on one GeForce RTX 4090 GPU using Adam, with cross-entropy and goal-distance MSE losses.

C.3. Training

  • Training used one GeForce RTX 4090 GPU, Adam for 5 epochs, an initial learning rate of 5, and batch size 12.Source: §C.3 · p. 13
  • The training objective combined cross-entropy loss with MSE measuring distance between the goal point and current position.Source: §C.3 · p. 13
  • AerialVLN’s look-ahead guidance used a step parameter of 10.Source: §C.3 · p. 13

D. Additional Analysis

Category-level evaluation generally favors GSM-enhanced AerialVLN, while flood-inundation simulations show reduced performance across models but retain GSM’s effectiveness.

Category-level performance

  • AerialVLN+GSM generally performs best across goal categories; CMA+GSM and Seq2Seq+GSM improve over their baselines but remain less effective.Ground and others categories challenge baseline methods, and GSM integration helps mitigate those difficulties.Source: D. Additional Analysis · p. 13, Table 6 · p. 14

Disaster scenarios

Category-level performance

Loading 2406.14240v3…