PaperlayerView exact revision on arXiv

Source-linked AI summary

SatNav: A Scalable Benchmark for Long-Horizon UAV Vision-Language Navigation from Satellite Imagery

Jiajun Jiang, Chunliang Hua, Zichun Chen, Yanxing Wu, Zeyuan Yang, Jie Song, Xiao Hu

arXiv:2609.31507v1cs.CVcs.RO

TL;DR

Urban UAV VLN needs long-range memory and geospatial grounding, while existing benchmarks are constrained by costly 3D assets and limited long-range coverage. SatNav builds a city-scale benchmark from satellite imagery and OpenStreetMap data, with tasks for loop progress, landmark grounding, and route reasoning. Experiments report challenges in generalizing to unseen regions and stopping reliably, and describe transfer to real-flight UAV observations.

  • Problem

    Urban UAV VLN requires memory over extended routes, but existing benchmarks rely on costly reconstructed 3D assets and provide limited evaluation of long-range memory and geospatial reasoning.

    Source: §1 · pp. 1–2

  • Method

    SatNav uses satellite imagery and OpenStreetMap annotations to construct city-scale navigation episodes for Boundary, Landmark, and Route tasks, and introduces SwiftVLN for memory-design comparisons.

    Source: §1 · pp. 2–3

  • Results

    Existing classical and LVLM-based agents face challenges generalizing to unseen geographic environments and stopping reliably; satellite-trained models also transfer to real-flight UAV observations.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The benchmark and SwiftVLN provide a setting for evaluating long-horizon memory and geospatial reasoning, including satellite-to-UAV transfer and memory-design comparisons.

    Source: §1 · p. 2

    Navigation uses four discrete actions at fixed altitude and excludes paths requiring other movements, favoring smooth routes over winding ones.

    Source: §6 · p. 10

Abstract

from arXiv · show

Urban uncrewed aerial vehicle (UAV) vision-language navigation (VLN) requires agents to follow instructions across extended urban spaces, inherently demanding long-term memory and geospatial grounding. However, scaling existing benchmarks remains difficult because of their reliance on costly reconstructed 3D assets, limiting geographic diversity and episode scale. To address this, we introduce SatNav, a scalable, long-horizon UAV VLN benchmark built from high-resolution satellite imagery. SatNav targets city-level navigation missions and uses satellite crops as approximations of UAV nadir views for visual observations. Through an automated cue-to-episode pipeline, SatNav constructs 118K episodes from 59 scenes across 18 cities, with an average trajectory length of 379 m. To stress-test long-horizon memory and geospatial reasoning, SatNav defines three task families: Boundary, Landmark, and Route, targeting loop progress tracking, landmark-based spatial grounding, and route following with counting cues. Benchmarking classical VLN agents and recent agents based on large vision-language models (LVLMs) on SatNav shows that city-scale navigation remains challenging. We further introduce SwiftVLN, a modular framework with switchable memory components, and conduct systematic memory-design ablations. Finally, satellite-to-UAV transfer experiments show that satellite-trained navigation models can operate on real-flight UAV observations, showing the practical relevance of SatNav. Our project page: https://eku127.github.io/SatNav/

1 Introduction

SatNav addresses the limited scalability and long-range coverage of urban UAV VLN benchmarks with a satellite-imagery benchmark for city-scale navigation. It evaluates memory and geospatial reasoning across three task families, benchmarks existing agents, introduces SwiftVLN, and tests transfer to real-flight UAV observations.

Motivation

  • Existing UAV VLN benchmarks are constrained by costly reconstructed 3D assets and provide limited coverage of long-range navigation requiring memory over extended routes.Indoor benchmarks also have limited spatial scale and navigable area for testing long-distance memory and temporal consistency.Source: §1 · pp. 1–2

Approach

  • Satellite crops approximate top-down UAV views and allow city-scale construction using rotate-and-crop operations instead of complex 3D rendering.The approach relies on large-scale geospatial structures being preserved in nadir views.Source: §1 · p. 2

Task design

  • Boundary, Landmark, and Route tasks target progress tracking, landmark-based spatial grounding, and route following with decision-point tracking.The benchmark also provides detailed, natural, and concise instruction styles.Source: §1 · pp. 2–3

Experiments

  • The paper benchmarks classical and LVLM-based agents, introduces SwiftVLN for memory-design ablations, and reports satellite-to-UAV transfer to real-flight observations.The introduction describes the transfer as evidence of SatNav’s practical relevance.Source: §1 · pp. 2–3

Benchmark

  • SatNav automatically constructs 118K episodes across 18 cities from paired satellite imagery and OpenStreetMap annotations.The benchmark’s average trajectory length is 379 m.Source: §1 · pp. 2–3

2 Related Work

Prior work has advanced indoor and aerial VLN, as well as memory methods for long-horizon navigation. The supplied review identifies limited support for city-scale evaluation of long-range memory and geospatial reasoning.

2.1 Indoor and Aerial VLN Benchmarks

  • Indoor benchmarks expanded instruction following and longer-horizon settings, but limited spatial extent and topological diversity constrain long-range memory testing; aerial benchmarks often emphasize local navigation and flight control.The review notes that many aerial benchmarks also rely on complex 3D assets or simulation pipelines.Source: §2.1 · p. 3

2.2 Memory Modeling in Long-Horizon VLN

  • Long-horizon VLN methods include map-based memory, selective frame sampling, pose-aware encoding, and token compression, but evaluations offer limited support for comparing these mechanisms at city scale.The review names representative strategies and their associated works.Source: §2.2 · p. 3

3 SatNav Bench

SatNav formulates satellite-view navigation as instruction-conditioned action prediction and defines three task families for long-horizon memory and spatial reasoning. Its shared cue-to-episode pipeline uses satellite imagery and OpenStreetMap annotations to build a geographically diverse dataset with dedicated seen and unseen test splits.

3.1 Task Definition

  • Boundary tasks test progress tracking and stopping around closed-region perimeters; Landmark tasks use visual landmarks to ground heading at one or two turns.Boundary variants include partial-arc, full-loop, and extended-loop; Landmark variants include one-turn and two-turn decisions.Source: §3.1 · p. 4
  • Route tasks follow roads, waterways, or hybrid paths, using intersections and ordinal cues to test junction decisions and counting successive decision points.Instructions can specify cues such as turning at the first bridge or stopping at the third crossroad.Source: §3.1 · p. 4

3.2 Geospatial Data Resources

  • Aligned satellite imagery and OpenStreetMap annotations provide visual and structured geographic cues for Boundary, Landmark, and Route episodes across 18 cities and 59 scenes.The dataset can be extended by selecting new regions and applying the same pipeline.Source: §3.2 · p. 4

3.3 Episode Generation

  • A shared cue-to-episode pipeline extracts validated geospatial cues, constructs task-specific trajectories, and converts cue sequences into navigation instructions.Candidate trajectories are filtered and refined, with long, coherent paths prioritized to require long-term memory.Source: §3.3 · p. 5
  • Instructions encode task-specific trajectory cues and are composed in detailed, natural, and concise styles, then rewritten and checked for semantic alignment.Each packaged episode contains scene identification, task metadata, an instruction, and a reference trajectory.Source: §3.3 · p. 5

3.4 Dataset Statistics and Splits

  • 118,494 episodes span 59 scenes in 18 cities across five continents, with a mean trajectory length of 379 m and median of 340 m.Boundary, Landmark, and Route account for 25.5%, 38.1%, and 36.3% of episodes, respectively.Source: §3.4 · p. 5
  • Test Seen uses new trajectories and instructions in familiar scenes, while Test Unseen uses three scenes in three new cities to assess geographic generalization.Both test splits maintain the same task distribution to reduce task-imbalance effects in seen-versus-unseen comparisons.Source: §3.4 · p. 6

4 SwiftVLN

SwiftVLN provides a flexible VLN framework for comparing memory designs, combining short-term dialogue history with interchangeable long-term memory mechanisms. A visual adapter also allows SatNav-trained policies to operate on real UAV nadir images using paired training data.

4.1 Framework Overview

  • SwiftVLN enables controlled evaluation of interchangeable memory designs and adaptation of different LVLMs to VLN.It refactors StreamVLN’s training and evaluation pipeline within ms-swift.Source: §4.1 · p. 6
  • Short-term memory retains recent image-action turns with a sliding window, while long-term memory processes observations outside that window.The window removes oldest turns at capacity while retaining overlap; long-term mechanisms include frame sampling, input augmentation, and memory compression.Source: §4.1 · p. 6

4.2 Satellite-to-UAV Generalization

  • A Transformer adapter trained on 24K paired UAV-satellite images maps UAV visual tokens into the frozen encoder’s satellite feature space, enabling the SatNav-trained policy to use real UAV images.The adapter uses bidirectional contrastive and cosine similarity losses and is placed between the vision encoder and the SatNav-trained backbone at inference.Source: §4.2 · p. 7

5 Experiments

Experiments compare classical and LVLM-based agents on SatNav, evaluate SwiftVLN memory variants, and test real-world transfer. Results show persistent geographic generalization challenges, benefits from selected memory designs, and feasible but difficult real-UAV deployment.

5.1.1 Baselines and Ablation Settings

  • Baselines include classical Seq2Seq and CMA and LVLM-based methods; SwiftVLN memory ablations hold non-memory components and training protocols fixed against a reference with an 8-turn window and 8 uniformly sampled history frames.The reference window has no overlap; evaluation reports SR, OS, SPL, NE, and average Steps, with success thresholds of 10 m or 30 m for Landmark.Source: §5.1.1 · p. 7, §5.1.2 · pp. 7–8

5.2 Baseline Comparison on SatNav

  • Performance drops from Test Seen to Test Unseen across methods; LVLM-based methods substantially outperform classical baselines, while StreamVLN* leads SR, SPL, and OS and SwiftVLN has the lowest NE on both splits.Navigation-pretrained initialization consistently improves performance; UniNaVid’s continued variant improves SR by 24.6 and 16.3 percentage points on Test Seen and Test Unseen, respectively.Source: Table 1 · p. 8, §5.2 · p. 8

5.3 Memory Design Analysis

  • Removing long-term memory reduces performance, window overlap improves on the reference setting, and temporal-biased sampling brings small but consistent gains over uniform sampling.Random history-frame sampling performs poorly; recent observations are described as informative for the next action, with earlier frames serving as trajectory anchors.Source: §5.3 · pp. 8–9, Table 2 · p. 9
  • Relative pose encoding brings stable gains, while initial-frame prompting improves OS but not SR; map memory and GTC underperform the reference, whereas STC outperforms GTC.The authors suggest a shared vision encoder may cause a representation compromise for map memory; STC preserves coarse temporal order by clustering within temporal segments.Source: §5.3 · p. 9, Table 2 · p. 9

5.4 Real-World Experiments

  • Real-world evaluation uses 18 episodes across Boundary, Landmark, and Route, with six per family; flights show feasibility of deployment with lightweight visual adaptation, but local ambiguity and execution uncertainty remain challenging.A DJI Matrice 4D transmits nadir images for online SwiftVLN inference on an RTX 4090 workstation.Source: §5.4 · p. 9, Table 3 · p. 9

6 Limitations

SatNav’s limitations include its restricted action space, reference-trajectory training, and a small real-flight evaluation constrained by system latency.

6 Limitations

  • Four discrete actions at fixed altitude exclude paths requiring other movements, favor smooth routes, and limit coverage of winding paths.Adding altitude and speed changes, takeoff, and landing could support a wider range of UAV missions.Source: §6 · p. 10
  • Training on reference trajectories provides limited experience with states reached after agent errors, and the current pipeline does not support reinforcement fine-tuning.The authors suggest interaction-based training could help recovery from wrong turns and improve long-route stopping decisions.Source: §6 · p. 10
  • System latency limits long-duration real-flight trials, so the 18-episode evaluation provides preliminary evidence of satellite-to-UAV transfer.The current transfer method also uses a simple visual adapter.Source: §6 · p. 10

7 Conclusion

SatNav uses satellite imagery and OSM annotations to evaluate memory-intensive, long-horizon UAV VLN without relying on reconstructed 3D assets. The conclusion reports challenges in geographic generalization and reliable stopping, while observation cropping and scene selection define the benchmark’s operating scope.

7 Conclusion

  • Existing classical and LVLM-based agents still struggle to generalize to unseen geographic environments and stop reliably; memory ablations and real-flight demonstrations reveal both design guidance and transfer challenges.The authors describe satellite-trained model operation on UAVs as feasible, while noting remaining satellite-to-UAV transfer challenges.Source: §7 · p. 10
  • Boundary, Landmark, and Route tasks make long-range progress tracking, geospatial grounding, and route reasoning explicit evaluation targets.SatNav is presented as a scalable benchmark for memory-intensive long-horizon UAV VLN built from satellite imagery.Source: §7 · p. 10

A.1 Observation Cropping

  • The observation crop corrects Web Mercator scale distortion, aligns its vertical axis with heading, and samples a 100 m × 100 m ground footprint.The frame uses a side length of 100k Web Mercator meters, where k = 1 / cos(ϕt); crops are resized to 448 × 448 pixels and approximate a UAV nadir view at 50 m altitude with a 90° horizontal field of view.Source: §A.1 · p. 15

A.2 Scene Collection

  • Scenes use zoom-level-19 satellite imagery at about 0.3 m per pixel, cover about 14 km² on average, and span 59 scenes across 18 cities.Selection uses fixed criteria for interpretable imagery, usable OSM annotations, and geospatial diversity; scene types include residential, commercial, industrial, park, coastal, riverine, and mixed urban areas.Source: §A.2 · p. 15

B UAV Benchmark Comparison

SatNav compares aerial-navigation benchmarks on scale and diversity, then describes how a fixed automatic pipeline creates episodes from geospatial cues. Its task-specific procedures refine boundaries, localize landmarks, and construct executable routes.

B UAV Benchmark Comparison

  • SatNav achieves the strongest overall scale among compared benchmarks, includes eight trajectory types, and reports vocabulary size 4.7K and lexical diversity 0.316.The authors attribute its scalability mainly to satellite imagery and map annotations rather than manually reconstructed 3D assets; lexical diversity uses a route-controlled protocol.Source: B UAV Benchmark Comparison · pp. 15–16, Table 5 · p. 16

C Automatic Episode Generation Details

  • The episode pipeline applies geospatial cue extraction, quality control, trajectory instantiation, instruction construction, and episode packaging across scenes and cities.The same fixed automatic procedure is used to produce VLN episodes from satellite imagery and OSM-derived cues.Source: C Automatic Episode Generation Details · p. 16

C.1 Detailed Boundary Pipeline

  • Boundary construction refines OSM polygons into image-based masks, then samples verified perimeter anchors to create partial-arc, full-loop, or extended-loop episodes.SAM receives the polygon bounding box and an interior point; retained anchors have nearby visually distinctive cues, and the refined boundary supplies the executable reference path.Source: §C.1 · pp. 16–17, Figure 6 · p. 16

C.2 Detailed Landmark Pipeline

  • Landmark construction uses coarse-to-fine image localization, places decision waypoints within 30 m of landmarks, and expresses turns through relative-view cues.Waypoint geometry is quantized to SatNav’s action space so demonstrations can be executed exactly through the evaluation interface.Source: §C.2 · p. 17

C.3 Detailed Route Pipeline

  • Route construction converts OSM roads and waterways into metric graphs, then retains smooth, locally visible trajectories with unambiguous counting cues for instruction generation.Instructions encode edge types, node categories, turn directions, and ordinal positions; Figure 7 shows representative graph node types.Source: §C.3 · pp. 17–18, Figure 7 · p. 18

C.4 Episode Quality Review

A manual audit accepts 94.3% of sampled candidate episodes before filtering. The section also details prompt construction, SwiftVLN’s query-and-memory pipeline, and several alternative long-term memory representations.

C.4 Episode Quality Review

  • 2,829 of 3,000 audited candidates are accepted (94.3%); rejection rates are 5.9% for Boundary, 8.9% for Landmark, and 2.3% for Route before manual filtering.Reviewers checked task family and subtype, decision points, spatial and counting cues, and stopping conditions; rejected episodes are removed before release.Source: §C.4 · p. 18, Table 8 · p. 19

D Instruction Generation Prompts

  • Four reusable prompt templates separate visual cue perception or localization from language rewriting of validated trajectory records.The landmark prompts use closed-category selection, re-audit candidates, and local verification; rewriting changes surface phrasing while preserving navigation semantics.Source: D Instruction Generation Prompts · pp. 19–21

E.1 Overall Pipeline

  • At each model-query round, SwiftVLN combines the instruction, current observation, short-term dialogue, and long-term memory to decode an action chunk; long-term memory updates when the window slides.The short-term window has a maximum capacity and preserves recent context; generated actions execute sequentially before the next model query.Source: §E.1 · pp. 21–22

E.2 Prompt Construction

  • The constructed prompt combines the instruction, long-term memory, active short-term dialogue, and current observation into the context used to decode the next response.The example contains two completed image-action turns in short-term context.Source: §E.2 · p. 22

E.3 Memory Design Details

  • Frame-sampling memory retains historical images, while initial-frame prompting adds the starting observation and pose encoding modulates image tokens using relative pose.The default retains eight uniformly sampled frames; the initial image uses 256 visual tokens, and pose-conditioned FiLM parameters are projected from a relative pose vector.Source: §E.3 · pp. 22–23
  • GTC clusters historical tokens jointly, STC clusters them across eight temporal segments, and map memory replaces history frames with global and local top-down maps.STC preserves coarse chronology; map memory uses 1000 m global and 400 m local crops, with Figure 9 showing the paired maps.Source: §E.3 · p. 23, Figure 9 · p. 23

E.4 Training Details

Training keeps SwiftVLN’s core optimization hyperparameters consistent across memory-module variants, while the satellite-to-UAV adapter learns from paired views and baseline models are adapted to a shared SatNav action interface.

E.4 Training Details

  • Memory-module variants share core optimization hyperparameters, although their GPU-hour training costs may differ.SwiftVLN and its variants are trained on 8 NVIDIA H100 80GB GPUs.Source: §E.4 · p. 23

E.5 Satellite-to-UAV Adapter Details

  • 24,467 paired UAV–satellite images (21,216 training and 3,251 validation) support adapter training.Each satellite image is aligned with its paired UAV image in viewing extent and orientation.Source: §E.5 · p. 24
  • The adapter transforms UAV visual tokens while leaving satellite features unchanged, then combines cosine and bidirectional contrastive losses for alignment.The adapter is a 2-layer token-level Transformer with 8 attention heads.Source: §E.5 · p. 24

F.1 Baseline Adaptation

  • Baselines use a unified forward, left, right, and stop action space, with model-specific changes to inputs, prompts, or action generation.The LVLM adaptations include fixed-frame histories, queued multi-step actions, and single-word action outputs.Source: §F.1 · pp. 24–25

F.2 Baseline Training Details

The real-world system runs an adapted SwiftVLN model on a workstation receiving UAV video and state data through a controller. Video transmission, rather than model inference, is the main runtime bottleneck.

F.2 Baseline Training Details

  • Baseline models are trained on 8 NVIDIA H100 80GB GPUs, following their public recipes with adjustments for the hardware and unified SatNav setting.Training configurations for LVLM baselines are summarized in Table 11.Source: §F.2 · p. 25, Table 11 · p. 26

G.1 Real-world Deployment

  • The deployed Qwen2.5-VL-3B SwiftVLN runs on one RTX 4090 workstation, receiving video and UAV state relayed through a DJI RC Plus 2 controller.The system uses RTMP for video and MQTT for state and control communication.Source: §G.1 · p. 26, Figure 11 · p. 26
  • 3–5 seconds of RTMP-over-Wi-Fi video latency typically exceeds the model’s roughly 250 ms inference time, making communication the primary runtime bottleneck.Latency between the UAV and controller is described as nearly negligible.Source: §G.1 · p. 26

G.2 More Real-world Flight Results

Real-flight tests cover Boundary, Landmark, and Route tasks, with examples of landmark-guided turning and route following. The tests also expose stopping and system-reliability challenges, while the paper sets out safeguards and intended-use boundaries for real-flight research.

G.2 More Real-world Flight Results

  • Real-flight examples cover all three task families, including landmark-guided turning and route following.Figure 15 presents Landmark and Route examples.Source: §G.2 · p. 26
  • A lake-boundary trial continued circling after completing the loop instead of stopping autonomously.The authors identify stop calibration and error recovery as future-work directions.Source: §G.2 · pp. 26–27

H Broader Impacts and Responsible Use

  • SatNav is intended for research and benchmark evaluation; the paper flags surveillance and military misuse, privacy, and physical safety concerns.It advises minimizing identifiable imagery collection and respecting restrictions around sensitive sites.Source: H Broader Impacts and Responsible Use · p. 27
  • Real-flight studies should use approved, controlled test areas, qualified supervision, geofencing, manual override, and emergency procedures.The paper also notes that physical deployment requires obstacle avoidance, altitude control, and safe recovery.Source: H Broader Impacts and Responsible Use · p. 27

I.1 Task-specific Baseline Performance

Per-task diagnostics distinguish stopping errors on Boundary, landmark localization on Landmark, and route-sequence errors on Route. Additional experiments examine backbone flexibility, satellite-to-UAV transfer, and sensitivity to benchmark settings.

I.1 Task-specific Baseline Performance

  • On Boundary, CMA reaches the goal region in 92.4% of seen episodes but succeeds in only 12.4%, with 80.05 steps, indicating loop-like behavior.StreamVLN and SwiftVLN also retain seen-split OS–SR gaps of 14.2 and 12.6 percentage points, respectively.Source: §I.1 · p. 27, Table 12 · p. 28
  • On Landmark, Seq2Seq and CMA achieve 0.0% SR, with NE exceeding 230 and 780, respectively; SwiftVLN’s SR drops from 71.6% seen to 58.5% unseen.The reported pattern points to difficulty grounding landmarks and sensitivity to cross-city variation.Source: §I.1 · p. 27, Table 12 · p. 28
  • On Route, unseen NE rises from 35.60 to 87.02 m for StreamVLN*, 36.35 to 90.03 m for SwiftVLN, and 47.85 to 115.72 m for UniNaVid*.The authors associate these increases with intersection and counting errors accumulating over time.Source: §I.1 · pp. 27–28, Table 12 · p. 28
  • Table 12 diagnoses distinct task failures: Boundary stopping, Landmark localization, and Route sequence-following and counting.The OS–SR gap measures episodes reaching the success region without successful termination.Source: §I.1 · pp. 27–28, Table 12 · p. 28

I.2 Effect of Base LVLM Backbones

  • SwiftVLN is evaluated with alternative LVLM backbones under the same task interface, memory configuration, action space, and training protocol.The experiment includes Qwen2.5-VL-7B and Qwen3-VL models; Table 13 reports performance.Source: §I.2 · p. 28, Table 13 · p. 29

I.3 Transfer Evaluation with Rendered UAV Observations

  • The adapter improves navigation under both clean and perturbed rendered UAV observations, while exposure and jitter reduce performance for both variants.The evaluation uses 94 episodes rendered from real-world 3D assets.Source: §I.3 · p. 29, Table 14 · p. 29

I.4 Sensitivity to Benchmark Configuration

  • Reducing crops to 224 × 224 pixels or changing spatial coverage substantially lowers SR; finer actions improve SR on both splits but roughly double training and evaluation time.Lower satellite-image resolution causes smaller decreases; Table 15 compares variants changing one factor from the default.Source: §I.4 · p. 30, Table 15 · p. 30
Loading 2609.31507v1…