Air-Ground Collaborative Vision-and-Language Navigation via Shared Bird's-Eye Maps

Shuning Zhang; Liang Li; Yunheng Wang; Tao Wang; Yihang Kang; Renjing Xu

Source-linked AI summary

Air-Ground Collaborative Vision-and-Language Navigation via Shared Bird's-Eye Maps

Shuning Zhang, Liang Li, Yunheng Wang, Tao Wang, Yihang Kang, Renjing Xu

arXiv:2609.03483v1Sep 3, 2026cs.ROcs.AI

TL;DR

Air-ground collaborative VLN lacks a stable training-free cooperation mechanism, despite complementary UAV and UGV viewpoints. AGC-VLN couples 3D-SPF on the UAV with VLM road-path planning on the UGV through a shared bird’s-eye map, achieving a 77.0% joint success rate on 100 closed-loop Town10HD episodes. The baseline remains bounded by VLM target mis-anchoring, latency, and simulator-provided synchronized poses.

Problem
Air-ground collaborative VLN lacks a demonstrated working method, and prior evaluations found unstable cooperation or degraded performance from naive communication and coupling.
Method
AGC-VLN couples UAV 3D-SPF with UGV VLM road-path planning through a shared bird’s-eye map containing teammate and target annotations.
Results
77.0% joint success rate was achieved on 100 closed-loop Town10HD episodes, with the shared-map system outperforming the weaker individual UAV agent by +27.0%.
Takeaways & Limitations
The UAV’s global bird’s-eye context and the UGV’s road-following execution provide complementary roles for training-free air-ground collaboration.
Takeaways & Limitations
The baseline uses VLM target anchoring, 3 s VLM latency, and simulator-synchronized poses, limiting robustness to mis-anchoring, fast-moving targets, and real-world localization requirements.

Abstract

from arXiv · show

Air-ground collaborative Vision-and-Language Navigation (VLN) pairs an unmanned aerial vehicle (UAV) with a global bird's-eye view and an unmanned ground vehicle (UGV) with a local first-person view, yet the setting remains largely unexplored: existing training-free methods solve single-agent tasks but offer no collaboration mechanism, and a recent CARLA-Air evaluation found no stable cooperative behavior across five state-of-the-art VLA models; naive semantic communication or bidirectional coupling even degrades performance. We establish AGC-VLN (Air-Ground Collaborative VLN), the first training-free baseline for air-ground collaborative VLN. The key insight is that training-free methods decompose navigation into VLM-based semantic reasoning and deterministic geometric execution, exposing a collaboration interface: the UAV's global view, over which it renders the UGV's reported pose and the VLM-anchored target as CAR/GOAL markers with distance labels, yielding a shared bird's-eye map. From this map, the UGV acquires global spatial context its first-person view cannot provide, plans a road-following path with a frozen VLM, and executes it under closed-loop control; in parallel, the UAV runs 3D-SPF, a spatial-search upgrade of SPF that localizes the target in the downward view and flies toward it. On 100 closed-loop episodes in CARLA-Air's Town10HD scene, AGC-VLN reaches a 77.0% joint success rate, a collaboration gain of +27.0% over the weaker individual agent (the UAV, 50.0%), and exceeds the strongest published single-agent baseline (Travel UAV, 53.0%) by 24.0 points, stemming from the complementarity of the UAV's global view and the UGV's road-following execution. Project page: https://github.com/ZSN2024/AGC-VLN.

I. INTRODUCTION

AGC-VLN addresses the lack of working air-ground collaborative VLN by extending training-free semantic-geometric navigation with a shared bird’s-eye map. The UAV supplies global context while the UGV performs road-level planning and execution.

Motivation: Air-ground collaboration combines the UAV’s wide-area coverage with the UGV’s ground-level reachability, supported by CARLA-Air and AirGroundBench infrastructure.CARLA-Air synchronizes CARLA and AirSim, while AirGroundBench provides 115 closed-loop episodes across 11 environments.
Problem: Five aerial VLA models failed to convert single-agent ability into stable cooperative behavior, while text hints and bidirectional coupling often degraded performance.The diagnosed deficits were partner-state anchoring, low-latency coordination, and team-level objective alignment.
Approach: AGC-VLN extends the training-free semantic-reasoning and deterministic-execution decomposition with a collaboration interface based on a shared bird’s-eye map.The UAV renders the UGV pose and VLM-anchored target as map annotations, avoiding a learned cross-agent representation.
Approach: The UAV provides global spatial context, while the UGV uses that context for road-level planning and closed-loop execution in its own role.This division of labor complements the UAV’s global view and the UGV’s ground-level execution.
Positioning: Training-free navigation uses frozen foundation models orchestrated by deterministic glue, but prior systems remained single-agent and did not define inter-agent communication.AGC-VLN applies this paradigm to the air-ground collaboration problem.

III. METHOD

The method turns the UAV’s downward view into a shared bird’s-eye map by combining its own view with the UGV’s reported pose and a VLM-anchored target. The UGV receives this map and uses it for path planning and execution.

UAV Module: At each roughly 3-second decision step, the UAV renders the shared map and simultaneously runs its own 3D-SPF navigation.The two functions operate in parallel within the collaboration loop.
UAV Module: The UAV projects the UGV pose into its downward image and anchors the target against the target photo using a frozen VLM.Both annotations share the target localization used by 3D-SPF.
Map Rendering: The map marks the UGV as CAR, the target truck as GOAL, the UAV at image center, and the straight CAR→GOAL reference line.These visual markers expose teammate and target geometry to the UGV.
Map Rendering: Distance labels give UAV→GOAL and CAR→GOAL measurements, and the fully annotated image is transmitted with the UAV pose for deterministic inverse projection.The map is shared through shared memory together with the annotation-time UAV pose.

2) 3D-SPF (Three-Dimensional SPF):

3D-SPF extends SPF by having a frozen VLM predict both the target pixel and a discrete vertical action, enabling spatial search beyond fixed-altitude planar flight.

3D-SPF: The UAV’s frozen VLM receives the downward image and target photo, then returns the target pixel coordinates and a height command.The height command is descend, hold, or ascend.
3D-SPF: Flat-ground ray casting converts the predicted pixel position into a world coordinate using the downward field of view and altitude.This conversion assumes planar ground.
3D-SPF: The UAV flies horizontally toward the converted world target with proportional velocity, then changes altitude by 10 m according to the predicted command.After the vertical adjustment, it hovers until the next decision.

3) Coordinate Transformation:

The system reconciles AirSim and CARLA coordinates through one-time calibration and uses flat-ground ray casting to connect image pixels with world positions.

Coordinate Transformation: AirSim reports positions in NED coordinates while CARLA uses a world frame with opposite z-axis direction.The UAV and UGV therefore require a coordinate-frame conversion.
Coordinate Transformation: A constant offset obtained by one-time calibration converts between the AirSim and CARLA coordinate systems.The calibration uses the drone positions reported by both simulators.
Coordinate Transformation: Flat-ground ray casting and its inverse perform the pixel↔world conversion used to align drone and overhead views.The conversion assumes a flat ground plane.

1) Road Path Planning:

The UGV converts the UAV’s annotated bird’s-eye map into a road-following world path, then tracks it with closed-loop control. Shared pose and map updates provide complementary global context and local driving capability.

Road Path Planning: The frozen VLM plans 10 pixel waypoints along visible roads between the CAR and GOAL markers.The first and last waypoints match the vehicle and target, while intermediate points follow roads, turn at intersections, and avoid buildings.
Road Path Planning: A sanity check discards paths whose endpoints clearly deviate from CAR or GOAL.
World-Path Conversion: Each pixel waypoint is inverse-projected into NED coordinates and converted into CARLA world coordinates using the offset.
Closed-Loop Path Tracking: The UGV tracks the resulting world path point by point using wrapped heading error and distance-scaled throttle.Six driving primitives support forward, turning, reverse, and reverse-turning behavior; stalled vehicles trigger reverse escape.
Collaboration Mechanism: Asynchronous shared memory exchanges the UGV pose to the UAV and the annotated bird’s-eye map back to the UGV.The UAV provides global spatial context, while the UGV contributes road-level planning and execution.

IV. EXPERIMENTS

The experiments evaluate AGC-VLN as a training-free, parallel air-ground system in synchronized CARLA-Air episodes. UAV and UGV threads share state, annotate maps, plan paths, and execute until success or timeout.

Research Questions: The study asks whether training-free agents can compose into a working collaborative system, improve over individual operation, and reveal actionable failure modes.
Episodes: Experiments use 100 closed-loop episodes across 50 Town10HD spawn points, with a Mini Cooper UGV navigating toward an HGV target.Each episode provides a target-truck photo and natural-language description.
Evaluation: The task counts success when either agent reaches the target within ϵ, while the first successful agent waits for the other.
Metrics: The evaluation reports UAV and UGV success rates, joint success rate, SPL, NE, and collaboration gain.Collaboration gain is defined as SRjoint minus the weaker individual success rate.

B. Baselines

The comparison includes published aerial VLN systems and single-agent lower bounds. These baselines reproduce alternative aerial methods and test UAV-only or UGV-only operation under the same task setting.

Published Single-Agent Methods: Published aerial baselines include OpenFly, FineCog-Nav, 3DG-VLN, and Travel UAV.They are reproduced under the same single-agent evaluation protocol for comparison.
Single-Agent Lower Bounds: Single-agent lower bounds include UAV-only SPF, UAV-only 3D-SPF, and a UGV-only VLM solving the entire episode alone.

C. Main Results (RQ1, RQ2)

AGC-VLN demonstrates positive collaboration gain in the main evaluation, with performance depending on the VLM backbone, UAV altitude, and map annotation richness. The results also clarify when the team’s aerial contribution disappears.

Main Results: 77.0% SRjoint combines 75.0% SRUGV and 50.0% SRUAV, yielding +27.0% collaboration gain over the weaker agent.The system also records 62.0% SPL and 10.9 m NE.
Main Results: 24.0 points separate AGC-VLN from the strongest published single-agent baseline, Travel UAV at 53.0%.
VLM Backbone: gemini-3.7-flash achieves the highest joint success rate at 77.0%, while alternative backbones reach 67.0%–73.0%.The alternatives weaken UAV success to 7.0%–47.0%.
UAV Altitude: At 60 m, UAV success peaks at 50.0% and UGV success reaches 75.0%, producing the best joint rate of 77.0%.At 30 m, joint success falls to 28.0%; at 90/120 m, UAV localization fails at 0.0% and the UGV alone delivers 77.0%/72.0%.
Map Annotation Richness: Full map annotation reaches 77.0% joint success, whereas CAR+GOAL+reference line reaches 75.0% and CAR+GOAL reaches 70.0%.GOAL-only scores 60.0%, below the raw unannotated image at 65.0%, because it lacks the teammate’s CAR anchor.

G. Failure Source Analysis (RQ3)

Failure analysis identifies path planning as the dominant joint-failure stage, while UAV-side failures primarily involve global target localization. The ablations and representative episodes connect these errors to road-path validity and target anchoring.

Failure attribution: 61% of jointly failed episodes arise from UGV path planning, where predicted paths collapse onto buildings, trees, or map borders.These failures cause crashes or stalls 16–73 m from the goal.
Failure attribution: 17% of joint failures are UAV-side localization errors, comprising unstable tracking at 13% and mis-anchoring at 4%.Mis-anchoring occurs when the UAV locks onto a visually similar but incorrect truck.
Per-agent analysis: The UAV fails mainly from global localization at 76%, whereas the UGV fails mainly from path planning at 60%.The UGV rescues 27 of the UAV’s 50 failures, compared with 2 rescues in the reverse direction.
Representative episodes: Representative failures show two distinct mechanisms: UAV mis-anchoring toward a wrong truck and UGV path planning that collides with a tree.In successful episodes, the annotated map supports a VLM road path that both agents track to the goal.

V. REAL-ROBOT EXPERIMENT CASE

The authors deploy the same training-free air-ground pipeline on physical UAV and UGV hardware. The case demonstrates coordinated map sharing, 3D target approach, and road-path execution across six time steps.

Shared-map collaboration: The UAV annotates and transmits a bird’s-eye map, while the UGV uses a VLM to plan a road path from that shared view.The UAV pose is supplied by LiDAR odometry, and the pipeline retains the simulation’s decision cadence.
Hardware and setup: The real-robot team uses a UAV with Mid360 LiDAR and RealSense D435i and an omnidirectional UGV with a forward camera.The deployment uses top-down and third-person views over six time steps.
Execution: The physical deployment drives both agents to the target: the UAV approaches with 3D-SPF and the UGV steers along its planned road path.The sequence shows the UAV climbing, sending the map, and both vehicles moving toward the target.

VI. LESSONS AND LIMITATIONS

AGC-VLN argues that global-view communication and deterministic execution make air-ground collaboration effective without learned cross-agent coupling, while acknowledging localization, latency, and deployment constraints.

Lessons for Air-Ground Collaboration: Global bird’s-eye communication supplies the UGV with relative teammate-target positions and road topology missing from its local view.The UAV renders the UGV pose and VLM-anchored target for direct consumption by the UGV’s VLM.
Lessons for Air-Ground Collaboration: View complementarity, rather than richer messages, is identified as the source of collaboration gains.The UAV contributes global spatial context, while the UGV contributes road-level planning and execution.
Lessons for Air-Ground Collaboration: Deterministic projection and inverse projection confine errors by preventing one platform’s VLM mistake from propagating to the other.This contrasts with the error amplification attributed to bidirectional velocity coupling.
Limitations: AGC-VLN deliberately uses a frozen VLM for target anchoring, accepts 3 s latency, and relies on synchronized simulator poses instead of GPS/SLAM.These choices retain a lightweight baseline but introduce mis-anchoring risk, restrict the setting to quasi-static scenes, and limit direct field-deployment realism.
Limitations: AGC-VLN achieves a positive collaboration gain on 100 closed-loop Town10HD episodes, with residual failures concentrated on global localization and wrong-truck reference binding.The paper identifies robust cross-view re-identification, learned verification, and latency-robust coordination as follow-up directions.