Source-linked AI summary
AerialVLA: A Vision-Language-Action Model for UAV Navigation via Minimalist End-to-End Control
Peng Xu, Zhengnan Deng, Jiayan Deng, Zonghua Gu, Shaohua Wan
TL;DR
UAV vision-language navigation requires continuous control in dynamic 3D environments, while existing methods depend on dense oracle guidance or external landing detectors. AerialVLA uses minimalist dual-view perception, fuzzy onboard prompts, and unified 3-DoF control with intrinsic landing. On unseen maps, it achieves 37.58% SR versus LongFly's 11.27% SR, while also reaching state-of-the-art performance in seen environments.
Problem
UAV-VLN must support autonomous visual interpretation and continuous control in complex 3D environments, but existing approaches rely on dense oracle guidance and external object detectors.
Method
AerialVLA maps raw visual observations and fuzzy onboard directional prompts to continuous 3-DoF control and an intrinsic landing signal through a minimalist end-to-end policy.
Results
37.58% SR versus LongFly's 11.27% SR on unseen maps, while AerialVLA achieves state-of-the-art 47.96% SR and 38.54% SPL in seen environments.
Takeaways & Limitations
The results support a minimalist autonomy-centric paradigm that generalizes across unseen targets and novel map geometries without dense oracle guidance or explicit object detectors.
Takeaways & Limitations
Reactive control can struggle with global backtracking in repetitive structures and may default to safe hovering when targets are severely occluded.
Abstract
from arXiv · showhide
Vision-Language Navigation (VLN) for Unmanned Aerial Vehicles (UAVs) demands complex visual interpretation and continuous control in dynamic 3D environments. Existing hierarchical approaches rely on dense oracle guidance or auxiliary object detectors, creating semantic gaps and limiting genuine autonomy. We propose AerialVLA, a minimalist end-to-end Vision-Language-Action framework mapping raw visual observations and fuzzy linguistic instructions directly to continuous physical control signals. First, we introduce a streamlined dual-view perception strategy that reduces visual redundancy while preserving essential cues for forward navigation and precise grounding, which additionally facilitates future simulation-to-reality transfer. To reclaim genuine autonomy, we deploy a fuzzy directional prompting mechanism derived solely from onboard sensors, completely eliminating the dependency on dense oracle guidance. Ultimately, we formulate a unified control space that integrates continuous 3-Degree-of-Freedom (3-DoF) kinematic commands with an intrinsic landing signal, freeing the agent from external object detectors for precision landing. Extensive experiments on the TravelUAV benchmark demonstrate that AerialVLA achieves state-of-the-art performance in seen environments. Furthermore, it exhibits superior generalization in unseen scenarios by achieving nearly three times the success rate of leading baselines, validating that a minimalist, autonomy-centric paradigm captures more robust visual-motor representations than complex modular systems.
1 Introduction
UAV vision-language navigation must handle continuous control in complex 3D environments, but existing systems depend on dense oracle guidance and external detectors. AerialVLA addresses these limitations with minimalist perception, fuzzy onboard prompts, and unified end-to-end control with intrinsic landing.
- Motivation: UAV-VLN requires autonomous navigation through full 3D state spaces while interpreting changing visual viewpoints under gravitational and inertial constraints.The capability targets applications such as search and rescue and remote inspection where GPS may be unreliable or target coordinates unknown.
- Limitations of Existing Methods: Existing approaches use dense oracle directional hints, turning navigation agents into passive instruction followers rather than active spatial reasoners.These hints are injected directly into input prompts from ground-truth trajectories.
- Limitations of Existing Methods: External object detectors frequently trigger landing, creating a disjointed perception-control loop that weakens robustness when detectors fail in open-world scenarios.The policy learns movement but relies on a separate black box to decide when to stop.
- AerialVLA: AerialVLA establishes a unified perception-action loop by replacing exact oracles with fuzzy onboard hints and mapping raw observations directly to continuous physical signals.The design supports active visual grounding, spatial reasoning, and integrated precision stopping within one policy.
- Contributions: The framework combines dual-view perception, fuzzy directional prompting, and numerical-tokenized 3-DoF control with intrinsic landing.Its contributions include streamlined front-and-down visual input, IMU-derived prompts, and continuous action outputs.
2 Related Work
UAV navigation research spans high-level planning, reasoning-centric foundation-model methods, and end-to-end control, each with different autonomy and physical-control trade-offs. AerialVLA targets the gap between hierarchical planning and constrained continuous-control systems through fuzzy prompting and intrinsic landing.
- UAV Navigation Benchmarks: UAV benchmarks cover instruction following, object navigation, spatial reasoning, mission planning, and continuous 3D navigation precision.TravelUAV is adopted to evaluate continuous maneuvering capabilities.
- High-Level Planning and Waypoint Prediction: High-level planning methods predict spatial waypoints for low-level controllers, supporting modular long-horizon, collaborative, or mission-generation workflows.Representative systems include CMA, AVDN, CityNavAgent, SkyVLN, AeroDuo, and UAV-VLA.
- Training-Free and Reasoning-Centric Approaches: Training-free reasoning approaches use frozen foundation models for waypoint or code generation, but 2D waypoint unprojection can violate physical constraints and sequential generation creates inference latency.SPF predicts 2D waypoints for heuristic 3D unprojection, while TypeFly generates code primitives for task planning.
- End-to-End Continuous Control: Existing end-to-end continuous-control systems map observations to flight signals but remain mainly confined to structured racing or discrete gate-selection environments.CognitiveDrone also separates lower-frequency auxiliary VLM reasoning from control.
- AerialVLA: AerialVLA bridges hierarchical planning and constrained continuous control by combining fuzzy directional prompting with intrinsic landing for open-world maneuvering without external detectors or heavy reasoning chains.Its architecture processes multimodal inputs end-to-end and decodes numerical tokens into control actions.
3 Method
AerialVLA combines minimalist dual-view perception, fuzzy sensor-derived prompting, geometry-consistent supervision, and unified numerical control to support reactive UAV navigation and landing. Its behavior-cloning objective predicts discretized action tokens from visual observations and structured prompts.
- Minimalist Dual-View Perception: AerialVLA uses only front and down views, concatenated into one composite image and processed by SigLIP-DINOv2 visual encoding.The dual-view design reduces multi-camera redundancy while retaining cues for forward navigation and target grounding.
- Fuzzy Directional Prompting: Fuzzy directional prompts discretize onboard-sensor-derived target bearing into coarse semantic hints instead of dense step-by-step oracle guidance.The coarse formulation introduces spatial ambiguity and supports reactive visual grounding without precise trajectory alignment.
- Geometry-Consistent Supervision: Geometry-consistent filtering removes ambiguous straight-flight labels when lateral space is clear, while retaining them near obstacles as valid evasion maneuvers.The procedure evaluates lateral bearing, expert yaw rate, and lateral depth, removing approximately 4% of training frames.
- High-DoF Control via Numerical Tokenization: The unified action space represents altitude, forward progression, and heading through continuous 3-DoF commands, with landing triggered by LAND or near-zero predicted displacement.Terminal frames use the zero-displacement vector ⟨0, 0, 0⟩ and LAND token, integrating navigation and termination in one objective.
- High-DoF Control via Numerical Tokenization: AerialVLA maps three discretized action dimensions to existing numerical tokens, dequantizes them into physical offsets, and executes them through velocity-duration control.The model predicts integer tokens in 99 bins; commands are converted to motion using a constant 1.0 m/s cruise speed.
- Training Objective: Training uses frame-level behavior cloning to maximize the autoregressive likelihood of expert action tokens conditioned on visual observations and structured prompts.The action tokens are predicted dimension by dimension for x, z, and ψ.
4 Experiments
AerialVLA is evaluated as a fully autonomous end-to-end policy on TravelUAV across seen and unseen settings, with comparisons against modular and generalist baselines. It achieves strong performance, generalization, qualitative robustness, and ablation-supported design benefits.
- Experimental Setup: Evaluation uses TravelUAV’s UAV-Need-Help task, with 7,922 training trajectories and Seen, Unseen Object, and Unseen Map test splits.Metrics include Navigation Error, Success Rate, Oracle Success Rate, and Success weighted by Path Length.
- Baselines: AerialVLA is compared with heuristic, hybrid VLM, and recent generalist baselines, whose systems use specialized decoders or auxiliary modules.TravelUAV additionally relies on Grounding DINO for target detection, while NavFoM and LongFly use specialized planning or spatiotemporal components.
- Quantitative Results: 47.96% SR and 38.54% SPL on Seen environments surpass LongFly by +11.57% SR and +7.47% SPL.On the Hard split, AerialVLA’s SR advantage widens to +12.36%; its intrinsic stopping mechanism also supports terminal maneuvers without external oracle triggers.
- Quantitative Results: 56.60% SR on Unseen Objects shows that AerialVLA maintains superiority when target categories are novel.The reported result is associated with directly grounding novel visual concepts to control actions rather than relying on explicit object detectors.
- Quantitative Results: 37.58% SR and 28.22% SPL on Unseen Maps are approximately three times LongFly’s 11.27% SR, demonstrating strong zero-shot adaptability.The approach relies on instantaneous observations rather than accumulated spatial memory.
- Qualitative Analysis: Qualitative trajectories show precision maneuvering in clutter and active error correction after approaching distractors.The agent aligns horizontally before descending in clutter and can hover, recognize a mismatch, climb, and resume searching under ambiguity.
- Ablation Study: Ablations show robustness to raw demonstrations, while redundant five-view inputs reduce unseen-map SR from 37.58% to 21.71%.Custom action tokens also cause significant SR and SPL drops across all splits, supporting the dual-view and numerical-tokenization designs.
5 Limitations and Future Work
AerialVLA’s minimalist reactive design has limitations in global backtracking and extreme out-of-distribution exploration. Future work targets lightweight memory and reinforcement-learning fine-tuning.
- Limitations: Reactive control can struggle with global backtracking in highly repetitive urban structures such as the NewYorkCity map.The limitation follows from prioritizing instantaneous control over explicit historical memory.
- Limitations: Behavior cloning makes the policy conservative in extreme out-of-distribution scenarios, including dense-canopy occlusions in ModularPark.In that setting, the agent defaults to safe hovering instead of executing complex multi-stage exploration.
- Future Work: Future work will add lightweight memory mechanisms for global reasoning and explore reinforcement-learning fine-tuning for active exploration.These directions are intended to address the reactive-memory and static-demonstration trade-offs.
6 Conclusion
AerialVLA establishes a minimalist end-to-end Vision-Language-Action paradigm for UAV navigation, using onboard fuzzy hints and one autonomous policy for cruising and precise termination. The approach decouples the agent from dense oracle guidance and explicit object detectors, with evaluations reporting zero-shot generalization across unseen targets and novel map geometries.
- AerialVLA rethinks UAV vision-language navigation through a minimalist end-to-end Vision-Language-Action paradigm.
- The policy uses onboard fuzzy hints and unifies cruising with precise termination into a single autonomous policy.
- AerialVLA completely decouples the agent from dense oracle guidance and explicit object detectors.
- Evaluations report zero-shot generalization across unseen targets and novel map geometries after discarding redundant spatial memory and complex auxiliary modules.