PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild

Xiaolou Sun, Wufei Si, Wenhui Ni, Yuntian Li, Dongming Wu, Fei Xie, Runwei Guan, He-Yang Xu, Henghui Ding, Yuan Wu, Yutao Yue, Yongming Huang, Hui Xiong

arXiv:2602.09657v1cs.RO

TL;DR

UAV VLN commonly assumes detailed instructions and predetermined routes, whereas unknown outdoor environments provide only coarse guidance and require autonomous planning and obstacle avoidance. AutoFly addresses this gap with an end-to-end VLA model, pseudo-depth features, progressive training, and an autonomous-navigation dataset. It achieves 3.9% higher navigation success rates than state-of-the-art baselines with consistent performance across simulated and real outdoor environments.

  • Problem

    Existing UAV VLN methods and datasets rely heavily on detailed instruction-following and insufficient real-world data, limiting their fit for autonomous navigation in unknown outdoor environments.

  • Method

    AutoFly combines a pseudo-depth encoder, progressive two-stage visual-depth-language-action alignment, and a dataset containing continuous autonomous workflows and extensive real-world trajectories.

  • Results

    3.9% higher navigation success rates than state-of-the-art VLA baselines are reported, with consistent performance across simulated and real outdoor environments.

  • Takeaways & Limitations

    AutoFly supports autonomous UAV navigation without detailed instructions using minimal linguistic guidance and coarse positional or directional information.

  • Takeaways & Limitations

    AutoFly has limited global exploration in large-scale environments and limited environmental context because it relies on forward-facing RGB imagery and monocular depth estimation.

Abstract

from arXiv · show

Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations, serving as a cornerstone task in Embodied AI. Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes. However, real-world outdoor exploration typically occurs in unknown environments where detailed navigation instructions are unavailable. Instead, only coarse-grained positional or directional guidance can be provided, requiring UAVs to autonomously navigate through continuous planning and obstacle avoidance. To bridge this gap, we propose AutoFly, an end-to-end Vision-Language-Action (VLA) model for autonomous UAV navigation. AutoFly incorporates a pseudo-depth encoder that derives depth-aware features from RGB inputs to enhance spatial reasoning, coupled with a progressive two-stage training strategy that effectively aligns visual, depth, and linguistic representations with action policies. Moreover, existing VLN datasets have fundamental limitations for real-world autonomous navigation, stemming from their heavy reliance on explicit instruction-following over autonomous decision-making and insufficient real-world data. To address these issues, we construct a novel autonomous navigation dataset that shifts the paradigm from instruction-following to autonomous behavior modeling through: (1) trajectory collection emphasizing continuous obstacle avoidance, autonomous planning, and recognition workflows; (2) comprehensive real-world data integration. Experimental results demonstrate that AutoFly achieves a 3.9% higher success rate compared to state-of-the-art VLA baselines, with consistent performance across simulated and real environments.

1 INTRODUCTION

AutoFly addresses the mismatch between detailed waypoint-based UAV navigation and unknown outdoor settings where only coarse guidance is available. It combines autonomous VLA navigation with a pseudo-depth encoder, progressive training, and a dataset centered on real-world autonomous behavior.

  • Current UAV VLN methods rely on step-by-step instructions specifying predetermined flight paths, explicit waypoints, and maneuvers.
  • AutoFly performs autonomous UAV navigation in unknown outdoor environments using concise language and coarse positional or directional guidance.The model directly outputs actions for navigation based on coarse guidance.
  • The model uses a pseudo-depth encoder and progressive two-stage training to align visual, depth, linguistic, and action representations.Pseudo-depth features are derived from RGB inputs to enhance spatial reasoning and multimodal alignment.
  • The proposed dataset emphasizes continuous planning, obstacle avoidance, and object recognition while incorporating extensive real-world trajectories.This shifts data collection beyond discrete instruction-following workflows and supports sim-to-real transfer.
  • 3.9% higher navigation success rates than state-of-the-art VLA baselines are reported, with consistent performance across simulated and real outdoor environments.

2 RELATED WORK

Related work largely extends instruction-following and modular UAV navigation paradigms, while AutoFly targets autonomous decision-making with coarse guidance. Its design adds pseudo-depth spatial representations and unifies high-level navigation primitives in an end-to-end VLA model.

  • UAV VLN extends language-guided navigation to three-dimensional flight with variable terrains and dynamic obstacles.
  • Unknown real-world scenarios often provide only coarse positional or directional guidance, motivating end-to-end autonomous navigation.
  • High-Level Navigation Primitives for UAVs: Traditional UAV systems commonly implement path planning, object recognition, and obstacle avoidance as independent modules.
  • High-Level Navigation Primitives for UAVs: AutoFly integrates path planning, object recognition, and obstacle avoidance into a unified end-to-end VLA architecture for coordinated decision-making.
  • Vision-Language-Action Models: Unlike RGB-only approaches, AutoFly extracts spatial representations from monocular RGB inputs through a pseudo-depth encoder without additional depth sensors.
  • AutoFly takes RGB observations and linguistic instructions, combines its high-level actions with initial coarse-guidance actions, and forms action sequences.

3 METHOD

AutoFly formulates UAV navigation as an end-to-end policy that combines RGB observations, language, and coarse guidance to produce executable control actions. Its method integrates pseudo-depth spatial representations, multimodal action decoding, autonomous-navigation data, and behavior-balancing strategies.

  • VLA Model for Autonomous Navigation: AutoFly learns a policy mapping RGB observations, language, and coarse positional or directional guidance to low-level UAV control actions.The policy aims to generate collision-free trajectories under UAV kinodynamic constraints.
  • VLA Model for Autonomous Navigation: The architecture combines a vision-language model, pseudo-depth encoder, and action de-tokenizer for autonomous navigation.The pseudo-depth encoder derives spatial representations from monocular RGB inputs, while the de-tokenizer converts model outputs into executable actions.
  • VLA Model for Autonomous Navigation: Pseudo-depth processing generates depth maps from RGB inputs and projects patch embeddings into the visual feature space for dimensional and spatial alignment.The projector preserves depth-aware spatial relationships while matching depth-token and vision-token dimensions.
  • VLA Model for Autonomous Navigation: The action de-tokenizer maps discrete language-model action tokens into continuous three-dimensional action vectors.AutoFly uses the final 256 LLaMA2 vocabulary tokens as the action-mapping space.
  • Autonomous Navigation Dataset: The dataset combines simulated and real-world trajectories for autonomous planning, obstacle avoidance, and object recognition.It includes diverse AirSim environments with irregular obstacles and trajectories generated from random starts toward designated targets.
  • Autonomous Navigation Dataset: Dataset rebalancing addresses long-horizon behavior imbalance by increasing balanced exposure to navigation behaviors during training.Obstacle avoidance otherwise dominates target-seeking phases; the imbalance is quantified by approximately 0.36 nats of KL divergence from uniformity.
  • Autonomous Navigation Dataset: The dataset introduces average obstacle encounter as a metric for environmental complexity and obstacle-avoidance capability.This supplements navigation success rates with a quantitative measure of a critical but under-measured behavior.

4 EXPERIMENTS

The experiments evaluate AutoFly using standard navigation metrics, real-world deployment, transfer-learning tests, and component ablations. Results support the value of pseudo-depth encoding, specialized depth projection, trajectory rebalancing, and depth-vision-language alignment.

  • Evaluation Details: Success rate, collision rate, and path efficiency rate assess navigation success, obstacle collisions, and planning efficiency across successful trials.Success requires both proximity and orientation according to the dataset’s evaluation protocol.
  • Real-World Performance: AutoFly is evaluated in indoor laboratory and outdoor campus-forest environments using 10 object instances and 20 independent trials per target in each setting.The outdoor setting includes irregular trees, swaying branches, and unstructured vegetation.
  • Transfer Learning: Increasing simulation data produces progressive performance gains during simulation-to-real transfer, even with limited real-world fine-tuning.The study varies simulation-to-real data ratios to evaluate transfer-learning efficacy.
  • Pseudo-Depth Encoder Ablation: 47.9% success and 21.9% collision rates with pseudo-depth outperform 44% success and 24.5% collision without it.The ablation attributes the improvement to introducing spatial information.
  • Depth Projector Validation: 47.9% success, 21.9% collision, and 77.3% path efficiency with the specialized depth projector outperform direct SigLIP and DINOv2 depth processing.The authors link the gap to general-purpose methods’ difficulty extracting geometric patterns from depth maps.
  • Depth-Vision-Language Alignment Analysis: Depth-vision-language alignment achieves 47.9% success with a shared Siamese projector, versus 43.3% with separate projectors and 26.7% with direct depth input.The shared projector has a 3.3% advantage over the non-Siamese variant.

5 CONCLUSION

AutoFly is an end-to-end Vision-Language-Action model for autonomous UAV navigation using minimal linguistic guidance. Its reported advantages include a 3.9% higher navigation success rate than state-of-the-art baselines and consistent performance across simulated and real environments.

  • Conclusion: AutoFly uses a pseudo-depth encoder, a multimodal dataset, progressive training, and Siamese MLP projectors for cross-modal alignment.These components support navigation from minimal linguistic guidance.
  • Conclusion: 3.9% higher navigation success rates than state-of-the-art baselines are reported for AutoFly.The conclusion frames this result as enabling autonomous navigation without detailed instructions.
  • Conclusion: AutoFly maintains consistent performance across simulated and real outdoor environments.The conclusion identifies applications including search and rescue, environmental monitoring, and autonomous delivery systems.

A.1 ABSTRACT

The appendix supplies supplementary materials covering data collection, experiments, methodology, deployment, visualizations, and limitations. It also identifies dedicated sections for real-world results and future work.

  • Extended Data Collection Details: Extended data collection details cover dataset construction, dataset splits, an RL-based collection algorithm, and dataset rebalancing methodology.These materials are listed under the first supplementary section.
  • Extended Experimental Results: Extended experimental results include robot setup, vision encoder ablations, challenging-scenario evaluation, and simpler alternative approaches.These materials are listed under the second supplementary section.
  • Extended Methodology Explanation: Extended methodology explanation includes baseline method architectures.This material is listed under the third supplementary section.
  • Extended Deployment Strategy: Extended deployment strategy covers distributed architecture, network communication, model acceleration, and parallel inference.These materials are listed under the fourth supplementary section.
  • Visualizations and Discussion: Visualizations and discussion include simulation visualizations and real-world experimental results.The appendix lists these materials under a dedicated visualizations section.
  • Limitations and Future Work: A separate appendix section addresses limitations and future work.The section is listed as item 6.

A.2.1 DATASET CONSTRUCTION

The dataset combines simulated and real-world flight data and evaluates navigation using proximity and orientation criteria. Simulated scenarios and object instances are divided across training, seen-test, and unseen-test categories to assess generalization.

  • Dataset Construction: The dataset combines 12 custom 70m × 70m AirSim scenes with real-world data collected through manual flights in controlled laboratories.This dual-source construction integrates simulation and real-world data acquisition.
  • Scene Configuration: Simulated scenes contain colored pillars, tree clusters, and stacked boxes, with intra-scene obstacles for avoidance and boundary targets for recognition.Object instances are randomly positioned at scene boundaries from a predefined pool.
  • Evaluation Criteria: Navigation success requires both spatial proximity and proper target orientation rather than proximity alone.This criterion is intended to validate spatial reasoning and target-oriented navigation.
  • Success Metrics: A successful episode requires reaching within 5 meters of the target with angular deviation ≤15 degrees.These thresholds represent the dataset’s stated UAV operational requirements.
  • Scenario Splits: The scenarios comprise 10 seen training scenarios, 2 unseen scenarios, and 2 reconfigured seen scenarios for generalization evaluation.The reconfigured scenarios alter layouts while retaining identical constituent elements.
  • Object Splits: Object instances use a 50/50/10 training/test-seen/test-unseen split across vehicles, furniture, and animals.The split supports evaluation across diverse semantic object categories.

A.2.2 DATASET SPLIT

The dataset combines over 13K episodes and 2.5M image-language-action triplets for training, with evaluation spanning seen and unseen scenes and targets in simulation.

  • The training set contains 10 scenes, 50 object instances, over 13K episodes, and 2.5M image-language-action triplets.Each sample uses the format [observation, command, action].
  • Evaluation uses 4 testing scenes, including 2 training scenes and 2 completely unseen scenes, with 60 targets comprising 50 seen and 10 unseen targets.The setup tests generalization across both environments and targets.
  • The evaluation design contains 4 conditions with 30 trials per condition, totaling 7,200 evaluation episodes.

A.2.3 DATA COLLECTION ALGORITHM BASED ON RL

The data collection algorithm replaces costly manual demonstrations and weak traditional planners with reinforcement-learning obstacle-avoidance agents trained using SAC and depth-only inputs.

  • Manual collection is prohibitively expensive, while A* and D* degrade in dense-obstacle environments and produce longer, suboptimal trajectories than expert demonstrations.
  • Specialized reinforcement-learning agents are trained independently for each scene to output velocity commands directly, avoiding trajectory-to-command conversion.The agents use CNN architectures with downsampling layers and MLP heads, trained with Soft Actor-Critic on depth-only inputs.
  • Each trained agent achieves a 95% success rate in its respective scene during evaluation.This supports the effectiveness of the automated collection strategy for generating high-quality training data at scale.
  • The critic network learns from replay-buffer samples by minimizing the squared difference between its Q-value estimate and target value y.The passage defines Qθ(s, a) as expected cumulative reward and y as the supervision target.
  • For stochastic policies, the target combines immediate reward with discounted future value when episodes continue, while terminated episodes use only the immediate reward.The minimum of two target-network Q estimates mitigates Q-value overestimation.
  • The entropy coefficient is optimized alongside policy behavior, with target entropy empirically set to −dim(A), where A is the action space.This reflects the stated joint optimization of policy entropy and expected cumulative reward.
  • Statistical comparisons across navigation efficiency, safety, and completion metrics find RL trajectories comparable in quality to expert demonstrations.The comparison uses identical scenarios and supports scalable automated collection while maintaining training-data quality.

A.2.4 DATASET REBALANCING

The dataset-rebalancing framework segments trajectories into obstacle-avoidance and target-seeking phases, measures their imbalance, and resamples sub-trajectories toward a target distribution. The broader training and evaluation setup combines simulation and real-world data, while ablations assess perception and pseudo-depth contributions to navigation performance.

  • Trajectory segmentation: Trajectory segmentation separates obstacle-avoidance and target-seeking sub-trajectories to address long-horizon behavioral imbalance.The segmentation function assigns phase 1 to obstacle avoidance and phase 2 to target seeking; Grounding DINO detects the transition when the target exceeds a confidence threshold.
  • Imbalance measurement: The dataset contains approximately 73% obstacle-avoidance and 27% target-seeking data, corresponding to an imbalance of approximately 0.36 nats.Imbalance is measured by KL divergence from a uniform phase distribution.
  • Resampling: For a uniform target distribution, resampling downsamples obstacle avoidance with w1 ≈ 0.68 and upsamples target seeking with w2 ≈ 1.85.Stratified resampling samples phase-specific sub-trajectories with or without replacement according to the resulting weights.
  • Vision-encoder ablation: DINO-SigLIP fusion achieves the best vision-encoder result at 47.9% success rate, improving 1.3% over SigLIP alone.SigLIP reaches 46.6%, compared with 43.1% for CLIP and 45.2% for DINO; the reported analysis attributes the fusion benefit to complementary spatial and visual-language strengths.
  • Pseudo-depth ablation: The pseudo-depth encoder improves navigation across challenging scenarios, with the largest reported gain occurring for dynamic obstacles at +9.9% success rate.The dynamic-obstacle setting also shows a 9.5% collision-rate reduction, while the dense-cylinder and dense-forest analyses report scenario-specific gains.
  • Alternative approaches: Adding training data improves success rate by +1.2%, but remains below the pseudo-depth encoder’s +3.9% gain.Data augmentation and a stronger vision encoder yield +0.3% and +0.7% success-rate improvements, respectively.
Loading 2602.09657v1…