Source-linked AI summary
See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation
Chih Yao Hu, Yang-Sen Lin, Yuna Lee, Chih-Hai Su, Jie-Ying Lee, Shr-Ruei Tsai, Chin-Yang Lin, Kuan-Wen Chen, Tsung-Wei Ke, Yu-Lun Liu
TL;DR
Autonomous UAV navigation must combine visual reasoning, language understanding, and action control while generalizing beyond curated training data. SPF addresses this gap with training-free 2D waypoint grounding, geometric 3D action conversion, and adaptive closed-loop control, achieving strong simulated and real-world results.
Problem
Existing UAV navigation policies depend on limited curated demonstrations, while direct VLM text actions are unsuitable for precise actions and 3D aerial control.
Method
SPF uses VLMs to iteratively mark 2D image waypoints, unprojects them into 3D motion commands, and adaptively scales travel distance in closed loop.
Results
SPF achieves 93.9% success in 23 simulated tasks and 92.7% in 11 real-world tasks, outperforming prior methods.
Takeaways & Limitations
SPF provides a model-agnostic, hardware-friendly approach for zero-shot navigation across long-horizon, ambiguous, and dynamic tasks.
Takeaways & Limitations
Performance remains sensitive to VLM inaccuracies, prompt phrasing, imprecise adaptive depth estimation, and inference latency of approximately 1–3 seconds for highly dynamic obstacles.
Abstract
from arXiv · showhide
We present See, Point, Fly (SPF), a training-free aerial vision-and-language navigation (AVLN) framework built atop vision-language models (VLMs). SPF is capable of navigating to any goal based on any type of free-form instructions in any kind of environment. In contrast to existing VLM-based approaches that treat action prediction as a text generation task, our key insight is to consider action prediction for AVLN as a 2D spatial grounding task. SPF harnesses VLMs to decompose vague language instructions into iterative annotation of 2D waypoints on the input image. Along with the predicted traveling distance, SPF transforms predicted 2D waypoints into 3D displacement vectors as action commands for UAVs. Moreover, SPF also adaptively adjusts the traveling distance to facilitate more efficient navigation. Notably, SPF performs navigation in a closed-loop control manner, enabling UAVs to follow dynamic targets in dynamic environments. SPF sets a new state of the art in DRL simulation benchmark, outperforming the previous best method by an absolute margin of 63%. In extensive real-world evaluations, SPF outperforms strong baselines by a large margin. We also conduct comprehensive ablation studies to highlight the effectiveness of our design choice. Lastly, SPF shows remarkable generalization to different VLMs. Project page: https://spf-web.pages.dev
1 National Yang Ming Chiao Tung University, 2 National Taiwan University
The paper concerns vision-language models, zero-shot UAV navigation, and 2D-to-3D waypoint prompting.
- SPF targets zero-shot UAV navigation guided by vision-language models.
- The framework uses 2D-to-3D waypoint prompting for aerial control.
- Its scope combines visual-language modeling with unmanned aerial navigation.
1 Introduction
SPF addresses the difficulty of generalizing autonomous UAV navigation across visual reasoning, language understanding, and action control. It replaces text-based action generation with grounded 2D waypoint prediction, lifting waypoints into 3D commands for zero-shot, closed-loop navigation, and reports strong benchmark results.
- Conventional end-to-end policies rely on curated expert demonstrations but struggle with unseen environments and task instructions because training data are limited in scale and diversity.
- Text-generation approaches are ill-suited to precise floating-point actions and lack aerial-navigation training for predicting 3D movements.
- SPF predicts image-space waypoints, converts them into 3D actions, and adaptively scales travel distance for closed-loop UAV control.
- SPF is designed to generalize to novel scenes and free-form instructions without task-specific training.
- 63% success-rate margin over prior state-of-the-art methods establishes SPF’s new DRL simulation benchmark result.
2 Related Work
Related work spans learned UAV navigation policies and training-free VLM-based control. The paper motivates SPF as a zero-shot alternative to data-limited policy learning and direct textual action generation.
- Imitation- and reinforcement-learning methods train UAV policies from expert demonstrations or environmental rewards for navigation and control.
- Limited training-data scale and diversity restrict the generalization of end-to-end navigation frameworks to unseen environments and instructions.
- Training-free VLM approaches convert visual and language inputs into UAV actions, including text-based actions and semantic-map representations.
3 Method
SPF formulates UAV navigation as iterative 3D target reaching, using a VLM to ground language instructions in image-space waypoints and a geometric controller to execute them. Adaptive step sizing, 2D-to-3D unprojection, obstacle awareness, and closed-loop replanning support zero-shot navigation from RGB inputs.
- SPF formulates UAV navigation as an iterative target-reaching process in 3D space, mapping observations and language instructions to feasible displacement vectors.
- 3.1 VLM-based Obstacle-Aware Action Planning: The VLM converts vague instructions and current visual observations into interpretable waypoint decisions that can be converted into UAV control signals.
- 3.4 Reactive Control Loop Execution: A closed-loop controller repeatedly updates observations, replans with the VLM, and executes synchronized velocity commands without policy optimization, skill libraries, external depth sensors, or training.
- 3.1 VLM-based Obstacle-Aware Action Planning: The VLM outputs image coordinates (u, v) and a discretized depth label dVLM representing intended forward travel distance rather than sensed depth.
- 3.1 VLM-based Obstacle-Aware Action Planning: Obstacle-avoidance mode constrains waypoint generation toward the goal while avoiding intersections with detected object bounding boxes.
- 3.2 Adaptive Travel Distance Scaling: A nonlinear scaling curve converts dVLM into an adjusted step size dadj, with larger steps in open areas and smaller movements near targets and obstacles.
- 3.3 2D-to-3D Waypoint Transformation: The system unprojects (u, v, dadj) through a pin-hole camera model into a body-frame 3D displacement (Sx, Sy, Sz), then decomposes it into pitch, yaw, and throttle commands.
4 Experimental Results
SPF achieves strong simulation and real-world navigation performance, while ablations show that structured waypoint grounding, VLM choice, and adaptive travel scaling are central to its effectiveness and efficiency.
- Performance Evaluation: 93.9% average success rate in simulation, outperforming PIVOT (28.7%) and TypeFly (0.9%).SPF is especially strong on obstacle avoidance, long-horizon, and search tasks.
- Performance Evaluation: 92.7% average success rate in real-world experiments, completing all evaluated tasks while baselines often failed and achieving faster completion times.The evaluation covered five representative tasks, including obstacle avoidance, long horizon, reasoning, and follow categories.
- Performance Evaluation: SPF produces smoother, more effective trajectories than TypeFly and PIVOT in both simulated and real-world qualitative comparisons.The comparisons examine obstacle avoidance and reaching targets across simulator and real-world flights.
- Ablations: 100% success rate versus 7% for plain VLM and 40% for PIVOT demonstrates the benefit of structured 2D waypoint grounding for navigation.The ablation compares text-based action prediction, candidate-point selection, and SPF waypoint prompting.
- Ablations: SPF generalizes across VLMs, reaching 100% success with Gemini 2.5 Pro, Gemini 2.0 Flash, and GPT-4.1, while weaker models also achieve high success.Claude 3.7 Sonnet and Llama 4 Maverick reach 93.3%, and Gemini 2.0 Flash-Lite reaches 87%.
B Method Details
The method converts normalized VLM waypoint coordinates and an adaptive travel distance into a 3D displacement, then maps that displacement to UAV control primitives and durations.
- The structured VLM output provides waypoint coordinates and adaptive step size, which are normalized before position calculation.
- Normalized waypoint coordinates and adaptive distance are converted into a 3D displacement vector using the camera’s field-of-view angles.Sx = Unorm · dadj · tan(α), Sy = dadj, and Sz = Vnorm · dadj · tan(β).
- The 3D displacement vector is transformed into yaw, pitch, and throttle control primitives for the UAV.
- Each control primitive’s duration is calculated from its displacement and a predefined primitive speed.
B.2 UAV Command Queue Implementation
The command queue implementation translates waypoint-derived motion parameters into timed UAV commands, using predefined speeds and queued control primitives.
- The command queue sends enqueued roll, pitch, throttle, and yaw-rate commands to the UAV through a Python SDK.The implementation uses DJITelloPy and includes a reset command to stop pitch rate.
- Yaw, pitch, and throttle primitives use predefined speeds to determine their respective execution durations.
- The implementation is organized as Python-like pseudocode for a UAV command queue.
- The control pipeline represents waypoints as normalized 2D coordinates, converts them to a 3D displacement vector, and derives primitive durations.
- The camera’s horizontal and vertical field-of-view angles parameterize the spatial conversion from normalized waypoint coordinates.
C.1 Task Specifications and Evaluation Protocols
The evaluation classifies each UAV trial as success or failure using collision, task-completion, target-visibility, and distance criteria, with prompts covering simulated and real-world tasks.
- A trial succeeds when the requested task is completed or the target is visible within the specified final distance without collision.The distance threshold is 1 meter in the real world and 1–5 meters in simulation.
- A trial fails if the UAV collides or the target is not visible in the final egocentric camera view.
- The benchmark includes detailed prompt examples for simulated and real-world tasks.
C.2 Implementation Details
The implementation uses asynchronous VLM inference and low-level control with adaptive scaling, while qualitative findings are provided through demonstration videos.
- Adaptive scaling uses s = 10, L = 10, dmin = 0.1m, and p = 1.8.
- VLM inference runs at approximately 0.3–1 Hz, low-level commands at approximately 10 Hz, and end-to-end latency is approximately 1.5–3 seconds.The latency is primarily attributed to VLM inference time.
- Unless otherwise specified, experiments use Gemini 2.0 Flash as the VLM backend.
- Qualitative experimental results are presented as demonstration videos viewable through an included browser interface.
C.4 Experiment Setup of Adaptive Travel Distance Scaling
A real-world ablation compared adaptive and fixed step-size control across three tasks designed to test long-horizon planning and reasoning. The adaptive controller reduced completion times while maintaining or improving success rates.
- The ablation used a DJI Tello EDU, low-level rc velocity commands, three tasks, and five trials per controller configuration.The tasks targeted long-horizon planning and reasoning in real-world scenarios.
- 61s to 28s, with 100% SR, on “Fly to the cones and the next.” under adaptive control.The adaptive controller more than halved completion time relative to the fixed baseline.
- 50.25s to 35.20s and 80% to 100% SR on “I’m thirsty. Find something that can help me.”Adaptive control improved both completion time and success rate for this reasoning task.
- 47s to 30s, with 100% SR, on “It’s raining...” under adaptive control.The adaptive controller substantially reduced completion time while preserving full success.