Source-linked AI summary
Vision-Language Models as copilots for Autonomous UAV Navigation: Analysis of Latency and Reliability in Degraded Environments
Hiago Sodre, Sebastian Barcelona, Vincent Sandin, Pablo Moraes, Ahilen Mazondo, Igor Nunes, William Moraes, André Kelbouscas, Ricardo Grando
TL;DR
Real-time closed-loop UAV navigation needs VLMs to provide semantic guidance while meeting latency and structured-command reliability requirements. This paper evaluates an asynchronous VLM copilot over an FSM in SITL simulation, and reports that structured reasoning integrity matters more to mission success than latency alone.
Problem
Method
The paper evaluates an asynchronous VLM copilot advising an FSM, with latency-compensation mechanisms, in a SITL UAV navigation framework.
Source: §1 · p. 1Results
0.255 m mean tracking error and 78.6% mission success rate were achieved by Llama 3.2 Vision (11B), the most robust degraded-environment configuration; mission success depended more on structured reasoning integrity than latency alone.
Source: §5 · p. 6, §4.2 · p. 6Takeaways & Limitations
When asynchronous inputs are properly handled, semantic reasoning reliability is more important than low response latency for degraded-environment navigation.
Source: §5 · p. 6Further work could optimize local inference through bit quantization without sacrificing reasoning.
Source: §5 · p. 6
Abstract
from arXiv · showhide
The integration of Vision-Language Models (VLMs) in autonomous Unmanned Aerial Vehicles (UAVs) offers unprecedented semantic reasoning capabilities. However, real-time closed-loop navigation requires not only low inference latency but also obedience to structured flight commands. This paper proposes a hybrid FSM-VLM control architecture for UAVs in GPS-free environments. The system combines a deterministic Finite State Machine (FSM) for low-level physical control with an asynchronous VLM copilot for high-level semantic pathfinding. We evaluate three models with different parameter scales in a Software-In-The-Loop (SITL) simulation. The framework isolates and measures syntax errors at the format level versus semantic hallucinations at the logic level in a normal and degraded scenario. This study demonstrates that parameter scaling, and not pure latency, remains the primary bottleneck for the safe and compatible integration of VLM into autonomous flights.
1. INTRODUCTION
The introduction motivates VLMs for UAV navigation where visual ambiguity challenges classical perception, but real-time control also demands low latency and reliable structured outputs. It proposes an asynchronous VLM copilot over an FSM, with latency compensation and a SITL evaluation framework.
1. INTRODUCTION
- VLMs add semantic context to perception when visual cues are ambiguous, occluded, or damaged, while high inference latency and unstructured outputs challenge real-time robotic control.Prior end-to-end visuomotor and reinforcement-learning approaches are described as lacking the interpretability of a structured communication protocol.Source: §1 · p. 1
- The proposed hybrid architecture pairs an asynchronous VLM copilot with a traditional Finite State Machine for UAV navigation.The framework targets GPS-free indoor navigation and combines semantic understanding with autonomous recovery.Source: §1 · p. 1
- The system uses adaptive speed scaling and pose-snapshot rewind to maintain flight stability during asynchronous VLM inference delays.The drone slows while the model processes an image and can return to the image-capture position to act on identified intersections.Source: §1 · pp. 1–2
- The evaluation pipeline compares VLM architectures across parameter scales, latency, semantic correctness, and mission completion in closed-loop SITL simulation.Its degraded-environment protocol quantifies robustness under partial infrastructure failure and examines when semantic reasoning outperforms reactive baselines.Source: §1 · p. 2
2. RELATED WORKS
Related work describes VLMs as a route to semantic visual reasoning, while noting latency and structured-output reliability challenges in real-time robotics. The section identifies a gap in closed-loop evaluation of structured reasoning failures and frames this paper as a controlled evaluation framework.
2. RELATED WORKS
- Classical perception performs less effectively when visual cues are ambiguous, occluded, or degraded, limiting higher-level decision-making based on geometric features.VLMs are presented as adding semantic reasoning to perception.Source: §2 · p. 2
- VLMs show visual-understanding and instruction-following capabilities, but real-time robotic use remains limited by latency and the lack of guarantees for structured outputs.The section names CLIP, BLIP-2, and LLaVA as examples.Source: §2 · p. 2
- Prior visuomotor and reinforcement-learning methods demonstrate data-driven sequential decision-making but do not address free-form generative outputs in safety-critical control loops.The section notes that these methods typically operate within predefined action spaces.Source: §2 · p. 2
- Systematic closed-loop evaluation of VLM structured reasoning is missing, and structural failures versus semantic inconsistencies remain underexplored in real-time settings.The paper responds with a controlled framework to quantify failure modes and examine latency and structured-reasoning integrity across model scales.Source: §2 · p. 2
3. SYSTEM OVERVIEW AND EXPERIMENTAL DESIGN
A deterministic FSM handles low-level flight control while an asynchronous VLM provides semantic advice; the study evaluates this hybrid design across UAV missions and normal and degraded conditions.
3.1 Hybrid UAV Navigation System
- The FSM runs deterministic OpenCV-based control at 30–50 Hz, while asynchronous VLM decisions enter as advisories without interrupting the control loop.The architecture separates low-level flight stability from high-level semantic reasoning.Source: §3.1 · pp. 2–3, Figure 2 · p. 2
3.2 Missions Design
- Benchmark missions cover delivery, tower inspection, urban search and rescue, and a full sequential run ending in precision landing.The individual tasks include marker-centered delivery, QR-code inspection during orbits, and multi-altitude facade search; the full run combines all three missions.Source: §3.2 · p. 3
3.4 Structured VLM Interface
- A predefined JSON schema governs VLM outputs; validation lets the system distinguish invalid formats from semantically incorrect decisions before FSM acceptance.Responses that violate the schema are discarded to prevent unsafe behavior.Source: §3.4 · p. 3, §3.5 · p. 3
3.5 Metrics and Data Collection
- Collected measures include average and maximum latency, mean and peak tracking error, query counts, valid outputs, format errors, and logic errors.Format errors are invalid JSON responses; logic errors are semantically incorrect decisions.Source: §3.5 · p. 3
3.6 Experimental Protocol
- The simulated quadrotor uses downward RGB for line and QR detection and forward-facing RGB-depth stereo for obstacle avoidance.The protocol compares four configurations under identical low-level control, sensor, and mission settings.Source: §3.6 · p. 4
3.3 VLM Models and Benchmark Configuration
- The comparison includes a no-VLM FSM baseline and Moondream (1.8B), LLaVA-LLaMA3 (8B), and LLaMA 3.2 Vision (11B), run locally using Ollama.Across configurations, low-level control, sensors, and mission parameters remain identical.Source: §3.3 · p. 3, §3.6 · p. 4
3.6 Experimental Protocol
- With 50% of QR textures corrupted, the degraded environment caused catastrophic benchmark failure, while large VLMs showed superior resilience.The corrupted textures could not be decoded by the classical QR reader, forcing reliance on the VLM or failure at intersections.Source: §3.6 · p. 4
4. RESULTS AND DISCUSSION
In normal conditions, VLMs preserved flight stability, but their format integrity and mission success varied; in degraded conditions, the larger Llama 3.2 Vision model performed best despite the highest latency. The results associate degraded-scene mission success more with structured reasoning integrity than inference speed, while backtracking helped the VLM system maintain mission continuity.
4.1 Performance Across Environmental Configurations
- All configurations remained stable in normal visual conditions; Moondream 2 had the lowest Mean Tracking Error at 0.161 m, versus 0.191 m for FSM-CV.Llama 3.2 Vision measured 0.167 m and Llava-Llama3 0.202 m Mean Tracking Error.Source: §4.1 · pp. 4–5
- Normal-condition mission success and format integrity diverged: FSM-CV and Llama 3.2 Vision reached 100.0% Success Rate, while Moondream 2 failed despite 0.702 s latency and 14.5% Format Integrity.Llava-Llama3 reached 99.5% Format Integrity and 80.0% Success Rate.Source: §4.1 · p. 5
- In degraded conditions, the FSM-CV baseline failed despite a 0.798 m Max Tracking Error; Llama 3.2 Vision reached 78.6% Success Rate, while Llava-Llama3 reached 33.3% and Moondream 2 reached 1.4% Format Integrity.Moondream 2’s mean and maximum tracking errors rose to 2.744 m and 28.30 m, respectively.Source: §4.1 · p. 5
4.2 Discussion on Resilience and Reasoning Integrity
- In degraded Mission 1, the VLM system used latency-aware backtracking to return to image-capture coordinates before turns, maintaining mission continuity despite 4–7 s inference latency.The CV baseline failed at the intersection amid visual noise and QR-code blur.Source: §4.2 · p. 5, Figure 7 · p. 5
- 78.6% Success Rate: Llama 3.2 Vision (11B) also achieved 100.0% Format Integrity and the lowest VLM tracking error in degraded conditions.Its mean tracking error was 0.255 m and maximum was 1.519 m, despite the highest average latency (5.67 s).Source: §4.2 · pp. 5–6, Table 1 · p. 6
- Latency alone did not explain mission performance: tracking errors stayed stable when guidance was semantically correct, while Llama 3.2 Vision was over eight times slower than Moondream 2 on average.The FSM maintains low-level control while the VLM contributes high-level reasoning.Source: Figure 8 · p. 5, §4.2 · p. 6
5. CONCLUSIONS
The study evaluates asynchronous VLM copilots for UAV navigation in compromised environments and finds that the most robust degraded-environment configuration used Llama 3.2 Vision (11B). Its results suggest semantic reasoning reliability matters more than low response latency when asynchronous inputs are properly handled.
5. CONCLUSIONS
- The study evaluates the relationship between inference latency and semantic reasoning integrity under real-time constraints across local VLM backends and computer vision.It focuses on compromised environments rather than proposing a new model architecture.Source: §5 · p. 6
- Llama 3.2 Vision (11B), with Latency-Aware Backtracking and Adaptive Speed Scaling, achieved 0.255 m mean tracking error and 78.6% mission success in degraded environments.The paper concludes that semantic reasoning reliability is more important than low response latency when asynchronous inputs are properly handled.Source: §5 · p. 6
- Future work could explore bit quantization to reduce the local inference bottleneck without sacrificing reasoning.Source: §5 · p. 6