PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenFlyV2: An onboard vision-language-action model for aerial perception and navigation

Junli Liu, Zhongrui You, Chenhui Li, Zhen Li, Yuyang Tian, Qizhen Chen, Delin Qu, Haoming Song, Dong Wang, Xuelong Li, Zhigang Wang, Bin Zhao

arXiv:2610.39287v1cs.RO

TL;DR

Aerial robots must unite visual-language understanding and continuous navigation while operating within onboard computational limits. OpenFlyV2 combines a shared vision-language representation with motion-aware memory and action generation, and achieves a 48% success rate in physical flight without real-world aerial training data.

  • Problem

    Aerial systems must combine scene understanding, language grounding, spatial reasoning, and closed-loop navigation onboard, despite limited compute and demands from changing viewpoints and 3D motion.

    Source: §1 · p. 2

  • Method

    OpenFlyV2 uses a shared vision-language backbone for understanding and an action expert for flight, with motion-adaptive memory and motion-structured priors to reduce onboard computation.

    Source: §1 · p. 3

  • Results

    48% aggregate navigation success across four real-world environments, using a single checkpoint without real-world flight training data.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results show that broad aerial vision-language capabilities and closed-loop flight can coexist in one model operating on a resource-constrained aerial platform.

    Source: §1 · p. 3

    Reliable aerial VLN in real-world environments remains a substantial challenge, and practical systems still require research on detecting deviations, recovering from errors, and resuming navigation under changing observations.

    Source: §3 · p. 7

Abstract

from arXiv · show

Human cognition flexibly draws on a shared perceptual foundation to understand environments, attend to referenced entities, reason about spatial relationships, and act. Endowing aerial robots with an analogous functional organization remains challenging, as existing systems are often specialized for individual capabilities, dependent on offboard computation, or unable to sustain closed-loop operation on resource-constrained platforms. We present OpenFlyV2, a unified onboard vision-language-action architecture that uses a shared vision-language representation to support semantic understanding, language-grounded perception, spatial reasoning, and continuous aerial control within a single model. A compact action expert converts the shared multimodal context into short-horizon flight waypoints, while motion-aware temporal memory and action priors reduce the computational cost of closed-loop execution. We construct \dataset{}, a simulation-based dataset comprising approximately 70,000 navigation trajectories and 60,000 vision-language instruction--response pairs. OpenFlyV2 generates action chunks, each comprising 12 waypoints, entirely onboard at 10 Hz using an NVIDIA Jetson Orin NX. It transfers zero-shot from simulation to physical quadrotor flight, achieving a 48\% navigation success rate across four real-world environments without real-world aerial training data. Extensive results demonstrate that broad vision-language competence and continuous closed-loop flight can coexist within a single model on a resource-constrained aerial platform. Code will be made publicly available at https://github.com/Ideal-ljl/AeroBrain.

1. Introduction

Aerial robots need to coordinate broad visual and language understanding with continuous onboard navigation, but capability integration, scalable supervision, and computational efficiency remain coupled challenges. OpenFlyV2 combines these functions in one model and uses motion-aware memory and action priors to support efficient flight.

Motivation

  • Aerial VLN must connect linguistic goals with visual observations and action while also supporting scene semantics, language grounding, and spatial reasoning.Offboard inference avoids some local computation limits but makes navigation dependent on communication availability and latency.Source: §1 · p. 2

Challenges

  • Integrating diverse vision-language capabilities, scalable aerial supervision, and low-latency onboard execution poses three coupled design challenges.Frequent UAV interruptions reduce average flight speed and mission efficiency.Source: §1 · p. 2

Architecture

  • OpenFlyV2 uses a shared vision-language backbone for understanding and a compact action expert to generate short-horizon continuous flight commands from the same multimodal context.Capabilities are accessed according to task demands.Source: §1 · p. 3

Efficiency mechanisms

  • Motion-Adaptive Memory compresses flight history into bounded event-level tokens, while Motion-Structured Prior initializes waypoint generation around a context-conditioned motion prototype.The methods reduce the costs of processing visual history and generating waypoint sequences for onboard closed-loop navigation.Source: §1 · p. 3

Training data

  • The training pipeline generated approximately 70,000 language-conditioned navigation trajectories and 60,000 aerial perception-and-reasoning instruction–response pairs across 13 environments.Task-specific samples came from simulation or rendering, without real-world flight demonstrations for policy training; language annotations were automatically generated and then human-verified.Source: §1 · p. 3

Evaluation overview

  • A single 1-billion-parameter checkpoint was evaluated across vision-language capabilities, simulated navigation, onboard inference, and physical flight, including 48% navigation success across four real-world environments without real-world training data.On a Jetson Orin NX, it generates action chunks at 10 Hz and supports closed-loop navigation entirely onboard.Source: §1 · p. 3

2. Results

OpenFlyV2 combines aerial perception and navigation in one model, trained and evaluated across shared environments. It performs competitively in aerial perception and simulated navigation, while onboard physical trials achieve a 48% aggregate success rate.

2.1. System overview

  • The system uses a 0.8B Qwen3.5-VL backbone and a 0.2B action expert; during navigation, current and historical observations condition waypoint prediction for a low-level controller.MAM compresses visual-action history and MSP initializes action generation from a context-conditioned motion distribution.Source: §2.1 · p. 3

2.2. Shared environments for aerial navigation and visual question answering

  • OpenFlyV2-130K pairs 72,352 navigation trajectories with 60,006 VQA pairs across 13 shared environments, covering five perception-and-reasoning task types.Nine environments were used for training and four were excluded from both VLN and VQA training; navigation actions and task-specific observations were generated in simulation or rendering.Source: §2.2 · pp. 3–4, Figure 2 · p. 5

2.3. Aerial vision-language understanding and spatial reasoning

  • With a 0.8B vision-language backbone, OpenFlyV2 performed broadly comparably to GPT-5.5 and surpassed Qwen3.5-9B and Gemini 3 Flash across five aerial-perception tasks.It exceeded GPT-5.5 in grounding accuracy (67.1% versus 61.6% Acc@0.5) and had lower distance-estimation errors (47.32 versus 52.62 m mean absolute error; 0.27 versus 0.34 relative error).Source: §2.3 · p. 4

2.4. Vision-language navigation in simulated environments

  • OpenFlyV2 outperformed the tested vision-language navigation models in seen and unseen environments, with success rates of 62.88% and 52.67%, respectively.It also had lower terminal navigation error than DualVLN in both splits and the highest oracle success rate and success weighted by path length in both splits.Source: §2.4 · p. 4

2.5. History compression and single-step action generation

  • MAM reduced input tokens from 446 to at most 218 while representing a longer flight history, and cut action-expert denoising time on the Jetson Orin NX from 80 ms to 20 ms.MAM improved all four navigation metrics at both integration budgets; with one step, SR rose from 49.82% to 53.45% and NE fell from 49.18 to 37.54 m.Source: §2.5 · pp. 4–5
  • Adding MSP to MAM raised one-step SR from 53.45% to 57.09% and reduced NE from 37.54 to 33.55 m.The combined model exceeded the two-step base on all four metrics, while MSP’s observed advantage was concentrated at the single-step budget.Source: §2.5 · p. 5

2.6. Real-world onboard aerial vision-language navigation

  • 48% aggregate success across 100 physical-flight trials, pooled over two scene groups and instruction types, demonstrates transfer to onboard navigation.The same policy generated 12-waypoint action chunks at 10 Hz on a Jetson Orin NX, with visual processing, memory updates, and action generation running onboard.Source: §2.6 · pp. 5–6

3. Discussion

OpenFlyV2 outperforms the tested navigation methods while combining aerial perception and navigation onboard; its ablations distinguish history compression from motion initialization. Physical trials extend language-conditioned landmark interpretation and task progression into an onboard perception–action loop, while error recovery and broader evaluation remain open directions.

Discussion

  • OpenFlyV2 outperforms tested navigation methods in seen and unseen simulated environments, where Figure 4B reports SR, OSR, SPL, and NE over 800 seen and 300 unseen episodes.Success requires an explicit stop within 25 m of the goal.Source: §3 · p. 6, Figure 4 · p. 7
  • Supporting perception and navigation within the aircraft’s computational constraints broadens how operators can use language to interact with the robot.Source: §3 · p. 6
  • MAM compresses flight history to reduce input tokens and action-expert denoising time while improving navigation; MSP improves single-step generation through structured motion initialization.The ablations indicate these mechanisms address distinct resource constraints.Source: §3 · p. 6
  • Physical trials bring language-conditioned landmark interpretation and task progression into a fully onboard perception–action loop.Source: §3 · pp. 6–7
  • Detecting task deviations, recovering from errors, and resuming navigation under changing observations remain directions for further research.Source: §3 · p. 7
  • Dialogue, reference tracking across requests, concurrent navigation and question answering, and broader evaluation across environments and operating conditions are proposed for future study.Source: §3 · p. 7

4. Materials and Methods

OpenFlyV2 combines a shared vision-language backbone with an action expert for waypoint generation, supported by aerial training data, compressed motion history, and structured action generation. Its methods span training, evaluation, and onboard execution.

4.1. OpenFlyV2 architecture

  • A shared vision-language backbone supports textual responses or conditions 12-waypoint action chunks, while updated prefix caching avoids reprocessing unchanged visual and language tokens during refinement.Actions use body-frame motion increments plus a stop signal; subsequent prediction cycles incorporate new observations.Source: §4.1 · pp. 7–8

4.2. Dataset generation and annotation

  • The data pipeline generated navigation trajectories and aerial perception-and-reasoning pairs from AirSim, Unreal Engine, and reconstructed 3D Gaussian Splatting scenes, with human verification retaining approved samples.Trajectory generation sampled landmarks and unoccluded viewpoints; GPT-5 generated instructions and question–answer pairs, and 60% of generated samples passed verification.Source: §4.2 · pp. 8–9

4.3. Motion-Adaptive Memory

  • Motion-Adaptive Memory segments past actions into bounded events and fuses action-conditioned visual features with event-age information, replacing dense historical images with compact motion-conditioned context.Each event uses action segments and images anchored at its start and end; memory capacity is bounded by the number of events and tokens per event.Source: §4.3 · p. 9

4.4. Motion-structured action generation

  • Motion-Structured Prior clusters training trajectories into four prototypes, adjusts a selected prototype to the current context, then samples an action source that the expert refines through conditional flow matching.Prototype adjustment controls horizontal extent, vertical profile, and accumulated yaw; covariance retains variation across action steps and coordinates.Source: §4.4 · p. 10

4.5. Training

  • Training jointly supervised vision-language and navigation tasks, introduced MAM and MSP in a second stage, then post-trained only the action expert with group-relative optimization.The final evaluation and deployment used one flow-integration step without added transition noise, retaining the sampled MSP source.Source: §4.5 · pp. 10–11

4.6. Evaluation protocols

  • Evaluation covered vision-language tasks, simulated navigation in seen and unseen environments, and 100 physical trials across two seen and two unseen locations.Physical success required stopping near the intended destination within the time limit; two-landmark tasks also required reaching both landmarks in order.Source: §4.6 · p. 11

4.7. Onboard deployment

  • Asynchronous chunk execution generates the next action chunk while the current one runs, conditioning on committed commands and training the expert to continue them.The latest observation and updated memory inform the uncommitted portion of each new chunk; a low-level controller tracks the resulting waypoints.Source: §4.7 · p. 11

Related papers

Loading 2610.39287v1…