PaperlayerView exact revision on arXiv

Source-linked AI summary

WareFly-VLA: A Vision-Language-Action Framework for UAV Navigation and Human Tracking in Smart Warehouses

Thinh D. Le, Son T. Nguyen, Duong Q. Nguyen, Dung D. Le, Ngo Anh Vien, H. Nguyen-Xuan

arXiv:2610.08526v1cs.ROcs.CV

TL;DR

Language-conditioned UAV control in warehouses remains underexplored, with few benchmarks combining appearance-grounded descriptions, continuous flight actions, and realistic industrial environments. WareFly-VLA introduces a human-demonstrated dataset and unified VLA benchmark; results show that aerial control remains challenging, with continuous action heads outperforming discrete tokenization and single-frame learning reliable mainly for forward motion.

  • Problem

    Language-conditioned UAV control is underexplored, and existing benchmarks rarely combine appearance-grounded target descriptions, continuous low-level actions, and realistic warehouse environments.

    Source: §1 · p. 2

  • Method

    WareFly-VLA pairs human-teleoperated warehouse demonstrations with language descriptions and continuous flight actions, and evaluates open-source VLA models using a leakage-free episode-level benchmark.

    Source: §1 · p. 3, §9 · p. 24

  • Results

    Current aerial control remains unsolved: π0 reaches r≈0.5 on the forward and heading channels, no model beats a constant-mean baseline on lateral or vertical channels, and flow-matching heads consistently outperform discrete action tokenization.

    Source: §8 · p. 21, §9 · p. 24

  • Takeaways & Limitations

    The synchronized dataset and released baselines support world-model research and further evaluation of language-grounded aerial autonomy.

    Source: §9 · p. 24

    All episodes come from one simulated Isaac Sim warehouse, so cross-environment and sim-to-real transfer are not assessed.

    Source: §8.2 · p. 24

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have achieved impressive results in robotic manipulation and ground-mobile navigation, yet language-conditioned control of unmanned aerial vehicles (UAVs) in smart warehouses remains largely unexplored, hindered by the lack of benchmarks that jointly provide continuous low-level flight actions, fine-grained natural-language target descriptions, and realistic industrial environments. This paper introduces WareFly-VLA, a photorealistic UAV VLA framework and dataset for language-guided human search, localization, and tracking in warehouse environments. It contains 507 human-teleoperated flight episodes and 8,504 high-resolution RGB transitions collected in NVIDIA Isaac Sim, each paired with a human-written appearance description of the target worker and a synchronized four-degree-of-freedom control command. Two aerial tasks are covered: target approach and person following, under occlusion, long-range search, altitude variation, and clutter. A unified benchmark of four open-source VLA architectures (SmolVLA, GR00T N1.7, pi_0 and OpenVLA) is established under a leakage-free episode-level protocol at two control rates. The results show that language-conditioned aerial control in warehouses is far from solved: performance drops substantially under strict generalization settings, continuous action modeling consistently outperforms discrete action tokenization, only the forward channel is reliably learnable from a single frame, and current foundation-model interfaces transfer poorly from ground and humanoid embodiments to aerial platforms. The synchronized video, language, action, pose, and difficulty annotations further support world-model research. The dataset, baselines, and evaluation protocol are released to support language-grounded aerial autonomy in smart warehouses.

1. Introduction

VLA research has focused mainly on manipulation and ground-mobile robots, leaving language-conditioned UAV control underrepresented despite its distinct 3D motion and visual challenges. WareFly-VLA addresses the benchmark gap with human-demonstrated warehouse search and following data and a unified evaluation of open-source VLA models.

Research gap

  • Existing VLA datasets and benchmarks concentrate on manipulation and ground-mobile platforms, while UAVs combine continuous 3D motion with rapid viewpoint changes, target-scale variation, and occlusion.These sensing and control differences make transferring ground and manipulation VLA approaches to aerial systems nontrivial.Source: §1 · p. 2

Motivation

  • Language-guided human following combines appearance-based target identification, active search, persistent target association, and continuous flight control in one task.Unlike conventional UAV tracking, the target is specified by language rather than a bounding box or visual template.Source: §1 · p. 2

Contribution

  • WareFly-VLA pairs human-teleoperated warehouse trajectories with synchronized RGB observations, appearance-grounded descriptions, and continuous 4-DoF flight actions.Its two tasks are approach-and-locate and person-following, with varied layouts, distances, altitudes, occlusions, and clutter.Source: §1 · p. 2
  • A unified benchmark evaluates open-source VLA architectures with a leakage-free episode-level split, and its results show that language-conditioned UAV control remains challenging.The work also releases dataset artifacts, benchmark configurations, training and evaluation tools, and identifies world-model research directions.Source: §1 · p. 3

2. Related Work

Prior VLA work has advanced generalization across tasks and embodiments, but aerial applications remain underexplored. Aerial navigation datasets often supervise waypoints or routes rather than continuous actions, while existing UAV VLA benchmarks do not jointly address appearance-grounded human search and following.

2.1. Vision–Language–Action models

  • VLA systems have advanced generalization across tasks and robot embodiments, but benchmarks remain centered on manipulation, mobile navigation, and humanoid control, with aerial datasets combining language and continuous flight supervision scarce.Examples include PaLM-E [1], RT-1 [2], RT-2 [3], OpenVLA [5], and π0.5 [7].Source: §2.1 · p. 3

2.2. Language-grounded aerial navigation

  • Aerial language-navigation datasets such as AerialVLN [15] and UAV-VLN [18] generally frame navigation as waypoint prediction, route planning, or localization rather than direct low-level control.This leaves limited supervision for translating language instructions into continuous flight actions.Source: §2.2 · pp. 3–4

2.3. UAV VLA datasets and benchmarks

  • Existing UAV VLA efforts address racing, mission planning, or simulation-specific navigation, while appearance-based human search and following has received little attention.No publicly available benchmark combines appearance-grounded target descriptions, continuous 4-DoF actions, human-teleoperated demonstrations, and warehouse search-and-follow tasks.Source: §2.3 · p. 4

3. The WareFly-VLA dataset

WareFly-VLA is designed for language-guided aerial search and following, using human demonstrations, appearance-based instructions, and low-level flight actions. The dataset pairs realistic warehouse scenarios with a consistent teleoperation policy and synchronized observations, controls, and simulator states.

3.1. Dataset design principles

  • The dataset centers on language-guided human search and following, with targets specified by observable appearance rather than bounding boxes or visual templates.Its design uses human demonstrations, appearance-grounded language, and continuous low-level flight actions.Source: §3.1 · p. 4

3.2. Simulation environment

  • The simulated warehouse includes racks, conveyors, forklift lanes, mixed lighting, and visual challenges such as clutter, reflective surfaces, viewpoint changes, and partial occlusions.A forward-facing RGB camera records at 1920×1440 pixels, while simulator state provides vehicle pose.Source: §3.2 · p. 4

3.3. Human-Teleoperated Data Collection

  • Human operators search for, approach, and follow workers on routes unknown to them, producing acquisition, viewpoint adjustment, temporary target loss, recovery, and following behaviors.RGB observations, control inputs, and simulator states are recorded synchronously, then subsampled to 1 FPS.Source: §3.3 · p. 5

3.4. Teleoperation policy

  • The teleoperation policy cycles through search states and sends the pilot to Approach Target when the worker appears; unsuccessful search legs return to Look Around.The policy uses an initial yaw sweep, checks whether the drone has room to fly, scans shelf aisles when open, and moves to an open viewpoint when constrained.Source: §3.4 · p. 5, Figure 1 · p. 6

3.5. Action representation

  • Each action is a four-dimensional body-frame motion command derived from consecutive simulator poses, with translation, altitude, and heading updates.The local-coordinate representation is invariant to global orientation and compatible with UAV controllers operating in body coordinates.Source: §3.5 · p. 7

3.6. Language annotation

  • Human annotators describe workers through observable clothing, colors, accessories, and protective equipment, avoiding identity and scene-specific position references.Multiple paraphrases are reviewed for factual consistency and language quality.Source: §3.6 · p. 7

4. Dataset analysis

WareFly-VLA combines varied human-flown warehouse trajectories with continuous UAV actions and appearance-grounded instructions. Its action and language distributions, challenge tags, and comparison with related datasets characterize the benchmark’s scope and diversity.

4.1. Quantitative statistics

  • The dataset contains 507 episodes and 8,504 non-terminal transitions at 1920 × 1440 resolution, with varied pursuit distances, motion pace, and flight paths.Episodes have distinct dynamics, spanning short-range maneuvers and long-distance approaches.Source: §4.1 · p. 7

4.2. Action space characterization

  • Forward motion averages 0.86 ± 0.69 m/frame, while lateral corrections vary widely and altitude stays nearly constant in most episodes.∆yℓ has a 0.42 m standard deviation and range [−5.12, +4.01] m; ∆z has mean −0.000 m and standard deviation 0.013 m; ∆ψ has standard deviation 0.355 rad and range [−1.25, +2.20] rad.Source: §4.2 · pp. 7, 9

4.3. Linguistic diversity and instruction analysis

  • Appearance instructions combine warehouse PPE terms with diverse colors and compositional clothing descriptions, while 15 task-verb families add paraphrase variation.Approach and follow dominate; thirteen additional verb families include “accompany,” “pursue,” and “move towards.”Source: Figure 2 · p. 8, §4.3 · pp. 9–11

4.4. Challenge tags and difficulty axes

  • Difficulty tags mark find, close, far, height_change, obstructed, and extended conditions for evaluation across independent challenge axes.“Far” denotes target distance >10 m; “height_change” marks altitude change >0.5 m, and “obstructed” requires the target to pass fully behind warehouse infrastructure for at least one frame.Source: §4.4 · p. 11

4.5. Comparison with related datasets

  • WareFly-VLA uniquely combines UAVs, language-based appearance descriptions, continuous low-level actions, and industrial warehouse settings, using photorealistic simulation rather than physical capture.The authors note that its episode count is modest because each episode requires manual teleoperation and a uniquely identifying hand-authored worker description.Source: §4.5 · pp. 11–12, Table 3 · p. 12

5. Benchmark protocol

The benchmark evaluates four open-source VLA architectures using shared episode-level splits, fine-tuning protocols, and complementary action-prediction metrics. On the held-out validation split, π0 performs best overall, but lateral and vertical prediction remain difficult.

5.1. Benchmark models

  • The benchmark selects OpenVLA-7B, GR00T N1.7, π0/OpenPI, and SmolVLA to span action-generation paradigms, parameter scales, and embodiment assumptions.The selected architectures cover discrete action tokenization, diffusion-based action generation, and flow matching.Source: §5.1 · p. 12, Table 4 · p. 13

5.2. Data splits

  • A fixed episode-level split assigns 431 episodes to training and 76 to validation, with no validation episode entering training at either control rate.Trajectory-level partitioning prevents temporal leakage between training and evaluation samples.Source: §5.2 · pp. 12–13

5.3. Training protocol

  • Models are fine-tuned on WareFly-VLA training data from public checkpoints using a single RGB image and language instruction, with compatibility mappings for UAV controls.At 10 FPS, only π0 and SmolVLA are retrained; OpenVLA and GR00T are excluded for stated compute or divergence reasons.Source: §5.1 · p. 12, §5.3 · p. 13

5.4. Evaluation metrics

  • MAE measures action-prediction fidelity, while Pearson correlation measures temporal-structure reproduction; both are compared with a constant-mean baseline.Low error can result from regressing toward the mean, whereas high correlation does not guarantee accurate action magnitudes.Source: §5.4 · p. 13
  • On 76 shared held-out episodes, π0 performs best overall, SmolVLA follows, OpenVLA generalizes poorly, and GR00T produces no meaningful predictions.No model surpasses the constant-mean baseline on ∆yℓ or ∆z.Source: Figure 8 · p. 14

6. Results and analysis

Across held-out episodes, only forward displacement is reliably predictable from one frame; the generalization gap is large and appears in both per-sample predictions and trajectory rollouts.

6.1. Main results

  • At 1 FPS, π0 reaches r=0.65 on ∆xℓ while ∆z is at chance (|r|≤0.06 for every model); at 10 FPS, only ∆xℓ is clearly better than naive in MAE.At 1 FPS, π0 also leads SmolVLA on all reported dimensions, and flow-matching models outperform discrete-token OpenVLA.Source: §6.1 · pp. 14–15, Table 6 · p. 15, Figure 9 · p. 16

6.2. Generalization gap

  • On ∆xℓ, SmolVLA’s MAE rises from 0.083 (train) to 0.468 (val), a 5.6× degradation, versus π0’s 0.150 to 0.413 (2.8×).SmolVLA fits the training data best but generalizes worse than the larger π0; OpenVLA’s smaller relative gap reflects underfitting and 256-bin quantization.Source: §6.2 · p. 15

6.3. Per-sample prediction quality

  • On held-out samples, π0 preserves a positive but range-compressed forward trend, while lateral and vertical predictions collapse near the training mean.The corresponding training-split predictions hug the identity diagonal, making the generalization gap visible.Source: §6.3 · p. 16, Figure 10 · p. 16

6.4. Qualitative analysis

  • SmolVLA under-reaches large forward displacements and misses lateral corrections; integrated predictions drift on curved paths as lateral and heading errors accumulate.The held-out rollouts retain gross forward progression but diverge where corrections are mispredicted.Source: §6.4 · pp. 16–17, Figure 11 · p. 17

7. Deployment-oriented analysis

Deployment-oriented results expose trade-offs among model cost, inference speed, and trajectory fidelity: SmolVLA and π0 occupy different cost–accuracy points, while π0 drifts least in rollouts.

7.1. Cost–accuracy frontier

  • SmolVLA and π0 define the cost–accuracy Pareto frontier: SmolVLA is 7× smaller and faster to fine-tune, while π0 is most accurate on held-out data.OpenVLA-7B is dominated, and neither frontier model reaches deployable accuracy on this benchmark.Source: §7.1 · pp. 17–18, Figure 13 · p. 19

7.2. Inference latency and control rate

  • At 10 FPS, π0 sustains 15.9 Hz and SmolVLA 31.6 Hz, clearing the 10 Hz loop budget; OpenVLA-7B violates that budget and GR00T is fast but inaccurate.SmolVLA’s effective rate benefits from four-step action chunking, with bimodal latency; π0’s latency is steady.Source: §7.2 · p. 18, §7.3 · p. 19, Figure 14 · p. 20

7.3. Open-loop rollout drift

  • π0 drifts least in open-loop rollouts at both rates: ADE/FDE are 2.55/4.73 m at 1 FPS and 3.12/5.64 m at 10 FPS.FDE exceeds ADE for every model, reflecting compounding drift; rollout drift distinguishes π0 from SmolVLA where per-step MAE approaches the baseline.Source: §7.3 · pp. 18–19, Table 8 · p. 20, Figure 15 · p. 21

8. Discussion

The discussion attributes weak off-axis control to single-frame observability and identifies continuous action modeling as an advantage. It proposes world-model directions while bounding conclusions to open-loop, single-environment simulation data.

8. Discussion

  • Flow-matching policies outperform discrete-token OpenVLA, particularly for low-amplitude altitude and heading commands where quantization can obscure the action range.The discussion attributes this advantage to continuous action representations.Source: §8 · p. 21
  • A single frame provides cues for forward motion but not the temporal information needed for lateral and vertical corrections, leaving those channels near zero correlation.The discussion identifies limited temporal observability, rather than model capacity, as the primary bottleneck.Source: §8 · p. 21, §8.1 · p. 21
  • Per-step MAE is frame-rate sensitive because higher sampling rates shrink actions and can make constant-mean predictions deceptively accurate.The authors recommend complementing MAE with correlation, normalized skill scores, and closed-loop trajectory evaluation.Source: §8 · p. 21

8.1. World-model roadmap enabled by WareFly-VLA

  • WareFly-VLA aligns RGB, language, action, pose, and difficulty supervision with JEPA-style representation learning, action-conditioned latent dynamics, and language-conditioned prediction.The roadmap frames these streams as inputs to predictive modules.Source: §8.1 · p. 22, Figure 17 · p. 23
  • The roadmap extends latent models to action-sequence planning, sim-to-real transfer, and a deployment-feedback loop; tagged occlusion recovery and rollout drift provide evaluation hooks.Deployment failures return to data collection as new episodes.Source: §8.1 · p. 23

8.2. Limitations

  • Conclusions are bounded by open-loop scoring, 507 episodes, one Isaac Sim warehouse, and a single forward-facing RGB stream; cross-environment and sim-to-real transfer are not assessed.The authors expect closed-loop deployment to amplify reported gaps and identify richer observations and larger collections as extensions.Source: §8.2 · pp. 23–24

9. Conclusion

The conclusion presents WareFly-VLA as a leakage-free benchmark for language-conditioned drone person-following and reports persistent generalization and control challenges. It also releases the dataset and code and outlines a world-model research roadmap.

9. Conclusion

  • WareFly-VLA evaluates four open-source VLA architectures on a shared leakage-free episode split with bootstrap confidence intervals at 1 and 10 FPS.The benchmark concerns language-conditioned drone person-following in a photorealistic industrial warehouse.Source: §9 · p. 24
  • The strongest model reaches only r≈0.5 on forward and heading channels, while no model beats a constant-mean baseline on lateral or vertical channels.The conclusion also reports large train-to-validation gaps, an edge for continuous flow matching over discrete tokens, and reduced per-step MAE discrimination at higher rates.Source: §9 · p. 24
  • The paper charts a world-model roadmap and releases the dataset, baseline implementations, and analysis code under an open license.It invites extensions with additional baselines, warehouse environments, and closed-loop hardware deployment.Source: §9 · p. 24

Appendices

The appendices specify a shared leakage-free reproducibility protocol and an appearance inventory whose combinatorial sampling diversifies workers and distractors.

Appendices

  • The appendices collect the reproducibility protocol, PPE inventory, failure analysis, robustness and cross-model analyses, and extended 10 FPS results.This overview identifies the scope of the appended material.Source: Appendices · p. 25

Appendix A. Reproducibility Protocol

  • 507 episodes are split into 76 validation and 431 training episodes using one serialized manifest shared by every model adapter.Episodes are sorted by UUID before a NumPy permutation seeded with 0; terminal frames with identically zero actions are excluded from training and evaluation.Source: Appendix A · p. 25
  • Latency measures only the observation-to-action model call, with 10 warm-up and 50 timed iterations and CUDA synchronization.The median and p95 exclude data-loading and simulator-update overhead.Source: Appendix A · p. 25

Appendix B. Worker Appearance and PPE Inventory

  • Each episode samples a unique configuration across six PPE categories, while distractor workers typically share two or three attribute categories with the target.The inventory spans headwear, upper and lower body, footwear, hand protection, and accessories, supporting varied appearance vocabulary and non-trivial target finding.Source: Appendix B · p. 26

Appendix C. Failure Mode Analysis

The appendix diagnoses forward-range compression and off-axis prior collapse in per-frame control, and adds cross-model and optimization diagnostics.

Appendix C. Failure Mode Analysis

  • Forward predictions compress toward the training mean, lateral and vertical predictions collapse to their priors, and rare verbs do not trigger an error spike.Forward pursuits reach 5.62 m/frame, with predictions near half the true large-action magnitude; off-axis low MAE matches the constant-mean baseline rather than learned control.Source: Appendix C · p. 26
  • The diagnosed forward and off-axis failures concern commands supervised by temporal target-relative state, motivating multi-frame observations or velocity-from-pose conditioning.The per-frame ground-truth pose state is released to support this research direction.Source: Appendix C · p. 26

Appendix D. Additional Robustness and Cross-Model Analyses

  • The appendix diagnostics corroborate the ranking π0 > SmolVLA > OpenVLA > GR00T and forward-channel-only learnability.These are presented as finer-grained corroboration of the headline per-dimension results.Source: Appendix D · p. 27

Appendix D.1. Training dynamics

  • OpenVLA’s loss shows an overfitting onset around epoch 3, while π0 plateaus before a sharp drop near step 1000; their objective-specific loss scales are not directly comparable.The OpenVLA onset is attributed to LoRA adapter saturation; π0’s drop occurs when the action expert aligns with VLM hidden states.Source: Appendix D.1 · p. 27, Figure D.1 · p. 27

Appendix D.2. All-model scatter comparison

  • SmolVLA shows steeper forward-range compression, OpenVLA shows discrete-token staircase bands, and GR00T predictions fall two orders of magnitude outside the empirical action range.The scatter diagnostics associate these patterns with SmolVLA’s larger generalization gap, OpenVLA’s quantization, and GR00T’s normalization failure, respectively.Source: Appendix D.2 · p. 27

Appendix D.3. Error cumulative-distribution analysis

Appendix diagnostics show that held-out errors are concentrated in action dimensions and episode types: forward motion is most learnable, while lateral and vertical predictions remain near the naive baseline. Across analyses, the flow-matching models lead, but range compression, generalization gaps, and shared hard episodes persist.

Appendix D.3. Error cumulative-distribution analysis

  • On held-out ∆xℓ, OpenVLA’s median |err| is 0.73 m versus 0.42 m for flow-matching models; π0 and SmolVLA nearly coincide, while lateral and vertical errors track naive.GR00T’s CDF lies one-to-two orders of magnitude above the empirical scale.Source: Appendix D.3 · pp. 27–28, Figure D.4 · p. 28

Appendix D.4. Train–val gap analysis

  • SmolVLA’s ∆xℓ MAE rises from 0.083 (train) to 0.468 (val), a +0.385 m gap; its low training error masks the steepest held-out fall-off.OpenVLA’s smaller gap (+0.149) reflects underfitting, while π0’s gap is +0.263.Source: Appendix D.4 · pp. 28–29, Table D.2 · p. 29

Appendix D.5. Error vs. action magnitude

  • On held-out ∆xℓ, all working models’ errors grow with action magnitude; π0 is slightly less compressed than SmolVLA, while OpenVLA adds a magnitude-independent quantization offset.The policies under-reach the largest expert displacements.Source: Appendix D.5 · pp. 29–30, Figure D.6 · p. 30

Appendix D.6. Per-episode error heatmap

  • Across nearly every episode, π0 and SmolVLA have lower MAE than OpenVLA, which is far below GR00T; long approach/locate episodes with large heading changes challenge all working models.The row-aligned errors indicate difficulty is shared by the episode rather than specific to one model.Source: Appendix D.6 · p. 30, Figure D.7 · p. 31

Appendix D.7. Linguistic robustness: Per-verb-family breakdown

Appendix D.8. Within-episode error drift; Appendix D.9. Side-by-side qualitative comparison

Appendix E.1. Correlation and per-episode ranking summaries

  • π0 beats SmolVLA on 64% of episodes; both flow-matching models beat OpenVLA on ≥84%, and every model beats GR00T on 100%.The pairwise results show the ranking is not merely an averaging artifact.Source: Appendix E.1 · p. 33

Appendix E.2. Directional accuracy and calibration at 10 FPS; Appendix E.3. SmolVLA companion diagnostics and the 10 FPS error CDF

  • At 10 FPS, SmolVLA’s per-sample scatter and open-loop rollouts show the same range-compression and curve-drift behavior as π0, but slightly more pronounced.Source: Appendix E.3 · p. 34
Loading 2610.08526v1…