WorldVLN: Autoregressive World Action Model for Aerial Vision-Language Navigation

• VLA models remain limited because they recognize objects and parse instructions without explicitly modeling temporal, geometric, and causal world evolution under actions.Source: §1 · pp. 1–2
• Autoregressive world modeling predicts candidate movement consequences and selects actions whose anticipated states best satisfy the instruction.Source: §1 · p. 2
• WorldVLN adapts a latent autoregressive video transformer to predict short-horizon world transitions, decode waypoint actions, and reinsert newly observed states into context after execution.Source: §1 · p. 2
• Action-aware GRPO optimizes autoregressive waypoint decisions through online rollout rewards that reflect trajectory, task, and reference-policy outcomes.Source: §1 · p. 2
• WorldVLN achieves state-of-the-art performance on indoor and outdoor aerial VLN benchmarks and demonstrates zero-shot generalization on a real-world drone platform.Source: §1 · p. 2

• VLA models extend vision-language models with action heads but remain limited in navigation because they do not explicitly model action-conditioned world dynamics.Source: §2 · p. 3
• World models provide visual-temporal priors, while WAMs couple latent world prediction with action generation and autoregressive variants support temporal memory and closed-loop feedback.Source: §2 · p. 3
• Existing autoregressive WAMs remain underexplored for aerial navigation, where viewpoint changes, continuous 3D motion, and accumulated state errors make autoregressive updating important.Source: §2 · p. 3
• Supervised imitation can suffer from covariate shift and accumulated errors, while reinforcement learning for autoregressive WAMs faces action-outcome mismatch and multi-step credit assignment.Source: §2 · p. 3

• Aerial VLN is modeled as partially observable sequential decision-making with egocentric observations and a policy conditioned on instruction and interaction history.Source: §3 · p. 3
• Each waypoint action contains relative 3D translation and relative yaw, and its execution updates the agent pose before producing a new egocentric observation.Source: §3 · p. 3
• Navigation repeats the observe–act process until a stop action or maximum horizon, with success defined by the final position being within threshold ϵ of the target.Source: §3 · p. 3

4.1 Model architecture

• WorldVLN reinterprets predicted latent video segments as short-horizon world-state transitions and decodes them into waypoint actions.Source: §4.1 · p. 4, Figure 1 · p. 4
• After executing decoded actions, real observation latents replace predicted latents in the autoregressive context, creating a closed-loop observe–act–update rollout.Source: §4.1 · p. 4

4.2.1 Stage 1: Supervised training

• Stage 1 supervises the latent backbone with instruction-video pairs and trains the action decoder on video-trajectory pairs to recover expert waypoint actions.Source: §4.2 · p. 4, §4.2.1 · p. 5, Figure 2 · p. 5

4.2.2 Stage 2: Action-aware GRPO

• Stage 2 performs online autoregressive rollouts and applies Action-aware GRPO to optimize waypoint decisions using segment-level navigation consequences.Source: §4.2 · p. 4, §4.2.2 · pp. 5–6
• The reward design combines local trajectory accuracy, global task outcome, and reference-policy regularization, with temporal decay assigning greater weight to earlier decisions.Source: §4.2.2 · p. 6
• Group-normalized segment advantages update the backbone-and-decoder policy through a clipped GRPO objective, aligning actions with downstream observations and final success.Source: §4.2.2 · p. 6

5.1 Quantitative results

• WorldVLN achieves the best benchmark results, reaching 79.12% and 78.02% average SR on UAV-Flow-Sim and 41.76% full-set SR on IndoorUAV-VLA.These results exceed the strongest baselines by 13.51, 12.24, and 14.60 percentage points, respectively.Source: §5.1 · p. 7
• WorldVLN’s advantage is larger on difficult settings, including 16.08-point and 33.64-point SR gains on IndoorUAV-VLA Medium and Hard splits.It also performs especially well on spatially precise UAV-Flow-Sim tasks such as Approach, Land, Move, Shift, and Ascend/Descend.Source: §5.1 · p. 7

5.2 Case analysis

• Qualitative cases show more accurate landmark grounding and waypoint trajectories than VLA baselines in outdoor object maneuvers and indoor landmark navigation.WorldVLN executes precise relative positioning around a car and follows the instructed staircase-to-wall turn.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3 Ablation study

• Autoregressive SFT improves success rates by 5.7+ percentage points on UAV-Flow and IndoorUAV compared with full-sequence SFT using the same backbone and action decoder.The autoregressive variant preserves coherent visual-spatial representations through repeated observation updates, whereas full-sequence prediction exhibits semantic drift and scene collapse.Source: Figure 4 · p. 8, §5.3 · p. 9
• Action-aware GRPO adds over 10 points after Stage-1 SFT nearly saturates and produces trajectories closer to the intended circular maneuver.The result indicates that rollout-based optimization improves action consequences beyond visual plausibility and strengthens goal-directed behavior.Source: Figure 4 · p. 8, §5.3 · p. 9

5.4 Zero-shot generalization in real-world deployment

• WorldVLN transfers from simulation to real UAV deployment without additional real-world fine-tuning in indoor and outdoor scenarios.The real-world cases involve close-range object alignment indoors and vertical navigation outdoors.Source: §5.4 · p. 9, Figure 5 · p. 9

6 Conclusions, Limitations, and Future Works

• WorldVLN presents an autoregressive world-action model and Action-aware GRPO, achieving over 12 percentage-point gains over VLA baselines and promising zero-shot real-world transfer.The conclusion frames the approach as a route toward spatial action tasks and potentially broader embodied domains.Source: §6 · p. 10
• The experiments cover relatively short-range aerial navigation, leaving long-horizon VLN for future exploration.This is an explicit scope boundary of the current validation.Source: §6 · p. 10
• Real-world deployment relies on server-side inference because of backbone computation costs, limiting fully onboard execution.Future work targets model compression, inference acceleration, and scalable long-horizon latent prediction.Source: §6 · p. 10

• WorldVLN may support infrastructure inspection, search and rescue, and disaster assessment while reducing manual operation and human exposure to hazardous environments.Source: §A.1 · p. 15
• The paper does not treat current results as evidence of readiness for unsupervised operation in public or safety-critical environments.Source: §A.1 · p. 15
• Real-world deployment should remain supervised and constrained by geofencing, speed and altitude limits, emergency stops, state-estimation checks, and applicable UAV regulations.Source: §A.1 · p. 15

• The study focuses mainly on short-range aerial navigation and short-horizon waypoint generation, leaving long-horizon and complex navigation capabilities for future validation.Source: §A.2 · p. 15
• Real-world evaluation uses controlled or enclosed environments, so robustness to illumination changes, adverse weather, dynamic obstacles, crowds, and GPS denial remains insufficiently evaluated.Source: §A.2 · p. 15
• The large autoregressive backbone still requires server-side inference rather than fully onboard deployment, and the study lacks extensive multi-seed experiments.Source: §A.2 · p. 15

Architecture overview

• The backbone combines a text encoder, discrete video tokenizer, spatiotemporal autoregressive Transformer, and action decoder to transform instructions and observations into navigation actions.Source: §A.3 · p. 15

Visual tokenization

• Image or historical-video inputs become multi-scale visual pyramid conditions, allowing both observation types to share the same autoregressive Transformer.Source: §A.3 · p. 15, Figure 6 · p. 16

World backbone

• Within each clip, prediction proceeds coarse-to-fine across spatial scales, while successive clips are predicted autoregressively to model temporal evolution.Source: §A.3 · p. 16
• Predicted token blocks are aggregated into future latent world states rather than decoded as RGB video, then passed to the action decoder.Source: §A.3 · p. 16, Figure 6 · p. 16

Action decoder

• The action decoder embeds latent features, applies factorized temporal and spatial attention, and regresses continuous UAV displacement and yaw actions.Source: §A.3 · pp. 16–17, Figure 7 · p. 17

World-backbone supervised fine-tuning

• Training uses one initial frame and three 16-frame future segments, with temporal compression mapping each future clip to four latent steps for short-horizon decisions.Source: §A.4 · p. 17
• Segment-level supervised fine-tuning conditions each target on ground-truth history and retains video decoding as a training-time interface for plausible future prediction.Source: §A.4 · pp. 17–18

Action-decoder training

• A video-to-action teacher trained on real clips and expert actions supervises latent-to-token alignment before the Action Decoder is trained on ground-truth latents.Source: §A.4 · p. 18

Action-aware GRPO

• Action-aware GRPO samples four online autoregressive rollouts, executing predicted waypoint segments and feeding each resulting observation back into context.Source: §A.4 · p. 18
• The method combines trajectory consistency, endpoint-based task progress, and CE-style reference alignment rewards to balance expert similarity, goal reaching, and preservation of the video prior.Source: §A.4 · pp. 18–19
• Reference alignment limits drift from the pretrained behavior because optimizing rollout rewards alone can degrade visual consistency or collapse the latent imagination capability.Source: §A.4 · p. 19
• Temporal decay γ = 0.9 weights earlier segment decisions more heavily because early errors affect later observations, actions, and accumulated drift.Source: §A.4 · p. 19

A.5 Details of experimental setup

• UAV-Flow and IndoorUAV-VLA provide complementary evaluation of fine-grained outdoor flight skills and short-horizon indoor action composition.UAV-Flow uses fixed-template and open-vocabulary instructions, whereas IndoorUAV-VLA evaluates Easy, Medium, and Hard action-composition levels with SR and NDTW.Source: §A.5 · p. 19, Figure 8 · p. 20, Figure 9 · p. 21

A.6 Details of results on UAV-Flow

• 79.12% Fixed SR and 78.02% Open SR on UAV-Flow show robust language-to-flight control across instruction settings.Approach reaches 97.62% Fixed SR and 95.24% Open SR, while Land reaches 92.59% Fixed SR and 98.15% Open SR.Source: §A.6 · p. 21

A.7 Details of results on IndoorUAV

• 41.76% SR and 13.48% NDTW on IndoorUAV-VLA Full demonstrate performance on both final-pose success and trajectory consistency.The Medium and Hard splits further test short-horizon composition of translation, rotation, vertical movement, and other primitives.Source: §A.7 · p. 21

A.8 Details of VLA vs WAM

• The VLA-versus-WAM comparison controls training splits, waypoint normalization, and evaluation protocol while contrasting direct observation-to-action prediction with latent world prediction followed by decoding.OpenVLA and WorldVLN are evaluated on the same UAV-Flow-Sim splits, with each retaining its native optimizer and training configuration.Source: §A.8 · pp. 21–22

A.9 Details of Real-World Deployment

• WorldVLN executes language-conditioned waypoint navigation on a real UAV in indoor motion-capture and outdoor GPS/LiDAR environments without using those localization measurements as model inputs.The platform uses onboard visual acquisition and flight control, while high-level inference runs on a ground-station server; the experiments verify executability and environmental adaptability.Source: §A.9 · pp. 22–23, Figure 10 · p. 22