PaperlayerView exact revision on arXiv

Source-linked AI summary

Skytopia: Monocular Drone Navigation with Action-Conditioned Latent World Models

Yuhang Zhang, Rangya Zhang, Yujing Shang, Zhuoyuan Yu, Weiying Wang, Steven Yang, Qingsong Yan, Chao Yan, Mir Feroskhan

arXiv:2609.26007v1cs.ROcs.AI

TL;DR

Monocular drone navigation must work from a forward-facing camera with few depth and scale cues, while existing navigation world models retain predictions in action generation. SKYTOPIA trains an action-conditioned latent policy but discards its predictor for deployment, outperforming baselines in simulation across three goal modes and reaching goals in real-world flights without fine-tuning.

  • Problem

    A forward-facing monocular camera offers few reliable depth and scale cues, while existing navigation world models feed predictions into action generation at every control step.

    Source: §1 · p. 2

  • Method

    SKYTOPIA trains an action-conditioned latent world-model policy with forward prediction and inverse dynamics objectives, then deploys the policy without its predictor.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    57.8%, 66.0%, and 49.0% success rates let SKYTOPIA surpass seven baselines across point-goal, image-goal, and goal-free simulation settings.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The same policy was deployed without fine-tuning on a physical drone, reaching goals in unseen indoor, open outdoor, and woodland environments.

    Source: §5 · p. 9

    Future observations serve as target states for dynamics supervision and are excluded from policy inputs.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Monocular drone navigation requires reaching a goal in an unseen environment from a single forward-facing camera, which offers few cues for depth and scale. World models address this by modelling how observations evolve under actions, but they are built to be executed: the prediction is produced at deployment and fed back into action generation at every control step. We argue that what a policy needs from a world model is not the prediction but the representation required to produce it: in flight the executed action explains almost all of the change between observations, so prediction reduces to reprojecting a static scene under a known displacement. We therefore introduce skytopia, a policy built on an action-conditioned latent world model, and the 3D Gaussian Splatting platform on which it is trained. A forward objective predicts the representation of the next observation from the intended motion, and an inverse objective recovers that motion from the predicted transition. Because the prediction never reaches action generation, the predictor is discarded and one policy serves point-goal, image-goal, and goal-free navigation. Simulation experiments show that skytopia outperforms every baseline under all three specifications, attaining 57.8%, 66.0%, and 49.0% success rate, while discarding the predictor removes 59.4% of the inference cost. The same policy is subsequently deployed on a physical drone without fine-tuning and reaches goals in indoor, open outdoor, and woodland environments.

1 INTRODUCTION

Monocular drone navigation is difficult because a forward-facing camera provides few reliable cues for depth and scale, while flight demands action-sensitive prediction without deployment-time delay. SKYTOPIA trains an action-conditioned latent world model and discards its predictor at deployment, outperforming baselines in simulation across three navigation modes.

1 INTRODUCTION

  • A single forward-facing camera provides few reliable depth and scale cues, making perception the dominant bottleneck for aerial navigation.Aerial workspaces are often open and weakly textured.Source: §1 · p. 2
  • Existing navigation world models are weakly conditioned on actions and retained at deployment, risking geometry-poor predictions and latency that becomes distance flown without a new command.In flight, action-image motion is the only reliable source of depth, and drones cannot pause for computation.Source: §1 · p. 2
  • SKYTOPIA combines a 3D Gaussian Splatting simulation platform with action-conditioned forward and inverse dynamics objectives, then removes the predictor for deployment.Its platform provides 32.5 million frames from 94 indoor and outdoor scenes; one deployed policy supports three goal modes.Source: §1 · p. 2, Figure 2 · p. 3
  • 57.8%, 66.0%, and 49.0% success rates let SKYTOPIA surpass seven baselines across point-goal, image-goal, and goal-free navigation.It exceeds the strongest baseline by at least 15 points on a 2B backbone without scene-specific adaptation.Source: §1 · p. 3

2 RELATED WORK

Prior navigation work includes direct observation-to-action policies, vision-language policies, and world models that predict observations or latent representations. SKYTOPIA targets aerial monocular navigation with training-only dynamics objectives and no deployed predictor.

2 RELATED WORK

  • Direct-mapping navigation policies fit demonstrated actions without modelling the dynamics that produce them, which the passage says makes them fragile under distribution shift.Earlier policies reach coordinate or image goals, while later work scales across robots and datasets.Source: §2 · p. 3
  • Vision-language navigation policies use semantic priors but ground representations in appearance and map observations directly to actions, unlike SKYTOPIA’s geometry-oriented setting.Instruction following is orthogonal to goals specified as coordinates, images, or no target.Source: §2 · p. 3
  • Explicit world models synthesize future frames, while latent models predict compact representations and plan over imagined rollouts; both retain deployment-time prediction.The passage describes explicit synthesis as computationally expensive and latent prediction as less costly.Source: §2 · p. 3
  • WorldVLN uses a closed-loop video backbone and FutureNav uses pooled descriptors with a small discrete action set; SKYTOPIA instead acquires geometry during training and retains no predictor.The passage identifies the prior systems’ designs as unsuited to open, weakly textured scenes observed through one camera.Source: §2 · p. 4

3 METHOD

SKYTOPIA trains a navigation policy with action-conditioned latent dynamics objectives, using predicted transitions to supervise representations and action tokens rather than to condition deployment. The policy generates goal-conditioned or goal-free action chunks, while its dynamics components are removed after training.

3.1 PROBLEM FORMULATION

  • SKYTOPIA formulates offline visual navigation as predicting action chunks from monocular observations, proprioceptive states, and optional metric or image goals.The goal inputs define point-goal, image-goal, and goal-free modes.Source: §3.1 · p. 4

3.2 OVERVIEW

  • A shared vision–language backbone produces separate action tokens for latent prediction and action generation, with deployment retaining only the backbone and action head.The architecture uses forward prediction and inverse dynamics objectives to train motion-related tokens.Source: §3.2 · p. 4

3.3 ACTION-CONDITIONED LATENT WORLD MODEL

  • Forward dynamics predicts future latent states from preceding states and intended-motion tokens, while an inverse head recovers executed commands from predicted transitions.A frozen video encoder supplies latent targets; with K = 3, inverse dynamics recovers the first six commands of a seven-command chunk, and randomly dropping preceding states encourages use of predicted states.Source: §3.3 · pp. 4–5

3.4 GOAL-CONDITIONED ACTION GENERATION

  • Metric goals are appended to proprioception and goal images remain spatial views; conditional flow matching generates action chunks across point-goal, image-goal, and goal-free modes.Training samples goal-input combinations, while the action head learns a distribution over continuous chunks to accommodate multiple valid trajectories.Source: §3.4 · pp. 5–6

3.5 TRAINING OBJECTIVE AND DEPLOYMENT

  • The training objective combines action, forward-dynamics, and inverse-dynamics losses, but neither predicted states nor predicted actions enter deployment action generation.The encoder and both dynamics heads are removed after training, so the dynamics objectives supervise the backbone without additional inference cost.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments evaluate SKYTOPIA on unseen simulated scenes, test transfer to physical flight, and examine the dynamics objectives, learned geometry, and deployment cost. Across these evaluations, the policy performs well across goal specifications, transfers without fine-tuning, and can discard its predictor at deployment.

4.1 EXPERIMENTAL SETUP

  • On 5 held-out scenes, configurations are evaluated over 200 episodes per goal specification under each of 3 random seeds, against direct-action, vision–language–action, and world-model baselines.The benchmark comprises 94 reconstructed scenes; episodes use start–goal pairs sampled from mutually reachable free-space positions.Source: §4.1 · p. 6

4.2 SIMULATION RESULTS

  • SKYTOPIA achieves the highest SR, OS, and SPL and the lowest NE and TTS across all three goal specifications, outperforming OmniVLA in goal reaching and path efficiency while reducing collisions.The comparison includes goal-free navigation, where the policy receives no target specification.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 REAL-WORLD FLIGHT

  • Without fine-tuning, the simulation-trained policy transfers to indoor, open outdoor, and woodland flight; mean SR is 56.7% for image-goal and 41.7% for goal-free navigation, while point-goal SR is 55% indoors.Representative flights show obstacle avoidance and successful arrival under each goal specification.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4 ABLATION STUDIES

  • Removing both dynamics objectives reduces success and path efficiency and increases collisions across all three modes; either objective improves on action-only training, while using both gives the strongest overall navigation performance.The variants compare action-only training, forward prediction, inverse dynamics, and the full model.Source: §4.4 · p. 8, Table 2 · p. 8

4.5 ANALYSIS

  • Joint dynamics training yields lower depth- and displacement-probe errors than ablated variants, making geometry and motion more accessible in frozen features without direct supervision of either quantity.The probes use frozen backbone features and report means and standard deviations over 4 episode splits.Source: §4.5 · p. 9, Figure 6 · p. 9
  • Discarding the predictor after training removes 59.4% of FLOPs and 52% of latency; the deployed model runs faster than NavMorph, OmniVLA, and NoMaD while achieving higher success across all three modes.NWM requires three orders of magnitude more computation and time per step; SKYTOPIA uses dynamics prediction during training, not action generation at deployment.Source: §4.5 · p. 9

5 CONCLUSION

SKYTOPIA combines a 3DGS simulation platform with action-conditioned latent world-model training for monocular drone navigation. The policy transfers to physical flight without fine-tuning, while persistent memory and dynamic-scene robustness remain open limitations.

5 CONCLUSION

  • SKYTOPIA combines a 3DGS simulation platform with action-conditioned latent world-model training for monocular drone navigation.Forward and inverse dynamics objectives train a policy for three navigation modes without retaining the predictor at deployment.Source: §5 · p. 9
  • The same policy flies without fine-tuning in unseen indoor, open outdoor, and woodland environments, with successful arrival and obstacle avoidance despite changes in appearance and flight conditions.The supported goal settings include metric goals indoors and image-goal and goal-free navigation across all three sites.Source: §5 · p. 9
  • Persistent spatial memory for long-horizon navigation is absent, and robustness to dynamic obstacles and temporally changing scenes remains unexplored.Source: §5 · p. 9

A.1 PLATFORM AND DATA COLLECTION

The platform reconstructs captured environments for photorealistic simulation and physical interaction, then uses them to generate navigation demonstrations. It spans indoor, urban, and vegetation scenes and records observations, commands, and state for training.

A.1 PLATFORM AND DATA COLLECTION

  • Synchronized LiDAR point clouds and RGB images support 3DGS appearance reconstruction, while a mesh from the reconstruction supplies collision geometry through a 10 cm-voxel SDF.The simulator uses 3DGS for rendering and the SDF for physical interaction.Source: §A.1 · pp. 12–13, Figure 7 · p. 13
  • The platform contains 94 scenes—18 indoor, 48 urban outdoor, and 28 vegetation environments—with 75,657 m2 of navigable area and an average of 805 m2 per scene.The reconstruction pipeline also produces meshes for collision geometry.Source: §A.1 · p. 13, Figure 7 · p. 13
  • In Isaac Sim, rigid-body dynamics run at 100 Hz and observations and commands are recorded at 50 Hz; the forward-facing camera renders 256×256 RGB images.Each action specifies body-frame velocity and yaw rate, while the recorded 13-dimensional state includes velocities, relative goal vector, and body orientation quaternion.Source: §A.1 · p. 13
  • An expert planner generates 1,000 trajectories per scene, yielding 94,000 trajectories and 32.5 million frames over 181 hours of flight.The planner searches shortest paths on a mesh-derived occupancy grid and adjusts speed using obstacle clearance and path curvature.Source: §A.1 · p. 13

A.2 ANALYSIS OF THE DYNAMICS OBJECTIVES

The dynamics objectives align latent states with temporally corresponding observations and commands while restricting prediction to past and current windows. The analysis explains how fixed targets avoid a constant zero-loss solution, while inverse supervision addresses the separate risk that prediction ignores actions.

A.2 ANALYSIS OF THE DYNAMICS OBJECTIVES

  • Eight frames form four latent states using tubelet size 2; forward prediction covers three transitions, while inverse supervision recovers the first six commands of a seven-command chunk and the action objective supervises all seven.The shared inverse head recovers two commands from each state pair.Source: §A.2 · p. 13, Figure 8 · p. 14
  • The forward head's causal mask lets tokens attend within the current and preceding windows, but not to future windows when predicting the next latent state.Source: §A.2 · p. 14, Figure 8 · p. 14
  • A conditionally independent Laplace model for latent coordinates turns the negative log-likelihood into summed absolute prediction errors across transitions, motivating an ℓ1 objective.The constant likelihood term is absorbed into the forward-loss weight; the authors note that absolute error limits the influence of large residuals from occlusion or newly visible regions.Source: §A.2 · p. 14
  • Freezing the encoder prevents the target and predictor from jointly reaching a constant zero-loss solution, because distinct fixed targets cannot all match one constant prediction.The passage states that this argument does not ensure action use: correlated preceding states may still support prediction without action tokens.Source: §A.2 · pp. 14–15
  • Inverse supervision addresses the separate action-use limitation by requiring command recovery from predicted transitions.Source: §A.2 · p. 15

A.3 ARCHITECTURE

SKYTOPIA combines a vision-language backbone and diffusion action head with forward and inverse latent-dynamics heads. Special query tokens separate transition prediction from action generation, and only the backbone and action head remain at deployment.

A.3 ARCHITECTURE

  • Only the backbone and action head are retained at deployment; the forward and inverse heads are training components.The backbone is Qwen3-VL-2B-Instruct, while the forward and inverse heads contain 160M and 2.1M parameters, respectively.Source: §A.3 · p. 15
  • The action head is a 160M-parameter diffusion transformer with 16 layers and hidden width 768.The frozen target encoder is V-JEPA 2 ViT-L/16, with 300M parameters and 256 × 256 input resolution, encoding 8-frame clips with tubelet size 2.Source: §A.3 · p. 15
  • Eight prediction tokens per transition and 32 action tokens per backbone pass are inserted at fixed input positions.The backbone hidden states at those positions form the action tokens; the image-goal prompt additionally identifies the final image as the goal view.Source: §A.3 · p. 15

A.4 TRAINING CONFIGURATION

Training uses separate learning rates for the backbone, action head, and dynamics heads, with action normalization based on training-set extrema. Goal indicators are jointly sampled, with temporally offset goal views and photometric jitter when image goals are used.

A.4 TRAINING CONFIGURATION

  • Actions are normalized to [−1, 1] using training-set extrema, and the same statistics recover physical units at deployment.Training uses separate learning rates for the backbone, action head, and dynamics heads, and runs on 8 H200 GPUs for approximately 56 hours.Source: §A.4 · p. 15
  • The four joint point-goal and image-goal indicator combinations are sampled with probabilities 0.20, 0.30, 0.40, and 0.10, respectively.When an image goal is present, its view is sampled 0.5–4.0 seconds after the current frame, and both views receive photometric jitter of strength 0.3.Source: §A.4 · p. 15

A.5 EVALUATION PROTOCOL

Evaluation uses closed-loop rollouts in held-out scenes, with success requiring arrival within 0.5 m without collision. The protocol reports navigation, safety, and efficiency metrics, and image-goal navigation supplies updated visual subgoals along the reference path.

A.5 EVALUATION PROTOCOL

  • An episode ends at success, collision, out-of-bounds, or timeout after 3000 control steps; success requires stopping within 0.5 m without collision.Collision uses the scene SDF, and out-of-bounds means leaving the reconstruction; the oracle success threshold is 1.0 m closest approach.Source: §A.5 · p. 15
  • The protocol reports terminal distance, success and oracle-success rates, collision and out-of-bounds frequencies, successful-episode steps, and SPL path efficiency.SR is the fraction of successful episodes; OS is the fraction approaching within the oracle threshold.Source: §A.5 · p. 16
  • Image-goal navigation updates the visual subgoal as the drone advances, supplying a local target when the destination is initially outside the camera view.The planner provides a sequence of subgoals along its reference path.Source: §A.5 · p. 16

A.6 BASELINE ADAPTATION

The appendix describes aerial baseline adaptations, real-world flight conditions, ablations, and additional analyses. Results indicate that spatial goal features and conditional flow matching benefit navigation, while performance remains comparatively strong over long routes and visual perturbations.

A.6 BASELINE ADAPTATION

  • Baseline methods are retrained for aerial navigation on the SKYTOPIA dataset and receive monocular RGB streams; waypoint outputs use the demonstration controller for velocity commands.The appendix summarizes these adaptations and computational costs.Source: §A.6 · p. 16, Table 5 · p. 17

A.7 REAL-WORLD DEPLOYMENT SETUP

  • Real-world tests use a Tello with a forward-facing 720p, 30-FPS camera; inference runs at approximately 7 Hz on a ground computer and speed is limited to 1 m s−1.Indoor flights use OptiTrack pose updates for metric goals, while outdoor flights use onboard IMU orientation without motion-capture measurements.Source: §A.7 · p. 16, Figure 9 · p. 16
  • The three flight sites vary from a 6 m × 6 m room to a 30 m × 30 m woodland area, testing transfer across appearance, obstacles, and visibility.Routes use manually specified starts and goals; image-goal flights use visual subgoals recorded on a prior flight, while an operator judges arrival in goal-free navigation.Source: §A.7 · pp. 16–17

A.8 ADDITIONAL ABLATIONS

  • Pooling goal features substantially reduces image-goal success and path efficiency, while point-goal and goal-free performance remain close to the full model.The reported effect is concentrated in the mode using a goal image; the authors attribute this to spatial correspondences supporting visual alignment.Source: §A.8 · p. 17, Table 6 · p. 17
  • Replacing conditional flow matching with regression reduces success and path efficiency across all three goal modes.The authors propose that flow matching models multiple valid action sequences that regression may average into unsuitable commands.Source: §A.8 · pp. 17–18, Table 6 · p. 17

A.9 ADDITIONAL ANALYSIS

  • Across three goal modes, SKYTOPIA maintains higher success and path efficiency across route lengths, with a substantial advantage on the longest routes.The analysis groups episodes by planner path length; baseline performance declines more sharply at the longest horizon.Source: §A.9 · p. 18, Figure 10 · p. 18
  • Under illumination changes, image noise, and motion blur, SKYTOPIA has the highest success rate and retains a larger fraction of clean performance than baselines.Motion blur causes its largest decline, consistent with sensitivity to spatial image cues.Source: Figure 11 · p. 18, §A.9 · p. 19
  • The action-recovery probe finds that preceding-state-only inputs predict commands effectively, and an unrelated next state preserves much of the aligned pair's performance.To reduce this shortcut, training independently zeros the preceding state with probability 0.3, encouraging command recovery to use the predicted next state.Source: §A.9 · p. 19, Table 7 · p. 19
Loading 2609.26007v1…