PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffWAM: A Fast and Efficient Navigation World Action Model

Mo Zhu, Yuze Wu, Xijie Huang, Xiao Cui, Fei Gao, Xin Zhou

arXiv:2609.39763v1cs.ROcs.AI

TL;DR

Continuous UAV instructions require executable trajectories, but video-model navigation pipelines often generate future video and reconstruct motion. DiffWAM reads motion directly from predictive features and first-frame geometry; its configurations lead compared methods on endpoint success across three benchmarks, and DiffWAM-1000 reports a trajectory RMSE of 0.3492 m.

  • Problem

    Video-model navigation commonly relies on future-video synthesis and geometric reconstruction, while continuous, structured UAV instructions call for executable trajectories.

    Source: §1 · p. 2

  • Method

    DiffWAM maps multi-level predictive features and first-frame geometry to camera trajectories, while FastDreamer overlaps prediction with flight and schedules timestamp-aware handoffs.

    Source: §1 · pp. 2–3

  • Results

    DiffWAM configurations achieve the highest endpoint-based success rates among compared methods on three benchmarks; on DiffWAM-1000, trajectory RMSE is 0.3492 m.

    Source: §1 · p. 3, §6 · p. 31

  • Takeaways & Limitations

    Predictive-to-motion grounding provides a direct alternative to generate-then-reconstruct pipelines for continuous UAV navigation.

    Source: §6 · p. 31

    Computation hides waiting only when a valid, feasible proposal is ready for its scheduled handoff; estimated latency is not a deterministic runtime guarantee.

    Source: §4.2 · p. 15

Abstract

from arXiv · show

Pretrained video foundation models encode rich semantic and spatiotemporal priors for embodied navigation, yet converting these priors into UAV motion typically requires expensive future-video synthesis and geometric reconstruction. We investigate whether the motion implicit in future visual prediction can instead be recovered directly from the predictive representations of a frozen video model. To this end, we present DiffWAM, a geometry-conditioned navigation world-action model that directly transforms multi-level predictive features into continuous camera trajectories. Its Grid-Motion module preserves spatial-temporal motion associations, while Latent2Pose grounds them with first-frame geometry to recover metrically meaningful 3D motion. Complete video rollouts and geometric reconstruction are required only for offline supervision, eliminating future-video decoding and multi-frame reconstruction during deployment. We further introduce FastDreamer, which overlaps predictive and geometric computation with ongoing flight and performs timestamp-aware asynchronous trajectory handoff for continuous UAV execution. DiffWAM achieves a trajectory RMSE of 0.3492 m and an endpoint success rate of 74.40% on the 1,000-sample DiffWAM-1000 benchmark, while representative real-world experiments demonstrate complex behaviors including constrained traversal, orbiting, S-shaped flight, and multi-stage navigation. An onboard DiffWAM-Flash implementation further reaches 1.08 s model-pipeline latency on NVIDIA Jetson AGX Thor. These results demonstrate that predictive video representations can be efficiently grounded into continuous 3D motion, providing a direct alternative to generate-then-reconstruct navigation pipelines. Project page: https://zzmmzzm.github.io/diffwam.github.io/.

1 Introduction

The introduction identifies a gap between understanding where to go and producing continuous, executable UAV trajectories, and asks whether predictive video features can directly reveal motion. DiffWAM grounds those features with first-frame geometry, while FastDreamer coordinates prediction with ongoing flight.

1 Introduction

  • Existing vision-language navigation systems are better at deciding where to go or what to do next than translating complex instructions into continuous executable trajectories with spatial constraints.Video foundation models learn how scenes evolve over time, motivating investigation of whether their intermediate predictive representations contain recoverable navigation motion before full video generation.Source: §1 · p. 2
  • DiffWAM directly maps multi-depth predictive features to time-indexed camera poses, using Grid-Motion for spatial-temporal associations and geometry-conditioned Latent2Pose for metric grounding.The method preserves spatial-temporal structure and conditions decoding on geometry from the initial observation.Source: §1 · pp. 2, 4
  • Complete video generation and geometric reconstruction provide offline trajectory supervision, while the student learns from early predictive features and first-frame geometry with frozen backbones.At deployment, complete video rollout, video decoding, and multi-frame geometric reconstruction are removed.Source: §1 · pp. 2–3
  • DiffWAM achieves a trajectory RMSE of 0.3492 m and an endpoint success rate of 74.40% on DiffWAM-1000, with real-world flights demonstrating structured and multi-stage motions.The reported behaviors include constrained traversal, orbiting, S-shaped motion, landing, and multi-stage navigation.Source: §1 · p. 3
  • FastDreamer overlaps motion-proposal preparation with flight and uses observation timestamps and scheduled handoffs to account for UAV motion during inference.It extends trajectory prediction toward continuous, multi-stage vision-language navigation.Source: §1 · pp. 3–4

2 Related Work

Related work spans video-model action and navigation, predictive-representation grounding, and asynchronous execution. DiffWAM and FastDreamer are positioned around continuous camera trajectories and handoffs aligned with the moving vehicle.

2.1 Video World Models for Embodied Action and Navigation

  • Video-model navigation ranges from future-video generation and reconstruction [12] to latent transitions and waypoints [13], while Fast-WAM [7] and Faster-WAM [14] show action prediction without complete rendering at test time.DiffWAM instead grounds intermediate predictive representations of a frozen video model into continuous camera trajectories.Source: §2.1 · p. 4

2.2 Predictive Representation Grounding and Distillation

  • DiffWAM uses first-frame geometry for spatial reference and metric scale, distilling motion from completed rollouts into a direct trajectory readout rather than matching teacher logits or action distributions.The discussion relates this approach to knowledge and policy distillation [20]; π3 [15] reconstructs camera motion and geometry from image collections, while MoGe-2 estimates metric geometry from one image.Source: §2.2 · pp. 4–5

2.3 Latency-Aware Continuous Execution

  • Prior latency-aware approaches include receding-horizon execution, overlapping action generation with execution, asynchronous semantic reasoning, and pose-aware alignment; FastDreamer prepares proposals during flight and ties them to observation time and handoff state.This addresses the spatial inconsistency that can arise as a mobile robot moves between observation and action activation.Source: §2.3 · p. 5

3 DiffWAM: Grounding Predictive Representations into Motion

DiffWAM reads early predictive features from a frozen video model and combines them with first-frame geometry to produce time-indexed camera poses. Its staged training uses completed video rollouts for offline trajectory supervision, while deployment omits future-video decoding and multi-frame reconstruction.

3 DiffWAM: Grounding Predictive Representations into Motion

  • DiffWAM maps retained predictive features and first-frame geometry to camera-pose trajectories through Grid-Motion and Latent2Pose, with completed rollouts reserved for offline supervision.The video backbone and MoGe2 remain frozen during readout training.Source: §3 · p. 5

3.1 Problem Formulation

  • Over a nominal five-second horizon, each pose gives orientation and position relative to the initial camera frame, but the finite pose sequence alone does not guarantee dynamically feasible flight.Pose queries have prescribed physical-time offsets; interpolation and executable trajectory construction belong downstream.Source: §3.1 · p. 6

3.2 Efficient Predictive Motion Readout

  • Grid-Motion preserves the unpooled spatial layout while associating predictive grids across time and with geometry-conditioned anchors before pooling motion-memory tokens.The architecture uses learned soft correspondences rather than measured optical flow or imposed task-specific trajectory templates.Source: Figure 3 · p. 8, §3.2 · pp. 8–9

3.3 Geometry-Guided Trajectory Decoding

  • The decoder conditions pose prediction on first-frame geometry and scene-level scale, using 38 learned future-pose queries to produce three translation coordinates and a six-dimensional rotation representation.When no valid depth estimate is available, the implementation uses unit scale.Source: §3.3 · pp. 9–10

3.4 Geometry-Guided Predictive Distillation

  • Geometry-guided predictive distillation pairs early features with trajectories reconstructed from their own completed rollouts, training the readout to recover teacher-associated motion.S1 initializes the readout from corpus trajectory supervision; S2 adds same-rollout correspondence, though the reconstructed references remain subject to generation and reconstruction errors.Source: §3.4 · pp. 10–12

3.5 Inference

  • At deployment, predictive and geometry branches can run concurrently, with no completed video, online reconstruction, or teacher trajectory required; a downstream planner checks feasibility.The predicted poses are proposals, not certified collision-free commands; calibrated extrinsics and state estimation map them to the execution frame.Source: §3.5 · pp. 12–13

4 FastDreamer: Inference and Continuous Execution

FastDreamer connects DiffWAM to continuous UAV control by sharing prompt-rewriting weights, overlapping proposal preparation with flight, and aligning updates to a scheduled future handoff. Its timing and transition checks account for the changing vehicle state and preserve task requirements when activating a new trajectory.

4 FastDreamer: Inference and Continuous Execution

  • FastDreamer connects DiffWAM to continuous UAV control through shared-weight prompt rewriting, overlapping proposal preparation with flight, and scheduled future handoffs.The framework emphasizes onboard deployment of DiffWAM-Flash on the Thor platform.Source: §4 · p. 13

4.1 Shared-Weight Prompt Rewriting and Parallel Proposal Inference

  • The rewriter preserves active-subtask constraints using mission-manager context, while geometry estimation runs in parallel with rewriting and prediction.The mission manager, not frame-wise rewriting, updates task progress and declares completion; DiffWAM-Flash uses one truncated predictive evaluation, and neither variant decodes future video online.Source: §4.1 · pp. 13–14

4.2 Flight-Time Compute

  • Preparation is hidden from execution only when a valid, feasible update is ready for its scheduled handoff; zero overrun alone does not establish acceptance or timely activation.The latency estimate is an admission condition, not a deterministic runtime guarantee; fallback must begin by the reserved deadline if no usable update is ready.Source: §4.2 · p. 15
  • An early proposal waits for its scheduled activation and is rechecked; the figure contrasts four backbone evaluations for DiffWAM with one for DiffWAM-Flash.Both configurations stop the final evaluation after DiT block 35 without future-video decoding.Source: Figure 4 · p. 15

4.3 Prospective Handoff and Safety Boundary

  • Prospective handoff predicts the activation state from the latest timestamped estimate and committed motion, then requires revalidation if timing or state mismatch invalidates the transition.Capture-time camera poses are lifted into the world frame, and transition planning must retain maneuver order and task requirements rather than merely matching the endpoint.Source: §4.3 · p. 16
  • At handoff, commanded references must remain continuous, but reference continuity does not guarantee that the actual vehicle exactly matches the nominal reference.For repeated or self-intersecting motions, elapsed time or geometric proximity alone cannot establish task progress.Source: §4.3 · pp. 16–17

5 Results

The experiments evaluate DiffWAM across benchmark, simulation, real-world, deployment, and ablation settings. They address comparative performance, execution of structured motion, onboard feasibility, and the effects of design choices.

5 Results

  • The evaluation spans benchmark comparisons, simulation and real-world execution, onboard deployment, and ablations of DiffWAM’s design choices.The questions include performance across aerial-navigation tasks, structured-motion execution, onboard pipeline feasibility, and contributions of representation, predictive computation, geometry, and training choices.Source: §5 · p. 17
  • The results section first establishes datasets, metrics, baselines, and protocol, then reports quantitative comparisons and simulation results.It next evaluates real-world navigation and onboard deployment before presenting systematic ablations.Source: §5 · p. 17

5.1 Experimental Setup

The experimental setup combines synthetic aerial-navigation training data with a 1,000-sample evaluation benchmark spanning diverse task families. It assesses trajectory and endpoint accuracy, task completion, and deployment efficiency under a scene-separated protocol.

5.1.1 Training Datasets

  • Training combines FlyMirage’s synthetic 3D environments and dynamically feasible trajectories with NavGen’s video-model-generated task-conditioned trajectories, totaling more than 510K samples.FlyMirage contributes about 1,100 scenes and over 110K trajectories; NavGen contributes approximately 400K trajectories, including complex motions such as cave traversal and orbiting.Source: §5.1.1 · pp. 17–18

5.1.2 Evaluation Datasets

  • DiffWAM-1000 contains 1,000 test samples across 18 task types, grouped into basic motion, object interaction, spatial navigation, and scene understanding.The families account for 10%, 65%, 15%, and 10% of the evaluation set, respectively; object-interaction tasks include target-relative motion such as orbiting and side-specific passing.Source: §5.1.2 · p. 18, Figure 5 · p. 18

5.1.3 Evaluation Metrics

  • Evaluation measures endpoint and trajectory accuracy, rotation error, closed-loop task success, and deployment efficiency.Trajectory errors are computed in the initial camera frame without post-hoc rigid transformation or scale alignment; success criteria include task-specific conditions such as traversal direction, orbit coverage, and landing status.Source: §5.1.3 · pp. 19–20

5.1.4 Baselines and Evaluation Protocol

  • Benchmark comparisons use common instructions and observations where interfaces permit, scene-level data splits, and a shared endpoint-success criterion across the three datasets.Endpoint success does not establish completion of traversal direction, orbit coverage, intermediate motion, or landing requirements; Fig. 6 supplies qualitative trajectory examples.Source: §5.1.4 · p. 20, Figure 6 · p. 21

5.2 Benchmark Results and Analysis

DiffWAM leads the compared methods on all three endpoint benchmarks, while selected trajectories illustrate motion patterns that endpoint scores cannot fully assess.

5.2.1 DiffWAM Performance

  • Selected trajectories show direct target approaches adapting lateral motion to the instruction and initial viewpoint.Examples include approaching the bicycle from the right and observing the boat’s right side.Source: §5.2.1 · p. 22
  • Constrained-traversal and orbit examples require intermediate path geometry, including alignment through openings and direction-dependent arcs around objects.For full orbits, endpoint proximity alone may not reveal an incorrect intervening path.Source: §5.2.1 · p. 22
  • The landing sequence is a qualitative descent example, not an independently measured landing error or physical landing-success result.The examples illustrate several task-dependent trajectory shapes, while benchmark scores quantify endpoint performance separately.Source: §5.2.1 · p. 22

5.2.2 Overall Benchmark Performance

  • On DiffWAM-1000, success rates are 92.00% for basic motion, 70.35% for object interaction, 74.03% for spatial navigation, and 84.00% for scene understanding.The gains over the strongest family-specific competitors are 6.00, 13.73, 23.26, and 28.00 percentage points, respectively.Source: §5.2.2 · pp. 22–23
  • 56.77%, 91.42%, and 74.40% are DiffWAM’s success rates on IndoorUAV-VLA, UAV-FLOW-Sim, and DiffWAM-1000, leading WorldVLN by 17.45, 11.18, and 16.00 percentage points.All comparisons use the shared endpoint criterion.Source: §5.2.2 · p. 22, Table 2 · p. 23
  • Endpoint performance does not by itself verify traversal direction, orbit coverage, or landing completion.The reported success-rate criterion is endpoint-based and distinct from task-specific closed-loop completion.Source: §5.2.2 · p. 23, Table 2 · p. 23

5.3 Real-World Experiment

Physical demonstrations cover structured single- and multi-stage UAV behaviors, while onboard latency measurements show a 1.080 s DiffWAM-Flash pipeline on Jetson AGX Thor.

5.3 Real-World Experiment

  • The real-world setup spans cloud-assisted inference on two Orin NX platforms and complete onboard inference on an AGX Thor platform.Nine representative cases are selected qualitative demonstrations, not a complete record of evaluation trials.Source: §5.3 · p. 23, §5.3.1 · p. 24

5.3.1 Real-World Navigation Performance

  • Indoor physical demonstrations include orbiting, selecting an instructed opening, S-shaped flight, and traversal through a circular frame.The behaviors depend on intermediate trajectory geometry, not only direct object approach.Source: §5.3.1 · p. 24
  • Outdoor demonstrations approach a logo-marked wall and a fire hydrant identified by its relative position among candidate objects.Together, indoor and outdoor cases illustrate execution across different scene appearances and spatial-language relations.Source: §5.3.1 · p. 24

5.3.2 Long-Horizon Real-World Navigation

  • Case IV combines target disambiguation, side-specific bypassing, and landing, while Case IX requires passing the fan before changing target to the rock formation.The missions illustrate trajectory proposals used in multi-stage physical navigation with downstream planning and execution.Source: §5.3.2 · p. 24

5.3.3 Onboard Inference Performance

  • 1.080 s is DiffWAM-Flash’s P50 model-pipeline latency on AGX Thor, versus 3.796 s for DiffWAM; both report 97.60 GiB peak memory.The measured pipeline latency is not end-to-end navigation-update latency.Source: Table 3 · p. 26, §5.3.3 · pp. 26–27
  • Latency tests used 45 measurements per configuration and excluded instruction rewriting, external communication, and file export.The reported values therefore do not include all preparation and execution-interface stages of a navigation update.Source: §5.3.3 · p. 26
  • On eight H20 GPUs, DiffWAM-Flash’s plotted total is 835.2 ms, compared with 3,605.5 ms for DiffWAM and 10,080.2 ms for NavDreamer.The displayed DiffWAM pipelines omit future-video decoding and π3 reconstruction stages retained by NavDreamer.Source: Figure 9 · p. 26, §5.3.3 · p. 27

5.4 Ablation Studies

Ablations compare trajectory readouts, feature depths, predictive evaluations, supervision volume, and loss recipes; their outcomes depend on the metric and tested setting.

5.4 Ablation Studies

  • The ablations cover five design choices, with matched settings held fixed except for the factor under study where stated.The five-case architecture comparison is separate from the complete benchmark evaluation, and the backbones remain frozen.Source: §5.4 · p. 27

5.4.1 Different WAM Architectures

  • 0.3492 m RMSE and 0.4357 m FDE are DiffWAM’s lowest errors among the compared architectures; its head latency is 37.35/38.84 ms P50/P95.Compared with Faster-WAM, DiffWAM reduces RMSE by 50.19% and FDE by 59.63%, while its head is faster than Fast-WAM and Faster-WAM but not every architecture.Source: Table 4 · p. 27, §5.4.1 · pp. 27–28

5.4.2 Feature-Layer Selection

  • Mixed layers 15/25/35 yield the lowest positional errors, with 0.3492 m RMSE and 0.4357 m FDE; middle layers yield the lowest RoE at 2.4314°.The mixed configuration’s RoE is 2.9179°, so the best layer combination differs by pose metric.Source: §5.4.2 · pp. 28–29, Table 5 · p. 29

5.4.3 Predictive Computation

  • Four world-model evaluations reduce RMSE from 0.5012 to 0.3492 m, ADE from 0.4423 to 0.3151 m, and FDE from 0.6865 to 0.4357 m versus one evaluation.Four evaluations have the lowest reported errors across metrics but higher inference latency; the one-evaluation setting is used for DiffWAM-Flash.Source: Table 6 · p. 29, §5.4.3 · p. 29

5.4.4 Training-Data Volume

  • Growing generated supervision from 1k to 64k pairs reduces RMSE from 0.8203 to 0.4873 m, FDE from 1.4011 to 0.6884 m, and RoE from 9.9409° to 2.7894°.The models use the same 20,000 updates and global batch size of 32.Source: Table 7 · p. 30, §5.4.4 · p. 30

5.4.5 Loss Functions and Depth Normalization

  • Neither untuned two-term loss guarantees lower errors: the unnormalized formulation reaches 0.8120 m RMSE, versus 0.7288 m for the legacy composite objective.The untuned depth-normalized formulation further increases positional errors to 1.0728 m RMSE and 1.5092 m FDE.Source: Table 8 · p. 30, §5.4.5 · p. 30

6 Conclusion

DiffWAM converts frozen video-model predictive representations into language-conditioned camera trajectories, with completed rollouts and geometric reconstruction used for offline supervision. Its reported configurations achieve the highest endpoint-based success rates among compared methods on three benchmarks.

6 Conclusion

  • Multi-level features, Grid-Motion associations, and geometry-conditioned Latent2Pose connect predictive motion information to an observed spatial reference and estimated metric scale.Deployment avoids completing future-video synthesis; completed rollouts and geometric reconstruction instead provide offline trajectory supervision, with video and geometry backbones frozen during motion-readout training.Source: §6 · p. 31
  • FastDreamer combines shared-weight instruction rewriting, parallel predictive and geometric processing, flight-time scheduling, and timestamp-aware prospective trajectory handoff.Source: §6 · p. 31
  • Endpoint-based success rates were 56.77% on IndoorUAV-VLA, 91.42% on UAV-FLOW-Sim, and 74.40% on DiffWAM-1000, highest among compared methods.Physical-flight examples illustrate constrained traversal, orbiting, S-shaped motion, and multi-stage navigation. DiffWAM-Flash records 1.080/1.101 s P50/P95 model-pipeline latency on AGX Thor, excluding instruction rewriting and external communication.Source: §6 · p. 31
  • Ablations favor mixed-depth features, additional world-model evaluations, larger generated-supervision sets, and a tuned two-term loss for positional accuracy, while the legacy objective retains the lowest orientation error.Source: §6 · p. 31
  • Further evaluation should quantify task-specific closed-loop completion, physical-scale accuracy, and complete navigation-update latency across unseen environments.Source: §6 · p. 31

Related papers

Loading 2609.39763v1…