PaperlayerView exact revision on arXiv

Source-linked AI summary

Hydra: A Navigation World Action Model with Discrete Latent Planning and Continuous Flow-Matching Execution

Mohammad Nazeri, Alexandyr Card, Samira Huber, Anuj Pokhrel, Yujun Wang, Ruben Hammele, Daeun Song, Sören Pirk, Xuesu Xiao

arXiv:2608.28995v1cs.ROcs.CV

TL;DR

Reactive policies lack foresight, while external-planner world models face representation misalignment and costly pixel-space evaluation. Hydra addresses this with unified discrete latent planning and continuous Flow Matching execution, and physical deployments show stronger goal-directed planning than world-model baselines while matching or exceeding reactive-policy execution.

  • Problem

    External-planner world models operate on decoupled manifolds and must evaluate sampled candidates through costly pixel-space decoding, limiting real-time physical control.

  • Method

    Hydra jointly represents visual states, poses, and actions in a unified manifold, uses discrete VQ codebooks and Kinematic-Perceptual Cost for planning, then applies Flow Matching for continuous execution.

  • Results

    Hydra outperforms continuous world-model baselines in real-time goal-directed planning while matching the execution precision of dedicated behavior-cloning policies.

  • Takeaways & Limitations

    The unified discrete manifold supports latent-space safety evaluation and planning without the pixel-decoding bottleneck of continuous baselines.

  • Takeaways & Limitations

    Hydra faces a trade-off between codebook expressivity and search latency, since larger vocabularies improve behavioral coverage but increase nearest-code retrieval cost.

Abstract

from arXiv · show

World models let robots imagine possible futures, but exploiting this capability for real-time control is bottlenecked by a representation misalignment: the generative model and the planner operate on decoupled manifolds, so the planner has no shared structure to search over and must instead decode every candidate back into high-dimensional pixel space to evaluate it. This decoding step is a major obstacle to real-time control on physical hardware. In this paper, we present Hydra, a discrete World Action Model that closes this gap by moving the planner, both the sampler and the evaluator, inside the model. Hydra establishes a unified latent manifold over visual states, physical poses, and control actions, then compresses this manifold through modality-specific Vector-Quantized bottlenecks into discrete vocabularies of kinodynamic intents and visual states. Because candidates are now drawn directly from this shared manifold, sampling is informed by the model's own understanding of the observation rather than proposed blind, and evaluation happens natively within the discrete space: candidates are ranked by a Kinematic-Perceptual Cost, without ever decoding to pixels. We term this Discrete Latent Planning (DLP). Because planning over discrete intents alone cannot supply the smooth, continuous commands physical actuation requires, Hydra pairs DLP with conditional Flow Matching, which maps each selected intent to a continuous trajectory for execution. Evaluated on two physical robotic platforms, Hydra outperforms state-of-the-art world models in goal-directed planning, while matching or exceeding the closed-loop execution capabilities of leading reactive foundation policies.

1 Introduction

Hydra addresses the mismatch between world-model generation and external planning by placing sampling and evaluation inside a shared latent manifold. Its discrete planning and continuous execution design supports real-world navigation without pixel-space evaluation.

  • Reactive policies cannot simulate candidate-action consequences, leaving them vulnerable to local minima and unable to plan.
  • External-planner world models search an unconstrained space whose size grows with horizon length, forcing sparse sampling or prohibitive computation.
  • Hydra jointly learns world dynamics, robot dynamics, and action policies in a bounded latent manifold, so sampling is observation-informed and evaluation avoids pixel decoding.
  • Discrete Latent Planning searches a finite vocabulary of physically feasible kinodynamic intents rather than an unbounded continuous action space.
  • Kinematic-Perceptual Cost ranks discrete candidates using uncertainty and quantization error, after which Flow Matching converts the selected intent into continuous poses or actions.
  • Hydra improves planning over NWM while matching dedicated behavior-cloning policies on execution precision in physical deployment.

2 Related Work

Prior navigation systems trade off reactive efficiency, generative foresight, and planning cost. Hydra’s related-work position combines early multimodal alignment with discrete latent search to reduce continuous-planning waste.

  • Reactive Navigation Policies: Reactive navigation policies map observations directly to actions but lack foresight about future environmental consequences.
  • Generative World Models: Generative world models provide foresight but often separate prediction from planning, creating latency, physical inconsistency, or weak coupling between modalities.
  • Planning with a Model: Traditional model-based planners such as MPPI and CEM waste computation sampling implausible trajectories in unconstrained continuous spaces.
  • Planning with a Model: WorldPlanner discretizes its MCTS search space with a diffusion model, whereas Hydra discretizes kinodynamic and visual representations into finite learned vocabularies.
  • Early vs In-context Alignment: In-context alignment interleaves modalities and requires attention to solve both cross-modal alignment and temporal correlations.
  • Early vs In-context Alignment: Hydra instead fuses observation and action into one token before sequence modeling, reserving model capacity for physical dynamics.

3 Hydra

Hydra formulates navigation as unified multimodal prediction and planning over discrete codebooks. It searches and evaluates latent intents before decoding only the selected trajectory into executable continuous controls.

  • Navigation Paradigms: Reactive policies directly output immediate actions or waypoints but cannot predict future visual or spatial consequences.
  • Navigation Paradigms: Decoupled visual world models condition predicted images on sampled continuous actions, requiring infeasible-sequence evaluation and modality decoding.
  • Navigation Paradigms: Hydra jointly models visual observations, poses, and actions over a horizon, with the planner operating inside the model manifold.
  • Unified Encoding: The encoder fuses multimodal history with horizon-length prediction tokens into context tokens that compress where, what, and how information.
  • Discrete Latent Planning: DLP searches the learned discrete manifolds of visual, pose, and action codebooks instead of sampling blindly in a dense continuous domain.
  • Kinematic-Perceptual Cost: KPC combines goal, kinodynamic-prior, entropy, and quantization-error terms to evaluate candidate safety within latent space.
  • Inference and Execution: Iterative refinement evaluates candidates in embedding space and calls Flow Matching only once to decode the elite trajectory.
  • Global Guidance: High guidance scales can push latent vectors outside the VQ codebook, causing downstream decoders to produce physically impossible dynamics.

4 Experiments

Hydra’s experiments evaluate generative grounding, real-time discrete planning, and physical path following. Across these tests, Hydra preserves long-horizon temporal consistency, plans quickly with high obstacle-navigation success, and matches or exceeds reactive baselines under sparser guidance, while exposing horizon- and visibility-dependent limitations.

  • Generative Grounding: Hydra outperforms NWM in FVD and maintains temporal grounding across the full 16-second horizon, whereas NWM decouples from commanded actions after approximately 8 seconds.The evaluation uses 100 held-out RECON clips generated at 4 FPS under teacher forcing.
  • Real-Time Planning: NWM requires over 500 seconds for one CEM search, while continuous VertiFormer plans in approximately 0.7 seconds but achieves 0% success with obstacles.Both baselines expose complementary weaknesses of continuous search: catastrophic latency or collision-prone sample inefficiency.
  • Real-Time Planning: Hydra plans in approximately 0.9 seconds, a roughly 500× speedup over NWM, and achieves 10/10 success in unobstructed scenarios and 8/10 in occluded and blind-corner scenarios.The discrete manifold supplies physically feasible candidate trajectories without pixel decoding.
  • Latent Collision Detection: Quantization error and predictive entropy rise together near occlusions and fall after hallucinated navigable resolutions, but visually complex foliage can trigger false positives.These signals provide collision-relevant information, yet dense textures can resemble genuine obstacle-induced reconstruction difficulty.
  • Local Path Following: With sparse waypoints, Hydra’s local-following variants match or exceed reactive baselines’ success rates and intervention counts in every tested scenario despite receiving less frequent correction.Reactive baselines receive dense goal images sampled at 2 Hz, whereas Hydra receives more widely spaced waypoints.
  • Local Path Following: CFG degrades on Long Distance, while Sampling is more reliable for long-horizon following because it stays within the valid codebook, trading speed on shorter scenarios.CFG reaches 138 ± 7.7 seconds and 2.4 ± 0.9 interventions on Long Distance; the reported failure is linked to compounded manifold deviation.
  • Local Path Following: Hydra’s planning mode reaches 90% success on Obstructed Goal and 100% on Indoor Corridor, but only 40% on Long Distance because iterative search reacts too late at an unrecoverable fork.On Long Distance, planning also records 119 seconds traversal time and 3.4 ± 0.6 interventions.
  • Ablations: Activating Cimg when the goal is not visible reduces success from 80% to 20%, so subsequent experiments disable the image-goal term in blind-corner conditions.Without a visual goal anchor, visually plausible candidates can minimize the term without being geometrically correct.

5 Limitations and Future Work

Hydra’s discrete representation introduces deployment limitations involving safety signals, dynamic-agent representation, and the trade-off between vocabulary expressivity and real-time latency. These limitations constrain reliability and generalization in visually complex, socially dynamic, or novel environments.

  • Hydra’s discrete representation limits downstream navigation behavior in empirical deployment.The paper frames this as a fundamental challenge arising from the discrete representation itself.
  • Manifold Rejection Confuses Visual Complexity with Genuine Risk: Manifold Rejection can confuse visually complex safe scenes with genuine collision risks.Foliage and grass can produce high quantization error despite being traversable, while homogeneous obstacles may be under-flagged; false positives and false negatives remain possible.
  • Semantic Omission and Agent Erasure: Hydra frequently erases or smooths pedestrians into the background in complex social scenarios.The paper attributes this challenge partly to the limited visual codebook, with |Vi| = 2048.
  • Representational Capacity Trades Expressivity for Latency: Larger VQ codebooks could improve generalization to novel environments but increase nearest-codebook retrieval cost.Vocabulary size therefore creates a direct trade-off between representational expressivity and real-time latency.

6 Conclusion

Hydra bridges generative planning and real-time physical execution by unifying visual states, poses, and actions in a discrete latent space. Physical deployments show real-time goal-directed planning at deployment-compatible scale and latency, while the approach remains bounded by codebook expressiveness.

  • Conclusion: Hydra unifies visual states, poses, and actions through early fusion and a Vector-Quantized bottleneck, restricting planning to learned kinodynamic primitives.Its Kinematic-Perceptual Cost evaluates safety and search largely in latent space without pixel decoding.
  • Conclusion: Hydra outperforms continuous baselines such as NWM and VertiFormer in real-time goal-directed planning during extensive physical deployments.The deployments operate at a scale and latency compatible with eventual onboard deployment.
  • Conclusion: Hydra’s effectiveness depends on whether its representation is expressive enough to capture behaviors omitted by the current codebook.The authors identify adaptive codebook capacity and explicit dynamic-agent representations as future directions.

A.1 Single-Step Image Quality

On held-out RECON data, Hydra provides competitive single-step visual prediction with fewer parameters and substantially fewer function evaluations than NWM, although VertiFormer leads PSNR.

  • A.1 Single-Step Image Quality: Hydra achieves competitive DreamSim and LPIPS reconstruction against NWM/B and VertiFormer on held-out RECON data.The comparison evaluates single-step visual predictive fidelity.
  • A.1 Single-Step Image Quality: Hydra uses 50 NFEs for multimodal encoding, compared with NWM’s 250 NFEs, while maintaining competitive structural reconstruction with a smaller parameter footprint.VertiFormer achieves the highest PSNR, but the passage notes that PSNR is misaligned with dynamic perception.

A.2 Long-Horizon Qualitative Comparison

Across a 16-second held-out RECON video-generation comparison, Hydra maintains command-grounded visual predictions while NWM’s continuous latents accumulate errors and drift from the actions.

  • A.2 Long-Horizon Qualitative Comparison: After approximately 8 seconds, NWM’s visual context rapidly degrades and decouples from commanded actions, whereas Hydra remains physically command-aligned through 16 seconds.The figure provides the qualitative counterpart to the FVD comparison in Table 1.

A.3 Analysis of the Kinodynamic Codebook

Hydra’s learned kinodynamic codebook grounds discrete planning in absolute physical scale and constrains search to plausible maneuvers. Continuous Flow Matching then restores fine-grained trajectories for execution, while latent distances and uncertainty support candidate evaluation without pixel decoding.

  • A.3 Analysis of the Kinodynamic Codebook: Continuous action spaces preserve fine-grained control but are sample-inefficient; Hydra uses Flow Matching to decode discretized primitives into continuous execution trajectories.The discretized actions can be interpreted as the mean of the trajectory decoded by the Flow-Matching head.
  • A.3 Analysis of the Kinodynamic Codebook: Hydra’s learned codebook gives distances between entries a meaningful relation to the world-model manifold, supporting topological safety evaluation through KPC.This meaning depends on the codebook and world model sharing a coupled representation.
  • A.3 Analysis of the Kinodynamic Codebook: Unlike normalized baselines, Hydra’s unnormalized pose codebook preserves absolute physical scale, separating low-speed indoor clusters below 0.5m from high-speed off-road bounds above 1.0m.The learned pose codebook spans dense central clusters for low-speed maneuvers and sparse outer regions for high-speed dynamics.
  • A.3 Analysis of the Kinodynamic Codebook: Contextual Manifold Shift occurs when incompatible visual context and velocity conditioning move predictions toward the environmental sub-manifold where the commanded speed is physically valid.The passage frames this as resolving conflicting modalities rather than a standard generative failure.
  • A.3 Analysis of the Kinodynamic Codebook: Bounding searches to the physically grounded codebook prevents OOD speed-context mismatches and permits deployment-time behavior control by masking selected codes.Examples include masking high-speed codes to prevent fast driving or masking right-turn codes.
  • A.3 Analysis of the Kinodynamic Codebook: Hydra’s RViz visualization shows a branching tree of discrete candidate trajectories, with JKPC selecting the lowest-cost branch for execution.The search considers a small vocabulary of plausible topological maneuvers rather than an unconstrained continuous spread.

A.5 Failure Analysis

Hydra exhibits failure modes tied to visual codebook capacity and goal-blind initialization, while its design choices address representation stability and planning efficiency.

  • Visual prediction failures: Pedestrians progressively disappear from Hydra’s SCAND predictions despite remaining visible in the ground-truth future frames.The failure is attributed to visual codebook quantization smoothing small human regions into the static background.
  • Planning failures: Goal-blind seed initialization can miss an early turn because goal information enters only indirectly through later Kinematic-Perceptual Cost updates.With N = 3 iterations, the planning budget may leave too little time for the goal signal to steer the candidate population.
  • Design tradeoff: Goal-conditioned seeding resolves near-turn failures but clusters candidates and reduces search diversity.This exposes a tradeoff between faster convergence toward the goal and broader exploration of trajectory space.

C Architecture and Implementation Details

Hydra combines early-fused multimodal representations, modality-specific VQ codebooks, and conditional Flow Matching decoders within a modular architecture designed for physical navigation.

  • Architecture: Hydra modularly separates visual-proprioceptive encoders, a unified spatio-temporal TransformerEncoder, discrete codebooks, and multimodal latent flow decoders.The architecture combines a Transformer backbone with modality-specific VQ codebooks and Flow Matching decoders.
  • Training: A cosine-annealed infilling curriculum decays fully observable training probability from 0.5 to 0.1 over 10 warmup epochs.The remaining probability is split between Forward Kinodynamics and Inverse Kinodynamics masking states.
  • Architecture: Early fusion sums visual, pose, and action modalities before Transformer processing without introducing extra fusion parameters.Sinusoidal positional encoding is then added to the unified representation.
  • Decoding: Conditional Rectified Flow Matching synthesizes continuous trajectories and high-fidelity visual latents from discrete intent vectors.AdaLN-Zero modulates the flow-matching framework, with capacity tailored to each modality’s spatial-temporal and latency constraints.
  • Decoding: Action and pose heads omit self-attention because iterative O(N^2) attention is prohibitive for high-frequency real-time inference.The visual head instead uses a full DiT block to preserve spatial and temporal structural coherence.
  • Training: Training uses 100,355 samples spanning off-road, indoor, outdoor, and social navigation scenarios.The synchronized tuples contain observations, poses, and actions for joint multimodal learning.
  • Training: Training is end-to-end with the AutoencoderKL frozen, while rollout targets vary uniformly between 1 and 3 seconds.The maximum horizon is H = 12 steps at 4Hz.

D.1 Projection Capacity and Codebook Utilization

Hydra’s linear projection improves codebook utilization and stabilizes discrete-to-continuous decoding, while DLP combines two stochasticity sources for diverse, valid candidate trajectories.

  • Projection capacity: 608.36 image codebook perplexity with the linear projection exceeds 465.42 with the 2-layer MLP, despite 100% code usage for both.The result indicates more uniformly distributed visual representations under the strict linear bottleneck.
  • Projection capacity: The deterministic anchoring head stabilizes randomly initialized codebooks before continuous Flow Matching decodes them.Without the head, early generation shows severe instability and spatial smearing; with it, structural details emerge earlier.
  • Two sources of stochasticity: Site 1 perturbs continuous intent queries before decoding, producing genuinely different candidate trajectory shapes with coarse-to-fine noise annealing.The perturbations are temporally smoothed so alternatives remain physically coherent rather than jittery.
  • Two sources of stochasticity: Site 2 applies Top-K Gumbel-max discretization, adding local exploration while restricting candidates to nearby valid codebook vectors.The mask limits sampled intents to the physically plausible manifold, and temporal smoothing supports stable control.
  • Two sources of stochasticity: Both stochasticity sites are necessary: Site 1 changes the model’s hypotheses, while Site 2 makes their discretization robust without collapsing to one nearest neighbor.Their smoothing and annealing schedules serve different search purposes.
  • Candidate generation: DLP re-quantizes mixed and pure codebook candidates before decoding, ensuring that only valid discrete codes enter the decoder.Candidate budgets can combine pure codebook entries with convex mixes of the scene-conditioned prior and vocabulary.
  • Candidate evaluation: Candidates are decoded with a deterministic pose head so their scores remain comparable across the batch.This avoids the high per-seed variance of stochastic ODE decoding for maneuver-code candidates far from the prior.
Loading 2608.28995v1…