PaperlayerView exact revision on arXiv

Source-linked AI summary

Hydra-0: Action Flow for Generalist World Modeling and Control

Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield, Soha Pouya, Chenran Li, Yan Chang

arXiv:2608.18077v1cs.RO

TL;DR

Existing world models often remain tied to their training embodiment, motivating a transferable interface for predicting action consequences across heterogeneous robots and settings. Hydra-0 conditions a generalist world model on action flow and inverts the same interface for control, achieving lower motion error, strong open-loop evaluation agreement, and human-demonstration-based robot action generation. Its scope remains limited by qualitative wrist-camera control evidence, open-loop evaluation, and sensitivity to projection errors.

  • Problem

    Existing world models remain tied to their training embodiments, leaving a need for transferable foundation world models across embodiments, tasks, and environments.

  • Method

    Hydra-0 represents commanded robot motion as camera-plane action flow for forward video prediction and infers compatible robot motion from desired object flow for inverse control.

  • Results

    90.40% lower robot-motion error and 60.16% lower object-motion error than the action-conditioned Cosmos 2.5 baseline accompany zero-shot and data-efficient transfer, while RoboLab replayed and reference success rates correlate at r=0.96.

  • Takeaways & Limitations

    Action flow serves as a shared control interface connecting heterogeneous training data, visual prediction, open-loop policy evaluation, and robot control.

  • Takeaways & Limitations

    The evaluation is limited to open-loop policy evaluation and a qualitative DROID wrist-camera proof of concept, with broader settings left for future work.

Abstract

from arXiv · show

We introduce Hydra-0, a generalist world model conditioned on action flow, which represents robot actions as pixel motion. This shared visual interface enables generalist world modeling and control by learning action consequences across embodiments, tasks, environments, and video-generation backbones. Our best configuration achieves 90.4% lower robot-motion error and 60.2% lower object-motion error than our action-conditioned baseline, while supporting zero-shot composition and data-efficient adaptation. On the RoboLab benchmark, Hydra-0 achieves a Pearson correlation of r=0.96 between replayed and reference success rates. Finally, we uncover an emergent inverse mode of this interface: a world action model that predicts compatible robot motion from desired object flow transferred from a human demonstration. A trained action head maps the resulting latent features to executable actions without requiring task-specific expert robot demonstrations. Together, these results demonstrate the potential of action flow as a shared control interface connecting heterogeneous training data, open-loop policy evaluation, and robot control.

1. Introduction

Hydra-0 uses action flow—camera-plane motion of visible robot or object points—as a shared interface for world modeling and control across embodiments, tasks, environments, and video backbones. It supports forward prediction of robot-motion consequences and an inverse mode that infers compatible robot motion from desired object flow.

  • Action-flow interface: Action flow represents robot or object motion as sparse, image-plane trajectories, avoiding direct dependence on embodiment-specific commands.The representation describes visible motion in visual-observation coordinates and can encode both embodiment motion and task intent.
  • Action-flow interface: During forward prediction, executable commands are converted through controller-and-physics rollouts, robot geometry, and camera calibration into embodiment action flow.The resulting pixel-aligned trajectories condition the video model to predict the command’s visual consequences.
  • Inverse control: The inverse world action model infers compatible robot motion from desired object flow transferred from a human demonstration and maps latent motion features to executable actions.The action head operates without task-specific expert robot demonstrations.
  • Results: 90.40% lower robot-motion error and 60.16% lower object-motion error are achieved by the best configuration versus the action-conditioned Cosmos 2.5 baseline.The system also supports zero-shot composition and data-efficient transfer to robot–deformable-object interaction.
  • Results: Pearson correlation r=0.96 links replayed and reference success rates across 5 RoboLab policies in open-loop policy evaluation.The paper also demonstrates real-robot control by converting human-demonstrated object flow into executable actions.

2. Method

Hydra-0 represents robot and object motion as action flow in the image plane, enabling video prediction across heterogeneous embodiments and conditioning sources. The method constructs, samples, and injects these trajectories into a video backbone to predict future observations.

  • Action-flow construction: Action flow represents robot motion as camera-plane trajectories, allowing one interface to describe arms, grippers, and human hands without exposing native action spaces.The representation tracks visible points and their visibility across a prediction horizon.
  • Action-flow construction: Geometry-aware construction projects visible robot-surface points into the camera plane using controller-and-physics rollouts, robot geometry, and camera calibration.Visibility is retained only for projected points satisfying depth, image-bound, and rendered-depth checks.
  • Training-time sampling: Training samples four conditioning modes—Embodiment, Object, All, and None—to provide action, task-intent, grounding, or dropout signals.Embodiment selects acting-body tracks, while Object supplies manipulated-object motion or desired-motion flow.
  • Action-flow construction: Video-only construction recovers dense image-plane trajectories from videos and segments them with grounded embodiment and object masks when robot metadata or calibration are unavailable.The same trajectory representation is produced for training without privileged geometric metadata.
  • Video prediction: The resulting world model predicts future observations from action flow, using flow matching and backbone adaptation for efficient long-horizon generation.The pretrained backbone is largely frozen while selected embeddings and LoRA modules are trained; autoregressive conversion and DMD2 support robotics deployment.
  • Visual conditioning: Action-flow trajectories propagate first-frame latent features through time into a motion condition that is concatenated with noisy video latents before DiT patch embedding.A presence gate distinguishes trajectory-conditioned locations from unmodified visual context.

3. Datasets

Hydra-0's training corpus combines robot and human interaction data, emphasizing deformable-object manipulation and standardized camera-stream windows. IWS is reserved for data-efficiency evaluation rather than training.

  • Training corpus: The corpus combines seven sources covering robot and human interactions, including single-arm, bimanual, humanoid, egocentric-hand, and deformable-object demonstrations.DROID is the primary robot source; other sources include ABC-130k, MolmoAct2, EgoDex, Deform360, XVLA-Soft-Fold, and H1-Fold-Clothes1.
  • Evaluation split: The Interactive World Simulator tasks are excluded from the training corpus and used only for data-efficiency evaluations.This separation supports held-out assessment of task-specific adaptation.
  • Training corpus: The dataset selection emphasizes deformable-object interactions involving cloth, cable, rope, bags, and paper.The main corpus is filtered toward deformable-object interaction data because this is the paper's focus.
  • Preprocessing: Episodes are standardized into 480p, 16-fps, non-overlapping 81-frame windows of approximately five seconds, with tracks, captions, latents, and available annotations.The corpus uses a shared per-episode, per-camera layout for training.
  • Filtering: Source-specific filtering removes static windows, frozen-gripper windows, and DROID episodes with contentless language annotations.A separate 17-frame DROID wrist-camera stream is restricted to deformable-object episodes and remains a standalone viewpoint.

4. Experiments

The experiments test actionable flow across backbones, transfer settings, data-efficiency regimes, rollout speed, policy evaluation, and inverse control. Results show stronger motion fidelity, transfer after multi-embodiment mid-training, efficient generation, benchmark-aligned evaluation, and desired-flow-to-action conversion.

  • Implementation: Action flow is implemented across Cosmos 2.5 and two Wan2.2 variants using the same track sampling and motion-feature aggregation, with backbone-specific injection adapters.The evaluated Wan2.2 variants are I2V-A14B and TI2V-5B.
  • Actionable flow conditioning: Replacing Cosmos 2.5's native relative 6D action with actionable flow improves displayed PSNR, SSIM, gripper-flow EPE, FID, and FVD estimates across all five datasets.Object-flow EPE also improves on all four datasets where it is reported; VLM scores are mixed because they measure broader perceptual criteria.
  • Data efficiency: Multi-embodiment mid-training provides the strongest zero-shot transfer: at 0% task-specific data, Ours (MT) outperforms Ours (PT) on LPIPS, object-flow EPE, and FVD across all six IWS tasks.The IWS comparison uses matched nested subsets of task-specific training data.
  • Data efficiency: At 100% task-specific data, Ours (MT) has the lowest LPIPS and FVD on all six IWS tasks and the lowest flow EPE on four.Most gains arrive by 20%; beyond that point, per-task changes are at most 3.4% in LPIPS, 6.7% in flow EPE, and 6.8% in FVD.
  • Rollout efficiency: Distillation reduces network evaluations to 15, reaches 62.0 frames per second, and provides a 16.0× generation-only speedup.The reported speedup excludes guidance and VAE/pixel decoding, so it is not end-to-end.
  • Open-loop policy evaluation: RoboLab generated success rates correlate with reference rates at Pearson's r=0.96 and Spearman's ρ=0.93, with 5.7 percentage points mean absolute error and matching policy rankings.Real-world generated rollouts qualitatively preserve different task outcomes, providing preliminary evidence beyond RoboLab.
  • Inverse control: Desired object flow from a held-out human demonstration conditions Hydra-0 to predict compatible robot motion, which a trained action-output head converts into executable actions.The input excludes embodiment flow, so the rollout predicts the robot motion needed to drive the desired object flow.

5. Related Work

Prior work uses action-conditioned video simulators for policy evaluation, improvement, and interaction, but Hydra-0 focuses on a shared visual action interface for cross-embodiment training and open-loop evaluation.

  • Neural Simulators and Robotic World Models: Latent and video-generation world models support planning or visually inspectable prediction of future observations.Latent models learn compact dynamics for imagined-action planning, while video-generation models synthesize future observations for physical-world simulation.
  • Action-Conditioned Video Simulators: Action-conditioned video simulators have been applied to policy ranking, policy improvement, training, and interactive rollouts.The cited systems study policy ranking, imagined trajectories for improvement or training, and nominal or out-of-distribution policy behavior.
  • Hydra-0: Hydra-0 maps executable commands to visible image-plane motion, using the same video-prediction interface for cross-embodiment training and open-loop policy evaluation.Its focus is a shared visual action interface rather than a backbone-specific conditioning mechanism.
  • Motion and Geometric Interfaces: Motion Prompting and related methods condition video generation on sparse or dense trajectories, masks, skeletons, pointmaps, or optical flow.These approaches include trajectory transport, semantic masks, embodiment masks, two-dimensional kinematic skeletons, and geometric or flow interfaces.

6. Limitations

The evaluation remains limited in grasp precision, sensing, camera settings, and policy-evaluation mode, leaving several deployment scenarios for future work.

  • Limitations: The world action model can exhibit centimeter-scale grasp imprecision, potentially related to limited depth awareness.Generated rollouts can also leave grasp and contact state ambiguous, including whether an object was successfully secured.
  • Limitations: Additional depth, tactile, and force modalities are proposed to reduce grasp and contact ambiguities.This is presented as a future direction rather than an evaluated solution.
  • Limitations: The wrist-camera experiment is only a qualitative DROID proof of concept, with broader camera motion, mobile manipulation, and systematic simulation evaluation left for future work.The stated scope boundary includes wrist-camera simulation and mobile-manipulation settings.
  • Limitations: The current evaluation is limited to open-loop policy evaluation; closed-loop evaluation remains future work.The paper does not report closed-loop evaluation in the supplied passage.

7. Conclusion

Hydra-0 presents kinematically grounded action flow as a visual interface for transferable world simulation across embodiments and video backbones. It also supports policy evaluation and inverse control, connecting visual prediction with robot action execution.

  • Conclusion: Kinematically grounded action flow provides a visual control interface for transferable neural world simulation across robot embodiments and video backbones.The conclusion frames action flow as the paper’s central interface.
  • Policy Evaluation: Pearson correlation r=0.96 links replayed and reference success rates across RoboLab policies.This result concerns the interface’s policy-evaluation capability.
  • Inverse Control: Desired object flow from a held-out human demonstration can generate compatible robot motion without embodiment-flow input.A trained action head maps latent motion features to executable robot actions without task-specific expert robot demonstrations.
  • Conclusion: Action flow connects visual prediction, open-loop policy evaluation, and robot control.The conclusion presents these capabilities as connected through a shared control interface.

8. Appendix

The appendix adds DROID qualitative results and details how metadata-rich robot data are converted into projected tracks and sampled into shared motion tensors.

  • Flow Construction: Metadata-rich data produce tracks by sampling visible robot-surface points, propagating them with link transforms, and projecting them into the camera plane.At deployment, transforms come from candidate commands executed through the controller and Isaac Lab physics simulation; recorded data use observed robot states.
  • Training-Time Track Sampling: Four sampling modes—none, embodiment, object, and all—select which trajectories populate the common motion tensor.Unavailable modes are removed and remaining probabilities renormalized, with a fallback to none when no valid mode remains.
  • Training-Time Track Sampling: The canonical I2V training mixture samples none, embodiment, object, and all modes with probabilities 0.05, 0.40, 0.40, and 0.15, respectively.These probabilities are stated in the listed mode order.
  • Additional Qualitative Evaluation: Figure 11 shows a matched DROID validation sample at six synchronized timestamps, pairing full-size Action Flow conditioning with clean generated rollouts.The Ours row omits the flow overlay while showing the corresponding generated rollout.
  • Training-Time Track Sampling: Embodiment and object modes sample 1–128 tracks, while all mode samples 256–1024 tracks when the required pools are available.The all-mode pool combines embodiment, object, and unassigned scene tracks before one subset is drawn.

8.4. Motion-Feature Aggregation

The method aggregates tracked motion features at destination cells using Gaussian locality, retaining only the strongest contributions and deriving a bounded presence gate.

  • Tracks are pooled into latent-frame windows, with visibility determined by any visible sample and destinations computed as means of visible positions.Source features are bilinearly sampled from the initial-frame latent feature map.
  • At each destination cell, the method retains the K = 2 largest raw Gaussian weights.The Gaussian locality is controlled by β = 220.
  • The propagated feature is formed without softmax or weight-sum normalization.
  • The raw-weight sum is clamped to [0, 1] to produce the presence gate.

8.5. Wan2.2 Implementation

Wan2.2 I2V injects propagated motion features through its existing visual-conditioning pathway while preserving the native initial-frame condition.

  • Wan2.2 I2V uses 16 VAE-feature channels and four mask channels for visual conditioning.
  • The first latent frame retains the native I2V condition, while later frames combine propagated feature mass with pristine features weighted by one minus the presence gate.
  • The presence gate is broadcast across the four mask channels.

8.6. Wan2.2 TI2V-5B Lightweight Variant

The lightweight Wan2.2 TI2V-5B variant supplies motion through a separate side input and uses direct tuning for few-step autoregressive generation.

  • 8.6. Wan2.2 TI2V-5B Lightweight Variant: Wan2.2 TI2V-5B uses a 49-channel motion side input containing 48 propagated feature channels and one presence-gate channel.The side input is concatenated with noisy latents and the DiT input projection is widened.
  • 8.6. Wan2.2 TI2V-5B Lightweight Variant: The motion side input remains constant during denoising and is neither noised nor used to re-anchor the latent state.
  • 8.6. Wan2.2 TI2V-5B Lightweight Variant: Direct tuning uses cleancontext teacher forcing and a block-causal mask while retaining the flow-matching objective.The stage does not require ODE initialization or intermediate distillation.

8.8. Few-Step Distillation Details

Few-step distillation trains a LoRA student against frozen real-score and online fake-score models, then evaluates generated clips with a multi-criterion judge.

  • 8.8. Few-Step Distillation Details: The student generates seven-latent-frame autoregressive chunks with four denoising steps before applying the DMD score-difference objective.
  • 8.8. Few-Step Distillation Details: An online fake-score critic is trained on student samples, and the real–fake score difference updates the LoRA student.The procedure performs five critic updates per generator update.
  • 8.8. Few-Step Distillation Details: Generated 81-frame clips are judged from 16 uniformly spaced frames using four physical and temporal criteria.
  • 8.8. Few-Step Distillation Details: The four ratings cover physical plausibility, temporal consistency, object permanence, and motion realism on a 1–5 Likert scale.Ratings are averaged within clips and then across clips.
Loading 2608.18077v1…