PaperlayerView exact revision on arXiv

Source-linked AI summary

DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation

Nisarga Nilavadi, Ralf Römer, Moritz Reuss, Michael Krawez, Tobias Jülg, Angela P. Schoellig, Rudolf Lioutikov, Wolfram Burgard

arXiv:2609.10506v1cs.ROcs.CV

TL;DR

Full 7-DoF manipulation remains difficult because latent world models poorly predict fine-grained spatial and rotational changes. DUET-DINO jointly conditions side- and wrist-view predictors in latent space and plans across both views, outperforming single- and independent-dual-view baselines across diverse manipulation tasks. The approach reaches 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks.

  • Problem

    Existing action-conditioned latent world models struggle to reliably predict fine-grained translational, rotational, and gripper-state changes needed for full 7-DoF manipulation.

    Source: §I · p. 1

  • Method

    DUET-DINO jointly predicts side- and wrist-view future latents using cross-view conditioning and plans full 7-DoF actions with a normalized dual-view goal cost.

    Source: §I · pp. 1–2, B. DUET-DINO: Simultaneous Cross-View… · p. 3

  • Results

    DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Complementary global-scene and gripper-centric observations support latent planning across the full 7-DoF action space.

    Source: §I · pp. 1–2

    CEM evaluation of many candidate actions through the world model is computationally expensive and limits real-time control.

    Source: §V · p. 8

Abstract

from arXiv · show

Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, we introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning. By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. DUET-DINO is trained from scratch on DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. We further show that while V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning. The code and model checkpoints will be open-sourced. Project page: https://utn-air.github.io/DUET-DINO

I. INTRODUCTION

Latent world models remain challenged by full 7-DoF manipulation because fine-grained translation, rotation, and gripper-state changes are difficult to predict reliably. DUET-DINO addresses this with jointly conditioned side- and wrist-view modeling for full-action-space planning.

Motivation and contribution

  • World models learn representations and dynamics from large-scale data, but applying latent world models to real-world manipulation remains challenging.They can predict future robot-manipulation states as latent representations, pixels, or rewards.Source: §I · p. 1
  • Full 7-DoF planning requires reliable prediction of fine-grained translational, rotational, and gripper-state changes for precise manipulation.These demands arise in spatially diverse reaching, angled grasping, orientation-intensive pick-and-place, and insertion.Source: §I · p. 1
  • DUET-DINO jointly models static side-view workspace context and wrist-view gripper-centric geometry through cross-view latent conditioning.Each view predictor exploits complementary information from the other while remaining specialized to its own camera.Source: §I · p. 1
  • DUET-DINO consistently outperforms static single-view and naive dual-view baselines on spatially diverse, orientation-intensive, and multi-goal angled-lift tasks.The comparison spans hardware and simulated benchmarks.Source: §I · p. 1

II. RELATED WORK

Prior work includes generative video and latent world models, pixel- and latent-space planning, and CEM-based action optimization. DUET-DINO builds on these directions while targeting the unresolved challenge of fine-grained full 7-DoF control with synchronized side and wrist cameras.

Generative video world models

  • Generative video methods plan through synthesized future videos, neural trajectories, policy scoring, or multi-view video diffusion.Examples include Cosmos, Wan, DreamGen, WorldGym, and Ctrl-World.Source: A. World Models in Robotics · p. 2

Latent world models

  • Latent world models predict learned representations instead of reconstructing pixels, with JEPA- and DINO-based approaches supporting action-conditioned prediction and planning.V-JEPA 2 adds an action-conditioned predictor, while DINO-WM and DINOv3 use dense visual features for latent prediction.Source: A. World Models in Robotics · p. 2
  • Existing latent world models largely rely on a single external camera and struggle with fine-grained 7-DoF control.DUET-DINO explicitly targets this open challenge with dual-view modeling.Source: A. World Models in Robotics · p. 2

Planning in pixel vs. latent space

  • Pixel-space planning couples decisions to generated-video fidelity, whereas latent-space methods optimize against learned representations for manipulation planning.The cited pixel-space approaches use generated images or videos as planning intermediates.Source: B. Visual Planning with World Models · p. 2

Action proposals and optimization

  • CEM iteratively refines Gaussian action proposals and is commonly used for highly non-convex planning objectives.Other approaches learn structured proposals or combine sampling with gradient steps.Source: B. Visual Planning with World Models · p. 2

Problem formulation

  • DUET-DINO uses synchronized static side-camera and dynamic wrist-camera streams for tabletop manipulation.Its formulation aims to predict future dual-view latents for zero-shot goal-conditioned planning.Source: A. Problem Formulation · pp. 2–3

B. DUET-DINO: Simultaneous Cross-View World Modeling

DUET-DINO encodes synchronized side and wrist observations with a frozen DINOv3 encoder, exchanges information through cross-attention, and predicts each view’s future latent with a view-specific action-conditioned head. Training jointly combines teacher-forced and autoregressive prediction losses.

Training objective

Cross-view latent modeling

  • A frozen DINOv3 encoder independently maps side- and wrist-view observations to patch-wise latent representations.The latent shape is determined by the number of image patches and encoder latent dimension.Source: B. DUET-DINO: Simultaneous Cross-View… · p. 3
  • Cross-attention lets each view’s latent query complementary information from the other view before prediction.Target-view latents serve as queries, while the other-view latents provide keys and values.Source: B. DUET-DINO: Simultaneous Cross-View… · p. 3

View-specific prediction

  • Separate predictor heads use cross-view-conditioned latents, robot state, and action to predict the next latent for their respective views.The heads preserve view-specific dynamics while using complementary information from the other camera.Source: B. DUET-DINO: Simultaneous Cross-View… · p. 3

Training objective

C. Visual Planning

DUET-DINO performs zero-shot goal-conditioned planning in a joint dual-view latent space. It rolls out candidate full end-effector action sequences with CEM and evaluates them using a normalized cost comparing predicted and goal latents across both views.

Goal-conditioned latent planning

  • DUET-DINO plans full end-effector motions by generating action sequences that drive predicted dual-view latents toward a goal observation.The planning problem is formulated in the learned side- and wrist-view latent space over a fixed receding horizon.Source: C. Visual Planning · pp. 3–4

CEM action optimization

  • CEM samples Gaussian-distributed action sequences, rolls them out through DUET-DINO, and evaluates their deviations from the goal latents.The process repeats across CEM iterations using candidate imagined latent trajectories.Source: C. Visual Planning · p. 4

Dual-view planning cost

  • The dual-view planning cost sums per-view costs and normalizes them by the corresponding zero-action rollout cost.This produces the normalized dual-view objective used to compare candidate action sequences.Source: C. Visual Planning · p. 4

CEM action optimization

  • A weighted top-k CEM update gives greater influence to lower-cost action sequences when updating the proposal distribution.Candidate weights are computed with a softmax over negative costs.Source: C. Visual Planning · p. 4

IV. EXPERIMENTS

The experiments section covers dataset processing, world-model training, experimental setup, evaluation protocol, and results for DUET-DINO and baseline methods.

  • The experiments cover dataset processing, world-model training, experimental setup, evaluation protocol, and results for DUET-DINO and baseline methods.Source: §IV · p. 4

A. Implementation and Training Details

DUET-DINO is trained and evaluated on DROID and RoboArena using varied visual encoders, camera views, and predictor architectures.

B. Experimental Setup

The evaluation tests full 7-DoF latent planning across reach, angled-reach, and grasp-and-lift tasks using single-view, independent dual-view, and DUET dual-view predictors.

  • The tasks require full 7-DoF end-effector control and include reach, angled reach, angled grasp, and angled lift-to-home.Source: B. Experimental Setup · p. 5
  • The comparison includes single-view predictors, independently trained dual-view predictors, and jointly trained DUET predictors with cross-view attention.Source: B. Experimental Setup · p. 5
  • Evaluation uses the RoboLab simulator with a Franka arm, Robotiq gripper, wrist camera, and static side-view camera.Source: B. Experimental Setup · p. 5

C. Reach Task

Reach evaluation measures whether the end effector reaches ten tabletop objects within a 5 cm position tolerance, with DUET-DINO benefiting from coordinated dual-view planning but requiring more computation.

  • Reach success requires the end-effector position to be within 5 cm of the goal within 100 steps.Final position error is measured at the final planning step, including residual error after successful completion.Source: C. Reach Task · p. 5
  • 92% success is achieved by DUET-DINO on reach, outperforming single-view and independent dual-view alternatives across ten target objects.Table I aggregates success rate and final position error over 100 evaluation runs per model.Source: C. Reach Task · p. 5, Table I · p. 5
  • DUET-DINO succeeds on corner-positioned Coffee Pot targets requiring coordinated translation and rotation while initially outside the wrist camera’s view.Source: C. Reach Task · p. 6
  • Dual-view planning is roughly 2× slower than single-view planning, while DUET-DINO requires 15–17 seconds per planning step in this setup.Source: C. Reach Task · p. 6

D. Angled Reach Task

Angled-reach experiments test full 7-DoF planning under orientation-intensive poses, visual distribution shifts, and real-robot constraints. DUET-DINO improves fine-grained pose planning over single-view and independent dual-view alternatives, while hardware performance is limited by reduced planning budgets.

Fixed Background

  • V-JEPA 2 and trained side-view predictors struggle with fine-grained rotation, while DINOv3 wrist-view prediction supports stronger orientation planning.The trained side-view models fail on the most rotation-intensive cases, and independent dual-view V-JEPA variants do not improve performance.Source: D. Angled Reach Task · p. 6
  • 72.5% success on angled-reach tasks shows DUET-DINO improves full-pose planning over the evaluated alternatives.The model reduces both final position and angular errors, whereas the independent DINOv3 dual-view model improves position error but worsens angular error.Source: D. Angled Reach Task · p. 6

Diverse Backgrounds and Distractor Objects

  • Under visual distribution shifts, evaluation spans 100 runs per model across five backgrounds, one-to-five-object scenes, and four angled-reach tasks.The setup adds visually similar distractor objects while varying tabletop scenes and target objects.Source: D. Angled Reach Task · p. 6, Table III · p. 6

Hardware Runs

  • DUET-DINO achieves the highest success rate on challenging real-robot angled-reach poses, outperforming the independent dual-view predictor.The hardware evaluation uses three tasks with target poses near table corners and both clockwise and counterclockwise orientations.Source: D. Angled Reach Task · p. 6, Table IV · p. 6
  • Hardware success is lower than simulation because planning uses one CEM iteration with 500 candidate actions and 15 planning steps.Unsafe actions are rejected, and wrist-view prediction produces unsafe actions more frequently.Source: D. Angled Reach Task · pp. 6–7

E. Angled Grasp and Angled Lift-to-Home

Sequential angled-reach, grasp, and lift-to-home tasks test multi-goal planning, while ablations and camera perturbations probe the model’s mechanisms and robustness. DUET-DINO completes most lift subtasks, benefits from cross-view attention, and remains more robust to camera shifts than single-view baselines.

Angled Grasp and Angled Lift-to-Home

  • Most DUET-DINO runs complete angled reaching, grasping, and lifting, whereas the DINOv3 wrist-view predictor often loses task context during lift-to-home.The full lift task requires maintaining the grasp while returning the object to the robot’s home pose.Source: E. Angled Grasp and Angled Lift-to-Home · p. 7

Cross-view Cross Attention

  • Removing cross-view attention lowers reach success to 81% and angled-reach success to 42.5%, confirming its importance for fine-grained dual-view planning.The ablation also yields 10.7±15.4 cm reach FPE and 4.3±3.1 cm FPE with 55.5±52.8° FAE on angled reach.Source: F. Ablation · p. 7

Camera Perturbations

  • Under camera-pose shifts, DUET-DINO and independent DINOv3 dual-view models achieve 66.7% success, versus 0% for both single-view side predictors.The perturbations include ±20 cm in x/y, ±10 cm in z, and ±11.5° in roll, pitch, and yaw.Source: F. Ablation · p. 7

Representation Analysis

  • Patch-wise latent-distance analysis compares DINOv3 and V-JEPA 2 feature maps using lower ℓ1 distance as greater similarity to a reference patch.The maps contain 196 DINOv3 patches at 14 × 14 and 256 V-JEPA 2 patches at 16 × 16.Source: Figure 6 · p. 7, Figure 5 · p. 7

Training Data

  • DROID-only training gives DUET-DINO 78% reach success and 72.5% angled-reach success, improving spatial planning but not orientation planning.This comparison uses 94.5k training steps and isolates the effect of additional training data.Source: F. Ablation · p. 7

G. Embeddings Analysis

The embedding analysis compares DINOv3 and V-JEPA 2 for representing action-induced visual changes, especially in wrist-camera predictions. DINOv3 preserves temporal and spatial correspondence patterns more faithfully, whereas V-JEPA 2 wrist-view predictions underestimate fine-grained dynamics.

Wrist-view correspondences

  • DINOv3 preserves coherent wrist-view correspondences over semantically related object regions and closely matches the encoded next-frame pattern.The correspondence analysis compares target-object patches across current, ground-truth next, and predicted next feature maps using ℓ1 distance.Source: G. Embeddings Analysis · p. 8
  • V-JEPA 2 produces noisier wrist-view patch correspondences, indicating weaker representation of fine-grained action-conditioned dynamics in that view.The comparison focuses on whether predicted low-distance patches preserve the target-object correspondence pattern.Source: G. Embeddings Analysis · p. 8

Implications for DUET-DINO

  • DUET-DINO combines cross-view latent conditioning with DINO representations to model translational, rotational, and gripper-induced scene changes.Its planning procedure uses computationally expensive CEM evaluation over many candidate actions, limiting real-time control.Source: §V · p. 8

Related papers

Loading 2609.10506v1…