Source-linked AI summary
DUET-DINO: Simultaneous Cross-View World Modeling for Latent Planning in Robot Manipulation
Nisarga Nilavadi, Ralf Römer, Moritz Reuss, Michael Krawez, Tobias Jülg, Angela P. Schoellig, Rudolf Lioutikov, Wolfram Burgard
TL;DR
Full 7-DoF manipulation remains difficult because latent world models poorly predict fine-grained spatial and rotational changes. DUET-DINO jointly conditions side- and wrist-view predictors in latent space and plans across both views, outperforming single- and independent-dual-view baselines across diverse manipulation tasks. The approach reaches 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks.
Problem
Existing action-conditioned latent world models struggle to reliably predict fine-grained translational, rotational, and gripper-state changes needed for full 7-DoF manipulation.
Method
DUET-DINO jointly predicts side- and wrist-view future latents using cross-view conditioning and plans full 7-DoF actions with a normalized dual-view goal cost.
Results
DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks.
Takeaways & Limitations
Complementary global-scene and gripper-centric observations support latent planning across the full 7-DoF action space.
Takeaways & Limitations
CEM evaluation of many candidate actions through the world model is computationally expensive and limits real-time control.
Abstract
from arXiv · showhide
Action-conditioned latent world models predict future visual representations, enabling zero-shot goal-conditioned robot planning and control. However, their predictions for fine-grained spatial and rotational actions are unreliable for full 7-DoF end-effector control. To address this gap, we introduce DUET-DINO, a simultaneous cross-view latent world model that jointly learns action-conditioned predictions from static side- and wrist-camera observations through cross-view conditioning. By exploiting complementary global scene and gripper-centric information, DUET-DINO enables latent planning over the full 7-DoF action space. Across spatially diverse reach, orientation-intensive angled-reach, and multi-goal grasp-and-lift tasks, DUET-DINO consistently outperforms single-view and independent dual-view baselines, achieving 92% success on reach, 72.5% on angled-reach, and 60.0% on lift tasks. DUET-DINO is trained from scratch on DROID and RoboArena datasets and generalizes robustly under visual distribution shifts. We further show that while V-JEPA 2 wrist-view predictions underestimate visual dynamics induced by fine-grained actions, DINOv3 predictions better capture action-conditioned scene changes, leading to stronger downstream planning. The code and model checkpoints will be open-sourced. Project page: https://utn-air.github.io/DUET-DINO
I. INTRODUCTION
Latent world models remain challenged by full 7-DoF manipulation because fine-grained translation, rotation, and gripper-state changes are difficult to predict reliably. DUET-DINO addresses this with jointly conditioned side- and wrist-view modeling for full-action-space planning.
- Motivation and contribution: World models learn representations and dynamics from large-scale data, but applying latent world models to real-world manipulation remains challenging.They can predict future robot-manipulation states as latent representations, pixels, or rewards.
- Motivation and contribution: Full 7-DoF planning requires reliable prediction of fine-grained translational, rotational, and gripper-state changes for precise manipulation.These demands arise in spatially diverse reaching, angled grasping, orientation-intensive pick-and-place, and insertion.
- Motivation and contribution: DUET-DINO jointly models static side-view workspace context and wrist-view gripper-centric geometry through cross-view latent conditioning.Each view predictor exploits complementary information from the other while remaining specialized to its own camera.
- Motivation and contribution: DUET-DINO consistently outperforms static single-view and naive dual-view baselines on spatially diverse, orientation-intensive, and multi-goal angled-lift tasks.The comparison spans hardware and simulated benchmarks.
II. RELATED WORK
Prior work includes generative video and latent world models, pixel- and latent-space planning, and CEM-based action optimization. DUET-DINO builds on these directions while targeting the unresolved challenge of fine-grained full 7-DoF control with synchronized side and wrist cameras.
- Generative video world models: Generative video methods plan through synthesized future videos, neural trajectories, policy scoring, or multi-view video diffusion.Examples include Cosmos, Wan, DreamGen, WorldGym, and Ctrl-World.
- Latent world models: Latent world models predict learned representations instead of reconstructing pixels, with JEPA- and DINO-based approaches supporting action-conditioned prediction and planning.V-JEPA 2 adds an action-conditioned predictor, while DINO-WM and DINOv3 use dense visual features for latent prediction.
- Latent world models: Existing latent world models largely rely on a single external camera and struggle with fine-grained 7-DoF control.DUET-DINO explicitly targets this open challenge with dual-view modeling.
- Planning in pixel vs. latent space: Pixel-space planning couples decisions to generated-video fidelity, whereas latent-space methods optimize against learned representations for manipulation planning.The cited pixel-space approaches use generated images or videos as planning intermediates.
- Action proposals and optimization: CEM iteratively refines Gaussian action proposals and is commonly used for highly non-convex planning objectives.Other approaches learn structured proposals or combine sampling with gradient steps.
- Problem formulation: DUET-DINO uses synchronized static side-camera and dynamic wrist-camera streams for tabletop manipulation.Its formulation aims to predict future dual-view latents for zero-shot goal-conditioned planning.
B. DUET-DINO: Simultaneous Cross-View World Modeling
DUET-DINO encodes synchronized side and wrist observations with a frozen DINOv3 encoder, exchanges information through cross-attention, and predicts each view’s future latent with a view-specific action-conditioned head. Training jointly combines teacher-forced and autoregressive prediction losses.
- Training objective: DUET-DINO trains cross-attention blocks and predictor heads jointly from scratch while keeping the visual encoder frozen.Training uses large-scale robotic manipulation data.
- Cross-view latent modeling: A frozen DINOv3 encoder independently maps side- and wrist-view observations to patch-wise latent representations.The latent shape is determined by the number of image patches and encoder latent dimension.
- Cross-view latent modeling: Cross-attention lets each view’s latent query complementary information from the other view before prediction.Target-view latents serve as queries, while the other-view latents provide keys and values.
- View-specific prediction: Separate predictor heads use cross-view-conditioned latents, robot state, and action to predict the next latent for their respective views.The heads preserve view-specific dynamics while using complementary information from the other camera.
- Training objective: The dual-view objective combines teacher-forcing one-step prediction with an autoregressive rollout loss.The autoregressive component is computed over a rollout horizon of K = 2.
C. Visual Planning
DUET-DINO performs zero-shot goal-conditioned planning in a joint dual-view latent space. It rolls out candidate full end-effector action sequences with CEM and evaluates them using a normalized cost comparing predicted and goal latents across both views.
- Goal-conditioned latent planning: DUET-DINO plans full end-effector motions by generating action sequences that drive predicted dual-view latents toward a goal observation.The planning problem is formulated in the learned side- and wrist-view latent space over a fixed receding horizon.
- CEM action optimization: CEM samples Gaussian-distributed action sequences, rolls them out through DUET-DINO, and evaluates their deviations from the goal latents.The process repeats across CEM iterations using candidate imagined latent trajectories.
- Dual-view planning cost: The dual-view planning cost sums per-view costs and normalizes them by the corresponding zero-action rollout cost.This produces the normalized dual-view objective used to compare candidate action sequences.
- CEM action optimization: A weighted top-k CEM update gives greater influence to lower-cost action sequences when updating the proposal distribution.Candidate weights are computed with a softmax over negative costs.
IV. EXPERIMENTS
The experiments section covers dataset processing, world-model training, experimental setup, evaluation protocol, and results for DUET-DINO and baseline methods.
- The experiments cover dataset processing, world-model training, experimental setup, evaluation protocol, and results for DUET-DINO and baseline methods.
A. Implementation and Training Details
DUET-DINO is trained and evaluated on DROID and RoboArena using varied visual encoders, camera views, and predictor architectures.
- Training and evaluation use DROID and RoboArena robot-manipulation datasets.
- Baselines vary by visual encoder, camera view, and architecture, including single-view predictors and the proposed dual-view predictor.
B. Experimental Setup
The evaluation tests full 7-DoF latent planning across reach, angled-reach, and grasp-and-lift tasks using single-view, independent dual-view, and DUET dual-view predictors.
- The tasks require full 7-DoF end-effector control and include reach, angled reach, angled grasp, and angled lift-to-home.
- The comparison includes single-view predictors, independently trained dual-view predictors, and jointly trained DUET predictors with cross-view attention.
- Evaluation uses the RoboLab simulator with a Franka arm, Robotiq gripper, wrist camera, and static side-view camera.
C. Reach Task
Reach evaluation measures whether the end effector reaches ten tabletop objects within a 5 cm position tolerance, with DUET-DINO benefiting from coordinated dual-view planning but requiring more computation.
- Reach success requires the end-effector position to be within 5 cm of the goal within 100 steps.Final position error is measured at the final planning step, including residual error after successful completion.
- 92% success is achieved by DUET-DINO on reach, outperforming single-view and independent dual-view alternatives across ten target objects.Table I aggregates success rate and final position error over 100 evaluation runs per model.
- DUET-DINO succeeds on corner-positioned Coffee Pot targets requiring coordinated translation and rotation while initially outside the wrist camera’s view.
- Dual-view planning is roughly 2× slower than single-view planning, while DUET-DINO requires 15–17 seconds per planning step in this setup.
D. Angled Reach Task
Angled-reach experiments test full 7-DoF planning under orientation-intensive poses, visual distribution shifts, and real-robot constraints. DUET-DINO improves fine-grained pose planning over single-view and independent dual-view alternatives, while hardware performance is limited by reduced planning budgets.
- Fixed Background: V-JEPA 2 and trained side-view predictors struggle with fine-grained rotation, while DINOv3 wrist-view prediction supports stronger orientation planning.The trained side-view models fail on the most rotation-intensive cases, and independent dual-view V-JEPA variants do not improve performance.
- Fixed Background: 72.5% success on angled-reach tasks shows DUET-DINO improves full-pose planning over the evaluated alternatives.The model reduces both final position and angular errors, whereas the independent DINOv3 dual-view model improves position error but worsens angular error.
- Diverse Backgrounds and Distractor Objects: Under visual distribution shifts, evaluation spans 100 runs per model across five backgrounds, one-to-five-object scenes, and four angled-reach tasks.The setup adds visually similar distractor objects while varying tabletop scenes and target objects.
- Hardware Runs: DUET-DINO achieves the highest success rate on challenging real-robot angled-reach poses, outperforming the independent dual-view predictor.The hardware evaluation uses three tasks with target poses near table corners and both clockwise and counterclockwise orientations.
- Hardware Runs: Hardware success is lower than simulation because planning uses one CEM iteration with 500 candidate actions and 15 planning steps.Unsafe actions are rejected, and wrist-view prediction produces unsafe actions more frequently.
E. Angled Grasp and Angled Lift-to-Home
Sequential angled-reach, grasp, and lift-to-home tasks test multi-goal planning, while ablations and camera perturbations probe the model’s mechanisms and robustness. DUET-DINO completes most lift subtasks, benefits from cross-view attention, and remains more robust to camera shifts than single-view baselines.
- Angled Grasp and Angled Lift-to-Home: Most DUET-DINO runs complete angled reaching, grasping, and lifting, whereas the DINOv3 wrist-view predictor often loses task context during lift-to-home.The full lift task requires maintaining the grasp while returning the object to the robot’s home pose.
- Cross-view Cross Attention: Removing cross-view attention lowers reach success to 81% and angled-reach success to 42.5%, confirming its importance for fine-grained dual-view planning.The ablation also yields 10.7±15.4 cm reach FPE and 4.3±3.1 cm FPE with 55.5±52.8° FAE on angled reach.
- Camera Perturbations: Under camera-pose shifts, DUET-DINO and independent DINOv3 dual-view models achieve 66.7% success, versus 0% for both single-view side predictors.The perturbations include ±20 cm in x/y, ±10 cm in z, and ±11.5° in roll, pitch, and yaw.
- Representation Analysis: Patch-wise latent-distance analysis compares DINOv3 and V-JEPA 2 feature maps using lower ℓ1 distance as greater similarity to a reference patch.The maps contain 196 DINOv3 patches at 14 × 14 and 256 V-JEPA 2 patches at 16 × 16.
- Training Data: DROID-only training gives DUET-DINO 78% reach success and 72.5% angled-reach success, improving spatial planning but not orientation planning.This comparison uses 94.5k training steps and isolates the effect of additional training data.
G. Embeddings Analysis
The embedding analysis compares DINOv3 and V-JEPA 2 for representing action-induced visual changes, especially in wrist-camera predictions. DINOv3 preserves temporal and spatial correspondence patterns more faithfully, whereas V-JEPA 2 wrist-view predictions underestimate fine-grained dynamics.
- Wrist-view correspondences: DINOv3 preserves coherent wrist-view correspondences over semantically related object regions and closely matches the encoded next-frame pattern.The correspondence analysis compares target-object patches across current, ground-truth next, and predicted next feature maps using ℓ1 distance.
- Wrist-view correspondences: V-JEPA 2 produces noisier wrist-view patch correspondences, indicating weaker representation of fine-grained action-conditioned dynamics in that view.The comparison focuses on whether predicted low-distance patches preserve the target-object correspondence pattern.
- Implications for DUET-DINO: DUET-DINO combines cross-view latent conditioning with DINO representations to model translational, rotational, and gripper-induced scene changes.Its planning procedure uses computationally expensive CEM evaluation over many candidate actions, limiting real-time control.