PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

Junyan Li, Ruizhi Li, Yu Liu, Xiangshuo Liu, Mingchao Sun, Hongyu Pan, Mu Xu, Lue Fan, Zhaoxiang Zhang

arXiv:2610.12468v1cs.ROcs.CV

TL;DR

Existing robot datasets have calibration gaps and limited coverage of unsuccessful interactions, motivating more reliable action-conditioned world models. DREAMTRUE aligns rendered action conditions with recorded videos and uses counterfactual post-training guided by an embodied video reward model. It ranks first in the AgiBot World Challenge 2026 world-model track and reports improved action following and interaction plausibility on AgiBot.

  • Problem

    Existing robot datasets can have inaccurate camera calibration and limited coverage of unsuccessful interactions, leaving challenges for training action-faithful, physically plausible world models.

    Source: §1 · pp. 1–2

  • Method

    DREAMTRUE calibrates rendered action conditions against recorded videos and post-trains on counterfactual actions using rewards learned from human annotations of video defects.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    DREAMTRUE ranks first in the AgiBot World Challenge 2026 world-model track; its AgiBot action-following nDTW is 0.8772, and human-assessed interaction defects fall from 48.12% to 6.25%.

    Source: §1 · p. 3, §4.2 · p. 7

  • Takeaways & Limitations

    The resulting predictions enable more accurate estimation of policy outcomes, supporting world-model evaluation of robot actions before execution.

    Source: §5 · p. 10

    Because the action representation, generator, and VLM reward model operate in image space, occlusion or limited views may permit visually plausible but physically incorrect interactions.

    Source: §5 · p. 10

Abstract

from arXiv · show

We present DreamTrue, a multi-view, cross-embodiment robot world model for action-faithful and physically plausible video prediction. Training such a model on existing robot datasets faces two obstacles: imprecise calibration can impair action following, while limited coverage of unsuccessful interactions can bias predictions toward successful outcomes. To improve action following across embodiments, we render action trajectories into image-space conditions and introduce offline geometric calibration to align these conditions with the target videos. To broaden interaction coverage, we introduce counterfactual post-training, modifying recorded action trajectories and generating future videos under a wider range of actions and contact configurations. To provide feedback on these predictions without paired ground-truth futures, we construct a human-annotated video dataset covering robot, object, and interaction defects and use it to train an embodied video reward model. Its scores guide reinforcement-learning post-training toward more physically plausible interaction outcomes. On AgiBot, DreamTrue attains state-of-the-art action following, while reducing the human-assessed interaction defect rate from from 48.12% to 6.25%. Notably, our model ranks first in the world model track of the AgiBot World Challenge 2026. The project page can be found at https://brave-eai.github.io/DreamTrue.

1 Introduction

Existing robot data can misalign rendered action conditions with videos and provide limited coverage of unsuccessful interactions. DREAMTRUE combines offline geometric calibration with counterfactual, reward-guided post-training; on AgiBot, it achieves an nDTW score of 0.8772 and reduces the human-assessed interaction defect rate from 48.12% to 6.25%.

1 Introduction

  • Existing robot datasets can lack accurate camera calibration and broad coverage of missed grasps, slipping objects, and unsuccessful contact, contributing to implausible predictions under failure-inducing actions.A model may predict an object rising after a missed grasp, potentially overestimating action success and misleading action selection or policy evaluation.Source: §1 · pp. 1–2
  • DREAMTRUE renders robot actions for cross-embodiment prediction and uses counterfactual trajectories plus a human-trained video reward model to improve interaction plausibility without paired future videos.The annotations cover robot embodiment, object consistency, and interaction plausibility; reward scores guide reinforcement-learning post-training.Source: §1 · p. 2
  • nDTW reaches 0.8772 among evaluated methods for action following, while reward-guided post-training reduces the human-assessed interaction defect rate from 48.12% to 6.25%.Both results are reported on the AgiBot benchmark.Source: §1 · p. 3
  • Offline geometric calibration is released for three datasets, covering 153,666 retained episodes and over 1,660 hours.The calibration aligns action conditions with recorded videos without dedicated calibration sequences.Source: §1 · p. 3
  • The video defect corpus covers 44.9K videos and includes 30.4K defect annotations for robot, object, and interaction defects.These human annotations train the embodied video reward model.Source: §1 · p. 3

2 Related Work

Related work develops action-conditioned robot world models, visual action representations, and evaluation or reward methods for generated embodied videos. DREAMTRUE emphasizes geometric alignment from existing recordings and rewards that assess robot–object interaction defects under counterfactual actions.

Action-conditioned robot world models

  • Action-conditioned world models use video-generation priors for robot motion and environmental responses, with prior work spanning multi-view prediction, policy learning, and failure-informed interaction coverage.Examples include Ctrl-World, Genie Envisioner, GE-Sim 2.0, DreamDojo, PlayWorld, and FACT (Li et al., 2025b; Yang et al., 2026b; Guo et al., 2025; Liao et al., 2026; Qiu et al., 2026; Gao et al., 2026; Yin et al., 2026; Peng et al., 2026).Source: §2 · p. 3

Geometric action representations and robot–camera calibration

  • Visual action representations depend on robot–camera geometry; prior calibration approaches may require dedicated sequences or depth, whereas DREAMTRUE recovers camera parameters and mounting offsets from RGB demonstrations.The passage describes BridgeV2W, Masked Visual Actions, EnerVerse-AC, classical hand–eye calibration, markerless methods, and PointWorld (Chen et al., 2026; Alzayer et al., 2026; Jiang et al., 2025a; Tsai & Lenz, 1989; Chen et al., 2023; Hong et al., 2024; Huang et al., 2026; Wen et al., 2025).Source: §2 · p. 3

Embodied video evaluation and reward-based post-training

3 Method

DreamTrue predicts multi-view future video from initial observations, task instructions, and actions, using calibrated image-space action conditions and reward-guided post-training to improve action following and interaction plausibility.

3.1 Problem Formulation and Overview

  • Given initial multi-view observations, a task instruction, and an action sequence, the framework predicts future videos through calibration and two training stages.The stages are supervised cross-embodiment training and reward-guided post-training for more plausible robot–object interactions.Source: §3.1 · p. 4

3.2 Offline Geometric Calibration

  • Camera parameters and applicable arm-mounting offsets are jointly refined from robot, background-over-time, and synchronized-view correspondences.Robot correspondences use reprojection error; background and synchronized-view correspondences impose ray coplanarity, and the optimized parameters produce robot renderings for action conditioning.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Stage I: Supervised Training with Cross-Embodiment Action Representation

  • Actions from different embodiments are converted into rendered image-space conditions, including robot appearance, masks, depth, gripper state, and camera-motion cues, for supervised multi-view future prediction.The conditions enter the video generator through a VACE branch, while paired future videos provide training supervision.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Stage II: Counterfactual Post-Training with Embodied Video Rewards

  • Counterfactual post-training perturbs recorded end-effector trajectories and uses human-labeled robot, object, and interaction defects to guide reward-based generator updates.The reward model predicts defects in three dimensions; defect-based rewards apply to all predictions, while recorded-action predictions also receive a PSNR reward against paired futures.Source: §3.4 · pp. 5–6

4 Experiments

Experiments assess action following and interaction quality, transfer across embodiments and environments, policy-outcome alignment, geometric calibration, and reward-model defect recognition.

4.1 Experimental Setup

  • The experiments use four jointly trained datasets totaling 2,232 hours of filtered multi-view trajectories, with action following and visual fidelity measured alongside human-assessed defect rates.The setup compares DreamDojo, GE-Sim 2.0, Genie Envisioner, EnerVerse-AC, and Stage-I versus Stage-II; defect labels cover embodiment, object consistency, and interaction plausibility.Source: §4.1 · pp. 6–7

4.2 Action Following and Interaction Plausibility

  • 0.8772 and 0.8831 nDTW under recorded and counterfactual actions, respectively; object defects fell from 31.88% to 3.12%, and interaction defects from 48.12% to 6.25%.DREAMTRUE also ranked first in the AgiBot World Challenge 2026 world model track for overall score, action following, and visual quality.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 7
  • With the initial scene fixed, predictions show grasped objects moving with grippers and ungrasped objects remaining on the table across varied grasp targets and single- or dual-arm actions.The qualitative comparison reports that the full model avoids unsupported scanner motion and object disappearance or duplication seen in comparison examples.Source: Figure 5 · p. 7, §4.2 · p. 8, Figure 6 · p. 8

4.3 Prediction across Embodiments and Environments

  • On DROID, PSNR rose from 22.00 to 22.95 and LPIPS fell from 0.162 to 0.073 versus Ctrl-World; qualitative transfer included unseen scenes and WidowX250 without fine-tuning.The held-out evaluation also covers RoboMIND 2.0 and RoboTwin 2.0, while the shared checkpoint produces predictions on four training datasets.Source: Table 3 · p. 8, §4.3 · pp. 8–9, Figure 7 · p. 8

4.4 Policy Outcome Evaluation

  • 12.92 to 7.08 percentage points: success-rate MAE fell across 24 task–policy–setting combinations, with Spearman ρ = 0.937 and aggregate bias of +0.42 pp.The model’s predictions cluster near the ideal diagonal in easy and hard settings, while baseline rollouts show upward-shifted estimates, especially at low success rates.Source: Figure 8 · p. 9, §4.4 · p. 9

4.5 Effect of Geometric Calibration

  • 1.28 dB and 1.30 dB PSNR gains on AgiBot and DROID followed training with refined calibration; inference-only refinement also improved all reported metrics on both datasets.The ablation measures synchronous position error as the pixel distance between predicted and ground-truth end-effector positions across temporally aligned frames.Source: Table 4 · p. 9, §4.5 · p. 10
  • .228 and .212 mean-IoU gains on AgiBot and DROID, respectively, with improvements on 97.1% and 91.6% of episodes.On DROID, RGB-only calibration outperformed PointWorld on 75.5% of 38,356 episodes; Figure 9 compares IoU against both dataset-provided calibration and PointWorld.Source: §4.5 · p. 9, Figure 9 · p. 10

4.6 Reward Model Evaluation

  • 86.33% average accuracy and 74.21% macro-F1 on 4,893 held-out clips support defect recognition across robot, object, and interaction dimensions.The human-annotated validation set evaluates the reward model’s agreement with judgments of those three defect types.Source: §4.6 · p. 10

5 Conclusion and Limitations

DREAMTRUE combines calibration and reward-guided post-training for action-faithful, physically plausible prediction, while its image-space design has a stated limitation under occlusion or limited views.

5 Conclusion and Limitations

  • The paper concludes that calibrated action conditions and counterfactual reward-guided post-training improve action following, prediction fidelity, and robot–object interaction plausibility.The resulting predictions support more accurate policy-outcome estimation before robot actions are executed.Source: §5 · p. 10
  • The authors release refined calibration results for 153,666 episodes across three datasets and defect annotations for 44.9K videos.Source: §5 · p. 10
  • Because the action representation, generator, and VLM reward model operate in image space, occlusion or limited views may permit visually plausible but physically incorrect interactions.Source: §5 · p. 10

A Model, Data, and Condition Construction

The supplied passage distinguishes the supervised Stage-I checkpoint from the counterfactually post-trained Stage-II checkpoint.

A Model, Data, and Condition Construction

  • “Ours (w/o RL)” denotes the supervised Stage-I checkpoint, while “Ours (full)” denotes the Stage-II checkpoint produced by counterfactual post-training from that same model.Source: A Model, Data, and Condition… · p. 15

A.1 Implementation and Training

Training uses a frozen base generator and reward model in Stage II, updating only the DiT and VACE LoRA branches; cross-embodiment results use no dataset-specific fine-tuning.

A.1 Implementation and Training

  • The generator predicts 100 future frames from three synchronized 240 × 320 views, conditioned on each view’s first RGB frame and the task instruction.DROID uses two external cameras and one gripper camera; the other three-view layouts use a primary camera and two wrist cameras.Source: §A.1 · p. 15
  • Stage I jointly trains the DiT and VACE LoRA branches with a 3D convolutional geometry encoder; Stage II freezes the base generator, geometry encoder, and reward model, updating only the DiT and VACE LoRA branches.The model has approximately 0.7B trainable parameters, with LoRA rank 128 and a Stage-II learning rate of 5×10−6.Source: §A.1 · p. 15
  • Stage I samples AgiBotWorld-Beta, DROID, RoboMIND 2.0, and RoboTwin 2.0; Stage II reuses recorded conditions and counterfactual edits, applying paired-future fidelity only when an observed future exists.The four datasets provide 2,232 hours of filtered trajectories, and cross-embodiment results use no dataset-specific fine-tuning.Source: §A.1 · p. 15

A.2.1 Implementation Details Offline Geometric Calibration

The appendix details how calibration is fitted and applied to real-world data, how robot conditions are rendered, and how feasible counterfactual trajectories are constructed.

A.2.1 Implementation Details Offline Geometric Calibration

  • Calibration groups episodes by collection metadata, fits a randomly selected episode, tests other episodes, and progressively calibrates additional episodes with poor alignment.AgiBot and RoboMIND use fixed initialization, while DROID uses 32 initializations; mounting offsets are optimized for RoboMIND and self-collected Piper, and simulator geometry is used for RoboTwin and transferred WidowX250.Source: §A.2.1 · pp. 15–16
  • Evaluation episodes may receive additional calibration after quality checks, without using ground-truth frames from the prediction interval; calibrated geometry is then frozen for rendering conditions.The calibration workflow spans raw demonstrations, geometric calibration, motion filtering, and two training stages.Source: §A.2.1 · p. 16, Figure 10 · p. 16
  • Released per-episode calibration covers 153,666 episodes and 1,660.31 h across AgiBotWorld-Beta, DROID, and RoboMIND 2.0.Records include camera intrinsics, extrinsics, and distortion; DROID and RoboMIND 2.0 records also include mounting offsets and calibration-quality metrics.Source: §A.2.1 · p. 16

A.2.2 Rendered Channels

  • Preprocessing stores robot RGB, normalized depth, masks, camera metadata, and robot state; RGB and depth are VAE-encoded, while masks and Plücker maps enter the geometry encoder.Link-level segmentation and simulator metadata support preprocessing and evaluation but are not model inputs.Source: §A.2.2 · p. 16

A.2.3 Gripper Channel Convention

  • Gripper openings are mapped to background RGB intensities from 0 to 255, with left and right grippers encoded in red and green channels.These continuous state cues are included in rendered RGB before VAE encoding.Source: §A.2.3 · p. 16

A.2.4 Counterfactual Trajectory Construction

  • Counterfactual trajectories perturb the demonstrated final end-effector pose, interpolate from the unchanged initial pose, and use inverse kinematics to generate modified joint sequences.Initial observations and instructions remain unchanged; calibration is fixed, robot and wrist-camera renderings are recomputed, and only kinematically feasible conditions are retained.Source: §A.2.4 · p. 17

B Reward Model, Annotation, and Metrics

This section describes the human defect labels and reward-model corpus used for post-training, along with the metrics for reward validation and action following.

B.1 Embodied Video Reward Model and Human Annotation

  • Annotations distinguish Embodiment, Object, and Interaction defects; reward-model labels are binary, while downstream defect rates use majority labels from three independent annotators.The dimensions cover robot duplication, disappearance, structural or material defects; object appearance, disappearance, deformation, material or texture defects; and contact–motion mismatch or implausible interaction.Source: §B.1 · p. 17
  • The reward model predicts three defect-label distributions in one structured response and assigns each dimension a negative defect-probability reward.It is fine-tuned on ground-truth videos and human-annotated predictions, and receives only the video and a fixed prompt when scoring.Source: §B.1 · p. 17
  • The reward-model corpus spans AgiBot, DROID, RoboMIND, RoboTwin, and Bridge predictions, with 31,600 training and 4,893 validation paths split by source episode.Bridge is used for reward-model training and validation, not for training the multi-view video generator.Source: §B.1 · p. 17
  • The released annotation corpus covers 44.9K original videos and 30.4K defect labels; 36,493 distinct video paths remain for training and validation.The retained set contains 31,600 training videos and 4,893 validation videos.Source: §B.1 · p. 17

B.2 nDTW Action-Following Metric

  • Probability-based MAE compares predicted defect probabilities with binary annotations, while nDTW measures gripper-trajectory agreement for action-following evaluation only.nDTW does not enter Stage-II post-training; Embodiment, Object, and Interaction defects are evaluated separately.Source: §B.1 · p. 18, §B.2 · p. 18

C.1 Evaluation and Human Annotation Details

The evaluation uses shared conditions across methods and human annotation of counterfactual videos, with agreement and defect-rate reporting specified separately.

C.1 Evaluation and Human Annotation Details

  • All methods receive the same initial observation, task instruction, and counterfactual action semantics, and are evaluated on the same conditions and metrics.DreamTrue receives rendered robot conditions and camera-ray encodings through its native conditioning interface.Source: §C.1 · p. 18
  • Three annotators independently assess 960 counterfactual videos for Embodiment, Object, and Interaction defects, with identities hidden and video order randomized.Every video has triplicate annotations; a dimension is defective when at least two annotators identify a defect, and defect dimensions may overlap.Source: §C.1 · pp. 18–19
  • The recorded-action quality table aggregates six displayed dimensions into EWMScore-P and highlights the best, second-best, and third-best scores.The table caption identifies the evaluation as WorldArena quality on recorded actions.Source: Table 7 · p. 19
  • Table 8 reports observed agreement and Fleiss κ, while Table 9 reports majority-vote defect rates with Wilson 95% confidence intervals.Pairwise comparisons use two-sided exact McNemar tests and are descriptive, without multiple-comparison correction.Source: §C.1 · p. 19, Table 8 · p. 19

C.2 Policy Evaluation Details

Policy-outcome evaluation compares world-model rollouts with simulated outcomes across bimanual tasks, while supplementary figures show additional OOD and cross-embodiment predictions.

C.2 Policy Evaluation Details

  • Evaluation covers beat block hammer, handover block, and place dual shoes across Easy clean scenes and Hard scenes with randomized unseen textures.The 60 fixed scenarios comprise 10 random seeds per task under each visual setting; each was verified as solvable by expert teleoperation.Source: §C.2 · p. 19
  • Four open-source VLA policies execute across the 60 cases, yielding 240 episodes; human annotators judge each world-model rollout against RoboTwin task-success conditions.Success labels are aggregated by majority vote, with annotators blinded to model and policy identity.Source: §C.2 · pp. 19–20
  • The policy benchmark reports calibration, ranking, and episode-classification metrics, including success-rate MAE across 24 task–policy–setting combinations.MAE is the absolute gap between rollout and simulated success rates, each measured over 10 episodes; positive bias denotes optimistic hallucination of failed manipulations as completions.Source: §C.2 · p. 20
  • Our model produced 12 false positives versus Ctrl-World’s 26, with 87.63% precision, 88.54% recall, and 90.42% accuracy; Ctrl-World precision was 76.15%.The comparison covers 240 policy episodes, including 96 ground-truth successes and 144 failures.Source: §C.2 · pp. 20–21, Table 11 · p. 21

D Additional Out-of-Distribution Rollouts

E Additional Cross-Embodiment Rollouts

Loading 2610.12468v1…