PaperlayerView exact revision on arXiv

Source-linked AI summary

AutodidactWAM: Cross-Modal Self-Distillation from Generated Video to Robot Actions

Sergei Kurchev, Iaroslav Kolomiets, Miguel Altamirano Cabrera, Artem Lykov, Dzmitry Tsetserukou

arXiv:2610.08119v1cs.RO

TL;DR

On an unseen dexterous humanoid embodiment, Cosmos 3’s generated video can look plausible while its co-generated actions are mis-targeted. AutodidactWAM recovers action targets from the model’s own video and fine-tunes its action-related layers. The hybrid objective achieves the strongest reported real-robot results among the trained configurations, including transfer to a held-out object.

  • Problem

    On a previously unseen dexterous humanoid, plausible generated video coexists with systematically mis-targeted native actions, which achieve ≈17% pre-grasp and ≈7% full-task success.

    Source: §I · p. 2

  • Method

    AutodidactWAM recovers action targets from generated video and uses them to align the model’s action prediction without additional task-specific teleoperation.

    Source: §I · p. 2

  • Results

    DPO+SFT+DTW reaches 90% pre-grasp and 20% full-task success on Oreo, and 80% and 30%, respectively, on a held-out object.

    Source: §IX · p. 8

  • Takeaways & Limitations

    Plain contrastive Flow-DPO reaches 0% closed-loop success despite saturating validation preference accuracy, while the hybrid objective combines preference supervision, supervised fitting, and trajectory anchoring.

    Source: §IX · p. 8

    The recovered action is an estimate that inherits extractor bias, and transfer to substantially different objects or tasks without retraining is untested.

    Source: §VIII · p. 7

Abstract

from arXiv · show

World-action models (WAMs) such as Cosmos 3 jointly generate future video and robot actions from an observation and instruction. Adapting one such model with a lightweight LoRA fine-tune to a previously unseen robot, a Unitree G1 humanoid with five-fingered BrainCo hands, exposes a video-action asymmetry: the video renders plausible task executions, while the co-generated action is systematically mis-targeted. We evaluate closed-loop real-robot trials at three cumulative stages: pre-grasp, grasp, and pick-and-place. The native action succeeds only approximately 17%, 10%, and 7% of the time, respectively, and performs worse on held-out objects. We propose AutodidactWAM, a hand-pose estimator trained without teleoperation, followed by inverse kinematics, that runs on the model's generated video to recover action estimates. Paired with the native prediction, these recovered actions provide preferred targets for fine-tuning only the action-related layers, while the generated video is teacher-forced. We compare supervised relabeling with a rectified-flow adaptation of Diffusion-DPO. After one-time embodiment adaptation, self-distillation requires no additional task-specific teleoperation. The recovered-action gate reaches approximately 75%, 47%, and 42% pre-grasp, grasp, and pick-and-place success, compared with 17%, 10%, and 7% for the native action. A hybrid objective combining preference supervision, supervised target fitting, and Cartesian trajectory anchoring (DPO+SFT+DTW) performs best: on Oreo, the training object, it reaches 90% pre-grasp and 20% full-task success; on a held-out object, it reaches 80% and 30%. Plain Flow-DPO reaches 0% success despite 1.000 validation preference accuracy, indicating that the combination of training objectives, rather than the contrastive objective alone, drives the observed gains.

I. INTRODUCTION

Adapting Cosmos 3 to a previously unseen dexterous humanoid reveals that plausible generated video can coexist with systematically mis-targeted native actions. AutodidactWAM recovers action targets from generated video and uses them to fine-tune the action-related layers without additional task-specific teleoperation.

I. INTRODUCTION

  • Teleoperation is slow, robot-specific, and must be repeated for new embodiments, especially for dexterous multi-fingered hands.WAMs such as Cosmos 3 jointly predict future video and actions from vision and language.Source: §I · p. 1
  • The recovered trajectory comes from a frozen hand-pose and inverse-kinematics stack applied to each generated rollout, then is paired with the native prediction to form a target.The paper compares supervised relabeling with a rectified-flow adaptation of Diffusion-DPO.Source: §I · p. 2
  • ≈17% pre-grasp and ≈7% full pick-and-place success: the native action remains mis-targeted despite plausible generated video.The reported rates are from closed-loop trials on the real robot.Source: §I · pp. 1–2
  • AutodidactWAM recovers executable action targets from the model’s generated video and uses them to align its action prediction without additional task-specific teleoperation.The method pairs recovered and native actions to fine-tune action-related layers while teacher-forcing the generated video.Source: §I · p. 2
  • A synthetic data engine generates self-distillation pairs from a small set of static observations using seed, prompt, and object-placement diversification.The generated examples require no manual annotation.Source: §I · p. 2

II. RELATED WORK

AutodidactWAM differs from prior approaches by repairing a world-action model’s action output from its own generated video rather than using video to drive a separate controller. Its method creates paired action trajectories from one rollout, then iteratively fine-tunes the action-related layers.

II. RELATED WORK

  • Unlike prior video-as-control work, AutodidactWAM uses generated video to repair the same model’s action branch rather than drive a downstream controller.The recovered target is used contrastively against the model’s native action, not only as a regression target.Source: §II · p. 2
  • Prior methods address the video–action gap with inverse models or policy optimization; AutodidactWAM instead corrects the action branch offline with hand-pose reconstruction and IK.Its preference pairs contrast the recovered action with the model’s own native action for the same generation.Source: §II · p. 2

III. METHOD

  • A single rollout produces video and native action; a frozen perception stack reads the video to recover a second action trajectory, which becomes the preferred target.The generated video is teacher-forced so fine-tuning updates only the action tokens.Source: §III · p. 2
  • The action pair is regenerated from the updated model at a fixed step interval, keeping the native action on-policy and the recovered target tied to current video.The method’s stages are generation, action recovery, and action-branch fine-tuning.Source: §III · p. 2

A. Preliminaries: Cosmos 3 as a joint video–action flow model

Cosmos 3 jointly denoises video and action tokens in one rectified-flow generator. AutodidactWAM turns each rollout into a preferred-versus-native action pair, with targets chosen according to the training objective and refreshed from the current policy.

A. Preliminaries: Cosmos 3 as a joint video–action flow model

  • Cosmos 3 jointly denoises video and action latent tokens in one rectified-flow network, rather than using separate video and action subnetworks.At inference, it emits a video rollout and a 57-D action chunk for the G1.Source: A. Preliminaries: Cosmos 3 as a joint… · p. 3

B. From one generation to a contrastive pair (a+, a−)

  • A frozen perception stack maps generated video to an estimated action trajectory, which is paired with the native action from the same forward pass.The recovered trajectory is a reconstruction, not ground truth.Source: B. From one generation to a… · p. 3
  • The preferred target is the recovered trajectory for SFT or a mix of that estimate and the round-starting prediction for Flow-DPO.For Flow-DPO, the pair is meaningful because the video is visually verifiable while the native action can be mis-targeted.Source: B. From one generation to a… · p. 3
  • At the start of each round, both actions are regenerated from the current checkpoint, and the reported reference model is an EMA of the policy.The EMA reference uses decay 0.999; a fully frozen reference is untested.Source: B. From one generation to a… · p. 3

C. Recovering ˆa: the RoboHaMeR extractor

AutodidactWAM recovers action trajectories from generated video, then trains the action-related layers with supervised or preference-based objectives. Flow-DPO contrasts the recovered target with the native action, while the combined objective also includes supervised fitting and trajectory anchoring.

C. Recovering ˆa: the RoboHaMeR extractor

  • The frozen extractor detects hands, estimates wrist and finger pose from synthetic renders, and uses differential arm IK to produce a filtered 57-D action trajectory.On a held-out physical scene, actuator-stroke MAE is ≈0.9% and 3-D keypoint MAE is ≈0.7 mm; these bound precision, not cross-scene generalization.Source: C. Recovering ˆa: the RoboHaMeR… · p. 3

D. Distillation objectives

  • The flow-matching residual evaluates an action chunk under weights φ against the rectified-flow target, using action tokens alone and conditioning context κ.The interpolated action is at = (1 − t)a + tn.Source: D. Distillation objectives · p. 3

1) Supervised relabeling (SFT baseline):

2) Flow-DPO:

  • Flow-DPO adapts Diffusion-DPO to rectified flow by replacing the noise-prediction error with a flow-matching residual, since actions lack tractable sequence likelihoods.Its EMA reference starts from the same post-adaptation checkpoint as the trainable policy.Source: 2) Flow-DPO · p. 3
  • Unlike SFT, Flow-DPO scores preferred and rejected actions under identical conditioning and noise, moving along the direction separating them.The pipeline’s combined objective adds SFT and Cartesian/DTW anchoring to the preference term.Source: 2) Flow-DPO · p. 3, Figure 2 · p. 4

3) Cartesian target anchor (the DTW term):

The DTW term anchors updates to physical target positions, addressing a blind spot in the preference margin. It compares FK-mapped predicted and target wrist trajectories, which may drift out of phase.

3) Cartesian target anchor (the DTW term):

E. Teacher-forcing the video: updating the action branch only

The update teacher-forces generated video so preference gradients affect only the action branch. The implementation uses a shared training configuration across SFT and DPO variants, with distinct schedules and cached recovered actions.

E. Teacher-forcing the video: updating the action branch only

  • Teacher-forcing the generated video keeps preference gradients out of the video branch and confines updates to action tokens, action-interface layers, and LoRA adapters.Each step uses four action-only forward passes, avoiding a full joint denoising rollout.Source: E. Teacher-forcing the video: updating… · p. 4
  • Distillation starts from the embodiment-adapted checkpoint, whose native action trace supplies a− without task- or scene-specific tuning.Source: F. Where the generated video comes from · p. 4

G. Implementation

  • The shared trainer updates 16.9M parameters, using rank-64 LoRA plus action-interface tensors while freezing everything else.The base is Cosmos 3 Nano, with the optimizer limited to LoRA weights and action-interface tensors.Source: G. Implementation · p. 4
  • SFT uses a fixed manifest of 105 single-source episodes and 500 optimizer iterations, with checkpoint selection by the DTW-path metric.The targets are RoboHaMeR-extracted once; there is no on-policy refresh or reference model.Source: G. Implementation · pp. 4–5
  • Flow-DPO uses 20 rounds of 25 optimizer steps, with on-policy generations, anchor-pool replay, and an EMA reference.Its 500 iterations match the SFT optimizer-step budget; rollback-on-regression was disabled for the reported run.Source: G. Implementation · p. 5
  • Generation uses 129 frames, 10 sampling steps, and guidance 4.5, with seeds and prompt paraphrases diversifying samples.Source: G. Implementation · p. 5
  • The frozen perception model reads each generated video and resamples its recovered action onto the native action’s time grid.Source: G. Implementation · p. 5

IV. SYNTHETIC DATA ENGINE

The synthetic data engine pairs model-generated videos and native actions with recovered actions, using composited anchors rather than filmed scenes. Its data supply can expand through seeds, prompt variants, and object placements, although scaling curves remain future work.

IV. SYNTHETIC DATA ENGINE

  • Composited hand and package images create anchors that yield video, native-action, and recovered-action tuples without requiring filmed scenes.The object can be placed at arbitrary reachable positions in software, and the resulting pairs are cached in a manifest.Source: §IV · p. 5
  • Seeds, prompt paraphrases, and object placements multiply the synthetic-pair supply; each additional pair costs one model pass and one perception pass, with no robot time or human labeling.A scaling sweep is left for future work.Source: §IV · p. 5
  • Self-distillation still requires an initial G1 teleoperation adaptation, and unusable generations are screened using anchor-pose divergence or an optional plausibility judge.Examples of unusable generations include hallucinated hands, dropped objects, and frozen grippers.Source: §IV · p. 5

A. Task, embodiment, and checkpoints

The recovered-action gate outperforms the native action in closed-loop pre-grasp trials on the real robot. The evaluation focuses on one tabletop task and scores outcomes at three cumulative stages.

A. Task, embodiment, and checkpoints

  • The evaluation uses Oreo-package pick-and-place on a Unitree G1 with five-fingered BrainCo hands, starting from a LoRA-adapted checkpoint.The native action reaches approximately 17% pre-grasp success and 0% full pick-and-place success on Oreo.Source: A. Task, embodiment, and checkpoints · p. 5

B. Metrics and protocol

  • Closed-loop success is scored cumulatively at pre-grasp, grasp, and pick-and-place; 20 real-robot trials per object and condition use a held-out set.Offline proxies include windowed RMSE and wrist-position trajectory RMS gap; the wrist DTW/path is tracked during training.Source: B. Metrics and protocol · p. 5
  • A matching video-side closed-loop score has not been run, so the video–action asymmetry is argued through the action-side protocol and reported results.Source: B. Metrics and protocol · p. 5

C. Validating the recovered actions (ˆa gate)

  • The recovered-action gate is tested directly in closed loop because self-distillation lacks a matched ground-truth trajectory for comparison.The perception model’s precision on a held-out physical scene separately bounds its noise floor.Source: C. Validating the recovered actions… · pp. 5–6
  • Approximately 75% of recovered actions succeed at pre-grasp, versus approximately 17% for the native action in closed-loop trials.The gate replays recovered actions open-loop on the real robot under the three-stage protocol.Source: C. Validating the recovered actions… · p. 6

D. Ablation grid

The pilot compares native actions, recovered actions, supervised relabeling, plain Flow-DPO, and a hybrid objective in closed-loop real-robot trials. DPO+SFT+DTW performs best among trained policies, while the recovered-action gate exceeds the native action at each stage.

D. Ablation grid

  • The pilot evaluates five conditions on Oreo and two held-out objects; the fixed-video-source Czs-versus-Bzs comparison isolates contrastive optimization from supervised relabeling.Further objectives and video sources were not evaluated.Source: §VI · p. 6, D. Ablation grid · p. 6

VI. RESULTS

  • DPO+SFT+DTW reaches 0.9 Oreo pre-grasp versus SFT’s 0.7 and 0.2 pick-and-place versus 0.1; on held-out pink, success rises from 10% to 80% pre-grasp and 10% to 30% full-task versus Pre-DPO.The pilot covers three evaluation objects and reports success over 20 trials per object and condition.Source: Table I · p. 6, §VI · p. 6
  • Extracted actions reach approximately 75%, 47%, and 42% pre-grasp, grasp, and pick-and-place success, versus Pre-DPO’s approximately 17%, 10%, and 7%.Recovered actions are replayed open-loop from zero-shot generated video before distillation.Source: Table I · p. 6, §VI · p. 6
  • SFT improves pre-grasp success on Oreo and the pink toy to 0.7 and 0.5 versus Pre-DPO’s 0.2 and 0.1, but degrades sharply at grasp and pick-and-place relative to the gate.The authors attribute this pattern to inheriting extractor errors without a signal about how the native action is wrong.Source: §VI · p. 6
  • Plain Flow-DPO reaches 0% closed-loop success at every stage on every object despite validation preference accuracy of 1.000; DPO+SFT+DTW adds SFT and Cartesian-anchor terms.The two conditions use the same 500-iteration schedule.Source: §VI · pp. 6–7
  • The reported real-robot pilot is limited to those five conditions; additional objectives and video sources are future work, not pending results.The authors state they did not have time to run the further conditions.Source: §VI · p. 7

VII. ANALYSIS

In the completed plain-DPO run, validation preference accuracy does not track wrist DTW/path or closed-loop success. Further training runs are needed to address several open questions, including video-side scoring and scaling.

VII. ANALYSIS

  • Validation preference accuracy reaches 1.000 while wrist DTW/path ranges 25–75 mm around the unchanged base, and the same run has 0% closed-loop success.The authors suggest per-frame wrist translation errors can accumulate into absolute Cartesian drift, but caution that this single run is not conclusive.Source: §VII · p. 7
  • Further runs are needed to measure video-branch performance, test reward hacking and the vision–language screen, explain β under gradient clipping, and chart data-engine scaling.These questions are described as blocked on further training runs.Source: §VII · p. 7

VIII. LIMITATIONS

The conclusion reports stronger closed-loop results for hybrid self-distillation than plain Flow-DPO. The limitations bound these findings by estimator bias, proxy-metric risks, and the pilot’s narrow scope.

VIII. LIMITATIONS

  • The recovered action is an estimate that inherits Robo-HaMeR’s domain gap and generation artefacts, so its residual bias carries into distillation targets.The passage also notes a possible reward-hacking mode and says transfer to substantially different objects or tasks without retraining is untested.Source: §VIII · p. 7
  • Offline proxies can miss the documented preference-margin failure mode, making closed-loop evaluation the only reliable gate for conclusions in this setting.The cited proxies include windowed RMSE, the action-drift readout, and teacher-forced validation loss.Source: §VIII · p. 7
  • Screening is scored only for correlation with recovered-action error, not used as an active filter; its discarded-generation and label-noise trade-off remains future work.Source: §VIII · p. 7
  • The pilot uses one object, Oreo, and covers only four of the defined design points; the proposed multi-object scale-up and several variants remain unrun.Cross-object transfer and any trade-off against Oreo performance are left for future testing.Source: §VIII · p. 7

IX. CONCLUSION

  • 90% pre-grasp and 20% full pick-and-place on Oreo, and 80% and 30% on a held-out object, were achieved by DPO+SFT+DTW; plain Flow-DPO reached 0% despite 1.000 validation preference accuracy.The conclusion attributes the gains to combining preference supervision, supervised target fitting, and Cartesian trajectory anchoring.Source: §IX · p. 8

Related papers

Loading 2610.08119v1…