Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos

• Moto addresses the high cost of action-labeled robotics data by asking whether autoregressive pre-training on interaction-rich videos can improve robot learning.Source: §1 · p. 2
• The method prioritizes motion over static visual detail because motion reflects state changes, aligns with low-level actions, and is hardware-agnostic.Source: §1 · p. 2
• Moto encodes visual dynamics into unsupervised latent motion tokens, pre-trains Moto-GPT by next-token prediction, and transfers the learned priors through co-fine-tuning.Source: §1 · p. 2
• Experiments report interpretable motion tokens, plausible trajectory prediction, trajectory-rationality assessment, and stronger robot-policy performance than training without motion priors.Source: §1 · p. 2
• The contributions combine unsupervised motion-language learning, action-free video pre-training, and a co-fine-tuning strategy for robot manipulation.Source: §1 · p. 2

• Vision-language-action studies generate robot actions from observations and instructions, often using vision-language pre-training or diverse action-labeled robot data.Source: §2 · p. 2
• Moto differs by using latent motion tokens as an intermediate language for generative video pre-training and robot control.Source: §2 · pp. 2–3, Figure 2 · p. 3
• Robot-learning methods also improve control through multiple views, depth, action chunking, policy diffusion, or latent-skill decomposition.Source: §2 · p. 3
• Video-based robot learning has used contrastive representations, video plans, generative policy pre-training, and autoregressive prediction based mainly on pixels or patch-level tokens.Source: §2 · p. 3

Overview

• Moto first trains a Latent Motion Tokenizer, then pre-trains Moto-GPT on motion-token sequences, and finally co-fine-tunes a robot action policy.Source: §3.1 · p. 3

Latent Motion Tokenizer

• The tokenizer extracts motion features from successive frames, quantizes them into latent tokens, and reconstructs the later frame from the earlier frame and tokens.Source: §3.2 · pp. 3–4

Motion Token Autoregressive Pre-training

• Videos are represented by concatenating token chunks from consecutive frame pairs into chronological motion trajectories for autoregressive modeling.Source: §3.2 · p. 4, §3.3 · p. 4
• Moto-GPT predicts each motion token from language, visual features, and preceding tokens, with M = K*T representing tokens across video length.Source: §3.3 · p. 4

Co-fine-tuning for Robot Manipulation

• During fine-tuning, action query tokens feed an MLP action head that predicts robot actions, while the original motion-token objective is retained.Source: §3.4 · p. 4
• The action loss combines Smooth-L1 losses for continuous displacements with binary cross-entropy for gripper state, alongside motion loss in the total objective.Source: §3.4 · p. 4

Latent-motion interpretability

• Figure 4 evaluates whether latent motion tokens produce consistent effects across starting observations and discriminative effects across different token choices.Source: Figure 4 · p. 5

Video imitation

• Figure 5 tests whether motion-token sequences extracted from demonstrations preserve movement semantics when decoded from a different initial frame.Source: Figure 5 · p. 5

Robot benchmarks

• SIMPLER evaluates three Google Everyday Robot tasks using 109k pre-training trajectory videos and 73k action-labeled expert trajectories for fine-tuning.Source: §4 · p. 5
• Generalization tests vary manipulated-object appearance or add irrelevant visual distractors.Source: §4 · p. 5

5.1. Latent Motion Token as an Interpretable Motion Language

• Latent motion tokens reconstruct fine-grained frame dynamics, support controllable motion variation, and transfer consistently across starting observations and robot embodiments.Ground-truth tokens reconstruct authentic next frames, while different token chunks vary motion orientation and scale and identical chunks produce consistent effects.Source: §5.1 · pp. 5–6, Figure 6 · p. 6
• 79.7% classification accuracy on 34 CALVIN tasks shows latent motion tokens retain semantic information while reducing representation size to 8 tokens per frame.This matches the accuracy of using ViT features for all eight frames, despite reducing input features from 196 to 8 tokens per frame.Source: §5.1 · p. 6, Table 1 · p. 6

5.2. Moto-GPT as a Useful Motion Prior Learner

• Moto-GPT predicts motion trajectories from language instructions and uses autoregressive likelihoods to assess whether trajectory videos are successful, failed, or random.The predicted token sequences generate plausible video trajectories, while log-likelihoods provide a potential reward signal for trajectory rationality.Source: §5.2 · p. 6, Figure 7 · p. 6

5.3. Moto-GPT as an Effective Robot Policy

• After fine-tuning, Moto-GPT significantly outperforms Moto without motion tokens and performs competitively with larger models on SIMPLER and in the unseen CALVIN environment.Moto-GPT uses a 98M-parameter GPT backbone, no action labels during pre-training, and only static-camera RGB inputs in the reported comparison.Source: §5.3 · p. 6, Table 2 · p. 7, Table 3 · p. 7
• Adding SSV2 human videos further improves performance, while Moto-GPT reaches 52.5% success with only 1% labeled data compared with 0% for the from-scratch variant.Latent motion tokens also provide a shared interface for translating human motions into semantically aligned robot actions.Source: Figure 9 · p. 7, §5.3 · p. 8, Figure 10 · p. 8
• Ablations show that retaining latent motion tokens and continuing their prediction during co-fine-tuning are both important for outperforming alternatives.Moto-IML and Moto-DM benefit from pre-training but remain behind Moto-GPT, indicating that direct attention and motion-token prediction help preserve learned priors.Source: §5.3 · p. 8, Figure 12 · p. 8

6. Conclusion and Discussion

• Moto bridges generative video pre-training and precise robot control by translating motion-related video priors into robot actions without hardware-specific action labels.The paper presents latent motion tokens as the interface enabling this transfer.Source: §6 · p. 8
• The method extends beyond robot videos to Internet-scale human videos and enables cross-embodiment knowledge transfer.The paper identifies scaling the pre-training dataset and improving fine-tuning as future directions.Source: §6 · p. 8

7.1. SIMPLER and CALVIN

• SIMPLER evaluates Pick Coke Can, Move Near, and Open/Close Drawer on the Google Everyday Robot embodiment.The benchmark includes varied object orientations for grasping and object-relocation instructions.Source: §7.1 · p. 13, Figure 13 · p. 13
• CALVIN evaluates five-task sequences with unconstrained language instructions on a Franka Emika Panda across environments A, B, C, and unseen D.Training uses environments A, B, and C to test zero-shot generalization to D.Source: §7.1 · p. 13, Figure 14 · p. 13, §7.2 · p. 14

7.2. Real-world Robot Experiments

• Real-world evaluation uses Pick-Place Banana, Close Laptop, and Disassembly on a FANUC LR Mate 200iD, with randomized initial object positions and novel-object or visual-distractor conditions.Each task is tested 10 times, and generalization changes object appearance or adds irrelevant objects.Source: §7.1 · p. 13

7.4. Data and Training Sets

• SIMPLER pre-training uses 109k OXE trajectory videos, while fine-tuning uses 73k action-labeled RT-1 trajectories; CALVIN and real-world experiments use their respective demonstration sets.The real-world experiments collect 90 teleoperated demonstrations, and SIMPLER data are entirely real-world rather than simulated.Source: §7.2 · pp. 13–14

7.3. Compared Models

• Comparisons include action-token policies, large vision-language models, image-editing approaches, pixel-prediction models, and visual-encoder variants.The baseline sets are defined separately for SIMPLER and CALVIN.Source: §7.3 · p. 14

7.5. Ablations

• Ablations compare training from scratch without motion tokens, retaining tokens without their fine-tuning loss, and removing tokens during fine-tuning.These variants isolate pre-training, token retention, and co-fine-tuning contributions.Source: §7.3 · p. 14

8.1. Latent Motion Tokenizer Training

• Videos are downsampled so successive frames contain sufficiently distinct visual motion for Latent Motion Tokenizer training.For OXE videos, the reported sampling interval is one frame every three frames, or ∆t = 3.Source: §8.1 · p. 14
• Tables 4 and 5 specify the Latent Motion Tokenizer’s trainable modules and training hyperparameters.The tokenizer is trained on four 40G GPUs.Source: §8.1 · p. 14, Table 5 · p. 15

8.2. Moto-GPT Training

• Tables 6 and 7 specify Moto-GPT’s implementation and training hyperparameters, with the Action Head included only during fine-tuning.The model handles a maximum video length of three frames and uses the same downsampling rate in pre-training and fine-tuning.Source: Table 6 · p. 15, Table 7 · p. 15, §8.2 · p. 15

8.3. Real-world Training

• The real-world model is further pre-trained for 5 epochs on OXE videos plus 90 newly collected trajectory videos, then fine-tuned with robot action labels for 20 epochs.This additional pre-training and fine-tuning schedule is reported for the real-world experiments.Source: §8.2 · p. 15

• Discrete motion tokens from an initial and next frame are decoded with the initial frame to reconstruct the target frame.Source: Figure 15 · p. 15
• The reported analyses cover both token-based visual reconstruction and autoregressive prediction of latent motion tokens.Source: Figure 15 · p. 15, Table 8 · p. 15
• Table 8 evaluates Moto-GPT using top-K accuracy for predicting ground-truth latent motion tokens.The evaluation uses a 128-size codebook and validation splits from the pre-training datasets.Source: Table 8 · p. 15

• SSV2 experiments suggest potential for human-to-robot motion transfer through latent motion tokens.Source: §10 · p. 16
• Moto-GPT transfers motion priors from videos, including human videos, to robot action prediction after fine-tuning on action-labeled trajectories.The authors describe this transfer as particularly beneficial in low-resource scenarios.Source: §10 · p. 16
• Beyond imitation learning for manipulation, Moto may serve as a reward model for trajectory rationality or as a vivid environment simulator.Source: §10 · p. 16
• The authors propose extending Moto toward reinforcement learning robustness and robotic tasks such as navigation and locomotion.Source: §10 · p. 16