Source-linked AI summary
mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAs
Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, Elvis Nava
TL;DR
VLAs must learn physical dynamics from costly robot demonstrations because their image-text pretraining is static, whereas video pretraining captures dynamic and procedural information. mimic-video pairs a pretrained video backbone with a flow-matching inverse-dynamics action decoder conditioned on latent visual plans. It reports state-of-the-art simulated and real-world performance, with 10x sample-efficiency improvement and 2x faster convergence than traditional VLA architectures.
Problem
Static image-text pretraining leaves VLAs to infer physical dynamics and temporal dependencies from scarce, expensive expert robot trajectories.
Method
mimic-video grounds robot control in a pretrained generative video model’s latent representations and uses a flow-matching action decoder as an inverse dynamics model.
Results
mimic-video achieves state-of-the-art performance across simulated and real-world manipulation tasks while improving sample efficiency by 10x and convergence speed by 2x over traditional VLA architectures.
Takeaways & Limitations
Pretrained generative video representations provide a more robust signal for policy learning than vision-language-action pretraining alone within the evaluated manipulation settings.
Takeaways & Limitations
The current model uses a single-view video backbone, lacks a unified large-scale cross-embodiment model, and evaluates only a focused set of real-world tasks.
Abstract
from arXiv · showhide
Prevailing Vision-Language-Action Models (VLAs) for robotic manipulation are built upon vision-language backbones pretrained on large-scale, but disconnected static web data. As a result, despite improved semantic generalization, the policy must implicitly infer complex physical dynamics and temporal dependencies solely from robot trajectories. This reliance creates an unsustainable data burden, necessitating continuous, large-scale expert data collection to compensate for the lack of innate physical understanding. We contend that while vision-language pretraining effectively captures semantic priors, it remains blind to physical causality. A more effective paradigm leverages video to jointly capture semantics and visual dynamics during pretraining, thereby isolating the remaining task of low-level control. To this end, we introduce mimic-video, a novel Video-Action Model (VAM) that pairs a pretrained Internet-scale video model with a flow matching-based action decoder conditioned on its latent representations. The decoder serves as an Inverse Dynamics Model (IDM), generating low-level robot actions from the latent representation of video-space action plans. Our extensive evaluation shows that our approach achieves state-of-the-art performance on simulated and real-world robotic manipulation tasks, improving sample efficiency by 10x and convergence speed by 2x compared to traditional VLA architectures.
I. INTRODUCTION
mimic-video addresses the data-efficiency limits of VLA policies by grounding robot control in pretrained video representations that encode dynamics and procedures. It combines a generative video model with a flow-matching action decoder and reports strong simulated and real-world results.
- Motivation: Static image-text pretraining leaves VLAs to learn physical dynamics and temporal procedures from scarce, expensive robot demonstrations.This creates a data-efficiency bottleneck for scalable manipulation learning.
- Motivation: Internet-scale video captures how objects move, deform, and react to forces, providing dynamic and procedural information absent from static image-text pairs.
- Related work: Prior video-policy methods often require fully generating future frames at every control step, making inference computationally prohibitive.
- Approach: mimic-video grounds control in latent representations of a generative pretrained video model and introduces a Video-Action Model combining video modeling with robot control.
- Results: 10x sample efficiency and 2x convergence speed improvements are reported against traditional VLA architectures, alongside state-of-the-art results across simulated and real-world manipulation tasks.
- Novelty: The approach directly uses internet-scale video priors and intermediate latent states to avoid full video generation and heuristic tracking.
III. CASE STUDY: HOW DOES VIDEO GENERATION QUALITY AFFECT ROBOT POLICY PERFORMANCE?
The case study examines whether pretrained video representations separate future-video prediction from action execution. It motivates mimic-video’s use of intermediate video latents and partial denoising for control.
- Case study motivation: Video models are hypothesized to encode images, physical dynamics, and visual action plans, leaving the action decoder with a lighter control-learning role.
- Case study design: The oracle study compares action decoding from predicted video latents with decoding from ground-truth future-video latents to disentangle prediction from execution.
- Architecture: mimic-video couples a pretrained language-conditioned video backbone with a lightweight flow-matching action decoder functioning as an inverse dynamics model.
- Flow matching: Flow matching interpolates clean data at τ = 0 with Gaussian noise at τ = 1 and learns a generating vector field for this probability path.
- Partial denoising: Partial denoising stops the video-generation process at an intermediate τ > 0, producing latent representations used to condition action prediction.
B. Architecture Formulation
The architecture formulates robot control as action-sequence prediction conditioned on visual observations, language, proprioception, and latent video representations. Video and action flow models are trained with the same flow-matching objective.
- Policy formulation: The policy predicts an action sequence from multiple RGB images, a language instruction, and the robot’s proprioceptive state.
- Model components: mimic-video uses two flow-matching models: a pretrained video model and a lightweight action decoder.
- Latent conditioning: The video model’s hidden states are extracted from noisy future-video encodings at a chosen flow time and used as control representations.
C. Video Model
mimic-video uses a pretrained video model to produce latent visual plans, then conditions an action decoder on intermediate representations for efficient robot-action sampling. Partial denoising avoids full video reconstruction, while τv=1 supports real-time inference.
- C. Video Model: Cosmos-Predict2 provides the video-model backbone, processing context frames and noisy future-frame latents with language conditioning.The implementation uses five context frames and a 2B latent Diffusion Transformer.
- C. Video Model: The action decoder conditions robot-action generation on intermediate video representations rather than requiring pixel-level future-video reconstruction.It uses cross-attention to hτv within each decoder layer.
- C. Video Model: Partial denoising extracts structural semantic features from intermediate video states and passes them to the action decoder for action generation.The decoder then fully denoises the action trajectory into a robot-action chunk.
- C. Video Model: τv is task-dependent but generally near 1; τv=1 requires one video-backbone forward pass and balances policy performance with inference speed.At τv=1, the additional video-processing step becomes redundant, facilitating real-time inference.
F. Training
VAM training separates video-backbone adaptation from action-decoder learning using disjoint parameter sets. The frozen-backbone decoder is trained with independently sampled video and action flow times for robustness and efficiency.
- F. Training: The video backbone is adapted with LoRA on robotics videos while preserving pretrained temporal reasoning capabilities.This stage aligns the generalist backbone with the visual domain and dynamics of robotic tasks.
- F. Training: The action decoder is trained from scratch to regress the action flow field conditioned on representations from the frozen video backbone.The backbone remains frozen during this second training stage.
- F. Training: Training samples independent video and action flow times, τv and τa, during each iteration to improve robustness across inference noise levels.The video and action time distributions are specified separately for their respective training processes.
- F. Training: The decoupled training scheme is reported to be more sample-efficient and faster to converge than comparable VLA baselines.The claim is tied to the separate backbone and decoder training design.
V. EXPERIMENTS
Experiments assess mimic-video across simulated benchmarks and real-world dexterous manipulation, including generalization, multi-task precision, embodiment diversity, and data efficiency. Comparisons include VLA and diffusion-policy baselines under standardized evaluation settings.
- V. EXPERIMENTS: The evaluation spans SIMPLER, LIBERO, and real-world humanoid-hand experiments across single-arm, multi-task, and bimanual dexterous settings.These setups probe generalization under visual shifts, precise multi-task manipulation, and contact-rich control.
- V. EXPERIMENTS: Real-world bimanual tasks use two 16-DoF mimic hands on Panda arms, with workspace, wrist-camera, and proprioceptive observations.The tasks are package sorting and tape stowing, both requiring long-horizon manipulation.
- V. EXPERIMENTS: The real-world action decoders use only 1h 33m and 2h 14m of task-specific data for sorting and stowing, respectively.The video backbone is finetuned on a broader 200-hour robotics-video corpus.
- V. EXPERIMENTS: Comparisons include a multi-view DiT-Block Policy, published methods such as Octo and OpenVLA, and a π0.5-style VLA baseline.The π0.5-style baseline uses a VLM checkpoint with a separately trained flow-matching decoder.
A. Direct Evaluation across Diverse Robot Platforms
mimic-video performs strongly across SIMPLER-Bridge, LIBERO, and real-world bimanual dexterous manipulation. The evaluations also examine task-specific τv optimization and robustness to occlusion with scarce task-specific data.
- A. Direct Evaluation across Diverse Robot Platforms: mimic-video achieves the strongest average success rate across all four SIMPLER-Bridge tasks, matching or surpassing state-of-the-art baselines.Task-optimized τv can further improve performance with modestly increased computation.
- A. Direct Evaluation across Diverse Robot Platforms: Adjusting τv enables inference-time specialization to individual task dynamics without retraining the fixed model.This strategy trades modest additional computation for further performance gains.
- A. Direct Evaluation across Diverse Robot Platforms: On LIBERO, mimic-video outperforms most state-of-the-art methods despite training action data from scratch on task-specific data.It achieves significantly higher success rates than the comparable π0.5-style VLA baseline.
- A. Direct Evaluation across Diverse Robot Platforms: In real-world bimanual manipulation, mimic-video significantly surpasses both DiT-Block Policy baselines while using only a single workspace camera view.The comparison includes workspace-only and multi-view variants under heavy grasping occlusions.
B. Data Efficiency and Convergence Speed
Conditioning action decoders on video-model representations substantially improves sample efficiency and convergence relative to a π0.5-style VLA baseline. Mimic-video reaches comparable success with far less action data and converges faster to a higher asymptotic success rate.
- Sample Efficiency: 10% of the training data suffices for mimic-video to reach the maximum success rate achieved by the VLM-conditioned decoder.This result is reported across LIBERO-Goal, LIBERO-Spatial, and LIBERO-Object task suites.
- Convergence Speed: Mimic-video’s action decoder converges significantly faster and to a higher asymptotic success rate than the π0.5-style VLA decoder.The advantage persists even though the VLA baseline received task-specific action data during FAST-pretraining.
C. Trade-offs between Video Fidelity and Action Performance
Mimic-video does not require fully reconstructed future video for effective action prediction. Intermediate or highly noisy video representations can provide stronger policy performance and lower action reconstruction error, while also reducing inference cost.
- Policy Performance: τv = 1 yields the best autonomous policy performance in the SIMPLER experiments, despite corresponding to pure noise in the video-generation process.The result contrasts with the intuition that higher-fidelity video should produce better policies.
- Action Reconstruction: τv ≈ 0.4 produces the lowest action reconstruction MSE on held-out BridgeDataV2 data.Reconstruction error increases toward both clean and pure-noise video latents.
- Action Reconstruction: Action prediction error increases sharply as conditioning moves from the intermediate optimum toward τv = 0, or full reconstruction.The authors attribute this behavior to non-trivial intermediate representations and distribution effects in the video model.
- Inference Trade-offs: Operating at τv = 1 requires only a single forward pass of the video backbone, producing the highest average performance and fastest inference speed.This setting avoids iterative video denoising during action sampling.
- Inference Trade-offs: Partially denoised plans mitigate distribution shift while reducing the computational cost of video generation.The discussion frames partial denoising as a dual benefit for policy conditioning and inference efficiency.
- Scope Boundary: The current model uses a single-view video backbone and evaluates only a focused set of real-world tasks.The authors identify multi-view architectures, cross-embodiment training, and broader manipulation diversity as future directions.
APPENDIX
The appendix specifies the observation, action, preprocessing, and representation settings used across the evaluated datasets. These details cover both single-arm and dexterous manipulation configurations.
- Representation and Rendering: Images are extracted or rendered at 480 x 640 px, while orientations use 6-dimensional vectors corresponding to the top two matrix rows.
- Observation Space: Observations include absolute end-effector pose and continuous gripper joint state.This observation specification is listed for the corresponding single-arm settings.
- Action Space: Actions encode future end-effector pose relative to the proprioceptive pose and gripper control over the action chunk.The gripper is continuous in one specification and binary in another.
- Preprocessing: The preprocessing removes 3046 non-informative language labels and the first state and null-action from each episode.Another preprocessing step removes episodes whose replayed actions do not produce successful rollouts.
- Dexterous Settings: Dexterous observations additionally include relative end-effector poses and previous end-effector and hand actions.The dexterous action space includes relative end-effector pose and absolute hand-joint actions.
C. Video-Action Model Learnings •
The appendix identifies representation and training choices that affect policy performance, including intermediate video features, observation horizon, decoder training, and noisy video conditioning. These findings support partial denoising as a central design choice.
- Video Representation: Video model layer k = 19 yields the strongest policy performance, with success decreasing toward the initial and final layers.The authors suggest that the optimal layer should ideally be learned rather than fixed.
- Video Representation: A five-frame video observation horizon performs better than conditioning on only the current observation.
- VLA Baseline: Layer 11 produces the highest SIMPLER-Bridge success rates when used for cross-attention in the FAST-pretrained VLM.
- VLA Baseline: For SIMPLER-Bridge, extending decoder training beyond 2–3 epochs does not significantly improve success rates after the frozen FAST backbone converges.For LIBERO, slightly extending FAST pretraining beyond convergence yields modest downstream gains.
- Video Denoising: Stopping video generation early and conditioning on noisy visual plans performs substantially better than fully denoising predictions.The authors identify the cutoff flow time τv as a critical inference hyperparameter.