PaperlayerView exact revision on arXiv

Source-linked AI summary

Video Generation Models in Robotics -- Applications, Research Challenges, Future Directions

Zhiting Mei, Tenny Yin, Ola Shorinwa, Apurva Badithela, Zhonghe Zheng, Joseph Bruno, Madison Bland, Lihan Zha, Asher Hancock, Jaime Fernández Fisac, Philip Dames, Anirudha Majumdar

arXiv:2601.07823v1eess.SYcs.RO

TL;DR

Robotics needs expressive and trustworthy models of physical interactions beyond language-only abstractions and costly real-world evaluation. This survey reviews video models as embodied world models, their robotics applications, and the challenges limiting reliable integration. It identifies applications spanning imitation learning, reinforcement learning, policy evaluation, and visual planning, alongside hallucinations, instruction-following failures, safety gaps, and computational costs.

  • Problem

    Trustworthy robotics requires expressive modeling of physical interactions, but language-based abstractions and current video models have important capability and reliability limitations.

  • Method

    The survey reviews video-model architectures, conditioning modalities, robotics applications, and open challenges, organizing applications into four broad classes.

  • Results

    The survey identifies robot data generation and action prediction, dynamics and rewards modeling, policy evaluation, and visual planning as the main robotics application classes.

  • Takeaways & Limitations

    Video models can serve as embodied world models because they learn fine-grained spatiotemporal relationships governing real-world state evolution.

  • Takeaways & Limitations

    Current video models remain constrained by hallucinations, including violations of physics, and by limited long-horizon generation for robotics tasks.

Abstract

from arXiv · show

Video generation models have emerged as high-fidelity models of the physical world, capable of synthesizing high-quality videos capturing fine-grained interactions between agents and their environments conditioned on multi-modal user inputs. Their impressive capabilities address many of the long-standing challenges faced by physics-based simulators, driving broad adoption in many problem domains, e.g., robotics. For example, video models enable photorealistic, physically consistent deformable-body simulation without making prohibitive simplifying assumptions, which is a major bottleneck in physics-based simulation. Moreover, video models can serve as foundation world models that capture the dynamics of the world in a fine-grained and expressive way. They thus overcome the limited expressiveness of language-only abstractions in describing intricate physical interactions. In this survey, we provide a review of video models and their applications as embodied world models in robotics, encompassing cost-effective data generation and action prediction in imitation learning, dynamics and rewards modeling in reinforcement learning, visual planning, and policy evaluation. Further, we highlight important challenges hindering the trustworthy integration of video models in robotics, which include poor instruction following, hallucinations such as violations of physics, and unsafe content generation, in addition to fundamental limitations such as significant data curation, training, and inference costs. We present potential future directions to address these open research challenges to motivate research and ultimately facilitate broader applications, especially in safety-critical settings.

1 Introduction

Video models offer controllable, high-fidelity representations of physical interactions, while robotics applications increasingly use them as embodied world models. This survey organizes those applications and highlights limitations that hinder trustworthy integration.

  • Video models synthesize high-fidelity videos conditioned on text, robot actions, and video frames, capturing aesthetic effects and physical interactions.Their capabilities arise from recent diffusion and flow-matching breakthroughs and training on internet-scale data.
  • Language models support unified robot policies but remain limited in their ability to represent intricate physical interactions.
  • Robotics uses video models as embodied world models for cost-effective policy evaluation, dynamics and reward prediction, data generation, action prediction, and visual planning.The survey identifies four application classes: imitation learning, reinforcement learning, policy evaluation, and visual planning.
  • Video models hallucinate physically unrealistic content, struggle with instruction following, and incur substantial data curation, training, and inference costs.The survey also identifies inadequate content safeguards as a challenge to trustworthy deployment.
  • The survey builds on prior reviews by combining video-model architectures, controllable generation, world modeling, and their robotics applications and challenges.It also explains the organization of these topics across the survey.

2 Background

This section introduces world models and video-generation foundations, emphasizing why video models offer expressive physical-world representations beyond simplified simulators. It also reviews diffusion-based architectures and their denoising formulation.

  • World Models: World models predict how an agent’s environment evolves through interactions, supporting policy learning when real-world data collection is costly or unsafe.They can enable scalable data collection with little to no real-world interaction.
  • World Models: Physics-based simulators approximate physical laws for computational feasibility, limiting realism for non-rigid objects and contributing to the sim-to-real gap.Increasingly complex manipulation tasks exacerbate these challenges.
  • World Models: Markovian state-based world models predict the next state from the current state and action, typically combining an encoder, dynamics predictor, and rewards predictor.Recent dynamics predictors use transformers or diffusion for more expressive prediction.
  • Video World Models: Video world models instead learn spatiotemporal mappings that capture environmental evolution across space and time without explicitly modeling a Markovian state.Their scene representations may be implicit or explicit, and diffusion and flow-based methods improve expressiveness for physical-world modeling.
  • Diffusion Video Models: Diffusion models generate data by reversing gradual noising: a forward process corrupts samples, while a learned reverse process iteratively removes noise.Training commonly predicts ground-truth noise, with velocity-based parameterization as an alternative.

3 Applications of Video Models in Robotics

Video models act as high-fidelity world models for robotics, supporting data generation, action prediction, policy learning, evaluation, and safety testing. Their applications span imitation learning, reinforcement learning, scalable policy evaluation, and visual planning.

  • Video models generate high-quality future physical-world predictions without requiring prohibitive large-scale action-labeled datasets.
  • Implicit and explicit video world models represent evolving scenes within the video model or through concrete 3D representations such as depth maps, camera raymaps, or voxel grids.
  • Video models support four robotics applications: imitation-learning data generation and action prediction, reinforcement-learning dynamics and rewards modeling, policy evaluation, and visual planning.
  • 3.1 Cost-Effective Data Generation and Action Prediction in Imitation Learning: Robot actions can be inferred from generated videos using modular pose-tracking methods or end-to-end latent-action and inverse-dynamics models.
  • 3.3 Scalable Policy Evaluation: Video models offer scalable policy evaluation and can test robustness and safety in out-of-distribution environments by constructing altered scenes and safety-critical tasks.

4 Evaluating Video Models

Video-model evaluation requires metrics covering perceptual quality, temporal coherence, diversity, physical consistency, and semantic alignment. Existing benchmarks show that visual quality and temporal consistency can improve while physical-law adherence and safety remain problematic.

  • Evaluation spans perceptual quality, temporal coherence, generation diversity, physical commonsense, and semantic alignment with input prompts.
  • 4.1 Metrics for Evaluating Video Models: Frame-level metrics include PSNR, SSIM, CLIP similarity, and inception score, but pixel-based measures may miss higher-order structures relevant to human perception.
  • 4.1 Metrics for Evaluating Video Models: Spatiotemporal metrics such as FVD, KVD, and FVMD evaluate video representations using image quality, motion features, or higher-order temporal variation.
  • 4.2 Benchmarks for Evaluating Video Models: Benchmarks find that video models can produce high visual quality while violating physical consistency, instruction-following requirements, or safety guidelines.
  • 4.2 Benchmarks for Evaluating Video Models: Physical-commonsense benchmarks assess laws including optics, thermodynamics, gravity, friction, and object interactions, with scaling gains that plateau relatively quickly.

5 Open Challenges and Future Directions

The survey identifies physical inconsistency, hallucinations, and unsafe generation as barriers to trustworthy robotic use of video models, and outlines physics-aware and affordance-based directions for addressing them.

  • Open Challenges and Future Directions: The survey identifies open research challenges in robotics video-model applications and directions for trustworthy integration.The broader research agenda extends beyond robotics to other video-generation applications.
  • Hallucinations and Violations of Physics: Video models hallucinate temporally inconsistent frames and violate physical laws, limiting trustworthy use in policy evaluation and visual planning.Multi-view inputs, especially wrist-camera views, can reduce hallucinations but have limited effect on physics violations.
  • Hallucinations and Violations of Physics: Physically unrealistic solid and fluid interactions reflect missing understanding of material properties, momentum, impenetrability, and mass conservation.Prompt engineering and scaling do not adequately resolve these failures, motivating novel architectures and training techniques.
  • Hallucinations and Violations of Physics: Physics-based priors and simulations can improve physical consistency, but existing approaches rely on ad-hoc solutions that reduce generality and ease of implementation.The survey proposes natively encoding physical laws through efficient training techniques and model architectures.
  • Hallucinations and Violations of Physics: Affordance maps that encode feasible interaction regions and contact hotspots could guide video synthesis toward more physically consistent outcomes.Future work should develop effective strategies for incorporating these affordance signals during generation.

5.2 Uncertainty Quantification

Video-generation uncertainty quantification remains underexplored and difficult because spatial-temporal correlations, computational cost, and absent confidence expression undermine standard approaches.

  • Uncertainty Quantification: Uncertainty quantification for image and video generation remains largely underexplored, despite its use for assessing trustworthiness in traditional neural networks.Video models pose additional challenges because spatial and temporal relationships are complex.
  • Uncertainty Quantification: Standard Bayesian uncertainty assumptions fail for video generation because frames across timesteps are correlated rather than independent and identically distributed.This dependence complicates direct extension of conventional uncertainty methods.
  • Uncertainty Quantification: Video-generation computational costs impede ensemble-based uncertainty quantification, while models cannot express or verbalize their confidence.Consequently, inference-time black-box language-model uncertainty methods cannot be directly applied.
  • Uncertainty Quantification: S-QUBED estimates task-level semantic uncertainty for text-to-video generation, whereas C3 jointly trains video generation and latent-space uncertainty quantification.These methods represent early efforts to quantify uncertainty for controllable video generation.

5.3 Instruction Following

Video models often fail to transfer requested actions and controls into generated videos, compromising robotic data generation and imitation learning; multimodal conditioning is a proposed remedy.

  • Instruction Following: Text-conditioned video models may generate the requested agents while only partially following, or entirely omitting, the specified actions.Existing guidance mechanisms provide insufficient supervision for consistent prompt adherence.
  • Instruction Following: Video models struggle to generate high-quality text annotations and to control camera motion through prompts.These failures persist even when other video components have high fidelity.
  • Instruction Following: Requested static viewpoints can trigger training-distribution camera motion, causing non-adherence to the input prompt.Models tend to mimic camera movement present in their training videos despite instructions to avoid panning.
  • Instruction Following: Instruction and camera-control failures corrupt robot training data, degrading end-effector goal poses and limiting imitation-learning effectiveness.Static-camera assumptions support accurate 3D pose back-projection, while incorrect actions undermine expert demonstrations.
  • Future Directions: Multimodal conditioning combines language, images, or predicted future states to improve instruction following during video synthesis.Examples include interleaved language-image prompts and VLM-derived conditional embeddings.

5.4 Evaluating Video Models

Robotics needs evaluation metrics that emphasize physical consistency and predictive accuracy, while video models also require broader and more general safety evaluation and guardrails.

  • Evaluating Video Models: A unified evaluation framework is lacking because existing video metrics generally measure perceptual quality or semantic consistency rather than robotics-relevant physical consistency and predictive accuracy.Researchers therefore lack suitable metrics for assessing video models in robotic applications.
  • Evaluating Video Models: Existing benchmarks are limited by simulator environment complexity and visual quality, while newer ground-truth-free benchmarks assess quality, alignment, motion, and temporal consistency.These axes broaden evaluation but do not eliminate the need for robotics-relevant assessment.
  • Safe Content Generation: Many video models lack adequate safety guardrails and can generate crime, offensive activities, violence, or misinformation.This propensity hinders integration into sensitive real-world applications.
  • Safe Content Generation: Existing video-safety methods are often task-specific, and current benchmarks cover only a limited range of unsafe-content criteria.The survey calls for versatile guardrails and more comprehensive safety benchmarks for robotics use.

5.6 Safe Robot Interaction

Video models can help robots assess proposed actions before execution, but safe interaction remains underexplored and difficult to generalize beyond training distributions.

  • Robot safety includes physical safety, avoiding collisions, and semantic safety, avoiding commonsense-harmful situations.
  • Video models can predict outcomes of proposed robot actions, enabling safety assessment without real-world execution.
  • Existing safety methods primarily use Markovian state-based world models rather than spatiotemporal latent spaces.
  • Generalization beyond training data remains challenging in long-tail and safety-critical scenarios because world models are limited by their training distributions.

5.7 Action Estimation

Action estimation can make generated videos useful for imitation learning, but latent-action and inverse-dynamics approaches face expressiveness, data, compute, and generalization constraints.

  • Generated videos usually lack action labels, while existing action-estimation methods often fall short of imitation learning’s accuracy requirements.
  • Latent action models infer actions between frames using fixed action-code primitives, whose expressiveness depends strongly on codebook size.
  • Larger latent codebooks create training instability and higher compute costs, limiting practical scaling.
  • Inverse-dynamics models require extensive data to cover robot actions and generalize poorly outside their training distributions.
  • Small latent codebooks improve interpretability but do not scale to complex tasks requiring larger codebooks.

5.8 Long Video Generation

Robotic tasks require video world models to generate much longer horizons than current systems typically support, motivating methods for efficient long-context generation.

  • Robotic tasks often last minutes, but current video models typically generate only a few seconds.
  • Veo 3.1 generates 8-second videos, while Wan 2.5 generates 10-second videos.
  • Long-horizon generation remains an open challenge despite strong short-duration video performance.
  • Long-video architectures use frame compression, sampling optimization, or hierarchical frameworks to preserve consistency over extended horizons.
  • Long-Context Tuning is constrained by quadratic self-attention cost, whereas Mixture of Contexts uses sparse attention routing without compressing history.
  • Diffusion Forcing enables variable-horizon generation by training tokens with independent noise levels.

5.9 Data Curation Costs

Training high-fidelity video models depends on diverse, well-curated data, yet preparing such datasets requires costly preprocessing and remains constrained by dataset scale.

  • High-quality and diverse training data strongly influence fidelity, especially for text-conditioned and action-conditioned video models.
  • Common dataset problems include inaccurate captions, blurry videos, and temporally inconsistent clips with rapid shot changes.
  • Video curation typically requires splitting, filtering, and annotating candidate clips before training.
  • Recent datasets filter clips using scores for aesthetics, temporal consistency, motion fidelity, and caption descriptiveness.
  • These filtering approaches improve training-data quality but produce relatively small datasets.

5.10 Training and Inference Costs

Video models face substantial training and inference costs that constrain access and limit real-time robotics use. Research directions therefore focus on compression, caching, and accelerated sampling to improve efficiency.

  • $200k training cost for OpenSora 2.0 illustrates the substantial compute required by cost-effective state-of-the-art open-source video models.Billions of parameters and classifier-free guidance add to training overhead, limiting research access to large groups.
  • About 12 video frames per second on an NVIDIA A100 GPU highlights the slow inference that challenges real-time closed-loop visual planning.Visual planners require timely feedback during execution for robustness.
  • Spatial and temporal compression reduce costly attention operations during video generation to accelerate training and inference.Dreamer 4 sparsifies temporal attention, while OpenSora uses deep compression autoencoders for greater downsampling ratios.
  • Feature caching, shortcut models, and consistency models accelerate video synthesis while targeting temporal continuity or reduced sampling steps.Wan uses chunk-based synthesis with feature caching, while consistency models enable single-step denoising.

6 Conclusion

The survey reviews video models as embodied world models for robotics, organizing their architectures, conditioning modalities, capabilities, applications, and open challenges. It emphasizes that fine-grained spatiotemporal modeling supports physically consistent future predictions and motivates broader research and application.

  • The survey categorizes robotics applications into imitation-learning data generation and action prediction, reinforcement-learning dynamics and rewards modeling, policy evaluation, and visual planning.
  • Video models learn fine-grained spatiotemporal relationships governing real-world state evolution, which is essential for physically consistent future predictions.
  • The survey identifies prevailing architectures, conditioning modalities, key capabilities, critical challenges, and directions intended to motivate broader applications.
Loading 2601.07823v1…