Source-linked AI summary
EVA: Aligning Video World Models with Executable Robot Actions via Inverse Dynamics Rewards
Ruixiang Wang, Qingming Liu, Yueci Deng, Guiliang Liu, Zhen Liu, Kui Jia
TL;DR
Video world models can produce visually plausible rollouts that violate robot kinematic and embodiment constraints, creating an executability gap for IDM-decoded actions. EVA uses IDM-derived rewards to reinforcement-learn smoother, bounded video trajectories, and experiments show improved rollout quality and downstream execution. The method remains limited by its focus on kinematic smoothness rather than contact dynamics and by expensive diffusion-based generation.
Problem
Video world models lack explicit executability constraints, so visually coherent rollouts can induce infeasible or unstable robot motions when decoded by IDMs.
Method
EVA trains an IDM on real robot data and uses its action-derived rewards to reinforce smooth, embodiment-consistent video generation.
Results
EVA reduces embodiment-specific artifacts, improves action smoothness and plausibility, and improves downstream execution across RoboTwin and a real robotic platform.
Takeaways & Limitations
Action-space priors from IDM-based reward alignment can improve the executability of video world models for robotic manipulation while preserving instruction adherence.
Takeaways & Limitations
The reward models kinematic feasibility and smoothness but not contact dynamics such as forces, friction, or torques, while diffusion generation remains computationally expensive.
Abstract
from arXiv · showhide
Video generative models are increasingly used as world models for robotics, where a model generates a future visual rollout conditioned on the current observation and task instruction, and an inverse dynamics model (IDM) converts the generated frames into executable robot actions. However, current video world models lack explicit executability constraints. As a result, visually coherent rollouts may still violate rigid-body and kinematic consistency, producing unstable or infeasible control commands when decoded by an IDM. We refer to this mismatch between visual generation and physically executable control as the executability gap. While this gap can be mitigated at inference time using techniques such as rejection sampling, such approaches are inefficient due to the high cost of video generation. In this paper, we leverage the executability gap as a training signal and introduce Executable Video Alignment (EVA), a reinforcement-learning post-training framework for aligning video world models. EVA trains an inverse dynamics model on real robot trajectories and repurposes it as a reward model that evaluates generated videos through the action sequences they induce, encouraging smooth motions measured by velocity, acceleration, and jerk while penalizing actions that violate embodiment constraints. Importantly, the reward remains informative even when generated videos contain severe visual artifacts, since such artifacts typically translate into unstable or out-of-bound actions. Experiments on the RoboTwin benchmark and a real bimanual robot show that EVA reduces embodiment-specific artifacts in generated rollouts and improves downstream task execution success.
1 Introduction
Video world models can generate visually coherent but physically infeasible robot trajectories, creating an executability gap between visual generation and IDM-decoded actions. EVA addresses this gap by using IDM-derived rewards to post-train video generators, improving rollout quality and downstream execution.
- Motivation: Visually coherent rollouts can violate rigid-body and kinematic constraints, causing IDMs to produce abrupt, unstable, or out-of-bounds control signals.The paper terms this mismatch between visual generation and physically executable control the executability gap.
- Motivation: Inference-time rejection sampling can mitigate the executability gap but is inefficient because video generation is computationally expensive.
- Approach: EVA trains an inverse dynamics model on real robot data and repurposes it as a dense reward for smoothness and embodiment compliance.The reward encourages low velocity, acceleration, and jerk while penalizing out-of-bound actions.
- Approach: EVA evaluates generated videos through their implied actions, allowing reinforcement-learning post-training to align video distributions with robot-executable trajectories.
- Results: EVA produces smoother, more plausible action sequences and more realistic videos, reducing kinematic artifacts and improving downstream execution stability.
2 Related Works
Video world models support visual planning and broad task generalization, but their decoupled IDM extraction pipeline lacks physical feedback. Existing generative-model alignment mainly targets visual or semantic fidelity, whereas this work targets physical executability.
- Video World Models for Robotics: Video world models can synthesize future interaction observations for robotics and support data simulation or visual planning.
- Embodied Visuomotor Policies: Decoupling video-based future imagination from action generation may improve generalization, but open-loop IDMs cannot correct upstream visual-generation errors.
- Alignment and Post-Training: Existing reinforcement-learning and preference-optimization methods for generative models primarily improve human aesthetics, text alignment, motion quality, or visual quality.
- Alignment and Post-Training: EVA shifts the alignment target from visual and semantic fidelity toward physical executability using action-derived feedback.
3 Preliminaries
The video generator models latent video distributions with flow matching, transporting Gaussian noise to decoded videos through a learned velocity field. Flow-GRPO converts deterministic flow sampling into a stochastic trajectory distribution so group-relative policy optimization can fine-tune the model with rewards.
- Flow-matching-based Video Generation: A pretrained 3D VAE encodes pixel-space videos into compact latent representations, where generative modeling is performed for efficiency.
- Flow-matching-based Video Generation: Flow matching linearly interpolates Gaussian noise and a latent video sample, training a velocity field to approximate the transport direction.The objective minimizes the squared error between x_1 − x_0 and the predicted velocity v_θ(x_t,t,c).
- Flow-matching-based Video Generation: During inference, solving the velocity-field ODE transports noise into a latent video that is subsequently decoded by the VAE.
- Group Relative Policy Optimization: Flow-GRPO constructs a stochastic process with matching flow marginals, yielding an SDE and trajectory distribution suitable for policy optimization.
- Group Relative Policy Optimization: GRPO computes advantages from group reward statistics without learning a value function, then optimizes the drift with a clipped objective and KL regularization.
4 Method
EVA aligns pretrained video generators with executable robot behavior by scoring IDM-decoded actions for smoothness and embodiment-limit compliance. The method trains an IDM on robot trajectories, uses it to evaluate generated videos, and fine-tunes the generator with the resulting bounded reward.
- 4 Method: EVA scores generated videos through IDM-implied actions, then fine-tunes the video generator toward physically plausible robot motions.The reward favors smooth action sequences that respect kinematic limits.
- 4 Method: The IDM predicts executed robot actions from a centered temporal window of visual frames and is trained by supervised regression on robot trajectories.The context radius k determines the temporal window around each target action.
- 4 Method: The IDM uses convolutional spatial features, spatial softmax keypoint-like coordinates, and an MLP to predict actions.This representation is described as more stable than global pooling when decoding actions from generated rollouts.
- 4 Method: For a generated video, the frozen IDM produces joint commands from which velocity, acceleration, and jerk are computed by finite differences.Huber penalties are applied to acceleration and jerk to penalize non-smooth motions.
- 4 Method: The penalty additionally captures violations of robot velocity and acceleration bounds before being mapped into a bounded fine-tuning reward.The reward scale is estimated from pretrained-model rollouts, and its decay rate controls how penalties affect reward values.
5 Experiments
EVA is evaluated through embodiment-focused rollout ratings, RoboTwin task execution, real-robot deployment, and isolated IDM decoding. Across these evaluations, alignment reduces visual and kinematic artifacts, improves executable actions and task success, while remaining limited by unmodeled contact dynamics.
- Experimental Setup: The evaluation uses a latent video diffusion world model, human ratings focused on embodiment-specific artifacts, and IDM-decoded action execution on RoboTwin and a physical robot.The setup uses a Wan2.1-14B DiT backbone, supervised fine-tuning, GRPO with LoRA, 21 RoboTwin tasks, and real-robot demonstrations.
- Visual Rollout Quality: 83.8% Perfect execution is achieved in human-rated rollout evaluation, with Kinematic plausibility improving by +20.9% over EVA (w/o RL) while instruction adherence is maintained.Human raters assess kinematic plausibility, interaction plausibility, instruction adherence, and perfect execution across 210 prompts.
- Simulation Policy Execution on RoboTwin: EVA achieves the best overall RoboTwin benchmark performance, consistently improving task success over EVA (w/o RL), especially on contact-rich tasks.The benchmark evaluates IDM-decoded actions across 21 bimanual tasks; improvements are particularly pronounced for ClickBell, OpenLaptop, and TurnSwitch.
- Real-World Deployment: On the real bimanual robot, EVA improves over EVA (w/o RL) on both seen and out-of-distribution tasks, while unaligned models can produce unstable or out-of-distribution commands.The real-world evaluation covers five seen and five OOD tasks, and success requires completing the objective without safety interruption or human intervention.
- IDM Evaluation: The IDM reaches an average execution success rate of 89.52% across 21 RoboTwin tasks using ground-truth video demonstrations.This isolated evaluation supports using the IDM as a kinematic bridge and dense reward model when visual trajectories are physically valid.
- Failure Modes: Observed failure modes include implausible kinematics, wrong contact, and incorrect goals, with morphological deformation, ambiguous articulation, penetration, and missing contact among the examples.These inconsistencies can become unstable or out-of-distribution control commands during IDM decoding and lead to execution failures.
6 Conclusion
EVA treats the executability gap as a training signal by using IDM-implied actions to reward feasible video rollouts. The method reduces embodiment-specific artifacts and improves downstream execution while preserving instruction adherence.
- EVA uses an IDM-based reward to evaluate generated videos through the action sequences they imply, targeting the executability gap.The reward penalizes embodiment violations and unstable motion in decoded actions.
- The conclusion illustrates task prompting with a blue-towel wiping instruction and reports zero-shot generation on out-of-distribution tasks.
- EVA reduces embodiment-specific artifacts in generated rollouts and improves downstream execution success while preserving instruction adherence.
A Detailed Experimental Results
Experiments evaluate EVA’s reward validity and task performance across RoboTwin tasks and rollout-quality analyses. The reward is associated with artifact-free videos and successful simulated executions, while EVA improves over the unaligned model across many tasks.
- EVA remains competitive with VLA baselines while improving over the unaligned video world model across a broad range of RoboTwin tasks.Table S1 contains per-task results for all 21 tasks.
- Reward validity: The reward analyses test whether IDM-based scores capture both visual plausibility and physical executability.
- Reward validity: Artifact-free rollouts generally receive higher reward scores than rollouts with visible embodiment-related kinematic artifacts.Figure S1(a) summarizes three representative tasks with 10 rollouts per group.
- Reward validity: Successful simulated executions generally correspond to higher reward scores than failed executions across three representative tasks.Generated videos are decoded into action sequences with the IDM before simulator execution.
C Detailed Training Analysis
Training analysis covers supervised fine-tuning, GRPO alignment, reward trends, reward hacking, and IDM evaluation. The reward generally rises during alignment but can support degenerate behaviors under prolonged optimization.
- Supervised fine-tuning: The video model is first supervised-fine-tuned on 49-frame clips before GRPO alignment.Training uses separate simulation and real-world datasets and resolutions.
- Training dynamics: The training reward generally increases during simulation and real-robot alignment, providing a stable optimization signal.Figure S3 distinguishes raw reward from its exponential moving average.
- GRPO training: GRPO post-training samples eight candidate videos per prompt, updates only rank-32 LoRA parameters, and keeps the backbone frozen.
- Reward hacking: Reward hacking can emerge during prolonged GRPO optimization because the reward emphasizes smoothness and embodiment feasibility without directly enforcing task completion.Observed failures include incorrect interactions, unrealistic link lengths, and nearly static behavior.
D Ablation Study on the IDM
The IDM ablation evaluates explicit spatial modeling, while the real-world study uses a dual-arm AgileX platform across in-distribution and out-of-distribution tasks. Spatial modeling substantially improves inverse-dynamics accuracy and success.
- Ablation Study on the IDM: Replacing spatial softmax with global average pooling reduces IDM test accuracy from 0.9864 to 0.7738 and test success rate from 89.52% to 84.29%.Test accuracy counts action predictions within ±0.05 radians of ground truth.
- Real-world evaluation: Real-world experiments use a dual-arm AgileX Cobot Magic tabletop platform supporting single-arm and dual-arm manipulation.
- Real-world evaluation: The real-world task set includes both in-distribution and out-of-distribution tasks.
F Scaling Embodied Data via Zero-Shot Generation
EVA supports scalable embodied data augmentation by generating diverse synthetic scenes and video trajectories whose motion remains consistent with target embodiment constraints. Zero-shot OOD examples illustrate this generation capability.
- Aligned video rollouts can be decoded into stable and feasible actions, supporting a scalable pipeline for embodied data augmentation.The approach addresses data scarcity by generating visual rollouts suitable for downstream action decoding.
- A text-to-image generator creates diverse synthetic initial scenes, which the aligned world model expands into embodiment-consistent video trajectories without human teleoperation.This fully synthetic pipeline enables large-scale generation of diverse embodied trajectories.
- Figure S5 shows zero-shot EVA-finetuned video generation on out-of-distribution tasks, with each row pairing a conditioning image and synthesized sequence.The figure presents representative generated sequences for OOD tasks.