PaperlayerView exact revision on arXiv

Source-linked AI summary

PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration

Han Wang, Zijun Wang, Shuoshuo Xue, Rui Cao, Fengjiao Chen, Xiaodan Liang, Roy Ka-Wei Lee

arXiv:2607.16602v3cs.CV

TL;DR

Action-conditioned world models have limited explicit PAV supervision and rely on expert ID action–video pairs, while OOD actions lack corresponding videos. PAVXploreRL trains a pretrained latent world model with PAV-focused rewards using ID data and video-free OOD action exploration. It improves pretrained models across benchmarks and supports more reliable policy evaluation.

  • Problem

    Existing world models provide limited explicit supervision for physical plausibility, action adherence, and visual fidelity, and rely on expert ID action–video pairs despite lacking videos for OOD actions.

    Source: §1 · pp. 1–2

  • Method

    PAVXploreRL uses a pretrained latent world model with complementary PAV rewards and trains on ID action–video pairs plus perturbed OOD actions without paired videos.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    PAVXploreRL consistently improves pretrained models, with an average gain of 5.6% across benchmarks and better PAV properties.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework improves the pretrained model's capability as a reliable action policy evaluator.

    Source: §5 · p. 9

    The base world model lacks explicit multi-view consistency, and long rollouts and reward computation make training costly and primarily suited to post-training.

    Source: §5 · pp. 9–10

Abstract

from arXiv · show

Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives-Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively referred to as PAV-while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. However, existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and generalize poorly beyond expert data. To address this, we propose PAVXploreRL, a reinforcement learning framework built on a pretrained latent world model that explicitly optimizes PAV objectives through reward-driven training. To improve action generalization, our method jointly leverages ID trajectories and noise-driven OOD action exploration, without paired video supervision. Experiments show that PAVXploreRL consistently outperforms pretrained baselines, achieving a 5.6% average gain across benchmarks and producing higher-quality PAV properties. As a policy evaluator, it also yields more reliable performance estimates and reduces the overestimation bias of prior expert-only world models such as Ctrl-World. Code: https://github.com/Social-AI-Studio/PAVXploreRL

1 INTRODUCTION

Action-conditioned world models serve as virtual environments for policy evaluation, but existing approaches have limited explicit PAV supervision and rely on expert ID action–video data. PAVXploreRL combines PAV-focused rewards with ID trajectories and video-free OOD action exploration, improving pretrained models and policy evaluation.

1 INTRODUCTION

  • Action-conditioned world models can act as virtual environments for policy evaluation, reducing the cost and risk of real-world deployment.The paper focuses on models that take robot actions as input to simulate future observations.Source: §1 · p. 1
  • Existing approaches rely primarily on pixel-level similarity or non-adapted VLM scores, with limited explicit supervision for physical plausibility, action adherence, and visual fidelity.These are the three PAV targets: plausible dynamics, faithful action following, and visually detailed outputs.Source: §1 · p. 1
  • Expert-collected ID action–video pairs predominantly show successful trajectories, while corresponding videos for unseen OOD actions are absent.This leaves OOD action outcomes difficult to integrate into standard training pipelines.Source: §1 · pp. 1–2
  • PAVXploreRL uses three complementary rewards with a pretrained latent world model and explores OOD actions by perturbing ID actions without paired video supervision.The rewards target physical consistency, action execution, and visual fidelity; the model is retrained on robotic data for multi-step prediction.Source: §1 · p. 2
  • PAVXploreRL improves pretrained models by an average of 5.6% across benchmarks and yields higher-quality PAV properties and more reliable policy evaluation.Experiments also report reduced policy overestimation after OOD action exploration.Source: §1 · p. 2

2 RELATED WORKS

Related work spans text-conditioned and action-conditioned world models, evaluation of physical, action, and visual properties, and reinforcement learning for world models. The paper positions its approach against limitations in PAV measurement and video-dependent OOD reward evaluation.

World Model

  • Action-conditioned world models take robot actions as input and predict future observations, and can serve as policy evaluators or support policy training.The related work contrasts them with text-conditioned models that function as action policies.Source: §2 · p. 2

Physical-Action-Visual Targets Evaluation

  • The paper evaluates physical plausibility with action-conditioned VJEPA-2, action adherence with embodiment segmentation, and visual fidelity with reference-free perceptual, statistical, and temporal metrics.The reference-free visual reward addresses the lack of paired videos in the OOD branch.Source: §2 · p. 3

Reinforcement Learning of World Models

  • Prior world-model RL methods either incur high memory and training costs or depend on ground-truth video rewards, limiting OOD action exploration.The paper adopts Diffusion-NFT and introduces a ground-truth-free reward based on VJEPA-2 latent future predictions.Source: §2 · p. 3

3 METHOD

PAVXploreRL adapts a pretrained video diffusion model into an action-conditioned world model and post-trains it with reward-weighted reinforcement learning. Its reward design distinguishes ID action–video pairs from OOD actions without paired videos, while the VJEPA-2 training pipeline adds embodiment-focused inputs and multi-step prediction.

3.1 WORLD MODEL ARCHITECTURE AND TRAINING

  • Action embeddings enter the temporal pathway through AdaLN and time embeddings, while Framepack compresses sampled history for long-horizon interaction.After SFT convergence, RL updates only the action encoder and LoRA parameters.Source: §3.1 · p. 3
  • Rollouts generate groups of future frames from historical frames and actions, then use trajectory-level rewards to weight DiffusionNFT denoising updates.Positive targets encourage reward-aligned paths and negative targets suppress deviations toward suboptimal paths.Source: §3.1 · pp. 3–4

3.2 REWARD FUNCTION DESIGN

  • ID rewards combine pixel, VJEPA encoder, and visual-fidelity scores, while OOD rewards without paired videos combine VJEPA predictor and visual-fidelity scores.The unified reward maps predicted frame sequences to scalar scores; group-level weights control the reward contributions.Source: §3.2 · pp. 4–5
  • OOD actions are generated by independently perturbing non-static ground-truth action segments through temporal shifts, magnitude bias, and smooth temporal noise.Human inspection indicates that the resulting trajectories deviate substantially from ground-truth actions.Source: §3.2 · pp. 4–5
  • A zero-action sequence regularizes predictor-based rewards because latent prediction can favor static but high-quality predictions.Under zero action, the predicted latent representation is intended to remain static and differ from world-model-generated videos.Source: §3.2 · p. 5
  • VJEPA rewards combine raw-frame and embodiment-focused streams, with the latter emphasizing robot-motion consistency with executed actions.The visual-fidelity reward supplements VJEPA-2 with frame-level and temporal no-reference metrics for sharpness, perceptual quality, motion consistency, structural coherence, and color stability.Source: §3.2 · p. 5

3 METHOD

  • Table 1 compares interactive long-horizon generation on Agibot and Droid validation sets using 9.6-second autoregressive trajectories conditioned on three sequential 3-step action chunks.Other baselines follow their respective training settings.Source: Table 1 · p. 6

3.3 VJEPA-2 TRAINING PIPELINE

  • VJEPA-2 training freezes its encoder, trains an action-conditioned predictor, and adds embodiment-focused frames and autoregressive accumulated loss across the predicted chunk.The pipeline first uses one-step prediction, then introduces random raw or embodiment-focused frame sampling and multi-step rollout training.Source: §3.3 · p. 6

4 EXPERIMENTS

PAVXploreRL improves the pretrained model across Agibot and DROID, while ablations and policy-evaluator tests report benefits from its reward components and more accurate real-world performance estimates.

4 EXPERIMENTS

  • The evaluation uses Agibot Alpha, with 92K trajectories across 36 tasks, and DROID, with 76K trajectories across 86 tasks.The DROID evaluation follows the Ctrl-World validation set; the Agibot test uses three sampled trajectories per task, yielding 220 clips.Source: §4 · pp. 6–7

4.2 ABLATION STUDY

  • On AgiBot, adding either the ID VJEPA encoder reward or OOD VJEPA predictor reward improves on pixel-only RL, and their combination performs best.Table 2 evaluates the reward branches and removals of individual auxiliary components after 1K RL steps.Source: Table 2 · p. 7, §4.2 · p. 8

4.3 VLA EVALUATION

  • Compared with Ctrl-World, PAVXploreRL reduces success-rate overestimation and more accurately estimates real-world performance under suboptimal-policy failures.The qualitative comparison notes that Ctrl-World can predict success when the policy fails in the real world, while PAVXploreRL reflects collisions and loose grasps.Source: Figure 4 · p. 8, §4.3 · p. 9, Figure 5 · p. 9

4.1 MAIN RESULTS

  • PAVXploreRL improves the pretrained model and achieves the best overall performance among single-view baselines across Agibot and DROID.The reported gains are particularly evident in SSIM, LPIPS, and flow consistency; RL also enables the model to surpass several baselines on DROID.Source: §4.1 · p. 8
  • On Agibot rollouts, PAVXploreRL improves physical plausibility, action adherence, and visual fidelity over the pretrained model and representative alternatives.Examples include object disappearance under occlusion, missed fine-grained cloth manipulation, and motion blur in the pretrained model; other baselines show instruction-following or long-horizon consistency failures.Source: Figure 3 · p. 7, §4.1 · p. 8
  • In a blinded evaluation of 100 randomly sampled AgiBot trajectories, participants preferred PAVXploreRL over the pretrained model across all three PAV dimensions and overall quality.The detailed evaluation protocol and results are reported in Appendix A.4.Source: §4.1 · p. 8

4.2 ABLATION STUDY

  • Removing static regularization, embodiment-focused rewards, or visual fidelity reward consistently degrades performance, supporting their complementary contributions.Further analyses of the corresponding reward weights are reported in Appendix A.6.Source: §4.2 · p. 8

4.3 VLA EVALUATION

  • The policy-evaluator study compares π0 and π0.5 against Ctrl-World on five Droid tasks, evaluating each task over 10 episodes in both the real world and learned world model.Task progress is scored by completing ordered subtasks, with each successful step contributing 1/N of total progress.Source: §4.3 · p. 9

5 CONCLUSION

PAVXploreRL is a model-agnostic RL pipeline that jointly optimizes Physical Plausibility, Action Adherence, and Visual Fidelity using in-distribution and out-of-distribution actions without paired video supervision. It improves quantitative and qualitative evaluations and action-policy evaluation, with limitations in the base model’s multi-view consistency and the computational cost of training.

5 CONCLUSION

  • The pipeline trains on in-distribution and out-of-distribution actions without paired video supervision, using a unified reward to jointly optimize the PAV targets.The framework is described as model-agnostic.Source: §5 · p. 9
  • The work focuses primarily on RL design, while the base world model does not explicitly incorporate components such as multi-view consistency.The framework is described as orthogonal to these design choices and readily extendable to future world-model architectures.Source: §5 · pp. 9–10
  • Empirical results show consistent gains in quantitative metrics and qualitative evaluations, alongside improved capability as a reliable action-policy evaluator.The paper also reports that training time and reward computation remain substantial, making the method primarily suitable as post-training.Source: §5 · p. 10
  • Long rollouts and reward computation incur substantial computational cost despite DiffusionNFT, making the method primarily suitable as a post-training method.Future work could test generality and robustness with advanced pretrained world models.Source: §5 · p. 10

ETHICS STATEMENT

The work focuses on action-conditioned video generation for robotics research and reports a blinded human evaluation without collecting personal or sensitive participant data. Generated videos should be treated as simulated predictions, not guaranteed representations of real-world outcomes, particularly in safety-critical robotics applications.

ETHICS STATEMENT

  • The work focuses on action-conditioned video generation for robotics research.It reports a blinded human evaluation and does not collect personal or sensitive participant data.Source: ETHICS STATEMENT · p. 10
  • Generated videos are simulated predictions rather than guaranteed representations of real-world outcomes, particularly in safety-critical robotics applications.The authors do not anticipate direct harmful impacts.Source: ETHICS STATEMENT · p. 10

A.1 VISUAL FIDELITY REWARD COMPONENTS

The Visual Fidelity reward combines complementary non-reference metrics to regularize spatial sharpness, perceptual realism, and temporal consistency. Its components target frame detail and naturalness as well as motion, structure, color, and edge stability over time.

A.1 VISUAL FIDELITY REWARD COMPONENTS

  • The overall visual-fidelity reward combines complementary non-reference metrics to regularize spatial sharpness, perceptual realism, and temporal consistency.The components are summarized in Table 3.Source: §A.1 · p. 15, Table 3 · p. 15
  • Laplacian sharpness discourages blur, while NIQE-inspired naturalness captures perceptual artifacts without reference supervision.Flow Distribution penalizes static collapse, excessive dynamics, and degenerate motion patterns.Source: §A.1 · p. 15
  • Temporal Gradient suppresses flickering, Interframe SSIM enforces adjacent-frame structural coherence, and Phase Coherence preserves stable spatial structure over time.Phase Coherence constrains temporal drift of Fourier phase representations.Source: §A.1 · p. 15
  • Color Stability penalizes temporal shifts in histogram-based color distributions, while Gradient Consistency reduces texture flickering and preserves edge stability.Gradient Consistency minimizes temporal variance in Sobel gradient magnitudes.Source: §A.1 · p. 15

A.2 VJEPA-2 TRAINING PIPELINE

The VJEPA-2 training pipeline first uses one-step prediction from raw frames, then extends to autoregressive multi-step prediction over horizon T. The appendix identifies this as a supplementary illustration and includes figures on validation PSNR across training steps and a blinded human evaluation.

A.2 VJEPA-2 TRAINING PIPELINE

  • Stage 1 trains VJEPA-2 on raw frames with one-step prediction; Stage 2 extends it to autoregressive multi-step prediction over horizon T.Stage 2 iteratively propagates from an initial frame using frames sampled from raw or embodiment-focused views.Source: Figure 6 · p. 15
  • Figure 7 plots validation-set PSNR over training steps during pretraining and reinforcement learning for the Agibot and Droid datasets.The supplementary illustration is identified as relating to the VJEPA training pipeline shown in Figure 6.Source: §A.2 · p. 15, Figure 6 · p. 15, Figure 7 · p. 16
  • Table 4 reports trajectory-level average preference rates for a blinded human evaluation of the pretrained model and PAVXploreRL on 100 randomly sampled AgiBot validation trajectories.Each trajectory is evaluated by two independent evaluators.Source: Table 4 · p. 16

A.3 MODEL PERFORMANCE OVER TRAINING STEPS

Pretraining converges at about 80K steps, while further SFT yields only marginal PSNR gains; the authors use 80K pretraining steps and 5K RL steps.

A.3 MODEL PERFORMANCE OVER TRAINING STEPS

  • 80K pretraining steps and 5K RL steps are selected to balance performance and efficiency after pretraining converges and further SFT gives marginal gains.The authors report PSNR on validation sets for both datasets across pretraining and RL steps.Source: §A.3 · p. 16

A.4 HUMAN EVALUATION

The section reports human evaluation of PAV quality and examines reward coefficients. Moderate VJEPA and reward-function coefficient settings perform best, with static-frame rejection identified as essential to the reward diagnostic.

A.4 HUMAN EVALUATION

  • Two independent evaluators compare pretrained and RL predictions against ground-truth futures on 100 randomly sampled trajectories across 36 AgiBot validation tasks.They rate physical plausibility, action adherence, visual fidelity, and overall quality; Table 4 reports preference rates and Cohen’s κ.Source: §A.4 · p. 16

A.5 ABLATION STUDY OF VJEPA REWARD

  • Moderate VJEPA reward weights perform best, with λVJEPA = 0.2 and αOOD = 0.1 selected for AgiBot after 1K RL steps.The authors report that excessively large values can degrade performance.Source: §A.5 · p. 16, Table 5 · p. 17

A.6 REWARD FUNCTION COEFFICIENT SELECTION

  • The coefficient-selection benchmark assumes a well-designed reward ranks high-quality videos highest and penalizes degraded ones.The authors run this offline diagnostic to avoid the high computational cost of RL training.Source: §A.6 · pp. 16–17
  • Removing λstatic makes ground-truth videos consistently score below temporally static videos; λemb and λVF further improve diagnostic accuracy, with selected weights 0.8, 0.5, and 0.1.The diagnostic compares reward rankings across 12 video variants on the AgiBot validation set.Source: Table 6 · p. 17, §A.6 · p. 17
  • The offline coefficient diagnostic treats rewards as a classifier, measuring how often the ground-truth video receives the highest reward among 12 variants.Variants include model predictions, perturbed, truncated, slowed, and static videos.Source: §A.6 · p. 17
Loading 2607.16602v3…