Source-linked AI summary
WorldReward: Reward Modeling for Camera-Conditioned World Models
Yibin Wang, Zehan Wang, Junshu Tang, Zhimin Li, Yujie Zhou, Jiazi Bu, Pengyang Ling, Feng Han, Zhixiong Zhang, Long Xing, Shengyuan Ding, Ziang Li, Cheng Jin, Yuhang Zang, Jiaqi Wang, Tianyu Pang
TL;DR
Camera-conditioned world-model rewards must relate commanded actions to coherent visual outcomes, but existing geometry- and image-based signals evaluate these properties separately and whole-video judging can dilute local evidence. WorldReward uses chunked, structured VLM pairwise judgments refined through auditing and human review, achieving the highest agreement across action, appearance, and motion dimensions and improving RL post-training outcomes.
Problem
Existing rewards separately assess trajectory execution or frame quality, while long-video judgments can miss short-lived local action evidence needed to evaluate coherent interactive generation.
Method
WorldReward divides paired videos into action-aligned chunks, reasons over structured visual evidence with a VLM, votes chunk decisions into separate preferences, and trains on refined reasoning-augmented supervision.
Results
WorldReward achieves the highest agreement with human preferences on action consistency, appearance quality, and motion quality, and improves both action execution and visual quality in HY-WorldPlay 1.5 RL post-training.
Takeaways & Limitations
Unified, localized action–video preferences provide effective reward signals for evaluating and post-training camera-conditioned world models across generation horizons.
Takeaways & Limitations
Preference-data assignment is restricted to controls supported by each model interface, with lower pair frequencies for Matrix-Game models.
Abstract
from arXiv · showhide
Camera-conditioned world models generate interactive videos in which commanded actions should induce the expected scene changes while appearance, geometry, and temporal dynamics remain coherent. Existing rewards assess these requirements separately: geometry-based rewards estimate trajectory execution but cannot judge the visual quality of the executed motion, whereas image-based rewards measure frame quality without capturing action execution or temporal dynamics. We posit that a vision-language model (VLM) offers a shared reasoning space for relating actions to their visual outcomes. However, judging a complete long video against its full action sequence creates a lengthy, noisy context in which short-lived local action evidence can be missed or diluted. We present WorldReward, a VLM-based pairwise preference reward model that unifies action-consistency and visual-quality evaluation for camera-conditioned world models. WorldReward decomposes paired videos into action-aligned chunks, organizes each chunk into structured visual evidence, and aggregates chunk-level decisions by voting into separate video-level action and visual-quality preferences. To train it, we construct a large-scale reasoning-augmented preference dataset using structured judgments generated by a frontier VLM and refined through tool-based agent auditing and targeted human review. We further introduce WorldReward-Bench, a human-annotated benchmark measuring reward-model agreement with human preferences across action consistency, appearance quality, and motion quality. WorldReward achieves the highest agreement on all three dimensions, exceeding GPT-5.5 by 3.42, 1.45, and 3.56 percentage points, respectively. When used for RL post-training of HY-WorldPlay 1.5, it consistently improves both action execution and visual quality across short- to long-term horizons.
1 Introduction
WorldReward addresses the coupled, localized, and long-horizon challenges of evaluating camera-conditioned world-model videos by using a VLM to jointly assess action consistency and visual quality. It grounds pairwise judgments in action-aligned chunks and structured evidence, then shows strong agreement with human preferences and benefits for RL post-training.
- Motivation: Camera-conditioned world-model rewards must jointly assess commanded-motion execution, appearance, geometry, and temporal dynamics.The same visual change can signal correct or incorrect execution depending on the command, while trajectory accuracy alone does not ensure visual coherence.
- Limitations of existing rewards: Geometry-based rewards measure trajectory consistency but miss temporal stability, dynamic plausibility, and generation artifacts.Image-based rewards score sampled frames independently and can overlook flickering, motion discontinuities, appearance drift, and inconsistent dynamics.
- WorldReward: WorldReward uses one VLM to derive action-consistency and visual-quality preferences from the same localized action–video evidence.This shared evaluation replaces separate geometry and image-quality systems with a unified pairwise preference model.
- WorldReward: Long video pairs are divided into temporally aligned chunks of four consecutive actions, judged from structured visual evidence, and aggregated by voting into separate global preferences.The evidence includes source-image and caption context, frame grids, and action-level first-to-last transitions.
- Data and evaluation: The training dataset combines frontier-VLM chunk reasoning, tool-using agent auditing, and targeted human calibration of paired world-model outputs.The benchmark contains 760 paired generations with independent labels for action consistency, appearance quality, and motion quality.
- Results: WorldReward achieves the highest human-preference agreement across three dimensions and improves action execution and visual quality when used for RL post-training.The RL improvements hold across short- to long-term horizons and are corroborated by GPT-5.5 and human evaluators under the same pairwise protocol.
2 Related Work
Related work spans interactive world models, RL and preference optimization for visual generation, VLM-based visual judges, and geometry-based rewards. These approaches provide foundations for interactive generation and preference learning, but the cited reward signals separately characterize camera-action geometry or frame-level visual quality.
- Camera-conditioned world models: Camera-conditioned world models extend future-observation prediction with continuous viewpoint control, interactive keyboard or mouse inputs, real-time streaming, and long-range history conditioning.This line of work includes explicit camera trajectories for viewpoint manipulation and scene exploration.
- RL post-training for visual generation: RL and preference optimization have been applied to image and video generation through policy gradients, reward backpropagation, direct preference optimization, human feedback, and self-distillation.Within flow-model RL, GRPO variants and pairwise preference rewards explore alternative policy-update strategies.
- Visual preference and reward models: Visual reward models use either discriminative scalar scoring from human preferences or generative VLM judges that compare candidates and produce multi-aspect reasoning.VLM judges can also be reinforced with agentic tool use and visual reasoning.
- Reward signals for world models: Geometry-based world-model rewards recover camera trajectories or scene structure and compare estimated motion with commanded trajectories.Related signals also use inverse dynamics or geometric and perceptual consistency, but these characterize geometric execution rather than visual quality.
3 Method
WorldReward evaluates camera-conditioned world-model videos through action-aligned chunks, using structured visual evidence to produce separate action-consistency and visual-quality preferences that are aggregated globally. Its training combines reasoning-augmented supervision, agent auditing, human calibration, and matched candidate comparisons for post-training.
- 3 Method: WorldReward evaluates whether generated scene changes follow prescribed camera/action trajectories while preserving visual faithfulness, temporal coherence, and structural integrity.It treats action execution and visual quality as coupled evaluation dimensions.
- 3 Method: Each paired video is divided into action-aligned chunks, avoiding the long, noisy context of judging a complete video and action sequence at once.The chunk-level design localizes short-lived motion evidence and links each action to its corresponding visual transition.
- 3 Method: Each chunk uses the source image, caption, paired frame-grid, and action-level panels to compare local transitions and assess temporal consistency, dynamics, artifacts, and structure.The visual-quality judgment jointly considers temporal consistency, dynamic generation quality, and artifact/structure integrity.
- 3 Method: Chunk-level action and visual-quality winners are assigned as A, B, or Tie, then aggregated by voting into separate global preferences for the complete video pair.Tie chunks favor neither candidate, and equal A/B vote counts produce a global Tie.
- 3 Method: The preference dataset uses diverse trajectories, paired world-model outputs, VLM reasoning, agent auditing, and targeted human calibration to provide supervision for separate action-control and visual-quality preferences.Among agent-revised samples, 87.0% of proposed corrections were confirmed by human reviewers.
- 3 Method: WorldReward post-training uses temporally aligned rollout clips and shared action-induced scene interpretations rather than independently computed heterogeneous reward models.Candidate clips share a common prefix and diverge only after the final frame of that prefix.
4 Experiments
Experiments evaluate WorldReward through held-out human preference agreement, generalization, post-training gains, and targeted ablations. Results show stronger alignment and improved action execution and visual quality, with structured evidence, reasoning supervision, annotation refinement, and combined rewards each contributing.
- Reward-Model Evaluation: WorldReward achieves the highest agreement with human preferences across action consistency, appearance quality, and motion quality.It surpasses proprietary VLM judges, visual preference models, and geometric trajectory estimators across the three dimensions.
- Reward-Model Evaluation: WorldReward generalizes across translation, rotation, and compound trajectories, attaining the highest action and motion agreement across trajectory groups.It also achieves the highest appearance agreement on rotation and compound trajectories, whose consecutive controls require associating multiple actions with visual changes.
- World-Model Post-Training: 1.58–2.78 points: WorldReward improves combined-action accuracy over WorldCompass across three generation horizons, while HPSv3 also increases in all six settings.Basic-action accuracy improves by 2.28–5.81 points; the longest horizon reaches HPSv3 scores of 1.02 and 3.94 versus 0.73 and 3.72 for WorldCompass.
- World-Model Post-Training: WorldReward-RL executes translations and rotations more distinctly than HY-WorldPlay 1.5 while preserving cleaner, more coherent scene structure throughout trajectories.The qualitative improvement is consistent with the action-accuracy and visual-quality gains reported quantitatively.
- Ablation and Discussion: 75.94% average agreement: agent-harness quality control raises agreement from 68.69%, and human review adds a further 1.39-point improvement.The gain from auditing concentrates in action consistency and motion quality, while human review produces the best agreement across all three dimensions.
5 Conclusion
WorldReward jointly evaluates action consistency and visual quality by reasoning over action-aligned chunks and aggregating their decisions into video-level preferences. WorldReward-Bench shows the model achieves the highest human-preference agreement across action, appearance, and motion, while RL post-training improves action execution and visual quality across horizons.
- WorldReward jointly evaluates action consistency and visual quality for camera-conditioned world models.It uses a VLM-based pairwise reward model rather than separate trajectory and frame-quality systems.
- Long videos are decomposed into action-aligned chunks, structured visual evidence is analyzed, and chunk decisions are aggregated into video-level preferences.This local-to-global design supports separate action-consistency and visual-quality preferences.
- WorldReward-Bench measures agreement with human preferences across action, appearance, and motion, where WorldReward achieves the highest agreement on all three dimensions.
- WorldReward improves both action execution and visual quality when used for RL post-training of HY-WorldPlay 1.5 across generation horizons.