Source-linked AI summary
WorldCompass: Reinforcement Learning for Long-Horizon World Models
Zehan Wang, Tengfei Wang, Haiyu Zhang, Xuhui Zuo, Junta Wu, Haoyuan Wang, Wenqiang Sun, Zhenwei Wang, Chenjie Cao, Hengshuang Zhao, Chunchao Guo, Zhou Zhao
TL;DR
Long-horizon video world models have limited post-training support, while existing RL pipelines do not directly fit their autoregressive generation. WorldCompass redesigns RL for this setting and substantially improves interaction accuracy and visual quality across varied scenarios.
Problem
Long-horizon video world models remain largely pre-training-based, while existing RL pipelines do not directly fit their autoregressive, long-sequence generation.
Method
WorldCompass redesigns RL for world models with clip-level rollouts, complementary interaction and visual-quality rewards, and negative-aware fine-tuning.
Results
WorldCompass substantially improves interaction accuracy and visual quality across durations and action complexities when post-training WorldPlay.
Takeaways & Limitations
The findings support post-training as a valuable approach for strengthening fundamental capabilities in interactive video-based world models.
Takeaways & Limitations
Reliable metrics for visual-quality drift and spatial-memory retention are lacking, leaving large-scale RL vulnerable to cumulative quality degradation in long-duration generation.
Abstract
from arXiv · showhide
This work presents WorldCompass, a novel Reinforcement Learning (RL) post-training framework for the long-horizon, interactive video-based world models, enabling them to explore the world more accurately and consistently based on interaction signals. To effectively "steer" the world model's exploration, we introduce three core innovations tailored to the autoregressive video generation paradigm: 1) Clip-level rollout Strategy: We generate and evaluate multiple samples at a single target clip, which significantly boosts rollout efficiency and provides fine-grained reward signals. 2) Complementary Reward Functions: We design reward functions for both interaction-following accuracy and visual quality, which provide direct supervision and effectively suppress reward-hacking behaviors. 3) Efficient RL Algorithm: We employ the negative-aware fine-tuning strategy coupled with various efficiency optimizations to efficiently and effectively enhance model capacity. Evaluations on the SoTA open-source world model, WorldPlay, demonstrate that WorldCompass significantly improves interaction accuracy and visual fidelity across various scenarios.
1. Introduction
Current open-source video-based world models remain largely confined to pre-training, motivating WorldCompass, an RL post-training framework for more accurate and consistent long-horizon exploration. WorldCompass redesigns RL for autoregressive, interactive generation and improves WorldPlay’s interaction accuracy and visual quality across varied scenarios.
- Current open-source video-based world models remain predominantly confined to pre-training and typically learn action following implicitly from pixel supervision.
- WorldCompass applies reinforcement learning post-training to teach long-horizon video-based world models to explore more accurately and consistently using interaction signals.
- Clip-level rollout improves rollout efficiency and reward-signal granularity while requiring models to rely on imperfect predictions, mitigating exposure bias.
- WorldCompass redesigns the RL process around autoregressive, interactive, and long-horizon generation to enable efficient training and fine-grained feedback.
- RL post-training on WorldPlay substantially improves interaction accuracy and visual quality across short- and long-term durations and basic or composite actions.
2. Related Work
Related work develops video-based world models for interactive future-state prediction, action control, and long-video generation. Existing methods largely rely on pre-training or parallel diffusion pipelines, whereas WorldCompass addresses the need for post-training RL in sequential, long-horizon autoregressive world models.
- Video-Based World Models: World models predict future states from observations and actions while adhering to physical and geometric laws, enabling interaction with environments.The Genie series demonstrates the potential of video-based world models for embodied intelligence and content creation.
- Video-Based World Models: Recent methods support long-video generation through variable-timestep autoregression, camera control through embedded signals, or combinations of both.Diffusion Forcing addresses long autoregressive clips, while other works govern camera movements using discrete or continuous controls.
- Post-Training for World Models: Pre-training methods implicitly learn action following from pixel supervision, limiting performance on action switching and complex composite actions.The proposed approach instead adds post-training with direct supervision for action fidelity and visual quality.
- RL for Generative Models: RL has progressed from autoregressive LLMs to diffusion-model post-training through preference optimization and GRPO-based on-policy methods.DeepSeek-R1 illustrates large-scale on-policy RL for autoregressive LLMs, while DiffusionDPO, Flow-GRPO, and Dance-GRPO adapt RL to diffusion models.
- RL for World Models: Existing diffusion-model RL pipelines cannot be directly applied because world models generate sequentially and often require very long-horizon autoregressive sequences.This architectural difference motivates a novel RL framework for the task.
3. WorldCompass
WorldCompass adapts reinforcement learning to long-horizon interactive video world models through clip-level rollouts, complementary rewards, and efficient policy optimization. Together, these components improve exploration consistency while suppressing reward hacking.
- Framework overview: WorldCompass addresses rollout, reward-design, and optimization challenges in autoregressive interactive video world models.The framework targets how to generate rollouts, evaluate interactive generation reliably, and optimize world modeling efficiently with RL.
- Clip-level rollout strategy: Clip-level rollouts generate G candidate clips at a target index after sampling the preceding n −1 clips once.This reuses identical historical observations, improving rollout efficiency and producing more consistent, fine-grained, comparable rewards.
- Complementary reward functions: The reward design separately evaluates interaction following and visual quality, combining action adherence with HPSv3-based text-visual alignment and aesthetics.Interaction following averages independently calculated rotation and translation accuracies, while visual quality averages HPSv3 scores from frames sampled every 4 frames.
- Complementary reward functions: The two reward functions act as mutual regularizers, balancing objectives to suppress reward hacking caused by autoregressive error accumulation.This balancing prevents the model from optimizing one objective at the expense of the other and supports more robust training.
- Efficient RL algorithm: Negative-aware fine-tuning uses rollouts from different initial noises and directly optimizes the flow matching objective to diversify camera-trajectory exploration.The method responds to SDE sampling over the same noise, which diversifies visual scenes but leaves camera movement virtually unchanged.
4. Experiments
Experiments on two WorldPlay variants and 600 test cases show that WorldCompass improves interaction following and visual quality across action types and video lengths. Ablations attribute these gains to clip-level rollouts, complementary rewards, DiffusionNFT, and efficiency optimizations.
- Evaluation Setup: WorldCompass is evaluated on HunyuanVideo-1.5-8B and Wan2.2-5B across basic and composite actions and short, medium, and long videos.The evaluation uses 600 WorldPlay test cases, with videos of approximately 125, 253, and 381 frames.
- Quantitative Results: WorldCompass substantially improves interaction following and visual quality across base-model versions, scenarios, and video lengths.Reward scores increase for both interaction following and visual quality during RL training, including on challenging action inputs.
- Quantitative Results: Action accuracy is evaluated by matching every 4th frame to its corresponding action condition, making the assessment rigorous.Errors at 10%–30% primarily reflect failure to understand or execute actions, whereas 50%–60% errors mainly involve latency during action switching.
- Ablation Studies: Clip-level rollout is crucial because sample-level rollout produces sparse, coarse rewards that weaken action-following gains and can mislead optimization.Sample-level rollout provides only minor visual-quality gains, while long videos make its reward density overly sparse.
- Ablation Studies: Complementary rewards, DiffusionNFT, and efficiency strategies address reward hacking, rollout diversity, and computational overhead.Using only the Interaction Following score improves action accuracy but degrades visual quality and can destabilize training; the efficiency strategies reduce training overhead by 50% while maintaining competitive results.
5. Conclusion
WorldCompass introduces an online reinforcement learning framework tailored to interactive, long-horizon, autoregressive world models. Its redesigned diffusion-model training pipeline includes clip-level rollouts that provide fine-grained rewards while improving rollout efficiency.
- 5. Conclusion: WorldCompass introduces a novel online reinforcement learning framework specifically tailored to world models.The framework addresses the interactive, long-horizon, and autoregressive nature of world-modeling tasks.
- 5. Conclusion: The framework redesigns the reinforcement learning training pipeline for diffusion models to address world-modeling characteristics.
- 5. Conclusion: Its clip-level rollout strategy provides fine-grained rewards while boosting rollout efficiency for long-term autoregressive video generation.
A. Limitation
WorldCompass identifies a limitation in reliably evaluating visual-quality drift and spatial-memory retention during long-form video generation. Without direct reward constraints, large-scale RL can cause cumulative quality degradation, currently mitigated through fewer iterations and a reduced learning rate.
- Reliable metrics are lacking for visual-quality drift and spatial-memory retention in long-form video generation.
- The missing evaluation signals leave rewards without direct constraints to penalize visual-quality drift.
- Large-scale RL training for long-duration autoregressive video generation therefore causes cumulative quality degradation.
B. More Qualitative Results
Additional qualitative results compare WorldCompass outputs before and after RL training across four distinct scenarios using a consistent action sequence. The videos are accompanied by reconstructed 3D scenes and camera trajectories to facilitate visualization.
- Qualitative comparisons: WorldCompass results before and after RL training are compared across multiple distinct scenarios with reconstructed 3D scenes and camera trajectories.The comparisons appear in Figures 5–8.
- Action sequence: Figures 5–8 use the same sequence: W+A moves forward-left during the first half, followed by → turning right during the second half.Each figure presents one visualization case under this action sequence.