PaperlayerView exact revision on arXiv

Source-linked AI summary

WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models

Bohai Gu, Yueyang Yuan, Taiyi Wu, Dazhao Du, Jian Liu, Xiaoyi Pang, Jie Zhang, Xiaocheng Lu, Haobin Zhong, Xiaotong Zhao, Alan Zhao, Song Guo

arXiv:2608.04964v1cs.AIcs.LG

TL;DR

Long-horizon video world models lack direct supervision for measuring accumulated transition error because arbitrary action sequences have no known future state. WorldCycle uses reversible action cycles with spatial-closure and temporal-consistency rewards, reducing state-returning drift by up to 44% and improving composite-action accuracy nearly 4× over the baseline.

  • Problem

    Long-horizon video world models lack effective supervision for directly measuring accumulated transition error because arbitrary action sequences have no available reference future state.

  • Method

    WorldCycle uses reversible action cycles with joint spatial-closure and temporal-consistency rewards to provide dense, annotation-free supervision for long-horizon transition consistency.

  • Results

    State-returning drift decreases by up to 44%, while composite-action accuracy improves nearly 4× over the baseline.

  • Takeaways & Limitations

    Reversible cycles provide a controlled way to expose and correct long-horizon transition errors, with gains extending to out-of-distribution composite actions.

  • Takeaways & Limitations

    The cycle construction does not directly apply to irreversible or partially irreversible processes that cannot recover their initial state through inverse controls.

Abstract

from arXiv · show

Interactive video world models are essential for long-horizon planning and exploration, yet they suffer from compounding errors. Post-training methods such as reinforcement learning (RL) can improve these models, but they hit a verification bottleneck: for arbitrary action sequences, no ground-truth future state exists to measure long-term drift. Our key insight is that reversible action cycles make this verification possible: a sequence composed with its inverse must analytically return to the initial state, yielding annotation-free supervision on long-horizon correctness. Building on this, we introduce WorldCycle, a self-verifiable RL framework that constructs closed action cycles and their repeated executions from ordinary action sequences, and optimizes two complementary rewards: a spatial closure reward enforcing symmetry between mirrored forward and reverse segments, and a temporal consistency reward aligning states across repeated cycle executions. These rewards force the model to learn actions as consistent state operators rather than memorized temporal patterns, and extend naturally to out-of-distribution composite action cycles that the base model handles poorly. We further release CycleBench, a diagnostic benchmark for state-returning ability under complex action structures. WorldCycle reduces state returning drift by up to 44% and lifts composite-action accuracy nearly 4x over the base model, providing a vital foundation for physically grounded world models.

Introduction · Related Work

Interactive video world models enable action-conditioned simulation for embodied interaction, planning, and learning, but long-horizon improvement is limited by missing supervision for accumulated error. WorldCycle addresses this bottleneck with reversible action cycles, self-verifiable rewards, and CycleBench for state-returning consistency.

  • Introduction: Interactive video world models generate future visual observations from an initial observation and user or agent actions, supporting embodied interaction, robot learning, visual planning, and game-like simulation.They extend generative video models into interactive simulators.
  • Introduction: Reinforcement-learning post-training improves video world models, but existing methods mainly optimize short-horizon visual quality or per-step action alignment.Long-horizon improvement lacks ground-truth future states for measuring accumulated error under arbitrary action sequences.
  • Introduction: Reversible action cycles provide annotation-free, self-verifiable supervision by requiring inverse actions to return the model to its initial state.This perspective targets state-returning failures in long-horizon video world models without requiring ground-truth trajectories.
  • Introduction: WorldCycle introduces spatial cycle-closure and temporal state-consistency rewards to reduce autoregressive drift and generalize to out-of-distribution composite action cycles.The rewards operate at the trajectory level and complement one another.
  • Introduction: 44%: WorldCycle reduces state-returning drift by up to 44% and improves composite-action accuracy nearly 4× over the baseline.CycleBench benchmarks state-returning consistency across reversible, repeated, and composite actions.
  • Related Work: Classical world models learn predictive environment dynamics for control and planning, often using compact internal or latent representations.Recent video world models instead predict environment evolution directly in pixel space using advances in video generation and visual world modeling.
  • Related Work: Recent video world-model objectives primarily pursue fidelity along a single forward direction, requiring each step to look correct and follow the current action.Long-range coherence is commonly treated as an appearance- or geometry-level property rather than state-returning correctness.

Method

WorldCycle turns reversible action cycles into annotation-free, trajectory-level verification for long-horizon video world models. It combines spatial closure and temporal consistency rewards with a curriculum and efficient rollout optimization to improve compositional dynamics beyond the base model’s training distribution.

  • Problem formulation: The learned world model composes action transformations over autoregressive rollouts, but residual errors accumulate because arbitrary long-horizon target transformations are unavailable for direct verification.This verification bottleneck forces existing post-training objectives to rely on locally verifiable proxies.
  • Reversible verification: Reversible actions form a transformation group whose trajectory followed by its inverse analytically equals identity, providing a content-independent, annotation-free verifier.The identity target is independent of scene content, initial state, and intermediate observations.
  • Cycle rewards: WorldCycle constructs closed forward-inverse cycles and jointly rewards mirrored spatial states and phase-aligned observations across repeated cycles.Spatial closure densely identifies drift within one cycle, while temporal consistency tests whether repeated returns produce consistent subsequent evolution.
  • Optimization: Training begins with spatial-only warm-up before activating both rewards, while trailing-chunk backpropagation preserves trajectory-level supervision at a per-step cost comparable to single-clip training.Joint optimization avoids catastrophic forgetting and gradient conflict between the spatial and temporal objectives.
  • Composite action generalization: WorldCycle reduces accumulated residual propagation in unseen composite combinations of translations, rotations, and their compositions, encouraging action-composition algebra rather than trajectory memorization.Composite actions lie outside the base model’s pretraining distribution, making this generalization a direct target of cycle-closure optimization.

CycleBench

CycleBench evaluates accumulated transition errors in long autoregressive rollouts without requiring ground-truth future states. It uses reversible, closed action trajectories and correspondence-based frame comparisons to measure whether generated states return to analytically identical states.

  • Benchmark purpose: CycleBench measures how transition errors accumulate over extended autoregressive rollouts when correct future states are unavailable for arbitrary open trajectories.The benchmark replaces direct long-horizon scoring with comparisons between rollout states that should coincide.
  • Benchmark construction: Reversible and closed action trajectories compose to the identity, making discrepancies between expected-to-coincide rollout states observable estimates of accumulated error.This construction provides annotation-free evaluation of state-returning behavior.
  • Evaluation metrics: CycleBench uses RoMa as an independent dense correspondence estimator and defines frame-pair distance as the mean 2D displacement of valid pixel matches.Pairs with too few valid matches receive the image diagonal as their distance.
  • Evaluation metrics: The benchmark reports three complementary metrics, including ESC↓ (Endpoint State Closure), which measures distance between generated endpoints and the corresponding initial states.The supplied passage introduces ESC↓ but truncates before describing the remaining metrics.

Experiments

Experiments show that WorldCycle improves cycle-consistency beyond scaling alone, while its complementary rewards and training schedule support long-horizon stability, composite-action generalization, and visual quality. Qualitative comparisons likewise show closer return views to the starting frame.

  • Scaling Alone Does Not Solve State Consistency: Lingbot World v2 (14B) has the highest HPSv3 quality but ESC, RPS, and RCS are 1.4–8.3× worse than WorldCycle.The result indicates that larger model capacity alone does not solve state-returning consistency.
  • Qualitative Comparison: Return-frame crops show methods ending closer to the starting view in both composite-action and simple-action settings.The qualitative comparison uses 253 frames and displays outward and return-phase keyframes for each method.
  • Reward Ablation: Removing temporal supervision raises long-term ESC from 0.163 to 0.212, while removing spatial supervision causes broader failures across settings.Temporal reward mainly supports long-range stability and composite-action generalization; spatial reward anchors approximately closing cycles.
  • Visual Quality: The full WorldCycle model achieves the highest HPSv3 score (10.42) among variants, a 16% gain over the base model (8.98).Cycle-consistency improvements do not come at the cost of visual fidelity.

Conclusion

WorldCycle uses reversible action cycles to provide dense, annotation-free supervision for long-horizon video world models. Its spatial-closure and temporal-consistency rewards promote consistent state operators and generalization to composite action cycles beyond pretraining.

  • Framework: WorldCycle turns reversible action cycles into dense, annotation-free supervision for long-horizon video world models.The framework is presented as self-verifiable reinforcement learning.
  • Learning objectives: Spatial closure and temporal state consistency rewards jointly encourage actions to behave as consistent state operators.These rewards address reward sparsity and temporal drift.
  • Generalization: The framework generalizes to composite action cycles outside the base model’s pretraining distribution.This extends its usefulness beyond the action structures represented during pretraining.
  • Evaluation: The paper introduces CycleBench as a benchmark for evaluating state-returning ability.The supplied passage identifies CycleBench as the first benchmark evaluating this capability, though the sentence is truncated.

Benchmark Details

CycleBench evaluates long-horizon state-returning consistency by constructing reversible or closed trajectories whose ideal composition is the identity. It probes accumulated transition error across distinct action structures, rollout horizons, and action complexities using complementary correspondence-based metrics.

  • Benchmark design: CycleBench constructs reversible and closed trajectories with analytically identity transformations, making discrepancies between states that should coincide observable measures of accumulated transition error.This bypasses the lack of ground-truth future states for arbitrary open action trajectories.
  • Tasks: T1 measures mirrored forward–reverse agreement, T2 tests endpoint closure on non-retracing paths, T3 measures repeated-cycle drift, and T4 probes error propagation across cascaded cycles.Together, the tasks distinguish path-wise symmetry, endpoint closure, repeated execution stability, and cross-cycle contamination.
  • Evaluation settings: The benchmark varies rollout horizon and action complexity through matched 125-, 253-, and 381-frame settings plus 125-frame composite-action rollouts.Composite controls combine motion components such as simultaneous translation and rotation and are out of domain for the base model’s pretraining distribution.
  • Metrics: CycleBench reports ESC for final closure residual, RPS for mirrored forward–reverse discrepancies, and RCS for phase-aligned drift across executions, with lower values indicating better consistency.RPS provides a dense path-wise diagnostic, while RCS captures progressive amplification during repeated or cascaded execution.
  • Metrics: CycleBench supplements state-consistency metrics with Action Accuracy and HPSv3 visual quality to check that cycle improvements preserve local action following and visual fidelity.RoMa provides an independent dense correspondence estimator, decoupling evaluation from the CoTracker-based correspondence reward used during post-training.

WorldCycle Training Algorithm

WorldCycle training constructs repeated closed action cycles from sampled hierarchy levels and evaluates EMA-model rollouts with complementary rewards. The procedure combines spatial closure, temporal consistency, action-following, and visual-quality supervision.

  • Each training step samples a hierarchy level and concatenates an action block with its inverse to construct a closed action cycle.
  • The closed action cycle is repeated to produce a long-horizon trajectory for training.
  • An EMA reference model generates independent rollouts used to compute spatial closure, temporal consistency, action-following, and visual-quality rewards.

Additional Analysis Details

WorldCycle’s post-training optimization from a WorldPlay initialization steadily improves all three cycle-consistency metrics, bringing them below their initial values.

  • Training Dynamics: All three cycle-consistency metrics decrease steadily during post-training and converge below their initial values.The analysis examines optimization dynamics when WorldCycle is initialized from WorldPlay.

VBench Video Quality Evaluation

WorldCycle’s training is evaluated for perceptual video quality across six VBench dimensions, independently of its CycleBench reward. The evaluation indicates that distribution action sequences do not degrade quality.

  • VBench Video Quality Evaluation: WorldCycle is assessed on six VBench dimensions: Aesthetic Quality, Subject Consistency, Background Consistency, Imaging Quality, Temporal Flickering, and Motion Smoothness.These dimensions provide a perceptual video-quality evaluation.
  • VBench Video Quality Evaluation: The six VBench dimensions are computed independently of the CycleBench reward, providing a complementary quality signal.This separates perceptual quality assessment from the training reward.
  • VBench Video Quality Evaluation: Distribution action sequences do not introduce quality degradation.This result supports the claim that WorldCycle training preserves perceptual video quality.

Discussion and Limitations

WorldCycle uses reversible action cycles as controlled, analytically verifiable probes for exposing and correcting long-horizon transition errors without ground-truth video supervision. However, its current cycle construction does not directly extend to irreversible or partially irreversible dynamics.

  • Discussion: Reversible cycles expose and correct long-horizon transition errors that are otherwise difficult to observe.Closed trajectories provide an analytically known composed transformation for verification.
  • Discussion: Mirrored and repeated states provide dense verification without requiring ground-truth video supervision.The framework uses analytically known closed-trajectory transformations to supervise long-horizon correctness.
  • Limitations: The current cycle construction cannot directly handle irreversible or partially irreversible processes such as deformation, contact-rich interaction, and persistent state changes.Applying inverse controls generally cannot recover the initial state in these dynamics.
  • Limitations: Extending self-verifiable supervision to irreversible processes requires constraints such as conservation laws or equivalent terminal states.The passage identifies analytically verifiable alternatives as directions for extension.

Relation to Concurrent Cycle-Based Methods

Concurrent methods also use cyclic constraints to improve video world models, but WorldCycle applies them specifically to reinforcement-learning post-training of pretrained interactive models. Its analytically known action-cycle closures provide self-verifiable, annotation-free reward feedback rather than serving only as differentiable structural regularization.

  • Concurrent methods: Cycle-World introduces reverse-prediction cycle consistency for long-horizon video generation using an auxiliary reverse model to reconstruct preceding latent chunks.The reverse model is also reused for inference-time latent correction.
  • Optimization role: Both concurrent approaches formulate cycle consistency as a differentiable loss for model training or fine-tuning.WorldCycle instead focuses on reinforcement-learning post-training of a pretrained interactive world model.
  • Optimization role: WorldCycle uses analytically known action-cycle closure relations as self-verifiable reward functions, converting sampled long-horizon rollouts into annotation-free scalar feedback.The action cycles are therefore more than structural regularizers for model training.
Loading 2608.04964v1…