RLVR-World: Training World Models with Reinforcement Learning

• RLVR-World targets task-specific accuracy and perceptual quality because MLE and teacher-forced objectives can diverge from multi-step transition prediction requirements.Source: §1 · p. 2
• RLVR-World unifies language and video world modeling as autoregressive prediction, encoding current states and actions as question tokens and next states as response tokens.Source: §1 · p. 2

Language world models

• +30.7% accuracy on text-based game state prediction and +15.1% F1 on web page state prediction demonstrate gains from RLVR fine-tuning of language world models.Source: §1 · p. 2

Video world models

• +9.2% relative improvement on LPIPS for robot manipulation trajectory prediction is achieved with only a few hundred RLVR gradient steps.Source: §1 · p. 2

• Reinforced world models are demonstrated in downstream policy evaluation and model-predictive control, motivating RLVR as a broader post-training paradigm for generative models.Source: §1 · p. 2

World models

• World models generate next states conditioned on current states and actions across text, video, sensory data, games, robotics, and autonomous driving [10] [17].Source: §2 · p. 3
• Autoregressive tokenization provides a route to unified multimodal world models, including language states, visual inputs, proprioception, and actuator trajectories [65, 8, 15, 54, 74].Source: §2 · p. 3

RL for generative models

• RLVR replaces learned reward models with verifiable rewards and has advanced language reasoning, while visual generative models have optimized measurable metrics through reinforcement learning [56] [3] [12].Source: §2 · p. 3
• RLVR-World identifies world modeling as a natural RLVR application because prediction accuracy can directly serve as a task-aligned reward across modalities.Source: §2 · p. 3

Visual tokenization

• Discrete visual tokenizers encode images into latent representations, quantize them through a codebook, and decode token maps back into images for autoregressive modeling.Source: §3 · p. 3
• Independent frame tokenization creates excessively long video sequences, motivating compressive tokenization that exploits temporal redundancy and shared context tokens.Source: §3 · p. 3

Group relative policy optimization

• GRPO samples response groups and estimates advantages relative to the group, eliminating the value function required by PPO.Source: §3 · p. 3
• GRPO retains a clipped objective with a KL-divergence penalty, adapting reinforcement-learning post-training to generative sequence models.Source: §3 · p. 3

• Figure 2 presents language models predicting verbal transitions from verbal actions and video models predicting future visual observations from action vectors.Source: Figure 2 · p. 4

4.1 Problem Formulation

• World models approximate the transition distribution p(s_t+1 | s_t−k+1:t, a_t−k+1:t), including partially observable and higher-order Markov settings.Source: §4.1 · p. 4

4.2 World Models as Sequence Modeling

• Different modalities are converted into tokens using text tokenization, learned visual tokenizers, or quantized continuous values, enabling a general decoder-only Transformer formulation.Source: §4.2 · p. 4
• Templates construct question sequences q(s, a) from current states and actions and response sequences o(s′) from next states, with the formulation extending beyond first-order transitions.Source: §4.2 · p. 4

4.3 Prediction Metrics as Verifiable Rewards

• RLVR post-training samples candidate responses, decodes them with modality-specific procedures, and compares predictions with the ground-truth next state to compute rewards.Source: §4.3 · p. 5
• The reward sign reverses metrics where lower values are better, such as visual mean squared error or perceptual loss, so the RL objective directly favors better predictions.Source: §4.3 · p. 5
• RLVR-World optimizes a prediction-process MDP rather than the original environment MDP, permits domain-specific inputs, outputs, and rewards, and is compatible with multiple RL algorithms.Source: §4.3 · p. 5

5.1 Text Game State Prediction

• The text-game benchmark contains 76,369 transitions from 31 games, with 2,954 high-quality transitions reserved for testing.The simulator maps context, state, and action to the next state, task reward, and completion indicator.Source: §5.1 · p. 5
• RLVR improves language world models on text-game transitions, with tailored rewards yielding +44.8% accuracy on unchanged and +9.6% on changed cases for the 1.5B model.The 7B model surpasses GPT-4 overall, although changed-case accuracy remains limited by base-model capacity.Source: §5.1 · p. 5

5.2 Web Page State Prediction

• Web-page prediction represents accessibility-tree changes explicitly and uses their F1 score as a verifiable RLVR reward.Exact item-change matching defines correctness, while precision and recall are combined through their harmonic mean.Source: §5.2 · p. 6
• RLVR improves LLM performance on verbal state-transition tasks beyond its established applications in mathematics and coding.This finding covers the broader language-world-model result across the evaluated tasks.Source: §5.1 · p. 5, §5.2 · p. 6

5.3 Application: Model Predictive Control for Web Agents

• Reinforced language world models enable more powerful WebArena agents through model-predictive control.The pipeline proposes candidate actions, predicts their outcomes, scores them with a value model, and executes the highest-scoring action.Source: §5.3 · pp. 6–7
• Direct comparison with WMA is unavailable because the web setup predicts precise item changes rather than natural-language descriptions.The experiments therefore compare RLVR-trained systems only with their base models.Source: §5.3 · p. 6

6.1 Setup

• Video trajectories encode observations and actions as unified token sequences, while rewards sum decoded-frame L1 and LPIPS losses over predicted steps.Single-step and multi-step prediction are evaluated with MSE, PSNR, SSIM, and LPIPS.Source: §6.1 · pp. 7–8

6.2 Main Results

• RLVR-World improves the base RT-1 model across all visual metrics, with better accuracy and perceptual quality after only hundreds of RLVR steps.Continuing multi-step pre-training for 150k additional steps still produced an LPIPS score of 14.5, substantially behind RLVR fine-tuning.Source: §6.2 · p. 8, §6.1 · p. 8
• RLVR-World achieves performance comparable to DINO-WM overall and surpasses it by a significant margin on the challenging Granular dataset.The comparison covers PushT, Rope, and Granular under single-step prediction.Source: §6.2 · p. 8, Table 4 · p. 8

6.3 Model Analysis

• Direct metric optimization reduces multi-step repetition from 48.6% to 9.9%, while metric-specific rewards generally produce the strongest performance on their corresponding metrics.An extended repetition-penalty reward maintains comparable prediction performance while eliminating repetition artifacts.Source: §6.3 · pp. 8–9
• At one-shot inference, RLVR-World can outperform the base model’s best-of-5 result, but the base model catches up and surpasses it when sampling reaches 100 candidates.This exposes a current RLVR limitation in test-time scaling.Source: §6.3 · p. 9
• Increasing GRPO group size improves convergence speed and final performance by increasing sample diversity and expanding exploration.Figure 4 also evaluates test-time sample scaling and metric-oriented optimization on RT-1.Source: §6.3 · p. 9, Figure 4 · p. 9

6.4 Application: Real2Sim Policy Evaluation

• RLVR-trained video world models improve real-world policy evaluation, yielding more accurate estimates than the base model and smaller real-to-sim discrepancies than handcrafted simulators.The application evaluates the models as scalable real-world simulators.Source: §6.4 · pp. 9–10, Figure 5 · p. 9, Figure 6 · p. 10

Discussion and Limitations

• RLVR training typically converges within hundreds of steps, so further gains require addressing bottlenecks in models, data, and algorithms.The discussion frames breaking this performance barrier as an open algorithmic challenge.Source: §7 · p. 10
• Whether RLVR improves out-of-distribution world-model generalization, especially for counterfactual reasoning over OOD actions, remains unresolved.The authors specifically call for study in sequential decision-making settings.Source: §7 · p. 10
• The video experiments use two-stage training on the same dataset, whereas general-purpose video world models could enable broader supervised-to-reinforcement post-training.The authors distinguish this limitation from their language-world-model experiments.Source: §7 · p. 10
• RLVR-World’s core insight is model-agnostic, but diffusion-model GRPO methods fall outside the current study.The authors identify broader application across model classes as future work.Source: §7 · p. 10
• Classical visual metrics align better with world modeling than MLE but do not fully capture user-intended qualities such as physical rules and temporal consistency.More sophisticated task-aligned rewards are therefore still needed.Source: §7 · p. 10

A.1 Text Game State Prediction

• The language-world-model experiments use 76,369 transitions from 31 text games, with a curated 2,954-transition test set.The dataset is ByteSized32-State-Prediction.Source: §A.1 · p. 15
• The simulator predicts JSON world-state transitions, task rewards, and completion, distinguishing unchanged actions from actions that modify the state.Changed cases are harder because the next state differs from the current state.Source: §A.1 · p. 15
• State-difference prediction outputs only objects whose properties changed, reducing formatting and output-complexity demands on the 1.5B model.This design follows the original task formulation.Source: §A.1 · p. 15
• SFT with LoRA is used to make the weak base model suitable for RL, using 4,237 reject-sampled responses and selecting the epoch-10 checkpoint.The base model initially achieved 0.08% accuracy on changed cases.Source: §A.1 · p. 15
• RLVR compares predictions with ground truth using either a binary exact-correctness reward or a task-specific reward combining all-property accuracy, changed-property accuracy, and binary correctness.The task-specific reward is weighted with α1 = 0.1, α2 = 1, and α3 = 0.2.Source: §A.1 · p. 15

A.2 Web Page State Prediction

• Web-page state prediction extracts changed accessibility-tree items, filters examples above 5,000 tokens, and evaluates exact item matches with F1-based rewards.Filtering leaves approximately 7,000 trajectories from the original 14,000, and unchanged states are represented by a special None item.Source: §A.2 · pp. 16–17

A.3 Model Predictive Control for Web Agents

• Experiments use the official WebArena environment across Shopping, CMS, Reddit, GitLab, and Mapping domains.The environment runs on an AWS EC2 instance with Docker.Source: §A.3 · p. 17
• The controller samples 20 candidate actions, retains the three most frequent, predicts each next state, and selects the action with the highest averaged value score.Each value query is repeated 20 times, with scores ranging from 1 to 5.Source: §A.3 · p. 17
• A summarization model extracts the top 10 salient predicted state changes and converts them into a natural-language transition summary for value scoring.The summary is provided together with the task objective.Source: §A.3 · p. 17
• Domain-specific web-agent performance is reported in Table 6, including relative gains from RLVR.The table marks these gains with Δ.Source: §A.3 · p. 17, Table 6 · p. 17

Dataset and implementation

• The RT-1 experiments use 87,212 trajectories with 256 × 320 observations and 13-dimensional actions, leaving 1% for testing.Additional training-process hyperparameters are listed in the referenced architecture and tokenizer tables.Source: §A.4.1 · pp. 17–18, Table 7 · p. 18, Table 8 · p. 20

Visual tokenizer

• A compressive tokenizer reduces per-frame token counts by conditioning frame encoding on a separately tokenized context frame.The per-frame tokenizer independently encodes frames, whereas the compressive tokenizer uses cross attention and n ≪ N tokens per frame.Source: §A.4.1 · p. 18

Sequence modeling formulation

• Single-step and multi-step prediction are formulated as autoregressive token sequences that alternate visual-state and action tokens.The single-step sequence predicts the next frame after a context history, while the multi-step sequence appends predicted frames and actions across the trajectory.Source: §A.4.1 · p. 19

Autoregressive transformer

• The 138M-parameter LLaMA-style Transformers are trained separately for single-step and multi-step prediction, with earlier context frames preventing information leakage.The models match GPT-2 small in size and use distinct training setups for the two prediction horizons.Source: §A.4.1 · p. 19

RLVR

• GRPO fine-tunes the pretrained Transformers with rewards defined as the negative prediction loss over generated frames and ground truth.The RLVR procedure is specified for both single-step and multi-step prediction, while visual tokenizers remain frozen during fine-tuning.Source: §A.4.1 · p. 19

Learning curves

• RLVR consistently improves the base model across PushT, Rope, and Granular, while next-token prediction tends to overfit smaller datasets and requires early stopping.RLVR continues improving validation prediction performance, indicating greater robustness and generalization in these experiments.Source: §A.4.2 · p. 21, Table 9 · p. 22

Extended analysis

• The model underperforms DINO-WM on Rope, which the authors associate with overfitting on the relatively small dataset.The analysis contrasts this with recurrent latent-state models using large-scale pretrained visual encoders.Source: §A.4.2 · p. 21
• Joint training followed by RLVR improves Rope LPIPS from 0.0208 to 0.0165.The experiment combines Rope and Granular data before fine-tuning on each individual dataset.Source: §A.4.2 · p. 22, Table 10 · p. 22
• RLVR-trained models can slightly improve performance on another task, but held-out performance later degrades because the base model lacks broad, generalizable representations.The authors present this as preliminary evidence for cross-task transfer alongside overfitting limitations.Source: §A.4.2 · p. 22, Table 11 · p. 22

Model-predictive control

• Rope and Granular experiments omit model-predictive control because the official DINO-WM code lacks the required configurations and details.This limits direct control comparisons on those two datasets.Source: §A.4.2 · p. 23
• RLVR also improves control performance on PushT to results comparable with DINO-WM in model-predictive-control evaluation.The reported control comparison follows the DINO-WM planner configuration.Source: §A.4.2 · pp. 22–23, Table 12 · p. 23

A.5 Real2Sim Policy Evaluation

• The study compares base and RLVR-World models with RT-1 checkpoints, RT-1-X, and SIMPLER-VM and SIMPLER-VA [37].RT-1 checkpoints include beginning, 15%-trained, and converged variants, while SIMPLER uses physical simulation requiring task-specific environment development.Source: §A.5 · pp. 23–24
• Evaluation covers drawer-opening and closing tasks, while pick tasks produce false positives because training data lacks failed grasp attempts.Drawer tasks include failure cases such as slipping grasps, making them more suitable for evaluating both the base model and RLVR-World.Source: §A.5 · p. 24
• The protocol generates 1,440 trajectories from 30 real-world initial frames per task using four policies, two world models, and six drawer tasks.World models predict seven future frames with sliding-window generation, and each trajectory is capped at 112 frames.Source: §A.5 · p. 24
• Human annotation replaces GPT-4o and Gemini 2.0 Flash because their success judgments varied with prompts and lacked consistent criteria across scenes.One annotator labels success or failure using only each trajectory’s final frame and task description.Source: §A.5 · p. 24
• Success rates for real2sim policy evaluation are reported in Table 13 for the evaluated methods.The evaluation compares world models against SIMPLER-based representations for policy evaluation.Source: Table 13 · p. 24, §A.5 · p. 24

B Computational Resources

• Robotic-manipulation training uses substantial compute, including 360–480 tokenizer-pretraining GPU hours, 500–530 transformer-pretraining GPU hours, and 3.5–10 hours of RLVR post-training.The experiments use A100 GPUs, accelerate for pretraining, and verl for RLVR post-training.Source: B Computational Resources · p. 25

C Broader Impact

• RLVR directly optimizing task performance may extend to visual counting, object detection, and OCR, while improving autonomous-agent simulation accuracy and reducing harmful behavior risk.Its effectiveness remains bounded by the underlying base model, so stronger foundation world models are still needed; RLVR may also reinforce autoregressive modeling by leveraging the LLM ecosystem.Source: C Broader Impact · p. 25

D Prompt Examples

• The prompt appendix specifies JSON game-state transitions, web-state prediction formats, action categories, and reasoning requirements for language world-model experiments.Text-game prompts define modified, removed, and score fields, while web prompts use accessibility trees, URLs, previous and current actions, and new/deleted/updated item sections.Source: D Prompt Examples · pp. 25, 26, 29
• The next-state prediction prompt supplies the objective, URL, observations, previous and current actions, and summarized state changes, requiring rationale followed by a formatted prediction.The prompt supports browser actions including clicks, typing, hovering, key presses, scrolling, tab management, navigation, and stopping with an answer when required.Source: D Prompt Examples · p. 31
• The web-state evaluator assesses each next webpage state against the user’s intent, rewarding progress toward the task and suggesting alternatives when the state fails or errors.It covers information seeking, site navigation, content modification, and general completion, while warning against repeated-typing corruption and overly general destinations.Source: D Prompt Examples · p. 32
• The evaluation output uses a 1–5 Likert scale, from failure or regression (1) to an optimal state directly aligned with the task goal (5).Scores 2–4 represent non-meaningful contribution, neutrality, or helpful progress, respectively.Source: D Prompt Examples · p. 32