RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL

Yinzhou Tang; Yu Shang; Yinuo Chen; Bingwen Wei; Xin Zhang; Shu'ang Yu; Liangzhi Shi; Chao Yu; Chen Gao; Wei Wu; Yong Li

Source-linked AI summary

RoboScape-R: Unified Reward-Observation World Models for Generalizable Robotics Training via RL

Yinzhou Tang, Yu Shang, Yinuo Chen, Bingwen Wei, Xin Zhang, Shu'ang Yu, Liangzhi Shi, Chao Yu, Chen Gao, Wei Wu, Yong Li

arXiv:2512.03556v1Dec 3, 2025cs.ROcs.CV

TL;DR

Generalizable embodied policies remain difficult because IL can overfit expert trajectories and RL lacks a unified reward across diverse tasks. RoboScape-R uses a world model as an RL environment and derives endogenous rewards from learned transition dynamics, achieving an average 37.5% improvement over baselines in out-of-domain scenarios.

Problem
IL can overfit expert trajectories, while RL lacks a unified and general reward signal for multi-scene generalization.
Source: §1 · p. 1
Method
RoboScape-R uses a world model as an RL environment and derives endogenous rewards from its learned understanding of state transition dynamics.
Source: §1 · pp. 1–2, §3.1 · p. 4
Results
37.5% average performance improvement over baselines is achieved in out-of-domain scenarios.
Source: §1 · p. 2
Takeaways & Limitations
World-model-based intrinsic rewards support more generalizable policy training across diverse scenarios than exogenous reward designs.
Source: §5 · p. 8
The framework is restricted to short-duration scenarios because stable autoregressive rollouts currently reach only 300 frames with a 48-frame window.
Source: §7 · p. 11

Abstract

from arXiv · show

Achieving generalizable embodied policies remains a key challenge. Traditional policy learning paradigms, including both Imitation Learning (IL) and Reinforcement Learning (RL), struggle to cultivate generalizability across diverse scenarios. While IL policies often overfit to specific expert trajectories, RL suffers from the inherent lack of a unified and general reward signal necessary for effective multi-scene generalization. We posit that the world model is uniquely capable of serving as a universal environment proxy to address this limitation. However, current world models primarily focus on their ability to predict observations and still rely on task-specific, handcrafted reward functions, thereby failing to provide a truly general training environment. Toward this problem, we propose RoboScape-R, a framework leveraging the world model to serve as a versatile, general-purpose proxy for the embodied environment within the RL paradigm. We introduce a novel world model-based general reward mechanism that generates ''endogenous'' rewards derived from the model's intrinsic understanding of real-world state transition dynamics. Extensive experiments demonstrate that RoboScape-R effectively addresses the limitations of traditional RL methods by providing an efficient and general training environment that substantially enhances the generalization capability of embodied policies. Our approach offers critical insights into utilizing the world model as an online training strategy and achieves an average 37.5% performance improvement over baselines under out-of-domain scenarios.

1. Introduction

RoboScape-R addresses limited generalization in embodied policy learning by using a world model as a training environment with an intrinsic, universal reward. The framework targets RL’s difficulty with consistent rewards across heterogeneous tasks and reports improved out-of-domain performance.

IL often overfits expert trajectories, while RL struggles to generalize across task families and lacks a universal reward for heterogeneous environments.Source: §1 · p. 1
World models can forecast observations from states and actions, learn transition dynamics from embodied scenarios, and serve as internal simulation environments.Source: §1 · p. 1
Existing world-model RL approaches require explicit rewards and termination signals, while external reward systems inherit limitations of artificially prescribed rewards.Source: §1 · pp. 1–2
37.5% performance improvement in out-of-domain scenarios is achieved by policies trained within the RoboScape-R environment.Source: §1 · p. 2
RoboScape-R deploys a world model as the primary simulation environment for training generalizable embodied policies.Source: §1 · p. 2
RoboScape-R combines a dual-world-model pipeline for action processing and instruction-following transitions with a reward derived directly from the world model.Source: §1 · p. 2

2. Related Works

Prior work uses world models mainly for prediction, data generation, or offline policy evaluation, while RL methods generally depend on rule-based or proxy-based rewards. These reward paradigms limit online world-model training and multi-task policy generalization.

2.1. Embodied World Models

Existing world models primarily generate future observations and are used for data generation or offline policy evaluation rather than online RL environments.Source: §2.1 · p. 2
World-model RL approaches use additional reward models or manually designed rewards, which can produce unstable rewards and limited generalization.Source: §2.1 · p. 2

2.2. RL for Embodied Policy Training

RL improves robust strategy learning through environment interaction and trial and error, but many embodied methods optimize rule-based or proxy-based exogenous rewards.Source: §2.2 · p. 2

3. Methods

RoboScape-R uses a dual-world model environment to provide predicted observations, generalized rewards, and scalable policy optimization across multiple environments. Its reward combines model-generated dense and sparse signals, while training initializes diverse environments, rolls out policies, and updates them from collected trajectories.

3.1. World Model with General Reward

RoboScape-R uses an action world model for predicted observations and a text world model for reward generation through a predicted goal observation.The two models share an autoregressive Transformer-based architecture but use distinct parameters and control signals.Source: §3.1 · p. 3
The model architecture tokenizes observations, encodes actions or instructions, predicts future tokens and done signals with spatial-temporal blocks, and decodes visual observations.The text model uses a T5 instruction encoder, while the action model uses an MLP for 7-DoF actions and cross-attention for control injection.Source: §3.1 · pp. 3–4
The general reward is endogenous and unsupervised, combining dense guidance for convergence and exploration with sparse feedback for generalization and avoiding local optima.The dense term compares current and goal observations, while the sparse term is determined by the predicted done signal.Source: §3.1 · p. 4

3.2. World Model-based RL Training

Environment initialization uses the text world model to generate a goal observation from the initial observation and instruction, taking the first frame whose done signal exceeds threshold θ.This provides a predicted final-state prior for each environment.Source: §3.2 · pp. 4–5
During rollout, the action world model generates next observations and sparse rewards, dense similarity rewards are computed against each goal observation, and combined rewards are stored for policy updates.The policy is optimized from collected trajectories using a policy-gradient update.Source: §3.2 · p. 5
The world-model environment supports diverse RL algorithms and policies through general observation, reward, and done interfaces.Its multi-environment initialization allows a policy to interact with multiple predicted environmental dynamics during training.Source: §3.2 · pp. 4–5

4. Experiments

The experiments evaluate RoboScape-R across in-domain, out-of-domain, multi-task, and reward-module settings using ManiSkill and a world-model environment. Results show that the world-model environment and endogenous reward support effective training and stronger generalization than competing reward designs.

4.1. Experimental Settings

The dataset contains 80 scenes and more than 300k optimal and suboptimal trajectories covering 20 objects, two containers, two tables, and four tasks.Each trajectory includes text instructions, action sequences, and multi-view observations.Source: §4.1 · p. 5
RoboScape-R is evaluated across in-domain, out-of-domain, and multi-task settings using four manipulation tasks, with policies trained in ManiSkill or the world-model environment.The evaluation uses MLP and OpenVLA policies optimized with PPO, including Pick and Place, Push, Pull, and Move to Aim tasks.Source: §4.1 · p. 5, §4.3 · p. 7

4.3. Overall Performance

The world-model reward achieves the best success rate while nearly matching rule-based training efficiency, whereas exogenous rewards converge more slowly and perform worse.The authors attribute this to the world-model reward being unsupervised and stable, while exogenous rewards directly fit manually designed labels and become unstable.Source: Figure 2 · p. 6, §4.3 · p. 7
In-domain RL in both ManiSkill and the world-model environment improves performance, supporting the world model as an effective RL training environment.The in-domain evaluation trains in an initial scene and evaluates in the same scene with different environments.Source: Table 1 · p. 6, §4.3 · p. 7
World-model training significantly improves out-of-domain performance by exposing policies to additional environments rather than fitting only properties of the initial environment.Out-of-domain evaluation uses a new scene with the same task, including unseen objects and unseen combinations of seen objects and containers.Source: §4.2 · p. 6, §4.3 · p. 7, Figure 5 · p. 8
For multi-task training, the world-model general reward improves both Pick and Place and Move to Aim, unlike two simulator rewards that raise one task’s success rate while lowering the other’s.The authors connect this balance to using one unified reward instead of separate unaligned rewards across environments.Source: Table 2 · p. 6, §4.3 · pp. 7–8

4.4. Detailed Analysis of the Reward Module

In reward-module analysis, rule-based and world-model rewards generalize to new scenes, while embedding-based and proxy-based rewards perform worse because they lack comparable scene coverage or state grounding.Only the world-model-trained policy succeeds in the described out-of-domain trajectory visualization; the rule-based and embedding-based policies fail on different manipulation steps.Source: Figure 3 · p. 6, Figure 4 · p. 7, §4.4 · p. 8

5. Conclusion and Future Works

RoboScape-R uses the world model as an RL environment that provides intrinsic rewards, producing more generalizable policies than exogenous reward designs. The authors identify adaptation to complex real-world tasks and reducing the Sim2Real gap as future potential.

RoboScape-R uses the world model as a universal environment simulator with an intrinsic reward signal for training generalizable policies.The framework lets the world model provide the environment interaction needed for policy training.Source: §5 · p. 8
The world model-based intrinsic reward yields more generalizable policies and outperforms existing exogenous reward designs.Source: §5 · p. 8
Future work may extend the paradigm to complex real-world tasks and reduce the Sim2Real gap.Source: §5 · p. 9

6. Broader Impacts

The world model can function as a scalable RL environment that interacts with policy actions and supplies observations and unified endogenous rewards. These rewards derive from its understanding of diverse tasks and support multi-task policy generalization.

The world model interacts with policy actions while providing next-frame observations and rewards.Source: §6 · p. 11
Unified endogenous rewards derived from the world model’s understanding of diverse tasks facilitate multi-task generalization.Source: §6 · p. 11

7. Limitations

The framework is currently constrained by short-horizon rollout stability and by the assumption that the world model has learned real-world dynamics accurately.

Stable rollout is limited to 300 frames with a 48-frame window, restricting the framework to short-duration tasks.Long-range complex tasks such as folding clothes are not yet supported.Source: §7 · p. 11
The framework assumes that the world model has fully learned real-world dynamic transitions.This limitation depends on the fundamental performance of the world model itself.Source: §7 · p. 11

8. Supplemented Evaluation Results

The supplementary evaluation defines in-domain and out-of-domain settings, describes a diverse ManiSkill dataset with optimal and suboptimal trajectories, and examines world-model controllability under extreme actions. The world model responds to such actions, whereas other models may generate meaningless observations.

8.1. Task Setting for In-domain and Out-of-domain Evaluation

In-domain evaluation changes initial states within the training environment, whereas out-of-domain evaluation recombines seen objects and containers in different environments.The out-of-domain examples swap containers between object-placement tasks.Source: §8.1 · p. 11, Figure 6 · p. 11

8.2. Supplemented Description for the Self-Collected Dataset

The self-collected ManiSkill dataset covers four tasks, two tables, two containers, and twenty objects, with both optimal and suboptimal trajectories.Motion planning generates trajectories, while waypoint modifications and perturbations produce suboptimal variants.Source: §8.2 · p. 11, Figure 7 · p. 12, Figure 8 · p. 13

8.3. Evaluation for the World Model Controlability

World-model environment training requires controllable observations under actions that may exceed the discrete action distribution represented in the collected dataset.Source: §8.3 · p. 11
The evaluated world model responds to extreme unseen actions, while other world models may produce meaningless observations.The reported robustness is attributed to cross-attention-based action injection and comprehensive pretraining data.Source: §8.3 · p. 12, Figure 9 · p. 13