PaperlayerView exact revision on arXiv

Source-linked AI summary

RISE: Self-Improving Robot Policy with Compositional World Model

Jiazhi Yang, Kunyang Lin, Jinwei Li, Wencong Zhang, Tianwei Lin, Longyan Wu, Zhizhong Su, Hao Zhao, Ya-Qin Zhang, Li Chen, Ping Luo, Xiangyu Yue, Hongyang Li

arXiv:2602.11075v2cs.RO

TL;DR

VLA policies remain brittle in complex manipulation, while physical on-policy reinforcement learning is costly and constrained. RISE addresses this gap with a Compositional World Model that generates imagined rollouts and advantages for policy improvement. Across three real-world tasks, it consistently outperforms RL and IL baselines, with reported gains exceeding 35% absolute performance.

  • Problem

    VLA models remain brittle under complex physical dynamics, while real-world on-policy reinforcement learning is limited by execution cost, safety risk, resets, and insufficient on-policy data.

  • Method

    RISE uses a Compositional World Model that separates controllable dynamics prediction from value estimation and trains robot policies through imagined rollouts and advantage conditioning.

  • Results

    RISE significantly outperforms RL and IL baselines across all three tasks, reaching 85% success on brick sorting and 95% on box closing, with a 40% margin in backpack packing.

  • Takeaways & Limitations

    RISE demonstrates that world models can serve as effective learning environments for improving policies on challenging real-world manipulation tasks.

  • Takeaways & Limitations

    RISE remains constrained by world-model accuracy and coverage, including physically implausible transitions in rare or underrepresented scenarios.

Abstract

from arXiv · show

Despite the sustained scaling on model capacity and data acquisition, Vision-Language-Action (VLA) models remain brittle in contact-rich and dynamic manipulation tasks, where minor execution deviations can compound into failures. While reinforcement learning (RL) offers a principled path to robustness, on-policy RL in the physical world is constrained by safety risk, hardware cost, and environment reset. To bridge this gap, we present RISE, a scalable framework of robotic reinforcement learning via imagination. At its core is a Compositional World Model that (i) predicts multi-view future via a controllable dynamics model, and (ii) evaluates imagined outcomes with a progress value model, producing informative advantages for the policy improvement. Such compositional design allows state and value to be tailored by best-suited yet distinct architectures and objectives. These components are integrated into a closed-loop self-improving pipeline that continuously generates imaginary rollouts, estimates advantages, and updates the policy in imaginary space without costly physical interaction. Across three challenging real-world tasks, RISE yields significant improvement over prior art, with more than +35% absolute performance increase in dynamic brick sorting, +45% for backpack packing, and +35% for box closing, respectively.

I. INTRODUCTION

RISE addresses brittle real-world manipulation by replacing costly physical on-policy interaction with imagination-based policy improvement. Its Compositional World Model separates controllable dynamics prediction from value-based evaluation to generate learning signals for self-improvement.

  • Motivation: VLA models remain brittle in complex physical dynamics because imitation learning depends on expert demonstrations and suffers from exposure bias.Small deviations from the expert manifold can produce compounding errors, motivating reinforcement learning.
  • Motivation: Real-world on-policy reinforcement learning is constrained by serial execution, manual monitoring, costly resets, and limited on-policy data.These constraints push prior methods toward offline data with substantial distribution shift.
  • Approach: RISE builds a Compositional World Model that factorizes world modeling into dynamics prediction and value estimation.The decomposition allows each component to use architectures and training objectives suited to its role.
  • Approach: RISE combines imagined dynamics, value-based advantages, and advantage-conditioned training to perform on-policy reinforcement learning in imagination.The dynamics and value modules interact in imagination space to provide policy-optimization signals.
  • Approach: The dynamics model predicts future observations from multi-view history and proposed action sequences, while the value model evaluates progress toward task completion.Advantages are computed from value differences between imagined future observations and the initial observation.
  • Evaluation: The evaluation suite covers dynamic brick sorting, backpack packing, and box closing, spanning moving-object handling, deformable manipulation, and precise control.These procedures include sorting bricks on a moving conveyor, packing clothes, and folding and tucking a box flap.

B. Reinforcement Learning

RISE formulates robot policy learning as an MDP and estimates action quality through advantages relative to policy performance. It then uses probabilistic inference to bias policy generation toward actions associated with improvement.

  • MDP Formulation: RISE models decision-making as an MDP with observations, task instructions, action sequences, horizon, and rewards.The policy generates an action sequence at each timestep and seeks to maximize expected return.
  • Advantage Estimation: The advantage function quantifies the quality of an action sequence relative to average policy performance.RISE estimates this quantity using the advantage formulation described in Eq. (2).
  • Policy Improvement: To stabilize improvement over a reference policy, RISE constructs a target action distribution by weighting reference actions according to their probability of improvement.The weighting depends on the reference policy's advantage and a temperature-like exponent β.
  • Policy Improvement: Bayes' rule converts the improvement likelihood into a density-ratio form for policy optimization.This provides the probabilistic inference formulation used to derive the target policy.
  • Policy Improvement: With β = 1, the simplified objective conditions the policy on discretized advantages to guide generation toward high-return trajectories.The reference-policy prior cancels in the derivation under this setting.

III. METHODOLOGY

RISE builds a compositional world model that separates controllable future-state prediction from value estimation, then uses imagined outcomes to produce learning signals for policy improvement. Its dynamics and value components are trained with task-specific objectives designed for action controllability, stable progress estimation, and failure sensitivity.

  • Compositional World Model: RISE separates dynamics prediction from value estimation so each world-model component can use an independently optimized architecture.The dynamics model predicts future observations, while the value model evaluates imagined states to derive advantages.
  • Controllable Dynamics Model: The dynamics model generates future visual states conditioned on historical context and candidate action chunks, supporting imagined interaction for reinforcement learning.RISE uses a pretrained video diffusion model and further action-labeled fine-tuning to improve controllability.
  • Policy Improvement: Imagined futures are evaluated to derive advantages that guide policy improvement, while the world model is used only during training and adds no inference-time computational overhead.The compositional world model provides an interactive learning environment in imaginative space.
  • Progress Value Model: RISE evaluates imagined futures with a value model that combines temporal progress regression and Temporal-Difference learning.Progress regression supplies a dense temporal signal, while TD learning incorporates successful demonstrations and failure rollouts.
  • Progress Value Model: The combined value objective uses intermediate zero rewards and terminal +1/−1 rewards to improve sensitivity to successful and failed episodes.The final objective is LV = Lprog + LTD, combining learning stability with error sensitivity.

B. Policy Warm-up on Real-world Experience

RISE first warm-starts the policy with offline real-world experience, then iteratively rolls out imagined actions and trains the behavior policy using evaluated advantages. This procedure anchors behavior to physically plausible demonstrations while enabling policy improvement through the world model.

  • Policy Warm-up on Real-world Experience: Offline warm-up anchors the policy to a physically plausible behavior distribution before on-policy improvement, reducing careless exploration later.The data includes expert demonstrations, successful and failed policy rollouts, and human-intervened corrections.
  • Self-Improving Loop: The self-improving loop alternates rollout and training stages using the compositional world model as an interactive simulator.Rollout data are generated in imagination, and the behavior policy is subsequently trained under advantage conditioning.
  • Rollout Stage: The rollout policy is prompted with an optimal advantage to infer actions with positive intent before their imagined consequences are evaluated.The dynamics model synthesizes future visual states, and the value model computes the action's actual advantage.
  • Rollout Stage: Evaluated advantages are discretized into uniform bins representing the practical utility of proposed actions in the current state.This advantage signal supports advantage-conditioned policy training on imagined rollout data.
  • Results: RISE reports superior performance to baselines across three diverse real-world tasks spanning dynamic sorting and precise packing.The supplied table caption states that success rates and scores were evaluated across all three tasks.

D. Implementation Details

RISE is evaluated against imitation- and reinforcement-learning baselines on three dexterous, long-horizon real-world tasks, with implementation and data details specified for each variant. The experiments report strong performance and ablations of offline-data mixing and modular design.

  • Experimental Setup: The evaluation uses dynamic brick sorting, backpack packing, and box closing on a dual 7-DoF AgileX robot with absolute joint control.The tasks cover dynamic interaction, deformable-object manipulation, and precise bimanual coordination.
  • Baselines: RISE is benchmarked against π0.5, DAgger, PPO, DSRL, and RECAP under closely matched compute budgets.The baselines span state-of-the-art imitation learning, online reinforcement learning, diffusion-latent RL, and advantage-conditioned offline RL.
  • Main Results: RISE achieves 85% and 95% success rates on the brick and box tasks, respectively, while yielding a 40% margin in Backpack Packing.Online adaptation with PPO and DSRL is reported as unstable, including a Dynamic Brick Sorting drop from 35% to 10%.
  • Ablation Setup: Overall performance peaks at an offline-data ratio of 0.6, indicating that balanced offline data is important for complex generalization.The ablation isolates the role of offline-data allocation during RL training.
  • Main Results: Overall, RISE significantly outperforms all RL and IL baselines across all tasks with consistently high success rates.The results include both Success Rate and Stage-wise Score.

C. Ablation Study

The ablations examine data integration and world-model design, showing that online actions and states, visual priors, task-centric dynamics, progress estimation, and temporal-difference learning each support performance. Dynamics-model comparisons also assess motion accuracy and perceptual quality against prior models.

  • Dynamics-Model Reliability: RISE produces temporally coherent and physically consistent dynamics, whereas Cosmos and Genie Envisioner exhibit distortion, blurring, and physical inconsistency.The qualitative comparison is shown against Ground Truth.
  • Dynamics-Model Reliability: RISE shows superior motion accuracy and perceptual quality across real-world tasks and the Bridge dataset.Table V evaluates PSNR, SSIM, LPIPS, FVD, and optical-flow EPE, where lower EPE indicates better motion accuracy.
  • Online Integration: Adding online actions raises success from 35% to 40%, while adding online states further raises it to 70%.The results attribute these gains to expanded action exploration and a richer generated-state training distribution.
  • Dynamics Model: Removing visual pretraining drops sorting accuracy by 32.15%, while omitting task-centric design reduces completion by 30%.These ablations support visual priors and filtering task-relevant distractions in the dynamics model.
  • Value Model: Removing progress regression lowers success by 20%, and omitting temporal-difference learning causes a 35% decline.Both components contribute to value-model evaluation of imagined outcomes.

V. RELATED WORK

Prior work uses world models and reinforcement learning to improve robot control, but abstract dynamics and physical-world interaction constraints limit scalability for challenging manipulation. RISE addresses this gap by using imagination as an online learning environment while acknowledging remaining realism and compute limitations.

  • World Models: Earlier robotic world models often used low-capacity latent dynamics that could not capture rich visual and contact dynamics required for real-world manipulation.Recent generative models renewed interest in high-fidelity observation modeling.
  • Robotic Reinforcement Learning: Simulation-based RL benefits from cheap, resettable, parallel interactions, unlike physical-world interactions that are serial, slow, and labor-intensive.These physical constraints leave prior real-world RL reliant on heavily reused off-policy data and limited online hardware interaction.
  • RISE: RISE performs on-policy reinforcement learning through imagination, replacing physical training interactions with a compositional world model that estimates states and advantages.The framework targets scalable online improvement for dynamic, deformable-object, and bimanual tasks.
  • Limitations: RISE remains constrained by the accuracy and coverage of its learned world model, which can produce physically implausible transitions in rare or underrepresented scenarios.The paper identifies uncertainty-aware imagination and explicit physical constraints as future directions.
  • Limitations: The framework shifts the main bottleneck from physical interaction to computation, while retaining a need for real-world data and further tuning of simulated-to-real balance.High-fidelity world-model training is computationally expensive, especially in compute-constrained settings.

APPENDIX

The appendix provides supplementary visualizations, experimental details, related-work comparisons, failure analysis, and additional results. It also reports that RISE outperforms competing RL alternatives and benefits from its training strategies.

  • The appendix includes additional experiments, ablations, qualitative visualizations, implementation details, related work, broader impact, licenses, and failure analysis.
  • Performance drops from high to low advantage bins, supporting diverse task performance under advantage conditioning.The evaluation compares high, neutral, and low advantage bins.
  • RISE achieves higher results than RECAP and DSRL despite their extended training or increased real-world interactions.
  • RISE improves success rate from 50% to 85%, a +35% improvement, with only 9k additional steps.RECAP saturates at 30% to 50%, while DSRL saturates at 5% to 10%.
  • Pre-training improves video-generation fidelity, while task-centric training yields sharper and more physically coherent predictions.Without these strategies, ablations show action misalignment and severe blurring.

X. REAL-WORLD EXPERIMENTAL DETAILS

The real-world experiments use fine-grained, multistage evaluation across autonomous trials and an asynchronous bimanual control framework. Task-specific scoring captures incremental progress and failure points beyond binary success.

  • Each task is decomposed into sub-goals with a total score of 10, averaged across 20 autonomous trials.This rubric addresses multistage and long-horizon planning.
  • Dynamic Brick Sorting scores manipulation robustness and classification accuracy through successful grasps and correct placement.The score is capped at 10 for completely clearing the workspace.
  • Backpack Packing and Box Closing use four milestone-based phases to identify degradation at intermediate sub-goals.The phases include interaction with deformable objects and precision-critical operations such as zipping or tab insertion.
  • The asynchronous controller executes VLA action chunks with horizon H = 50 at 30 Hz in joint space.This bridges low-frequency policy inference and high-frequency robotic control.
  • The experiments use separate human demonstrations, policy rollouts, and human corrections across the three tasks.Box Closing includes 540 human corrections in addition to demonstrations and policy rollouts.

B. Dynamics Model

The dynamics-model appendix describes multi-view action-conditioned video prediction, training and inference configurations, and comparisons against alternative policy-training strategies. The broader pipeline uses frozen dynamics and value models during policy self-improvement.

  • B. Dynamics Model: The dynamics model predicts future multi-view RGB observations conditioned on action sequences and uses Flow Matching with Logit-Normal timestep scheduling.Inputs come from top-down and bilateral wrist cameras at 192 × 256 resolution.
  • B. Dynamics Model: The model is evaluated through multi-view rollouts on real-world tasks using a bimanual platform with wrist-mounted and top-down cameras.The platform has 6 DoF per arm, 1-DoF grippers, and 30 Hz control.
  • B. Dynamics Model: The dynamics and value models remain frozen during the self-improving loop for policy optimization.
  • B. Dynamics Model: Pre-training and task-centric batching improve prediction fidelity, action adherence, motion coherence, and resistance to visual artifacts.Ablated variants exhibit action misalignment and blurred predictions.
  • B. Dynamics Model: RISE performs on-policy reinforcement learning in a learned world model rather than relying on costly off-policy real-world interactions.The world model generates new states and assigns advantages to action chunks.

XII. CONCEPTUAL COMPARISONS WITH HIGHLY-RELATED WORK

RISE uses a learned world model to support on-policy reinforcement learning through imagined interaction. Its qualitative rollouts show visual and logical consistency, while failures remain tied to task-specific physical and temporal challenges.

  • RISE enables on-policy reinforcement learning with a learned world model that generates new states and assigns advantages to action chunks.Prior methods rely on off-policy data from costly real-world interactions.
  • The world model produces high-fidelity rollouts whose reward trends distinguish expert, optimized, and suboptimal action sequences.Suboptimal placement causes reward to drop when the arm moves toward the wrong target.
  • RISE failures arise from temporal inconsistency, deformability, compliance, tight geometric tolerances, and bimanual synchronization errors.These factors produce tracking, grasping, stowing, zipper, loading, flap, and tab failures across the tasks.

XV. ADDITIONAL RELATED WORK ON VLA MODELS

VLA models integrate vision-language understanding with low-level robot control, while RISE-related evidence emphasizes imagined value prediction, controllable rollouts, and robust manipulation across diverse regimes.

  • VLA models: VLA models integrate general-purpose vision-language models with low-level robotic control, using discretized, chunked, diffusion-based, or flow-matching action generation.These models are primarily trained on teleoperated robot datasets, with simulator, wearable-device, synthetic, and internet data also contributing.
  • Model implementation: The dynamics model is built on Genie Envisioner and pretrained using the Agibot World and Galaxea public datasets.Bridge is used for some comparisons, while Cosmos-Predict2.5 serves as a baseline.
  • Value modeling: Value prediction distinguishes successful and failed real-world outcomes, while joint losses improve critical-step detection but reduce numerical stability.The qualitative ablation contrasts a progress-only variant with the jointly trained value model.
  • Imagined rollouts: Imagined rollouts can follow different actions from the same initial state, producing positive rewards for successful sorting and negative outcomes for failed sorting.The visualization contrasts placing a blue brick in the blue basket with mistakenly placing it in the yellow basket.
  • Manipulation regimes: RISE is evaluated across dynamic brick sorting, deformable backpack packing, and high-precision bimanual box closing, with inference failures involving tracking, grasping, deformability, and alignment.These tasks span moving objects, cloth manipulation, and coordinated bimanual control.
Loading 2602.11075v2…