PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots

Chenhao Li, Andreas Krause, Marco Hutter

arXiv:2504.16680v3cs.ROcs.AIcs.LG

TL;DR

Offline MBRL promises data-efficient robotic control from fixed datasets, but distribution shift and compounding errors make long-horizon predictions unreliable, especially with noisy, biased, and partially observed real-world data. The paper combines RWM-U, an autoregressive world model with epistemic uncertainty estimation, and MOPO-PPO, uncertainty-penalized PPO for offline policy learning. Across simulated and real manipulation and locomotion tasks, the resulting policies outperform model-free and uncertainty-unaware baselines, while moderate real-data fusion surpasses simulation-only online model-free baselines.

  • Problem

    Offline MBRL must handle distribution shift and compounding long-horizon errors in noisy, biased, and partially observed robotic datasets without additional interaction.

    Source: §1 · p. 2, §4.1 · p. 4

  • Method

    The paper combines RWM-U's autoregressive world modeling and epistemic uncertainty estimation with MOPO-PPO's uncertainty-penalized PPO optimization.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Across simulated and real robotic tasks, the method consistently outperforms model-free and uncertainty-unaware model-based baselines, while real-data fusion surpasses simulation-only online model-free baselines.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Offline RL can reuse simulation and real-world robotic experience across tasks while maintaining cautious generalization beyond the dataset.

    Source: §1 · p. 2, §7 · p. 10

    Performance remains bounded by dataset coverage and long-horizon model errors, while real-world datasets may lack risky but informative transitions.

    Source: §6 · p. 10

Abstract

from arXiv · show

Reinforcement Learning (RL) has achieved impressive results in robotics, yet high-performing pipelines remain highly task-specific, with little reuse of prior data. Offline Model-based RL (MBRL) offers greater data efficiency by training policies entirely from existing datasets, but suffers from compounding errors and distribution shift in long-horizon rollouts. Although existing methods have shown success in controlled simulation benchmarks, robustly applying them to the noisy, biased, and partially observed datasets typical of real-world robotics remains challenging. We present a principled pipeline for making offline MBRL effective on physical robots. Our RWM-U extends autoregressive world models with epistemic uncertainty estimation, enabling temporally consistent multi-step rollouts with uncertainty effectively propagated over long horizons. We combine RWM-U with MOPO-PPO, which adapts uncertainty-penalized policy optimization to the stable, on-policy PPO framework for real-world control. We evaluate our approach on diverse manipulation and locomotion tasks in simulation and on real quadruped and humanoid, training policies entirely from offline datasets. The resulting policies consistently outperform model-free and uncertainty-unaware model-based baselines, and fusing real-world data in model learning further yields robust policies that surpass online model-free baselines trained solely in simulation.

1 INTRODUCTION

Offline RL can reuse fixed robotic datasets, but narrow coverage and distribution shift make long-horizon model-based learning unreliable without online correction. The paper proposes combining uncertainty-aware world modeling with uncertainty-penalized policy optimization for offline real-robot control.

  • Offline RL learns from fixed datasets, eliminating online correction while requiring dynamics modeling, rollouts, and policy optimization to remain robust to distribution mismatch.Source: §1 · p. 2
  • Biased datasets with narrow state-action coverage make out-of-support predictions unreliable, while strict model-free constraints limit generalization beyond observed transitions.Source: §1 · p. 2
  • RWM-U combines autoregressive long-horizon prediction with ensemble epistemic uncertainty, while MOPO-PPO adapts uncertainty-penalized optimization to PPO for 100-step offline rollouts.Source: §1 · p. 2
  • Across simulation and real quadruped and humanoid tasks, offline policies outperform model-free and uncertainty-unaware baselines, with real-data fusion surpassing simulation-only online model-free baselines.Source: §1 · p. 2

2 RELATED WORK

Prior MBRL work improves long-horizon planning through learned dynamics and synthetic rollouts, while uncertainty-aware methods address distribution shift mainly in controlled settings. Their reliable application to offline real-world robotics remains underexplored.

2.1 MODEL-BASED REINFORCEMENT LEARNING

  • MBRL learns predictive dynamics models to generate synthetic rollouts, progressing from Gaussian-process and local-linear models to scalable neural and latent-space world models.Source: §2.1 · p. 2
  • RWM uses self-supervised autoregressive simulation for long-horizon robotic dynamics, but its offline applicability remains underexplored because online interaction can correct model errors.Source: §2.1 · p. 3

2.2 UNCERTAINTY ESTIMATION AND DISTRIBUTION SHIFT MITIGATION

  • Bootstrap ensembles and related techniques estimate model uncertainty to mitigate distribution-shift overestimation in offline MBRL.Source: §2.2 · p. 3
  • Uncertainty-aware optimization methods penalize unreliable predictions or impose conservatism, yet reliable deployment on real robots still requires accurate long-horizon modeling and stable policy learning.Source: §2.2 · p. 3

3 PRELIMINARIES

Offline MBRL uses a learned world model to generate synthetic training rollouts without environment interaction, but long-horizon prediction errors can degrade policies. RWM provides autoregressive robotic dynamics modeling, while its lack of uncertainty estimation limits offline use.

3.1 OFFLINE MODEL-BASED REINFORCEMENT LEARNING

  • The problem is modeled as a partially observable Markov decision process with state, action, observation, transition, reward, and discount components.Source: §3.1 · p. 3
  • Offline MBRL trains from fixed transition datasets without interaction, using learned dynamics to generate synthetic rollouts beyond the observed trajectories.Source: §3.1 · p. 3
  • Out-of-distribution predictions accumulate model error over long horizons, requiring accurate dynamics and robust error handling for reliable policy optimization.Source: §3.1 · p. 3

3.2 ROBOTIC WORLD MODEL

  • RWM is a self-supervised, dual-autoregressive neural simulator that forecasts future observations over long horizons without domain-specific inductive biases.Source: §3.2 · p. 4
  • RWM supports robust forecasting and zero-shot hardware deployment, but lacks uncertainty estimation and therefore relies on online interaction to correct model errors.Source: §3.2 · p. 4

4 APPROACH

RWM-U augments autoregressive dynamics modeling with ensemble uncertainty to support reliable long-horizon rollouts in offline settings. MOPO-PPO uses that uncertainty to penalize risky imagined transitions and optimize policies toward high-confidence model regions.

4.1 UNCERTAINTY QUANTIFICATION WITH UNCERTAINTY-AWARE ROBOTIC WORLD MODEL

  • RWM-U uses ensemble-based Gaussian prediction heads to estimate aleatoric uncertainty from predicted variance and epistemic uncertainty from disagreement across ensemble means.The ensemble is applied after a shared recurrent feature extractor, while the ensemble mean supplies the predicted next observation.Source: §4.1 · p. 4, Figure 2 · p. 5
  • Autoregressive prediction feeds prior predictions back into the model, enabling uncertainty-aware trajectory forecasting with uncertainty propagated over long horizons.This addresses offline settings where model errors cannot be corrected through additional environment interaction.Source: §4.1 · pp. 4–5, Figure 3 · p. 6

4.2 POLICY OPTIMIZATION ON UNCERTAINTY-PENALIZED IMAGINATION

  • MOPO-PPO adapts uncertainty-penalized model-based policy optimization to PPO, discouraging policies from exploiting unreliable model predictions.Its training loop rolls out uncertainty-aware trajectories in imagination, computes penalized rewards, and updates the policy with PPO.Source: §4.2 · pp. 5–6
  • The uncertainty penalty encourages policies to focus on high-confidence dynamics regions while allowing cautious generalization beyond the behavioral distribution.The approach combines uncertainty-aware rollouts with trust-region optimization to improve robustness and reliability in offline MBRL.Source: §4.2 · p. 5

5 EXPERIMENTS

Across simulation and real-robot experiments, RWM-U and MOPO-PPO provide uncertainty-aware offline model-based control across autoregressive prediction, diverse tasks, and mixed real-world data. The method improves reliability when uncertainty penalties and data mixtures are appropriately calibrated.

5.1 AUTOREGRESSIVE UNCERTAINTY ESTIMATION

  • RWM-U’s epistemic uncertainty closely follows autoregressive prediction error, supporting its use as a trust metric for long-horizon policy optimization.After the model switches to fully autoregressive prediction, accumulated error increases while epistemic uncertainty tracks the same trend; aleatoric uncertainty remains low.Source: §5.1 · pp. 6–7

5.2 UNCERTAINTY-PENALIZED IMAGINATION

  • λ = 1.0 balances exploration and model reliability, whereas λ ≤0.5 causes hallucinated-rollout overfitting and λ ≥2.0 produces overly conservative behavior.Small penalties lead to unstable or failed locomotion, while large penalties restrict policies to ineffective low-uncertainty behavior.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 GENERALITY ACROSS ROBOTIC ENVIRONMENTS

  • Mixed datasets achieve the highest rewards because broader state-action coverage reduces distribution-shift effects, while Expert data can overfit specific dynamics.The evaluation spans Reach-Franka, Velocity-ANYmal-D, and Velocity-Unitree-G1 using Random, Medium, Expert, and Mixed datasets.Source: §5.3 · p. 8
  • Value-based and uncertainty-unaware baselines are especially weak in locomotion, where model overfitting can cause collisions and falls.CQL and SAC-based model-based methods remain competitive in manipulation, but MBPO and MBPO-PPO can exploit inaccurate predictions in complex locomotion tasks.Source: §5.3 · p. 8
  • MOPO-PPO consistently outperforms uncertainty-unaware baselines across robotic tasks, particularly in complex locomotion settings.Its uncertainty-aware rollouts improve policy stability by penalizing transitions where the learned model is unreliable.Source: §5.3 · pp. 7–8, Figure 5 · p. 8

5.4 OFFLINE LEARNING WITH REAL-WORLD DATA

  • Appropriate uncertainty penalization enables stable hardware locomotion, while insufficient or excessive penalties cause collisions, failure to walk, or ineffective stationary behavior.Figure 7 evaluates deployment on ANYmal D and Unitree G1 under varying uncertainty-penalty levels.Source: §5.4 · pp. 8, 10, Figure 7 · p. 9, Figure 6 · p. 9
  • 0.91 ± 0.03 is achieved on ANYmal D with 800K simulated and 200K real transitions, exceeding the data-collecting policy’s 0.79 average and online model-free simulation baseline.Moderate real-world data reduces domain shift, but excessive reliance on limited real trajectories reduces performance and increases failure rates.Source: §5.4 · pp. 8, 10, Table 1 · p. 10

6 LIMITATIONS

Offline MBRL remains limited by dataset coverage and compounding long-horizon model errors, despite uncertainty-aware training mitigating these effects.

  • Dataset coverage and compounding model errors can still degrade policy generalization, although uncertainty-aware training substantially mitigates these effects.Real-world datasets may also lack risky but informative transitions needed for world-model and policy learning.Source: §6 · p. 10
  • Simulation complements real-world data by safely supplying rare, critical transitions, creating a trade-off between realism and experience diversity.Real-world data mitigates domain shift, while simulation enriches coverage of rare experiences.Source: §6 · p. 10

7 CONCLUSION

The paper presents RWM-U with MOPO-PPO as a principled pipeline for fully offline MBRL that generalizes cautiously and performs robustly on robotic hardware.

  • RWM-U combined with MOPO-PPO addresses compounding model errors and distribution shift through epistemic uncertainty estimation and uncertainty-penalized PPO.The approach supports uncertainty-propagated long-horizon rollouts and cautious generalization beyond the dataset.Source: §7 · p. 10
  • Across simulated manipulation and locomotion tasks and real ANYmal D and Unitree G1 robots, the method consistently outperforms model-free and uncertainty-unaware model-based baselines.The authors describe this as the first successful deployment of offline MBRL on physical hardware.Source: §7 · p. 10
  • The framework is intended to provide a robust and scalable method for learning world models for complex robotic tasks.The work reports no human subjects or sensitive data and notes potential dual-use risks.Source: ETHICS STATEMENT · p. 11
  • The implementation is described as reproducible from the training and implementation details in Sections A.1–A.3, with code planned for release upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1.1 OBSERVATION AND ACTION SPACES

The appendix specifies the observation and action interfaces for the world model and velocity-tracking policy, including privileged information used to improve long-term prediction.

  • The world-model observation space includes base velocities, gravity measurements, joint positions, joint velocities, and joint torques.Source: §A.1.1 · p. 14, Table S2 · p. 14
  • Privileged information supplies an additional learning objective that implicitly embeds information needed for accurate long-term predictions.Source: §A.1.1 · p. 14, Table S3 · p. 14
  • The action space consists of joint-position targets.Source: §A.1.1 · p. 14, Table S4 · p. 14
  • The policy observation space includes base velocities, gravity measurements, velocity commands, joint positions, and joint velocities.Source: §A.1.1 · p. 14

A.1.2 REWARD FUNCTIONS

The appendix details velocity-tracking rewards, model and policy architectures, training configuration, real-world data collection, and the offline locomotion challenges motivating the design.

A.1.2 REWARD FUNCTIONS

A.2.1 RWM-U

  • RWM-U uses a GRU base with MLP heads that predict the next observation and privileged information, including its mean and standard deviation.Source: §A.2.1 · p. 16, Table S7 · p. 16

A.2.2 MOPO-PPO

A.3 TRAINING PARAMETERS

A.4 REAL-WORLD DATA COLLECTION

  • The autonomous data pipeline collects approximately one hour and 200K transitions while enforcing a safety boundary and using fallback commands when sampled poses leave it.Source: §A.4 · p. 17, Figure S8 · p. 18

A.5 DISTINCT CHALLENGES

  • Value-based off-policy methods are difficult to deploy reliably for high-dimensional locomotion, motivating model-based and on-policy formulations.Correlated proprioceptive trajectories and distribution shift can produce brittle policies with poor hardware transfer.Source: §A.5 · pp. 18–19
  • RWM-U propagates ensemble disagreement over long autoregressive trajectories, while MOPO-PPO applies uncertainty penalties over rollouts up to 100 steps.The pipeline is demonstrated on a physical quadruped and humanoid using fixed offline datasets.Source: §A.5 · p. 19

A.6 ONLINE VS. OFFLINE REINFORCEMENT LEARNING

  • Fully offline RL cannot use new interaction to correct model errors or expand coverage, so uncertainty estimation must detect underexplored states and constrain unreliable policy behavior.Source: §A.6 · pp. 19–20
Loading 2504.16680v3…