PaperlayerView exact revision on arXiv

Source-linked AI summary

Online World Modeling Enables Real-World Inverse Reinforcement Learning from Observation

Tyler Han, Bat Nemekhbold, Siyang Shen, Rohan Baijal, Richard Ebock, Harine Ravichandiran, Sanghun Jung, Kevin Huang, Byron Boots

arXiv:2602.24121v2cs.RO

TL;DR

Real-world robot learning from observation remains limited by the need for rewards, action supervision, or engineered simulations. MPAIL2 learns visual manipulation online from observation alone and demonstrates positive real-world transfer across tasks, while matched RL and Behavior Cloning baselines achieve 0% and 12% success.

  • Problem

    Real-world Inverse Reinforcement Learning from Observation remains impractical despite its potential to avoid reward shaping, action supervision, and hand-engineered simulations.

  • Method

    MPAIL2 jointly learns latent world, reward, value, and policy models online from observations and interaction without rewards, simulation, pre-training, or action supervision.

  • Results

    MPAIL2 demonstrates positive online transfer across real-world visual manipulation tasks, while matched RL and Behavior Cloning baselines reach only 0% and 12% success.

  • Takeaways & Limitations

    MPAIL2 supports scalable real-world robot learning from video-only demonstrations through self-guided online interaction and positive transfer.

  • Takeaways & Limitations

    The encoder assumes that latent state can be observed instantaneously, overlooking persistent information dynamics from hardware delays, tracking errors, and SLAM.

Abstract

from arXiv · show

Current methods in robot learning are fundamentally bottlenecked by one or more of: hand-designed rewards, simulation modeling, or action supervision (e.g. teleoperation) each requiring significant domain expertise, engineering effort, and robot-operator labor. Towards eliminating these bottlenecks, this work pursues observational learning via Inverse Reinforcement Learning from Observation (IRLfO) in which only access to task observations (e.g. video) is assumed. Due to the challenging setting and limitations of RL methods, IRLfO has thus far remained impractical for real-world robot learning. Here, we present the first IRL method to learn visual manipulation in the real world from scratch, and the first real-world demonstration of positive online transfer across visual manipulation tasks from scratch. In under 40 minutes, MPAIL2 learns pick-and-place from scratch to 82% success, where RL and BC with equal interaction and demonstration budgets reach only 0% and 12% despite their reward and action supervision. Interactive project page with training videos: https://uwrobotlearning.github.io/mpail2/

1 Introduction

The paper introduces MPAIL2, an observation-only, online IRLfO framework that learns visual manipulation directly on real robots without rewards, simulation, pre-training, or action supervision. From scratch, MPAIL2 achieves 82% pick-and-place success in under 40 minutes and enables positive online task transfer, addressing IRL’s real-world brittleness and sample-efficiency limitations.

  • Motivation and approach: MPAIL2 enables real-world IRLfO training directly on-robot, avoiding MPAIL’s dependence on thousands of rollouts and simulation-based interaction.The approach is motivated by MPAIL’s real-world robustness while removing the Real-to-Sim-to-Real bottleneck.
  • Method: MPAIL2 jointly improves encoder, dynamics, reward, value, and policy models through online off-policy latent world modeling using only observations and interaction.The framework requires no rewards, simulation, pre-training, or action supervision.
  • Experiments: Simulated and real-world experiments identify latent world modeling and planning as critical for practical real-world IRLfO.The paper also demonstrates positive online task transfer and video-only robot learning directly in the real world.
  • Evaluation: Over 96 independent real-world training runs were conducted on-robot from scratch, supporting multi-seed evaluations across real-world IRL and RL tasks.The evaluations benefit from reduced training-from-scratch times and the removal of privileged and online supervision.

2 Related Work

Related work studies learning from observation as a way to avoid hand-designed rewards and teleoperation, but practical methods remain data-, supervision-, and interaction-intensive. This work therefore investigates latent world modeling and online planning to improve real-world (I)RL sample efficiency and robustness.

  • Learning from Observation: LfO theoretically avoids hand-designed rewards and teleoperation but often requires extensive prior data, direct action supervision, or lengthy demonstrations.Such methods can also produce large models dependent on domain knowledge and embodiment-specific data.
  • Inverse Reinforcement Learning from Observation: IRLfO reduces demonstration quantity by replacing direct supervision with self-supervised RL-like exploration and interaction, while jointly optimizing a reward function and policy.This joint optimization worsens the interaction-efficiency limitations of existing real-world RL algorithms.
  • World Modeling & Online Planning: Latent world modeling and online planning are investigated as sample-efficiency tools because planning-based RL performs efficiently in simulation and is more robust than policy networks for real-world robot learning.The passage also notes a theoretical unification of planning-based methods and diffusion-based BC as implicit policies.

3 Model Predictive Adversarial Imitation Learning 2

MPAIL2 addresses observation-only IRL in a partially observable setting without action or reward supervision by replacing observation-based, on-policy learning with latent world modeling and off-policy optimization. Its encoder, dynamics, reward, value, multi-step policy, and MPPI planner jointly support sample-efficient online planning and transfer.

  • Problem: The learner observes trajectories from an optimal expert policy in a POMDP while state transitions, rewards, terminal states, and actions remain unknown.The formulation assumes only observation trajectories and no action or reward supervision.
  • Approach: MPAIL2 replaces observation-based dynamics and on-policy reward/value optimization with latent world modeling and off-policy objectives to improve real-world sample efficiency.The method introduces an encoder, latent dynamics model, multi-step policy, and corresponding off-policy architectures.
  • Encoder & Dynamics: The encoder and dynamics learn task-agnostic latent representations self-supervised from observed transitions, making reward-independent representations that support online planning and transfer.The latent prediction loss depends only on observed transitions and excludes reward and value heads.
  • Inferred Reward: Off-policy inferred-reward training preserves coverage across previous interactions, stabilizes reward learning, generalizes across states, and supports reliable evaluation of sampled online plans.The method addresses the failure mode in which on-policy reward learning converges to a constant.
  • Value, Multi-step Policy & Planner: A Q-ensemble and off-policy λ-return value objective support recovery, demonstration generalization, beyond-horizon reasoning, and stable policy updates, while the multi-step policy warm-starts MPPI planning.The policy receives supervision through inferred reward and value rather than expert action labels.

4 Experiments

Experiments evaluate MPAIL2 and baselines across simulated and real-world manipulation tasks, finding that world modeling, planning, and off-policy optimization enable practical IRLfO. MPAIL2 also competes with more-supervised methods, improves beyond demonstrations through interaction, transfers positively, and supports video-only demonstrations.

  • Experiment Setup: Experiments evaluate Block Push and Mug-on-Plate on Franka and Pick-and-Place on Kinova Gen3, using human-operated demonstrations and programmatic random resets.Results report Best and Last checkpoints, with real-world evaluations across two independent robot platforms.
  • World Modeling and Planning: Latent dynamics modeling enables real-world IRLfO: MPAIL2 and [−P] are the only evaluated IRL methods with any real-world training successes.In simulation, MPAIL2 and [−P] also succeed 2–3× earlier than other methods, while off-policy training is identified as critical.
  • World Modeling and Planning: Planning mitigates adversarial IRL instability and improves actor robustness, while MPAIL2 shows little to no instability compared with [−P].Simulation curves exhibit decaying or inconsistent slopes associated with approximate drops in success rate; [−P] tends to drop without resurgence.
  • Scaling Real-World Learning: MPAIL2 learns beyond demonstrations through online interaction and can learn from video-only demonstrations using rewards conditioned on shared sensing modalities.Its planner can disregard policy output when it conflicts with reward and value, mitigating plasticity-loss effects during transfer.
  • IRLfO vs. RL/BC: MPAIL2 and its IRLfO ablations significantly outperform or compete with RLPD and BC despite receiving neither rewards nor action supervision.RLPD receives action supervision and dense reward or online hand-labeling, while BC receives action supervision; these signals are absent from IRL methods.
  • Scaling Real-World Learning: 90% success is achieved after 60 episodes (20 minutes) with transferred Block Push (rt.) weights, versus 88% after 140 episodes (50 minutes) from scratch.The transferred model reaches its highest performing model in less training, indicating positive representation transfer.

5 Conclusion

MPAIL2 enables real-world inverse reinforcement learning from observation, learning visual manipulation tasks from scratch in under 40 minutes without reward, action supervision, or simulation. It outperforms RL and behavioral-cloning baselines through self-guided online interaction.

  • Contribution: MPAIL2 enables real-world IRLfO for visual manipulation tasks from scratch in under 40 minutes using observation alone.The method requires neither reward nor action supervision nor simulation.
  • Contribution: MPAIL2 and its ablated IRL baselines do not suffer performance loss from reduced supervision and prior information compared with RL and BC.This finding addresses the historical impracticality of real-world IRL learning efficiency and robustness.
  • Contribution: MPAIL2 strongly outperforms RL and BC baselines by generalizing through self-guided online interaction.The conclusion identifies online interaction as the mechanism supporting performance beyond learning efficiency.

6 Limitations and Future Work … A.1.3 Planning

The paper identifies limitations in instantaneous latent-state observation and points to pre-training, recurrent observers, and probabilistic modeling as future directions. The appendix documents MPAIL2’s training, gradient penalty, and planning procedures, including MPPI-style planning with learned models.

  • 6 Limitations and Future Work: MPAIL2’s task-dependent components may benefit from pre-training, motivating integration with pre-trained world, policy, and reward models for learning and planning from observation.The passage also notes prior work demonstrating cross-embodiment generalization, but the supplied text is truncated before details are given.
  • 6 Limitations and Future Work: The encoder assumes that latent state can be observed instantaneously, which does not reflect hardware delay, PID tracking errors, and SLAM-related temporal information dynamics.These discrepancies persist through time in real-world robotics.
  • 6 Limitations and Future Work: Recurrent state-based observers and probabilistic modeling are proposed as potential approaches for resolving temporal observation discrepancies.The proposed directions address information dynamics that persist through time.
  • Appendix Contents: The appendix organizes MPAIL2’s algorithm and architecture, experiment setup, implementation, additional results, and additional discussion into sections A–E.The contents list includes training, planning, checkpoint evaluation, task setups, hyperparameters, baselines, world-model analysis, learning curves, and areas of improvement.
  • A.1.1 Training: MPAIL2 training repeatedly interacts using the planner, samples trajectories and task observations, updates encoder and dynamics, and updates the inferred reward.The algorithm uses a replay buffer and performs updates per episode.
  • A.1.2 Gradient Penalty: The gradient penalty uses uniform linear interpolations between expert latents eω(D) and buffer latents eω(B).The interpolated latent pairs are denoted (z̃, z̃′).
  • A.1.3 Planning: The planning algorithm largely follows MPPI, replacing its standard components with learned dynamics, reward, and value models.The objective is defined over candidate action plans.
  • A.1.3 Planning: Predicted rewards r̂ and values q̂ estimate the return of each plan τ̂(a).These predictions are used within the planning objective.

A.2 MPAIL2 Architecture · B Experiment Setup

MPAIL2 learns latent visual dynamics and contrastive transition rewards, then uses model-based planning and value estimation for actor-critic updates. Experiments target visual manipulation because planning-based IRL’s benefits for real-world object interactions remain unclear, while simulation, rewards, and action supervision are practical bottlenecks.

  • A.2 MPAIL2 Architecture: The encoder maps observations to latent states, while autoregressive dynamics predict future latent states from action inputs.Training minimizes a multi-step MSE loss against stopgradient targets from fresh encoder passes on subsequent observations.
  • A.2 MPAIL2 Architecture: The reward model minimizes agent transition scores and maximizes demonstration transition scores in latent space.A gradient penalty regularizes the reward over interpolated latent pairs.
  • A.2 MPAIL2 Architecture: MPAIL2 unrolls encoder, policy, dynamics, reward, and value components over horizon H to compute the λ-return for actor-critic updates.The policy produces action sequences, dynamics predict next latent states, rewards score transitions, and Q estimates returns.
  • A.2 MPAIL2 Architecture: The planner samples trajectory plans, estimates returns through model rollouts, and retains elite samples for the next sampling distribution.Non-elite samples receive zero weight, while the elite cutoff is given by the Kth largest return.
  • A.2 MPAIL2 Architecture: The reported implementation contains 7.1M parameters for an observation space with two RGB images and proprioception.The parameter summary reports 7114494 total parameters across the encoder, dynamics, reward, value, and sampling components.
  • B Experiment Setup: Visual manipulation is used to test whether planning benefits extend beyond self-prediction to real-world dynamics such as object interactions.The setup also treats simulation, rewards, and action supervision as practical bottlenecks for IRLfO.

B.1 Checkpoint Evaluation Results … B.5 Video-only Demonstration Setup

The appendix reports checkpoint evaluations alongside real-world setups for block pushing, pick-and-place, mug-on-plate, and video-only demonstrations. It specifies task instrumentation, demonstration collection, success criteria, reward limitations, and architectural modifications.

  • B.1 Checkpoint Evaluation Results: Table 3 reports the full checkpoint history by environment steps in simulation, with offline BC (Diffusion) performance constant across interaction budgets.The corresponding real-world results are reported in Table 4.
  • B.2 Real-World Block Push Setup: Block Push uses a Franka arm with fixed-table and wrist RGB cameras plus proprioception, while actions are defined as end-effector position velocities.The setup uses 64 × 64 RGB images and calibrated block positions from AprilTags.
  • B.2 Real-World Block Push Setup: 10 space-mouse demonstrations provide 1,451 transitions for Block Push, where success is credited when the block crosses the target y-threshold.Dense reward is available to RLPD from camera-calibrated block positions.
  • B.3 Real-World Pick-and-Place Setup: 10 keyboard-teleoperated Pick-and-Place demonstrations contain 1,025 transitions, while RLPD lacks dense rewards and rarely receives reward within the limited interaction budget.Real-world evaluation hand-labels four ordered stages: initialized, lifted, transported, and placed.
  • B.4 Real-World Mug-on-Plate Setup: Mug-on-Plate uses a Franka arm with the Block Push observation space, bounded end-effector position-velocity and gripper actions, and 15 teleoperated episodes totaling 1,802 transitions.Success means the mug rests stably on the plate; no automatic object tracking or dense RLPD reward is available.
  • B.5 Video-only Demonstration Setup: The video-only demonstration task replaces the usual fused encoder with a substate reward encoder, with external video represented as oext.The figures document the fused encoder used elsewhere and the substate reward encoder used for video-only demonstrations.
  • B.5 Video-only Demonstration Setup: Video-only Block Push experiments show that dynamics can inherit the fusion process while keeping latent representations compartmentalized, enabling reward on a subset of the latent state.This design supports scalable demonstration modalities such as third-person video.

C Implementation … D.2 Learning Curves and Ablations

The paper details MPAIL2’s implementation, hyperparameters, and baseline variants, then outlines evaluation tasks and additional learning-curve, ablation, efficiency, and stability analyses.

  • C.1 Hyperparameters: MPAIL2 uses shared hyperparameters across experiments unless otherwise specified, and only the reward model omits LayerNorm.
  • C.2 Baseline Implementations: [−P] (MAIRL) removes online planning and directly samples actions from a single-step policy while retaining equivalent encoder, dynamics, reward, and value losses.Multi-step losses are computed through autoregressive policy inference where applicable.
  • C.2 Baseline Implementations: [−PM] (DAC) removes online planning and model-based components, using latent observation encoding updated through value-target loss and a scaled learning rate.The authors note these modifications are not necessarily equivalent to MPAIL2.
  • C.2 Baseline Implementations: [−PMO] (AIRL) further restricts [−PM] replay sampling to interactions from the previous episode, while [−O] (MPAIL) replaces selected off-policy updates with an on-policy rollout buffer.For [−O], encoder and latent-dynamics updates retain off-policy JEP replay, while other MPAIL2 components remain unchanged.
  • C.2 Baseline Implementations: RLPD follows its cited implementation with the encoder improvement used in [−PM], while BC uses a diffusion policy with a U-Net denoiser trained for 500 epochs.BC implementation and parameters follow the cited reference.
  • D.1 Task Overview: The evaluation includes Pick and Place tasks requiring reaching, grasping, lifting, and placing objects beyond a target line or on a plate, plus Push tasks requiring pushing blocks beyond a target line.
  • D Additional Results: Figures 10 to 13 add Gymnasium learning curves, key-hyperparameter ablations, baseline-relative sample efficiency, and per-episode success-rate stability.These analyses constitute the reported additional results.

D.3 World Model Analysis · E Additional Discussion

D.3 analyzes MPAIL2’s learned world model through planning influence, causal rollout structure, latent-plan inspection, and qualitative improvement over training. E adds context on RLPD’s relative strength in state-based simulation and difficulty in visual real-world learning.

  • D.3 World Model Analysis: The learned policy contributes relatively little to the final optimized online MPPI plan.Figure 14 quantifies policy influence as the fraction of total MPPI score attributable to policy-seeded rollouts among elite samples at the final planning iteration.
  • D.3 World Model Analysis: Latent-plan visualizations let practitioners inspect whether dynamics, reward, or value needs more supervision or training.The reconstructed plans provide qualitative evidence that the robot’s world model improves over time.
  • D.3 World Model Analysis: MPAIL2 is evaluated through ablations of demonstrations, horizon lengths, and gradient penalty coefficients in Sim: Block Push.Additional analyses compare time efficiency and success-rate stability across methods.
  • D.3 World Model Analysis: Block trajectories remain static until contact, illustrating that MPAIL2 learns an implicitly causal world model.As the robot approaches, predicted block motion becomes more dynamic, and the model learns that block motion can only be influenced through contact.
  • D.3 World Model Analysis: By episode 100, real-world plans show the robot reaching, grasping, and placing the mug onto the plate.Early episodes are unstructured, unmodeled, and exploratory; later plans become structured around successful manipulation.
  • E Additional Discussion: RLPD is the second strongest method on Sim: Block Push (State) when given state observations, dense reward, and action-labeled demonstrations.Its difficulty is concentrated in the visual, real-world setting, where it does not yet exhibit successes within the small interaction budgets considered.

E.1 Why Is MPAIL2 Significantly More Sample Efficient? Building up from baselines.

MPAIL2’s sample efficiency improves incrementally through online rewards, representation and dynamics modeling, and planning. The planner makes real-world training practical, improves performance over BC, and supports transfer and earlier recovery behavior.

  • RLPD→DAC: DAC improves online over RLPD by replacing the hand-designed reward with an online reward model, although learning remains much slower than MPAIL2.With classifier-based rewards, the distinction partly reduces to reward regularization because DAC uses a Gradient Penalty.
  • RLPD→DAC: RLPD showed no improvement on Block Push after 300 episodes, while omitting reward and encoder pre-training kept comparisons fair by withholding their additional prior information and supervision.The withheld components rely on curated failure examples for the reward classifier and large-scale encoder pre-training.
  • DAC→MAIRL: MAIRL’s encoder and dynamics model significantly improve learning efficiency, ending repeated workspace-edge over-reaching earlier and producing numerous successes.An encoder was also provided to DAC, trained via gradients from the va…
  • MAIRL→MPAIL2: MPAIL2’s planner reduces training time enough for practical real-world learning, consistently improves upon BC despite BC’s privileged supervision, and enables transfer where MAIRL and BC show negative transfer.Recovery behavior is more robust and appears much earlier than with MAIRL.
  • BC (Diffusion): 82% success for MPAIL2 versus 12% for BC on Pick-and-Place reflects MPAIL2’s recovery after missed grasps, whereas BC often continues the demonstration trajectory without grasping the cube.BC’s rote memorization and interpolation can fail when execution deviates from the demonstrations; MPAIL2 often returns to reattempt a grasp.
  • MPAIL: Under matched low-interaction budgets, MPAIL’s on-policy rollout buffer covers substantially fewer state transitions than MPAIL2’s accumulated replay buffer, making its adversarial reward more prone to rapidly fi…The passage cautions that this poor performance does not show planning or latent dynamics are insufficient.

E.2 Discussion by Task … E.5 Areas of Improvement

Across tasks, MPAIL2’s performance reflects task dynamics, planning, and transfer through model-based representations, while remaining limited by instability, variability, and untested scalability. The discussion identifies planning and online policy resolution as central strengths and several priorities for improvement.

  • E.2 Discussion by Task: Block Push is more dynamically unstable in simulation, where BC generally underperforms [−P], [−PM], and MPAIL2, while higher friction and slower actuation improve real-world BC performance.BC often stays near expert trajectories after complete block destabilization.
  • E.2 Discussion by Task: Block Push training instability may arise because experts lack recovery demonstrations, making increasingly precise inferred rewards harder for learners to satisfy after sampled destabilization and recovery.
  • E.2 Discussion by Task: Real: Pick-and-Place appears to demand more generalization than simulation, with [−P] and BC both dropping below 16% in the real world.Sim: Pick-and-Place can be solved by relying on demonstrations, and BC’s high relative success suggests limited generalization is needed there.
  • E.2 Discussion by Task: MPAIL2 outperforms BC on average for Mug-on-Plate, a task that randomizes the plate’s target location and rewards precise placement at the plate center.BC tends to miss the precise center, whereas MPAIL2 tends to place the mug there.
  • E.3 Problem Learning Intuition: M = 0.05N policy fraction accompanies high task performance, while the policy contributes below 20% of Sim: Block Push’s elite planning score; removing planning substantially degrades all tasks.These findings support planning as the primary contributor rather than the explicit policy.
  • E.4 Why Does MPAIL2 Exhibit Transfer?: MPAIL2’s first from-scratch, real-world online transfer between tasks uses model-based representations and planning, with transfer hypothesized to arise from resolving new policies online.Planning can disregard a prior policy when it conflicts with the new task’s learned dynamics, reward, or value.
  • E.4 Why Does MPAIL2 Exhibit Transfer?: Dynamics-only transfer initializes the encoder and dynamics models from pre-trained weights while randomly initializing reward, value, and policy components.This experiment evaluates whether the latent representation transfers independently of the remaining learned components.
  • E.5 Areas of Improvement: Areas for improvement include scaling beyond the evaluated tasks, stabilizing adversarial reward learning, reducing first-success variability, and diagnosing collapsed plans at episode beginnings.The authors emphasize monotonic improvement with experience and note that planning visualizations mainly cover Sim: Block Push.
Loading 2602.24121v2…