A Bayesian Approach to Robust Inverse Reinforcement Learning

Ran Wei; Siliang Zeng; Chenliang Li; Alfredo Garcia; Anthony McDonald; Mingyi Hong

Source-linked AI summary

A Bayesian Approach to Robust Inverse Reinforcement Learning

Ran Wei, Siliang Zeng, Chenliang Li, Alfredo Garcia, Anthony McDonald, Mingyi Hong

arXiv:2309.08571v2Sep 15, 2023cs.LG

TL;DR

Offline model-based IRL must infer rewards and policies despite limited environment access and covariate shift. The paper jointly estimates reward and subjective dynamics with priors over expert-model accuracy, finding robust policies and state-of-the-art MuJoCo performance when high model accuracy is assumed a priori.

Problem
Offline model-based IRL faces limited access to deployment dynamics and robustness challenges from covariate shift.
Method
The framework simultaneously estimates the expert’s reward and subjective environment dynamics using priors that parameterize believed expert-model accuracy.
Results
The proposed algorithms achieve state-of-the-art performance in MuJoCo benchmarks, with robustness emerging when the expert is believed a priori to have a highly accurate model.
Takeaways & Limitations
Bayesian simultaneous estimation can produce robust offline model-based IRL policies without ad hoc pessimistic penalty design.
Takeaways & Limitations
BM-IRL can have less stable training dynamics and larger evaluation variance than RM-IRL.

Abstract

from arXiv · show

We consider a Bayesian approach to offline model-based inverse reinforcement learning (IRL). The proposed framework differs from existing offline model-based IRL approaches by performing simultaneous estimation of the expert's reward function and subjective model of environment dynamics. We make use of a class of prior distributions which parameterizes how accurate the expert's model of the environment is to develop efficient algorithms to estimate the expert's reward and subjective dynamics in high-dimensional settings. Our analysis reveals a novel insight that the estimated policy exhibits robust performance when the expert is believed (a priori) to have a highly accurate model of the environment. We verify this observation in the MuJoCo environments and show that our algorithms outperform state-of-the-art offline IRL algorithms.

1 Introduction

The paper addresses offline model-based IRL by jointly estimating the expert’s reward and subjective dynamics model, using priors over model accuracy to obtain robust policies without pessimistic penalty design.

Offline IRL faces limited deployment-environment access and robustness problems caused by covariate shift between training and deployment data.
Existing model-based offline IRL methods typically estimate dynamics first, then fix that model while learning the reward and policy.
The proposed Bayesian approach simultaneously estimates the expert’s reward function and internal dynamics model because expert decisions convey beliefs about the environment.
Priors parameterize assumed expert-model accuracy, and high prior accuracy yields robustness through worst-case planning outside the offline data distribution.
The resulting high-dimensional continuous-control algorithms outperform state-of-the-art offline IRL methods in MuJoCo without designing pessimistic penalties.

2 Preliminaries

The preliminaries formulate entropy-regularized MDPs and standard IRL objectives, then motivate offline model-based variants and robust alternatives for handling imperfect learned dynamics.

2.1 Markov Decision Process: The paper models behavior with infinite-horizon entropy-regularized MDPs defined by states, actions, dynamics, rewards, initial-state distribution, and discount factor.
2.2 Inverse Reinforcement Learning: Standard IRL seeks a reward whose entropy-regularized learner policy matches the expert’s state-action features.
2.2 Inverse Reinforcement Learning: Maximum discounted likelihood provides a related IRL formulation using expert trajectories, with the policy implicitly parameterized by reward parameters.
2.2 Inverse Reinforcement Learning: MCE-IRL and ML-IRL are equivalent for sufficiently large trajectories under linear rewards, while the stated formulation also permits nonlinear reward parameterization.
2.2 Inverse Reinforcement Learning: Conventional GAN-like IRL methods require true dynamics or a high-quality simulator, whereas offline model-based methods estimate dynamics and incur errors that scale with model error and estimated value.
2.3 Offline Model-Based IRL & RL: Robust alternatives train policies against worst-case dynamics in out-of-distribution states, avoiding explicit pessimistic penalties and connecting to robust MDP methods.

3 A Bayesian Approach to Model-based IRL

BM-IRL jointly estimates an expert’s reward and internal dynamics model under a prior controlling believed dynamics accuracy. With high prior accuracy, robustness emerges naturally, enabling scalable BM-IRL and RM-IRL algorithms for high-dimensional settings.

3 A Bayesian Approach to Model-based IRL: Jointly estimating the expert’s reward and internal dynamics model distinguishes BM-IRL from two-stage offline model-based IRL.The expert’s decisions inform both reward and subjective dynamics estimation, and the objective includes the internal-dynamics log likelihood weighted by λ.
3 A Bayesian Approach to Model-based IRL: The prior precision λ represents how accurate the expert’s environment model is believed to be.The framework estimates a posterior over reward and dynamics parameters using a chosen prior, with λ controlling the dynamics-accuracy assumption.
3.1 Naive Solution: Without regularization, reward and dynamics can trade off by making expert actions appear valuable through falsely optimistic dynamics.The accuracy prior helps disentangle genuinely high reward from optimistic internal dynamics.
3.2 Robust BM-IRL: High believed expert-model accuracy makes robustness emerge by planning against worst-case dynamics outside the offline data distribution.The resulting likelihood maximizes expert-trajectory reward while minimizing learner reward under adversarially trained estimated dynamics, connecting BM-IRL to robust MDPs.
3.3 Proposed Algorithms: The paper develops two scalable MAP-estimation algorithms: BM-IRL for the naive simultaneous approach and RM-IRL for efficient optimization at high λ.Both use a tractable surrogate objective and nested two-timescale stochastic optimization; RM-IRL simplifies updates by emphasizing the dynamics prior.
3.4 Performance Guarantees: The performance gap is linear in policy estimation error but quadratic in dynamics estimation error, making accurate dynamics estimation essential.The bound decomposes the real-environment gap between expert and learner and motivates priors that explicitly encourage dynamics accuracy.

4 Experiments

Experiments show that the dynamics-accuracy prior shapes BM-IRL behavior, while BM-IRL and RM-IRL achieve strong performance across MuJoCo offline IRL benchmarks.

4.1 Gridworld Example: The Gridworld study trained BM-IRL agents with λ ∈{0.001, 0.5, 10} on 100 expert trajectories and compared them with a fixed-dynamics two-stage IRL agent.The two-stage model was pretrained by maximum likelihood before reward estimation.
4.1 Gridworld Example: In Gridworld, increasing λ produced more accurate reward estimates concentrated near the true goal and fewer illegal transitions.BM-IRL agents with λ = 0.5 and λ = 10 localized high rewards more precisely than the λ = 0.001 and two-stage agents.
4.2 MuJoCo Benchmarks: The MuJoCo evaluation used 10 expert trajectories from D4RL, normalized scores over 5 random seeds, and comparisons with model-free and model-based offline IRL baselines.Baselines included Behavior Cloning, ValueDICE, ML-IRL, and CLARE.
4.2 MuJoCo Benchmarks: BM-IRL and RM-IRL outperformed other algorithms in 6/9 MuJoCo settings, with differences from the best algorithm very small in 2/9 additional settings.The only setting where ML-IRL significantly outperformed the proposed algorithms was HalfCheetah medium-replay.
4.2 MuJoCo Benchmarks: BM-IRL outperformed RM-IRL in 7/9 MuJoCo settings, although Hopper medium showed significantly lower mean performance and large variance.The authors attribute this to higher BM-IRL training instability, while its peak performance was on par with RM-IRL.

5 Limitations

BM-IRL has less stable training dynamics than RM-IRL, causing evaluation performance to alternate between near-optimal and medium-performance periods.

5 Limitations: BM-IRL can alternate between near-optimal and medium evaluation performance, producing larger variance than RM-IRL.The authors relate the instability to noisier two-sample-path learning signals and simultaneous dynamics-model training.

6 Conclusion

The paper concludes that Bayesian simultaneous estimation yields robust policies and state-of-the-art MuJoCo offline model-based IRL performance without ad hoc pessimistic penalties.

6 Conclusion: Bayesian simultaneous estimation gives rise to robust policies and algorithms achieving state-of-the-art performance on MuJoCo continuous-control benchmarks.The proposed algorithms do so without ad hoc pessimistic penalty design.
6 Conclusion: A future direction is identifying priors that robustly infer rewards and internal dynamics from sub-optimal and biased human demonstrators.

Reinforcement Learning

The paper situates its approach within Bayesian IRL and decision-aware model learning, emphasizing simultaneous inference of reward and dynamics and links between value-targeted objectives and robustness.

Bayesian IRL: Unlike earlier Bayesian IRL formulations with fixed dynamics during reward inference, simultaneous reward-and-dynamics estimation can infer demonstrators’ biased environment beliefs.
Decision-aware model learning: Decision-aware model-learning analyses suggest that value-targeted model objectives may be related to robust objectives.The cited perspective also notes that value-equivalent models shrink as the policy and value sets grow.

B.1 Proofs For Section 3.1

The appendix analyzes simultaneous reward–dynamics estimation, showing that unrestricted estimation is generally unidentifiable and that regularization or prior assumptions are needed for accurate reward recovery.

Gradient derivation: The policy log-likelihood gradient is derived in terms of Q-function gradients for both reward and dynamics parameters.The derivation uses discounted state-action occupancy measures and separates gradients with respect to θ1 and θ2.
Identifiability: Simultaneous reward–dynamics estimation without specific assumptions on the prior is generally unidentifiable.An alternative dynamics model can be paired with an alternative reward while preserving Bellman-consistent value functions and policies.
Implications: Existing simultaneous estimators without explicit or implicit regularization cannot generally recover the expert reward accurately.The appendix identifies SERD [18] as an example and links inaccurate reward estimation to poor performance guarantees.
Likelihood analysis: The discounted likelihood derivation decomposes the objective into a reward-related term and a dynamics-dependent correction term.The correction becomes negligible when the estimated dynamics are accurate under the expert distribution.
Proof technique: The bound on the correction term follows by applying Jensen’s, Hölder’s, and Pinsker’s inequalities.These inequalities connect the correction to divergence between the true and estimated transition distributions.

B.3 Proofs For Section 3.4

This appendix establishes a performance bound that decomposes learner–expert performance differences into policy and model errors, then describes the implementation of the proposed algorithms.

Performance decomposition: The decomposition separates real-environment performance difference, learner-distribution model disadvantage, and expert-distribution model advantage.This decomposition is based on a result from [54].
Performance bound: The performance gap is bounded by policy estimation error and dynamics estimation error under stated density-ratio and bounded-reward assumptions.The theorem uses ε_π for policy error, ε_P for dynamics error, R_max, and a bounded expert–learner marginal state-action density ratio.
Optimization: Dynamics gradients use a REINFORCE baseline b(s, a) = Qθ(s, a) − Rθ1(s, a) to reduce variance and normalize updates across mini-batches.The value function is estimated from a single action sample in the continuous-control setting.
Implementation setting: The implementation builds on the official RAMBO implementation and follows standard model-based reinforcement-learning practices for MuJoCo.The appendix specifies the implementation basis and benchmark setting but does not report additional benchmark outcomes here.

D.1.1 Dynamics Pre-training

The dynamics pre-training procedure uses an ensemble of probabilistic neural networks, selects elite models by validation error, and applies environment-specific rollout and training settings.

Model architecture: The dynamics model is an ensemble of K = 7 neural networks predicting Gaussian mean and covariance for state differences δ = s′ − s.Each network is a four-layer feedforward model with 200 hidden units and SiLU activations.
Model selection: The five best networks by mean-squared error on a 10% holdout set are retained as elites for rollouts.During rollouts, one elite model is randomly selected to generate the next state.
Environment-specific settings: Rollout hyperparameters vary by MuJoCo environment: Hopper and Walker2d require larger rollout steps, whereas HalfCheetah benefits from smaller rollout steps and larger rollout batches.The authors did not use different rollout hyperparameters for different datasets.
Training procedure: BM-IRL and RM-IRL use different reward and dynamics rollout batch sizes because BM-IRL obtains only the first real-path step from the dataset.The batch sizes are 1000 for BM-IRL and 256 for RM-IRL, and BM-IRL is trained for more epochs.
Loss weighting: The dynamics adversarial-loss weight is λ1 = 0.01 for both algorithms, while small λ1 values can cause severe model exploitation.The authors report that this setting outperformed the official RAMBO value of 0.0768.