Source-linked AI summary
BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators
Fabio Ramos, Rafael Carvalhaes Possas, Dieter Fox
TL;DR
Simulation-to-real transfer is limited by uncertainty and mismatch in simulator parameters and models. BayesSim uses likelihood-free Bayesian inference with a black-box simulator to estimate parameter posteriors, which support domain randomization that outperforms uniform-prior alternatives.
Problem
Simulation-trained robotics models and policies may not transfer reliably because simulator parameters and dynamics do not fully represent reality.
Method
BayesSim uses likelihood-free inference with a black-box simulator and physical observations to compute full posterior distributions over simulator parameters.
Results
Posterior-based domain randomization produces more robust and easier-to-train policies than randomization directly from the prior.
Takeaways & Limitations
BayesSim provides a principled Bayesian framework for simulator-parameter uncertainty and applies the resulting posterior to Sim2Real policy search.
Abstract
from arXiv · showhide
We introduce BayesSim, a framework for robotics simulations allowing a full Bayesian treatment for the parameters of the simulator. As simulators become more sophisticated and able to represent the dynamics more accurately, fundamental problems in robotics such as motion planning and perception can be solved in simulation and solutions transferred to the physical robot. However, even the most complex simulator might still not be able to represent reality in all its details either due to inaccurate parametrization or simplistic assumptions in the dynamic models. BayesSim provides a principled framework to reason about the uncertainty of simulation parameters. Given a black box simulator (or generative model) that outputs trajectories of state and action pairs from unknown simulation parameters, followed by trajectories obtained with a physical robot, we develop a likelihood-free inference method that computes the posterior distribution of simulation parameters. This posterior can then be used in problems where Sim2Real is critical, for example in policy search. We compare the performance of BayesSim in obtaining accurate posteriors in a number of classical control and robotics problems. Results show that the posterior computed from BayesSim can be used for domain randomization outperforming alternative methods that randomize based on uniform priors.
I. INTRODUCTION
BayesSim addresses the reality gap by inferring full posterior distributions over simulator parameters from physical observations, enabling posterior-based domain randomization for more robust policy learning.
- Simulation enables scalable, lower-cost training, but transfer to physical robots is hindered by incorrect parameters, simplified models, and numerical limitations.
- Domain randomization traditionally requires manual selection of simulation parameters and sampling distributions through iterative real-robot testing.
- BayesSim uses likelihood-free inference to compute full Bayesian posteriors over simulator parameters from small sets of real-system observations.
- The framework treats the simulator as a black box and does not require access to its internal differential equations.
- BayesSim represents parameter uncertainty beyond a single best-fitting solution, including potentially multimodal posterior structure.
- Posterior-based domain randomization produces policies that are more robust and easier to train than randomization directly from the prior.
III. PRELIMINARIES
The preliminaries frame likelihood-free inference and policy search: simulators generate observations when likelihoods are unavailable, while reinforcement learning optimizes discounted returns from environment interaction.
- Likelihood-free inference: Likelihood-free inference is needed when the simulator defines an intractable likelihood that can only be sampled through forward simulations.
- Likelihood-free inference: Approximate Bayesian computation accepts simulated samples whose sufficient statistics resemble physical observations, but convergence can be slow in high-dimensional parameter spaces.
- Reinforcement learning: An RL agent receives observations, selects actions, and obtains rewards over discrete timesteps, often under partial observability.
- Policy search: Policy search seeks policies that maximize the expected discounted sum of future rewards.
- Policy search: DDPG and PPO differ in experience collection, with DDPG off-policy and PPO on-policy sampling.
IV. BAYESSIM
BayesSim learns a conditional parameter density from simulator-generated pairs, then uses physical observations to estimate the posterior while accounting for the desired prior.
- BayesSim learns qφ(θ|x) from parameter–simulation-observation pairs generated under a proposal prior.
- The simulator maps sampled parameters θn to synthetic observations xn, creating training data for conditional density estimation.
- The learned conditional density is optimized through a log-likelihood objective over the generated parameter–observation pairs.
- After learning, BayesSim evaluates the conditional density at real observations to obtain an estimate of the simulator-parameter posterior.
- When proposal and desired priors differ, the posterior estimate is adjusted to account for the desired prior.
B. Mixture density random feature networks
BayesSim models the conditional posterior with a Gaussian mixture whose parameters are computed from Quasi Monte Carlo random Fourier features, with diagonal covariances and positivity constraints.
- The conditional density qφ(θ|x) is represented as a mixture of K Gaussian components.
- Quasi Monte Carlo random Fourier features replace a feedforward neural network when computing mixture coefficients, means, and covariances.
- Mixture coefficients are constrained to lie between zero and one and sum to one.
- Component means are defined as linear combinations of feature vectors.
- Covariance matrices are parameterized diagonally, assuming independence between simulator-parameter dimensions.
- The modified exponential linear unit enforces positive covariance values, while enough mixture components can reduce the restrictiveness of diagonal covariance assumptions.
C. Neural Network features
BayesSim can use neural-network features in a mixture density network to model the conditional distribution of simulator parameters. The experiments use a two-layer fully connected network with tanh activations.
- BayesSim uses neural-network features to create a model similar to a mixture density network.
- The feature extractor consists of two fully connected layers with nonlinear activations.
- The experiments use tanh as the activation function instead of the general sigmoid notation.
D. Quasi Monte Carlo random features
BayesSim represents mixture-density features with random Fourier features that approximate shift-invariant kernels. The approach supports kernel-based prior knowledge, uses quasi-Monte Carlo frequency sampling, and exhibits stable optimization behavior.
- Random Fourier features can parameterize the mixture density while remaining independent of input dimensionality.Their approximation converges to the original Hilbert space at O(1/√s), where s is the number of features.
- Mixture densities with random Fourier features are experimentally more stable to initialization and usually converge to the same local maximum.
- Random Fourier features approximate a shift-invariant kernel with a finite-dimensional feature dot product.The approximation is written as k(τ) ≈ Φ(x)^T Φ(x).
- The kernel choice allows prior knowledge to be incorporated into the approximation.For the RBF kernel, frequencies follow ω ∼ N(0, 2σ^-2I) and biases follow b ∼ U[-π, π].
- Halton sequences provide the quasi-Monte Carlo frequency samples used by the method.The cited work reports better convergence and lower approximation error than standard Monte Carlo.
E. Posterior recovery
BayesSim adjusts posterior calculations when the proposal and desired priors differ. In the paper, the assumed prior is uniform, while Gaussian proposals permit an analytical correction for a mixture and a single Gaussian.
- A posterior based on a mismatched proposal prior is corrected by weighting with p(θ)/˜p(θ).
- The paper assumes either a finite-support uniform prior or an improper uniform prior constant everywhere.
- With a Gaussian proposal, the ratio between a mixture and a single Gaussian can be computed analytically.
F. Sufficient statistics for state-action trajectories
BayesSim compresses long state-action trajectories into sufficient statistics before inference to avoid prohibitive model input dimensionality and computational cost. The representation includes trajectory-derived summary statistics such as means, variances, correlations, and transition differences.
- Long state-action trajectories are replaced by sufficient statistics before being input to the model.This strategy is adopted from Approximate Bayesian Computation to reduce input dimensionality and computational expense.
- The summary representation is defined as x = ψ(S, A), where S and A are sequences of states and actions.
- Candidate statistics include each time series’ mean, log variance, autocorrelation, and cross-correlation between series.
- The representation can use statistics of immediate future-state minus current-state differences.
- The feature map includes state and action dimensions, dot products, expectations, and variances.Ds denotes state-space dimensionality, Da action-space dimensionality, E expectation, and Var variance.
G. Example: CartPole posterior
BayesSim estimates unknown CartPole parameters from real observations, recovering dependent multimodal posteriors that can then support posterior-based domain randomization.
- BayesSim estimates the CartPole pole mass and length when both parameters are unknown, using uniform priors and simulated data.The experiment collects 1000 simulations before obtaining real observations.
- With 10 real trajectories, BayesSim recovers multimodal posteriors for masspole and pole length.The multiple modes reflect statistical dependencies, such as lower mass paired with longer length or the reverse.
- Posterior-based domain randomization samples simulator parameters from the inferred posterior while optimizing policy parameters β.This strategy is introduced to use the inferred posterior in policy training.
- The posterior mixture is sampled by selecting a mixture component according to α and then drawing θ from its Gaussian N(θ|µk, Σk).This approximates the expectation over simulator parameters during policy optimization.
- The experiments separately assess posterior accuracy and compare policy robustness under prior-based versus posterior-based randomization.These evaluations are organized around recovered posterior accuracy and robustness of trained policies.
A. Posterior recovery
BayesSim’s posterior-recovery analysis evaluates likelihood-free methods across seven control and robotics problems, emphasizing posterior concentration, stability, and multimodality. The experiments then examine how prior- versus posterior-based randomization affects policy robustness.
- Posterior quality: The evaluation compares Rejection ABC, ϵ-Free, and BayesSim with neural-network or random Fourier features across seven problems.Log predicted probability of the true parameter is averaged over five training and testing runs, with standard deviations reported.
- Posterior structure: BayesSim recovers multimodal posteriors when different parameter combinations explain the observations, such as longer poles paired with lighter masspoles.The recovered densities represent uncertainty over related parameters rather than only a single best-fitting explanation.
- Posterior quality: BayesSim generally produces higher log probabilities and lower standard deviations than Rejection ABC, while matching ϵ-Free means with greater stability.Random Fourier features usually yield higher log probabilities, whereas neural networks generally produce lower standard deviations.
- Posterior quality: For the Fetch slide problem, BayesSim with random Fourier features produces a posterior more peaked around the true friction value than alternative methods.The analysis reports random Fourier features as significantly superior in the robotics problems considered and slightly better than neural-network features.
- Policy robustness: CartPole policies randomized over BayesSim’s posterior are significantly more robust at the actual parameter value than policies randomized over a uniform prior.The posterior-trained policy also shows lower variance across multiple runs, while performance declines for shorter poles and excessively heavy masspoles.
- Policy robustness: In Fetch tasks, uniform-prior randomization performs well for closed-loop pushing but poorly for open-loop sliding, where BayesSim concentrates density near the true friction coefficient.The slide results show higher rewards around the true friction value, whereas uniform-prior performance is mostly flat.
VI. CONCLUSIONS
BayesSim combines Bayesian inference over robotics simulation parameters with domain randomization for policy search, producing more robust policies across parameter values than uniform-prior randomization. The framework remains dependent on meaningful trajectory statistics, with end-to-end representation learning identified as future work because simulation-heavy training may cause overfitting.
- BayesSim combines a Bayesian treatment of robotics simulation parameters with domain randomization for policy search.
- Posterior-based domain randomization produces more robust policies over multiple parameter values than uniform-prior randomization.
- BayesSim is presented as an initial Bayesian framework that can integrate black-box simulators and compute potentially multimodal parameter posteriors.
- Figure 5 compares prior- and posterior-randomized policies across simulation-parameter values for Fetch slide and Fetch push tasks.
- BayesSim relies on meaningful sufficient statistics for state-action trajectories, while end-to-end recurrent representations may overfit simulation data and become less sensitive to real parameter variation.