A survey on policy search algorithms for learning robot controllers in a handful of trials

Konstantinos Chatzilygeroudis; Vassilis Vassiliades; Freek Stulp; Sylvain Calinon; Jean-Baptiste Mouret

Source-linked AI summary

A survey on policy search algorithms for learning robot controllers in a handful of trials

Konstantinos Chatzilygeroudis, Vassilis Vassiliades, Freek Stulp, Sylvain Calinon, Jean-Baptiste Mouret

arXiv:1807.02303v5Jul 6, 2018cs.ROcs.AIcs.LGstat.ML

TL;DR

Physical-robot policy search is often too data-intensive for adaptation, creating the problem of learning effective controllers in only a few minutes or trials. The survey synthesizes prior-based and model-based strategies for micro-data reinforcement learning and concludes that successful methods combine them, while scalability and generic priors remain open challenges.

Problem
Robot policy search commonly requires too much interaction for physical systems, motivating learning in only a few minutes or a handful of trials.
Method
The survey organizes micro-data policy search around prior knowledge and learned surrogate models of expected return or dynamics.
Results
Successful micro-data algorithms combine prior knowledge with models, enabling robot policies to be learned in a few minutes of interaction.
Takeaways & Limitations
Future progress depends on scaling simultaneously to high-dimensional state and policy spaces and developing more generic, task-agnostic priors.

Abstract

from arXiv · show

Most policy search algorithms require thousands of training episodes to find an effective policy, which is often infeasible with a physical robot. This survey article focuses on the extreme other end of the spectrum: how can a robot adapt with only a handful of trials (a dozen) and a few minutes? By analogy with the word "big-data", we refer to this challenge as "micro-data reinforcement learning". We show that a first strategy is to leverage prior knowledge on the policy structure (e.g., dynamic movement primitives), on the policy parameters (e.g., demonstrations), or on the dynamics (e.g., simulators). A second strategy is to create data-driven surrogate models of the expected reward (e.g., Bayesian optimization) or the dynamical model (e.g., model-based policy search), so that the policy optimizer queries the model instead of the real system. Overall, all successful micro-data algorithms combine these two strategies by varying the kind of model and prior knowledge. The current scientific challenges essentially revolve around scaling up to complex robots (e.g., humanoids), designing generic priors, and optimizing the computing time.

I. INTRODUCTION

Robotic policy search is often too data-intensive for real-world adaptation, motivating micro-data reinforcement learning: learning in a few minutes or roughly a dozen trials. The survey organizes solutions around prior knowledge and learned surrogate models.

Motivation: Robotics cannot readily reproduce data-intensive successes requiring millions of games, extensive real-time play, or hundreds of thousands of physical grasps.Real-world interaction cannot be accelerated or parallelized like computation, making such requirements technically infeasible for many robots.
Motivation: Online adaptation is useful only when it occurs within a few minutes or approximately a dozen trials.The motivating cases include recovering from damage, adapting gait to terrain, and discovering new ways to manipulate objects.
Micro-data reinforcement learning: The survey calls learning by trial-and-error in a handful of trials “micro-data reinforcement learning.”The term emphasizes an absolute interaction limit rather than relative data efficiency.
Strategies: Micro-data policy search leverages prior knowledge about dynamics, policy parameters, policy structure, or expected return, and learns surrogate models of dynamics or expected return.The two strategies can use information before learning or models built from data gathered during learning.
Survey organization: The survey categorizes methods by priors on policy structure and parameters, models of expected return, and models of dynamics.It also discusses approaches outside these main categories and identifies field challenges and recommendations.
Problem formulation: The policy-search formulation optimizes parameters θ of a parameterized policy followed over a fixed episode horizon.The survey focuses on episode-based policy search and excludes planning-based methods that search directly in state-action or trajectory spaces.

III. USING PRIORS ON THE POLICY PARAMETERS/REPRESENTATION

Policy representation must balance expressiveness against searchability: it should represent near-optimal behavior while keeping the parameter space feasible to optimize. Lower-dimensional spaces are generally easier to search, but may restrict achievable performance or reuse.

Design trade-off: Policy design is guided by a trade-off between expressiveness and an efficiently searchable parameter space.The choice of how parameters map states to actions determines this balance.
Expressiveness: An expressive policy representation should approximate the optimal policy for a task while accepting a bounded margin of suboptimality.Absolute optimality is rarely required in robotics because small tracking errors and nonminimal command costs may be acceptable.
Searchability: A policy representation must also make its optimal parameters θ* feasible to find, and smaller dim(θ) generally improves searchability.This creates a trade-off between representing useful behaviors and keeping optimization tractable.

A. Hand-designed policies

Hand-designed policies use task knowledge to create compact parameter spaces that are easy to search. This can reduce optimization difficulty, but sacrifices generality and may leave substantial suboptimality unmeasured.

Hand-designed policies: Hand-tailoring a ball-acquisition policy reduced its parameter space to dim(θ)=4 and required 672 trials for optimization.The compact representation uses prior knowledge, after which policy search optimizes its parameters.
Limitations: Very low-dimensional hand-designed policies may have a large suboptimality margin because their lost expressive capacity is not estimated.A more expressive representation might have achieved higher reward, but the gap is unknown.
Limitations: Task-specific hand-designed policies are difficult to reuse for other tasks, limiting their transfer-learning capability.Their compactness is obtained partly by specializing the representation to the task for which it was designed.
Representability versus learnability: A universally expressive function approximator need not be learnable efficiently from data.The universal approximation theorem concerns representability, not the feasibility of finding the function from observations.

B. Policies as function approximators

Function approximators and trajectory-based policies provide reusable representations, while demonstrations and dynamical-system structure can improve initialization, robustness, stability, and searchability. DMPs are especially suitable for micro-data learning because their learned components receive one-dimensional phase inputs and preserve useful movement guarantees.

Function approximators: General policy representations, including linear policies, radial basis functions, and neural networks, can be applied across tasks but still require design choices such as basis count and shape.Non-parametric methods reduce some parameter choices but may require many basis functions and become slow.
Function approximators: Demonstrations can initialize policy parameters through supervised learning using state-action training pairs.This provides an initial parameter setting before policy-search refinement.
Trajectory-based policies: Trajectory-based policies include waypoint-based and dynamical-system-based forms and suit point-to-point or repetitive robotic movements.Dynamical-system policies combine function-approximation generality with robustness to perturbations and stability guarantees.
Dynamical movement primitives: Discrete DMPs combine a phase-based canonical system with spring-damper transformation systems and a function approximator for generating motion accelerations.The phase starts at 1 and converges to 0, while the motion state typically represents end-effector pose or joint angles.
Dynamical movement primitives: DMPs guarantee convergence to a goal ξg for discrete movements, while rhythmic DMPs generate repetitive motion independently of parameter values.The time constant ω controls whether the movement is slower or faster.
Dynamical movement primitives: DMP learning is facilitated because each dimension uses a function approximator with the one-dimensional phase variable s as input.The learned component is open-loop even though the overall DMP closes the loop on the state, simplifying learning.
Task generalization: Task-parameterized DMPs generalize task-specific parameters to variations described by a task vector q, while current applications are limited to learning from demonstrations.DMP parameters otherwise remain specific to a particular task.
Stability and adaptation: Time-dependent DMPs may fail to adapt to perturbations after some time, motivating autonomous stable controllers such as SEDS.SEDS uses dynamical systems to define time-independent, asymptotically stable policies.

D. Learning the controller

The controller-learning pipeline combines policy representations and initialization informed by prior knowledge with search procedures that select and evaluate policy parameters. Bayesian optimization learns a surrogate return model and uses acquisition functions to balance exploration and exploitation, achieving strong data efficiency in several robotic tasks.

D. Learning the controller: Controller parameters can include both reference-trajectory and low-level controller parameters, allowing them to be learned jointly.
D. Learning the controller: Expressive policies can represent many tasks but also create local minima, making initialization near the optimum important.
D. Learning the controller: Demonstrations provide policy initialization through recording, teleoperation, or kinesthetic teaching, although each modality has limitations.
D. Learning the controller: Policy structures derived from task or robot knowledge make representations expressive yet efficiently searchable, especially when combined with demonstrations.
D. Learning the controller: Bayesian optimization learns a surrogate expected-return model and selects the next policy by optimizing an acquisition function over its predictions and uncertainties.
D. Learning the controller: Bayesian optimization has shown strong robot-learning efficiency, including quadruped, biped, and soft-robot tasks requiring roughly 30–100 trials and at least an order-of-magnitude fewer data than competing methods.

B. Bayesian optimization with priors: using non-zero mean functions as a starting point for the search process

Bayesian optimization can incorporate priors as non-zero mean functions, simulator-derived kernels, trajectory-based behavior kernels, or multiple information sources. These combinations improve adaptation in several tasks, while the approach remains constrained by policy-space dimensionality and model-scaling issues.

B. Bayesian optimization with priors: IT&E uses a simulator-generated repertoire of about 15000 policies in a low-dimensional map, then applies Bayesian optimization during robot adaptation.The map can be 6-dimensional while the underlying policy space is 36-dimensional.
B. Bayesian optimization with priors: IT&E enabled a hexapod to adapt to damage conditions in less than 2 minutes and fewer than a dozen trials.
B. Bayesian optimization with priors: Prior and real-world observations are combined smoothly: real data correct predictions locally, while predictions revert to the prior farther from observed data.
B. Bayesian optimization with priors: Simulator-derived kernels can encode gait differences rather than Euclidean parameter distance, and learned kernels can approach hand-tuned performance while outperforming traditional Bayesian optimization.
B. Bayesian optimization with priors: Behavior-Based Kernels compare policies through trajectory densities, but evaluation was limited to low-dimensional, well-chosen policy spaces.
B. Bayesian optimization with priors: Combining learned dynamics models with Bayesian optimization can learn tasks in fewer than 10 episodes but scales poorly with both state-action and policy-space dimensionality.
B. Bayesian optimization with priors: MLEI balances prior likelihood against expected improvement, improving over random trials or mismatched hand-chosen priors and sometimes outperforming a real prior.

V. LEARNING MODELS OF THE DYNAMICS

Model-based policy search learns a surrogate dynamics model from observed trajectories and uses repeated model queries to estimate expected returns. Policy optimization can therefore proceed on the learned model rather than requiring every candidate evaluation on the robot.

V. LEARNING MODELS OF THE DYNAMICS: Model-based policy search learns robot dynamics from trajectory data and performs policy search on the resulting surrogate model.
V. LEARNING MODELS OF THE DYNAMICS: The section organizes dynamics-model learning around using trajectory data to approximate the system's transition behavior.
V. LEARNING MODELS OF THE DYNAMICS: The learned function ˆf(x, u) is used to estimate the expected return ˆJ(θ|τ1, · · · , τN).

A. Model-based Policy Search: alternating between updating the model and learning a policy in the model

Model-based policy search alternates between learning approximations of dynamics and immediate rewards and maximizing predicted long-term reward on those models. Its main practical advantage is policy-space independence, but available methods generally remain limited to relatively small state-action spaces.

A. Model-based Policy Search: The method alternates between learning models ˆf and ˆr and maximizing expected long-term reward on the learned model.
A. Model-based Policy Search: The iterative scheme begins with an initialized policy and dataset, updates the learned models from trajectory data, and evaluates ˆJ(θ|D).
A. Model-based Policy Search: Algorithm 4 collects state-action-reward samples, learns transition and immediate-reward models, and chooses θn+1 by maximizing predicted return.
A. Model-based Policy Search: Key design choices include initialization, model classes, the optimizer for predicted return, and the procedure for making long-term predictions.
A. Model-based Policy Search: Model-based policy search is usually more data-efficient than direct and surrogate-based methods because it depends less on policy-space dimensionality.
A. Model-based Policy Search: Practical model-based policy-search algorithms are generally available only for relatively small state-action spaces because they model transition dynamics.

1) Model learning:

Model-based policy search learns models of dynamics and rewards, then evaluates policies through simulated trajectories or deterministic approximations. Stochastic methods support parallel rollouts but can have high variance, whereas deterministic inference offers low-variance predictions and analytic gradients under stronger assumptions.

Model learning:: Learned models may be deterministic or probabilistic; probabilistic models provide uncertainty that can support more robust controllers and reduce exploitation of model bias.The survey notes that probabilistic models are usually more effective for model-based policy search because uncertainty enters long-term predictions.
Model learning:: Model-based policy search learns transition and reward models, then generates trajectories under candidate policy parameters instead of querying the real system.The learned models define transition probabilities and immediate rewards for rollouts.
Model learning:: Stochastic long-term prediction evaluates policies by sampling trajectories, using either ordinary Monte Carlo averages or PEGASUS with fixed random seeds to reduce sampling variance.Sampling-based evaluation is parallelizable and relatively easy to implement, but its predictions can have large variance that affects optimization.
Model learning:: PIPPS combines reparameterization and likelihood-ratio gradients into Total Propagation, while likelihood-ratio and combined gradients avoid the curse of chaos reported for reparameterization gradients.The reparameterization approach requires many rollouts for accurate gradient estimates.
Model learning:: Deterministic long-term prediction approximates trajectory distributions with linearization, sigma-point methods, or moment matching, usually using a Gaussian approximation.PILCO uses moment matching as a unimodal approximation minimizing KL-divergence to the true predictive distribution.
Model learning:: Deterministic inference provides low-variance predictions and analytic gradients, but intractable moment integrals limit applicability; PILCO therefore requires a known, differentiable reward function.These assumptions restrict when the required exact moments can be computed.
Model learning:: PILCO demonstrated data-efficient learning on benchmark tasks, including solving cart-pole swing-up with less than 20 seconds of interaction time.The survey also reports applications to a physical low-cost manipulator and simulated walking tasks.
Model learning:: Black-DROPS matches PILCO's data-efficiency while exploiting multi-core architectures to reduce computation time and allowing model, reward, and initialization components to be swapped easily.Its black-box formulation facilitates incorporating useful prior sources.

B. Using priors on the dynamics

Priors on dynamics reduce real-robot interaction by initializing model learning with simulators or other approximate dynamics and learning residuals. Successful approaches identify or update these priors before policy search, achieving adaptation in complex physical systems with few trials or under a minute.

B. Using priors on the dynamics: Dynamics priors initialize an approximate model, after which the learner estimates residual dynamics to reduce interaction time.The survey describes this as the general model-based policy search framework with an initialization strategy.
B. Using priors on the dynamics: Analytic or simulated data can create GP priors; PILCO with priors increased PILCO's data-efficiency on a real inverted pendulum using a simple prior model.The reward function is assumed known in these approaches.
B. Using priors on the dynamics: GP-MI combines model identification with non-parametric learning by selecting simulator parameters that best match observations before policy search.The tunable simulator parameters can represent physical quantities such as link lengths.
B. Using priors on the dynamics: Black-DROPS with GP-MI outperformed several prior-based and baseline methods on pendubot swing-up, even with misleading priors.It also learned walking policies for a damaged physical hexapod with 48D state and 18D action spaces in less than 1 minute of interaction time.
B. Using priors on the dynamics: VGMI uses Bayesian optimization to identify simulator mechanical parameters from real trajectories, then performs policy search on the updated simulator and outperformed PILCO on a physical dual-arm task.The procedure updates the simulator before policy optimization.
B. Using priors on the dynamics: A separate self-modeling procedure can execute actions, collect data, synthesize 15 candidate self-models, and select the most accurate model after around 15 cycles.The selected model is then used for policy search.
B. Using priors on the dynamics: Model-based policy search is presented as especially data-efficient when model uncertainty is included, while prior dynamics can reduce interaction requirements for high-dimensional or complicated systems.The remaining challenge is computational complexity.

A. Guided policy search

Guided policy search combines real-system trajectory optimization, learned local dynamics models, and supervised policy approximation. Related simulation and meta-learning approaches trade expensive prior computation for rapid physical adaptation, while scalability remains constrained by robot and observation dimensionality.

A. Guided policy search: Guided policy search combines local trajectory optimization on the real system, learned local dynamics, and supervised learning to approximate local controllers with one neural-network policy.Its two-loop structure couples real-robot execution with model learning and indirect policy search.
A. Guided policy search: Less data-efficient than model-based policy search but more data-efficient than traditional direct policy search, guided policy search still learned tasks in fewer than 100 episodes.Reported applications include 2D walking, peg-in-the-hole, and octopus-robot control, including high-dimensional and discontinuous settings.
B. Transferability approaches: Transferability approaches learn where simulation is accurate and require fewer than 25 physical-robot trials in most experiments.Reported examples include 15 trials for maze navigation, about 10 for quadruped walking, and 25 for damaged-leg adaptation.
B. Transferability approaches: Their main limitation is that they can find only policies whose simulated and real-world performance are similar.This favors settings such as static gaits over highly dynamic gaits.
C. Simulation-to-reality & meta-learning approaches: Simulation-to-reality and meta-learning approaches seek robustness across task or environment distributions, with meta-learning additionally targeting fast adaptation through few gradient steps.Reported results include less than 30 seconds of interaction for cartpole and double-pendulum control, while another six-legged-robot application required 30 minutes.
A. Scalability: Micro-data methods still face scaling challenges because model-based search struggles with high-dimensional state spaces, direct search needs low-dimensional policies, and raw-observation end-to-end learning remains inefficient.The survey frames these as complementary limitations rather than a single universally scalable approach.

B. Priors

Priors can sharply reduce policy-search interaction time, but their usefulness depends on how specifically they encode tasks and whether they remain relevant when conditions change. The survey highlights a trade-off between prior knowledge and learning, alongside open challenges in generic, robust, and quickly adaptable priors.

Prior knowledge: Priors can catalyze rapid policy search by shifting knowledge into policy structure, parameters, reward functions, or dynamical models.Examples include dynamic movement primitives, demonstrations, reward shaping, and models of intact robot dynamics.
Prior knowledge: Task-specific priors such as demonstrations accelerate learning but require expert knowledge for each task.The survey identifies task-agnostic priors, including physical-world properties, as a way to relax this requirement.
Prior selection: Priors can mislead policy search on new tasks, motivating methods that ignore irrelevant priors or actively select among alternatives.The survey also points to physical simulation as a generic way to generate priors offline.
Generalization and robustness: Micro-data methods often evaluate single tasks in fixed contexts, leaving generalization and robustness under varying conditions insufficiently assessed.Such evaluation is difficult because testing perturbations and varied starting conditions requires additional robot interaction.
Generalization and robustness: Fast adaptation should cover both changing goals and changing dynamics, while generalization can reduce interaction time when tasks or conditions change.The survey discusses goal-conditioned policies, behavioral repertoires, Bayesian optimization, meta-learning, and model identification as related approaches.

D. Interplay between planning, model-predictive control and policy search

The survey contrasts policy search with planning and model-predictive control, emphasizing that model-based methods can reduce interaction but may increase computation and depend on accurate state or dynamics models. It recommends matching algorithm families to robot dimensionality, available priors, and observation complexity.

Planning and control: When dynamics are known or learnable, control methods such as LQR, differential dynamic programming, and iLQG provide alternatives to policy search but can be computationally costly.LQR is optimal for linear dynamics with quadratic costs; nonlinear systems may be locally linearized or handled with related methods.
Planning and control: MPC enables online replanning by using shorter horizons and feedback from the real system, but sacrifices overall-task optimality.It is suited to short-term tasks such as trajectory tracking when a trajectory can be produced offline.
Planning and control: MPC relies on models that may be inaccurate early in learning and generally requires full system-state knowledge.Combining MPC with policy search can produce deployed policies that use raw observations and run faster than MPC online.
Computation time: PILCO required 3 weeks of computation for 20 episodes on a 3-link planar arm, while Black-DROPS with GP-MI required 24 hours for 26 pendubot episodes.Both examples illustrate that reducing robot interaction can shift costs into computation, especially with Gaussian-process queries.
Recommendations: Micro-data learning can achieve a few minutes of robot interaction, but scaling simultaneously to high-dimensional state and policy spaces remains a critical challenge.The survey proposes leveraging priors, using all trial data, active learning, uncertainty estimates, and expensive pre-mission computation.
Recommendations: For low-DOF robots, model-based policy search is recommended; for high-DOF robots with low-dimensional policy spaces, Bayesian optimization offers a computation–convergence trade-off.For complex robots, model-based policy search with dynamics priors is more data-efficient at increased computation cost; raw observations favor SimToReal with online adaptation.