Source-linked AI summary
Learning Parameterized Skills
Bruno Da Silva, George Konidaris, Andrew Barto
TL;DR
The paper asks how to construct reusable skills for distributions of related reinforcement-learning tasks without learning a separate policy for every variation. It estimates the topology of the policy manifold from solved examples, classifies tasks by manifold, and regresses task parameters to policy parameters; experiments examine dart throwing with an underactuated simulated arm. The policies vary smoothly within multiple disjoint manifolds, and a parameterized skill reduces additional policy updates from 22 when learning from scratch to 4 with 9 examples and 2 with 20 or more.
Problem
Learning a separate policy for every variation of a parameterized task is infeasible, motivating a skill that generalizes across related tasks.
Method
The method estimates policy-manifold topology from solved task examples, classifies tasks by manifold, and uses nonlinear regression to predict policy parameters.
Results
22 policy updates are required from scratch, versus 4 with 9 examples and 2 with 20 or more examples to reach the performance threshold.
Takeaways & Limitations
The framework constructs effective parameterized skills and identifies the number of qualitatively different strategies required across a task distribution.
Takeaways & Limitations
Selecting training tasks actively and handling unknown non-stationary task distributions remain open problems.
Abstract
from arXiv · showhide
We introduce a method for constructing skills capable of solving tasks drawn from a distribution of parameterized reinforcement learning problems. The method draws example tasks from a distribution of interest and uses the corresponding learned policies to estimate the topology of the lower-dimensional piecewise-smooth manifold on which the skill policies lie. This manifold models how policy parameters change as task parameters vary. The method identifies the number of charts that compose the manifold and then applies non-linear regression in each chart to construct a parameterized skill by predicting policy parameters from task parameters. We evaluate our method on an underactuated simulated robotic arm tasked with learning to accurately throw darts at a parameterized target location.
1. Introduction
The paper addresses the challenge of reusing reinforcement-learning skills across related tasks when a single policy cannot cover task variations. It proposes constructing a parameterized skill from a few solved instances by modeling how policy parameters vary with task parameters.
- Motivation: Hierarchical reinforcement learning uses temporally extended policies to abstract low-level control in complex problems.The options framework is presented as a widely used formalism in which high-level skills can be reused across tasks and contexts.
- Problem: A single policy is insufficient when a task requires varying actions, such as kicks with different forces and target locations.Learning one policy for every possible task variation is described as infeasible.
- Approach: The proposed method learns a few task instances, estimates the policy manifold, and predicts policy parameters for new task parameters.It identifies the manifold’s charts and fits nonlinear regression within each chart.
- Evaluation: The method is evaluated on an underactuated simulated robotic arm learning to throw darts accurately at parameterized target locations.
2. Setting
The setting considers policies for related MDP tasks drawn from a distribution and assumes their optimal parameters lie on a lower-dimensional, smoothly varying policy manifold. Because the manifold may have multiple charts, the method can discover qualitatively distinct policy families.
- Task distribution: The agent seeks a policy skill that maximizes expected return across MDP tasks sampled from a task distribution.Tasks are assumed to have sufficiently similar dynamics and reward functions to count as variations of one task.
- Parameterized skill: A parameterized skill maps task parameters to the policy parameters used for the task currently being solved.
- Training data: The training set consists of sampled task parameters paired with policy parameters that maximize return for those tasks.These pairs are used to construct a skill that approximately maximizes expected performance over the task distribution.
- Policy manifold: The framework assumes related-task policies lie on a lower-dimensional surface and vary smoothly as task parameters change.This assumption is motivated by smooth policy changes and locally related goals.
- Multiple charts: Multiple charts may be required when some policy-space points cannot be locally modified to solve new related goals.The method discovers these separate policy regions and integrates models for them.
3. Overview
The method combines manifold analysis, classification, and nonlinear regression to generalize from solved task instances to parameterized skills. It first determines which policy manifold applies, then predicts the corresponding policy parameters.
- Overview: Example task instances and solution policies are collected before training independent nonlinear regressors.Separate regression models are needed because policies for different task subsets may occupy disjoint manifolds.
- Procedure: The procedure samples tasks, estimates the number of policy manifolds, trains a task-to-manifold classifier, and fits regressors within each manifold.
- Execution: The classifier χ selects a manifold for a task, after which that manifold’s regression models map task parameters to policy parameters.
- Design choices: The framework leaves the policy representation, learning algorithm, classifier, and regression model unspecified so they can match the application.
4. The Dart Throwing Domain
The evaluation uses a simulated planar underactuated arm that must throw darts at targets around it. The domain combines nonlinear control, a constrained actuator, and a state representation that includes the dart’s release status.
- Environment: The simulated planar arm is mounted in a 3-meter-high, 4-meter-wide room and has three connected links.
- Actuation: A single motor applies torque only to the second joint, creating a difficult nonlinear underactuated control problem.
- Modeling assumption: An alternative multivariate regression design is available when policy features are not approximately independent conditioned on the task.
- Task: The arm holds and releases a dart at the end of its third link while aiming to hit a target of interest.
- State: The system state has seven features: six continuous link-angle and angular-velocity values plus one binary dart-held indicator.
5. Learning Parameterized Skills for Dart Throwing
The dart-throwing implementation combines DMP-based policy representations with PoWER policy search, then analyzes policy-space topology and uses classification and regression to construct parameterized skills.
- Policy representation: The implementation uses Dynamic Movement Primitives to represent smooth robotic-arm movements with linearly parameterized nonlinear differential equations.DMPs are selected because they are suited to multi-joint simulated robotic-arm control.
- Policy representation: The DMP dynamics use position, velocity, start and goal positions, temporal scaling, spring-like and damping terms, plus a learned nonlinear function to generate complex movements.The phase-dependent nonlinear function is represented with Gaussian basis functions, and a PID controller tracks the resulting trajectories.
- Policy representation: The resulting policy vector has 37 dimensions: a dart-release phase value, the DMP goal, and 35 Gaussian-basis weights.The release parameter specifies when the arm lets go of the dart.
- Policy learning: DMPs are combined with PoWER, which explores perturbed policy parameters through rollouts and updates them toward parameters inducing a success-weighted path distribution.Exploration uses structured, state-dependent perturbations, while rollout weighting can discard samples with very small importance weights.
- Topology and skill construction: ISOMAP estimates the number and membership of disconnected lower-dimensional policy manifolds, supplying labels for a classifier that maps task parameters to manifold identifiers.The experiment uses a linear classifier, although more powerful classifiers could be used generally.
- Topology and skill construction: Support Vector Machines with Gaussian kernels construct the nonlinear regression models that predict policy parameters within each manifold chart.The experiments use Gaussian-kernel SVMs with inverse variance width 5.0.
6. Experiments
Experiments show that policy solutions vary smoothly within multiple lower-dimensional manifolds, enabling parameterized skills to predict policies for unseen targets. In dart throwing, predictions improve with more training tasks and substantially reduce subsequent learning.
- Policy-manifold structure: Policies vary smoothly with task changes but occupy two disjoint lower-dimensional surfaces, reflecting qualitatively distinct movement strategies.The discontinuity occurs near targets directly above the arm, separating policies for targets on the left and right.
- Policy-manifold structure: The two manifolds are analyzed separately over target-angle ranges covering the right side and the left-side quadrants.The right-side range spans approximately 1.57 to 3.5 radians, while the other manifold covers targets to the left of the arm.
- Parameterized-skill construction: A linear task-policy separation supports using distinct regression models for the two qualitatively different movement types.The nonlinear fit is constructed independently within each manifold.
- Policy-manifold structure: ISOMAP estimates the skill manifold’s intrinsic dimensionality as 2, matching the dimensionality of the task-parameter space.Policies for similar tasks remain geometrically close in the solution space.
- Dart-throwing evaluation: With five sample trajectories and two additional policy updates, the predicted policy hits a novel middle-right-wall target perfectly.The examples included targets high on the ceiling and low on the right wall, while the prediction targeted an unseen location.
- Dart-throwing evaluation: 6% error is reached after 6 samples, stabilizing around 3% with approximately 15 samples for predicted policy parameters.Error is averaged over 15 unknown tasks and measures relative distance from known good solutions.
- Dart-throwing evaluation: 30cm average target distance is achieved with 24 samples, compared with 70cm after 10 samples before further learning.These measurements evaluate direct execution of predicted policies on novel tasks.
- Dart-throwing evaluation: 2 additional policy updates are required with 20 or more examples, versus 22 updates when learning from scratch.With 9 examples, the average requirement falls to 4 updates.
7. Related Work
Related work addresses hierarchical skills, task transfer, and parameterized policies, but prior methods leave gaps in constructing complete parameterizations for families of related tasks.
- Hierarchical and transferable skills: Hierarchical reinforcement learning uses temporally extended policies to abstract low-level control in high-dimensional tasks.The options framework is identified as the most widely used hierarchical reinforcement learning formalism.
- Hierarchical and transferable skills: Existing skill-transfer work adapts option termination criteria but does not predict complete policy parameterizations for new tasks.
- Task and value-function transfer: Several transfer methods reuse models or value functions between task pairs rather than constructing a parameterized solution from learned tasks.Some approaches also assume known mappings between source and target task features or require prior task-dynamics knowledge.
- Task and value-function transfer: A domain-dependent kicking method constructs a closed-form policy-parameter model by exhaustively testing a known relevant parameter and assuming a quadratic relation.
- Parameterized-skill methods: A DMP-based method maps task descriptions to metaparameters but assumes DMP policies and sufficient metaparameters for the task class.
- Parameterized-skill methods: Related neuroscience work supports structure learning in humans but does not propose a concrete method for identifying and constructing parameterized skills.
8. Conclusions and Future Work
The paper presents a general framework that learns parameterized skills from sampled task instances using classifiers and nonlinear regression over structured policy manifolds. It reports effective skills and strategy identification, while identifying task selection, changing task distributions, and efficient geometry analysis as future challenges.
- Conclusions: The framework generalizes a small number of sampled task instances by combining classifiers and nonlinear regression models.
- Conclusions: The approach is effective in practice because it exploits intrinsic policy-space structure and lower-dimensional manifolds for similar tasks.
- Conclusions: The framework constructs effective parameterized skills and identifies the number of qualitatively different strategies required by a task distribution.
- Future work: Active selection of training tasks remains an open problem for improving a skill's readiness under an expected future task distribution.
- Future work: Unknown changes in the task distribution require more general strategies than resampling instances from the existing training set.
- Future work: Efficient topology and geometry analysis remains challenging because global methods may require an unreasonable number of training tasks.Policy-search trajectories could provide additional manifold samples and support an off-policy construction approach.