PaperlayerView exact revision on arXiv

Source-linked AI summary

MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies

Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, Sergey Levine

arXiv:1905.09808v1cs.LGstat.ML

TL;DR

Learning reusable skills that transfer across tasks is difficult when complex agents must coordinate multiple subtasks simultaneously. MCP factorizes behavior into motor primitives that can be activated and composed multiplicatively, then transfers them to new tasks. Across increasingly complex control tasks, MCP outperforms prior methods and learns the only successful policy on the benchmark’s hardest task.

  • Problem

    Reusable skills must transfer across tasks, but complex high-dimensional agents may need to coordinate many subtasks simultaneously, making discrete combinations difficult to model.

  • Method

    MCP learns reusable motor primitives from pre-training and combines multiple primitives multiplicatively to form flexible policies for target tasks.

  • Results

    MCP outperforms prior skill-transfer methods as task complexity grows and is the only approach that learns a successful policy on the benchmark’s most challenging task.

  • Takeaways & Limitations

    Once trained, MCP primitives form an action space that supports structured exploration and novel combinations for long-horizon continuous-control tasks.

  • Takeaways & Limitations

    Transfer is considered only between related pre-training and transfer tasks, and incorporating temporal abstractions remains an important direction.

Abstract

from arXiv · show

Humans are able to perform a myriad of sophisticated tasks by drawing upon skills acquired through prior experience. For autonomous agents to have this capability, they must be able to extract reusable skills from past experience that can be recombined in new ways for subsequent tasks. Furthermore, when controlling complex high-dimensional morphologies, such as humanoid bodies, tasks often require coordination of multiple skills simultaneously. Learning discrete primitives for every combination of skills quickly becomes prohibitive. Composable primitives that can be recombined to create a large variety of behaviors can be more suitable for modeling this combinatorial explosion. In this work, we propose multiplicative compositional policies (MCP), a method for learning reusable motor skills that can be composed to produce a range of complex behaviors. Our method factorizes an agent's skills into a collection of primitives, where multiple primitives can be activated simultaneously via multiplicative composition. This flexibility allows the primitives to be transferred and recombined to elicit new behaviors as necessary for novel tasks. We demonstrate that MCP is able to extract composable skills for highly complex simulated characters from pre-training tasks, such as motion imitation, and then reuse these skills to solve challenging continuous control tasks, such as dribbling a soccer ball to a goal, and picking up an object and transporting it to a target location.

1 Introduction

MCP learns reusable motor primitives that can be transferred and multiplicatively composed into diverse behaviors. The method targets skill reuse for complex agents and reports strong transfer performance as task complexity grows.

  • Transferable skills can improve exploration and representations, but versatile reuse remains challenging for high-degree-of-freedom systems.The introduction contrasts learning from scratch with leveraging prior experience across new domains.
  • MCP learns reusable motor primitives that can be composed into a continuous spectrum of skills and transferred to new tasks.Its primitives can be recombined to yield different behaviors in a target domain.
  • Multiplicative composition activates multiple primitives simultaneously, providing a more flexible skill range than temporal sequencing alone.The method composes skills in space, whereas standard hierarchical models compose them in time.
  • Pre-training primitives on motion clips enables composed behaviors for challenging mobile manipulation tasks with complex simulated characters.The learned primitives show specializations resembling manually designed control structures and support structured exploration.
  • MCP substantially outperforms prior skill-transfer methods as task complexity grows and is the only approach to learn a successful policy on the benchmark’s most challenging task.

2 Preliminaries

The transfer framework pre-trains reusable skills for later tasks whose goals and rewards may differ. Existing additive hierarchical models activate one primitive at a time, creating a combinatorial limitation as simultaneous subtasks increase.

  • The framework trains an agent on pre-training tasks before applying learned skills to subsequent transfer tasks.Its objective is to acquire reusable skills that improve effectiveness on later tasks.
  • Successful transfer assumes related pre-training and transfer tasks, although pre-training need not cover every skill required downstream.This is an explicit scope condition for the transfer setting.
  • Additive hierarchical models select one primitive per timestep, restricting each action to a single primitive’s behavior.Temporal sequencing can produce complex behavior, but simultaneous subtasks remain constrained by the one-primitive selection.
  • As systems require more simultaneous subtasks, the combinations of behaviors become prohibitively challenging to model with single-primitive activation.

3 Multiplicative Compositional Policies

MCP composes multiple reusable action-distribution primitives simultaneously, then transfers fixed primitives to new tasks through a newly learned gating policy. Gaussian primitives yield a normalized Gaussian composite suitable for end-to-end continuous-control training.

  • 3 Multiplicative Compositional Policies: MCP activates multiple primitives simultaneously through multiplicative composition, enabling a continuous spectrum of skills.Each primitive contributes influence to the composite action distribution through a state- and goal-dependent weight.
  • 3 Multiplicative Compositional Policies: Unlike additive hierarchical policies, MCP combines primitive distributions before sampling rather than selecting one primitive at each timestep.The partition function normalizes the resulting composite distribution.
  • 3.1 Gaussian Primitives: Multiplicative composition of Gaussian primitives produces another Gaussian policy with component-wise means and variances.Independent Gaussian action parameters remain independent in the composite policy.
  • 3.1 Gaussian Primitives: Primitive variances can independently adjust each action parameter’s influence beyond the primitive’s overall gating weight.The composite policy can then be trained end-to-end with standard automatic differentiation.
  • 3.2 Pre-Training and Transfer: During transfer, MCP keeps primitive parameters fixed and trains a new gating policy to compose them for the target task.Pre-training learns the primitives from a shared set of tasks, while transfer changes the composition policy.
  • 3.2 Pre-Training and Transfer: The asymmetric model gives goals only to the gating function, encouraging primitives to specialize in distinct skills and supporting transfer across different goal spaces.The primitives depend only on the state, while the gating function observes state and goal.

4 Related Work

MCP builds on hierarchical and latent-space approaches to skill reuse while addressing their limitations in composing multiple primitives. It is evaluated on transfer tasks combining locomotion and object manipulation in complex simulated characters.

  • Hierarchical Policies: Hierarchical policies reuse low-level primitives through a gating function that selects a suitable primitive for each scenario.Standard hierarchical policies commonly train primitives on pre-training tasks before transferring them to new tasks.
  • Hierarchical Policies: Standard hierarchical policies activate one primitive at a time, limiting direct composition or interpolation of multiple skills.Their behaviors can still be sequenced temporally, but each timestep remains governed by a single primitive.
  • Latent Space Models: Latent-space models map latent representations to system controls and can support flexible skill interpolation through continuous latent variables.MCP represents latent skills as compositions of multiplicative primitives.
  • Transfer Tasks: The transfer tasks combine locomotion and object manipulation, requiring coordination across multiple body parts and temporally extended behaviors.Examples include carrying an object to a target and dribbling a ball to a goal.

5 Experiments

The experiments evaluate MCP for transferring motion-imitation skills to locomotion and manipulation tasks across simulated characters of increasing complexity. MCP improves transfer on challenging tasks, adapts to holdout directions, and learns primitives with distinct gait specializations.

  • Experimental Setup: Experiments test transfer across ant, biped, humanoid, and T-Rex characters, spanning locomotion and mobile manipulation tasks.The characters have 14, 23, 34, and 55 degrees of freedom, respectively.
  • Experimental Setup: Pre-training uses motion imitation, while transfer tasks include heading, carrying a box, and dribbling a soccer ball.Carry requires pickup, transport, and placement; dribble combines locomotion, balance, and contact-based ball manipulation.
  • Comparisons: MCP substantially improves learning speed and performance on challenging carry and dribble tasks, succeeding uniquely on Dribble: T-Rex.The comparison includes scratch, finetuned, hierarchical, option-critic, mixture-of-experts, and latent-space models.
  • Comparisons: MCP adapts quickly to holdout heading directions, whereas the latent space model overfits to pre-training directions.Scratch and finetune models also perform well because they operate directly in the underlying action space.
  • Primitive Specializations: During walking, primitives specialize by gait phase, with opposing activation patterns for left and right stance and distinct action clusters in PCA space.These specializations consistently appear across multiple training runs.

6 Conclusion

MCP learns transferable skills with multiplicative primitives that support structured exploration and novel combinations. It performs well on long-horizon tasks and improves relative to prior methods as task complexity grows, while reference-motion selection remains important.

  • 6 Conclusion: MCP learns and composes skills using multiplicative primitives that transfer to challenging continuous-control tasks.The primitives form a new action space for combining skills in novel ways.
  • 6 Conclusion: MCP enables more structured exploration and flexible combinations of primitives for eliciting new task behaviors.The conclusion characterizes this as a form of spatial abstraction.
  • 6 Conclusion: MCP is effective for long-horizon tasks and outperforms prior methods as task complexity grows.The paper identifies temporal abstractions as an important direction for future work.
  • 6 Conclusion: Pre-training requires care in selecting an expressive corpus of reference motions.Future work considers recovering sophisticated primitive skills without this supervision.

7 Additional Experiments

Additional experiments examine evaluation accounting and the number of primitives. They find that increasing primitive count can reduce learning efficiency, with 32 primitives exceeding the humanoid’s original action-space dimensionality.

  • Evaluation: Evaluation records environment timesteps rather than policy steps to measure the physical interactions required for learning.The hierarchical model selects a new primitive approximately every 30 environment timesteps, unlike the other models.
  • Primitive Count: Four, eight, sixteen, and thirty-two primitive MCP models are trained to assess the effect of primitive count.The experiments vary k across 4, 8, 16, and 32.
  • Primitive Count: No noticeable performance difference appears between 4 and 8 primitives, while learning efficiency decreases as primitive count increases.With 32 primitives, the weight dimension exceeds the humanoid’s 28-dimensional action space, reducing dimensionality-reduction benefits.

8 Reference Motions

The experiments use reference-motion data for pre-training, and the additional-experiments section includes a table describing learning curves for different primitive counts. The motion corpus contains walking and turning clips shared by the biped and humanoid.

  • Reference Motions: Biped and humanoid pre-training uses shared mocap clips of walking and turning motions from a publicly available database.The corpus contains 230 seconds of motion data, with upper-body joints removed when retargeting to the biped.

9 Transfer Tasks

The transfer tasks evaluate directional locomotion, soccer-ball dribbling, and long-horizon box transport. Their rewards combine movement toward objects or targets with proximity, velocity, and control-related objectives.

  • Heading: The heading task changes the target direction at every timestep and rewards the character for following it.The target direction is perturbed uniformly by ∇θt ∼ Uniform(−0.15rad, 0.15rad), and the goal is encoded as a horizontal-plane unit vector.
  • Carry: The carry task tests long-horizon mobile manipulation by requiring pickup, transport, and placement of a box.The box weighs 5kg; reward terms encourage approaching and staying near it, then moving it toward the target.
  • Dribble: Dribbling requires simultaneous locomotion, balance, and contact-based manipulation to move a soccer ball to a target.The ball and its target are randomly initialized within 10m-scale distances, while reward terms encourage approaching and staying near the ball and moving it toward the target.
  • Holdout: The holdout task evaluates Ant locomotion toward target directions that are excluded from pre-training.Pre-training uses directions θ̂ ∈ [0, 3/2π], whereas transfer uses directions sampled from θ̂ ∈ [3/2π, 2π].

10 Model Setup

The model setup compares MCP with scratch, finetuning, hierarchical, option-critic, mixture-of-experts, and latent-space alternatives. These models use PPO-based training or their respective option and latent-variable objectives, with asymmetric state-goal access used to support transfer.

  • Training: All models use PPO except option-critic, while value functions use TD(λ) and advantages use GAE(λ).The discount factor is γ = 0.95 during pre-training and γ = 0.99 during transfer tasks.
  • MCP: MCP uses a fully connected value function with 1024 and 512 hidden units followed by a linear output.Its listed hyperparameters are provided in Table 5.
  • Baselines: The scratch and finetuning baselines use two-layer 1024- and 512-unit ReLU policy networks with linear Gaussian-action outputs.Finetuning first imitates reference motion and then adapts to transfer tasks, adding goal inputs through input injection when needed.
  • Primitive-based baselines: The hierarchical, option-critic, and mixture-of-experts models represent policies through transferable low-level primitives and task-dependent selection mechanisms.The hierarchical model uses a PPO-trained softmax gating function; option-critic learns option specialization automatically; mixture-of-experts gates state-only primitives using state and goal inputs.
  • Latent-space baseline: The latent-space model encodes goals into Gaussian latent variables that condition the policy, with separate transfer encoders trained after policy pre-training.Pre-training embeds short motion clips, and the policy parameters are frozen while a new encoder is trained for each transfer task.
Loading 1905.09808v1…