Source-linked AI summary
Modular Multitask Reinforcement Learning with Policy Sketches
Jacob Andreas, Dan Klein, Sergey Levine
TL;DR
The paper addresses whether hierarchical reinforcement learning can benefit from task structure without detailed subtask supervision. It learns modular subpolicies from ungrounded policy sketches by sharing parameters across task-specific policies, and reports better performance in sparse-reward multitask settings while inducing reusable behaviors.
Problem
The paper asks whether abstract policy structure is sufficient for hierarchical reinforcement learning without explicitly grounding high-level actions in environmental representations or intermediate rewards.
Method
The method associates each sketch symbol with a modular low-level subpolicy and jointly optimizes concatenated task-specific policies by tying parameters across shared subpolicies.
Results
The approach substantially outperforms purely unsupervised methods and prior explicit-decomposition approaches while learning shared behavior in sparse- and delayed-reward tasks.
Takeaways & Limitations
Policy sketches provide a minimal, ungrounded scaffold that induces reusable interpretable subpolicies for zero-shot generalization and hierarchical adaptation.
Takeaways & Limitations
The formulation assumes tasks are annotated with sketches consisting of sequences of symbolic labels drawn from a fixed vocabulary.
Abstract
from arXiv · showhide
We describe a framework for multitask deep reinforcement learning guided by policy sketches. Sketches annotate tasks with sequences of named subtasks, providing information about high-level structural relationships among tasks but not how to implement them---specifically not providing the detailed guidance used by much previous work on learning policy abstractions for RL (e.g. intermediate rewards, subtask completion signals, or intrinsic motivations). To learn from sketches, we present a model that associates every subtask with a modular subpolicy, and jointly maximizes reward over full task-specific policies by tying parameters across shared subpolicies. Optimization is accomplished via a decoupled actor--critic training objective that facilitates learning common behaviors from multiple dissimilar reward functions. We evaluate the effectiveness of our approach in three environments featuring both discrete and continuous control, and with sparse rewards that can be obtained only after completing a number of high-level subgoals. Experiments show that using our approach to learn policies guided by sketches gives better performance than existing techniques for learning task-specific or shared policies, while naturally inducing a library of interpretable primitive behaviors that can be recombined to rapidly adapt to new tasks.
1. Introduction
The paper asks whether abstract policy sketches can provide the benefits of hierarchical reinforcement learning without specifying how subtasks map to observations, actions, or intermediate rewards. It introduces modular policies tied across tasks and evaluates them on sparse-reward environments.
- Motivation: The central question is whether abstract policy structure alone can replace fine-grained supervision for learning hierarchical policies.The paper contrasts sketches with explicit high-level actions, subgoals, and behavioral primitives.
- Motivation: Policy sketches describe task components as short symbolic sequences while leaving their meanings in observations and intermediate rewards unspecified.Shared symbols can indicate shared high-level behavior across tasks without defining how that behavior is implemented.
- Approach: The architecture associates each high-level action with a low-level subpolicy and ties shared subpolicy parameters while jointly optimizing task-specific policies.This produces modular policies trained across multiple tasks and reward functions.
- Approach: The modular design naturally induces an interpretable library of policy fragments that can be recombined across tasks and data conditions.The evaluation includes joint multitask learning, zero-shot sketch following, and adaptation to new tasks.
- Evaluation: The framework is evaluated in crafting, maze navigation, and quadrupedal locomotion environments with discrete or continuous control and delayed sparse rewards.These tasks require resource collection, key-and-door navigation, or traversing a narrow winding cliff.
- Evaluation: In tasks requiring four or five high-level actions, random task-specific policies essentially never discover the final reward signal.The passage states that these tasks cannot be solved without considering their hierarchical structure.
2. Related Work
The paper relates policy sketches to hierarchical reinforcement learning, structured deep models, and instruction following. Its distinction is learning executable low-level behavior from sparse rewards rather than relying on grounded abstractions, demonstrations, or hard-coded interpreters.
- Hierarchical reinforcement learning: The approach instantiates the options framework while learning modular subpolicies in a multitask setting.The paper places its agent representation within the broader family of hierarchical reinforcement learners.
- Hierarchical reinforcement learning: Policy sketches provide less burdensome supervision than grounded subgoals or feature-abstraction hierarchies while improving over fully unsupervised approaches.The paper describes this as a sometimes dramatic improvement and emphasizes the lower burden for human providers.
- Hierarchical reinforcement learning: Unlike meta-level policy learning, which selects among an existing inventory of complex primitives, this work learns the named primitives themselves from sketches.The paper characterizes the two learning problems as direct duals.
- Hierarchical reinforcement learning: Compared with HAMs, the approach decouples policy representation from the value function and therefore avoids their strong reward-function and state-representation assumptions.The paper reports that HAMs perform less effectively with arbitrary state representations where those assumptions do not hold.
- Structured deep models: Unlike structured deep models and programmer–interpreters, the composed policies are learned through interaction with sparse rewards rather than direct output-action or execution-trace supervision.The related models discussed include question answering, relational reasoning, neural programmers, and programmer–interpreters.
- Instruction following: Unlike instruction-following systems that execute symbolic action sequences through hard-coded interpreters, this work learns to execute complex actions.The paper frames policy-sketch learning as complementary to natural-language instruction following.
3. Learning Modular Policies from Sketches
The framework learns modular subpolicies from ungrounded task sketches, composes them into task-specific policies, and trains shared behaviors across tasks with task-specific critics and curriculum learning.
- Model: Each sketch symbol b is assigned a shared subpolicy πb, allowing corresponding high-level behaviors to be learned across annotated tasks without explicit grounding.The subpolicy may be a neural network mapping states to distributions over low-level actions and STOP.
- Model: A task-specific policy concatenates sketch-ordered subpolicies, transferring control to the next subpolicy whenever STOP is emitted.This induces a policy over the augmented state space S × B and is semi-Markov when projected onto S.
- Policy Optimization: The modular actor–critic objective uses one subpolicy per symbol but one critic per task because shared subpolicies participate in policies with different reward functions.Subpolicy gradients sum expected-reward gradients over all tasks in which the symbol appears.
- Policy Optimization: Task-conditioned critics reduce variance during joint modular-policy learning, and experiments report noticeable performance improvements over state-only conditioning.Alternative advantage estimators can also be maintained separately for each task.
- Curriculum Learning: Curriculum learning begins with short-sketch tasks, increases the sketch-length limit after reward improves, and samples tasks in proportion to 1−ˆErτ.This balances learning from easy tasks against the risk of overfitting before harder tasks are introduced.
4. Experiments
Experiments evaluate modular learning from sketches across crafting, maze, and cliff environments, comparing it with hierarchical and policy-gradient baselines. The modular approach achieves higher reward, faster convergence, and better held-out-task generalization, while state- and task-dependent critics and curricula improve training.
- Environments: The experiments cover crafting, maze navigation, and cliff traversal, spanning discrete interaction and high-dimensional continuous locomotion.Crafting includes resource acquisition and tool construction; maze tasks require keys and doors; cliff tasks require quadrupedal navigation along winding paths.
- Baselines: The modular model is compared with option–critic, Q automaton, independent-policy, and joint-policy baselines across all three environments.The joint model conditions on task identity, while the Q automaton maps states and action symbols to Q values.
- Multitask learning: Across environments, modular learning substantially outperforms baselines, reaching higher average reward and converging faster than policy-gradient alternatives.After learning simple tasks, the model also rapidly adapts to more complex tasks, including ones containing previously unseen high-level actions.
- Ablations: A critic conditioned on both state and task identity converges faster than critics using either dependence alone or a constant baseline.After 3 million episodes, the constant-baseline approach achieves less than half the full critic’s overall performance.
- Ablations: Curriculum sampling based jointly on sketch length and task performance produces the best convergence among the tested curriculum variants.The length curriculum increases sketch lengths iteratively, while performance weighting samples tasks inversely to their accumulated reward.
- Zero-shot and adaptation: In held-out crafting tasks, baselines obtain negligible reward, whereas the modular model performs comparatively well in zero-shot and adaptation settings.The joint model overfits the training tasks, and the independent model cannot discover enough reward to learn during adaptation.
5. Conclusions
The paper concludes that symbolic policy sketches can guide multitask reinforcement learning without grounding high-level actions in observations or intermediate rewards. Shared modular subpolicies support reusable, interpretable behavior for zero-shot generalization and hierarchical adaptation.
- 5. Conclusions: Associating each sketch symbol with a modular neural subpolicy enables behavior sharing across tasks with sparse and delayed rewards.The learned inventory contains reusable policy fragments that can be recombined across tasks.
- 5. Conclusions: The resulting subpolicy inventory supports zero-shot generalization when new sketches are available and hierarchical reinforcement learning when they are not.The conclusion presents sketches as an effective scaffold requiring no environmental grounding.
A. Tasks and Sketches
The appendix lists the complete set of tasks, sketches, and symbols used in the paper. Tasks marked with an asterisk are held out for generalization experiments but included in multitask training.
- A. Tasks and Sketches: Tasks marked with an asterisk are held out for generalization experiments and included in multitask training experiments.The appendix provides the complete task, sketch, and symbol inventory.