Language as an Abstraction for Hierarchical Deep Reinforcement Learning

Yiding Jiang; Shixiang Gu; Kevin Murphy; Chelsea Finn

Source-linked AI summary

Language as an Abstraction for Hierarchical Deep Reinforcement Learning

Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn

arXiv:1906.07343v2Jun 18, 2019cs.LGcs.AIcs.CLstat.ML

TL;DR

The paper asks how RL agents can learn compositional, long-horizon tasks while generalizing to novel goals despite difficult hierarchical abstractions. It uses language between high- and low-level policies, trains the low-level policy with instruction relabeling, and reports broad success across long-horizon manipulation tasks with compositional generalization.

Problem
Compositional long-horizon skills and generalization to novel concepts remain open challenges, while effective general abstractions for hierarchical RL are difficult to acquire.
Method
HAL uses language as the abstraction between hierarchical policies and extends goal relabeling to language instructions describing regions of state space.
Results
The approach learns diverse long-horizon manipulation tasks, including object reconfiguration and sorting, while outperforming state-of-the-art RL and hierarchical RL approaches and generalizing to systematically different instructions.
Takeaways & Limitations
Compositional language supports efficient, flexible, human-interpretable hierarchical control and is critical for learning diverse sub-skills and systematic generalization in the studied tasks.
Takeaways & Limitations
The method relies on a language supervisor with access to scene-describing instructions and uses an instruction set specific to the problem domain.

Abstract

from arXiv · show

Solving complex, temporally-extended tasks is a long-standing problem in reinforcement learning (RL). We hypothesize that one critical element of solving such problems is the notion of compositionality. With the ability to learn concepts and sub-skills that can be composed to solve longer tasks, i.e. hierarchical RL, we can acquire temporally-extended behaviors. However, acquiring effective yet general abstractions for hierarchical RL is remarkably challenging. In this paper, we propose to use language as the abstraction, as it provides unique compositional structure, enabling fast learning and combinatorial generalization, while retaining tremendous flexibility, making it suitable for a variety of problems. Our approach learns an instruction-following low-level policy and a high-level policy that can reuse abstractions across tasks, in essence, permitting agents to reason using structured language. To study compositional task learning, we introduce an open-source object interaction environment built using the MuJoCo physics engine and the CLEVR engine. We find that, using our approach, agents can learn to solve to diverse, temporally-extended tasks such as object sorting and multi-object rearrangement, including from raw pixel observations. Our analysis reveals that the compositional nature of language is critical for learning diverse sub-skills and systematically generalizing to new sub-skills in comparison to non-compositional abstractions that use the same supervision.

1 Introduction

The paper addresses compositional, long-horizon RL by using language as the abstraction between hierarchical policies. HAL combines language-conditioned low-level control, high-level language actions, instruction relabeling, and empirical analysis of compositional generalization.

Motivation: Compositional long-horizon skills and generalization to novel concepts remain open challenges in deep reinforcement learning.The paper frames the central question as how language’s compositional structure can support long-horizon tasks and systematic generalization.
Approach: HAL uses language as the interface between high- and low-level policies in hierarchical reinforcement learning.The high-level policy produces language instructions, while the low-level policy follows them.
Approach: Language abstractions let low-level policies be reused across high-level objectives without retraining and make high-level actions human-interpretable.Instructions can represent regions of states satisfying abstract criteria rather than a single complete goal state.
Training: The method extends goal relabeling to language instructions, enabling low-level policies to learn from many language goals despite binary completion rewards.Language instructions describe regions of state space, rather than individual target states.
Evaluation: The authors introduce a CLEVR-inspired object-interaction environment and report success on vision-based long-horizon manipulation tasks including reconfiguration and sorting.They also report outperforming state-of-the-art RL and hierarchical RL approaches, compositionality advantages over non-compositional abstractions, and systematic generalization to new instructions.

2 Related Work

Prior hierarchical RL methods use hand-engineered, learned, or goal-oriented abstractions, while language-based approaches have not fully exploited compositional instruction representations. This work positions language instruction relabeling as a generalization of goal relabeling for hierarchical control.

Hierarchical reinforcement learning: Hierarchical RL studies abstractions for decomposing Markov decision processes into temporally extended policies.Classical work often learns high-level policies over hand-engineered low-level policies or flexible options.
Hierarchical reinforcement learning: End-to-end low-level learning from final task rewards can scale poorly to complex domains unless task distributions are carefully designed.Alternative approaches add auxiliary rewards based on diversity, domain knowledge, or goals.
Goal-oriented methods: Goal-oriented rewards provide inductive bias with relatively little domain-specific engineering and have improved performance across multiple domains.The paper characterizes its approach as a generalization of goal-oriented HRL methods.
Language-conditioned HRL: The method extends goal relabeling from achieved states to language statements describing regions of state space.This connects language-conditioned low-level training to multi-goal reinforcement learning.
Language-conditioned HRL: Earlier language-guided HRL methods used instruction representations with insufficient diversity to exploit language compositionality over one-hot goals.The paper distinguishes its compositional language abstraction from those prior representations.

3 Preliminaries

The preliminaries define standard RL, goal-conditioned RL, Q-learning, and hindsight experience replay as foundations for the paper’s language-conditioned hierarchical framework. These concepts formalize policies, goals, value functions, and sparse-reward relabeling.

Standard reinforcement learning: Standard reinforcement learning models an MDP with state space S, action space A, transitions T, rewards R, and discount factor γ.The objective is to learn a policy maximizing expected discounted future reward.
Goal-conditioned reinforcement learning: Goal-conditioned RL augments the MDP with a goal space G and conditions the reward and policy on a goal.The explicit distinction between state and goal supports later discussion of language-conditioned goals.
Q-learning: Q-learning estimates the optimal discounted return Q*(s_t,a_t) and recovers a policy by selecting the action with maximal Q-value.In high-dimensional settings, function approximators are fit from replay-buffer transition tuples.
Hindsight experience replay: Hindsight experience replay relabels trajectory transitions with states achieved later in the trajectory and adjusts rewards accordingly.The resulting denser signal can act as an implicit curriculum of increasing difficulty.

4 Hierarchical Reinforcement Learning with Language Abstractions

HAL uses compositional language as the interface between high- and low-level policies in hierarchical reinforcement learning. The low-level policy follows instructions while the high-level policy selects them to solve temporally extended tasks.

Framework: HAL trains a 2-layer hierarchy in which the high-level policy produces language instructions for the low-level policy.The framework is called Hierarchical Abstraction with Language (HAL).
Policy training: HAL uses grounded language supervision that maps observations to distributions of language statements describing them.This supervision supports training the instruction-following low-level policy.
Policy training: The low-level policy is trained to follow language instructions, while the high-level policy acts in the instruction space using environmental rewards.The two policies operate over the shared instruction set G but are trained separately.
Low-level learning: Hindsight instruction relabeling relabels trajectory states with language descriptions of achieved states to address sparse low-level rewards.Unlike direct reward optimization, the method provides training examples from many instructions at once.
Framework: Language instructions let the high-level policy take semantically meaningful actions that span multiple low-level actions.This provides a structured action space for long-horizon exploration.
High-level learning: The high-level policy can use a smaller instruction subset because many language instructions are redundant or synonymous.When this subset is small, the problem becomes a discrete-action RL problem.

5 The Environment and Implementation

The paper introduces a MuJoCo and CLEVR-based environment for compositional object manipulation and evaluates HAL on six temporally extended tasks. Experiments use both state-based and image-based observations with distinct action parameterizations.

Environment: The environment combines MuJoCo physics with the CLEVR language engine to generate scenes and language descriptions for object manipulation.Scenes vary in object number and visual and physical properties.
Tasks: The evaluation covers six tasks: object arrangement, object ordering, object sorting, color ordering, shape ordering, and color-and-shape ordering.The first three use the standard environment, while the latter three use a more diverse environment.
Implementation: State-based observations encode object locations in R^10 with 40 actions, while image-based observations use 64×64×3 renderings with 800 actions.Image-based actions select a location on a 10×10 grid and one of eight push directions.
Implementation: The low-level policy encodes instructions with a GRU and predicts action Q-values, while the high-level policy is also a neural-network Q-function trained with Double DQN.The high-level action space contains 80 instructions in the standard environment and 240 in the diverse environment.

6 Experiments

The experiments evaluate language-based hierarchical reinforcement learning across representation scaling, systematic generalization, pixel observations, and long-horizon control tasks. Results consistently support compositional language and relabeling, while HAL solves tasks that competing approaches often cannot.

Experimental design: The experiments compare language with one-hot, non-compositional, and bag-of-words instruction representations, while also testing systematic generalization and vision-based observations.The full HAL method is additionally compared with DDQN, HIRO, and Option-Critic on temporally extended tasks.
Low-level policy: One-hot performance deteriorates as instruction sets grow, whereas language leverages shared structure and can improve with more instructions.The comparison uses increasing instruction-set sizes and identical instruction relabeling where applicable.
Low-level policy: The non-compositional representation makes no meaningful progress despite identical supervision, indicating that compositionality is critical for effective instruction learning.Without HIR relabeling, the reward signal is significantly sparser, further reducing performance.
Vision-based learning: With pixel observations, the language-based low-level policy reaches state-based performance after longer convergence, while one-hot achieves much worse performance with equal experience.The experiments also evaluate image-based low-level policies against one-hot and non-compositional representations.
Low-level policy: Language outperforms the non-compositional representation in zero-shot instruction generalization, including settings where training and test distributions have disjoint support.The systematic split withholds a word-position combination during training and evaluates it only at test time.
High-level policy: HAL solves all three sparse-reward high-level tasks consistently, with lower variance and better asymptotic performance than the compared baselines, and transfers to images without performance loss.DDQN solves only two of three tasks, while HIRO and Option-Critic do not learn a reasonable policy; in diverse pixel settings, high-level learning remains difficult and had not converged.

7 Discussion

The discussion presents HAL as an efficient, flexible, and human-interpretable language abstraction for hierarchical reinforcement learning, while identifying constraints in supervision, instruction diversity, and language generation.

Discussion: HAL uses language abstractions to provide efficient, flexible, and human-interpretable representations for long-horizon hierarchical control.The framework leverages relabeling and language compositionality to train language-conditioned low-level policies without engineered reward shaping.
Discussion: A language model can replace fixed instruction selection, potentially enabling more powerful and general-purpose high-level policies.Fixed instruction sets limit flexibility and create scaling issues for complex instructions.
Discussion: The language model is trained with disentangled embeddings, a generator, discriminator, decoder, and mutual-information objectives.The latent vector combines Gaussian noise with a uniformly sampled latent code, while the generator maps it to sentence representations.
Discussion: The generator objective combines adversarial, Huber reconstruction, and vector-quantization terms, with Huber loss preferred over squared error for stability.The discriminator uses a hinge-style objective, and the mutual-information constraint prevents collapsed embeddings.
Discussion: A high-level SAC policy outputs a continuous action that the generator maps into an instruction executed by the low-level policy.This separates language generation from low-level instruction execution in the hierarchical pipeline.

A.4 Experiments

The experiments use a procedurally generated object-interaction environment to test compositional, sparse-reward, temporally extended manipulation tasks across fixed and diverse object settings.

Experiments: The language-model agent performs comparably to the fixed-instruction agent on ColorSort.The comparison uses 10,000 randomly sampled instructions in the diverse setting.
Environment: The environment contains up to five objects represented by positions and one-hot color, shape, size, and material properties.An internal relation graph stores neighboring objects in four cardinal directions.
Environment: The point-mass agent manipulates objects, while unsatisfied language statements are re-evaluated after each interaction for trajectory relabeling.The point-mass can teleport, and statements becoming true can serve as goals or instructions.
Tasks: The diverse setting combines three shapes with five colors and evaluates color ordering, shape ordering, and joint color-and-shape ordering.Colors and shapes may repeat, but identical color-shape combinations do not, producing 3003 possible configurations.

B.3 Implementation details

The implementation builds hierarchical policies around language-conditioned low-level control, object-centric or image-based observations, and a high-level policy that selects instructions.

Policy architecture: Language statements are generated from functional programs executed on the scene’s relation graph.The programs query object properties and represent varied statements inspired by CLEVR.
State-based policy: The state-based low-level policy represents scenes as variable-sized sets of object states and actions.Each object has a state vector, and each action is represented by an object-specific feature.
State-based policy: A goal-conditioned self-attention UVFA embeds instructions, attends over pairwise object relations, and outputs state-action values.Shared networks process object pairs and combine instruction and scene representations into per-object values.
Vision-based policy: The vision-based policy uses 64 × 64 RGB images and discretizes the 2D action space into a 10 × 10 grid.The image encoder is FiLM-conditioned on the instruction embedding and feeds factorized action groups.
Limitations: The fixed-instruction high-level policy selects each sentence independently and therefore does not exploit language structure.The paper identifies language generation as an important direction for future work.
Hierarchical training: The overall hierarchy trains a low-level policy π_l(a|s,g) and a high-level policy π_h(g|s), with the high-level policy selecting instructions.The high-level policy uses Double DQN and rolls out the low-level policy for five steps per instruction.

C.2 Training the low-level policy

Both state-based and vision-based low-level policies use DDQN, with an additional movement reward in vision experiments to alleviate exploration in a high-dimensional action space.

Training: Both state-based and vision-based low-level policies are trained with DDQN.State-based experiments use binary completion rewards, while vision-based experiments add a movement bonus.
Training: Vision-based training adds a 0.25 reward when object positions change by a minimum threshold.The bonus addresses exploration in the high-dimensional action space R800.

C.3 Training the high-level policy

The high-level policy is trained over language instructions, while hindsight relabeling supplies additional instruction-conditioned experience to the low-level policy. The relabeling strategy emphasizes instructions fulfilled in near-future transitions and augments learning with object-level instructions.

C.4 Relabeling Strategy: HIR relabels trajectories with language instructions that become fulfilled in future transitions, addressing the underdetermined notion of distance between instructions.The method avoids token-sequence distance metrics because they do not capture the relevant state information.
C.4 Relabeling Strategy: Algorithm 2 trains the low-level policy with off-policy RL by adding relabeled instruction transitions to a replay buffer.The procedure initializes π_l(a|s, g), samples instructions, records transitions, and updates the policy from minibatches.
C.4 Relabeling Strategy: Future relabeling samples up to K future transitions, selects an instruction newly satisfied there, and discounts its reward by the future time gap.The experiments use K = 4.
C.4 Relabeling Strategy: Unary Relabeling adds a reward-1.0 transition for an object’s name whenever that object is moved, helping the agent learn object concepts.The operation uses the object’s language description, such as “large rubber red ball.”

D Experimental Details

The experiments train high-level policies by sampling language instructions for repeated low-level rollouts, while comparing language embeddings with alternative representations and HRL baselines. The setup also documents substantial training costs and difficulties in making algorithm comparisons equivalent.

D Experimental Details: Each of 600 instructions receives 1, 4, 10, or 20 one-hot bins, producing effective vector sizes of 600, 2400, 6000, or 12000.Goals are uniformly assigned to one of their corresponding bins and embedded with a two-layer MLP.
D Experimental Details: The high-level policy samples an instruction from its instruction set, executes the low-level policy for T′ substeps, stores the resulting experience, and updates with an RL algorithm.The high-level policy uses an instruction encoder and the trained low-level policy.
D.2 Non-compositional representation: The non-compositional comparison uses a lossless Seq2Seq autoencoder to encode the 600 instructions into continuous vectors while preserving their information content.Adding regularization, including dropout or a smaller bottleneck, decreases the resulting representation’s performance.
D Experimental Details: Bag-of-words represents each instruction by normalized token frequencies and embeds the resulting vector with a two-layer MLP.Token order is discarded in this representation.
D Experimental Details: HIRO uses an R10 object-location goal representation in this comparison and was unable to make good progress, while Option-Critic is evaluated with 8, 16, and 32 options.The authors note that different HRL algorithms are difficult to compare in an apples-to-apples manner.
D Experimental Details: Training takes about 2 days for state-based low-level and high-level policies, 6 days for image-based low-level policies, and 3 days for image-based high-level policies.All experiments run on a single Nvidia Tesla V100, and implementations are not deliberately optimized for performance.

E More Experimental Results and Discussions

On the diverse environment, training is slower than with a fixed number of objects, reflecting the greater diversity of the visual tasks. In visual-domain comparisons, DDQN fails to solve the problems, whereas language-space exploration provides more structured exploration reward.

E.1 Low-level policy for a diverse environment: Training performance is worse in the diverse environment than with a fixed number of objects given the same amount of experience.The authors attribute this to the greater diversity and difficulty of the visual tasks.
E.1 Low-level policy for a diverse environment: Figure 10 reports training instructions per episode for the diverse environment.The figure concerns low-level-policy training results.
E.2 Proposed environment, sparse reward, and structured exploration: DDQN solves 2 cases in the state-based environment but none in the visual domain, while language-space exploration yields higher random-exploration reward under sparse rewards.The authors hypothesize that pixel observations and a 20× larger action space make DDQN exploration difficult.