Source-linked AI summary
Grounding Language to Autonomously-Acquired Skills via Goal Generation
Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, Olivier Sigaud
TL;DR
Language-conditioned RL can express abstract goals, but most approaches depend on external instruction and reward, and direct language conditioning limits behavioral diversity. This paper proposes LGB, which separates skill learning from language grounding through semantic goals, and implements it in DECSTR. The reported findings show that semantic representations support language grounding, diverse instruction-compatible behaviors, and strategy switching.
Problem
Most language-conditioned RL agents depend on external instructions and rewards, while direct language conditioning couples skill learning to grounding and limits behavioral diversity.
Source: §1 · p. 1Method
LGB separates skill learning from language grounding through semantic goals; DECSTR learns spatial-relation skills and trains a language-conditioned generator to produce compatible goals.
Source: §1 · p. 2, §3.3 · p. 4, §3.5 · p. 5Results
The authors report that semantic representations facilitate language grounding and that goal generation supports diverse compatible configurations and strategy switching after failure.
Source: §5 · p. 9Takeaways & Limitations
The paper concludes that intermediate semantic representations can connect language to autonomously acquired skills while allowing multiple behaviors for an instruction.
Source: §5 · p. 9The semantic predicate set is domain-dependent because it characterizes the behaviors the agent can explore.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
We are interested in the autonomous acquisition of repertoires of skills. Language-conditioned reinforcement learning (LC-RL) approaches are great tools in this quest, as they allow to express abstract goals as sets of constraints on the states. However, most LC-RL agents are not autonomous and cannot learn without external instructions and feedback. Besides, their direct language condition cannot account for the goal-directed behavior of pre-verbal infants and strongly limits the expression of behavioral diversity for a given language input. To resolve these issues, we propose a new conceptual approach to language-conditioned RL: the Language-Goal-Behavior architecture (LGB). LGB decouples skill learning and language grounding via an intermediate semantic representation of the world. To showcase the properties of LGB, we present a specific implementation called DECSTR. DECSTR is an intrinsically motivated learning agent endowed with an innate semantic representation describing spatial relations between physical objects. In a first stage (G -> B), it freely explores its environment and targets self-generated semantic configurations. In a second stage (L -> G), it trains a language-conditioned goal generator to generate semantic goals that match the constraints expressed in language-based inputs. We showcase the additional properties of LGB w.r.t. both an end-to-end LC-RL approach and a similar approach leveraging non-semantic, continuous intermediate representations. Intermediate semantic representations help satisfy language commands in a diversity of ways, enable strategy switching after a failure and facilitate language grounding.
1 INTRODUCTION
The paper seeks agents that autonomously acquire diverse skills while also learning from language commands. It proposes LGB, which separates skill learning from language grounding through semantic goals, and presents DECSTR as an implementation.
1 INTRODUCTION
- Goal-conditioned RL offers a route to self-supervised skill acquisition, while language can express abstract state constraints such as opening a red door.Prior goal-conditioned methods often represent goals as target features, including object positions, agent positions, or images.Source: §1 · pp. 1–2
- Most language-conditioned RL agents rely on external instructions and rewards, while direct language conditioning couples skill learning with grounding.The authors argue that this coupling limits pre-verbal-like goal-directed learning, behavioral diversity, and switching strategies after failure.Source: §1 · p. 2
- LGB uses language to generate sets of matching semantic configurations, allowing one instruction to correspond to multiple possible goals.Skills are learned by targeting configurations from the semantic space, separately from learning to generate goals that satisfy language constraints.Source: §1 · p. 2
- LGB separates intrinsically motivated skill acquisition from language grounding through an intermediate semantic goal representation.Its four intended properties include self-generated goals and rewards, behavioral diversity for instructions, and strategy switching after failures.Source: §1 · p. 2
- DECSTR learns spatial-relation configurations in a manipulation environment, then trains a conditional variational auto-encoder to generate language-matched semantic goals.The paper evaluates the composed phases for instruction following.Source: §1 · p. 2
2 RELATED WORK
The related work contrasts DECSTR's autonomous, semantic-goal approach with language-conditioned and block-manipulation methods that rely on external supervision, engineered curricula, or fixed representations.
2 RELATED WORK
- Prior language-conditioned RL generally uses external instructions and rewards and does not decouple language acquisition from skill learning; IMAGINE is an exception to the external-supervision pattern.The authors contrast this with LGB's language-conditioned goal generation, which supports behavioral diversity and strategy switching.Source: §2 · p. 2
- Block-stacking systems by Lanier et al. (2019) and Li et al. (2019) stack up to 4 and 6 blocks, respectively, but use hand-defined curricula and target block positions.DECSTR instead builds its own curriculum and uses spatial-relation semantic goals.Source: §2 · p. 3
- Lynch & Sermanet (2020) decouple language and image goals in a shared representation, but use teleoperated data and deterministic merging that limits behavioral diversity and strategy switching.Other cited semantic approaches use manually generated or hand-engineered skills, or optimal-control skills, rather than autonomously acquired skills.Source: §2 · p. 3
3 METHODS
The methods define LGB's semantic-goal pipeline and implement it as DECSTR for block manipulation. The agent learns goal-conditioned skills, learns to generate language-compatible goals, and is evaluated across skill learning, language grounding, and instruction following.
3 METHODS
- The methods specify a semantic representation, an intrinsically motivated goal-conditioned learner, and a language-conditioned goal generator, whose phases can be composed for instruction following.The three phases are G→B, L→G, and L→G→B.Source: §3 · p. 3, §3.1 · p. 3
3.1 THE LANGUAGE-GOAL-BEHAVIOR ARCHITECTURE
- LGB's semantic goals define the behavioral and goal spaces; its goal generator should model the distribution of semantic configurations satisfying language constraints.The architecture is intended to generate its own goals and rewards while supporting behavioral diversity and strategy switching.Source: §3.1 · pp. 3–4
3.2 ENVIRONMENT
- DECSTR operates in a Fetch-based manipulation environment with three blocks, and a social partner sometimes initializes non-trivial scenes during exploration.The scene initialization assistance is not provided during offline evaluations.Source: §3.2 · p. 4
3.3 SEMANTIC REPRESENTATION
- DECSTR represents scenes with nine binary close and above predicates over three objects, yielding 35 physically valid configurations and a reward for matching the goal configuration.The predicate set defines the explored behavioral space and replaces separately specified goal spaces and reward functions.Source: §3.3 · p. 4
3.4 INTRINSICALLY MOTIVATED GOAL-CONDITIONED REINFORCEMENT LEARNING
- The goal-conditioned learner uses SAC and HER, selecting only experienced semantic goals with sampling biased toward high absolute learning progress and low competence.Goals are grouped into five equally filled buckets ordered by discovery time, which serves as a proxy for difficulty; the bucket estimates use recent success histories.Source: §3.4 · pp. 4–5
- DECSTR's object-centered policy and critic share a network across object pairs to transfer skills, with input arrangement aligning symmetric above relations.The implementation uses a Deep-Sets adaptation and represents goals with six predicates: three close and three above.Source: §3.4 · p. 5
3.5 LANGUAGE-CONDITIONED GOAL GENERATION
- The language-conditioned goal generator learns semantic configurations from an initial scene and an instruction describing a change in an object-pair relation.It is trained with a C-VAE on 5,000 interaction-derived triplets; repeated sampling supports alternative compatible goals and logical combinations of instructions.Source: §3.5 · p. 5
3.6 EVALUATION OF THE THREE LGB PHASES
- Evaluation measures skill-learning success across goals, goal-generation precision and recall across five generalization subsets, and instruction following across single, sequential, and logically combined commands.Instruction following includes 1-shot and 5-shot settings, sequences without resets, and counts of distinct achieved configurations.Source: §3.6 · p. 6
4 EXPERIMENTS
The experiments evaluate DECSTR across skill learning, language grounding, and instruction following, then compare semantic intermediate goals with direct language conditioning and continuous goals. DECSTR shows benefits over direct language conditioning, while continuous goals learn configurations faster but perform poorly in language grounding.
4.1 HOW DOES DECSTR PERFORM IN THE THREE PHASES?
- DECSTR masters all reachable configurations in its semantic representation space, learning buckets in discovery order as it shifts from easier to harder goals.Its full version performs on par with expert-defined buckets and significantly outperforms the ablations.Source: §4.1 · pp. 6–7
- The language goal generator generalizes across five contexts, including new configurations, new sentences, and combinations of new sentences and initial configurations.It can generate more goals from known configurations than were observed in training, and generalize from new initial configurations.Source: §4.1 · p. 7
- In instruction following, DECSTR handles instructions and their logical combinations, succeeds in 14.9±5.7 successive instructions, and shows strategy switching and behavioral diversity.Across 10 seeds repeating an instruction 50 times, it produces at least 7.8 configurations, averaging 15.6 and reaching 23 depending on the instruction.Source: §4.1 · p. 7
4.2 DO WE NEED AN INTERMEDIATE REPRESENTATION?
- DECSTR outperforms the end-to-end LB baseline on transition and sequence instruction following, while LB cannot generalize to logical combinations and has lower behavioral diversity.LB succeeds in 3.01 ± 0.43 random instructions in a row versus 14.9 for DECSTR; on the same instructions, LB produces 4.2 configurations on average versus 17.1 for DECSTR.Source: §4.2 · p. 8
4.3 DO WE NEED A SEMANTIC INTERMEDIATE REPRESENTATION?
- LGB-C masters all 35 semantic configurations faster than DECSTR, but learns only one coordinate-placement skill and cannot build a repertoire of semantic skills.It reaches SR=95% in 708·103 episodes versus 1238·103 for DECSTR; DECSTR also moves blocks less (∆dist = 26 ± 5 cm) and reaches goals faster (∆steps = 13±4).Source: §4.3 · p. 8
- LGB-C maintains reasonable precision on the first two language-grounding test sets but has low recall across all sets, and its poor grounding prevents instruction following.The authors suggest that lacking semantic representations of skills might explain the difficulty of training its language-conditioned goal generator.Source: §4.3 · p. 9
5 DISCUSSION AND CONCLUSION
LGB separates autonomous skill acquisition from language grounding through semantic goals, and DECSTR demonstrates how this supports varied instruction-compatible behaviors. The discussion also identifies fixed-size configurations and limited social-partner involvement as boundaries of the current work.
5 DISCUSSION AND CONCLUSION
- LGB uses an intermediate semantic representation to decouple sensorimotor skill learning from language grounding; DECSTR demonstrates the architecture rather than defining its only possible implementation.The approach could also benefit from advances in GC-RL or generative models.Source: §5 · p. 9
- Predicate-based representations let DECSTR learn abstract goals opportunistically, facilitate language grounding, and require less domain knowledge to define than enumerating goals and reward functions.The predicate set remains domain-dependent and defines the behaviors the agent can explore.Source: §5 · p. 9
- A single instruction can yield many matching goal configurations, letting DECSTR switch strategies after failure and generalize to new sentences or initial configurations.The generator can also combine goal sets for logical expressions of instructions.Source: §5 · p. 9
- DECSTR is limited to fixed-size configuration spaces; future LGB implementations could represent variable-sized configurations with sets of predicate-and-object vectors.The proposed representation could target partial predicate sets rather than all relations among objects.Source: §5 · p. 9
- The social partner's role was limited to helping the agent experience non-trivial configurations and describing its behavior in simplified language.The authors identify more intertwined skill-learning and language-grounding phases as future work.Source: §5 · pp. 9–10
A LGB PSEUDO-CODE
- The LGB pseudo-code presents separate Goal→Behavior and Language→Goal phases, followed by instruction-conditioned behavior with a strategy-switching loop.The algorithms are described as high-level pseudo-code for the three phases.Source: A LGB PSEUDO-CODE · p. 14
B SEMANTIC PREDICATES AND APPLICATION TO FETCH MANIPULATE
- DECSTR represents three blocks with close and above predicates; close is symmetric, whereas above is asymmetric and based on vertical position.The predicates apply to red, green, and blue blocks.Source: B SEMANTIC PREDICATES AND APPLICATION… · p. 14
C.1 INTRINSICALLY MOTIVATED GOAL-CONDITIONED RL
DECSTR explores reachable semantic configurations with intrinsically motivated, goal-conditioned reinforcement learning. It combines self-generated goals and rewards, curriculum-based sampling, and an object-centered policy architecture to learn manipulation skills.
Sensorimotor learning
- The agent discovers reachable configurations by targeting sampled goals, then updates its curriculum strategy and policy as it gathers trajectories.The curriculum strategy biases which goals to target and learn about; the policy is updated through reinforcement learning.Source: §C.1 · p. 14
Goal-conditioned RL
- The learning problem is a goal-conditioned MDP whose state includes sensory observations and semantic configuration, with the goal space equal to the configuration space.The reward is sparse and binary; policy updates use SAC and Hindsight Experience Replay (HER).Source: §C.1 · p. 15
Policy updates
- HER substitutes an achieved configuration for the original goal in past transitions, allowing learning from failures while goal sampling favors configurations selected by the curriculum.The curriculum samples goals according to learning-progress-based probabilities.Source: §C.1 · p. 15
Object-centered architecture
- A shared network encodes each object pair independently, and summed pair-specific outputs feed the policy and critic, supporting skill transfer across pairs.The implementation uses permutations of object pairs because they worked better than combinations in practice.Source: §C.1 · pp. 15–16
Implementation details
- Sensorimotor training uses 24 parallel workers, each alternating two data-collection episodes with 30 updates; an epoch totals 2,400 episodes.The experiments required at least 24 CPU cores and ran for about 72 hours per run.Source: §C.1 · pp. 16–17
C.2 LANGUAGE-CONDITIONED GOAL GENERATOR
DECSTR trains a conditional goal generator from social-partner interactions, then evaluates goal coverage, generalization, and logical instruction combinations. The appendix also specifies an expert-bucket oracle for comparison with automatic curriculum generation.
C.2 LANGUAGE-CONDITIONED GOAL GENERATOR
- A conditional variational auto-encoder generates compatible semantic goals from an initial configuration and a sentence describing a relation change, using 5,000 interaction triplets for training.The training set contains initial and final configurations paired with descriptions of a predicate change.Source: §C.2 · p. 17
- The five test sets assess compatible-goal coverage and generalization across held-out configurations, sentences, and combinations of the two.In test set 1, 80% of compatible final configurations are absent from training.Source: §C.2 · pp. 17–18
- Logical-expression tests include conjunctions and combinations of conjunctions and disjunctions, with some instructions negated.The tests use 100 simple conjunctions, 200 mixed conjunctions, and 200 compound expressions.Source: §C.2 · p. 18
- The generator uses an encoder-decoder network with sentence and initial-configuration conditions, trained with binary cross-entropy plus a weighted KL-divergence loss.The selected loss weight is β = 0.6; the reported training setup uses 150 epochs.Source: §C.2 · p. 18
D.1 EXPERT BUCKETS ORACLE
- The expert-buckets oracle replaces DECSTR's automatic bucket generation with five predefined groups based on configuration types and difficulty.Defining these buckets requires prior knowledge of unreachable configurations; bucket choice is the only stated difference from DECSTR.Source: §D.1 · p. 19
D.2 LGB-C BASELINE
LGB-C is a decoupled baseline that uses continuous target coordinates rather than semantic configurations as goals. Its Figure 5 illustrates sampling target positions for each block.
D.2 LGB-C BASELINE
- LGB-C represents each goal with three-dimensional target coordinates for the three blocks rather than a semantic configuration.Its goal vector has size 9.Source: §D.2 · p. 19
- The baseline retains DECSTR's goal selection and uses randomly sampled target positions, while omitting conditioning on current semantic and goal configurations.It also uses an object-centered inductive bias and a C-VAE for language grounding.Source: §D.2 · p. 19
- Figure 5 shows an example of the LGB-C baseline sampling target positions for each block.Source: Figure 5 · p. 19
E.1 COMPARISON DECSTR - LGB-C IN SKILL LEARNING PHASE
The section compares DECSTR and LGB-C during skill learning, but their success measures are not directly equivalent.
E.1 COMPARISON DECSTR - LGB-C IN SKILL LEARNING PHASE
- Figure 6 compares average success over 35 valid configurations, but the two agents are evaluated against different goal definitions.DECSTR is evaluated on reaching semantic goals; LGB-C is evaluated on reaching sampled specific goals within a 5 cm error margin.Source: §E.1 · p. 20, Figure 6 · p. 20
- LGB-C can satisfy its specific target while missing the corresponding semantic goal because of the target error margins.Source: §E.1 · p. 20
E.2 AUTOMATIC BUCKET GENERATION.
Automatic buckets are continually reassigned as configurations are discovered, with easy and hardest goals tending to align with expert-defined difficulty groups.
E.2 AUTOMATIC BUCKET GENERATION.
- Discovered configurations are evenly spread across five buckets, and a configuration can change buckets as new configurations appear.The figures show epochs 1, 50, and 100 across 10 runs, with undiscovered configurations shown in grey.Source: §E.2 · p. 20
- By 100 epochs, DECSTR has discovered all configurations, while intermediate discovery varies across runs.After epoch 1, all configurations from expert buckets 1 and 2 were discovered; some runs found a few others.Source: §E.2 · p. 20
- Easy expert groups and the hardest configurations tend to map to corresponding automatic buckets, but medium-difficulty groups vary across runs.Expert buckets 3 and 4 show differing automatic assignments, suggesting their discovery order varies between runs.Source: §E.2 · p. 20
E.3 DECSTR LEARNING TRAJECTORIES
DECSTR’s learning trajectories are described as consistent across runs, with automatic buckets reflecting discovery and learning order.
E.3 DECSTR LEARNING TRAJECTORIES
- Figure 7’s pie charts encode bucket attribution across runs; colors identify buckets 1–5 and grey marks undiscovered configurations.Source: Figure 7 · p. 21
- DECSTR’s learning trajectories appear uniform across runs, and buckets are learned in increasing order.The trajectories track internal competence, learning progress, and associated sampling probabilities.Source: §E.3 · pp. 20–21, Figure 8 · p. 21
- Configurations discovered earlier enter lower-index buckets and are learned first, making discovery time a useful proxy for goal difficulty.Source: §E.3 · p. 21
- DECSTR performs much better than random-goal conditions, which the authors take as evidence that automatic bucket generation produces useful goal clusters.Source: §E.3 · p. 21