Source-linked AI summary
Grounded Language Learning in a Simulated 3D World
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, Phil Blunsom
TL;DR
Grounding language in the physical world remains difficult for systems that rely on hard-coded or limited representations. The paper trains an agent in a simulated 3D environment using reinforcement and unsupervised learning, finding that it generalises language beyond prior experience and learns new words faster as semantic knowledge grows.
Problem
Grounding language in the physical world is difficult, while existing rule-based approaches do not generalise or scale beyond programmed domains.
Method
An agent learns to execute written instructions in a simulated 3D world by combining reinforcement learning with unsupervised learning from visual and linguistic inputs.
Results
The agent grounds language in objects, relations, actions, and plans, interprets novel word combinations and situations, and learns new words faster as semantic knowledge grows.
Takeaways & Limitations
The learned semantic representations support transfer to unfamiliar settings, multiple distinct tasks, and generalisation beyond the agent’s prior experience.
Abstract
from arXiv · showhide
We are increasingly surrounded by artificially intelligent technology that takes decisions and executes actions on our behalf. This creates a pressing need for general means to communicate with, instruct and guide artificial agents, with human language the most compelling means for such communication. To achieve this in a scalable fashion, agents must be able to relate language to the world and to actions; that is, their understanding of language must be grounded and embodied. However, learning grounded language is a notoriously challenging problem in artificial intelligence research. Here we present an agent that learns to interpret language in a simulated 3D environment where it is rewarded for the successful execution of written instructions. Trained via a combination of reinforcement and unsupervised learning, and beginning with minimal prior knowledge, the agent learns to relate linguistic symbols to emergent perceptual representations of its physical surroundings and to pertinent sequences of actions. The agent's comprehension of language extends beyond its prior experience, enabling it to apply familiar language to unfamiliar situations and to interpret entirely novel instructions. Moreover, the speed with which this agent learns new words increases as its semantic knowledge grows. This facility for generalising and bootstrapping semantic knowledge indicates the potential of the present approach for reconciling ambiguous natural language with the complexity of the physical world.
1. Introduction
The paper addresses the challenge of grounding language in the physical world without laborious rule coding. It proposes learning end-to-end in a continuous situated environment from pixel-level visual input.
- Grounded language understanding remains difficult because rule-based systems require laborious linguistic and physical knowledge coding.
- The proposed paradigm learns language understanding in a continuous, situated environment rather than relying on conventional rule-based approaches.
- Agents learn end-to-end grounding for linguistic expressions while completing tasks from pixel-level visual input.
- Initial experiments use a 3D environment where agents find and pick up objects from textual task descriptions.
2. Related work
Earlier grounded-language systems commonly used simpler environments, structured representations, modular pipelines, or fixed datasets. This work instead targets language grounding in continuous perceptual input through interactive learning.
- Learning semantic grounding without prior knowledge is difficult because linguistic expressions can have limitless possible referents.
- Prior grounded-language systems often operated in environments simpler than the continuous, noisy sensory experience encountered by humans.
- Semantic parsing typically maps language to formal semantic representations grounded in databases or knowledge graphs.
- Related systems connected language to perception or actions using modular pipelines.
- Other communication approaches grounded language in highly structured environments rather than continuous perceptible input.
- Image-captioning models generally learn from fixed datasets, unlike the interactive approach presented here.
3. The 3D language learning environment
The experiments place an agent in a configurable 3D world with continuous visual input and textual instructions. Tasks require active exploration, object identification, and generalisation across varied layouts and language combinations.
- The environment combines a language channel with a 3D simulated world where the agent receives continuous visual input and textual instructions.
- The agent actively explores by controlling its visual field and movement through the surroundings.
- Training episodes vary object shapes, colours, patterns, and positions while instructions identify target objects.
- Even simple reference tasks require hundreds of inter-dependent actions while learning compositional meanings tied to visual features.
- Additional tasks test predicate generalisation, productive phrase composition, and grounding language in relations and actions.
4. Agent design
The agent is a recurrent multimodal policy network that combines visual and linguistic inputs with reinforcement learning and auxiliary unsupervised objectives. Its design uses exploration, prediction, and shared representations to learn from sparse task rewards and immediate perceptual feedback.
- Visual input is encoded by a convolutional module, language by a recurrent module, and both are combined before a recurrent action module.
- The action module produces a probability distribution over motor actions and estimates state value for reinforcement learning.
- The network is organized into interconnected modules supplemented by auxiliary learning objectives.
- The policy is optimized toward higher discounted returns using Advantage Actor Critic.
- Auxiliary objectives address the weakness of learning only from infrequent object-selection rewards by exploiting immediate perceptual feedback.
- Temporal autoencoding predicts the next visual observation from the current observation and action, training shared visual and policy representations.
- Language prediction estimates instruction words from visual observations using parameters shared with the vision and language modules.
5. Experiments
The experiments show that grounded language learning benefits from auxiliary unsupervised objectives, existing semantic knowledge, compositional generalisation, and curriculum training. Agents learned to apply language to unfamiliar objects, phrases, and increasingly complex action-based tasks.
- 5.1 Role of unsupervised learning: Reinforcement learning alone produced no learning after millions of episodes, whereas combining reward prediction, value replay, language prediction, and temporal autoencoding produced the fastest vocabulary learning.The vocabulary task used single-word instructions to identify one of two objects in a room.
- 5.2 Word learning speed experiment: Agents with prior words learned new vocabulary faster than agents trained from scratch, and learning speed appeared to accelerate as more semantic knowledge was acquired.The comparison indicates that initial training also develops visual and motor capabilities, while existing lexical knowledge supports later word learning.
- 5.3 One-shot learning experiments: The agent productively composed familiar colour and shape words into novel phrases and inferred constituent concepts even when trained only on multi-word instructions.In the decomposition-composition condition, generalisation occurred after fewer training instances because each bigram conveyed more information.
- 5.3 One-shot learning experiments: Knowledge of larger and smaller transferred from trained shapes to unfamiliar shapes, while lighter and darker generalised above chance but less than perfectly to unfamiliar colours.The paper suggests that inferring human-like lightness relations in RGB space was especially difficult from few examples.
- 5.5 Multi-task learning: Curriculum training enabled transfer from simpler settings to more complex room-retrieval tasks and supported grounding language in actions, plans, and relationships.A single agent learned Selection, Next to, and In room tasks; transfer from a single-room setting eventually enabled learning in a larger two-room environment.
6. Conclusion
The agent learns grounded language in a simulated 3D world without pre-programmed knowledge, linking language, vision, and action. Its semantic representations support generalisation, knowledge reuse, and curriculum-based learning.
- 6. Conclusion: The agent executes multi-word instructions in a simulated three-dimensional world without pre-programming or hard-coded knowledge.Its distributed knowledge spans language, vision, and policy networks, covering modifiers, relations, actions, and concrete objects.
- 6. Conclusion: Its semantic representations interpret novel word combinations, apply known modifiers and relations to unfamiliar objects, and reuse existing knowledge while learning new concepts.
- 6. Conclusion: The agent exhibits active multimodal concept induction, transfer to unfamiliar settings, multiple-task learning, and synthesis of unsupervised and reinforcement learning.
- 6. Conclusion: Vocabulary growth accelerates, semantic knowledge is generalised and reused, and curriculum moderation improves learning outcomes in ways characteristic of human development.
A.1 Agent core
The agent’s core is a single neural architecture that combines visual perception, language encoding, multimodal mixing, and recurrent action selection. Different language encoders are used depending on whether word order matters.
- A.1 Agent core: The vision module processes each 84 × 84 RGB first-person observation with a three-layer convolutional network, producing vt ∈R64×7×7.
- A.1 Agent core: The language module encodes instruction words with an LSTM for order-sensitive tasks or a bag-of-words encoder for other tasks.Both encoders use 128-dimensional word embeddings and produce lt ∈R128.
- A.1 Agent core: The mixing module flattens visual features and concatenates them with language features into a shared representation mt.
- A.1 Agent core: The action module updates its recurrent state using mt and the previous state, then supports policy-based motor-action selection.The default action-state dimension is d = 256.
A.2 Auxiliary networks
The auxiliary networks add predictive objectives that connect perception, action, and language. Temporal auto-encoding predicts future visual input after an action, while language prediction estimates instruction words from visual observations.
- A.2 Auxiliary networks: The temporal autoencoder receives consecutive visual data points and a one-shot action representation, encoding the first observation before action-conditioned transformation.
- A.2 Auxiliary networks: The transformed visual encoding is passed through a deconvolutional network to predict the next visual input.The temporal autoencoder is optimised using mean-squared error between prediction and target observation.
- A.2 Auxiliary networks: The language-prediction network applies a shared vision encoder to visual features, transforms them to 128 dimensions, and predicts vocabulary-word probabilities.
- A.2 Auxiliary networks: Language prediction computes negative log likelihood for an instruction word selected as the target at each time-step.
Appendix B. Environment details
The simulated environment consists of configurable rooms, objects, language instructions, and reward functions. Objects vary across internal properties and spatial context, while rewards are designed to discourage degenerate strategies.
- Appendix B. Environment details: A simulated level specifies a room map, object specifiers, language, and a reward function, with objects drawn from a fixed inventory.
- Appendix B. Environment details: Rooms have randomly sampled unique floor colours, and object placement or spawn points may be specified or randomly sampled.
- Appendix B. Environment details: An object can be described by up to seven factors, including five internal properties, its room, and proximity to another object.
- Appendix B. Environment details: Rewards are attached to picking up a particular object, scaled to [−10; 10], and balanced where possible so random-agent expected reward is 0.This prevents high performance through degenerate strategies that ignore textual grounding.
Appendix C. Hyperparameters
The appendix documents the hyperparameter settings used throughout the experiments, distinguishing fixed settings from randomly sampled settings for training replicas.
- Experimental settings: Tables 1 and 2 document the parameter settings used throughout the experiments.The reported confidence bands equal ± one standard deviation on the mean under a normal-distribution assumption.
- Fixed hyperparameters: Table 1 lists agent hyperparameters fixed throughout the experimentation.
- Sampled hyperparameters: Table 2 lists hyperparameters randomly sampled to produce different training replicas of the agents.The table defines uniform sampling over [x, y] and loguniform sampling over [x, y] in log-space, favoring lower values.