Source-linked AI summary
Gated-Attention Architectures for Task-Oriented Language Grounding
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, Ruslan Salakhutdinov
TL;DR
Task-oriented language grounding requires agents to connect natural-language instructions with visual elements and actions from raw observations. The paper proposes an end-to-end 3D architecture using Gated-Attention multimodal fusion and policy learning, and reports stronger generalization than concatenation-based models across unseen maps and instructions.
Problem
Agents must map natural-language instructions to visual elements and actions while recognizing objects, exploring partial views, grounding attributes, and navigating to the correct object.
Method
The paper uses an end-to-end architecture that combines instruction and image representations with multiplicative Gated-Attention fusion and learns policies through reinforcement or imitation learning.
Results
Gated-Attention models outperform concatenation models for Multitask and Zero-Shot Generalization across three difficulty modes.
Takeaways & Limitations
The model learns attribute-sensitive associations and supports task execution from raw pixels on unseen instructions and maps in a 3D environment.
Abstract
from arXiv · showhide
To perform tasks specified by natural language instructions, autonomous agents need to extract semantically meaningful representations of language and map it to visual elements and actions in the environment. This problem is called task-oriented language grounding. We propose an end-to-end trainable neural architecture for task-oriented language grounding in 3D environments which assumes no prior linguistic or perceptual knowledge and requires only raw pixels from the environment and the natural language instruction as input. The proposed model combines the image and text representations using a Gated-Attention mechanism and learns a policy to execute the natural language instruction using standard reinforcement and imitation learning methods. We show the effectiveness of the proposed model on unseen instructions as well as unseen maps, both quantitatively and qualitatively. We also introduce a novel environment based on a 3D game engine to simulate the challenges of task-oriented language grounding over a rich set of instructions and environment states.
1 Introduction
The paper addresses task-oriented language grounding by learning to map natural-language instructions and raw visual observations to actions in a challenging 3D environment. It proposes Gated-Attention fusion and evaluates generalization to unseen instructions and maps.
- Task-oriented language grounding maps language to visual elements and actions so an agent can perform an instructed task.
- Agents must recognize objects from pixels, explore occluded or out-of-view environments, ground instruction concepts, interpret pragmatic language, and avoid incorrect objects.
- The architecture combines a state-processing module for joint instruction-image representations with a policy learner that predicts actions.
- Gated-Attention performs multimodal fusion through multiplicative interactions between language and image representations.
- The proposed end-to-end architecture requires no prior linguistic or perceptual knowledge and generalizes to unseen instructions and maps.
- The authors introduce a ViZDoom-based 3D environment with rich actions, objects, attributes, first-person views, and navigation scenarios.
2 Related Work
Prior work grounds language in objects, actions, or visual inputs, but often relies on semantic parsing, separate task policies, or 2D environments. This paper targets raw-pixel visual language grounding in a more challenging 3D setting.
- Prior research grounds language in objects, attributes, haptic signals, or robot actions through human-robot interaction and active learning.
- Semantic-parsing and neural approaches map navigational instructions to action sequences, while this paper emphasizes grounding visual attributes such as shape, size, and color.
- Earlier deep reinforcement-learning methods learned separate policies for individual tasks using raw visual information and deep Q-learning.
- Related instruction-execution studies include seen and zero-shot settings, but the cited 2D environments do not match this paper’s 3D setting.
- The paper addresses raw-pixel input, continuous agent positions, partial observability, and perception, exploration, and reasoning challenges without prior linguistic or perceptual knowledge.
3 Problem Formulation
The task is target-driven visual navigation conditioned on a natural-language description of an object. At each timestep, the agent observes first-person pixels and the instruction, then selects actions intended to reach the correct object.
- The agent receives a natural-language instruction describing a target object and must navigate to that object in an episodic environment.
- At each timestep, the state consists of the raw pixel-level first-person image and the language instruction.
- The learned policy maps observed states to actions that should complete the task by reaching the correct object before termination.
- Episodes terminate when the agent reaches any object or exceeds the maximum episode length.
- The formulation considers imitation learning with an oracle and reinforcement learning for learning the action policy.
4 Proposed Approach
The proposed architecture jointly represents raw visual observations and natural-language instructions, then learns actions through imitation or reinforcement learning. Its Gated-Attention fusion uses instruction-conditioned multiplicative interactions with convolutional image features and remains end-to-end differentiable.
- State Processing Module: The model processes images with a convolutional network and instructions with a GRU before forming a joint multimodal state representation.The policy learner uses this representation to predict the action at each timestep.
- Concatenation: Concatenation flattens and joins image and instruction representations as the baseline multimodal fusion method.This baseline follows prior methods cited by the paper.
- Gated-Attention: Gated-Attention transforms the instruction embedding into a sigmoid attention vector, expands it spatially, and multiplies it element-wise with convolutional feature maps.The attention vector has dimension d, matching the number of convolutional feature maps.
- Gated-Attention: The fusion mechanism uses multiplicative interactions between instruction representations and convolutional image feature maps, extending gated attention to verbal-visual multimodal fusion.The unit is differentiable, enabling end-to-end training.
- Gated-Attention: Gated attention lets the agent select visual attributes relevant to instructions such as color, shape, or their combination.Different instructions can require attending to green objects, pillars, or green pillars.
- Policy Learning Module: The policy-learning module supports Behavioral Cloning and DAgger for imitation learning, or A3C-based reinforcement learning with recurrent memory.The A3C pathway uses an LSTM to remember previously observed objects when exploration leaves them out of view.
5 Environment
The authors build a ViZDoom-based 3D environment in which agents execute language instructions by navigating to described objects. Randomized object configurations and Easy, Medium, and Hard spawning modes create varied task settings.
- Environment Design: The environment is built on the ViZDoom API and provides raw first-person visual information for instruction-following tasks.Successful completion requires reaching the object described by the instruction.
- Environment Design: Its customizable scenarios vary difficulty and support multiple object arrangements for task-oriented language grounding.The authors connect this flexibility to studying multi-task and zero-shot reinforcement learning.
- Instructions and Episodes: Each instruction combines one action, one object, and one or more attributes, while episodes randomize correct and incorrect objects and their locations.The environment provides 70 manually generated instructions and multiple episodes per instruction.
- Difficulty Modes: Easy uses fixed agent and candidate-object locations, Medium randomizes object locations within view, and Hard randomizes both agent and objects with possible visibility constraints.These modes progressively vary the starting configuration and whether candidate objects are visible.
6 Experimental Setup
The experiments evaluate language grounding across difficulty modes, generalization settings, and learning algorithms in a ViZDoom-based 3D environment. Models process raw visual and linguistic inputs and compare Gated-Attention with concatenation baselines.
- Evaluation Setup: Training uses 55 instructions, while 15 unseen attribute-object combinations are held out for zero-shot evaluation.Each episode contains one correct object, four incorrect objects, and the agent.
- Evaluation Setup: Experiments cover easy, medium, and hard environments and report model accuracy averaged over 100 episodes.A3C experiments use parallel threads and optimize value-function and policy-gradient losses.
- Evaluation Setup: Multitask Generalization evaluates unseen maps with training instructions, whereas Zero-shot Task Generalization evaluates unseen instructions and unseen maps.Accuracy measures reaching the correct object before the episode terminates.
- Model Comparisons: The baseline uses concatenation multimodal fusion, with matched CNN, GRU, and policy-learning architectures for fair comparison.A3C-Concat and BC-Concat serve as reinforcement-learning and imitation-learning baselines.
- Architecture and Learning: The network receives a 3x300x168 RGB image and an instruction encoded by a 256-unit GRU.Visual processing uses successive convolutional layers, while policy modules differ between reinforcement and imitation learning.
7 Results & Discussions
Gated-Attention models outperform concatenation models across multitask and zero-shot generalization, with stronger learning and accuracy in difficult settings. Attention visualizations suggest that the model represents object types and attributes, while reinforcement learning supports exploration under occlusion.
- Performance: 83% and 73% accuracy are achieved by GA in hard-mode Multitask and Zero-Shot Generalization, versus 24% and 12% for Concat.A3C models with GA also learn faster and converge to higher accuracy than Concat models.
- Performance: GA models outperform Concat models for both Multitask and Zero-Shot Generalization.The comparison covers the reported model families and environment difficulty settings.
- Learning Behavior: A3C with Gated-Attention is more robust to agent location because reinforcement learning explores more extensively and covers more state trajectories.Imitation learning performs poorly in medium and hard settings where exploration is needed.
- Policy Execution: A hard-mode execution explores with a 300 degree turn, handles initial occlusion, distinguishes short from tall green torches, and avoids the distractor.The trajectory reaches the short green torch despite no target initially appearing in the field of view.
- Attention Analysis: Attention dimensions cluster by object type and color, while instructions without a color refer across clusters associated with object type.Test instructions also receive attention patterns indicating recognition of unseen object attributes.
8 Conclusion
The paper presents an end-to-end architecture for grounding language in raw pixels in 3D environments using Gated-Attention fusion. Across reinforcement and imitation learning, the approach outperforms concatenation for multitask and zero-shot generalization and learns object attributes.
- Conclusion: The architecture supports task-oriented language grounding from raw pixels in a 3D environment for reinforcement and imitation learning.Its joint state representation uses multiplicative interactions between instruction and image representations.
- Conclusion: Gated-Attention models outperform concatenation models across Multitask and Zero-Shot task generalization and three difficulty modes.The result applies to A3C, Behavioral Cloning, and DAgger models using the respective fusion units.
- Conclusion: Attention-weight visualizations indicate that the agent recognizes object, color, and size attributes.These visualizations provide qualitative evidence about the learned multimodal representation.
A Doom objects
The ViZDoom environment contains objects varying in color, size, and type for task-oriented language-grounding experiments. Available object types include Columns, Torches, Armors, and Keycards.
- Environment Objects: The environment supports spawning several objects with varied colors and sizes.These variations provide attributes used in the grounding tasks.
- Environment Objects: The available object types are Columns, Torches, Armors, and Keycards.The experiments use several of these object categories.
B Instructions
The system uses 70 navigational instructions, each formed as “Go to the X,” with X drawn from the entries listed in Table 2.
- The training and testing set contains 70 navigational instructions.
- Each instruction follows the template “Go to the X,” where X is an entry from Table 2.
C Attention Maps
The attention maps group instructions by description or color, allowing comparison of attention-vector outputs across these attribute-based groupings.
- Attention maps are organized by instruction description and by color.
- Figure 11 presents attention-vector outputs for instructions grouped by description, with test instructions marked by asterisks.
- Figure 12 presents attention-vector outputs for instructions grouped by color, with test instructions marked by asterisks.