Source-linked AI summary
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequences
Hongyuan Mei, Mohit Bansal, Matthew R. Walter
TL;DR
The paper addresses mapping ambiguous natural-language navigation instructions to action sequences for autonomous agents. It introduces an end-to-end bidirectional LSTM encoder-decoder with multi-level alignment over language and observable world state. The model achieves state-of-the-art single-sentence execution and competitive multi-sentence performance despite limited data and no specialized linguistic resources.
Problem
Robots need to understand and execute free-form navigational instructions, which are ambiguous and complex and must be interpreted from local observable environments.
Method
A bidirectional LSTM encoder-decoder uses multi-level alignment to map instructions and the current world state to executable action sequences.
Results
The model achieves a new state-of-the-art on single-sentence execution and competitive results on multi-sentence execution with very small training datasets and no specialized linguistic resources.
Takeaways & Limitations
End-to-end sequence-to-sequence learning can map natural-language route instructions to action plans without prior linguistic structure, resources, or annotation.
Takeaways & Limitations
The multi-sentence setting remains more challenging than single-sentence navigation because errors cascade across individual sentences.
Abstract
from arXiv · showhide
We propose a neural sequence-to-sequence model for direction following, a task that is essential to realizing effective autonomous agents. Our alignment-based encoder-decoder model with long short-term memory recurrent neural networks (LSTM-RNN) translates natural language instructions to action sequences based upon a representation of the observable world state. We introduce a multi-level aligner that empowers our model to focus on sentence "regions" salient to the current world state by using multiple abstractions of the input sentence. In contrast to existing methods, our model uses no specialized linguistic resources (e.g., parsers) or task-specific annotations (e.g., seed lexicons). It is therefore generalizable, yet still achieves the best results reported to-date on a benchmark single-sentence dataset and competitive results for the limited-training multi-sentence setting. We analyze our model through a series of ablations that elucidate the contributions of the primary components of our model.
Introduction
The paper targets robust robot execution of ambiguous, complex navigational instructions. It proposes an end-to-end LSTM sequence-to-sequence model with multi-level alignment and reports strong benchmark performance without specialized linguistic resources.
- Motivation: Natural-language navigation is difficult because instructions are ambiguous, complex, variable in detail, and contain long-term dependencies among language and actions.The challenge is especially pronounced in unknown environments and when the relevant hallway or destination is uncertain.
- Approach: The model learns to map free-form instructions to executable action sequences from raw training sequence pairs, without prior linguistic knowledge.It learns word meanings, spatial relations, syntax, and compositional semantics directly from the paired data.
- Approach: A bidirectional LSTM encoder and decoder represent the instruction and generate actions conditioned on the current world state.The recurrent architecture is intended to capture temporal dependencies in instruction and action sequences.
- Approach: The multi-level aligner uses high- and low-level input representations to focus on instruction regions relevant to the current action.This design addresses information loss from alignment methods that use only high-level language abstractions.
- Results: The model achieves the best reported results on the 2,000-pair single-sentence task and remains competitive on the limited-training multi-sentence task without specialized resources.The evaluation also includes ablations of the encoder, representations, alignment, and bidirectionality.
Related Work
Prior grounded-language methods commonly rely on parsers, manually defined features, or probabilistic mappings to connect instructions with world referents. This paper instead formulates route interpretation as end-to-end neural sequence-to-sequence learning with alignment over language representations.
- Grounded language learning: Grounded language acquisition addresses the symbol grounding problem by associating linguistic elements with corresponding manifestations in the external world.Earlier approaches used manually prescribed mappings, while later methods learned associations from language in perceptual contexts.
- Parser-based methods: One class of prior methods learns parsers that convert free-form instructions into formal language or executable action specifications.These parsers may be learned with supervised or weakly supervised signals from instructions, actions, and world representations.
- Referent-mapping methods: Another class maps utterances to object, location, and action referents using probabilistic models of word-to-world associations.Interpreting a new instruction becomes inference in the learned model.
- Sequence-to-sequence formulation: The paper treats route interpretation as end-to-end sequence-to-sequence learning without prior linguistic structure, resources, or annotation.This formulation is motivated by the success of neural sequence-to-sequence methods in machine translation.
- Model design: The encoder-aligner-decoder adds the observable world state as an input to every decoder step and aligns actions with relevant instruction portions.The model uses multi-level alignment rather than relying only on high-level hidden annotations.
Task Definition
The task is to generate an action sequence for a previously unseen natural-language instruction using only the agent’s local, observable environment. The benchmark contains noisy route instructions paired with paths in virtual hallway worlds.
- Task formulation: Each example contains a variable-length instruction x(i), corresponding action sequence a(i), and observable environment representation y(i).The model predicts a(i) from an unseen pair (x(i), y(i)).
- Task formulation: The task covers isolated sentences and full paragraphs, corresponding to single-sentence and multi-sentence navigation settings.The multi-sentence setting requires executing a full paragraph.
- Observability: The model observes only the local environment within the agent’s line of sight while interpreting each instruction.This limited observability is part of the task definition rather than access to a complete map.
- Dataset: The benchmark contains free-form route instructions and action sequences across three virtual worlds of interconnected hallways.Hallways are identified by floor patterns and wall paintings, with objects placed at intersections.
The Model
The model maps natural-language instructions and local world states to action sequences with a bidirectional LSTM encoder, multi-level alignment, and an LSTM decoder.
- Architecture: The model treats route following as inference over action sequences conditioned on the instruction sequence and observable world states.It models P(a1:T | y1:T, x1:N) and selects actions by maximizing the learned posterior.
- Architecture: A bidirectional recurrent encoder converts the instruction into hidden annotations, while an aligner produces a time-specific context vector for decoding.The decoder uses the aligned context to represent the instruction at each action step.
- LSTM and Training: The model uses LSTMs to capture long-term dependencies across instruction and action sequences without exploding or vanishing gradients.The encoder is bidirectional, and the model is trained end-to-end with negative log-likelihood and back-propagation.
- Multi-level Aligner: The multi-level aligner combines word vectors and encoder states so the decoder can use both low-level words and high-level contextual representations.Including original word representations helps match salient instruction words directly to landmarks in the current world state.
- Decoder: The LSTM decoder conditions the next-action distribution on the previous decoder state, current world state, and aligned instruction context.Inference maximizes the conditional posterior to generate the desired action sequence.
- Inference: During inference, beam search maintains the k best hypotheses for single sentences and searches multi-sentence instructions sentence by sentence.A sequence ends when the decoder emits the “stop” action.
Experimental Setup
The experiments use the SAIL route-instruction corpus, represent local visible environments as directional bag-of-words states, and evaluate exact execution on single- and multi-sentence tasks.
- Dataset: The SAIL dataset contains 706 navigational instruction paragraphs from six instructors across 126 start-end pairs in three virtual worlds.The authors retain raw instructions, including spelling and grammatical errors.
- World State: The world state represents line-of-sight floor patterns, wall paintings, and unoccluded objects using directional bag-of-words vectors.The represented directions are forward, left, and right.
- Evaluation: Single-sentence success requires exactly matching the demonstrated final position and orientation, whereas multi-sentence evaluation disregards final orientation.The multi-sentence setting remains more challenging because errors can cascade across sentences.
- Training Protocol: The study uses three-fold cross-validation by map, with separate training and validation partitions for each held-out test map.A second vTest procedure uses the test map to choose the stopping iteration for comparison with some prior methods.
- Optimization: Training usually converges within 50 epochs using Adam, with early stopping based on the validation task metric.Validation log-likelihood was not well correlated with the task metric.
Results and Analysis
The model surpasses prior single-sentence results and remains competitive on multi-sentence navigation with limited data, while ablations show benefits from alignment, bidirectionality, and sentence encoding.
- Primary Result: 68.05 vDev and 70.56 vTest accuracy on single-sentence navigation surpass state-of-the-art results without specialized linguistic resources.The model also achieves 23.93 vDev and 27.91 vTest accuracy on the multi-sentence task, outperforming several specialized-resource methods and remaining competitive with state-of-the-art.
- Distance Evaluation: The evaluation measures whether the action sequence reaches the exact desired destination, while distance analysis also examines results landing within d nodes.The model often produces action sequences that reach points close to the desired destination.
- Multi-level Aligner Ablation: The multi-level aligner significantly improves performance over a standard high-level aligner by incorporating original input words alongside encoder annotations.The alignment visualizations show word-to-action correspondences for sentences in the instruction paragraph.
- Aligner Ablation: Learning alignment improves action-sequence accuracy over a No Aligner model using non-learned uniform weights.The No Aligner model retains connections between instructions and actions but does not learn selective alignment weights.
- Bidirectionality Ablation: A bidirectional encoder significantly improves accuracy compared with a unidirectional forward encoder.The comparison is reported as a bidirectionality ablation in Table 2.
- Encoder Ablation: Encoding the input sentence into a context representation yields a substantial gain over directly feeding word vectors to the decoder.The encoder’s sequential processing incorporates sentence-level information that helps distinguish instructions such as “turn right before” versus “turn right after.”
Conclusion
The paper presents an end-to-end bidirectional LSTM sequence-to-sequence model with a multi-level aligner for mapping instructions to action plans. It achieves state-of-the-art single-sentence execution and competitive multi-sentence results despite small datasets and no specialized linguistic resources.
- The approach maps natural language navigational instructions to action plans from local observable world state using a bidirectional LSTM-RNN and multi-level aligner.
- The model achieves new state-of-the-art single-sentence execution and competitive results on the more challenging multi-sentence domain.
- The results hold with very small training datasets and no specialized linguistic knowledge or resources.