Talk the Walk: Navigating New York City through Grounded Dialogue

1 INTRODUCTION

• Talk The Walk brings perception, action, and interactive dialogue together in a navigation task, addressing limitations of static text-only datasets and prior tasks that omit one or more of these elements.The guide knows the target and has a map but not the tourist’s location; the tourist sees the street view but lacks the map and target.Source: §1 · p. 1
• MASC repeatedly conditions spatial dimensions of convolution on communicated messages, and the paper uses localization models to establish baselines for the full task.The introduction also reports that the best model exceeds human performance with perfect perception and learned emergent communication, while setting a non-trivial natural-language baseline.Source: §1 · p. 2

2 TALK THE WALK

• The environment uses manually captured 360-degree views of five NYC neighborhoods, represented as grids; tourist movement is simplified to left, right, and forward actions.The control problem is limited to a 2D grid to keep the focus on interactive dialogue.Source: §2 · p. 2, §1 · p. 2

2.1 TASK

• Each task samples a 4x4 grid, a random start and orientation, and a random target known only to the guide; success requires reaching the target within three failed evaluations.The map omits street names and uses coarse landmark categories, so the dialogue must resolve uncertainty about the tourist’s location.Source: §2.1 · p. 3, §2 · p. 3

2.2 DATA COLLECTION

• Turkers were paired at random on Mechanical Turk and alternated between guide and tourist roles across tasks.The ParlAI interface rendered 360-degree images and dynamically displayed neighborhood maps.Source: §2.2 · p. 3

2.3 DATASET STATISTICS

• The dataset contains over 10,000 successful dialogues, with a 76.74% human success rate and more than 62 acts per successful task on average.The average dialogue includes more than 44 tourist actions, roughly 9 guide utterances, and 8 tourist utterances.Source: §2.3 · p. 3

3 EXPERIMENTS

• The full-task baseline uses trained localization models on random tourist walks, evaluating the tourist’s position whenever the guide predicts the target has been reached.The task combines landmark recognition, tourist localization, and natural-language instruction-giving; localization is needed to determine whether the target was reached.Source: §3 · p. 4

3.1 TOURIST LOCALIZATION

• The localization experiments assume perfect perception because landmark classification models barely outperform a random baseline, and they ignore orientation, implicitly assuming compass access.Under perfect perception, the tourist receives landmarks at the current location instead of an image view.Source: §3.1 · p. 4
• High localization accuracy requires communicating a short random path that includes both observations and actions, so the guide must decode multiple time steps and their spatial arrangement.The study probes communication from continuous vectors to emergent discrete symbols and natural language.Source: §3.1 · p. 4

3.1.1 FORMALIZATION

• In emergent communication, tourists take random actions and send a message based on their observations and actions; the guide predicts the target location from that message.The formalization represents the path with observations Z and actions A, and the communicated message with M.Source: §3.1.1 · p. 5
• Natural-language experiments use dataset utterances and the observations and actions since the prior tourist utterance, or generated messages conditioned on those observations and actions.Unlike emergent communication, actions cannot be sampled randomly because messages are coupled to human trajectories.Source: §3.1.1 · pp. 4–5

4.1 THE TOURIST

• Continuous and discrete tourist channels encode observations and actions separately, using position-gated landmark embeddings to retain observation order.The discrete channel samples messages from Bernoulli distributions and is trained with policy gradients because sampling is non-differentiable.Source: §4.1 · p. 5
• For natural-language messages, separate LSTMs encode observation and action histories, whose states condition a decoder trained with teacher forcing and optionally fine-tuned against guide loss.At test time, the tourist can use greedy, sampling, or beam-search decoding.Source: §4.1 · p. 6

4.2 THE GUIDE

• The guide extracts observations and actions from messages; for natural language, a bidirectional LSTM and attention produce embeddings, with a learned control embedding updated from prior control and observation.Source: §4.2 · p. 6

4.2.1 MASKED ATTENTION FOR SPATIAL CONVOLUTIONS (MASC)

• MASC treats communicated actions as spatial transitions: action embeddings form 3×3 masks over convolution weights, repeatedly updating the map before observation-to-location scoring.The motivation is to aggregate landmarks from surrounding locations rather than compare the tourist’s trajectory directly with individual landmark embeddings.Source: §4.2.1 · pp. 6–7

4.3 COMPARISONS

• Evaluation compares MASC with a shared ordinary-convolution No MASC model and a Bayes prediction upper bound for emergent-language localization.The upper bound represents irreducible error because multiple locations can match the same observations and actions.Source: §4.3 · pp. 7–8, Table 2 · p. 7

5.1 ANALYSIS OF LOCALIZATION TASK

• A single landmark observation yields only about 35% localization accuracy, leaving uncertainty in the tourist’s location to drive dialogue.If the guide knew the tourist’s exact location and orientation, it could instead give a list of instructions.Source: §5.1 · p. 8
• Observation-only localization plateaus around 57%, even for T = 3 actions, but exceeds 90% when actions are included.For random walks, high localization accuracy requires communicating a trajectory of observations and actions.Source: §5.1 · p. 8

5.2 EMERGENT LANGUAGE LOCALIZATION

• MASC significantly improves emergent-language localization: it reaches 56.09% for T = 1 action and 69.85% for T = 3, while no-MASC models plateau at 43%.Similar performance for continuous and discrete emergent communication suggests the discrete channel is not a limiting factor.Source: §5.2 · p. 8

5.3 NATURAL LANGUAGE LOCALIZATION

• Human utterances yield 16.17% test accuracy, and MASC outperforms no-MASC models for natural-language communication.Using up to five dialogue utterances improves localization only to 20.33%.Source: §5.3 · p. 8

5.4 LOCALIZATION-BASED BASELINE

• 88.33% for the continuous MASC model with T = 3 exceeded human performance of 76.74% on the full task, under a perfect-perception assumption.Humans may compensate for localization uncertainty through better guidance or path preferences.Source: Table 4 · p. 9, §5.4 · p. 9

5.3 NATURAL LANGUAGE LOCALIZATION

• Generated utterances refer to observed landmarks more successfully than human utterances, but performance deteriorates when models condition on multiple observations.This suggests the models cannot yet produce natural-language utterances about multiple time steps.Source: Table 5 · p. 9, §5.3 · p. 9
• Supervised and policy-gradient generated utterances improve accuracy by more than 10 points over human utterances, but remain slightly below the single-observation baseline.The models learn to ground utterances in a single landmark observation, not multiple observations and actions.Source: §5.3 · p. 9

5.4 LOCALIZATION-BASED BASELINE

• Human annotators took roughly 15 steps, compared with roughly 34 for the best random-walk model; they likely used some form of guidance.Source: §5.4 · p. 9

• Talk The Walk comprises crowd-sourced dialogues in which two human annotators collaborate to navigate to target locations in virtual NYC streets.Source: §6 · p. 10

• Earlier task-oriented dialogue corpora include the Map Task and Maze Game, while later datasets address visual grounding, knowledge-base-grounded discourse, or negotiation.Source: §7 · p. 13
• Embodied-AI work includes language-instruction navigation and neural navigation in indoor scenes and large cities, but the cited city and indoor navigation approaches lack multi-agent communication.Source: §7 · p. 13
• Grounded language learning links language to sensorimotor experience and has been applied across multimodal semantics, robotics, language representations, vision-language tasks, and multi-agent communication.Source: §7 · p. 13

• Emergent models use embedding size L = 500; natural-language models use 128-dimensional word embeddings and a bidirectional RNN with 256 units.Source: §8 · p. 13

9 ADDITIONAL NATURAL LANGUAGE EXPERIMENTS

• Tourist generation conditioned on a single observation achieves the best performance among the trajectory lengths investigated.Source: §9 · p. 13, Table 6 · p. 14

9.1.1 PATH LENGTH

• Conditioning tourist utterances on one observation (T = 0) is the most important evaluation hyperparameter for full-task performance.Full-task performance deteriorates as conditioned trajectory length increases.Source: Table 6 · p. 14, §9.1.1 · p. 14
• Training guide models on random rather than human trajectories yields only small improvements, with human trajectories generalizing to random paths.Random-path guide training uses the evaluation path length.Source: §9.1.1 · p. 14

9.1.2 EFFECT OF BEAM-SIZE

• Localization test accuracy falls from 29.05% to 20.87% when beam size increases from one to eight.Source: Table 7 · p. 14, §9.1.2 · p. 15

9.2 LOCALIZATION FROM HUMAN UTTERANCES

• MASC outperforms no-MASC across models trained on the last 1, 3, or 5 dialogue utterances, including guide utterances.Mean predicted trajectory length increases from 1 to 2 as more context is included.Source: Table 8 · p. 14, §9.2 · p. 15

10 VISUALIZING MASC PREDICTIONS

• The first action is always mapped to the correct state transition, but the second and third MASC values can be mixed or misassigned.In one example the third action corresponds to “no action”; in another, the second action appears as the third MASC.Source: §10 · p. 15, Figure 2 · p. 15

11 EVALUATION ON FULL SETUP

• Full-task localization accuracy is reported as an average over three runs.Source: Table 9 · p. 16, §11 · p. 16
• The full-task evaluation generates random tourist paths, obtains a message from the tourist, and uses the guide’s predicted location to assess whether the target was reached.Source: §11 · p. 17

12 LANDMARK CLASSIFICATION

• The dataset contains 307 landmarks in nine classes, with shops and restaurants most frequent and play fields and theaters relatively rare.Landmark recognition is treated as multi-label classification because a corner may contain multiple landmarks.Source: §12 · p. 18
• For each landmark, the method extracts orientation-specific views facing away from the intersection, using two 60-degree views per orientation.This yields eight views per 360-degree image.Source: §12 · pp. 18–19
• The study compares text-recognition features and ResNet features, combining embeddings across views before predicting landmark-class probabilities.Text features use a learned lookup table, while the ResNet representation is extracted from its penultimate layer.Source: §12 · p. 19
• Models are trained with binary cross-entropy weighted by inverse class frequency and an 80–20 class-conditional train-validation split.Training runs for up to 100 epochs with early stopping on validation loss.Source: §12 · p. 20
• All presented landmark-classification models struggle to outperform an all-positive baseline that predicts each landmark class as visible.The 256-dimensional ResNet features have slightly better validation precision but much worse recall and lower F1.Source: §12 · p. 20

13 DATASET DETAILS

• The dataset-statistics table is organized by neighborhood and dialogue status.Source: Table 11 · p. 20
• The split assigns entire 4×4 grids to train, validation, or test, with all four test-set intersections novel.The validation set includes at least one intersection absent from training.Source: §13 · p. 20
• The dialogue examples show guides using landmark descriptions and corrective instructions as tourists report observations and actions.Source: §13 · pp. 20–21

14 MECHANICAL TURK INSTRUCTIONS

• Figure 6 presents the instructions shown to Turkers before their first task.Source: Figure 6 · p. 22
• Figure 7 continues the set of instructions shown to Turkers before their first task.Source: Figure 7 · p. 23