Where Are You? Localization from Embodied Dialog

1 Introduction

• WAY addresses embodied dialog requiring goal-driven questions, informative answers, and active perception and navigation to gather discriminative observations.The Locator asks questions and gives instructions while the Observer may navigate and respond from first-person views.Source: §1 · p. 1, Figure 1 · p. 1
• The dataset was collected in reconstructed 3D buildings with two humans holding asymmetric information about the Observer’s location and environment.The Observer begins at a random location; the task ends when the Locator predicts the Observer’s position.Source: §1 · p. 1
• WAY contains about 6K English dialogs collected from more than 2K starting locations in reconstructed indoor environments.The Observer navigates from a first-person view without a map, while the Locator uses a detailed top-down map that omits the Observer’s location.Source: §1 · pp. 1–2
• LED localizes the Observer from a map and partial or complete dialog, while EVD models the Observer and CL models both agents.The paper notes that LED could potentially help emergency responders localize callers using descriptions matched to detailed indoor maps.Source: §1 · p. 2
• 32.7% of the time, the LingUNet-based baseline locates the Observer within 3 meters in unseen test environments, versus 70.4% for human Locators and 6.6% random chance.The paper also studies data augmentation and residual connections and probes dataset biases with unimodal, shuffled, and ablated-dialog experiments.Source: §1 · p. 2

2 Related Work

• WAY extends image-based dialog tasks to active perception and navigation in a situated 3D environment with a whole-building state space.The paper contrasts this setting with Cooperative Visual Dialog and GuessWhat?!, which ground dialog in natural images.Source: §2 · p. 2
• CVDN shares Matterport3D buildings and goal-oriented dialog with WAY, but focuses on navigation rather than localization.The authors say WAY’s localization focus encourages more descriptive first-person utterances; Talk the Walk instead uses an abstracted outdoor map.Source: §2 · p. 3
• WAY has higher densities of adjectives, nouns, and prepositions than related embodied perception datasets, suggesting more descriptive dialog.The comparison reports dataset size, dialog length, and parts-of-speech densities.Source: §2 · p. 3
• WAY builds on language-based localization and language-conditioned map models, including LingUnet, previously proposed for predicting visual goals in images.Related work also connects grounding language to maps with grounding referring expressions to images.Source: §2 · p. 3
• Unlike VLN and CVDN, where instructions assume a known start location, WAY uses dialog to localize the navigator.The paper argues that localization is often a precursor to navigation and describes WAY as addressing this gap.Source: §2 · p. 3

3 WHERE ARE YOU? Dataset

• WAY contains 6,134 embodied localization dialogs across 87 indoor environments built from Matterport3D reconstructions.The Locator sees perspective-rendered top-down maps, with a separate image for each floor in multi-story buildings.Source: §3 · p. 3

3.1 Collecting Human Localization Dialogs

• The collection sampled 2,020 episodes across 87 environments, used paired AMT workers, and split dialogs by environment to distinguish seen- from unseen-building evaluation.The train, val-seen, val-unseen, and test sets contain 3,967, 299, 561, and 1,165 dialogs, respectively.Source: §3.1 · p. 4

3.2 Dataset Analysis

• 72.5% of episodes were successful at the 3-meter threshold, and average human localization error was 3.17 meters.Each starting location had three annotations from separate Observer–Locator teams.Source: Figure 2 · p. 4, §3.2 · p. 5
• Human localization difficulty varies with starting-location distinctiveness, map interpretation, and environmental features such as repeated rooms and multiple floors.The paper notes that larger and multi-floor buildings tend to have greater localization errors.Source: §3.2 · p. 5
• Dialogs average five messages and 61 words; Observers describe their surroundings while Locators probe for details or request movement.Observer messages average 19 words versus 9 for Locators, reflecting their different roles.Source: §3.2 · p. 5
• 62.6% of episodes include Observer navigation; those episodes have lower average localization error than episodes without navigation, 2.70m versus 3.98m.The paper describes movement away from nondescript areas toward unambiguous locations as a way to reduce uncertainty.Source: §3.2 · p. 5

3.3 WHERE ARE YOU? Tasks

• WAY formalizes LED as final-location prediction from environment and dialog, EVD as predicting the Observer’s next action and message, and cooperative localization as modeling both agents.LED does not require navigation or text generation; cooperative localization can be evaluated as a team using LED accuracy.Source: §3.3 · pp. 5–6

4.1 LED Model from Top-down Views

• LingUNet-Skip combines a top-down environment map with dialog to produce a probability distribution over possible Observer locations.A bidirectional LSTM encodes the dialog; language-predicted kernels modulate encoder features in skip connections that the decoder combines into pixel predictions.Source: §4.1 · pp. 6–7

4.3 Results

• 32.7% of test locations were predicted within 3m by LingUNet-Skip, versus 70.4% for human Locators.The model improved on the best handcrafted baseline by 10% to 30% absolute across splits, yet had higher localization error than Center on val-unseen and test.Source: Table 2 · p. 7, §4.3 · p. 8

4.2 Experimental Setup

• The model minimizes KL-divergence to a ground-truth location smoothed with a 3m Gaussian, and evaluation uses Euclidean localization error and 3m/5m success thresholds.Training uses map augmentations and selects the checkpoint with the highest val-unseen Acc@3m.Source: §4.1 · p. 7, §4.2 · pp. 7–8

4.4 Ablations and Analysis

• Similar performance on val-seen and val-unseen suggests the model is not memorizing navigation-node locations; learning their distribution could instead provide a useful free-space prior.Source: §4.4 · p. 8
• On val-unseen Acc@3m, No Vision scored 7.8% and No Dialog 32.1%, versus 45.6% for the full model, indicating that the task is strongly multimodal.No Dialog also beat Center, 32.1% versus 29.8%, consistent with a visual centrality prior stronger than the center coordinate.Source: §4.4 · p. 9
• First-half dialog scored 46.2% versus 45.6% for full dialog; Observer-only scored 44.9% versus 33.3% for Locator-only, while shuffling rounds reduced val-unseen Acc@3m by 0.7%.The first half supplies coarser descriptions; Observer replies may also retain the influence of Locator prompts from data collection.Source: §4.4 · p. 9
• Without data augmentation, val-unseen Acc@3m dropped 9.9%; removing the residual connection dropped it 2.5%.Source: §4.4 · p. 9

• WAY provides a dataset for LED, Embodied Visual Dialog, and Cooperative Localization, covering localization from dialog history and modeling one or both agents.Source: §5 · p. 9
• LingUNet-Skip improves over simple baselines and model ablations on LED, but does not fully use the second half of the dialog.Source: §5 · p. 10
• The authors identify further exploration as an opportunity because WAY encompasses multiple embodied localization tasks.Source: §5 · p. 10

• Val-unseen human Acc@3m is 79.4%, compared with 70.4% on test; the authors associate val-unseen’s higher performance with smaller environments and discriminative features.The split is notably smaller; test environments are unseen, unlike val-seen environments included in training.Source: §6 · p. 12
• Navigation varies across environments: starting and ending positions barely change in one example but shift substantially in another, with no final positions in its long corridor.The passage attributes variation as likely related to building size and discriminative features, and notes long hallways often accompany long paths.Source: §6 · p. 12, Figure 7 · p. 13
• The collection interface pairs a chat history with a top-down map and floor controls for the Locator, while the Observer navigates through a first-person view.Source: §6 · p. 12, Figure 8 · p. 14
• The Observer’s first message averages 23.9 words and has many unique words, while later messages are largely conditioned on the other agent’s preceding message.The first message has a common structure but discriminative content; the authors say accurate dialog-history encoding is important for location estimation.Source: §6 · p. 12, Figure 9 · p. 15
• The appendix visualizes localization-error distributions across validation and test splits and shows predicted distributions alongside true locations, including two failure cases.The error CDFs can also be read as accuracy versus error threshold; example maps mark the 3-meter threshold and true location.Source: §6 · p. 12, Figure 10 · p. 16, Figure 11 · p. 16
• Figure 12 provides examples of dialogs from the WAY dataset.Source: Figure 12 · p. 17