Source-linked AI summary
Vision-and-Dialog Navigation
Jesse Thomason, Michael Murray, Maya Cakmak, Luke Zettlemoyer
TL;DR
Robots need dialog to request and understand assistance while navigating human environments, but existing navigation and dialog settings do not capture this combined challenge. The paper introduces CVDN and the Navigation from Dialog History task, using a multimodal sequence-to-sequence baseline to infer actions from target objects and dialog history. Agents perform better with more dialog history and mixed human-plus-planner supervision, while transfer to physical robots remains limited by simulation and sensing differences.
Problem
Robots navigating human environments need to communicate with people, yet dialog-enabled assistants generally do not actuate and mobile robots generally do not interact with users.
Method
The paper introduces 2050 human-human navigation dialogs in CVDN and defines NDH, which predicts navigation actions from a target object and dialog history.
Results
Agents perform better with more dialog history, while mixed human and planner supervision combines long-range exploration with short-range language-aligned accuracy.
Takeaways & Limitations
CVDN supports studying in-situ navigation interactions and training agents that navigate human environments, ask questions when unsure, or provide verbal assistance.
Takeaways & Limitations
Transfer to real-world robots is constrained by differences in sensing, noise, localization, and the simulator’s discrete graph-based navigation.
Abstract
from arXiv · showhide
Robots navigating in human environments should use language to ask for assistance and be able to understand human responses. To study this challenge, we introduce Cooperative Vision-and-Dialog Navigation, a dataset of over 2k embodied, human-human dialogs situated in simulated, photorealistic home environments. The Navigator asks questions to their partner, the Oracle, who has privileged access to the best next steps the Navigator should take according to a shortest path planner. To train agents that search an environment for a goal location, we define the Navigation from Dialog History task. An agent, given a target object and a dialog history between humans cooperating to find that object, must infer navigation actions towards the goal in unexplored environments. We establish an initial, multi-modal sequence-to-sequence model and demonstrate that looking farther back in the dialog history improves performance. Sourcecode and a live interface demo can be found at https://cvdn.dev/
1 Introduction
CVDN studies how robots and humans can use dialog to navigate photorealistic home environments. It introduces a human-human dataset and the NDH task, finding that longer dialog history and mixed supervision improve navigation.
- CVDN places two human collaborators in a photorealistic simulated environment, where a Navigator asks an Oracle for guidance while searching for a goal room.The Oracle uses privileged shortest-path-planner information to answer the Navigator’s questions.
- The dataset supports robots that ask targeted questions when unsure and agents that provide language guidance to people navigating unfamiliar places.Its dialogs contain nearly three times as many words and cover average paths more than three times longer than R2R.
- NDH trains agents to encode human-human dialog history and infer navigation actions toward a goal location in an unexplored environment.The task is built from CVDN dialogs and uses a sequence-to-sequence model as an initial baseline.
- Agents perform better with more dialog history and when training mixes human and planner supervision.The mixed supervision combines long-range exploration guided by human intuition with short-range accuracy aligned to language input.
2 Related Work and Background
CVDN extends vision-and-language navigation and dialog by combining ambiguous, underspecified instructions with two-sided natural-language interaction in a photorealistic environment. It builds on the Matterport R2R simulator’s graph-based navigation actions and visual observations.
- Related Work: CVDN begins with an ambiguous, underspecified hint and requires exploration plus dialog to resolve the goal location.Unlike many earlier tasks, the hint does not uniquely identify the goal or describe the steps needed to reach it.
- Related Work: CVDN differs from visual question answering and document dialog because its conversations require cooperation and receive new visual observations during navigation.Questions are not simply answered from a static image or document context.
- Related Work: Compared with prior navigation-help settings, CVDN uses two-sided human-human natural-language dialogs rather than templated planner sequences or one-sided guidance.The guiding interaction is situated in a photorealistic environment viewed by both speakers.
- Background: The Matterport simulator contains 90 house scans represented as visual panorama nodes connected by an adjacency matrix.Navigation distinguishes graph hop distance from metric distance.
- Background: At each timestep, agents turn, tilt, move forward to an adjacent node, or stop, receiving a new visual observation after non-stop actions.The forward action is available only when the agent faces an adjacent node.
3 The Cooperative Vision-and-Dialog Navigation Dataset
CVDN is a crowdsourced dataset of ambiguous, underspecified human-human navigation dialogs in simulated MatterPort homes, designed to support embodied agents that use language during goal-directed search. Its longer routes, richer language contexts, and diverse conversational phenomena distinguish the dataset from simpler navigation instructions.
- CVDN contains 2050 human-human navigation dialogs and over 7k navigation trajectories across 83 MatterPort houses.Dialogs use initial instructions that are both ambiguous and underspecified.
- Each prompt specifies a house scan, target object, starting position, and goal region, with targets selected from 81 object types.Goal-room hints identify an object but remain ambiguous because multiple rooms contain that object and underspecified because they omit the route.
- Navigators could move, ask natural-language questions, or guess the goal room, while paired Oracles answered through a chat interface.Incorrect guesses disabled further navigation and forced a question to the Oracle.
- CVDN’s longer routes and language contexts, together with complex dialog and navigation phenomena, create training conditions for navigation, question asking, and question answering.The paper focuses on navigation while identifying broader uses for the dataset.
- Human paths averaged 25.0 ± 12.9 steps and planner paths 17.4 ± 7.0, compared with 6.0 ± 0.85 for R2R.Navigator and Oracle utterances averaged 81.6 words per dialog, exceeding R2R’s 29 words per instruction by nearly a factor of three.
- Dialogs averaged about six utterances, including roughly three question-answer exchanges, with some extending to 26 utterances and about 5% containing no exchanges.No-exchange dialogs reflect successful goal finding from the initial hint alone and are described as lucky guesses.
- Over 90% of annotated dialogs contained egocentric references, while more than 10% exhibited conversational repair and some required prior dialog or navigation history.The annotation covered 100 dialogs and 342 utterances per speaker; annotators reached Cohen’s κ = .738 across phenomena labels.
4 The Navigation from Dialog History Task
The Navigation from Dialog History task converts cooperative dialogs into navigation instances in which an agent uses the target object and prior questions and answers to infer actions toward a goal region. The initial model encodes dialog history and decodes actions from visual observations, while the task provides both human and shortest-path supervision.
- CVDN is divided into Navigation from Dialog History instances, each pairing navigation actions with the preceding question-answer history.The broader dataset also supports question asking and answering, but this section focuses on navigation.
- Each instance contains a target object and a possibly empty sequence of Navigator questions and Oracle answers preceding navigation actions.Dialogs follow the repeating structure of navigation, question, answer, and subsequent navigation steps.
- The sequence-to-sequence baseline uses an LSTM to encode dialog history and initialize an LSTM decoder whose visual observations are fixed environment-frame embeddings.The decoder outputs navigation actions, and input ablations compare access to the target, recent exchanges, or the full dialog history.
- The dataset yields 7415 NDH instances, split into 4742 training, 382 seen-validation, 907 unseen-validation, and 1384 unseen-test instances.Splits preserve house-scan and dialog boundaries so instances from one dialog remain in the same fold.
- NDH provides Navigator steps and Oracle shortest-path steps as two supervision signals, with average lengths of 6.63 and 4.35 steps, respectively.Navigator sequences range from 1 to 40 steps, while Oracle sequences range from 0 to 5 steps.
- Performance is measured as progress toward the goal region, defined by the reduction in distance between the inferred path’s beginning and ending nodes.Distance to a goal region is the minimum topological distance from a node to any node in that region.
5 Experiments
The experiments evaluate a multimodal sequence-to-sequence baseline for Navigation from Dialog History, varying dialog context and supervision. Longer dialog history and mixed human-planner supervision improve navigation progress, especially in unseen environments.
- Model: The baseline encodes dialog history with an LSTM and uses visual observations to decode navigation actions through an LSTM decoder.The model is adapted from a sequence-to-sequence navigation baseline to encode entire dialog histories rather than single instructions.
- Training: 20,000 training iterations use student-forcing with cross-entropy against the next action along the shortest path to the supervision end node.Validation performance is checked every 100 iterations, and the best validation epoch is reported.
- Supervision: Mixed supervision uses the Navigator path when the Oracle endpoint lies on it, and otherwise uses the Oracle path.This acts as a binary confidence heuristic for filtering human demonstrations.
- Experimental setup: The experiments compare target-object, last-answer, prefacing-question, and full-dialog-history inputs, alongside oracle, navigator, and mixed supervision.They also include shortest-path, random, and unimodal baselines.
- Results: Using all dialog history significantly outperforms unimodal ablations in unseen environments.In unseen test environments, full history is statistically significantly better than target-object context alone; mixed supervision consistently outperforms oracle or navigator supervision.
6 Conclusions and Future Work
The paper introduces CVDN and NDH as resources for studying cooperative navigation through situated dialog. It finds benefits from dialog history and mixed supervision, while identifying simulation-to-real transfer and richer sequential modeling as open challenges.
- Conclusions: CVDN contains 2050 human-human navigation dialogs in a photorealistic simulated environment.The dialogs include egocentric visual grounding and references to dialog and past navigation history.
- Future work: CVDN is intended to support agents that navigate while asking questions and agents that provide verbal assistance to human navigators.The dataset also provides a scaffold for training two agents in tandem for end-to-end task-oriented dialog evaluation.
- Conclusions: NDH evaluates whether agents can map dialog-based instructions to navigation actions toward a goal.The task is defined using CVDN dialogs and supports studying navigation from conversational context.
- Conclusions: Mixed human-planner supervision combines long-range exploration from human intuition with short-range accuracy aligned with language input.This describes the complementary strengths reported for the two supervision sources.
- Limitations: Simulation-to-real transfer is limited by discrete graph navigation, RGB-only sensing, lower-quality vision, sensor noise, and localization issues.The authors suggest continuous-motion or virtual-reality data collection as possible aids for transfer to physical robots.
- Future work: Future models could jointly condition dialog and navigation history and formulate NDH as a sequential decision process using reinforcement learning.The motivation includes resolving past-reference instructions that depend on earlier navigation actions.
7 Appendix
The appendix analyzes target distributions, path structure, supervision, dialog history, and model variants in CVDN and NDH. It reports that fuller dialog history and mixed supervision improve navigation performance, while naive navigation-history concatenation does not.
- Additional CVDN Analysis: CVDN’s 81 target objects are most often objects frequent across houses, typically appearing two to four times and corresponding to bedrooms or bathrooms.
- Additional CVDN Analysis: CVDN paths share more nodes within scans than R2R paths because starting panoramas were selected far from potential goal regions.
- Additional CVDN Analysis: Human Navigator trajectories overlap substantially more with shortest-path planner trajectories than average within-scan paths, indicating closer adherence to shortest paths.
- Additional NDH Analysis: NDH shortest-path-supervised trajectories are shorter than R2R paths, whereas human Navigator paths are substantially longer; mixed supervision shifts progress toward positive goal movement.
- NDH Model Performance Statistical Comparisons: Across nearly all supervision comparisons, oracle, navigator, and mixed supervision produce statistically significantly different goal progress, with one unseen-validation exception.
- NDH Model Performance Statistical Comparisons: Using all dialog history significantly outperforms using only the target object in unseen validation and test environments, but not several shorter-history conditions.
- NDH Model Performance Statistical Comparisons: In unseen environments, the full-history model significantly outperforms most unimodal baselines, while vision-only performance benefits from familiar-house memorization.
- NDH Model Variants: Naively concatenating encoded navigation history with dialog history does not improve performance in seen or unseen environments.