Source-linked AI summary
Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, Yoav Artzi
TL;DR
The paper addresses joint vision-language reasoning in real-world visual navigation and spatial localization. It introduces Touchdown, a two-stage task and dataset, and finds that the resulting data demands complex spatial reasoning while challenging existing methods.
Problem
Existing vision-language resources largely use simpler visual input, motivating a task that captures richer spatial reasoning in real-life urban observations.
Method
The paper introduces Touchdown, where an agent follows navigation instructions through Google Street View and then resolves a spatial description to locate a hidden teddy bear.
Results
9,326 examples expose complex spatial reasoning challenges, while LINGUNET significantly outperforms several strong baselines on spatial description resolution.
Takeaways & Limitations
Touchdown demonstrates the value of real-life visual observations and goal-driven data collection for studying diverse spatial reasoning and learning challenges.
Takeaways & Limitations
Several paths were discarded because of updates in Street View data.
Abstract
from arXiv · showhide
We study the problem of jointly reasoning about language and vision through a navigation and spatial reasoning task. We introduce the Touchdown task and dataset, where an agent must first follow navigation instructions in a real-life visual urban environment, and then identify a location described in natural language to find a hidden object at the goal position. The data contains 9,326 examples of English instructions and spatial descriptions paired with demonstrations. Empirical analysis shows the data presents an open challenge to existing methods, and qualitative linguistic analysis shows that the data displays richer use of spatial reasoning compared to related resources.
1. Introduction
Touchdown studies joint vision-language reasoning through navigation in real urban imagery followed by spatial description resolution. The dataset is designed to capture diverse spatial reasoning and remains challenging for existing methods.
- The task requires reasoning about object properties, object arrangements, and changing relative positions while the agent moves through streets.
- Touchdown combines natural-language navigation through a Google Street View environment with locating a hidden teddy bear from a spatial description.The agent first reaches a goal position, then identifies the described location in the observed image.
- Workers write and follow instructions while hiding Touchdown, using successful execution as a measurable goal to discourage overly verbose descriptions.
- 9,326 complete examples yield 9,326 navigation tasks and 27,575 spatial description resolution tasks.Each example includes a navigation demonstration and Touchdown’s location.
- The spatial description model casts Touchdown localization as language-conditioned image feature reconstruction, significantly outperforming several strong baselines.
2. Related Work and Datasets
Touchdown extends vision-language research from simpler or simulated settings to interactive navigation with real-world observations. Its environment and language provide a larger and more complex setting than closely related resources.
- Prior work commonly studies static images, simulated interactive environments, visual question answering, navigation, and referring expression resolution.
- Touchdown focuses on an interactive environment with real-world observations rather than simulated visual input.
- Touchdown uses 29,641 panoramas, compared with R2R’s 90 environments averaging 119 panoramas each.
- The larger environment makes closely following instructions necessary because searching for the goal is unlikely even with many steps.
- Unlike referring expression resolution, SDR describes a specific location rather than discriminating among objects of the same type, producing more complex language.
3. Environment and Tasks
The environment is a graph of headed Street View panoramas, supporting two complementary tasks: egocentric navigation and allocentric spatial description resolution. Navigation predicts actions, whereas SDR predicts a target pixel.
- Environment: The New York City environment contains 29,641 360° panoramas connected by 61,319 undirected edges, with perspective images rendered for edge headings.
- Tasks: Navigation uses egocentric spatial reasoning about the agent’s relationship to observed objects, while SDR uses allocentric relations among observed objects.
- Navigation: Navigation executes FORWARD, LEFT, RIGHT, and STOP actions through a deterministic transition function over headed panorama states.
- Navigation: Navigation is evaluated with task completion, shortest-path distance, and success-weighted edit distance.
- Spatial Description Resolution: SDR identifies the (x, y) location referred to by a natural-language description in a 360° panorama, where Touchdown is hidden from the input.
- Spatial Description Resolution: SDR evaluation measures accuracy at 40, 80, and 120 pixel radiuses, consistency across images for unique sentences, and distance error.
4. Data Collection
TOUCHDOWN uses a treasure-hunt crowdsourcing process in which workers write, follow, validate, and segment instructions for navigation and spatial description resolution. The process propagates target locations across neighboring panoramas and uses goal-directed incentives and validation to produce usable task data.
- Data collection process: The data collection process splits into writing, following, target propagation, validation, and segmentation tasks.Workers write instructions while navigating to and hiding Touchdown; other workers follow them, propagate target locations, validate outcomes, and annotate task segments.
- Instruction writing: Workers write instructions along sampled routes that often end mid-block, encouraging descriptions of goal positions rather than only directions to intersections.Writers reorient to the route, describe object directions or relations to the agent, place Touchdown at a non-moving object, and cannot refer to text in images.
- Target propagation: Target propagation makes Touchdown’s location available in neighboring panoramas, allowing correct completion from semantically equivalent positions.The propagated locations support validation and permit success without stopping at the exact original panorama.
- Validation: Validation requires followers to navigate without the overhead map and click within a 90-pixel radius of Touchdown’s ground-truth location.A correct validation therefore requires reaching the exact goal panorama or one of the neighboring panoramas receiving the propagated target.
- Segmentation: Instruction segmentation labels consecutive navigation prefixes and SDR suffixes, which may overlap.This annotation enables the navigation and spatial description resolution tasks to be addressed separately.
5. Data Statistics and Analysis
TOUCHDOWN provides substantial validated data and exhibits longer, more linguistically complex instructions than related navigation resources. Qualitative analysis finds frequent entity tracking and diverse spatial reasoning, with egocentric relations concentrated in navigation and allocentric relations in SDR.
- Validation and dataset splits: 89.1% of examples were correctly validated, with 80.1% succeeding on the first attempt and 9.0% on the second.Although five attempts were allowed, 64% of tasks required one attempt and only 1.4% succeeded only after five attempts.
- Validation and dataset splits: TOUCHDOWN contains 9,326 complete-task examples, while SDR includes 25,575 examples from 9,326 unique descriptions.The complete-task and navigation splits contain 6,526 training, 1,391 development, and 1,409 test examples; SDR contains 17,880, 3,836, and 3,859 respectively.
- Data statistics: The mean instruction length is 108.0 tokens, paths average 35.2 panoramas, and SDR segments average 29.8 tokens.The navigation paths are longer than R2R’s 6.0-panorama average, and TOUCHDOWN has larger vocabulary and longer navigation instructions than related corpora.
- Linguistic analysis: TOUCHDOWN examples refer to 10.7 unique observable entities on average versus 3.7 in R2R, with nearly universal coreference to previously mentioned entities.The analysis also reports more reasoning about counts, sequences, comparisons, and spatial relationships than in R2R.
- Linguistic analysis: Navigation segments contain more egocentric spatial relations, whereas SDR segments require more allocentric reasoning between objects.The distinction reflects navigation reasoning about the agent’s relation to its surroundings and SDR reasoning about relations among observed objects.
6. Spatial Reasoning with LINGUNET
The SDR task is formulated as language-conditioned image reconstruction, using LINGUNET to predict Touchdown’s location over the observed panorama. LINGUNET combines text and visual representations to produce pixel-level probability distributions, and performs best among the evaluated systems.
- SDR predicts a distribution over Touchdown’s location across the entire observed image.
- LINGUNET is a language-conditioned UNET that incorporates language during image reconstruction.
- Text is encoded with a bidirectional LSTM, while panorama images are represented using pretrained RESNET18 features.
- LINGUNET fuses text-conditioned feature maps with convolutional and deconvolutional processing, then predicts a pixel distribution whose mode is the location estimate.
- Results: All learning systems outperform non-learning baselines and the language-free UNET, with LINGUNET performing best.
- Results: LINGUNET can resolve unique object anchors, but ambiguous repeated locations spread probability across similar regions and reduce exact localization.
7. Navigation Baselines
Navigation is evaluated against simple non-learning and learned baselines using supervised action prediction. Learned systems perform better than simple baselines, while RCONCAT outperforms GA across all three reported metrics.
- STOP, RANDOM, and FREQUENT provide non-learning navigation baselines based on immediate stopping, random actions, or repeatedly moving forward.
- GA and RCONCAT are evaluated as recent learned navigation models using RESNET18 image features.
- Navigation models are trained with supervised learning by maximizing the log-likelihood of actions in reference demonstrations.
- Results: Table 5 reports development and test results for three navigation evaluation metrics.
- Results: Learned baselines outperform STOP, FREQUENT, and RANDOM, while RCONCAT outperforms GA across all three metrics.
8. Complete Task Performance
The complete task combines the best navigation and SDR models in a simple pipeline and evaluates whether the agent finds Touchdown. The pipeline achieves 4.5% accuracy at an 80px threshold, versus 92% mean human accuracy on valid tasks.
- The full task uses a simple pipeline combining the best SDR and navigation models.
- 4.5% accuracy is achieved for complete-task Touchdown finding at an 80px threshold.
- Human workers achieve 92% mean accuracy on the set of valid tasks.
9. Data Distribution and Licensing
The project releases the environment graph, panorama-download scripts, collected data, and code under a CC-BY 4.0 license. Downloaded panoramas remain subject to Google’s policies, with additional released RESNET18 features available by request.
- Licensing: The released resources include environment panorama IDs and edges, scripts for downloading RGB panoramas, collected data, and project code.
- Licensing: These released components are provided under a CC-BY 4.0 license.
- Licensing: Downloaded panoramas must comply with Google’s policies, while RESNET18 RGB-panorama features are released through a request form.
10. Conclusion
The paper introduces TOUCHDOWN as a dataset for natural-language navigation and spatial reasoning with real-life visual observations. Its environment and data-collection process target diverse reasoning and learning challenges.
- TOUCHDOWN provides natural-language navigation and spatial reasoning tasks grounded in real-life visual observations.
- The two tasks require different reasoning and learning challenges, supporting separate or combined treatment.
- The dataset uses a large Google Street View environment with routes generated from sampled panoramas and traffic-following directions.
B. Additional Data Analysis
Additional analyses describe the evaluation setup, model implementations, and image representations used for navigation and spatial-description resolution. RESNET18 features outperform raw RGB images in the reported navigation comparison.
- Additional Data Analysis: Table 6 compares linguistic analyses of randomly sampled development examples from TOUCHDOWN, SAIL, and LANI.
- Additional Data Analysis: SDR predictions are evaluated at an 80-pixel threshold using gold-location comparisons in the predicted feature-map scaling.
- Additional Data Analysis: Navigation models encode instructions with LSTM networks and combine language, visual observations, and previous actions over time.
- Additional Data Analysis: The experiments include asynchronous supervised learning, ADAM optimization, early stopping, and raw-RGB comparisons.
- Experiments with RGB Images: RESNET18 features achieve better navigation performance than raw RGB images.
F.2. Single-modality Experiments
Single-modality and qualitative experiments show that visual and linguistic cues jointly matter, while model predictions can succeed on salient objects yet fail on precise spatial relations.
- Single-modality Experiments: Removing vision produces no meaningful navigation learning, whereas removing language still permits limited performance.
- Single-modality Experiments: Both language and vision are necessary for the navigation task.
- Qualitative Predictions: LINGUNET correctly localizes some targets, including the dumpster tarp and green trash barrel.
- Qualitative Predictions: 104px separates one prediction from the correct target, as the model focuses on the visually distinguished top of a sign instead of its bottom.
- Qualitative Predictions: Models can fail to resolve exact spatial descriptions even when identifying mentioned objects or salient signs.
- Qualitative Predictions: Simpler CONCAT and CONCATCONV models correctly predict one location where LINGUNET and TEXT2CONV fail.