Source-linked AI summary
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, Jason Baldridge
TL;DR
VLN research needs broader multilingual, larger-scale, and more finely grounded data, while existing path designs can encourage exploitable biases. RxR addresses these gaps with a multilingual dataset containing pose-aligned instructions and human demonstrations, and reports generally improved baselines from monolingual learning, RxR paths, and joint R2R–RxR training. Its pose traces also support further grounded-agent research, although language-level differences within the dataset remain difficult to attribute.
Problem
Existing VLN datasets have limited multilinguality, scale, fine-grained grounding, follower demonstrations, and path-bias controls.
Method
RxR combines native English, Hindi, and Telugu instructions with word-aligned guide pose traces, follower demonstrations, varied paths, and CNN-based instruction encoding.
Results
Baseline performance generally improves with monolingual learning, RxR follower and guide paths, and concatenated R2R–RxR multitask training.
Takeaways & Limitations
RxR expands research possibilities for embodied language agents by linking language to visual scenes through dense pose traces across three languages.
Takeaways & Limitations
Differences within RxR may reflect language, dialect, annotator pools, or other factors, and their contributions are unclear.
Abstract
from arXiv · showhide
We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provide results for a model that learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for research on embodied language agents in simulated, photo-realistic environments.
1 Introduction
RxR is a larger multilingual VLN dataset designed to address path and language-coverage gaps while adding dense spatiotemporal grounding. Baseline experiments cover monolingual, multilingual, and multitask settings.
- RxR expands VLN data with more paths, three languages, and annotators’ 3D pose sequences for dense spatiotemporal grounding.The languages are English, Hindi, and Telugu.
- Baseline performance generally improves with monolingual learning and with RxR follower and guide paths.
- Training on concatenated R2R and RxR annotations obtains across-the-board improvements through a simple multitask strategy.
- 126K instructions cover 16.5K sampled guide paths and 126K human follower demonstration paths.
2 Motivation
RxR targets multilinguality, scale, fine-grained grounding, human follower demonstrations, and path biases that limit existing VLN datasets. Its design uses native-language instructions, larger coverage, pose-aligned annotations, and paths intended to resist exploitable priors.
- Multilinguality: Existing VLN datasets were predominantly English, making generalization of findings to other languages unclear.
- Multilinguality: RxR provides independently authored English, Hindi, and Telugu instructions rather than translations, including English from US and Indian speakers.
- Scale: For each language, RxR contains 14K paths with three instructions per path, totaling 126K instructions and 10M words.
- Fine-Grained Grounding: Guides speak while moving, and RxR time-aligns transcribed words with their recorded 3D pose traces.
- Path Desiderata: RxR path desiderata require high length variance, indirect approaches, naturalness, and uniform viewpoint coverage.
3 Two-Level Path Sampling
RxR generates paths hierarchically: it samples simple room sequences, then finds constrained shortest panorama paths between sampled endpoints. Greedy selection favors non-shortest paths and improves panorama coverage.
- Graph Construction: The navigation representation uses a panorama graph P and a room graph R whose vertices encompass connected panorama subgraphs.
- Path Generation: Path generation enumerates simple room paths spanning at most five rooms and two building levels.
- Path Generation: For each room path, RxR constructs a constrained panorama graph and samples start and goal panoramas uniformly from the endpoint rooms.
- Path Generation: The generated panorama path is the shortest path between sampled endpoints within the room-path-induced graph.
- Greedy Selection for Coverage: Greedy selection chooses paths using length and panorama-coverage terms, while imposing 40m and 500-path-per-building limits.
- Path Statistics: RxR paths average 8 edges and 14.9m, with 44.5% not shortest and average length 27.4% above the shortest path.
4 Data Collection and Metrics
RxR collects multilingual navigation data with synchronized Guide and Follower trajectories, then evaluates path adherence alongside standard VLN metrics. Its analysis reports richer linguistic grounding, human-aligned visual attention, and baselines showing the challenge of its indirectly routed paths.
- Data Collection: Guides create spoken instructions while the simulator records camera poses, and dynamic time warping aligns ASR output with manually transcribed tokens.Each Guide annotation yields audio, tokenized timestamped text, and a timestamped 6-DOF pose trace.
- Dataset Scale: 126K instructions cover 16.5K sampled paths, with annotations distributed equally across English, Hindi, and Telugu.Most paths receive nine annotations: three per language across Guide instructions.
- Data Collection: Follower demonstrations objectively validate instructions and provide alternative pose traces that can be compared with Guide traces or used as additional training signals.Success is assessed by reaching within 3m of the final panorama, with failed instructions re-enqueued for further annotation.
- Dataset Analysis: RxR instructions average 78 words, while Guide and Follower tasks average 458 and 132 seconds respectively.Guide time is dominated by transcription, whose recordings average 60 seconds.
- Dataset Analysis: RxR instructions contain higher rates of entity references, coreference, and sequencing than R2R, although within-dataset language differences have multiple possible explanations.The analysis cannot isolate language effects from dialect, annotator-pool, or other factors.
- Evaluation: NDTW and SDTW receive primary emphasis because RxR includes indirectly routed paths, while Table 4 reports simple baseline comparisons on val-unseen paths.The table caption states that RxR is harder overall and less amenable to agents that tend to go straight.
5 Experiments
The experiments adapt an RCM-style agent to RxR’s long multilingual instructions and evaluate path types, multilinguality, grounding supervision, multitask transfer, and modality contributions. Results favor Follower-path training, monolingual learning, multitask training, and multimodal inputs, while grounding supervision and simple transfer show limitations.
- Agent: A CNN instruction encoder replaces the bidirectional LSTM for RxR’s longer instructions, while multilingual BERT supplies word embeddings.The agent retains an LSTM action decoder and uses visual attention over panoramic viewpoints and navigable directions.
- Agent: Panoramic viewpoint features and navigable-direction features are attended jointly with the instruction to rank next actions.The viewpoint representation contains 36 30° panorama intervals, and actions are scored by hidden-state similarity with direction encodings.
- Monolingual and multilingual results: Training with both Guide and Follower paths benefits all languages, while monolingual training outperforms multilingual training and cross-translations hurt performance.These comparisons correspond to Table 5 experiments 3–5.
- Spatiotemporal grounding supervision: Visual attention supervision gives mixed val-unseen results: NDTW and NE improve, but success-based metrics worsen relative to the multilingual baseline.Textual attention supervision did not improve performance, and the authors describe the investigation as preliminary.
- Multitask and transfer learning: The multitask model performs best on both RxR and R2R, whereas transfer learning between the datasets is much weaker than in-domain evaluation.The authors attribute the transfer gap to path-bias differences and RxR’s longer paths and richer language being out of domain for R2R.
- Unimodal ablations: The multimodal agent outperforms both unimodal ablations, while the language-only agent is better than the vision-only agent.Navigation-language cues such as turning and stair instructions remain informative without vision, whereas vision alone lacks the instructions.
6 Conclusion
RxR expands embodied-language navigation research through multilingual, bias-aware paths and dense pose-based grounding. Its word-to-pixel alignments and Follower perspective traces support further study of grounded pragmatics and agent training.
- Conclusion: RxR’s paths are designed so language plays a fundamental role in agent performance across three typologically diverse languages.The authors frame this evaluation as a way to avoid overfitting to one language and dataset.
- Conclusion: Word-level alignments to panorama pixels and Follower perspective traces enable grounded-language study and training in graph-based or continuous simulators.The authors state that these pose-trace possibilities have only begun to be explored.
A Supplementary Material
The supplementary material documents annotators, collection procedures, unavailable audio, and multilingual instruction examples. It also provides figures illustrating annotation views, pose alignment, and sample instructions across English, Hindi, and Telugu.
- Annotators: RxR involved 247 annotators, including contributors in the USA and India who were fluent in their assigned languages.Annotators were paid competitive local hourly wages and held standard contractor rights.
- Collection process: Quality control included separate Guide and Follower assignments, pilot-feedback rounds, annotation guidelines, and follow-up training or removal for lower-success annotators.Followers did not execute their own Guide instructions, and their feedback also supported better Guide instruction writing.
- Data release: Audio instructions were not released because at-home recordings required safety and privacy review during COVID-19.The authors hoped to include the audio in a future release.
- Instruction examples: Supplementary examples include English instructions from US and Indian English, Hindi instructions, and Telugu instructions.The examples contain landmark references and directional actions in each language.
ROOM-ACROSS-ROOM (RxR)
RxR was created by Google Research to advance multilingual vision-and-language navigation. It addresses dataset gaps through more paths, more instructions, bias reduction, and annotator pose sequences.
- Dataset purpose: RxR advances VLN in English, Hindi, and Telugu by addressing gaps in existing datasets.The stated gaps include path coverage, instruction scale, and annotators’ 3D virtual pose sequences.
- Dataset provenance: The dataset was created by Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, Jason Baldridge, and the Google Data Compute team on behalf of Google Research.Funding was provided by Google Research.
COMPOSITION
RxR pairs multilingual natural-language navigation instructions with trajectories through reconstructed 3D buildings, including timestamped pose traces for Guides and Followers. It provides 126K Guide instructions across 16.5K sampled paths, plus 126K human Follower demonstration paths, using recommended building splits.
- Data instances: Each instance pairs a natural-language navigation instruction with a trajectory through a Matterport3D building reconstruction.Trajectories are represented as viewpoint-id paths and sequences of virtual camera poses.
- Alignment: Guide and Follower pose traces and text transcriptions are timestamped and aligned.The dataset includes pose traces for both instruction annotators, with the Follower charged with following the Guide’s instructions.
- Scale and splits: 126K Guide instructions cover 16.5K sampled paths and 126K human Follower demonstration paths, split equally across English, Hindi, and Telugu.The dataset follows Matterport3D and R2R building splits.
- Missing data: Guide audio recordings are not provided.The dataset includes speech transcriptions but not the corresponding audio files.
- Dependencies: The dataset is based on Matterport3D building reconstructions and R2R viewpoint navigation graphs, rather than web resources.RxR is otherwise described as self-contained.
PREPROCESSING / CLEANING / LABELING
RxR preprocessing includes removing or re-annotating instances where Followers could not correctly follow Guide instructions. The dataset has already been used to train VLN agents and may support several related vision-and-language tasks.
- Cleaning and validation: Instances in which Followers could not correctly follow the Guide’s instruction were removed or re-annotated.The paper refers to Section 4 for the validation procedure and associated measures.
- Existing uses: RxR has been used to train vision-and-language navigation agents.The paper describes these experiments as part of its use of the dataset.
- Potential uses: Potential applications include instruction generation, visual referring-expression grounding, grounded dialogue, multilingual learning, and pre-training.
DISTRIBUTION
RxR is distributed openly through GitHub and Google Cloud Storage under a CC-BY license. Use of its Matterport3D-derived content remains subject to the Matterport3D Terms of Use.
- Availability: RxR is distributed via GitHub and Google Cloud Storage and was released on publication.
- License: RxR is released under a CC-BY license.
- Restrictions: Matterport3D-derived data is governed by the Matterport3D Terms of Use.
- Updates: The dataset is not planned to receive updates.