PaperlayerView exact revision on arXiv

Source-linked AI summary

Less is More: Generating Grounded Navigation Instructions from Landmarks

Su Wang, Ceslee Montgomery, Jordi Orbay, Vighnesh Birodkar, Aleksandra Faust, Izzeddin Gur, Natasha Jaques, Austin Waters, Jason Baldridge, Peter Anderson

arXiv:2111.12872v4cs.CVcs.CL

TL;DR

Indoor navigation instruction generators can be poorly grounded in what is visible, limiting the reliability of their descriptions. MARKY-MT5 detects landmarks in route panoramas and generates instructions conditioned on them, using grounded RxR annotations. Human wayfinders achieve 71% SR following its instructions on R2R, close to 75% for human instructions, while a gap remains on harder RxR-style paths.

  • Problem

    Indoor navigation instructions require egocentric movement guidance and visual references, but existing generated instructions can mention nonexistent objects or confuse spatial terms.

    Source: §1 · p. 1

  • Method

    MARKY-MT5 uses RxR pose traces and instructions to build grounded landmark annotations, then detects landmarks in route panoramas and generates instructions conditioned on them.

    Source: §1 · p. 2, §4 · p. 5, §5 · p. 6

  • Results

    71% SR on R2R paths for MARKY-MT5 nearly matches 75% for human instructions, while on RxR-style paths it reaches 62% versus 78% for human instructions.

    Source: RxR · p. 8

  • Takeaways & Limitations

    The work releases silver landmark data and over one million generated navigation instructions as data augmentation for training VLN agents.

    Source: §1 · p. 2

Abstract

from arXiv · show

We study the automatic generation of navigation instructions from 360-degree images captured on indoor routes. Existing generators suffer from poor visual grounding, causing them to rely on language priors and hallucinate objects. Our MARKY-MT5 system addresses this by focusing on visual landmarks; it comprises a first stage landmark detector and a second stage generator -- a multimodal, multilingual, multitask encoder-decoder. To train it, we bootstrap grounded landmark annotations on top of the Room-across-Room (RxR) dataset. Using text parsers, weak supervision from RxR's pose traces, and a multilingual image-text encoder trained on 1.8b images, we identify 971k English, Hindi and Telugu landmark descriptions and ground them to specific regions in panoramas. On Room-to-Room, human wayfinders obtain success rates (SR) of 71% following MARKY-MT5's instructions, just shy of their 75% SR following human instructions -- and well above SRs with other generators. Evaluations on RxR's longer, diverse paths obtain 61-64% SRs on three languages. Generating such high-quality navigation instructions in novel environments is a step towards conversational navigation tools and could facilitate larger-scale training of instruction-following agents.

1. Introduction

Indoor wayfinding instructions must combine egocentric movement guidance with references to the visual environment, yet existing generators can hallucinate objects or confuse spatial terms. MARKY-MT5 selects landmarks from route panoramas to condition multilingual instruction generation, and its instructions approach human-following success on R2R paths.

1. Introduction

  • Indoor instruction generators must provide egocentric movement guidance and refer to visual surroundings, unlike outdoor systems that can use cardinal directions, street names, or map databases.Prior indoor systems assume floorplans and landmark databases, while newer approaches attempt generation directly from visual inputs.Source: §1 · p. 1
  • Generated instructions can be stylistically correct yet difficult to follow because they reference nonexistent objects and confuse left and right.Human wayfinding success rates on unseen R2R environments were 36% for Speaker-Follower and 42% for EnvDrop.Source: §1 · p. 2
  • More visual input can hurt instruction generation because instructions mention only a fraction of the environment, encouraging spurious correlations and hallucinations.Human annotators look at less than 30% of the environment, and instructions reference only a fraction of the objects they view.Source: §1 · p. 2
  • RxR pose traces align spoken and transcribed instructions with what annotators viewed, enabling silver landmark descriptions in English, Hindi, and Telugu grounded to panorama regions.Text parsers identify landmark phrases; image-text co-embedding models and weak supervision from pose traces ground them in the environment.Source: §1 · p. 2
  • MARKY-MT5 first detects landmarks along a sequence of panoramas, then generates instructions conditioned on those landmarks and navigation actions.The released resources include silver landmark data and over one million generated navigation instructions intended as data augmentation for VLN agents.Source: §1 · p. 2
  • 71% SR on R2R paths for MARKY-MT5 nearly matches 75% for human instructions, while MARKY-MT5 reaches 62% on more challenging RxR-style paths versus 78% for human instructions.The introduction also reports 42% for previous models on R2R paths and 58% for MARKY-MT5 trained on full 360° panoramas.Source: §1 · p. 2

2. Related Work

Related work motivates landmark-centered guidance, places MARKY-MT5 among synthetic instruction generators, and connects its interpretable visual grounding to multimodal text generation.

2. Related Work

  • Landmarks encode relations among an object, its surrounding environment, and the wayfinder’s viewpoint, while prior work models scene-object-direction relations for VLN.MARKY-MT5 trains its detector on RxR human-referenced landmarks rather than explicitly designing for landmark saliency.Source: §2 · p. 2
  • Speaker-Follower systems generate instructions from path panoramas for data augmentation and path reranking, whereas MARKY-MT5 selects key landmarks instead of using entire panoramas indiscriminately.MARKY-MT5 builds on multitask multilingual T5 and explores multimodal pretraining tasks for generalization to unseen environments.Source: §2 · p. 2
  • Prior landmark-based generation used reinforcement learning to induce landmark grounding, while Pashevich et al. [44] use synthetic instructions to train VLN agents on ALFRED [55].Those synthetic expressions also serve as an additional decoding task; their objective is agent performance, whereas MARKY-MT5 aims for instructions people can follow.Source: §2 · pp. 2–3
  • MARKY-MT5’s selected-landmark intermediate representation is interpretable, unlike the black-box multimodal-to-text generator discussed for collaborative human-agent instruction generation.Related multimodal generation work includes mouse-trace-controlled captioning, text-image keyword-box alignment, and image-grounded dialogue.Source: §2 · p. 3
  • MARKY-MT5 encodes connected images interleaved with action descriptions and evaluates generated instructions through human downstream task performance rather than automatic metrics alone.Its architecture belongs to multimodal-input, text-output approaches including VL-T5 [12], MAnTiS, and SimVLM.Source: §2 · p. 3

3. Bootstrapping a Landmark Dataset

The authors bootstrap silver landmark annotations by extracting phrases from RxR instructions and grounding them to pose-trace frames. They refine the resulting image-text pairs to better align each phrase with its landmark.

3. Bootstrapping a Landmark Dataset

  • Navigation datasets lacked landmark annotations, and Matterport3D object boxes were not aligned to instructions or broad enough to cover their landmarks.A landmark annotation pairs an instruction span mentioning an object with a perspective image containing that object.Source: §3 · p. 3
  • The pipeline extracts temporally ordered noun-phrase landmarks with a multilingual dependency parser and stop lists, then grounds them to pose-trace frames and refines the selected views.RxR provides pose traces and word timestamps; refinement selects overlapping views with the highest MURAL alignment score.Source: §3 · pp. 3–5
  • The RxR splits yield 971k landmark phrases from 102k instructions, averaging 9.55 landmarks per instruction across 84,237 unique phrases.Source: §3 · p. 4
  • CTC aligns ordered landmark phrases to contiguous frame subsequences, using BLANK for frames without landmarks and assuming mention order matches observation order.Greedy decoding selects the highest-probability frame for each phrase to form landmark image-text pairs.Source: §3 · p. 4, Figure 3 · p. 4
  • On 100 Val-Unseen instructions, alignment precision is 56.5 with timestamp bias versus 49.6 without bias; timestamp bias alone scores 23.4.Source: §3 · p. 5

4. Landmark Detection

The landmark detector uses route-conditioned panoramas to select candidate landmarks along a path. It is trained on bootstrapped RxR annotations and returns a path-length-matched set of detections.

4. Landmark Detection

  • The detector treats landmark selection as object detection over route-conditioned 360° panoramas, rotating each pano to center the outbound direction and marking both inbound and outbound directions.Equirectangular distortion is described as minimal near the horizon, where most landmarks occur.Source: §4 · p. 5, Figure 5 · p. 5
  • Training maps perspective landmark images to enclosing equirectangular boxes and fine-tunes a COCO-pretrained Hourglass-104 CenterNet for 20k iterations on RxR silver landmarks.The detector uses a single landmark class and 512×256 images with batch size 128.Source: §4 · p. 5
  • At inference, the system pools each pano’s top three detections and returns the top T detections for a path of length T.This corresponds to at most three landmarks per pano and one per pano on average in the training data.Source: §4 · p. 5

5. Instruction Generation

The instruction generator turns selected landmark images and route actions into text using a multilingual mT5 model. Its template encodes landmark orientation and connecting movements, while auxiliary tasks support landmark realization and multilingual generation.

5. Instruction Generation

  • The generator extends the multilingual mT5 encoder-decoder by inserting selected landmark image representations into a templatic route description.Source: §5 · p. 5, Figure 6 · p. 6
  • The template orders landmarks along the route, encodes each landmark’s relative orientation and the actions between landmarks, and prompts the desired output language.It uses 12 orientation phrases and 12 action phrases, each representing a 30° arc; an outbound landmark is also included at each pano.Source: §5 · p. 6
  • An auxiliary rewrite task replaces silver landmark image features with their phrases, training the model to associate landmark representations with their textual realizations.Source: §5 · p. 6
  • The model is pretrained on image-caption translation with a pivot image, then fine-tuned on RxR in English, Hindi, and Telugu and decoded with argmax.The pretraining mixture uses CC3M and CC12M captions, translated into Hindi and Telugu; only the first half of each input caption is provided.Source: §5 · p. 6

6. Experiments

Experiments compare generated instructions with human wayfinding and prior generators using R2R and RxR evaluations. Training on the larger RxR dataset improves English human wayfinding performance.

Experiments

  • The study compares MARKY-MT5 with Speaker-Follower and EnvDrop, whose generators use full panoramic images represented by 36 directional features.A separate Speaker-Follower reimplementation is trained on RxR in English, Hindi, and Telugu.Source: §6 · p. 6
  • On R2R Val-Unseen paths, MARKY-MT5 reaches 70.8% SR versus 74.9% for human instructions and 42% for previous models.Table 1 reports 783 evaluations for each model.Source: Table 1 · p. 7
  • Human wayfinding evaluation involved 20,000 trials with 70 annotators and measured route similarity, instruction quality, visual search, and completion time.Reported route metrics include NE, SR, SPL, NDTW, and SDTW; visual search measures observed panoramic field.Source: §6 · p. 7
  • RxR training raises English success rate from 42.0% to 57.8% and SPL from 35.8% to 48.7%, while lowering navigation error from 6.0m to 3.9m.The comparison uses the same model trained on R2R or RxR.Source: §6 · p. 7

RxR

MARKY-MT5 nearly closes the human-instruction success-rate gap on R2R-level paths, but a gap remains on RxR paths. Landmark inputs and rewrite training improve results, while automatic text metrics can diverge from human evaluations.

RxR

  • RxR automatic results aggregate English, Hindi, and Telugu except for English-only SPICE; RxR annotations are longer and include more detailed descriptions and state verification.The automatic-evaluation table reports SR and NDTW for a state-of-the-art VLN agent.Source: Table 3 · p. 8, RxR · p. 8
  • On R2R, adding landmark modeling raises SR from 57.8% to 70.8% and SPL from 48.7% to 59.8%, while lowering navigation error from 3.9m to 2.9m.The authors attribute gains primarily to landmark modeling; transformer architecture and stronger image features have little impact without landmarks.Source: RxR · p. 8
  • On RxR, silver landmarks raise SR from 53.3% to 63.3% versus outbound landmarks, while NDTW rises from 52.1% to 57.0% and navigation error falls from 5.3m to 4.3m.The authors interpret this as evidence that the silver landmarks contain useful groundings.Source: RxR · p. 8
  • On RxR, silver landmarks yield 63.3% SR versus 61.5% with predicted landmarks, suggesting room to improve landmark detection.The silver-landmark evaluation is not deployed-system performance.Source: RxR · p. 8
  • On R2R-level paths, MARKY-MT5 reaches 71% SR versus 75% for human instructions and 42% for prior models; on RxR-style paths, the comparison is 62% versus 78%.The authors report that this nearly eliminates the gap on R2R-level paths, while a gap remains on RxR.Source: RxR · p. 8
  • A VLN agent obtains near-identical SR (56.5% vs. 55.7%) and NDTW (62.9% vs. 63.3%) for human and model-generated instructions, though humans can distinguish them.The reported pairs compare human and model-generated instructions, respectively.Source: RxR · p. 8
  • Rewrite training improves Val-Seen/Val-Unseen BLEU (+2.4/+1.8), CIDEr (+8.7/+7,1), and SPICE (+1.2/+0.7); pretraining improves BLEU and CIDEr but not SPICE.The pretraining comparison reports SPICE changes of -0.5/-0.2.Source: RxR · p. 8
  • Replacing silver with predicted landmarks sharply reduces BLEU, CIDEr, and SPICE despite human-evaluation results; the authors recommend automatic metrics only for comparing similar models.They attribute the textual-similarity reduction to differences in landmark choice and note that sampling landmark predictions permits diverse instructions.Source: RxR · p. 8

7. Conclusion

The paper proposes a two-stage visual-landmark approach for generating navigation instructions and reports near-elimination of the human-instruction gap on R2R paths. It also identifies context blindness during generation as a limitation.

7. Conclusion

  • The proposed two-stage approach generates navigation instructions directly from visual inputs.Its strength is focusing on visual landmarks.Source: §7 · p. 8
  • MARKY-MT5 nearly eliminates the gap between generated and human-written instructions on R2R paths, supported by a bootstrapped dataset of 971k grounded landmarks.Source: §7 · p. 8
  • The system is blind to environment context during generation and may produce pragmatic errors, such as “Leave the room” when multiple exits exist.The authors identify addressing this limitation as a potential route to further gains.Source: §7 · p. 8

8. Bootstrapping a Landmark Dataset

The silver landmark dataset is built by extracting ordered landmark phrases from multilingual RxR instructions and grounding them in pose-trace frames. The section also evaluates grounding precision and describes extraction errors and the distribution of landmarks.

8. Bootstrapping a Landmark Dataset

  • A multilingual dependency parser extracts entity mentions, expands each head with non-clausal dependents, and records the resulting phrase span.For example, “large brown chair” is retained as one phrase.Source: §8 · p. 12
  • Extraction can yield non-object phrases and errors from incomplete sentences; language-specific stoplists address non-objects, while incomplete-sentence errors remain rare.Examples include “steps,” “left,” and “Fancy.”Source: §8 · p. 12
  • For 100 sampled English instructions, authors manually aligned phrases to pose-trace frames and counted a grounding as correct when its frame fell within the human-selected subsequence.Precision was averaged across phrases within each instruction and then across instructions.Source: §8 · p. 12
  • Landmark phrases have a long-tailed frequency distribution, and their centers cluster near the outbound heading and around or below the horizon.Many unique phrases are semantically similar, such as “brown chair” and “white chair.”Source: §8 · p. 12

9. Landmark Detection

Landmark detection uses a single landmark class, with inference returning landmarks at a path-length-based ratio. The section also illustrates phrase filtering, grounding interfaces, and the spatial and frequency distributions of detected landmarks.

9. Landmark Detection

  • Using clustered phrase embeddings to create richer landmark classes slightly reduced the full model’s automatic evaluation scores.The authors suggest uncalibrated detection confidence across classes as a possible explanation.Source: §9 · p. 12
  • The detector returns one landmark per pano in the path, while landmark centers cluster near the outbound heading and around or below the horizon.The authors tested several landmark-to-path-length ratios before using 1:1.Source: §9 · p. 12, Figure 12 · p. 15
  • Language-specific, native-speaker-compiled stoplists exclude hard-to-localize phrases; phrase extraction identifies object heads and gathers constrained dependencies into full noun phrases.The stoplists cover English, Hindi, and Telugu.Source: Figure 7 · p. 13, Figure 8 · p. 13
  • The grounding interface lets annotators mark pose-trace frames where each extracted phrase is visible; examples include both correct groundings and failures.The examples include a lamp that cannot be found in the scene.Source: Figure 9 · p. 14, Figure 10 · p. 14
  • Common generic phrases such as “table” or “door” are frequent, while specifically described uncommon objects populate the long tail.Figure 11 also presents the 20 most frequent phrases and 20 phrases appearing only once.Source: Figure 11 · p. 15

10. Instruction Generation

Instruction-generation models are pretrained with cross-entropy and Adafactor, then finetuned from alternative pretrained models. The section also reports a landmark-to-path-length ratio comparison setup.

10. Instruction Generation

  • Pretraining uses cross-entropy loss and Adafactor, with batch size 128 for 1.45M steps.The downstream model’s final pretrained version is selected based on SPICE.Source: §10 · p. 15
  • Table 5 compares landmark-to-path-length ratios for MARKY-MT5 with Rewrite auxiliary training and CC3M/12M pretraining.Source: Table 5 · p. 15
  • The CC3M, CC12M, and combined CC3M+CC12M pretrained models are each finetuned for downstream use.Source: §10 · p. 15

11. Experiments

In automatic evaluation on RxR Val-Unseen, MARKY-MT5’s generated instructions achieve HAMT results close to human-written instructions. Human wayfinding and error analyses also examine performance across languages and the types and consequences of remaining errors.

11. Experiments

  • Wayfinders used PanGEA to navigate with an instruction and then rated its quality on a five-point Likert scale; evaluators were fluent in the instruction language.Source: §11 · pp. 15–16
  • RxR Val-Unseen human-wayfinding results are reported across English, Hindi, and Telugu, with patterns consistent across languages and 1,517 paths evaluated per language.Each path is evaluated once per language.Source: §11 · p. 16, Table 6 · p. 16
  • In the error analysis, 16% of instructions were error-free; among errors, weak descriptions accounted for 62.9%, versus 11.4% full hallucinations and 25.7% action-orientation issues.The analysis covered 110 randomly sampled English instructions generated for RxR Val-Unseen paths.Source: §11 · pp. 16–17, Table 7 · p. 17
  • Minor landmark-description flaws can often be overcome, whereas path errors are less recoverable because they can lead wayfinders onto an entirely wrong route.Source: §11 · p. 17
  • 55.7% vs. 56.5% success rate and 63.3% vs. 62.9% nDTW were obtained for generated versus human-written instructions with HAMT on RxR Val-Unseen.The results indicate similar performance for model-generated and human-written instructions in this automatic evaluation.Source: Table 8 · p. 17, §11 · p. 17
Loading 2111.12872v4…