PaperlayerView exact revision on arXiv

Source-linked AI summary

VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View

Raphael Schumann, Wanrong Zhu, Weixi Feng, Tsu-Jui Fu, Stefan Riezler, William Yang Wang

arXiv:2307.06082v2cs.AIcs.CLcs.CV

TL;DR

Urban vision-and-language navigation asks agents to ground human directions in real street scenes, with open-ended landmarks and spatial and temporal reasoning demands. VELMA uses an LLM prompted with verbalized instructions, trajectory, and visual observations, including landmark visibility. Its finetuned agent outperforms the previous state-of-the-art model in task completion, while two-shot experiments also report navigation skills.

  • Problem

    Urban VLN requires grounding human-written instructions in real-world panoramas and handling open-ended landmarks, spatial directions, and temporal reasoning.

    Source: §1 · p. 1, §4.1 · p. 3

  • Method

    VELMA prompts an LLM with verbalized task instructions, trajectory, and observations, using thresholded CLIP scores to represent visible instruction landmarks.

    Source: §5.2 · p. 4, §7 · p. 8

  • Results

    By 10%, finetuned VELMA-FT outperforms the previous state-of-the-art ORAR* in task completion.

    Source: §6.4 · p. 7

  • Takeaways & Limitations

    Two-shot results indicate that navigation skills can emerge when VLN is verbalized for an LLM relying on observation descriptions rather than direct image features.

    Source: §6.3 · pp. 6–7

Abstract

from arXiv · show

Incremental decision making in real-world environments is one of the most challenging tasks in embodied artificial intelligence. One particularly demanding scenario is Vision and Language Navigation~(VLN) which requires visual and natural language understanding as well as spatial and temporal reasoning capabilities. The embodied agent needs to ground its understanding of navigation instructions in observations of a real-world environment like Street View. Despite the impressive results of LLMs in other research areas, it is an ongoing problem of how to best connect them with an interactive visual environment. In this work, we propose VELMA, an embodied LLM agent that uses a verbalization of the trajectory and of visual environment observations as contextual prompt for the next action. Visual information is verbalized by a pipeline that extracts landmarks from the human written navigation instructions and uses CLIP to determine their visibility in the current panorama view. We show that VELMA is able to successfully follow navigation instructions in Street View with only two in-context examples. We further finetune the LLM agent on a few thousand examples and achieve 25%-30% relative improvement in task completion over the previous state-of-the-art for two datasets.

1 Introduction

Urban VLN requires an agent to ground human-written directions in real-world street scenes and reason over long trajectories. VELMA verbalizes the task, observations, and trajectory for an LLM, and reports few-shot and finetuned results for the task.

1 Introduction

  • The task combines real-world panoramas, a large road-network environment, and long routes, requiring instructions to be grounded in visible surroundings.Navigation directions include open-ended landmark references and directional indications intended to guide the agent along the path.Source: §1 · p. 1
  • VELMA verbalizes the navigation task, trajectory, and visual observations, appending new observations to the prompt as the LLM predicts each next action.Full trajectories average 40 steps.Source: §1 · p. 1, Figure 1 · p. 1
  • VELMA is presented as the first LLM-based agent for urban VLN, with few-shot results and state-of-the-art performance after finetuning.The work also modifies Touchdown to address limitations for few-shot agents.Source: §1 · p. 2

2 Related Work

Prior outdoor and indoor VLN agents use varied ways to represent visual input, while VELMA is distinguished by embodied, ad-hoc navigation from directly observed information. Urban VLN trajectories are substantially longer than the cited indoor LLM navigation setting.

Outdoor VLN

  • Outdoor urban VLN agents commonly fuse encoded text and image representations in sequence-to-sequence models or finetune pretrained vision-language transformers.Other approaches use semantic segmentation or CLIP-scored landmarks for route planning with full access to panorama information.Source: §2 · p. 2
  • Unlike prior CLIP-based route planning with full panorama access, VELMA plans ad hoc as an embodied agent using directly observed information.The cited route-planning work uses a non-urban graph of 300 nodes and plans a route in advance.Source: §2 · p. 2

Indoor VLN

  • Urban VLN trajectories average 40 steps, compared with six steps for the cited indoor LLM-based R2R agent that transcribes image content.The paper relates the shorter indoor trajectories to the feasibility of transcribing the entire image content.Source: §2 · p. 2

3 Urban VLN Environment

The section describes Touchdown’s Street View graph and its action mechanics, then addresses action-sequence mismatches at intersections and adds a dedicated direction-reversal action.

3 Urban VLN Environment

  • Touchdown uses 29,641 Street View panoramas connected in a directed graph covering lower Manhattan.The agent state comprises its position and heading, and its actions include FORWARD, LEFT, RIGHT, and STOP.Source: §3 · p. 2

3.1 Alignment Inconsistencies in Touchdown

  • At intersections, automatic heading alignment can make a semantically correct right turn require four FORWARD actions, misaligning actions with instructions.The modified transition rule avoids automatic rotation after FORWARD and selects left, forward, or right relative to all outgoing streets; the figure contrasts the resulting action sequences.Source: §3.1 · pp. 2–3, Figure 2 · p. 2

3.2 Turning Around

  • The added TURN AROUND action reverses the agent’s direction and aligns more directly with natural-language direction reversals.In the unmodified environment, reversal requires LEFT or RIGHT actions on regular street segments.Source: §3.2 · p. 3

4 Navigation Task

The task asks an agent to follow navigation instructions to a goal using observations and sequential actions. The section outlines the reasoning demands and datasets, while the overview figure depicts VELMA’s repeated action-and-observation loop.

4 Navigation Task

  • The agent predicts actions from instructions and current observations, repeats state transitions, and succeeds by stopping within one neighboring node of the target.The overview depicts the LLM action, new state, landmark visibility assessment, and verbalized observation being fed into the next prompt.Source: §4 · p. 3, Figure 3 · p. 4

4.1 Challenges

  • The task requires detecting open-ended street-scene landmarks, spatial reasoning about directions and stopping, and temporal reasoning over prior actions and observations.The table presents example instruction snippets for the required reasoning skills.Source: §4.1 · p. 3, Table 1 · p. 3

4.2 Datasets

  • Touchdown and Map2seq each contain around 10k instances and use geographically separated training and development/test routes in the unseen scenario.Touchdown instructions were written by annotators following Street View routes, whereas Map2seq instructions were written from maps and later validated in Street View; Map2seq starts facing the correct direction.Source: §4.2 · p. 3

5 LLM Agent

VELMA verbalizes the navigation task and environment observations as text for an LLM to use in predicting the next action. Its pipeline extracts instruction landmarks, scores their visibility, and verbalizes landmarks and intersections.

5 LLM Agent

  • At each timestep, VELMA concatenates the task description, navigation instructions, and observation/action history into a prompt for next-word prediction over five action literals.The language model is a black box with no vision capabilities.Source: §5 · p. 4

5.1 Landmark Extractor

  • Before navigation, an LLM extracts landmarks mentioned in the instructions using a prompt with five in-context instruction–landmark examples.The extractor runs before navigation starts.Source: §5.1 · p. 4

5.2 Landmark Scorer

  • CLIP compares landmark captions with panorama views, standardizes scores using views from about 20,000 training-area panoramas, and classifies a landmark as visible when its score exceeds threshold τ.The scorer is unsupervised, requires no annotations, and has τ as its only tunable parameter.Source: §5.2 · p. 4, Figure 4 · p. 5

5.3 Verbalizer

  • The verbalizer encodes intersections from navigation-graph outdegree and visible landmarks as text naming the landmark and its direction; it appends the observation to the agent prompt.The observation is empty when neither an intersection nor a landmark is detected.Source: §5.3 · p. 5

6 Experiments

Experiments assess VELMA in few-shot and finetuning settings, including landmark extraction, navigation metrics, and a vision ablation. Response-based finetuning yields the largest reported gains over prior state of the art.

6 Experiments

  • The experiments evaluate navigation performance in finetuning and in-context learning settings.The CLIP visibility threshold was set to 3.5 without systematic tuning to preserve the few-shot premise.Source: §6 · p. 5

6.1 Landmark Extraction

  • GPT-3 landmark extraction scores F1 96.3 on Touchdown and 99.6 on Map2seq, with precision/recall of 97.2/95.4 and 100/99.3, respectively.On average, 2.7 landmarks are extracted per instruction; around 58% of test landmarks are novel.Source: §6.1 · p. 5

6.2 Metrics and Baseline

  • Metrics are TC (stopping at the goal or an adjacent node), SPD (shortest-path distance), and KPA (correct key-point decisions); ORAR was retrained in the improved environment for fair comparison.Key points include the initial step, gold-route intersections, and target location.Source: §6.2 · pp. 5–6

6.4 Finetuning Results

  • 77% and 57% relative task-completion increases over prior state of the art are reported for Touchdown and Map2seq, respectively.These are the overall gains reported for the proposed training approach.Source: Table 2 · p. 6, Response-Based Learning · p. 7

6.3 Few-Shot Learning Results

  • Across model families, 2-shot KPA scales with parameter count, but LLaMa-2’s KPA of 48.3 on Touchdown and 57.7 on Map2seq corresponds to TC rates of only 2.1 and 3.2.GPT-3, GPT-4, and Mixtral are the only few-shot LLMs reported to achieve substantial task-completion rates.Source: §6.3 · p. 6, Figure 5 · p. 6

6.4 Finetuning Results

  • VELMA-FT exceeds ORAR* by 10% relative TC rate on Touchdown and 16% on Map2seq; the authors find text observations less prone to overfitting than image-feature fusion.VELMA-FT finetunes LLaMa-7b on all training instances for each dataset.Source: §6.4 · p. 7
  • RBL mixes teacher forcing with student trajectories, reusing successful trajectories and oracle shortest-path actions after failures; task completion increases by 2.9 and 7.5.The mixing parameter λ is set to 0.5.Source: Response-Based Learning · p. 7

6.5 Image Ablation

  • OpenCLIP detects landmarks better than original CLIP, while omitting landmarks further degrades performance, especially on Touchdown.Table 3 compares separately finetuned LLaMa-7b models on the development set.Source: §6.5 · p. 7, Table 3 · p. 7

7 Conclusion

VELMA verbalizes navigation instructions, visual observations, and trajectory for LLM action selection, and uses thresholded CLIP scores to identify visible landmarks. It achieves task completion rates of 10 and 23 in few-shot evaluation, and 26 and 47 after full-training-set finetuning, on Touchdown and Map2seq, respectively.

7 Conclusion

  • 10 and 23 task completion rates on Touchdown and Map2seq rise to 26 and 47, respectively, after full-training-set finetuning.The paper reports these as strong few-shot results and new state-of-the-art results after finetuning.Source: §7 · p. 8
  • VELMA prompts an LLM with verbalized task instructions, visual observations, and the agent’s past trajectory to infer its next action.An unsupervised pipeline extracts instruction landmarks and uses thresholded CLIP scores to determine their visibility in the current panorama.Source: §7 · p. 8
  • The agent is evaluated in a modified Street View-based Touchdown environment, including a fix for intersection alignments and a TURN AROUND action.The paper concludes that finetuning results show verbalization is not an inherent limitation for this task.Source: §7 · p. 8

A Finetuning Details

Finetuning uses official LLaMA-7b weights with LoRA adapters on q proj and v proj, Adam optimization, and greedy decoding.

A Finetuning Details

  • The experiments use official LLaMA-7b weights and finetune LoRA adapters for q proj and v proj.LoRA settings are r = 8, alpha = 16, dropout = 0.05, and no bias.Source: A Finetuning Details · p. 11
  • Adam is used with a learning rate of 0.0003, a 0.1 warmup ratio, and linear decay.Source: A Finetuning Details · p. 11
  • Training uses batch size 16 for 20 epochs, with greedy decoding for all experiments.Source: A Finetuning Details · p. 11

B Modified Environment

The modified Touchdown environment changes intersection action sequences, which the authors describe as more intuitive and necessary for few-shot agents.

B Modified Environment

  • The environment modifications are proposed in Section 3.1 for the environment introduced by Chen et al. (2019).Source: B Modified Environment · p. 11
  • Table 4 overviews action sequences for clearing 3-way, 4-way, and 5-way intersections in different directions under the original and modified implementations.Source: B Modified Environment · p. 11
  • The authors describe the modified intersection sequences as more intuitive and necessary for few-shot agents to interact with the environment.Source: B Modified Environment · p. 11

C Landmark Extraction

Instruction landmarks are extracted before navigation begins using a separate GPT-3 prompt with in-context examples; the absence of gold annotations precludes quantitative evaluation.

C Landmark Extraction

  • Landmarks are extracted before each run through a separate prompt fed to GPT-3.Source: C Landmark Extraction · p. 11
  • The prompt uses five paired instruction texts and extracted-landmark lists as in-context examples, compiled by hand, and is reused for every instance.Source: C Landmark Extraction · p. 11
  • Because there are no gold annotations for extracted landmarks, the paper cannot quantitatively evaluate extraction.Source: C Landmark Extraction · p. 11
  • Figures 8 and 9 show the prompts for Map2seq and Touchdown instructions, and Figure 10 shows GPT-3-extracted landmarks.Source: C Landmark Extraction · p. 11

D Landmark Scorer

VELMA uses landmark visibility scoring and verbalized prompts to support stepwise navigation in Street View. The appendix illustrates landmark extraction, scorer behavior, and the full prompt sequence.

D Landmark Scorer

  • For Touchdown instructions about traffic flow, VELMA compares CLIP scores for front and rear vehicle views and reports the inferred orientation before the first step.The higher-scoring phrase determines whether the agent faces with or against traffic.Source: D Landmark Scorer · p. 11

E Full Prompt Sequence

  • Touchdown and Map2seq routes average 40 steps, requiring agents to evaluate 200 panorama views per navigation instance.The displayed trajectory is shortened for visualization.Source: E Full Prompt Sequence · p. 11

D Landmark Scorer

  • The scorer standardizes CLIP similarity scores between landmark captions and training-area panoramas, then applies threshold τ to determine landmark visibility in a novel panorama.The score compares each panorama with the caption “picture of [landmark]”.Source: Figure 6 · p. 12

E Full Prompt Sequence

  • The prompt sequence appends verbalized visual observations at each step, while the LLM predicts the next action.Figure 7 depicts a completed route in which the agent predicts STOP at timestep 14.Source: E Full Prompt Sequence · p. 11, Figure 7 · p. 12
  • The improved Touchdown implementation makes intersection-clearing action sequences semantically aligned with their expected outcomes.Source: Table 4 · p. 13

D Landmark Scorer

Loading 2307.06082v2…