Source-linked AI summary
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, Jason Baldridge
TL;DR
VLN research has lacked clear evidence about language’s role because R2R paths are shortest-to-goal and common metrics emphasize arrival over route fidelity. The paper introduces CLS and R4R to evaluate instruction-following trajectories, finding that fidelity-oriented rewards improve CLS and reduce navigation error.
Problem
R2R’s shortest-to-goal paths and goal-oriented metrics make instruction fidelity difficult to evaluate in VLN.
Method
The paper proposes CLS, which measures trajectory conformity to the reference path, and constructs R4R by joining existing R2R paths and instructions.
Results
34.6% CLS versus 20.4% shows that CLS-rewarded agents outperform goal-oriented agents, while navigation error decreases from 8.45m to 8.08m on Validation Unseen.
Takeaways & Limitations
CLS and R4R provide a toolkit for measuring language understanding through path fidelity in VLN.
Takeaways & Limitations
SPL is ineffective or misleading for reference paths whose starts and ends are close because it assumes the shortest start-to-goal distance is optimal.
Abstract
from arXiv · showhide
Advances in learning and representations have reinvigorated work that connects language to other modalities. A particularly exciting direction is Vision-and-Language Navigation(VLN), in which agents interpret natural language instructions and visual scenes to move through environments and reach goals. Despite recent progress, current research leaves unclear how much of a role language understanding plays in this task, especially because dominant evaluation metrics have focused on goal completion rather than the sequence of actions corresponding to the instructions. Here, we highlight shortcomings of current metrics for the Room-to-Room dataset (Anderson et al.,2018b) and propose a new metric, Coverage weighted by Length Score (CLS). We also show that the existing paths in the dataset are not ideal for evaluating instruction following because they are direct-to-goal shortest paths. We join existing short paths to form more challenging extended paths to create a new data set, Room-for-Room (R4R). Using R4R and CLS, we show that agents that receive rewards for instruction fidelity outperform agents that focus on goal completion.
1 Introduction
The paper argues that R2R’s shortest-to-goal paths and goal-oriented metrics understate instruction following in VLN. It introduces CLS and R4R, showing that rewarding fidelity improves performance and makes language use measurable.
- Motivation: R2R paths and dominant metrics diminish the importance of language understanding by emphasizing direct goal completion over fidelity to the described route.All R2R reference paths are direct-to-goal shortest paths, while standard metrics can reward reaching the goal without following instructions.
- Contributions: CLS measures how closely an agent’s trajectory conforms to the entire reference path, addressing the insensitivity of existing metrics to instruction fidelity.The metric is proposed alongside R4R to evaluate whether agents follow the route described by language, not merely whether they arrive.
- Contributions: R4R composes existing R2R paths into longer, twistier paths, making deviations from the instructed journey more consequential and easier to evaluate.The construction uses path pairs whose goals and starts coincide or are sufficiently close.
- Results: 34.6% CLS versus 20.4% shows that CLS-rewarded agents outperform agents trained with goal-oriented rewards on R4R Validation Unseen.Navigation error also decreases from 8.45m to 8.08m in the same comparison.
- Results: 34.6 CLS versus 25.3 shows that the CLS-rewarded agent remains more sensitive to the full instruction when only the last five tokens are provided.The goal-oriented agent stays at CLS 20.4 regardless of whether the full instruction or only its last five tokens is available.
2 Extending R2R to create R4R
R4R extends R2R’s short, shortest-to-goal examples by joining compatible paths and their instructions. The resulting longer paths provide greater variety for studying conformity between language and navigation trajectories.
- R2R: R2R pairs natural-language instructions with graph paths through photo-realistic building and house interiors.Nodes represent possible positions, edges represent navigable direct paths, and each node provides an egocentric panoramic view.
- Motivation: Because shortest-to-goal paths decouple instruction conformity from reaching the destination, R2R’s largest path having only 6 edges limits evaluation of complete instructions.The paper therefore motivates larger and more diverse reference paths.
- Room-for-Room (R4R): R4R creates longer, twistier paths by joining paths whose endpoints are within a distance threshold, inserting the shortest connecting path when needed.For paths A and B, the connector C is empty when A’s endpoint equals B’s start; otherwise C is the shortest path between them.
- Room-for-Room (R4R): Each combination of instructions for joined paths A and B is included in R4R, producing N_A · N_B instruction combinations for the extended path.The dataset construction uses existing human annotations rather than additional human or low-fidelity machine annotations.
3 Evaluation Metrics in VLN
The paper argues that VLN evaluation should measure fidelity to the reference journey, not only whether agents reach the goal. It introduces CLS as a continuous path-fidelity metric that combines reference-path coverage with an appropriate length penalty.
- Desiderata: VLN metrics should compare all nodes in predicted and reference paths and penalize deviations even when they lead to the same goal.The desiderata also require larger discrepancies to receive larger penalties and an exact match to be the unique perfect score.
- Motivation: Goal-focused metrics can reward paths that reach the destination while deviating from the instructed trajectory.SPL accounts for success and normalized predicted-path length but ignores similarity to intermediary reference-path nodes.
- Existing Metrics: SPL evaluates destination success and path length but fails to capture reference-trajectory similarity and can assign optimal scores to multiple goal-reaching paths.These shortcomings violate the path-similarity and unique-optimum desiderata.
- Existing Metrics: SED compares action sequences with edit distance, but treats actions as identical or different without measuring how far their endpoints are in the graph.Figure 4 illustrates this distinction: SED gives zero to both alternatives, whereas CLS gives 0.89 to orange and 0.48 to red.
- CLS: CLS is the product of Path Coverage and Length Score, combining continuous coverage of reference nodes with a penalty for predicted paths shorter or longer than expected.Path Coverage uses exponentially decayed graph distance to the nearest predicted node, while Length Score compares predicted length with expected optimal length given coverage.
- CLS: CLS addresses the stated desiderata under an acyclic reference-path and shared-start assumption while remaining invariant to graph scale.Its graph-distance computations can be pre-computed with Dijkstra’s algorithm.
4 Agent
The agent combines visual and language encoders with a recurrent policy, then trains through behavioral cloning and policy-gradient updates. It compares goal-oriented and instruction-fidelity rewards, including CLS-based sparse rewards.
- Agent: The RCM agent is extended with reward functions based on CLS and success rate.This enables comparison between instruction-fidelity and goal-completion objectives.
- Navigator: The reasoning navigator learns a policy mapping the instruction and initial visual scene to actions while its LSTM state encodes prior scenes and actions.The state update uses the current visual representation and previous action.
- Navigator: Language instructions are encoded with fine-tuned GloVe embeddings and a bidirectional recurrent network into contextual representations.Tokens occurring fewer than five times are excluded from the restricted vocabulary, with others mapped to a shared OOV identifier.
- Navigator: At each step, attention pools panoramic visual features, while available actions represent navigable directions whose number varies by location.The panoramic view is discretized into 36 angles, combining CNN features with orientation features.
- Learning: Training uses behavioral cloning to initialize the policy, followed by REINFORCE policy-gradient updates using sampled trajectories and an advantage-scaled gradient.Interleaving policy-gradient batches with behavior-cloning batches, with the cloning ratio decaying, improves validation performance.
- Reward: The fidelity-oriented sparse reward combines successful termination with CLS against the instruction-associated reference path.The authors report no noticeable performance difference from adding CLS reward shaping at earlier time steps, so experiments use the sparse form.
5 Results
On R4R, fidelity-oriented rewards improve conformity to reference paths, while R2R results show that shortest-to-goal structure can make goal completion appear instruction-faithful. CLS exposes these differences more effectively than SPL in twistier settings.
- Language Ablations: Removing all but the last five instruction tokens lowers the fidelity-oriented agent’s CLS from 34.6% to 25.3% on Validation Unseen.The goal-oriented agent has the same CLS with full instructions and with only the last five tokens.
- R2R Performance: On R2R, goal-oriented and fidelity-oriented agents show no major differences because shortest-to-goal paths implicitly signal path conformity.An agent optimized for the target may therefore appear to follow instructions incidentally.
- Metric Analysis: SPL can favor the goal-oriented agent despite substantially poorer CLS conformity, making it misleading when reference paths twist or begin and end near each other.SPL assumes the optimal path is the shortest distance between starting and ending positions.
6 Conclusion
CLS and R4R provide tools for evaluating language understanding through path fidelity, while R4R remains challenging and leaves substantial room for improvement. The paper argues that future VLN agents and datasets should connect language to environmental action more effectively.
- CLS and R4R provide a better toolkit for measuring the impact of language understanding in VLN.
- 34.6 CLS is achieved by the authors’ RCM reimplementation on R4R Validation Unseen paths.
- 8.08 meters average navigation error for the best agent on R4R leaves substantial headroom relative to humans’ 1.61-meter average error on R2R.
- Path fidelity may be crucial in VLN scenarios where direct routes lead through traps, opponents, or dangerous areas.The paper gives games and search-and-rescue human-robot teams as examples.