Source-linked AI summary
Object-and-Action Aware Model for Visual Language Navigation
Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, Qi Wu
TL;DR
VLN methods often mix object descriptions and action specifications, despite their different visual and orientation counterparts. OAAM separates and matches these cues, and the combined model and path loss outperform prior methods on unseen scenarios, including a 0.40 CLS score on unseen R4R.
Problem
Existing approaches entangle object descriptions and action specifications, although they correspond to visual perceptions and viewpoint orientations, respectively.
Source: §1 · p. 3Method
OAAM separately attends to object- and action-centered instructions, matches them to visual perceptions and orientations, and combines their predictions with adaptive weights; a path loss encourages following the ground-truth path.
Source: §1 · p. 3, §3.3 · p. 8Results
Across unseen R2R and R4R, OAAM with path loss outperforms prior methods; unseen R4R CLS is 0.40, 3% higher than the second best.
Source: §4.4 · p. 11Takeaways & Limitations
The combined model and path loss show superiority over several state-of-the-art navigators, especially on unseen test scenarios.
Source: §5 · p. 12On Val Seen, the model ranks third behind FAST and RA, which the authors say may indicate weaker fit to the training data.
Source: §4.4 · p. 11
Abstract
from arXiv · showhide
Vision-and-Language Navigation (VLN) is unique in that it requires turning relatively general natural-language instructions into robot agent actions, on the basis of the visible environment. This requires to extract value from two very different types of natural-language information. The first is object description (e.g., 'table', 'door'), each presenting as a tip for the agent to determine the next action by finding the item visible in the environment, and the second is action specification (e.g., 'go straight', 'turn left') which allows the robot to directly predict the next movements without relying on visual perceptions. However, most existing methods pay few attention to distinguish these information from each other during instruction encoding and mix together the matching between textual object/action encoding and visual perception/orientation features of candidate viewpoints. In this paper, we propose an Object-and-Action Aware Model (OAAM) that processes these two different forms of natural language based instruction separately. This enables each process to match object-centered/action-centered instruction to their own counterpart visual perception/action orientation flexibly. However, one side-issue caused by above solution is that an object mentioned in instructions may be observed in the direction of two or more candidate viewpoints, thus the OAAM may not predict the viewpoint on the shortest path as the next action. To handle this problem, we design a simple but effective path loss to penalize trajectories deviating from the ground truth path. Experimental results demonstrate the effectiveness of the proposed model and path loss, and the superiority of their combination with a 50% SPL score on the R2R dataset and a 40% CLS score on the R4R dataset in unseen environments, outperforming the previous state-of-the-art.
1 Introduction
VLN methods often entangle object and action cues despite their different visual and orientation counterparts. OAAM separates these cues, adds a path loss to encourage path following, and reports strong benchmark results.
1 Introduction
- Existing approaches entangle object descriptions and action specifications, although the former relate to visual perceptions and the latter to viewpoint orientations.The paper argues this mixed encoding may limit matching between instructions and visual perceptions or orientations.Source: §1 · p. 3
- OAAM uses separate attention and matching modules for object-centered visual cues and action-centered orientations, then combines their predictions with context-dependent weights.The combination weights are derived from the instruction and visual context.Source: §1 · p. 3
- The path loss penalizes deviation by measuring each agent step against its nearest viewpoint on the ground-truth path.It is distinct from CLS, which finds the nearest agent-trajectory viewpoint for each ground-truth-path viewpoint; the authors report that the loss aids unseen-scenario generalization.Source: §1 · p. 3
- 0.40 CLS on R4R sets a new state of the art, according to the introduction's reported experiments.Source: §1 · p. 4
2 Related Work
Prior VLN work includes a range of navigation and training strategies, but generally does not separate object and action cues. The paper proposes progress-aware decomposition that matches each cue to its corresponding viewpoint information.
Vision-and-Language Navigation
- Prior VLN approaches include training strategies, data augmentation, progress monitoring, backtracking, and candidate-viewpoint search.Source: §2 · p. 4
- Earlier methods neglect to distinguish object descriptions from action specifications, which may limit matching to their corresponding visual perceptions and orientations.Source: §2 · p. 4
- The proposed decomposition matches object and action components to visual perceptions and orientations, respectively, while focusing on the instruction portion relevant to the agent's current location.Source: §2 · p. 5, Figure 2 · p. 5
3 Method
The method extends EnvDrop with an object-and-action aware navigator that separately matches instruction components to visual features and viewpoint orientations. It combines those predictions adaptively and adds a nearest-point loss to encourage trajectories to stay on ground-truth paths.
3 Method
- The method encodes an instruction, decomposes it into object- and action-centered representations, and uses their matching scores to compute action probabilities.During inference, it selects the navigation action with maximum probability.Source: §3 · pp. 5–6
3.1 Problem Setup
- At each step, the agent selects a navigable viewpoint using visual and orientation features, with 36 discrete views in its panoramic observation.Navigable viewpoints have orientation features and ResNet visual features.Source: §3.1 · p. 6
3.2 Baseline–EnvDrop
- EnvDrop encodes instructions with a bidirectional LSTM and uses attended visual features in its decoder to predict the next navigation action.The method adopts EnvDrop, which also uses back translation and global dropout to augment training data and environments.Source: §3.2 · p. 6
3.3 Object-and-Action Aware Model
- OAAM separately predicts actions from object-centered instruction matched to visual features and action-centered instruction matched to candidate-view orientations.The two attention modules account for navigation progress, and an adaptive module combines their predictions using weights derived from the attentive instruction.Source: §3.3 · pp. 7–8
3.4 Training Loss
- The model combines imitation and reinforcement learning losses with a nearest-point loss that encourages the agent to stay on ground-truth paths.The loss uses the shortest trajectory distance from the current viewpoint to a viewpoint on the ground-truth path.Source: §3.4 · p. 8
4 Experiments
Experiments evaluate OAAM and nearest point loss on R2R and R4R, using navigation metrics suited to each dataset. The combined approach leads the reported unseen and test comparisons, with complementary ablations and qualitative examples.
4.1 Dataset and Implementations
- R2R contains 7,189 paths and 21,567 instructions, while R4R concatenates R2R trajectories and instructions; experiments use precomputed ResNet features and real-then-augmented training.Augmented data are generated using speaker and environment-dropout techniques.Source: §4.1 · p. 9
4.2 Evaluation Metrics
- R2R uses SPL as its main metric, while R4R uses CLS and SDTW to measure navigation fidelity against the ground-truth path.R2R evaluation also includes SR, TL, and OSR.Source: §4.2 · p. 9
4.3 Ablation Study
- About 4% SR and 3% SPL gains on Val Unseen appear when OAAM and NPL combine, while either OA or AA alone drops SR about 20%.AA outperforms OA across splits; average combination weights are 0.18 for OA and 0.82 for AA. Top-three noun/verb attention tags score about 80% for OA and 75% for AA.Source: Table 1 · p. 10, §4.3 · pp. 10–11, Table 2 · p. 11
4.5 Qualitative Results
- Qualitative trajectories show OA and AA attention often matches relevant objects or actions; at some midway steps, only one module is correct but the combined prediction is correct.Other illustrated trajectories include incorrect midway decisions followed by reaching the goal.Source: §4.5 · pp. 11–12
4.4 Comparison to State-of-The-Art Navigators
- On R2R, OAAM ranks best on Val Unseen and Test at up to 50% SPL; on R4R, it scores 0.40 CLS unseen, 3% above second-best.On Val Seen, it ranks third on R2R and in the top three on R4R.Source: §4.4 · p. 11, Table 3 · p. 12, Table 4 · p. 12
5 Conclusion
OAAM separately matches object and action information in instructions to visual perceptions and orientation embeddings, and adds a path loss based on distance to the nearest ground-truth viewpoint. The paper reports stronger performance than several state-of-the-art methods, especially in unseen scenarios.
5 Conclusion
- The model outperforms several state-of-the-art navigation methods, especially in unseen test scenarios.The conclusion reports the overall comparison without giving metric values here.Source: §5 · p. 12
- OAAM separately highlights object and action information, matching them to visual perceptions and orientation embeddings, respectively.Its path loss uses distance to the nearest ground-truth viewpoint to encourage the agent to stay on the path.Source: §5 · p. 12
- In unseen-environment samples, darker instruction text indicates higher relative attention, numbered circles mark navigable viewpoints, and blue marks the final decision.Source: Figure 5 · p. 13
- In unseen-environment examples, OA and AA modules can each err while their combination predicts the correct action; the authors say all three modules are crucial to success.One example also shows an incorrect action at step 3 followed by arrival at the goal room.Source: Figure 6 · p. 14