Source-linked AI summary
Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang
TL;DR
VLN research has not fully exploited rich environmental semantics needed for navigation reasoning. AuxRN adds four self-supervised auxiliary reasoning tasks, and experiments report improved performance and generalizability, including leading benchmark results.
Problem
VLN methods have neglected rich environmental semantics, including navigation-graph structure and sub-trajectory information, limiting the signals available for navigation reasoning.
Method
AuxRN jointly uses trajectory retelling, progress estimation, angle prediction, and cross-modal matching as self-supervised auxiliary reasoning tasks.
Results
65% SPL after unseen-environment finetuning is 4% higher than the previous state-of-the-art result, while all auxiliary tasks together improve validation performance by 3.02% on seen and 2.78% on unseen environments.
Takeaways & Limitations
Auxiliary reasoning tasks improve VLN performance on seen and unseen environments and help the model generalize across environments.
Takeaways & Limitations
Angle prediction requires teacher-action supervision and is therefore not forwarded during reinforcement learning.
Abstract
from arXiv · showhide
Vision-Language Navigation (VLN) is a task where agents learn to navigate following natural language instructions. The key to this task is to perceive both the visual scene and natural language sequentially. Conventional approaches exploit the vision and language features in cross-modal grounding. However, the VLN task remains challenging, since previous works have neglected the rich semantic information contained in the environment (such as implicit navigation graphs or sub-trajectory semantics). In this paper, we introduce Auxiliary Reasoning Navigation (AuxRN), a framework with four self-supervised auxiliary reasoning tasks to take advantage of the additional training signals derived from the semantic information. The auxiliary tasks have four reasoning objectives: explaining the previous actions, estimating the navigation progress, predicting the next orientation, and evaluating the trajectory consistency. As a result, these additional training signals help the agent to acquire knowledge of semantic representations in order to reason about its activity and build a thorough perception of the environment. Our experiments indicate that auxiliary reasoning tasks improve both the performance of the main task and the model generalizability by a large margin. Empirically, we demonstrate that an agent trained with self-supervised auxiliary reasoning tasks substantially outperforms the previous state-of-the-art method, being the best existing approach on the standard benchmark.
1. Introduction
VLN agents must integrate sequential visual and linguistic information, but prior approaches neglect environmental semantics needed to understand activity, progress, and reachable actions. AuxRN adds four auxiliary reasoning tasks, improving performance on seen and unseen environments.
- VLN agents navigate 3D indoor environments by following natural-language instructions while receiving visual observations at each step.
- Prior VLN methods emphasize vision-language perception, historical encoding, and cross-modal matching, while neglecting rich semantic information in the environment.
- Environmental semantics matter because past actions affect future decisions, trajectory-instruction alignment is uncertain, progress labels can be noisy, and actions are constrained by neighboring graph nodes.
- AuxRN introduces trajectory retelling, progress estimation, angle prediction, and cross-modal matching as auxiliary reasoning tasks that use temporal context alongside single-step inputs.
- 3.45% improvement over the baseline on the validation set is achieved by AuxRN pretrained in seen environments with auxiliary reasoning tasks.
- 65% SPL after finetuning on unseen environments is 4% higher than the previous state-of-the-art result, ranking first in the VLN Challenge by SPL.
2. Related Work
Related VLN work combines reinforcement and imitation learning, data augmentation, panoramic action spaces, progress monitoring, and methods for improving generalization. AuxRN is presented against this progression as a framework that integrates auxiliary reasoning with vision-language navigation.
- VLN research has explored reinforcement learning, imitation learning, and supervised speaker-follower frameworks for navigation and data augmentation.
- The panoramic action space represents candidate navigation directions across a 36-view panorama and includes neighboring graph nodes plus a stop action.
- AuxRN embeds vision and language features, performs co-attention, supervises reasoning modules with auxiliary losses, and fuses attended features with candidate features for action prediction.
- Self-monitoring methods estimate progress toward the goal, while unsupervised pre-exploration and environmental dropout address the domain gap between training and testing data.
3. Method
AuxRN combines vision-language navigation with imitation and reinforcement learning, while jointly training four auxiliary reasoning objectives to exploit semantic information from trajectories and environments.
- Problem Setup: VLN represents instructions as word sequences, panoramic observations as 36 vision-orientation views, and actions as navigable neighbors plus a stop action.The number of neighboring candidates is not fixed.
- Vision-Language Forward: The model uses attention to fuse feature sequences with query vectors, then maintains trajectory-level visual history with an LSTM.Attention weights determine each feature’s contribution to the fused representation.
- Objectives for Navigation: Action prediction selects among reachable candidates using teacher actions for imitation learning, sampled probabilities for reinforcement learning, or the highest-probability candidate at test time.The navigation objective jointly optimizes imitation learning and reinforcement learning, using A2C for the latter.
- Auxiliary Reasoning Learning: AuxRN adds trajectory retelling, progress estimation, angle prediction, and cross-modal matching to provide semantic reasoning signals alongside action prediction.The four auxiliary losses are jointly trained end-to-end with the total objective Ltotal = LSpeaker + LProgress + LAngle + LMatching.
- Auxiliary Reasoning Learning: Trajectory retelling explains previous actions, while progress estimation predicts completed trajectory percentage and aligns current views with instruction words.The retelling objective is jointly optimized with the main task but is not trained in reinforcement-learning scenarios because student forcing can cause large deviations.
- Auxiliary Reasoning Learning: Cross-modal matching detects shuffled vision-language features to encode trajectory-instruction consistency, while angle prediction learns the teacher action’s orientation during imitation learning.Angle prediction is not forwarded in reinforcement learning because it requires a teacher angle for supervision.
4. Experiment
AuxRN is evaluated on R2R across seen and unseen environments, multiple training settings, auxiliary-task ablations, and visualizations. It consistently improves navigation performance, with evidence that the auxiliary tasks complement one another and support progress, consistency, and language-attention reasoning.
- Experimental Setup: The R2R dataset uses 90 housing environments split into training, seen-validation, unseen-validation, and test sets.Training includes 61 environments and 14,025 instructions; unseen validation includes 11 environments and 2,349 instructions.
- Experimental Setup: AuxRN training proceeds through pretraining, finetuning on augmented training data, and optional pre-exploration finetuning in unseen environments.Each training stage runs for 80K iterations.
- Test Set Results: AuxRN outperforms prior models across Single Run, Pre-explore, and Beam Search settings.Single Run and Pre-explore prioritize SPL, while Beam Search compares success rate.
- Test Set Results: 71% success rate is achieved with Beam Search, 2% higher than Environmental Dropout.This result establishes a state-of-the-art result for the Beam Search setup.
- Auxiliary Reasoning Task Comparison: Training all auxiliary tasks together improves validation performance by 3.02% on seen environments and 2.78% on unseen environments.Each task independently improves the baseline, while their combination produces further gains.
- Trajectory Retelling Ablation: Teacher forcing performs better than matching critic and student forcing for trajectory retelling, while shared-context teacher forcing shows higher word-prediction accuracy but lower unseen-validation SPL.The paper interprets the shared-context result as overfitting on the trajectory-retelling task.
- Progress Estimation Ablation: BCE loss with step-wise progress labels outperforms MSE-based alternatives, with a 0.14 higher error for Step-wise+MSE on seen validation and 0.16 higher on unseen validation.The reported performance advantage is 6.34% on seen validation and 1.58% on unseen validation.
- Visualization: AuxRN visualizations show increasing estimated progress and matching scores that approach 1 when the agent reaches the goal.The authors use these patterns to indicate accurate progress estimation and instruction-trajectory consistency.
5. Conclusion
AuxRN is a navigation-learning framework built around four auxiliary reasoning tasks. Experiments confirm quantitative and qualitative performance improvements for VLN.
- AuxRN facilitates navigation learning through four auxiliary reasoning tasks.
- Experiments confirm that AuxRN improves VLN performance quantitatively and qualitatively.