Source-linked AI summary
Language and Visual Entity Relationship Graph for Agent Navigation
Yicong Hong, Cristian Rodriguez-Opazo, Yuankai Qi, Qi Wu, Stephen Gould
TL;DR
Vision-and-language navigation must connect complex language instructions with scene, object, and directional visual clues, but prior methods rarely model these relationships explicitly. The paper introduces interacting language and visual entity graphs with message passing, achieving improved results on R2R and R4R benchmarks. The authors report new state-of-the-art performance on R2R and a 21% absolute SDTW improvement on R4R over the previous best.
Problem
R2R requires step-wise correspondence between complex visual clues and natural-language instructions, while prior agents rarely consider instruction-mentioned objects with strong localization signals.
Method
The method uses interacting language attention and language-conditioned visual graphs to model relationships among scene, object, and directional clues and propagate information for action prediction.
Results
The method improves over existing methods on R2R and R4R, achieving new state-of-the-art results on R2R and a 21% absolute improvement in SDTW over the previous best on R4R.
Takeaways & Limitations
Modeling relationships among scene, objects, directions, and language helps clarify instruction ambiguity and build a more comprehensive environmental perception.
Takeaways & Limitations
Objects are used only as visual features, not for progress monitoring, instance tracking, or reinforcement-learning reward shaping.
Abstract
from arXiv · showhide
Vision-and-Language Navigation (VLN) requires an agent to navigate in a real-world environment following natural language instructions. From both the textual and visual perspectives, we find that the relationships among the scene, its objects,and directional clues are essential for the agent to interpret complex instructions and correctly perceive the environment. To capture and utilize the relationships, we propose a novel Language and Visual Entity Relationship Graph for modelling the inter-modal relationships between text and vision, and the intra-modal relationships among visual entities. We propose a message passing algorithm for propagating information between language elements and visual entities in the graph, which we then combine to determine the next action to take. Experiments show that by taking advantage of the relationships we are able to improve over state-of-the-art. On the Room-to-Room (R2R) benchmark, our method achieves the new best performance on the test unseen split with success rate weighted by path length (SPL) of 52%. On the Room-for-Room (R4R) dataset, our method significantly improves the previous best from 13% to 34% on the success weighted by normalized dynamic time warping (SDTW). Code is available at: https://github.com/YicongHong/Entity-Graph-VLN.
1 Introduction
Vision-and-language navigation requires agents to align complex instructions with visual clues in unseen environments. The paper models relationships among scene, object, and directional clues with interacting language and visual graphs, improving benchmark performance.
- R2R requires agents to follow natural-language instructions through unseen photo-realistic environments without explicit alignment between sub-instructions and path segments.
- Previous agents often concatenate image and directional features, while rarely modeling instruction-mentioned objects that provide strong localization signals.
- Navigation instructions combine scene, object, and directional clues whose relationships clarify the intended action and destination.
- The proposed entity relationship graph models inter- and intra-modal relationships through language attention and language-conditioned visual subgraphs.The visual graph represents scene, object, and direction features; propagated information updates nodes used to determine action probabilities.
- On R2R, the end-to-end agent significantly outperforms the baseline and achieves state-of-the-art performance on the test unseen split.On R4R, it obtains a 21% absolute improvement in SDTW over the previous best.
2 Related Work
Prior vision-and-language navigation research developed reinforcement-learning, self-supervised, path-monitoring, and path-recovery approaches. Related graph-based work models relationships across activities, scenes, objects, and language, motivating graph representations for navigation.
- Vision-and-language navigation: Vision-and-language navigation research includes model-based and model-free reinforcement learning, self-supervised instruction augmentation, progress monitoring, and path-scoring or backtracking methods.
- Graphs for relationship modelling: Graph neural networks have been applied to model inter- and intra-modality relationships in activities and vision-and-language tasks.Prior representations include spatial-temporal human-object graphs and graphs over scene objects.
3 Language and Visual Entity Relationship Graph
The proposed graph models specialized language contexts and related visual entities to connect scene, object, and directional information during navigation. Language-conditioned message passing updates these visual nodes before predicting an action for each candidate direction.
- Graph-based navigation pipeline: At each navigation step, the agent encodes panoramic observations, candidate directions, detected objects, and the instruction before graph-based decoding.The visual input includes directional encodings and object features, while a bidirectional LSTM produces word representations.
- Language attention graph: The language attention graph extracts three specialized contexts corresponding to scene, object, and direction information.These contexts are obtained through independent soft-attention operations conditioned on the current agent state.
- Language attention graph: A second attention level models relational contexts between specialized language nodes, including scene-direction, scene-object, and object-direction relationships.The graph uses these relations to interpret directional instructions in their scene and landmark context.
- Language-conditioned visual graph: The language-conditioned visual graph represents scene, object, and direction features for each candidate direction and links them to the corresponding language contexts.Projected visual features are combined with specialized contexts through element-wise products, and the direction node carries temporal information from the previously selected action.
- Language-conditioned visual graph: Language-conditioned message passing exchanges relation-guided information among visual nodes, after which each node is updated by adding received messages to its initial feature.The updated scene, object, and direction nodes are combined to infer an action probability for each candidate direction.
- Training: The network is trained with combined imitation-learning and reinforcement-learning objectives to balance teacher-guided learning with exploration.The reinforcement-learning component uses rewards, while the objective weights imitation loss with coefficient λ.
4 Experiments
Experiments evaluate the graph agent on R2R and R4R using standard navigation, efficiency, and path-fidelity metrics, with comparisons, ablations, and attention visualizations. The results show improved state-of-the-art performance, benefits from relationship modeling and object clues, and complementary strengths between models with and without object visual features.
- Experimental setup: The R2R and R4R experiments use standard navigation, efficiency, and path-fidelity metrics, with object features extracted from 101 simplified categories.R2R includes TL, NE, SR, and SPL; R4R additionally uses CLS, nDTW, and SDTW.
- Benchmark results: On R2R, the method achieves a 5% absolute SPL improvement over EnvDrop on both unseen splits and the best test results on NE, SR, and SPL.Its test navigation error is 0.39m shorter than the previous best.
- Benchmark results: On R4R, nDTW and SDTW increase by 15% and 21%, respectively, over previous state-of-the-art performance on the validation unseen split.The authors interpret these gains as better instruction following and path fidelity.
- Ablation analysis: Ablations show that object features require relationship modeling to help, scene features are essential, and textual-visual connections contribute substantially to performance.Introducing object features also significantly decreases trajectory length, likely because objects provide strong localization signals.
- Object-clue analysis: The full model outperforms the no-object model across almost all instruction groups, with larger success-rate gains as the number of referenced objects increases.This indicates that object visual clues help the agent exploit object mentions in longer or more object-rich instructions.
- Attention analysis: Attention visualizations show specialized contexts focusing on scene, object, and direction terms, while relational attention links clues such as stairs, exit signs, and walking direction.At the first step, the relational context can suggest searching for an exit sign before it appears visually.
5 Conclusion and Future Direction
The paper uses a language and visual entity relationship graph to connect scenes, objects, and directional clues during navigation. It reports improved performance on R2R and R4R, while identifying richer object use as future work.
- Conclusion: The proposed graph exploits connections among scenes, objects, and directional clues to clarify instructions and build a comprehensive environmental perception.Its relationships span language and visual entities during navigation.
- Conclusion: The graph networks improve over existing methods on the R2R and R4R benchmarks and achieve new state-of-the-art performance.
- Future Direction: The current method uses objects as visual features rather than for progress monitoring, instance tracking, or reinforcement-learning reward shaping.The authors identify these uses as potential future research directions.
Broader Impact
The experiments use licensed indoor Matterport3D photos without recognizable individuals in a safe, confidential simulator. The work remains at an early stage toward real-world instruction-following robots.
- Broader Impact: Experiments use licensed indoor Matterport3D images without recognizable individuals and run in the safe, confidential Matterport3D Simulator.
- Broader Impact: The research is at an early stage of developing robots that follow human instructions in the real world.