Source-linked AI summary
Visual Semantic Navigation using Scene Priors
Wei Yang, Xiaolong Wang, Ali Farhadi, Abhinav Gupta, Roozbeh Mottaghi
TL;DR
The paper asks how semantic and functional priors can guide navigation toward objects in unfamiliar scenes, including unseen targets. It encodes those priors with Graph Convolutional Networks inside a deep reinforcement learning policy, and reports better navigation and generalization to unseen scenes and targets.
Problem
Semantic and functional priors help agents infer likely locations for target objects in unfamiliar scenes and transfer search strategies to unseen objects.
Method
Graph Convolutional Networks encode knowledge-graph priors and update them from current observations within an actor-critic deep reinforcement learning framework.
Results
The proposed model outperforms baselines on success rate and SPL, with proper knowledge graphs reaching SPL 38.5 versus 32.5 for fully connected and 30.1 ± 0.6 for random graphs.
Takeaways & Limitations
Semantic prior knowledge improves navigation performance and generalization to unseen environments and novel target objects.
Takeaways & Limitations
The current formulation lacks long-term memory and does not yet incorporate higher-order relationships between objects and scenes.
Abstract
from arXiv · showhide
How do humans navigate to target objects in novel scenes? Do we use the semantic/functional priors we have built over years to efficiently search and navigate? For example, to search for mugs, we search cabinets near the coffee machine and for fruits we try the fridge. In this work, we focus on incorporating semantic priors in the task of semantic navigation. We propose to use Graph Convolutional Networks for incorporating the prior knowledge into a deep reinforcement learning framework. The agent uses the features from the knowledge graph to predict the actions. For evaluation, we use the AI2-THOR framework. Our experiments show how semantic knowledge improves performance significantly. More importantly, we show improvement in generalization to unseen scenes and/or objects. The supplementary video can be accessed at the following link: https://youtu.be/otKjuO805dE .
1 INTRODUCTION
The paper argues that semantic and functional priors help agents plan searches for visible, unseen, and novel targets in unfamiliar scenes. It incorporates these priors into deep reinforcement learning and reports improved navigation and generalization.
- Motivation: Humans use functional relationships, such as mugs near coffee machines, to infer likely object locations in unfamiliar scenes.Prior experience with many kitchens supports these semantic and functional expectations.
- Motivation: Semantic priors also support navigation to unseen objects by transferring knowledge from familiar categories, such as using fruit locations to search for mangoes.The example predicts searching the fridge for a mango based on prior experience with apples and grapes.
- Approach: The proposed model uses Graph Convolutional Networks to encode prior knowledge within a deep reinforcement learning framework.The agent updates knowledge using current observations, previous knowledge, and prior information from large-scale scene-understanding datasets.
- Evaluation: Evaluation in AI2-THOR uses customizable near-photo-realistic environments with varied object configurations and randomized agent locations.The supplied passage introduces the evaluation setting but truncates the reported experimental findings.
- Contributions: The contributions include integrating reinforcement learning with knowledge graphs and improving navigation performance and generalization to unseen environments and novel target objects.These claims are summarized explicitly as the paper’s main contributions.
2 RELATED WORK
Prior navigation research includes geometry-based mapping, learning-based control, contextual reasoning, knowledge graphs, and zero-shot reasoning. This paper positions its contribution at their intersection by applying semantic context and graph knowledge to interactive navigation.
- Geometry-based navigation: Geometry-based navigation either assumes a known environment map or builds one during navigation.The paper distinguishes its approach by not relying on a map.
- Learning-based navigation: Learning-based navigation methods use deep learning or reinforcement learning for tasks including target-driven navigation, mapping, planning, auxiliary prediction, collision avoidance, and landmark-based memory.The cited methods span indoor, street, and game environments.
- Context and scene prior: Contextual reasoning has mainly been applied to static computer vision tasks, whereas this work applies it to interactive navigation with belief updates from observations.The paper frames interactive navigation as a distinct use of contextual information.
- Knowledge graphs: Knowledge graphs have been used for image classification, situation recognition, and zero-shot recognition, while this work uses them for reinforcement-learning-based visual navigation.The distinction concerns the interactive navigation setting.
- Reasoning about unknown environments or objects: Related zero-shot reinforcement learning studies address unseen language combinations, changed target-domain dynamics or states, and zero-shot imitation learning.These works provide context for reasoning about unknown environments or objects.
3 VISUAL SEMANTIC NAVIGATION
Visual semantic navigation asks an agent to reach a specified object category from a random location using egocentric RGB observations. The paper formulates this with actor-critic reinforcement learning and augments the policy with semantic and scene-prior information.
- 3.1 TASK DEFINITION: The task starts from a random scene location and succeeds when an instance of the semantic target category becomes visible within the required field of view and distance.The target is specified by a semantic label rather than an exact object appearance.
- 3.1 TASK DEFINITION: The agent observes an egocentric RGB state and samples actions from a policy conditioned on the semantic task objective.The policy maps visual and semantic inputs into action decisions.
- Architecture: The architecture combines current visual features, the semantic target-category feature, and features from activated portions of a knowledge graph.The knowledge graph supplies scene-prior information to the policy network.
- Architecture: Separate visual and semantic branches map their inputs into a joint visual-semantic feature embedding.The visual and semantic modalities are processed separately before fusion.
- Feature representations: The visual branch uses ResNet-50 features from the current and three previous observations, while the semantic branch embeds the target category with fastText.The supplied implementation passages specify the respective feature-processing pipelines.
- Actor-Critic policy network: The A3C actor-critic network concatenates visual and semantic representations, then outputs both a policy and a value for action selection.The fused input is a 1024-d feature vector, and the action is sampled from the predicted policy.
- Actor-Critic policy network: A single policy network is used across different scene examples, making the implementation more compact and generalizable.This contrasts with approaches using different policy networks for different scenes.
- Reward: The reward gives 10.0 for reaching a target within the step limit and penalizes each step by -0.01 when the target is not reached.With a stop action, success additionally requires being within 1 meter and facing the target.
4 GENERALIZATION WITH GRAPH CONVOLUTIONAL NETWORKS
The paper represents semantic priors as a knowledge graph and uses GCNs to extract scene-conditioned relational features for navigation. These features are passed to the policy model to incorporate current visual and semantic context.
- Graph Convolutional Network: GCNs dynamically update graph-based semantic knowledge as the agent receives information specific to the current environment.The GCN formulation propagates node features through the normalized graph adjacency structure across layers.
- Knowledge Graph Construction: The knowledge graph encodes spatial relationships between object categories and links known objects to novel categories without visual examples.
- Knowledge Graph Construction: The graph contains object-category nodes and relationship edges, built from Visual Genome relationships among categories appearing in AI2-THOR.Edges are retained when a relationship occurrence exceeds three.
- Incorporating Semantic Knowledge: Three GCN layers propagate information from scene-initialized nodes, producing a |V| dimensional semantic knowledge vector for the policy model.The first two layers output 1024-dimensional latent features, and the final layer outputs one value per node.
- Incorporating Semantic Knowledge: Each graph node combines semantic cues from word embeddings with visual cues from current-state image classification scores.Word embeddings come from fastText, while classification scores come from an ImageNet-pretrained ResNet-50.
- Incorporating Semantic Knowledge: The GCN output is mapped to a 512-dimensional embedding and concatenated with 1024-dimensional visual-semantic features before policy prediction.The resulting joint representation is 1536-dimensional.
5 EXPERIMENTS
The experiments evaluate knowledge-graph navigation in AI2-THOR across scene and object generalization settings, using Success Rate and SPL. Results show improved performance over baselines, with proper graph structure contributing to performance.
- Evaluation Framework: The evaluation uses AI2-THOR scenes across kitchens, living rooms, bedrooms, and bathrooms, with randomized training, validation, and testing room splits.Each room category contains 20 training, 5 validation, and 5 testing rooms.
- Evaluation Framework: Models are evaluated using Success Rate and SPL, which jointly reflect reaching the target and path efficiency.SPL incorporates success and the path length relative to the shortest path distance.
- Generalization Settings: The experiments test seen scenes with novel objects, unseen scenes with known objects, and unseen scenes with novel objects.The study also reports a seen-scene, known-object setting and evaluates 250 randomized episodes per scene.
- Results: The knowledge-graph model outperforms the baselines in both success rate and SPL, although performance drops sharply when scenes and target objects are both novel.The authors also report overfitting to known scenes and objects, and caution that settings are not directly comparable because scene and object configurations differ.
- Knowledge-Graph Ablation: SPL falls as object nodes or relations are removed, while fully connected and random graphs reach 32.5 and 30.1 ± 0.6 SPL versus 38.5 for the proper graph.The ablation is performed on kitchen scenes without the stop action, with the random-graph result repeated three times.
- Computation Cost: The GCN module adds only 0.12 GFLOPs over the approximately 4 GFLOPs A3C baseline.The paper characterizes this increase as marginal.
6 CONCLUSIONS
The paper integrates semantic and functional priors into deep reinforcement learning through Graph Convolutional Networks for navigation. It reports improved generalization to unseen scenes and targets, while identifying long-term memory and higher-order relationships as future directions.
- Graph Convolutional Networks encode semantic and functional priors and update them using observations from the current scene.The encoded prior knowledge informs the deep reinforcement learning navigation model.
- Prior knowledge improves generalization to unseen scenes and targets.
- The current formulation lacks long-term memory for learning more complex exploration strategies.
- Incorporating higher-order relationships between objects and scenes is identified as another future direction.
APPENDIX A NAVIGATION TARGETS
The appendix documents the navigation-target object categories and their training and testing splits for each AI2-THOR scene type.
- Table 4 lists the object categories used as navigation targets.
- The target categories are divided into training and testing sets.
- The table organizes training and testing splits by scene type in AI2-THOR.
APPENDIX B TRAINING CURVES
Training curves compare the proposed method with the A3C baseline using Success Rate and SPL. The proposed model converges faster and achieves better performance in both metrics.
- The proposed model converges faster than the A3C baseline.All models were trained three times with different initializations, and error bands represent standard deviation.
- The proposed model achieves better performance than A3C in both Success Rate and SPL.Performance is computed every 10 million iterations during training.
- Figure 5 reports Success Rate in the top row and SPL in the bottom row.
APPENDIX C IMPLEMENTATION DETAILS
The method is implemented in TensorFlow with an actor-critic policy network trained using a single NVIDIA GeForce GTX Titan X GPU and RMSProp optimization.
- The method is implemented in TensorFlow with an actor-critic policy network.
- Training uses a single NVIDIA GeForce GTX Titan X GPU with 20 threads.
- The initial learning rate is 7e −4 and decreases linearly during training.
- The network parameters are optimized with RMSProp.
APPENDIX D QUALITATIVE RESULTS
The appendix presents qualitative examples from the model on unseen scenes with novel target objects, showing the last eight frames alongside predicted actions.
- Qualitative examples show the model operating in unseen scenes with novel target objects.
- Each example includes the last eight frames generated before the reported action sequence.
- The examples pair visual observations with the corresponding actions predicted by the model.