Source-linked AI summary
Visual Representations for Semantic Target Driven Navigation
Arsalan Mousavian, Alexander Toshev, Marek Fiser, Jana Kosecka, Ayzaan Wahid, James Davidson
TL;DR
The paper asks which visual representations best support target-driven semantic navigation in unexplored environments without relying on metric semantic maps and path planning. It learns navigation policies from semantic and contextual representations, using segmentation and detection masks, and reports successful target navigation in 54% of cases while outperforming a non-learning baseline by 8%.
Problem
The paper studies how visual observations should be represented for target-driven semantic navigation in unexplored environments, where traditional approaches typically require 3D mapping and reliable localization.
Method
The approach learns a deep navigation policy from segmentation and detection masks, using optimal-path supervision to predict action progress toward the semantic goal.
Results
54% of cases reached the target, outperforming the non-learning-based map-and-detector baseline by 8%.
Takeaways & Limitations
Semantic visual representations support generalization to previously unseen environments and allow navigation policies to use synthetic and real images jointly without domain adaptation.
Abstract
from arXiv · showhide
What is a good visual representation for autonomous agents? We address this question in the context of semantic visual navigation, which is the problem of a robot finding its way through a complex environment to a target object, e.g. go to the refrigerator. Instead of acquiring a metric semantic map of an environment and using planning for navigation, our approach learns navigation policies on top of representations that capture spatial layout and semantic contextual cues. We propose to using high level semantic and contextual features including segmentation and detection masks obtained by off-the-shelf state-of-the-art vision as observations and use deep network to learn the navigation policy. This choice allows using additional data, from orthogonal sources, to better train different parts of the model the representation extraction is trained on large standard vision datasets while the navigation component leverages large synthetic environments for training. This combination of real and synthetic is possible because equitable feature representations are available in both (e.g., segmentation and detection masks), which alleviates the need for domain adaptation. Both the representation and the navigation policy can be readily applied to real non-synthetic environments as demonstrated on the Active Vision Dataset [1]. Our approach gets successfully to the target in 54% of the cases in unexplored environments, compared to 46% for non-learning based approach, and 28% for the learning-based baseline.
I. INTRODUCTION
The paper asks which visual representations best support target-driven navigation in unexplored environments. It proposes semantic and contextual representations with learned navigation policies, combining real and simulated training data.
- Problem: Target-driven semantic visual navigation asks an agent to reach a class-specified object in an unexplored environment.The motivating example is navigating to a refrigerator.
- Motivation: Traditional navigation commonly builds 3D metric and semantic maps before planning and control, requiring advance mapping and reliable localization.These approaches typically underuse general semantics and contextual cues during decision making.
- Approach: The proposed policy uses segmentation and detection masks as high-level semantic and contextual observations.The work also investigates combinations with raw RGB and depth inputs.
- Training: Optimal path planning supplies stronger supervision by estimating progress toward the goal, which the navigation model predicts to select actions.The paper investigates both memory-free and internally stateful LSTM models.
- Result: 54% of previously unseen environments reach the target successfully using the proposed model.The policy is trained and evaluated with both synthetic and real environments, including a detailed ablation study.
II. RELATED WORK
Prior visual-navigation work spans map-based planning, direct learned policies, reinforcement learning, imitation learning, and simulation-based embodied-perception research. These approaches differ in representations, supervision, architecture, generalization, and environmental realism.
- Map-based navigation: Classical navigation constructs maps through visual localization and mapping, then applies planning and control to execute trajectories.The environment model supports state-of-the-art planning and control strategies.
- Learned navigation: Learning-based methods bypass explicit mapping and pose estimation by learning navigation strategies directly from observations.They vary in architecture, complexity, training cost, generalization, and supervision.
- Reinforcement learning: Reinforcement-learning navigation has been evaluated in synthetic 3D mazes, with auxiliary losses proposed to address sparse rewards.The cited maze observations do not exhibit the complexity of real-world settings.
- Synthetic control: Synthetic data with domain randomization has been used to learn control strategies for object reaching and quadcopter flight.These related tasks use simulation to support control learning.
- Alternative learning methods: Other work applies value iteration networks, imitation learning, or a single traversal to learn navigation strategies.These methods represent distinct supervision and learning setups.
- Embodied perception: Active and embodied perception research has motivated simulation environments derived from SUNCG and Matterport3D for navigation, task planning, and visual question answering.Most visual-question-answering evaluation metrics do not explicitly account for navigation.
III. NAVIGATION MODEL
The navigation model treats target reaching from visual observations as a partially observable decision problem and learns policies over representations extracted from those observations. The paper emphasizes semantic masks because they encode scene structure while supporting real–simulation training without domain adaptation.
- A. Setup: The task is formulated as a POMDP in which the agent’s pose is unobservable and actions are discrete turns and translations.The reward is defined as the negative shortest-path distance from the current state to the target.
- A. Setup: The policy maps an observation representation and target class to actions that move the agent toward the target.The target is specified by its class label rather than a coordinate.
- B. Visual Representations: Detection masks encode object categories from Faster R-CNN, while semantic-segmentation masks encode NYU V2 scene categories.Both representations are spatial H × W mask stacks with category-specific channels.
- B. Visual Representations: Depth is represented by normalized sensor depth plus a validity mask, whereas RGB uses a pretrained ResNet-50 embedding.These modalities provide alternatives for comparison with semantic representations.
- B. Visual Representations: RGB produces different outputs across real and simulated domains, making it difficult to use directly across both.Domain-adaptation methods are described as difficult to train because they rely on relatively unstable generative-adversarial setups.
- B. Visual Representations: Detection and segmentation masks capture layouts, obstacles, and target locations while allowing large pretrained perception networks to remain outside controller training.The representations therefore retain navigation-relevant information while making controller optimization easier and more stable.
- B. Visual Representations: In simulation, object masks and bounding boxes come from renderer labels, eliminating the need for domain adaptation when combining synthetic and real data.The remaining gap is related to the quality of the real-data segmentation and detection.
C. Model
The model processes the current visual representation together with a one-hot target description, then combines modality embeddings and recurrent processing to produce navigation behavior.
- C. Model: The deep policy receives the current observation representation and a one-hot target-class vector that modulates network behavior.The target is supplied as a secondary input alongside the visual observation.
- C. Model: Each feature extractor uses convolutional layers and a fully connected layer to produce a 128-dimensional modality embedding, except RGB’s pretrained ResNet-50.The target embedding is also 128-dimensional before concatenation.
- C. Model: The concatenated modality and target embeddings pass through fully connected layers and an LSTM.This architecture combines multiple representations with internal state.
D. Training
Training uses shortest-path supervision to learn action costs from observations, then selects the action with the lowest predicted cost at test time. An LSTM implicitly models observation history.
- D. Training: Shortest-path planning provides stronger supervision by estimating each action’s progress toward the goal.Progress is defined as the reduction in shortest distance after taking an action.
- D. Training: The agent predicts action costs rather than imitating a single optimal action, allowing non-shortest actions that still lead toward the goal.This addresses states where multiple actions can follow successful paths.
- D. Training: Collision actions receive a cost of +2, goal-reaching actions receive −2, and stopping elsewhere receives +2.Other action costs lie in the range [−1, 1].
- D. Training: A neural network predicts state-action costs directly from observations, and the controller chooses the action with the lowest predicted cost.The policy therefore converts visual observations and the target class into test-time actions.
- D. Training: Although observation history is not explicit in the formulation, the model represents it implicitly through an LSTM component.Training uses real and simulated environments with discrete states and actions.
IV. EXPERIMENTS
Experiments evaluate navigation on real homes using two AVD generalization splits and supplement training with synthetic SunCG homes while retaining AVD for evaluation.
- IV. EXPERIMENTS: AVD contains dense scans of nine homes and supports similar-environment and different-environment train/test splits.The similar split uses different scans of the same home, whereas the different split uses different homes for training and testing.
- IV. EXPERIMENTS: The different-environment split tests generalization across homes, while the similar-environment split tests across scans of the same home.Objects and scanning locations can differ between scans in the similar split.
- IV. EXPERIMENTS: SunCG supplies an additional set of synthetically generated homes, with evaluations performed only on AVD real data.This setup measures how synthetic training data affects performance on real environments.
- IV. EXPERIMENTS: The task uses seven discrete movement or stopping actions and five semantic goal categories.The goals are dining table, refrigerator, television, couch, and microwave.
A. Training Details
Training combines random trajectories from real or synthetic environments, while evaluation measures whether agents reach target objects within a fixed interaction budget.
- A. Training Details: Each minibatch contains losses from 8 random trajectories sampled from AVD or SunCG.Trajectories use random environments, goals, and targets, then retain the first 20 locations.
- A. Training Details: The model uses 64 × 64 inputs for non-raw features and 299 × 299 inputs for raw images matching ResNet50’s training resolution.The optimizer is Adam with a learning rate of 10−4.
- A. Training Details: Evaluation runs agents for up to 100 steps, with the stop action terminating an episode early.Initial locations are randomly selected and fixed across experiments.
- A. Training Details: Success rate is the proportion of runs reaching the goal, defined as coming within 5 steps of any target-object instance.The distance threshold avoids degenerate observations when the object fills the field of view.
RGB vs Semantic Representation:
Semantic representations generalize better to different homes than RGB embeddings, while ResNet-based embeddings can overfit scans from homes seen during training. Adding depth appears to hurt semantic generalization.
- RGB vs Semantic Representation:: Semantic representations achieve higher performance on the different-environment split, where testing uses totally different homes.This pattern holds for both SSeg and Det representations.
- RGB vs Semantic Representation:: Figure 3 compares model success rates between similar and different AVD environment splits.Blue denotes similar environments and red denotes different environments.
- RGB vs Semantic Representation:: Adding depth appears to hurt the generalization of the proposed semantic representations.The passage reports this trend for the different-environment setting.
- RGB vs Semantic Representation:: ResNet-based image embeddings allow overfitting on scans from the same homes.They may therefore be more appropriate when deployment environments have been seen during training.
Use of Synthetic Environments:
Synthetic training data improves semantic representations on real AVD evaluation, while raw depth suffers from the reality gap and recurrence remains important.
- Synthetic-data augmentation: Synthetic data improves success for Det, SSeg, and their combination when evaluated on AVD.Training episodes are sampled from AVD or SunCG, while evaluation uses AVD only.
- Synthetic-data augmentation: Over 10% performance increases occur in some semantic-representation settings after adding simulated data.
- Reality gap: Raw Depth performance drops with synthetic-data augmentation because simulated depth is cleaner than noisy, incomplete Kinect depth on AVD.Depth values are clipped, perturbed, and randomly removed to mimic Kinect imperfections.
- Reactive vs recurrent model: Removing recurrence causes a significant success-rate drop, with state-less policies repeating wrong actions and oscillating despite additional training data.
- Object-class ablation: The object-class ablation removes each of six COCO superclasses to assess their contribution to Det-based navigation.
Importance of Different Object Classes:
Furniture objects are especially valuable for navigation because they are visible from far away and correlate with an agent’s location, whereas Food contributes little.
- Object-class importance: Furniture removal produces the largest success-rate drop among the evaluated COCO superclass subsets.
- Object-class importance: Furniture helps navigation because couches, tables, and beds are detectable from far away and correlate strongly with the agent’s location.
- Object-class importance: Food has little importance, likely because food objects are small and lack permanent locations.
- Comparison with baseline: The non-learning baseline receives the full environment graph and global pose, unlike the proposed method.
- Comparison with baseline: When the target is detected, the baseline back-projects it using pose, depth, and camera intrinsics, then selects a directed graph view and shortest path.
- Comparison with baseline: If the target is not visible, the baseline acts randomly, while graph planning avoids obstacles even in unexplored regions after detection.
- Learned trajectories: Learned paths can cross a home successfully, staying straight in narrow passages while sometimes turning around to explore.
V. CONCLUSION
The paper demonstrates semantic visual representations and learned policies for target-driven navigation, combining real and synthetic training without domain adaptation. The approach reaches targets in 54% of cases and outperforms the map-based baseline by 8%.
- Conclusion: Semantic segmentation and detection representations serve as observations for a deep navigation policy in target-driven visual navigation.
- Conclusion: The approach jointly trains navigation policies with synthetic CG data and real images without requiring domain adaptation.
- Conclusion: Ablation studies show that semantic segmentation and detectors improve generalization to previously unseen environments.
- Conclusion: Adding an RNN component has its most dominant effect when synthetic data is included in training.
- Conclusion: 54% of cases reach the goal, outperforming the non-learning map-and-detector baseline by 8%.
- Conclusion: The generalization improvement from simulated examples does not hold for depth used with both real and synthetic data.