Source-linked AI summary
Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J. Lim, Abhinav Gupta, Li Fei-Fei, Ali Farhadi
TL;DR
The paper addresses limited target generalization and data inefficiency in deep reinforcement learning for target-driven visual navigation. It introduces a target-conditioned actor-critic model and the AI2-THOR interactive simulation framework, reporting faster convergence, generalization across targets and scenes, and adaptation to a real robot with limited fine-tuning.
Problem
Deep reinforcement learning often requires costly retraining for new target goals and many trial-and-error episodes to converge.
Method
The paper combines a policy conditioned on the current state and visual target with AI2-THOR, a high-quality 3D physics-based environment for interactive data collection.
Results
The method generalizes to new targets and scenes, converges with fewer training samples than state-of-the-art DRL methods, and adapts from simulation to a real robot with small fine-tuning.
Takeaways & Limitations
Target-conditioned learning and interactive simulation support end-to-end visual navigation across targets, scenes, and discrete or continuous domains.
Abstract
from arXiv · showhide
Two less addressed issues of deep reinforcement learning are (1) lack of generalization capability to new target goals, and (2) data inefficiency i.e., the model requires several (and often costly) episodes of trial and error to converge, which makes it impractical to be applied to real-world scenarios. In this paper, we address these two issues and apply our model to the task of target-driven visual navigation. To address the first issue, we propose an actor-critic model whose policy is a function of the goal as well as the current state, which allows to better generalize. To address the second issue, we propose AI2-THOR framework, which provides an environment with high-quality 3D scenes and physics engine. Our framework enables agents to take actions and interact with objects. Hence, we can collect a huge number of training samples efficiently. We show that our proposed method (1) converges faster than the state-of-the-art deep reinforcement learning methods, (2) generalizes across targets and across scenes, (3) generalizes to a real robot scenario with a small amount of fine-tuning (although the model is trained in simulation), (4) is end-to-end trainable and does not need feature engineering, feature matching between frames or 3D reconstruction of the environment. The supplementary video can be accessed at the following link: https://youtu.be/SmBxMDiOrvs.
I. INTRODUCTION
The paper targets visual navigation with deep reinforcement learning while addressing poor target generalization and inefficient, costly training. It combines a target-conditioned policy with AI2-THOR simulation to support broader evaluation and data collection.
- Motivation: General DRL policies depend only on the current state, so changing the target requires learning new model parameters.Training DRL agents is computationally expensive.
- Target-driven model: The target-driven model conditions its policy jointly on the current state and visual target, avoiding retraining for each new target.Shared routes and scene regularities allow training across targets to provide reusable information.
- Simulation framework: AI2-THOR provides high-quality 3D scenes and physics-based interactions for efficiently collecting visual observations and action-reaction data.Agents can navigate indoor scenes and interact with objects at low and high levels.
- Evaluation: The evaluation covers target, scene, and real-world generalization, and reports improved data efficiency over state-of-the-art DRL methods.The real-robot demonstration uses a model trained in simulation.
- Main findings: The method generalizes across targets and scenes and reaches real-robot navigation with a small amount of fine-tuning.The framework supports visual interactions for agents through high-quality rendering.
II. RELATED WORK
Related work spans map-based, map-reconstructing, map-less, and deep reinforcement learning approaches to visual navigation. The paper distinguishes its method by directly conditioning on target images, avoiding explicit maps and feature-matching or reconstruction pipelines.
- Visual navigation: Map-based navigation methods use a global environment map, whereas this method is map-less and does not require a prior map.The paper also contrasts with methods that reconstruct maps online or build them through human-guided training.
- Visual navigation: The approach does not require feature matching, 3D reconstruction, or supervised training to recognize distinctive landmarks.These requirements are identified as differences from cited visual navigation methods.
- Reinforcement learning: Prior reinforcement learning work covers locomotion, motor primitives, obstacle detection, flight, mapping, large-scale learning, and Atari decision-making.The paper positions its task as visual navigation with more complex inputs than Atari games or constrained laboratory images.
- Deep reinforcement learning: Deep reinforcement learning methods include deep Q networks, asynchronous policy updates, and combinations of reinforcement learning with Monte Carlo tree search.The paper contrasts these applications with its target-driven navigation setting.
- Simulation frameworks: The related simulated-learning frameworks shown include ALE, ViZDoom, UETorch, Project Malmo, SceneNet, TORCS, SYNTHIA, and Virtual KITTI.Figure 2 presents screenshots for qualitative comparison.
- Target generalization: Other learning methods address generalization across target tasks, while this model takes the target goal directly as input without retraining.The paper also reports simulation-to-real-world generalization.
III. AI2-THOR FRAMEWORK
AI2-THOR integrates a physics engine with deep learning for interactive 3D-scene training and evaluation. Its direct communication and realistic image appearance are designed to support scalable learning and real-world generalization.
- Framework requirements: The framework must support actions, perceptual outcomes, interchangeable scene types, and detailed physics for movements and object interactions.A plug-and-play architecture is required so different scenes can be incorporated easily.
- Architecture: AI2-THOR integrates the Unity 3D physics engine with TensorFlow to connect rendered observations and agent control commands.The physics engine streams images to the learning framework, which returns commands to the simulated agent.
- Architecture: Direct physics-engine-to-learning-framework communication enables immediate environmental feedback for online decision-making.This is presented as an advantage over architectures that separate the physics engine from the controller.
- Realism: The framework aims to mimic real-world image appearance, including lighting, object appearance, textures, and background clutter.This design is described as important for generalizing to real-world images.
- Scene construction: The framework contains 32 household scenes across kitchens, living rooms, bedrooms, and bathrooms, averaging 68 object instances per scene.Scenes are created with textures and lighting modeled after reference images.
- Benefits and limitations: AI2-THOR offers scalable, cheaper, and safer training than operating robots in real houses, but synthetic scenes under-model real-world detail.The framework can also support fine-grained interactions such as force application, grasping, and opening or closing objects.
IV. TARGET-DRIVEN NAVIGATION MODEL
The paper formulates target-driven visual navigation as learning actions from current and target RGB images, using a deep reinforcement learning model that conditions policy on both. Its siamese actor-critic architecture supports shared representations and avoids retraining for new targets.
- Target-driven visual navigation seeks the minimum-length action sequence from the current location to a target specified by an RGB image.
- The model maps current and target images to 3D actions such as moving forward or turning right.
- Unlike goal-specific DRL models, the policy takes both the current state and target as inputs, avoiding retraining for each new target.
- The deep siamese actor-critic network represents current observations and targets in a shared embedding space to reason about their spatial arrangement.
C. Learning Setup
The learning setup uses command-level navigation actions and first-person RGB images for both observations and goals. A target image specifies the location and viewpoint the agent should reach.
- Action space: The agent uses four command-level actions: moving forward, moving backward, turning left, and turning right.
- Observations and goals: Both observations and goals are first-person RGB camera images.
- Observations and goals: Given a target image, the agent navigates to the location and viewpoint where that image was captured.
1) Action space:
The model is trained to minimize navigation trajectory length through goal-reaching and time-penalty rewards. Its policy approximator conditions actions on visual state and target representations, which are projected into a shared embedding space.
- Reward design: The reward is 10.0 upon task completion and -0.01 as an immediate time penalty to encourage shorter trajectories.
- Target-driven policy: The policy π maps the current observation image and target image to an action distribution using a deep neural network.
- Target-driven policy: Goal embeddings support knowledge transfer across targets, enabling generalization beyond a finite set of discrete goals.
- Network architecture: The siamese actor-critic network projects current and target images into the same embedding space to capture their relative spatial arrangement.
E. Training Protocol
Training uses a multi-target protocol based on asynchronous actor-critic learning, with separate navigation targets assigned to threads. Shared siamese layers receive gradients from all targets, while scene-specific layers receive within-scene updates.
- Training protocol: The training protocol uses parallel threads with different navigation targets, following the asynchronous update approach of A3C.
- Training protocol: The network uses shared parameters across tasks, allowing learning from multiple goals simultaneously rather than training each task separately.
- Training protocol: Gradients from navigation tasks update scene-specific layers within a scene and generic siamese layers across all targets.
- Experiments: The experiments evaluate shortest-trajectory navigation alongside transfer across new targets, new scenes, continuous space, and a real robot.
A. Navigation Results
The target-driven model improves data-efficient navigation after 100M frames and learns observation embeddings that preserve spatial layout.
- After 100M training frames, the target-driven model learns better navigation policies than state-of-the-art A3C methods.
- The final model’s average trajectory length is three times shorter than the single-branch model’s.The authors attribute this to scene-specific layers capturing characteristics that vary across scene instances.
- A3C learns more efficiently with four actor-learning threads per target than with one, while Q-learning converges slowly.
- t-SNE projections show observation embeddings whose relative spatial arrangement corresponds to the observations’ locations in a living room scene.
B. Generalization Across Targets
The model is evaluated on navigation to unseen targets and scenes, with transfer across scenes improving adaptation efficiency and requiring scene-specific adjustment.
- Generalization scope: The model generalizes to new targets within a scene and to new scenes using shared generic siamese layers.
- Across targets: Target-generalization tests vary the number of trained targets and the distance between trained and new targets, measuring success over 100 episodes per target.
- Across scenes: Transferring generic layers significantly improves data efficiency for learning in new environments compared with training from scratch.
- Across scenes: Applying a single-branch model trained on 16 scenes to new scenes without extra training performs worse than chance, highlighting the importance of adapting scene-specific layers.
D. Continuous Space
The same target-driven model is tested in continuous-space physics-based navigation and on a real robot, where simulation transfer accelerates convergence with fine-tuning.
- Continuous space: In a physics-controlled door-finding task, the model reaches the door in 15 average steps versus 719 for random agents.Training requires around 50M frames for a single target, and the setting includes forces, collisions, and sliding along heavy objects.
- Continuous space: The continuous-space experiment uses the same four actions while the physics engine controls the agent’s moves and turns.
- Real-world generalization: On a SCITOS robot, transferring and fine-tuning parameters from simulation converges 44% faster than training from scratch.The robot is evaluated on door and microwave targets after training at 28 discrete locations.
- Real-world generalization: The robot model remains robust to random starting points, noisy dynamics, varying step lengths, illumination changes, and object-layout changes.
VI. CONCLUSIONS
The paper presents a DRL framework for target-driven visual navigation aimed at more realistic settings. It addresses generalization and data-efficiency limitations while demonstrating simulation-to-real adaptation and end-to-end operation without explicit reconstruction.
- The framework targets more realistic visual-navigation settings than typical DRL environments.
- It addresses generalization across scenes and targets, improves data efficiency, and provides AI2-THOR for inexpensive interaction-data collection.
- The method generalizes to new targets and scenes excluded from end-to-end training and converges with fewer training samples than state-of-the-art DRL methods.
- A simulation-trained model adapts to a real robot with small fine-tuning and operates end to end without explicit feature matching or 3D reconstruction.
- Future work includes expanding AI2-THOR with more high-quality 3D scenes and adding physical-interaction and object-manipulation learning.