Source-linked AI summary
Towards Monocular Vision based Obstacle Avoidance through Deep Reinforcement Learning
Linhai Xie, Sen Wang, Andrew Markham, Niki Trigoni
TL;DR
Obstacle avoidance with monocular vision is difficult because RGB images lack direct 3D information, while conventional planners require parameter tuning. The paper proposes D3QN, combining monocular depth prediction with dueling and double-Q reinforcement learning. D3QN trains about twice as fast as normal DQN and transfers simulator-trained models directly to real robots, including environments with unseen dynamic objects.
Problem
Monocular obstacle avoidance is challenging because RGB imagery lacks 3D information, while conventional path planners require parameter tuning.
Method
D3QN uses a two-phase network that predicts depth from monocular RGB images and applies dueling and double-Q deep reinforcement learning for obstacle avoidance.
Results
D3QN trains about two times faster than normal DQN and simulator-trained models transfer directly to real-world obstacle-avoidance tasks.
Takeaways & Limitations
The trained network can avoid obstacles with monocular vision in new real-world scenarios, including environments containing previously unseen dynamic objects.
Abstract
from arXiv · showhide
Obstacle avoidance is a fundamental requirement for autonomous robots which operate in, and interact with, the real world. When perception is limited to monocular vision avoiding collision becomes significantly more challenging due to the lack of 3D information. Conventional path planners for obstacle avoidance require tuning a number of parameters and do not have the ability to directly benefit from large datasets and continuous use. In this paper, a dueling architecture based deep double-Q network (D3QN) is proposed for obstacle avoidance, using only monocular RGB vision. Based on the dueling and double-Q mechanisms, D3QN can efficiently learn how to avoid obstacles in a simulator even with very noisy depth information predicted from RGB image. Extensive experiments show that D3QN enables twofold acceleration on learning compared with a normal deep Q network and the models trained solely in virtual environments can be directly transferred to real robots, generalizing well to various new environments with previously unseen dynamic objects.
I. INTRODUCTION
Monocular vision offers lightweight, information-rich perception for obstacle avoidance, but limited range information makes collision avoidance difficult. The paper proposes a two-phase D3QN approach that combines monocular depth prediction with deep reinforcement learning and transfers simulation-learned policies to real robots.
- Monocular cameras are low-cost, lightweight, and broadly applicable, but their lack of direct 3D range information makes obstacle avoidance difficult.
- Conventional approaches infer obstacles or traversable spaces from visual information before applying path planners, which require manual parameter tuning and adapt poorly to new operating areas.
- Supervised and self-supervised end-to-end methods avoid complex planner modeling, but self-supervised policies remain bounded by their label-generating strategy.
- Deep reinforcement learning can explore policies without human supervision, but its weaker learning signal requires much more training data and complicates direct real-world use.
- The paper proposes a two-phase neural network using a fully convolutional depth predictor followed by a DQN with dueling architecture for parallel angular and linear action values.
- The contributions include faster end-to-end learning with limited computational resources, transfer of simulation-learned knowledge to new real-world scenarios, and extensive real-world experiments.
II. DEEP Q NETWORK FOR MONOCULAR VISION BASED OBSTACLE AVOIDANCE
Normal DQN can eventually perform obstacle avoidance from raw images, but it tends to overestimate Q-values and requires lengthy, computationally intensive training. The paper therefore introduces D3QN to improve performance and training efficiency.
- Normal DQN can learn obstacle-avoidance policies directly from raw images but tends to overestimate Q-values and train slowly.
- D3QN is introduced as an advanced architecture intended to improve both obstacle-avoidance performance and training efficiency.
A. Problem Definition
The monocular obstacle-avoidance task is formulated as sequential decision-making: the robot maps camera observations to actions, receives rewards, and transitions between observations. The objective is to maximize discounted future reward, approximated with deep neural networks.
- The robot selects a discretized action from a camera observation, receives a reward, and transitions to the next observation.
- The objective is to maximize cumulative future reward using a discount factor over the decision horizon.
- The Q-value represents the expected return for a state-action pair under a policy.
- The Bellman formulation expresses optimal value as the current reward plus the discounted optimal value available at the next time step.
- Deep Q-networks approximate the optimal Q-value function with neural networks instead of computing it directly across a large state space.
B. Dueling Network and Double Q-Network
D3QN combines dueling and double-Q mechanisms to estimate action values more effectively and improve learning efficiency for monocular obstacle avoidance.
- Dueling architecture: The dueling architecture separates state-value and action-advantage estimation before combining them into action Q-values.Two fully connected streams compute the value and advantage functions separately after convolutional layers.
- Training procedure: Training uses batches containing the current state, action, reward, and resulting state in the D3QN procedure.Figure 2 identifies element-wise addition, subtraction, and multiplication operations in the training procedure.
- Double-Q mechanism: The double-Q procedure uses the online network to select actions and the target network to evaluate them, reducing overoptimistic value estimation.The target value combines the current reward with the discounted value at the resulting state, and the error updates the online network.
- Combined design: D3QN exploits both techniques to improve data efficiency and speed learning.The paper presents the combined architecture as a way to accelerate training for obstacle avoidance.
C. From Appearance to Geometry
The approach transfers monocular RGB observations into a geometric representation by predicting depth, then trains with noisy depth to address simulation-to-reality differences.
- Transfer challenge: Simulation-to-reality transfer is challenging for vision-based techniques because virtual and real environments differ in appearance and illumination.The paper motivates deriving a geometric representation from RGB imagery to address this challenge.
- Depth prediction: The first D3QN stage predicts depth information from a single RGB image using a fully convolutional residual network.The predicted depth is intended as the geometric representation used by the obstacle-avoidance system.
- Depth prediction: Predicted depth is substantially noisier than Kinect depth because it is estimated by a neural network rather than measured by a 3D sensor.Figure 3 compares RGB, predicted depth, and Kinect depth, highlighting the predicted depth noise.
- Simulation-to-reality adaptation: Training depth images are corrupted with random noise and image blur to improve transfer and real-world generalization.The authors report that this corruption is critical for transferring models from simulation to reality.
D. Model and Training Settings
The model uses a convolutional dueling architecture with discretized velocity actions and a reward function that favors fast forward motion while penalizing collisions and stationary rotation.
- Model architecture: D3QN contains three convolutional layers and three fully connected layers organized into two dueling streams.The architecture is built from the dueling and double techniques.
- Action space: Actions independently control linear and angular velocities in discretized form rather than using simple commands such as turning left or right.This action representation produces separate velocity controls for the robot.
- Reward design: The instantaneous reward is r = v ∗ cos(ω)∗δt, with δt set to 0.2 second.Here, v and ω are the local linear and angular velocities, respectively.
- Training environments: Training uses two Gazebo simulation worlds, progressing from a simple environment to a complex scenario.The simulation environments are used to train the model before real-world testing.
- Reward design: A collision terminates an episode with an additional punishment of −10, while non-collision episodes end after 500 steps without punishment.The reward function also penalizes simply rotating on the spot and favors running as fast as possible.
III. EXPERIMENTAL RESULTS
The model is trained in progressively more complex Gazebo environments and then evaluated in real indoor settings, where D3QN outperforms DQN and DDQN in training speed and performance.
- Training efficiency: D3QN is compared with DDQN and DQN to assess training efficiency and performance and the contribution of dueling and double techniques.The comparison uses smoothed learning curves with average rewards acquired by the robot.
- Real-world evaluation: Real-world tests cover previously unseen indoor environments including libraries, museums, attics, and bedrooms.Figure 6 displays Q-values for linear and angular actions, with red bars marking greedily selected actions.
- Training efficiency: D3QN is about two times faster to train than the normal DQN.The paper also reports that D3QN outperforms the other models in both training speed and performance.
B. Real World Tests
Real-world experiments test whether the trained D3QN can select reasonable obstacle-avoiding actions from monocular vision across varied scenarios.
- The experiments directly test trained models in the real world.
- Action Prediction from Static Images: D3QN produces reasonable actions according to estimated Q values for images from previously unseen environments.The images include libraries, museums, attics, and bedrooms.
- Action Prediction from Static Images: Fig. 7 visualizes image streams alongside the steering actions selected by the robot at each step.
2) Tests in Three Different Scenarios:
Tests in doorways, corridors, offices, and cluttered environments show monocular-camera obstacle avoidance across static and dynamic real-world settings.
- Tests in Three Different Scenarios: The D3QN robot completes 20-second navigation tests in a doorway, corridor, and office.It passes an unseen printer, moves through a narrow gap, and operates safely among previously unseen objects.
- Tests in Three Different Scenarios: In the corridor, the robot navigates smoothly through the narrow space between an obstacle and the wall.
- Tests in a Cluttered Environment: Long-term experiments use a Vicon system to record ground-truth robot poses in a cluttered room with dynamic layouts and objects.
- Tests in a Cluttered Environment: The reward function favors straight motion over turning, leading the greedy policy to follow low-curvature loop trajectories.
- Tests in a Cluttered Environment: The robot avoids dynamic objects in two environments with different complexities using only a monocular camera.
IV. CONCLUSION
The paper proposes D3QN for monocular-RGB obstacle avoidance, trains it in simulation, and transfers it to real-world tasks.
- The proposed deep reinforcement learning algorithm performs obstacle avoidance using only monocular RGB images.
- D3QN combines dueling and double-network techniques to improve learning efficiency over normal DQN and learn from noisy depth predictions.
- The network can be trained solely in a simulator and directly transferred to real-world tasks.
- Future work will augment the network and use auxiliary losses for exploration and global navigation.