PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-based Deep Reinforcement Learning for Obstacle Avoidance in UAV with Limited Environment Knowledge

Abhik Singla, Sindhu Padakandla, Shalabh Bhatnagar

arXiv:1811.03307v1cs.ROcs.LG

TL;DR

UAV obstacle avoidance with monocular vision remains challenging in unknown, unstructured environments because the camera provides limited geometric information and prior controllers do not retain all relevant observations. The paper combines monocular depth prediction with recurrent deep reinforcement learning and temporal attention, reporting improved navigation over DQN and D3QN while reducing oscillatory motion and power wastage. Its depth predictions remain noisy, leaving visual quality as a stated improvement area.

  • Problem

    Monocular RGB images provide limited geometric information, while existing UAV controllers do not fully retain environmental information for obstacle-avoidance decisions.

  • Method

    The method predicts depth from monocular RGB images and uses recurrent deep reinforcement learning with temporal attention to retain observations and select avoidance actions.

  • Results

    The method reports significant improvements over DQN and D3QN and reduces oscillatory motion that wastes power during navigation.

  • Takeaways & Limitations

    Memory and temporal attention support safer maneuvering in environments without prior knowledge and with moving obstacles, including navigation near corners.

  • Takeaways & Limitations

    The depth prediction network produces noisy depth maps, and the authors identify improving visual quality as future work.

Abstract

from arXiv · show

This paper presents our method for enabling a UAV quadrotor, equipped with a monocular camera, to autonomously avoid collisions with obstacles in unstructured and unknown indoor environments. When compared to obstacle avoidance in ground vehicular robots, UAV navigation brings in additional challenges because the UAV motion is no more constrained to a well-defined indoor ground or street environment. Horizontal structures in indoor and outdoor environments like decorative items, furnishings, ceiling fans, sign-boards, tree branches etc., also become relevant obstacles unlike those for ground vehicular robots. Thus, methods of obstacle avoidance developed for ground robots are clearly inadequate for UAV navigation. Current control methods using monocular images for UAV obstacle avoidance are heavily dependent on environment information. These controllers do not fully retain and utilize the extensively available information about the ambient environment for decision making. We propose a deep reinforcement learning based method for UAV obstacle avoidance (OA) and autonomous exploration which is capable of doing exactly the same. The crucial idea in our method is the concept of partial observability and how UAVs can retain relevant information about the environment structure to make better future navigation decisions. Our OA technique uses recurrent neural networks with temporal attention and provides better results compared to prior works in terms of distance covered during navigation without collisions. In addition, our technique has a high inference rate (a key factor in robotic applications) and is energy-efficient as it minimizes oscillatory motion of UAV and reduces power wastage.

I. INTRODUCTION

UAV obstacle avoidance is difficult because aerial motion exposes the vehicle to diverse obstacles and monocular vision provides limited geometric information. The paper addresses this challenge with memory-based deep reinforcement learning that retains observations over time.

  • UAVs must avoid diverse stationary and moving obstacles, including structures irrelevant to ground vehicles.Indoor examples include furnishings and walking humans whose future motion cannot be predicted.
  • SLAM and SfM are unsuitable for real-time avoidance in several settings because online path planning is difficult and dynamic obstacles remain problematic.The cited limitations also include failure to detect untextured walls.
  • Monocular cameras are lightweight and low-cost, but extracting object range from their two-dimensional RGB images is challenging.This motivates monocular depth prediction for aerial obstacle avoidance.
  • Prior monocular depth-learning methods predict depth maps but do not address UAV obstacle avoidance and navigation.Other prior approaches use substantial hallway-image data or conventional CNN-based control strategies.
  • The proposed controller uses recurrent neural networks with Temporal Attention to store relevant observations and learn an obstacle-avoidance policy.The design is motivated by partial observability: past observations can inform later navigation decisions.

A. Organization of the Paper

The paper next presents its UAV obstacle-avoidance method, evaluates it in simulation, analyzes its advantages and limitations, and concludes with future improvements. The method seeks policies that avoid stationary and mobile obstacles while supporting a high-level path objective.

  • The next section describes the developed UAV obstacle-avoidance method.The paper then details experiments and simulation environments before presenting results, advantages, limitations, and future work.
  • The objective is to learn action policies that avoid both stationary and mobile obstacles during UAV navigation.The method can integrate with a high-level planner specifying an overall path objective, start position, and goal position.

A. Problem Definition

UAV obstacle avoidance is modeled as sequential decision-making under uncertainty and partial observability. The proposed model uses monocular-image depth observations and model-free deep Q-learning with recurrent memory to estimate better state information.

  • A. Problem Definition: UAV actions change position, orientation, visual feedback, and future states, making obstacle avoidance sequential decision-making under uncertainty.The next state is probabilistic after each action.
  • A. Problem Definition: Because a monocular camera has limited field of vision, the controller receives an observation rather than the full environment state, motivating a POMDP formulation.The model input is an RGB image, while its observation is the predicted depth map.
  • A. Problem Definition: The POMDP comprises environment states, feasible actions, transition probabilities, rewards, observations, and a discount factor.The transition function models state evolution from selected actions, while the reward provides feedback for those actions.
  • B. Model: The model predicts a depth map from a monocular RGB image, then selects among going straight, turning right, and turning left.The reward is designed using depth information, while unknown environment dynamics motivate reinforcement learning.
  • B. Model: Q-learning selects actions from estimated discounted rewards, while DQN uses a neural network to approximate Q-values for large state spaces.Experience replay and target-network updates support DQN training.
  • B. Model: Recurrency integrates observations over time to better estimate the underlying state in a partially observable environment.The proposed memory-augmented convolutional network therefore approximates Q-values from observations rather than full states.

C. Deep Recurrent Q-Network with Temporal Attention

The control network uses a deep recurrent Q-network with temporal attention to integrate recent depth-map observations and estimate action values for obstacle avoidance.

  • C. Deep Recurrent Q-Network with Temporal Attention: The architecture tracks past observations with a recurrent network and uses temporal attention to weight recent observations by decision-making importance.The recurrent network learns temporal dependencies, while attention assigns importance to recent observations.
  • C. Deep Recurrent Q-Network with Temporal Attention: At time t, convolutional layers process a sequence of L recent depth maps before an LSTM estimates Q-values from observations, hidden state, and actions.The hidden state is recursively determined from the preceding hidden state and observation.
  • C. Deep Recurrent Q-Network with Temporal Attention: Figure 2 specifies the number of filters, stride, and output size for each convolutional layer in the control network.The figure presents the complete temporal-attention model architecture.
  • C. Deep Recurrent Q-Network with Temporal Attention: Temporal-attention weights are computed from recurrent hidden states and convolutional feature vectors using learnable parameters and a tanh activation.The attention formulation uses w, b_a, W_a, and U_a as learnable parameters.
  • C. Deep Recurrent Q-Network with Temporal Attention: The attention weights are normalized with softmax and used to compute a context vector for subsequent Q-value prediction.The context vector is passed to a fully connected ReLU layer that outputs an approximated Q-value for each action.

D. Obtaining depth maps from RGB images

The depth network uses a conditional GAN to translate monocular RGB images into depth maps, combining adversarial training with an L1 objective for near-ground-truth outputs.

  • D. Obtaining depth maps from RGB images: The conditional GAN maps paired RGB images and depth maps using generator and discriminator ConvNets for intermediate depth prediction.The generator uses an encoder-decoder structure with skip connections, while the discriminator classifies depth images as real or fake.
  • D. Obtaining depth maps from RGB images: The discriminator learns from labeled real and generated depth images, while the generator learns to produce outputs that resemble ground-truth depth maps.Training alternates discriminator and generator updates for several epochs.
  • D. Obtaining depth maps from RGB images: The final objective combines the conditional-GAN loss with an L1 loss, controlled by an adjustable hyper-parameter λ.The L1 term is intended to generate images closer to ground truth.
  • D. Obtaining depth maps from RGB images: Compared with earlier methods, the approach learns a loss function adaptable to input data and is presented as domain independent for intermediate depth prediction.The stated application is obstacle avoidance using monocular RGB input.
  • D. Obtaining depth maps from RGB images: Figure 3 illustrates the conditional GAN architecture used for the depth network.The figure corresponds to the generator-discriminator image-translation model.

E. Remarks

The predicted depth maps remain partial descriptions of the UAV’s surroundings, so the system stores relevant observations over time for navigation decisions.

  • E. Remarks: Depth maps generated from the conditional GAN provide limited information about the visual geometry surrounding the UAV.The passage identifies this as the partial-information issue in monocular-image depth prediction.
  • E. Remarks: The LSTM stores and combines relevant partial information collected from earlier observations to support better navigation decisions.This connects depth prediction with the memory-based controller described earlier.

III. EXPERIMENTAL SETUP

The experiments use varied Gazebo environments and RGB-D data to train and test UAV collision-avoidance algorithms under simulated conditions.

  • III. EXPERIMENTAL SETUP: The conditional GAN is trained initially on 90,000 RGB-D image pairs collected from 22 simulated indoor environments.The environments include hallways, enclosed areas, varied floorings, and structured or unstructured obstacles.
  • III. EXPERIMENTAL SETUP: The designed Gazebo environments vary obstacle and wall colors, textures, sizes, and shapes.Figure 4 shows screenshots of these designed environments.
  • III. EXPERIMENTAL SETUP: RGB-D pairs are collected with a simulated Kinect mounted on the drone across possible viewpoints and are augmented by flipping, jitter, and appearance changes.Training uses the entire dataset for 20 epochs with batch size 4 on an NVIDIA Titan X.

B. Control Network and Simulation Settings

The control network learns action values from recent depth observations in progressively harder simulated environments, using rewards that penalize obstacle proximity and collisions while favoring forward motion. Transfer to physical systems is challenging because simulated and real depth imagery differ, including greater noise in predicted real-world depth maps.

  • Simulation settings: The UAV is trained in simulated environments whose complexity increases through narrower pathways, enclosed free space, and denser obstacles.
  • Control network: The network estimates observation-action values over the last L depth observations for “go straight,” “turn left,” and “turn right.”
  • Reward design: The reward penalizes actions when the UAV is closer than σ − r_drone to an obstacle, with r_drone = 0.292m and σ = 1.5m.
  • Reward design: A collision ends the episode with −10 penalty; otherwise, episodes run up to 1000 steps without penalty, and going straight adds +0.5 reward.
  • Simulation-to-real transfer: Applying the learned policy to physical robots is difficult because real and simulated visual information differ, while predicted real-world depth maps are noisier than simulated Kinect images.

IV. EXPERIMENTAL RESULTS

The experiments evaluate the depth network’s prediction quality and inference speed, then compare the proposed recurrent attention-based control network with baseline policies for obstacle avoidance. Results indicate that temporal memory and attention help under partial observability.

  • Depth network evaluation: Figure 5 shows depth maps generated from simulated-data training on unseen real-world images with variable illumination, color, and texture.The figure encodes far regions in red and near regions in blue.
  • Depth network evaluation: 20Hz average inference rate was observed for the depth model on an NVIDIA GeForce GTX 1050 mobile GPU with 8 GB RAM and an Intel Core i7 machine.Inference speed is evaluated because it is critical for robotic collision avoidance.
  • Depth network evaluation: Depth prediction quality was evaluated using L1 and cGAN loss on 5,000 unseen simulated samples and 2,500 fine-tuned real-world images.The evaluation covered unseen simulated data and fine-tuned real-world data.
  • Depth network evaluation: The depth network was assessed across manually collected data, simulated training data, mixed NYU2 and RGBD-human-explore testing data, and simulated data with fine-tuning.The passage identifies three evaluation configurations represented by rows in Table II.
  • Obstacle-avoidance control: The proposed Deep Recurrent Q-network with Temporal Attention was compared with baseline DQN, random, and straight policies using average steps until collision.DQN and the proposed model were trained in 12 simulated indoor environments containing hallways and rooms with varied obstacles.
  • Obstacle-avoidance control: Under partial observability, DQN performance was hindered, while adding attention to a memory network improved learning by retaining crucial information gathered over time.The comparison specifically contrasts the attention-augmented memory network with its no-attention counterpart.

1) Testing in Simulated environments:

The proposed UAV controller is evaluated across six simulated environments, including static obstacles, narrow mazes, poles, and randomly moving humans. It outperforms DQN across environments, operates at 60 Hz, and reduces wobbling-related power waste compared with D3QN.

  • Testing setup: Six simulated environments test navigation among scattered obstacles, narrow pathways, poles, and randomly moving human actors.The cafe-like environment trains with 3 actors but tests with 7, using 200 episodes per environment.
  • Inference: 60 Hz inference on an NVIDIA GeForce GTX 1050 mobile GPU supports the controller’s use in robotic applications.The reported rate is associated with the learned UAV model evaluated in the simulated cafe environment.
  • Training environments: Training curves compare the proposed network with DQN in open areas with scattered obstacles and maze-like environments with narrow pathways.A separate curve comparison considers enclosed areas containing scattered static obstacles of varying sizes and structures.
  • Obstacle avoidance: The proposed method outperforms DQN across different environments and smoothly avoids both static and dynamic obstacles.The learned policy selects actions such as turning left when a moving human obstructs the UAV’s view.
  • Energy efficiency: The controller minimizes oscillatory left-right motion that consumes power without producing displacement.Energy consumption per meter is computed from battery-level drop and distance covered, comparing the proposed method with D3QN.

V. DISCUSSION

The discussion highlights memory, temporal attention, simulated-data transfer, and energy-aware rewards as strengths of the approach. It also identifies noisy physical-world depth predictions and limited obstacle-avoidance scope as boundaries.

  • Strengths: A cGAN-based depth predictor trained on simulated data with limited fine-tuning shows transferability to real-world datasets.The authors describe this combination as novel and suitable for intelligent robotic applications because of its inference rate and transferability.
  • Strengths: Memory networks and temporal attention retain information from past observations for safer decisions in unknown environments and with moving obstacles.The discussion also reports competent maneuvering near corners, a challenging case for previously proposed controllers.
  • Strengths: The energy-aware reward function biases the UAV toward going straight except when obstacle avoidance requires turning.Compared with D3QN, the resulting trajectories are smoother and wobbling is minimized, reducing power wastage.
  • Limitations: The go-straight reward bias might be problematic at intersections requiring a right or left turn.The obstacle-avoidance controller can be integrated with a high-level planner responsible for computing paths between start and goal positions.
  • Limitations: Depth predictions from unseen physical-world images are noisy, although the noise-trained control network generalizes and adapts to them.The authors identify higher-fidelity simulated training data as a possible route toward smoother depth predictions.

VI. CONCLUSIONS AND FUTURE WORK

The paper concludes that Deep Recurrent Q-Networks with Temporal Attention improve UAV obstacle avoidance in cluttered, unseen environments by retaining information across observation sequences. The method performs well indoors, while outdoor validation and improved history abstraction remain future work.

  • The proposed Deep Recurrent Q-Network with Temporal Attention supports effective UAV obstacle avoidance in cluttered and unseen environments.It retains crucial information over long sequences of observations to address partial observability.
  • The controller predicts depth maps from monocular RGB images using a cGAN before selecting actions.
  • Experimental results show significant improvements over Deep Q-Network and D3QN algorithms.
  • Future work: The method works well for obstacle avoidance in indoor environments, but its performance in real outdoor environments remains to be tested.
  • Future work: Future directions include learning concise recurrent-history abstractions, incorporating scene prediction, and studying low-regret policies for UAV obstacle avoidance.
Loading 1811.03307v1…