PaperlayerView exact revision on arXiv

Source-linked AI summary

CAD2RL: Real Single-Image Flight without a Single Real Image

Fereshteh Sadeghi, Sergey Levine

arXiv:1611.04201v4cs.LGcs.CVcs.RO

TL;DR

Vision-based RL for real flight is constrained by data-intensive, collision-dependent trial and error, motivating simulation-only training. CAD2RL uses randomized synthetic environments and monocular RGB input to learn collision avoidance, and the resulting policies navigate real indoor environments while outperforming baselines despite some collisions.

  • Problem

    Vision-based RL for real autonomous flight requires extensive data and risky collision-based trial and error, leaving simulation-to-real generalization without real training images as an open question.

  • Method

    CAD2RL trains a deep RL collision-avoidance policy entirely on randomized synthetic monocular RGB images, directly predicting action values without explicit 3D reconstruction.

  • Results

    CAD2RL policies navigate real indoor environments using monocular images, outperform baselines, and show that randomized synthetic training can transfer despite some real-world collisions.

  • Takeaways & Limitations

    Purely simulation-trained deep navigation policies show potential for real-world indoor flight on quadrotors and other mobile robots.

Abstract

from arXiv · show

Deep reinforcement learning has emerged as a promising and powerful technique for automatically acquiring control policies that can process raw sensory inputs, such as images, and perform complex behaviors. However, extending deep RL to real-world robotic tasks has proven challenging, particularly in safety-critical domains such as autonomous flight, where a trial-and-error learning process is often impractical. In this paper, we explore the following question: can we train vision-based navigation policies entirely in simulation, and then transfer them into the real world to achieve real-world flight without a single real training image? We propose a learning method that we call CAD$^2$RL, which can be used to perform collision-free indoor flight in the real world while being trained entirely on 3D CAD models. Our method uses single RGB images from a monocular camera, without needing to explicitly reconstruct the 3D geometry of the environment or perform explicit motion planning. Our learned collision avoidance policy is represented by a deep convolutional neural network that directly processes raw monocular images and outputs velocity commands. This policy is trained entirely on simulated images, with a Monte Carlo policy evaluation algorithm that directly optimizes the network's ability to produce collision-free flight. By highly randomizing the rendering settings for our simulated training set, we show that we can train a policy that generalizes to the real world, without requiring the simulator to be particularly realistic or high-fidelity. We evaluate our method by flying a real quadrotor through indoor environments, and further evaluate the design choices in our simulator through a series of ablation studies on depth prediction. For supplementary video see: https://youtu.be/nXBWmzFrj5s

I. INTRODUCTION

The paper addresses indoor collision avoidance by learning directly from monocular images, avoiding explicit 3D reconstruction and reducing reliance on costly sensors. CAD2RL trains entirely in randomized simulation and transfers to real quadrotor flight without real training images.

  • Monocular-image control avoids explicitly modeling 3D environment structure while predicting collision-free commands with a deep convolutional network.The approach avoids separate obstacle detection and control by using reinforcement learning to predict collision probabilities from raw images.
  • RL-based flight training is difficult because it is data-intensive and requires collision-inducing trial and error on fragile, battery-limited quadrotors.
  • Randomized simulation transfers monocular RGB obstacle-avoidance policies to real indoor flight without requiring highly realistic rendering or real training images.Randomized appearances and lighting encourage invariant representations that capture geometric features and open spaces.
  • CAD2RL combines randomized synthetic environments with deep RL to train collision-avoidance policies and provide a simulation testbed for navigation learning.The synthetic hallways vary furniture placement, lighting, and textures.
  • Real-world experiments show that simulation-trained policies can navigate many indoor obstacles with monocular images, outperforming baselines despite experiencing some collisions.The evaluation includes real-world transfer experiments and simulated comparative studies.

II. RELATED WORK

Prior indoor navigation methods commonly reconstruct local 3D geometry before planning, whereas CAD2RL directly predicts collision likelihood from monocular images. Its simulation-based deep RL approach combines raw-image processing with action evaluation and avoids human demonstrations during training.

  • Standard indoor navigation first maps local geometry and then computes a collision-free path using mapping, localization, and motion planning.
  • CAD2RL sidesteps explicit 3D reconstruction by directly predicting collision probability from an image and candidate action.
  • Simulation enables a collision predictor trained with deep RL to generalize from synthetic environments to real-world domains with systematic discrepancies.
  • Unlike prior raw-visual control approaches, the method uses RL without human demonstrations and commands both heading and altitude in 3D.
  • The model predicts Q(It, at) from monocular observations and actions, then selects actions by maximizing the learned collision-avoidance value.Training uses reinforcement learning and a reward designed around collision avoidance and distance from obstacles.

A. Perception-Based Control

The perception-based controller represents navigation actions as image-space grid locations that become velocity commands. This representation supports flexible 3D maneuvering, while heuristic free-space pretraining initializes the policy.

  • The image is discretized into an M × M grid, with each bin representing a navigation action transformed into a normalized velocity command.
  • Image-space actions allow the vehicle to choose arbitrary navigation directions, change altitude, and negotiate tight turns around obstacles.
  • A heuristic pretraining phase initializes the model by predicting whether a one-meter collision-raycast direction intersects an obstacle.

C. Reinforcing Collision Avoidance

CAD2RL refines a free-space policy into a collision-aware Q-function using simulated rollouts and dense action evaluation. Simulation enables repeated resets, collision raycasts, and training without collisions, while requiring transfer to real environments.

  • The pretrained free-space model can miss future collisions, so deep RL fine-tunes it to estimate action-conditioned collision outcomes.Free space within 1 meter does not necessarily predict whether maneuvering or later dead-ends will cause collisions.
  • CAD2RL samples randomized states and orientations, rolls out the current policy, and evaluates multiple candidate actions at each observation.The action space contains M^2 actions derived from an M × M score map.
  • Each observation-action pair receives a Q-value representing the expected sum of future rewards, producing densely labeled training data.The resulting dataset contains observation, action, and return tuples for policy updates.
  • The algorithm combines fitted Q-iteration with Monte Carlo policy evaluation using multi-step rollouts of the current policy.Its simulated setup supports repeated rollouts from states and ground-truth collision raycasts while avoiding training-time collisions.
  • Randomized simulated training must bridge the gap between synthetic hallways and real-world environments at test time.The method therefore depends on generalization beyond the simulated training hallways.

D. Network Architecture

The system uses randomized synthetic indoor environments and evaluates collision-free flight across unseen synthetic and realistic settings. Its simulator varies geometry and appearance to support generalization beyond training scenes.

  • Network Architecture: The network represents the Q-function and initial free-space predictor with a dilated fully convolutional VGG16-based architecture.Its output is a 41 × 41 action grid and training uses stochastic gradient descent with cross-entropy loss.
  • Synthetic Training Environments: Synthetic hallways vary in connectivity, rooms, doors, furniture, textures, lighting, and other visual parameters.The environments were manually designed in Blender to represent structures found in real hallways.
  • Synthetic Training Environments: Training uses deliberately non-photorealistic but highly varied scenes so the model handles broader appearance variation than typical real hallways.The stated intuition is that variation in lighting and textures promotes generalizable models.
  • Evaluation Design: Evaluation targets generalization by testing in synthetic and real environments that were not seen during training.The test mazes differ in geometry, distractors, and obstacles, and use 100 unseen test textures.
  • Evaluation Design: A realistic 3D mesh provides a systematically different environment that serves as a proxy for real indoor performance.The evaluation fixes 60 random initialization points across hallway locations for baseline comparability.

1) Quantitative Evaluation:

In a realistically textured hallway, CAD2RL outperforms prior and baseline methods in collision-free flight. Its advantage includes longer flights and fewer failures at difficult geometric features.

  • CAD2RL maintains collision-free flight for 1.2 kilometers in about 40% of cases.The method substantially outperforms a supervised controller trained to predict 1 meter of free space.
  • CAD2RL outperforms LRS and other baselines by a substantial margin in the quantitative hallway comparison.The reported comparison is summarized in Figure 5 for a realistically textured hallway.
  • CAD2RL has fewer collisions at dead-ends, corners, and junctions than the LRS and free-space prediction controllers.The free-space controller can become trapped in rooms or fail near junctions and corners, while LRS is vulnerable at corners and poor initial locations.
  • The authors attribute the stronger medium-horizon behavior to reinforcement learning compared with directly supervised greedy methods.The trajectory visualization compares the 25 longest flights from each method.

B. Real World Flight Experiments

Real-world experiments fly CAD2RL-controlled quadrotors through varied indoor spaces, including hallways, rooms, junctions, furniture, and confined obstacles. The policy demonstrates recovery from drift and generalization to obstacles beyond walls.

  • Experiments use Parrot Bebop 1.0 and 2.0 quadrotors in indoor scenarios involving sharp turns and collision avoidance around furniture, walls, and fixtures.Both platforms are controlled through the ROS Bebop autonomy package.
  • Controlled hallway experiments compare CAD2RL with FS-pred across fixed velocities, platforms, environments, and initial conditions.The experiments are conducted in Cory Hall and Sutardja Dai Hall, which differ in floor plans, textures, and lighting.
  • CAD2RL experiences fewer collisions and longer expected safe flight than the baseline, while both methods perform better in Cory Hall than SDH.SDH has narrower hallways, glossy textureless walls, and stronger air currents.
  • The controller successfully navigates narrow doors, rooms, junctions, and a U-shaped maze built from chairs and boards.These scenarios include randomly appearing obstacles with varied appearances and colors.
  • The policy can recover from drift caused by imperfect stabilization, turbulence, and air currents, but particularly novel situations remain confusing.The model performs reasonably well on general obstacles and can often recover from mistakes.

C. Ablation Study for Real World Transfer

The ablation study tests how simulation randomization affects transfer to real images using a free-space prediction proxy. Randomized textures and lighting outperform fixed or realistic simulation alternatives.

  • Ablation design: The study uses free-space prediction from real RGB images with depth-camera labels as a lower-variance proxy for real-world transfer.The proxy evaluates whether each pixel contains an obstacle within a threshold distance or corresponds to free space.
  • Real-world transfer: Real indoor flight examples cover furniture, corners, windows, stairs, narrow corridors, junctions, confined mazes, and dynamic obstacles.The figure's red dots indicate CAD2RL's commanded flight direction.
  • Ablation design: The ablated training conditions compare fixed textures and lighting on 3 or 9 hallways, realistic textures and geometry, and randomized textures and lighting on 9 hallways.The realistic condition uses SLAM-based reconstruction data from Cory Hall, whereas the other conditions use synthetic hallways.
  • Evaluation: The evaluation uses per-pixel free-space probabilities, threshold sweeps, and precision-recall curves to measure prediction accuracy.The real RGB-D dataset contains 620 images from five hallways, with depth used to generate free-space labels.
  • Results: The randomized simulator produces the strongest free-space prediction results among the compared training conditions.Figure 8 reports precision curves with shaded standard-deviation regions.

VI. DISCUSSION

The paper presents simulation-only training for monocular-image obstacle avoidance and hallway following, then evaluates transfer to real flight and free-space prediction. It concludes that randomized synthetic scenes support real-world flight, while future combinations with real data may improve performance.

  • Discussion: The method trains deep neural network policies for obstacle avoidance and hallway following using only simulated monocular RGB images.The paper also introduces a simple and stable deep reinforcement learning algorithm for simulation.
  • Discussion: Randomized simulated scenes produce a model that can fly and avoid obstacles in the real world.The paper evaluates randomization quantitatively through a proxy free-space prediction task.
  • Discussion: The simulated evaluation outperforms several baselines and a prior end-to-end learning-based method.The stated aim is to assess the potential, benefits, and limitations of policies trained entirely in simulation.
  • Future work: Future work could combine simulated training with real data through finetuning or domain adaptation to improve real-environment performance.This is presented as a promising direction rather than as an evaluated result of the current method.

VII. APPENDIX

The appendix provides additional generalization experiments, evaluation procedures, and simulator details. It evaluates collision-free flight across varied initial states and compares CAD2RL with learned and ground-truth free-space controllers.

  • Additional experiments: Additional experiments test CAD2RL in three synthetic mazes with different hallway geometries, distractors, and obstacles.The test mazes are larger and more challenging than the training hallways, and use 100 unseen test textures.
  • Evaluation procedure: Collision-free flight is measured by running episodes until collision and counting the steps completed before termination.The maximum number of steps is fixed across experiments.
  • Evaluation procedure: Initial positions are sampled uniformly from each environment's free space to reduce bias from difficult locations such as dead ends, sharp turns, and doorways.The procedure is intended to test robustness across varied starting conditions.
  • Evaluation procedure: Performance curves report the percentage of trials reaching each traveled distance before collision.This reveals whether a policy tends to collide early or sustain longer collision-free flight.
  • Evaluation caveat: Complete collision-free flight from all randomized initial configurations is described as exceptionally difficult.The appendix notes that the same evaluation metric is used in the quantitative experiment.
  • Baselines: The comparison includes prior learning-based obstacle-avoidance methods, a supervised free-space predictor, and an upper-bound controller using perfect LIDAR or depth sensing.The ground-truth free-space controller represents the upper-bound performance of a free-space-prediction-based controller.

B. Results and Analysis

CAD2RL achieves the strongest learning-based collision-avoidance performance in simulated hallways and maintains high free-space prediction precision on unseen photo-realistic images. The results support randomized simulation as a transfer strategy despite a domain gap.

  • Collision-free flight: CAD2RL consistently outperforms the other baselines and FS-pred in hallway scenarios with and without furniture.FS-GT performs best without furniture, while furniture reduces its performance because of its greedy strategy; CAD2RL remains best among learning-based methods.
  • Free-space prediction: The free-space predictor reaches 80% precision on unseen photo-realistic images despite a 10% precision gap from synthetic images.The gap is attributed to domain shift, and the real hallways are wider than the synthetic hallways.
  • Free-space prediction: The free-space evaluation uses precision and Jaccard similarity for correctly labeling free-space and obstacle pixels.The test includes 4k rendered frames sampled uniformly along the hallways.
  • Collision-free flight: CAD2RL achieves the longest collision-free flights in simulated test hallways aside from the ground-truth FS-GT upper bound.The comparison is reported in Figure 10's quantitative results.

X. DETAILS ON REAL ENVIRONMENT TESTS

Real-flight tests used Parrot Bebop drones with center-cropped onboard images, and evaluated CAD2RL across randomized hallway environments. The controller navigated free spaces while avoiding collisions, despite occasional drift from flight disturbances.

  • Real drone platforms: Experiments used Parrot Bebop 1.0 and 2.0 platforms with different physical dimensions, weights, and maximum speeds.Both drones were controlled through the ROS Bebop autonomy package.
  • Camera processing: The onboard camera images were center cropped to remove fisheye distortion, making objects appear closer than their actual distance.
  • Flight outcomes: The controller successfully guided the drone through free spaces while avoiding collisions, although turbulence and imperfect stabilization sometimes caused lateral drift.The controller could recover from these disturbances.
  • Hallway environments: The test environments comprised 24 hallways based on 12 floorplans, with optional furniture drawn from 21 varied items.Furniture dimensions were slightly randomized to increase environmental diversity.
  • Hallway environments: Rendering and viewpoint diversity came from randomizing wall textures, lighting, furniture placement and identity, camera height, yaw perturbations, and camera position.Wall textures were selected from a pool of 200 possibilities.
Loading 1611.04201v4…