Source-linked AI summary
Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight
Katie Kang, Suneel Belkhale, Gregory Kahn, Pieter Abbeel, Sergey Levine
TL;DR
Real-world flight policies need diverse training data, but collecting it can be difficult and simulation does not capture all real-world dynamics. The paper combines real-world dynamics learning with simulation-trained perception, and reports improved collision avoidance on a nano aerial vehicle using limited real-world data.
Problem
Simulation misses complex real-world physical and visual phenomena, while fine-tuning with small real-world datasets can catastrophically overfit.
Source: §I · p. 1Method
The method learns vehicle behavior mainly from real-world data and transfers visual perception features learned in simulation into the real-world policy.
Source: §I · pp. 1–2Results
4× further compared to alternative methods, the approach flew through hallways with varied lighting and geometry.
Source: §I · p. 2Takeaways & Limitations
For nano aerial vehicle collision avoidance, task-specific reinforcement-learning features from simulation transfer better than unsupervised or supervised visual pretraining.
Source: §I · p. 2In curved hallways, 30% of trials ended in glass-door collisions; real-world training lacked glass doors, and many simulated glass doors were traversable.
Source: §V · p. 6
Abstract
from arXiv · showhide
Deep reinforcement learning provides a promising approach for vision-based control of real-world robots. However, the generalization of such models depends critically on the quantity and variety of data available for training. This data can be difficult to obtain for some types of robotic systems, such as fragile, small-scale quadrotors. Simulated rendering and physics can provide for much larger datasets, but such data is inherently of lower quality: many of the phenomena that make the real-world autonomous flight problem challenging, such as complex physics and air currents, are modeled poorly or not at all, and the systematic differences between simulation and the real world are typically impossible to eliminate. In this work, we investigate how data from both simulation and the real world can be combined in a hybrid deep reinforcement learning algorithm. Our method uses real-world data to learn about the dynamics of the system, and simulated data to learn a generalizable perception system that can enable the robot to avoid collisions using only a monocular camera. We demonstrate our approach on a real-world nano aerial vehicle collision avoidance task, showing that with only an hour of real-world data, the quadrotor can avoid collisions in new environments with various lighting conditions and geometry. Code, instructions for building the aerial vehicles, and videos of the experiments can be found at github.com/gkahn13/GtS
I. INTRODUCTION
Generalizing vision-based flight policies requires diverse training data, but collecting it can be difficult for fragile quadrotors. This work combines real-world learning of vehicle behavior with simulation-trained visual perception for collision avoidance.
I. INTRODUCTION
- Large, diverse training datasets can improve generalization, but gathering extensive real-world experience is difficult for fragile, safety-critical quadrotors.The paper motivates integrating more plentiful data sources into training.Source: §I · p. 1
- Real-world dynamics and visual phenomena are difficult to simulate accurately, and fine-tuning with small real-world datasets can catastrophically overfit.Air currents, drift, and turbulence are especially significant for small aerial vehicles.Source: §I · p. 1
- The approach learns vehicle behavior mainly from real-world data while using simulation to learn visual perception that generalizes to new environments.It separates perception and control, transferring simulated visual features while training control with real-world data.Source: §I · pp. 1–2
- Task-oriented visual features trained with reinforcement learning in simulation transfer better than unsupervised or supervised alternatives.The supervised comparison includes standard pre-training on large image-recognition datasets.Source: §I · p. 2
- The method flies 4× further than alternative methods and navigates hallways with varied lighting and geometry.The task uses a real-world nano aerial vehicle and an onboard monocular camera.Source: §I · pp. 1–2
II. RELATED WORK
Prior transfer approaches address simulation-to-reality mismatch through task-agnostic representations, simulator or reality adaptation, and fine-tuning. This work instead emphasizes task-specific perception transfer and adaptation to limited real-world data.
II. RELATED WORK
- The approach transfers task-specific perception layers from a simulation-trained policy, which the authors report is crucial for success.Prior domain-adaptation and feature-space methods generally learn task-agnostic perception models.Source: §II · p. 2
- Other transfer methods reduce simulation-reality gaps by improving the simulator, adapting reality, or randomizing visual or physical properties; this approach complements them by adapting models with limited real-world data.The passage cites simulator improvements,, reality adaptation, [8], visual randomization,, and physical randomization,,,.Source: §II · p. 2
- Unlike prior adaptation work evaluated on low-dimensional or simple dynamical systems, this approach targets raw images and a nano aerial vehicle’s nonlinear dynamics.It uses a scalable, sample-efficient end-to-end latent dynamics model.Source: §II · p. 2
- ImageNet perception layers followed by fine-tuning performed poorly compared with the proposed method for nano aerial vehicle collision avoidance.The comparison uses perception layers from a model trained on ImageNet.Source: §II · p. 2
- Unlike prior aerial-flight work using planning, expert demonstrations, or large real-world datasets, this work focuses on adapting with limited real-world data for a SWaP-constrained vehicle.The cited approaches include geometric mapping and planning,, expert-based methods,, [3], and large-data methods,.Source: §II · p. 2
III. PROBLEM FORMULATION
The formulation seeks a real-world policy from a small real-world dataset and a simulated dataset. The policy’s objective is to maximize expected discounted future reward.
III. PROBLEM FORMULATION
- The policy is parameterized by θ and selects actions from states to maximize expected discounted future rewards under the transition distribution.The discount factor γ determines how much distant rewards contribute to the objective.Source: §III · p. 2
- The robot’s next state follows an unknown transition distribution, and each action produces a task-specific reward.The formulation represents this as T(st+1|st, at) and rt.Source: §III · p. 2
- Training assumes access to a small dataset gathered by the robot in the real world alongside a simulator dataset.The resulting policy is intended to generalize in the real world.Source: §III · p. 2
IV. GENERALIZATION THROUGH SIMULATION
The method assigns complementary roles to real-world and simulated data: real experience informs vehicle dynamics, while simulation supplies visual diversity for learning perception. A simulation-trained task-specific Q-function provides perception layers for a real-world reward predictor.
IV. GENERALIZATION THROUGH SIMULATION
- Simulation reinforcement learning trains a task-specific Q-function, whose perception layers initialize the real-world action-conditioned reward predictor.The predictor estimates future rewards from the current state and future actions; selecting actions that maximize them yields a control policy.Source: Figure 2 · p. 3
- The authors report that real-world fine-tuning of the Q-function performs poorly, the reward predictor better suits limited real-world data, and task-specific simulation learning improves perception transfer.These are the three experimental findings summarized with the approach.Source: Figure 2 · p. 3
- Real-world data provides accurate signals about robot dynamics but limited visual diversity, while simulation provides abundant visual variety but unrealistic dynamics.The method uses the real world for learning dynamics and simulation for a generalizable visual perception system.Source: §IV · pp. 2–3
A. Real-World Policy Learning
Real-world policy learning uses an action-conditioned reward predictor to make learning sample-efficient from limited robot data. At test time, a finite-horizon controller optimizes planned actions against predicted future rewards and executes them in a receding-horizon loop.
A. Real-World Policy Learning
- The action-conditioned reward predictor takes the current image state and a sequence of planned actions as input and predicts rewards for the next H steps.Its image input is encoded by a convolutional network, then a recurrent latent dynamics module updates the representation H times before reward predictions are produced.Source: A. Real-World Policy Learning · pp. 3–4
- Training minimizes reward-prediction error on real-world data, while test-time control chooses an action sequence to maximize predicted future reward.The controller executes only the first action, then replans from the subsequent state in a receding-horizon MPC loop, using CEM for approximate optimization.Source: A. Real-World Policy Learning · p. 3
- Reducing learning to supervised reward prediction gives the approach favorable sample efficiency and greater stability than deep Q-learning in low-data regimes.Unlike image-sequence dynamics models, it predicts reward directly from the current image state and planned actions, avoiding the challenge of predicting a high-bandwidth sensory signal.Source: A. Real-World Policy Learning · pp. 3–4
B. Transferring Visual Perception Systems from Simulation
The approach uses simulation to learn task-specific visual features while reserving real-world data for policy learning, addressing overfitting to limited real-world experience. It transfers and freezes the simulation-trained perception layers in a real-world action-conditioned reward predictor.
B. Transferring Visual Perception Systems from Simulation
- The real-world policy can overfit its training data and fail to generalize to novel environments, motivating the use of simulation data.The paper notes the immense visual diversity of real-world environments as a challenge to generalization.Source: B. Transferring Visual Perception… · p. 4
- The simulated model is task-specific because the authors aim to distill task-relevant features, while using simulation for visual diversity rather than accurate NAV dynamics.The paper’s rationale is that simulators provide realistic, diverse visual scenes but do not accurately model complex real-world NAV dynamics.Source: B. Transferring Visual Perception… · p. 4
- A deep Q-function is trained in simulation because deep Q-learning requires large datasets and can learn long-horizon tasks that may improve its visual features.The Q-function represents expected future reward from a state and action, assuming optimal behavior thereafter.Source: B. Transferring Visual Perception… · p. 4
- The simulation-trained Q-function’s perception layers initialize the real-world policy and remain fixed during training to prevent overfitting to real-world data.The Q-function is trained by minimizing Bellman error on simulation-gathered state, action, reward, and next-state tuples; its actions are selected by maximizing the Q-function.Source: B. Transferring Visual Perception… · p. 4
C. Algorithm Overview
The algorithm first learns visual features from diverse simulated environments, then uses those features in a real-world policy trained on robot data. The perception parameters are held fixed during real-world training to support generalization to novel environments.
C. Algorithm Overview
- First, the method trains a deep Q-function with deep reinforcement learning in visually diverse simulated environments.Its perception layers provide the visual representation used in the real-world model.Source: C. Algorithm Overview · p. 4
- Next, the method creates an action-conditioned reward prediction model using the simulation-trained Q-function’s perception layers to process images.The real-world model uses the transferred visual perception system.Source: C. Algorithm Overview · p. 4
- Finally, the model is trained on robot-gathered real-world data without updating its perception parameters, with the aim of generalizing to novel environments.Source: C. Algorithm Overview · p. 4
V. EXPERIMENTS
The experiments evaluate whether simulation-trained perception combined with real-world data can transfer to unseen hallways. GtS completed the straight-hallway task and succeeded in most trials across three harder settings, while prior methods failed there.
V. EXPERIMENTS
- In an unseen straight hallway, GtS reliably flew the full length to the maximum flight time; the best prior method collided after a quarter of the hallway.The straight hallway was not included in real-world training.Source: Table I · p. 5, §V · pp. 5–6
- Simulation training used 17 million data points from 16 environments, while one hour of real-world collection in a single hallway yielded 14,000 points.The simulator used visually diverse scanned environments and simplified point-mass camera dynamics; real-world training held speed and height constant.Source: §V · p. 5
- The comparison included simulation-only and fine-tuned Q-function policies, real-world-only training, ImageNet and VAE perception transfer, and GtS.Table I records time to collision over five attempts per method in the unseen straight hallway.Source: §V · p. 5
- Sim-only usually collided at doors, Q-learning fine-tuning was inconsistent, and real-world-only training made limited progress, likely because of overfitting.GtS improved real-world performance over the compared methods and outperformed transfer from task-agnostic perception models.Source: §V · p. 6
- Across a tilted-camera straight hallway, a curved hallway with varying lighting, and a zig-zag hallway, GtS succeeded in most trials while the prior methods failed.The two comparison methods were Sim-only and VAE transfer; curved-hallway failures often involved glass doors.Source: Table II · p. 6, §V · p. 6
- Glass-door collisions remained a failure case because real-world training lacked glass doors and simulation often treated them as traversable.The paper notes that additional real-world data could help the vehicle learn from these errors.Source: §V · p. 6
VI. DISCUSSION
The approach pairs simulation-learned generalization with real-world adaptation to vehicle dynamics, and enables a nano aerial vehicle to fly through novel, complex hallways. The authors identify the separation between these steps as an opportunity for future integrated data collection.
VI. DISCUSSION
- The approach enables a nano aerial vehicle to fly through novel, complex hallway environments.The experiments also evaluate the method’s design decisions.Source: §VI · p. 6
- Simulation learns how to generalize, while real-world data adapts the simulated model to real-world dynamics.The method transfers simulation-trained perception layers to a sample-efficient, action-conditioned reward predictor trained on real-world data.Source: §VI · p. 6
- The approach currently treats simulation learning and real-world adaptation as separate steps; tightly integrated active data collection is proposed as future work.The proposed loop would let the robot leverage both simulated and real-world data during collection.Source: §VI · p. 6