Source-linked AI summary
HyperDrone: Simulation-to-Reality Transfer at Millions of Frames per Second
Jonas Eschmann, Giuseppe Loianno
TL;DR
Robot imitation learning can require costly real-world data collection, while proprietary vision-model data can limit reproducibility and introspection. HyperDrone combines a fast simulator and renderer, and the authors report successful transfer of a simulation-trained visual yaw-control policy to a real quadrotor. They present the simulator as a way to lower barriers to research on learned vision-based control.
Problem
Robot imitation-data collection has substantial costs, while proprietary vision-model data can hamper reproducibility, interpretability, and independent study of data-distribution changes.
Source: §1 · p. 2Method
HyperDrone combines a fast simulator for end-to-end visual-control training with a ray-tracing renderer and quadrotor dynamics simulation.
Source: §1 · p. 2, §3.1 · p. 4Results
The authors report successful real-world, entirely vision-based yaw tracking using a policy trained in simulation.
Source: §4.3 · p. 9Takeaways & Limitations
The authors expect HyperDrone to lower barriers to research on learned vision-based control policies.
Source: §6 · p. 9The transfer is a simple, low-frequency proof of concept that relies on an external position reference and RAPTOR as an end-to-end low-level controller.
Source: §5 · p. 9
Abstract
from arXiv · showhide
We present HyperDrone, an extremely fast quadrotor simulator focused on high-performance batch rendering that is more than 20x faster than existing state-of-the-art batch quadrotor simulators. HyperDrone combines highly optimized multirotor dynamics with HyperLux, a new ray-tracing engine that supports Nvidia OptiX, Vulkan, Apple Metal and WebGPU backends that takes full advantage of modern ray-tracing hardware, capable of running both components entirely on the GPU. HyperLux renders RGB, depth, semantic/instance segmentation, surface normals and optical flow observations at configurable levels of fidelity, including reflections, realistic lighting, motion blur, anti-aliasing, and Physically Based Rendering (PBR). Even in the highest-fidelity settings, HyperLux achieves millions of Frames Per Second (FPS) on consumer-grade GPUs while rendering realistic 3D scenes. We demonstrate the merits of HyperDrone by training a vision-based yaw-control policy purely in simulation and transferring it to a real small-scale quadrotor (Crazyflie) with a camera, running completely onboard.
1 Introduction
The introduction contrasts the growing use of imitation learning and pre-trained vision models in robotics with their less widespread adoption for aerial autonomy. It presents HyperDrone as a fast simulator for visual control and reports a simulation-trained yaw-control policy transferred to a real quadrotor.
1 Introduction
- Imitation learning from human demonstrations is less widely adopted for aerial autonomy than for manipulation and locomotion.Quadrotor work has instead emphasized imitation from privileged expert policies and pre-trained vision encoders.Source: §1 · p. 2
- Scaling robot imitation data may offer large returns, but data-collection costs and restricted access to commercial data pose reproducibility barriers.Source: §1 · p. 2
- Training entirely in simulation avoids requiring real-world data and offers more introspection than training with proprietary vision-foundation-model data.The authors note that proprietary training data hampers reproducibility and study of how data-distribution changes affect behavior.Source: §1 · p. 2
- HyperDrone combines a fast, large-batch ray-tracing renderer with highly optimized multirotor dynamics to support end-to-end visual-control training.HyperLux renders RGB, depth, semantic and instance segmentation, surface normals, and optical flow.Source: §1 · p. 2
- The authors demonstrate simulation-to-reality transfer of a yaw-control policy trained entirely in simulation.Source: §1 · p. 2
2 Related Work
Related quadrotor simulators trade rendering throughput, visual features, and hardware compatibility. The section motivates a new simulator focused on performance while supporting realistic visual effects.
2 Related Work
- Table 1 compares quadrotor simulators and rendering backends.Source: Table 1 · p. 2, §2 · p. 2
- Flightmare lacks batch rendering and was measured at approximately 230 FPS; it also requires licensed software to replace its default scenes.Source: §2 · p. 3
- VisFly supports quadrotor dynamics and batch rendering on recent GPUs, but lacks motion blur and anti-aliasing.Source: §2 · p. 3
- Aerial Gym supports depth and semantic segmentation, but not RGB for general 3D mesh scenes, and does not use modern GPU ray-tracing cores.It is also incompatible with Nvidia Blackwell GPUs, according to the passage.Source: §2 · p. 3
- HyperDrone is motivated by the need for high rendering performance alongside motion blur, anti-aliasing, and reflections.The authors focus on global-shutter cameras for now and identify rolling-shutter simulation as future support.Source: §2 · p. 3
3 Method
HyperDrone combines GPU-based ray-traced rendering and quadrotor dynamics to support visual simulation and control. The transfer demonstration trains a yaw-estimation student on synthetic images and deploys it on a camera-equipped Crazyflie.
3.1 Architecture
- The architecture keeps rendering and dynamics on the GPU, avoiding CPU–GPU memory transfers between those components.The dynamics simulator builds on the implementation in [24] and subsequent development.Source: §3.1 · pp. 3–4
- HyperLux uses OptiX, Vulkan, or Apple Metal for ray intersections, with software ray traversal as a WebGPU fallback.Its accelerated structure is used to find mesh intersections for supplied rays.Source: §3.1 · p. 4
3.2 Visual Simulation-to-Reality Transfer
- For transfer, a convolutional student estimates relative yaw from a channel-wise stack of target and current frames, while RAPTOR handles low-level and position control.Both teacher and student run on the GPU during training.Source: §3.2 · p. 4, §3.1 · p. 4
- Training uses diverse ProcTHOR indoor scenes and probe-ray rejection sampling to select initial positions at a specified minimum distance from objects.The method uses 64 rays in a deterministic Fibonacci-sphere distribution; at 64×64 resolution, the stated overhead is roughly 1.6%.Source: §3.2 · pp. 4–5
- The trained yaw commands are sent over UART from the camera-board inference code to the Crazyflie microcontroller for RAPTOR to consume.Inference runs on the camera board’s NPU.Source: §3.2 · p. 5
4 Results
HyperDrone combines high-fidelity visual rendering with high throughput, and its benchmarks show substantially faster performance than existing quadrotor simulators. A yaw-control policy trained in simulation also achieves successful real-world visual tracking.
4.1 Visual Results
- HyperDrone renders RGB, depth, segmentation, surface normals, and optical flow, while 2 × 2 averaged rays reduce aliasing.The visual comparisons include resolutions from 2048 × 2048 to 64 × 64, with anti-aliasing on and off.Source: §4.1 · pp. 6–7
- Motion blur is simulated by averaging N rays sampled throughout the shutter period; N = 32 looks realistic, while N = 2 suffices for transfer.The N = 2 blur appears relatively unrealistic, particularly at a 1/20 s shutter time.Source: §4.1 · p. 7, Figure 7 · p. 7
4.2 Performance
- Vulkan matched OptiX throughput on an RTX 5090, while Metal reached 1 500 309 FPS on Mac Studio and 97 376 FPS on iPhone 17 Pro Max.WebGPU was 9–27× slower than OptiX on the same GPU.Source: §4.2 · p. 7
- More than 20× the throughput of the fastest quadrotor simulators was achieved on an RTX 4090 16GB (80W).At comparable fidelity, Madrona was more than 5× slower; Table 2 compares simulator throughput at 64 × 64 resolution.Source: §4.2 · p. 8, Table 2 · p. 8
4.3 Visual Simulation-to-Reality Transfer
- The policy trained in about 4 hours on an RTX 5090 achieved successful, entirely vision-based yaw tracking in real-world experiments.The GPU process used about 6GB of memory and could run on an RTX 3060 8GB, with training taking 15 hours.Source: §4.3 · pp. 8–9, Figure 8 · p. 9
5 Limitations and Future Work
The demonstrated system has limits in camera modeling, collision avoidance, and task scope.
5 Limitations and Future Work
- Rolling-shutter simulation is unsupported; the current focus is on global-shutter cameras.The authors plan to support rolling shutter in the future.Source: §5 · p. 9
- Probe rays are implemented, but there is no proper collision-avoidance interface and they have not been tested for collision avoidance.Source: §5 · p. 9
- The transfer is a proof of concept for a simple, low-frequency task that relies on an external position reference and RAPTOR for low-level control.Source: §5 · p. 9
6 Conclusion
The authors present HyperDrone as a fast, efficient simulator for synthesizing and training vision-based quadrotor control policies, with the goal of making research on learned visual control more accessible and extensible.
6 Conclusion
- The authors present HyperDrone as a fast, efficient simulator for synthesizing and training vision-based quadrotor control policies.They expect community access to lower research barriers and support an open, extensible framework usable on consumer-grade hardware.Source: §6 · p. 9
A Multirotor Dynamics
The multirotor dynamics section identifies the model variables and state, but the supplied passages do not describe the dynamics equations.
A Multirotor Dynamics
- The model state is given as s := {p, q, v, ω, ωm}, with ωm identified as the motor state.The supplied equation fragment also includes p = v, but does not provide enough context to interpret that relation.Source: A Multirotor Dynamics · p. 13
- Table 3 is identified as listing the variables and parameters of the multirotor dynamics model.Source: Table 3 · p. 13
B Rendering Fidelity Modes
HyperDrone provides three compile-time rendering modes that trade fidelity against throughput while sharing ray generation, acceleration structures, and non-RGB outputs. The modes differ in RGB shading, from baked colors to glTF PBR lighting and reflections.
B Rendering Fidelity Modes
- All three modes share ray generation, acceleration structures, and depth, segmentation, surface-normal, and optical-flow outputs; only RGB shading varies.Compile-time feature flags make disabled features zero-cost at runtime.Source: B Rendering Fidelity Modes · p. 13
- Materials follow the glTF 2.0 model, allowing glTF assets to render without conversion.The model codifies PBR in ISO/IEC 12113:2022.Source: B Rendering Fidelity Modes · p. 13
- Low mode assigns each mesh a flat baked color from its material color factor and mean base-color texture, independent of orientation and lighting.Source: B Rendering Fidelity Modes · p. 13
- Medium mode samples base-color textures at hit points, applies view-dependent darkening without scene lights, and traces one reflection ray for metallic materials.Source: B Rendering Fidelity Modes · p. 14
- High mode uses glTF 2.0 PBR maps, authored punctual lights, ambient illumination, metallic reflections, and transparent continuation rays.Shadows, multi-bounce indirect illumination, and subsurface scattering are not modeled.Source: B Rendering Fidelity Modes · p. 14
C Training Pipeline
The training pipeline uses RAPTOR-generated trajectories to supervise a vision-based yaw estimator in simulation. It samples scenes, setups, and dynamics, then trains and validates the student across a curriculum of initial yaw offsets.
C Training Pipeline
- The student is trained by supervised regression on trajectories from the RAPTOR [8] teacher, with simulation, rendering, teacher, and student running in one process on one GPU.Source: C Training Pipeline · p. 14
- Each epoch uses two distinct training scenes with 64 simulated instances each, sampled setups and dynamics, and batched target frames cached for that epoch.The first 30 ProcTHOR training scenes are split into 25 training scenes and five held-out validation scenes.Source: C Training Pipeline · p. 14
- The teacher controls a 22-dimensional privileged state at 100 Hz, and each frame is labeled with body yaw relative to the target frame.The 50-step, 0.5-second rollout regulates the quadrotor toward its spawn position and level, zero-yaw attitude.Source: C Training Pipeline · p. 14
- The student receives current and target RGB frames in a 50 × 80 × 8 input stack and outputs two values through a 0.73 M-parameter convolutional and dense network.The network uses four stride-2 convolutions followed by three dense layers.Source: C Training Pipeline · pp. 14–15
- The initial-orientation bound increases from 0° to 90° in 1° steps when frontier-band RMSE falls below 25% of the bound, clipped to [2.5°, 5°].Validation uses held-out scenes and fixed yaw offsets up to ±32°.Source: C Training Pipeline · p. 15
- Table 5 summarizes the training-pipeline parameters.Source: Table 5 · p. 15
D Collision Queries
Collision queries reuse the ray-tracing pipeline to measure proximity against the scene mesh. Camera-frame probes follow a deterministic Fibonacci-spiral pattern, which is also used to annotate free-space spawn positions.
D Collision Queries
- Each camera traces N probe rays against the image-rendering acceleration structure and reads back each ray’s first-hit distance and hit flag.The outputs are zero-copy, like the image outputs.Source: D Collision Queries · p. 15
- Probe 0 follows the camera principal axis; the remaining probes use a Fibonacci spiral with equal steps in cos ψ and golden-angle azimuth increments.The pattern covers the sphere evenly for any N and is deterministic.Source: D Collision Queries · p. 15
- Scene positions are accepted for free-space spawning when over 72% of probes hit geometry, mean hit distance is below 45% of scene extent, and nearest-hit clearance exceeds 1 m.At runtime, the same probes measure proximity against the full scene mesh.Source: D Collision Queries · p. 15
E API
The Overview notebook demonstrates HyperDrone’s environment API by loading scenes and drone models, configuring and stepping the environment, and reading rendered outputs.
E API
- The example loads three ProcTHOR scenes and pairs each drone model with its matching dynamics model.The model list includes crazyflie_brushless, arpl, x500, and savagebee_pusher.Source: E API · p. 16
- Each scene becomes a world with four mutually visible agents, while agent 0 carries the sole front camera.The environment is JIT-compiled from Config settings.Source: E API · p. 16
- Runtime parameters merge over nominal values; the example starts drones at zero linear velocity, full propeller speed, and random angular velocity up to 1 rad/s per axis.Source: E API · p. 16
- Reset places drones at predefined spawn points in agent 0’s field of view, observe renders lazily, and step integrates 10 ms using normalized motor commands.The example applies full throttle to every agent.Source: E API · p. 17
- Reading outputs returns snapshots indexed by instances and selected views, with each scene represented as a row in the mosaic.The example reads RGB, depth, surface normals, semantic and instance segmentation, optical flow, and flow status.Source: E API · p. 17