PaperlayerView exact revision on arXiv

Source-linked AI summary

AlphaPilot: Autonomous Drone Racing

Philipp Foehn, Dario Brescianini, Elia Kaufmann, Titus Cieslewski, Mathias Gehrig, Manasi Muglikar, Davide Scaramuzza

arXiv:2005.12813v2cs.ROcs.CVeess.SY

TL;DR

Autonomous drone racing is difficult because high-speed flight degrades visual-inertial localization, while race courses demand rapid navigation with limited onboard computation. AlphaPilot combines CNN-based gate-corner detection, EKF fusion, global gate mapping, and near time-optimal planning to address these constraints. It completed the 74 m championship course in 11.36 s, reaching 8 m/s, and demonstrated navigation toward gates that were not directly visible.

  • Problem

    High-speed flight causes VIO performance to degrade through large optic flow and state-estimate drift, challenging autonomous navigation in drone racing.

  • Method

    The system fuses CNN-based gate-corner detection, VIO, and an EKF to compensate drift, build a global gate map, and plan near time-optimal paths.

  • Results

    11.36 s on the 74 m championship course, with a peak velocity of 8 m/s; the system also navigated toward gates that were not directly visible.

  • Takeaways & Limitations

    Global gate mapping and drift compensation supported high-speed racing and planning beyond immediately visible gates.

  • Takeaways & Limitations

    At 8 m/s, the system achieved a 60% success rate because VIO tracking failures and absent visible gates caused crashes after the first few gates.

Abstract

from arXiv · show

This paper presents a novel system for autonomous, vision-based drone racing combining learned data abstraction, nonlinear filtering, and time-optimal trajectory planning. The system has successfully been deployed at the first autonomous drone racing world championship: the 2019 AlphaPilot Challenge. Contrary to traditional drone racing systems, which only detect the next gate, our approach makes use of any visible gate and takes advantage of multiple, simultaneous gate detections to compensate for drift in the state estimate and build a global map of the gates. The global map and drift-compensated state estimate allow the drone to navigate through the race course even when the gates are not immediately visible and further enable to plan a near time-optimal path through the race course in real time based on approximate drone dynamics. The proposed system has been demonstrated to successfully guide the drone through tight race courses reaching speeds up to 8m/s and ranked second at the 2019 AlphaPilot Challenge.

I. INTRODUCTION

AlphaPilot targets high-speed autonomous drone racing, where visual and computational constraints degrade conventional localization. Its system combines gate-corner perception, drift compensation, global mapping, and near time-optimal planning to race through challenging courses.

  • I. INTRODUCTION: High-speed drone racing stresses autonomous navigation through large optic flow, motion blur, low light, and limited onboard computation.
  • I. INTRODUCTION: Conventional VIO performance degrades during agile flight because large optic flow makes sequential feature detection and tracking difficult, causing state-estimate drift.
  • I. INTRODUCTION: The system fuses CNN-based gate-corner detection with VIO in an EKF, using gate observations to compensate drift and construct a global gate map.
  • I. INTRODUCTION: Unlike approaches limited to the next gate, it uses arbitrary and simultaneous gate detections, supporting navigation when gates are not immediately visible.
  • I. INTRODUCTION: The paper extends earlier work with detailed gate-corner detection, VIO-gate fusion, path parameterization, and ablations of detection and planning choices.

II. ALPHAPILOT RACE FORMAT AND DRONE

The AlphaPilot challenge required identical onboard drones to autonomously traverse ordered gate sequences under uncertain gate positions. The paper describes the race setting, sensing hardware, and approximate rigid-body model used for autonomy.

  • A. Race Format: More than 400 teams entered qualification tests, with nine selected for the challenge and six finalists competing in the championship race.
  • A. Race Format: Each time-trial attempt required autonomous passage through distinct gates in the correct order and termination at a designated finish gate.
  • A. Race Format: The system architecture separates sensor interface, perception, state estimation, planning and control, and drone interface into five functional groups.
  • A. Race Format: Gate layouts and sequences were provided before races with approximately ±3 m horizontal uncertainty, requiring adaptation to actual gate positions.
  • B. Drone Specifications: The provided drone measured approximately 0.7 m in diameter, weighed 3.4 kg, and had a thrust-to-weight ratio of 1.4.
  • B. Drone Specifications: Its sensor suite included forward-facing stereo camera pairs, an IMU, a downward-facing laser rangefinder, and an onboard NVIDIA Jetson Xavier computer.
  • C. Drone Model: The drone is modeled as a rigid body with mass m, velocity-proportional rotor drag, translational position, and quaternion-based orientation.
  • C. Drone Model: Experimentally identified rotor-drag coefficients are d_x = 0.5 kg/s and d_y = 0.25 kg/s, with gravity and quaternion multiplication included in the motion model.

III. SYSTEM OVERVIEW

The system asynchronously processes camera, inertial, range, and gate data to estimate and correct vehicle state, map gates, and plan controlled motion. Gate-corner detections provide landmarks for EKF-based drift correction and downstream multi-gate planning.

  • III. SYSTEM OVERVIEW: The architecture contains sensor interface, perception, state estimation, planning and control, and drone interface components.
  • III. SYSTEM OVERVIEW: Two central forward-facing stereo cameras support gate detection and VIO, reducing image-processing load while preserving a large field of view.
  • III. SYSTEM OVERVIEW: An EKF fuses gate detections, VIO, and downward-facing laser measurements to estimate a global gate map and compensate VIO drift.
  • III. SYSTEM OVERVIEW: The system predicts state estimates forward by approximately 130 ms using IMU measurements to offset camera and VIO latency.
  • III. SYSTEM OVERVIEW: A near time-optimal path through the next N gates is replanned after gate passage or significant map or drift updates, then tracked by cascaded position and attitude control.
  • III. SYSTEM OVERVIEW: Gate detection, VIO, EKF estimation, and planning and control run asynchronously on separate CPU cores, with detection at 60 Hz and VIO at approximately 35 Hz.
  • III. SYSTEM OVERVIEW: The detector segments four gate-corner classes and four part-affinity fields from each image using a supervised mean-squared-error objective.
  • III. SYSTEM OVERVIEW: Known gate geometry constrains relative pose, while corner-to-gate association resolves ambiguity when multiple gates appear in one image.

1) Corner Maps:

Corner maps encode the likelihood of each gate-corner class at image pixels, while Part Affinity Fields represent directed connections between corners of the same gate. Per-gate maps are aggregated while preserving distinctions between nearby corners.

  • Corner Maps: Each corner-class map indicates the likelihood that a corner of class j exists at a specific image pixel.The maps have the same spatial size as the input image.
  • Corner Maps: The ground-truth corner-map value is modeled by a Gaussian centered at the nearest ground-truth corner of class j.The implementation uses σ = 7 pixel to accommodate small hand-annotation errors.
  • Corner Maps: Corner maps are generated separately for each gate and aggregated with a pixel-wise maximum to preserve distinctions between close corners.This aggregation retains the strongest response at each pixel across individual gate maps.
  • Part Affinity Fields: Each Part Affinity Field is a two-channel image vector pointing from one designated corner to its connected corner of the same gate.Fields are defined for the four gate-edge classes and apply to image points within distance d of the edge.
  • Part Affinity Fields: Part Affinity Fields for overlapping gates are aggregated by averaging the corresponding unit vectors at shared image points.The fields are first generated separately for each gate.

B. Stage 2: Corner Association

Stage 2 extracts candidate corners, scores possible connections using Part Affinity Fields, and assembles consistent gate detections through decomposed bipartite matching. The resulting associations support downstream state estimation and mapping.

  • B. Stage 2: Corner Association: Corner candidates are extracted by non-maximum suppression and thresholding, then combined exhaustively into candidate edges.Each candidate edge receives a score based on agreement with its corresponding Part Affinity Field, computed using a sampled line integral.
  • B. Stage 2: Corner Association: The global corner-to-gate assignment is NP-hard, so association is decomposed into independent bipartite matching problems for each edge class.This decomposition follows the edge structure of the gate detector.
  • B. Stage 2: Corner Association: The matching optimization selects compatible corner pairs while preventing two edges from sharing a corner, and is solved with the Hungarian method.The result is a set of edge candidates for every edge class.
  • B. Stage 2: Corner Association: Pairwise edge associations are extended into sets of associated edges for each gate.This produces complete gate-level detections from the independently matched edge classes.
  • B. Stage 2: Corner Association: The detector is implemented as a supervised U-Net-based neural network trained to predict corner maps and Part Affinity Fields.The architecture search is restricted to U-Net variants to balance computation time and accuracy.

A. Measurement Modalities

The EKF treats the VIO estimate as an input parameter and fuses it with nonlinear gate-detection and laser-rangefinder measurements. This supports efficient estimation while accounting for frame alignment and drift.

  • A. Measurement Modalities: All measurements through the camera frame time are passed to the EKF together with the VIO pose estimate.The VIO estimate is treated as a constant parameter rather than as a filter state, simplifying derivation and computation.
  • A. Measurement Modalities: Gate measurements consist of image pixel coordinates for identified top-left, top-right, bottom-left, and bottom-right corners.Gates are indexed separately, and all gates share the same width and height in the model.

1) Gate Measurements:

Gate-corner measurements are modeled by projecting known gate-frame corner locations into camera pixels with a pinhole-camera model. The resulting measurement function supports EKF linearization through its state-dependent Jacobian.

  • 1) Gate Measurements: Known gate-corner coordinates are projected into image pixels using the pinhole-camera model.The projection uses camera focal lengths and optical-center coordinates.
  • 1) Gate Measurements: The projected corner position depends on the camera pose and the transformation between the drone body frame and camera frame.These transformations connect the inertial, body, and camera coordinate frames.
  • 1) Gate Measurements: The EKF measurement Jacobian is derived by applying the chain rule to the projection and the state-dependent corner transformation.The state dependence includes gate position and orientation as well as frame alignment.
  • 1) Gate Measurements: Gate detection supplies sets of corner-pixel measurements for an unknown gate, with each measurement associated with a known corner class.These measurements are used to identify correspondences for the filter.

2) Gate Correspondences:

The system matches detected gate measurements to mapped gates using reprojection error and fuses vision, inertial, and range measurements with an EKF. For planning, it simplifies the drone to a point mass with bounded acceleration and derives time-optimal motion primitives.

  • Gate Correspondences: Gate correspondence is established by assigning each detection set to the mapped gate with the minimum summed corner reprojection error.The method first reprojects all candidate gate corners, then compares them with the measured corner set.
  • Gate Correspondences: For path planning, the drone is approximated as a point mass with bounded acceleration inputs, enabling closed-form approximate time-optimal motion primitives in real time.The simplification neglects the quadrotor’s full acceleration dynamics while retaining the dynamics considered most relevant for planning.

A. Time-Optimal Motion Primitive

The planner constructs time-optimal motion primitives under bounded acceleration and velocity constraints, then concatenates them through the ordered gates. A sampled shortest-path formulation makes near time-optimal multi-gate planning computationally efficient.

  • A. Time-Optimal Motion Primitive: 18?
  • A. Time-Optimal Motion Primitive: Time-optimal controls are bang-bang in acceleration, switching between the lower and upper acceleration bounds.Closed-form expressions exist for both the minimum time and switching times.
  • A. Time-Optimal Motion Primitive: A velocity constraint changes the solution to a bang-singular-bang profile, limiting speed while preserving the time-optimal formulation.The speed bound can keep trajectories within regimes where the simplified dynamics remain useful or reduce camera motion blur.
  • A. Time-Optimal Motion Primitive: The three-dimensional primitive uses the slowest axis minimum time as the common duration and scales acceleration on faster axes so all coordinates finish together.The common duration is T* = max(T*x, T*y, T*z), with a scaling parameter α ∈ [0, 1] for faster axes.
  • A. Time-Optimal Motion Primitive: The planner concatenates primitives through the gates and solves a shortest-path problem over M sampled velocities at each gate.Closed-form minimum-time expressions make graph construction and solution efficient.

C. Path Parameterization

Because the simplified motion primitives are not directly feasible for the full quadrotor dynamics, the planned path is converted into piecewise fourth-order polynomial segments for tracking. The controller uses cascaded position and attitude loops.

  • C. Path Parameterization: The simplified-dynamics path is infeasible for the full quadrotor model, so it is approximated by fourth-order time polynomials for accurate tracking.The path is divided into multiple segments of equal arc length.
  • C. Path Parameterization: Each segment is represented as p̄_k(t) = a_4,ks^4 + a_3,ks^3 + a_2,ks^2 + a_1,ks + a_0,k, where s is time since the segment begins.Velocity and acceleration are obtained by differentiating the polynomial.
  • C. Path Parameterization: The position-control design shapes vertical and horizontal position errors as second-order systems with specified time constants and damping ratios.The formulation separately parameterizes the inertial z axis and the horizontal x-y axes.
  • C. Path Parameterization: Figure 6 compares the fastest planned path with sampled alternatives, the VIO trajectory with its corrected estimate, and an executed multi-lap path.The figure uses color for the fastest planned path and gray for sub-optimal sampled paths.
  • C. Path Parameterization: The controller computes total thrust from the desired translational acceleration and the drone’s dynamics to track the parameterized path.The thrust calculation is inserted into the translational dynamics after the position-error response is specified.

B. Attitude Control

The attitude controller converts position-control acceleration demands into a reference orientation and prioritizes the body z-axis for translational control. The system completed the championship race and repeated multi-lap flights despite mostly invisible gates.

  • B. Attitude Control: The required acceleration determines the drone’s reference z-axis, while a reference yaw points the x-axis toward a position 5 m ahead.This makes the drone look in the direction of flight while prioritizing z-axis alignment for translation.
  • B. Attitude Control: The nonlinear attitude controller prioritizes z-axis alignment over yaw correction because z-axis orientation is crucial for translational dynamics.The attitude-error representation uses quaternion components and per-axis first-order time constants.
  • B. Attitude Control: 11.36 s: with a maximum primitive velocity of 8 m/s, the drone completed the 74 m championship course, averaging 6.5 m/s and reaching 8 m/s repeatedly.Only two other teams also completed the full race course; ground truth was unavailable, so the plotted positions are estimated and corrected positions.
  • B. Attitude Control: The drone successfully flew repeated multi-lap trajectories even when gates were usually invisible, using the global map and VIO estimate to plan to unseen gates.Repeated gate observations compensated VIO drift and enabled the drone to pass through the gate centers.

A. Gate Detection

The system combines gate detection with drift-corrected state estimation and receding-horizon trajectory planning. It achieved real-time deployment performance and completed the championship course at high speed, while performance degraded at the highest tested speed.

  • Gate detection: Gate-detector architectures are compared using IoU, precision, recall, and inference timing to balance detection accuracy against computational cost.The comparison covers U-Net variants evaluated on a separate 4k-image test set.
  • Gate detection: 10.5 ms per stereo-camera inference enables gate detection at the camera update rate on the Jetson Xavier, even under strong illumination changes and gate ranges of 2–17 m.The optimized network requires 3.86 GFLOPS in FP16 TensorRT execution.
  • State estimation: The EKF significantly improves gate-relative state estimation over pure VIO by using arbitrary and multiple gate detections to correct accumulated drift and initial yaw error.The corrected estimate prevents the large end-of-track discrepancy that could otherwise cause a gate collision.
  • Planning and control: Receding-horizon planning over N = 3 gates provides near time-optimal paths while keeping computation suitable for real-time replanning.The planner samples M = 150 states per gate, limits velocity to 8 m/s and the gate-normal angle to ϕmax = 30°, and replans in less than 2 ms on the Jetson Xavier.
  • Limitations: At 5 m/s the system achieved a 100% success rate, falling to 60% at 8 m/s because VIO failures and absent visible gates caused crashes after the first few gates.The authors suggest tightly integrating gate measurements into VIO or using perception-aware planning as possible remedies.
Loading 2005.12813v2…