PaperlayerView exact revision on arXiv

Source-linked AI summary

Fully neuromorphic vision and control for autonomous drone flight

Federico Paredes-Vallés, Jesse Hagenaars, Julien Dupeyroux, Stein Stroobants, Yingfu Xu, Guido de Croon

arXiv:2303.08778v1cs.ROcs.AIcs.CVcs.LGcs.NE

TL;DR

Resource-intensive vision and constraints on spiking-network training and embedded hardware have limited neuromorphic robotics. This paper presents a spiking event-to-control pipeline for a freely flying drone, which transferred from simulation to real flight and followed diverse ego-motion setpoints onboard.

  • Problem

    Neuromorphic robotic implementations have been limited by difficult SNN training and embedded processors’ restricted network capacity, while large vision networks also strain small robots.

    Source: §1 · pp. 1–2

  • Method

    The paper trains a spiking network to map raw event-camera data to ego-motion estimates and uses a simulator-evolved linear controller to produce drone commands.

    Source: A fully neuromorphic solution to… · p. 2, Neuromorphic state estimation:… · p. 11

  • Results

    Real-world tests showed successful sim-to-real transfer: the drone followed ego-motion setpoints for hovering, landing, and lateral maneuvers under constant yaw, with onboard operation at 200 Hz and 27 µJ per inference.

    Source: DISCUSSION AND CONCLUSION · pp. 9–10

  • Takeaways & Limitations

    The results suggest neuromorphic sensing and processing may bring deep neural networks within reach of small autonomous robots.

    Source: DISCUSSION AND CONCLUSION · p. 10

    The implementation ends at thrust and attitude commands because fully neuromorphic sensing, processing, and actuation hardware is not available.

    Source: DISCUSSION AND CONCLUSION · p. 10

Abstract

from arXiv · show

Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions due to the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present the first fully neuromorphic vision-to-control pipeline for controlling a freely flying drone. Specifically, we train a spiking neural network that accepts high-dimensional raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28.8k neurons, maps incoming raw events to ego-motion estimates and is trained with self-supervised learning on real event data. The control part consists of a single decoding layer and is learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to-real transfer of the fully learned neuromorphic pipeline. The drone can accurately follow different ego-motion setpoints, allowing for hovering, landing, and maneuvering sideways$\unicode{x2014}$even while yawing at the same time. The neuromorphic pipeline runs on board on Intel's Loihi neuromorphic processor with an execution frequency of 200 Hz, spending only 27 $\unicode{x00b5}$J per inference. These results illustrate the potential of neuromorphic sensing and processing for enabling smaller, more intelligent robots.

1. INTRODUCTION

The introduction frames neuromorphic vision-to-control as a way to address energy and deployment constraints, then presents a spiking pipeline for drone flight that separates vision and control training.

1. INTRODUCTION

  • Large ANN workloads strain resource-constrained robots, while SNN training and limited embedded hardware have kept prior robotic neuromorphic systems relatively simple.Prior examples include low-dimensional sensory inputs or limited drone control, rather than full vision-based control of freely flying drones.Source: §1 · pp. 1–2
  • Neuromorphic sensing and processing promise sparse, low-latency, energy-efficient perception and action, but event-camera pipelines commonly accumulate events into windows processed like image frames.Event-camera pixels transmit brightness changes asynchronously, while spiking processors can exploit binary activity for efficient processing.Source: §1 · pp. 1–2
  • The proposed pipeline processes raw events into corner optical flow and maps visual ego-motion to thrust and attitude commands for autonomous drone control.It assumes a downward-looking camera over a static flat surface and applies identical five-layer, 7.2k-neuron networks to four 16x16-pixel image corners.Source: A fully neuromorphic solution to… · pp. 2–3
  • Vision is trained self-supervised using contrast maximization, while a linear controller is evolved in simulation and transferred to the robot without retraining.During vision training, corner flow estimates parameterize a homography to derive dense flow; the controller is trained from simulated visual observables and setpoints.Source: A fully neuromorphic solution to… · pp. 3–4, Figure 2 · p. 4

RESULTS

The vision network produces optical-flow estimates that capture drone motion, and the learned controller transfers to real-world flight, though it often misses larger horizontal setpoints. Tests also show robustness across several flight and lighting conditions and high Loihi processing speed on sparse event sequences.

Robust vision-based state estimation

  • The spiking vision network produces optical-flow estimates that capture camera motion, including in a fast sequence with approximately 200 deg/s rotation.The network is evaluated without ground-truth information during training despite spike processing, limited field of view, self-recurrency, and hardware quantization.Source: Robust vision-based state estimation · p. 5
  • During fast-sequence processing, every vision-network layer stays below 50% activation; encoder 2 and 3 activity is less tied to input statistics than encoder 1 and pooling.The network was not explicitly trained to promote sparse activations.Source: Robust vision-based state estimation · p. 5
  • The EPE comparison finds that the ANN outperforms spiking variants, while corner-specific processing benefits the proposed architecture and Loihi quantization increases its error.Capping events at 90 per corner also reduces internal activity in the evaluated SNNs.Source: Robust vision-based state estimation · p. 5, Table 1 · p. 7

Control through visual observables: from sim to real

  • The hand-tuned PI controller reaches setpoints faster than the linear network controller and avoids its failure to reach larger horizontal targets.Both controllers overshoot at the fastest downward setpoint, where the drone has little time to correct before reaching the ground.Source: Control through visual observables:… · p. 7

Disco, darkness, squares and frisbees: examples of versatility and robustness

  • Landing remains successful under flickering and dim lighting, but state estimates diverge too much for landing in the darkest condition with almost no events.Flickering lights increase event counts without causing the same estimation failure.Source: Disco, darkness, squares and frisbees:… · p. 7

Control through visual observables: from sim to real

Disco, darkness, squares and frisbees: examples of versatility and robustness

  • The vision network and PI controller support square flight and straight-line spinning while tracking yaw, although square flight shows yaw drift.Rotating the sideways velocity setpoint by yaw angle produces the frisbee-like trajectory despite some X and Z drift.Source: Disco, darkness, squares and frisbees:… · p. 7, Figure 6 · p. 9

Improved inference speed and energy consumption on neuromorphic hardware

  • Loihi outperforms Jetson Nano by three to four orders of magnitude and improves execution frequency by one to two orders, with faster execution on sparser sequences.The in-memory benchmarks omit I/O and preprocessing; flight tests achieve 200 Inf/s, below the benchmarked Loihi frequencies.Source: Improved inference speed and energy… · pp. 7–9

DISCUSSION AND CONCLUSION

The paper demonstrates a fully neuromorphic vision-to-control pipeline operating on a freely flying drone, with successful sim-to-real flight across varied ego-motion tasks. Its onboard performance is promising, but hardware and bandwidth constraints limit how fully neuromorphic the complete drone can be.

DISCUSSION AND CONCLUSION

  • The drone accurately follows varied ego-motion setpoints in real-world flight, including hovering, landing, and lateral maneuvers while yawing.The experiments demonstrate successful sim-to-real transfer of the learned pipeline.Source: DISCUSSION AND CONCLUSION · p. 9
  • 28.6, 106.9, and 186.6 events/inference are the average inputs for the slow, medium, and fast sequences used in Table 2’s device energy and power comparison.The caption defines delta power as running minus idle power, used to calculate energy per inference.Source: Table 2 · p. 10
  • 200 Hz and 27 µJ per inference are achieved by the onboard vision-to-control pipeline.The paper presents these results as evidence of the pipeline’s potential, while noting hurdles to embedding it on drones under 30 g.Source: DISCUSSION AND CONCLUSION · pp. 9–10
  • The current pipeline ends at thrust and attitude commands; unavailable neuromorphic sensing and actuation and limited processor I/O constrain fuller integration.Spike bandwidth limits also led the authors to use a linear network controller.Source: DISCUSSION AND CONCLUSION · p. 10
  • Neuromorphic sensing and processing may bring deep neural networks within reach of small autonomous robots, potentially approaching the agility, versatility, and robustness of flying insects.The authors frame this as a possibility, despite the remaining limitations.Source: DISCUSSION AND CONCLUSION · p. 10

MATERIALS AND METHODS

The methods build a sequential spiking vision network that estimates planar motion from event-camera corner flows, then connect its outputs to a linear controller trained in simulation. Hardware-aware SNN training and sim-to-real strategies support deployment, while Loihi energy benchmarks measure the network rather than the complete pipeline.

Neuromorphic state estimation: spiking, sequential processing

  • The spiking network uses CUBA-LIF neurons, and quantization-aware training simulates Loihi’s integer-valued states and arithmetic to reduce deployment performance loss.The simulated and on-chip spiking neurons achieved a 100% matching score.Source: Neuromorphic state estimation:… · pp. 10–11

Neuromorphic state estimation: planar homography

  • The vision network predicts four corner displacements, which are used in a four-point parametrization to estimate the homography and derive optical flow.The method concatenates point-correspondence constraints from the four image corners to solve for the homography.Source: Neuromorphic state estimation: planar… · p. 11

Neuromorphic state estimation: self-supervised learning

  • Self-supervised contrast maximization trains the network to compensate for event motion blur, with lower focus loss indicating more accurate optical flow.A training buffer accumulates five successive event-flow tuples, or 25 ms of data, before backpropagation through time and a parameter update.Source: Neuromorphic state estimation:… · p. 11

From a vision-based state estimate to control

  • Four corner flows are converted to scaled velocities and yaw rate for control, assuming a static planar scene, small pitch and roll angles, and derotated optical flow.The method estimates the visual observables by inverting the flow relation across all four corners, then transforms them to the body frame.Source: From a vision-based state estimate to… · p. 12

Training control in simulation

  • A linear controller maps visual observables, drone attitude, and velocity setpoints to thrust and attitude commands, with its weights evolved in modified Flightmare.The mutation-only genetic algorithm uses a population of 100 and evaluates individuals over 16 setpoints, each repeated ten times.Source: Training control in simulation · pp. 12–13

From simulation to the real world

  • Simulation training uses domain randomization over attitude bias and initial conditions, and adds a translational drag model to the drone dynamics.For real-world transfer, the authors also use low-level controllers, calibrate attitude and hover-thrust biases, and smooth and scale visual observables and control outputs.Source: Training control in simulation · p. 13, From simulation to the real world · p. 13

Hardware setup

  • The real-world setup sends event-camera data through an UP Squared computer to Loihi for vision processing, then routes decoded flows and control commands through ROS and PX4.The complete pipeline runs at 200 Hz.Source: Hardware setup · p. 13

Energy benchmark

  • Loihi energy and execution benchmarks measure the network alone, using preloaded data rather than online streaming and excluding I/O and preprocessing.The authors note that these measurements are not representative of the complete drone pipeline.Source: Energy benchmark · pp. 13–14

SUPPLEMENTARY MATERIALS

The supplementary materials describe the simulated quadrotor and controller implementation, real-world control adjustments, and limitations of the linear controller. They also quantify the small but significant effect of tethering on the estimated vertical velocity.

Simulation

  • The simulated drone model combines rotor thrust and torque, motor dynamics, and rigid-body motion, while omitting rotor drag.Individual rotor thrust is modeled with a second-order function of motor speed.Source: Simulation · p. 17
  • The linear controller’s outputs are clamped, scaled into thrust and attitude commands, then passed through cascaded low-level rate and torque controllers.Thrust is compensated for drone attitude to partially address the linear controller’s shortcomings.Source: Simulation · p. 17

Real world

  • Real-world operation manually scales and smooths vision estimates and control outputs, calibrates hover thrust and attitude biases, and integrates yaw-rate commands into yaw angles.Yaw integration uses an approximate 0.005 s control-loop timestep.Source: Real world · p. 17

Linear-controller limitations

  • The linear controller cannot fully represent nonlinear effects of attitude and altitude on velocity, and some drift remains during sloped vertical flight.Horizontal flights therefore start at the same altitude; attitude-based compensation is ad hoc, and differences between simulation and the real drone affect setpoints.Source: Real world · pp. 17–18

Quantification of tether impact

  • The tether has a significant but small effect on estimated vertical velocity; its effects on estimated horizontal velocities are not significant.The comparison uses ten runs with and without a tether and bootstrapped differences of medians.Source: Quantification of tether impact · p. 18

Related papers

Loading 2303.08778v1…