A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones

Daniele Palossi; Antonio Loquercio; Francesco Conti; Eric Flamand; Davide Scaramuzza; Luca Benini

Source-linked AI summary

A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones

Daniele Palossi, Antonio Loquercio, Francesco Conti, Eric Flamand, Davide Scaramuzza, Luca Benini

arXiv:1805.01831v4May 4, 2018cs.ROcs.AIcs.NEeess.SP

TL;DR

Nano-drones have lacked the power and computing resources needed for onboard DNN-based visual navigation. The paper develops a parallel ultra-low-power deployment methodology on GAP8 and demonstrates closed-loop DroNet navigation on a CrazyFlie 2.0, achieving real-time operation at 6 fps and 64 mW, with scaling to 18 fps.

Problem
Complex DNN deployment remains difficult on nano-UAVs because memory and power constraints exceed the capabilities of conventional low-power MCU platforms.
Method
The paper maps and adapts DroNet for parallel execution on a GAP8-based PULP-Shield coupled to the CrazyFlie 2.0, using AutoTiler-managed tiled computation and memory transfers.
Results
6 fps at 64 mW achieves comparable UAV-control quality to the standard-sized baseline, while the system scales to 18 fps at 284 mW.
Takeaways & Limitations
Closed-loop DNN-based visual navigation can run entirely onboard a nano-UAV using a parallel ultra-low-power computing approach.

Abstract

from arXiv · show

Fully-autonomous miniaturized robots (e.g., drones), with artificial intelligence (AI) based visual navigation capabilities are extremely challenging drivers of Internet-of-Things edge intelligence capabilities. Visual navigation based on AI approaches, such as deep neural networks (DNNs) are becoming pervasive for standard-size drones, but are considered out of reach for nanodrones with size of a few cm${}^\mathrm{2}$. In this work, we present the first (to the best of our knowledge) demonstration of a navigation engine for autonomous nano-drones capable of closed-loop end-to-end DNN-based visual navigation. To achieve this goal we developed a complete methodology for parallel execution of complex DNNs directly on-bard of resource-constrained milliwatt-scale nodes. Our system is based on GAP8, a novel parallel ultra-low-power computing platform, and a 27 g commercial, open-source CrazyFlie 2.0 nano-quadrotor. As part of our general methodology we discuss the software mapping techniques that enable the state-of-the-art deep convolutional neural network presented in [1] to be fully executed on-board within a strict 6 fps real-time constraint with no compromise in terms of flight results, while all processing is done with only 64 mW on average. Our navigation engine is flexible and can be used to span a wide performance range: at its peak performance corner it achieves 18 fps while still consuming on average just 3.5% of the power envelope of the deployed nano-aircraft.

I. INTRODUCTION

Nano-UAVs offer mobile IoT sensing and analytics, but their tiny power and computational budgets have kept sophisticated onboard visual navigation out of reach. This work presents a hardware/software methodology and closed-loop deployment of CNN navigation on a resource-constrained nano-UAV.

Nano-UAVs can serve as mobile IoT hubs, smart sensors, and data collectors for surveillance, inspection, and related tasks.
CNN-based navigation is lighter than localization-mapping-planning, yet its computational and power requirements still exceed current nano-drone navigation budgets.
5% of a small UAV’s total power budget is available for onboard computation, with the remainder allocated to propulsion and low-level control.
CNN-based autonomous vision navigation was considered out of reach for nano-drones constrained to low-power single-core microcontrollers.
The proposed methodology deploys and adapts DroNet on a GAP8-based PULP-Shield attached to a 27 g CrazyFlie 2.0 nano-UAV.
6 fps at 64 mW provides comparable UAV-control quality to the standard-sized baseline, while performance scales to 18 fps at 284 mW.
The system demonstrates closed-loop, fully onboard vision-driven autonomous navigation and releases its hardware, code, datasets, and trained networks.

II. RELATED WORK

Prior nano-UAV systems generally relied on offboard computation or limited onboard sensing and guidance, while complex DNN deployment remained difficult on memory- and throughput-constrained MCUs. The paper addresses this gap with an onboard, ultra-low-power CNN navigation engine.

Edge-computing trends motivate autonomous nano-UAVs as mobile IoT end-nodes requiring substantial onboard intelligence.
Earlier nano-UAV navigation included offboard visual-inertial SLAM, onboard velocity estimation requiring low flight speeds, and optical-flow-based guidance.
NAVION achieved visual-inertial odometry at 171 fps and 24 mW, but addressed only one UAV functionality and had not been demonstrated on a real flying nano-drone.
Complex DNN deployment on low-power MCUs remains difficult because of scarce working memory, limited throughput, and missing software abstractions and libraries.
The proposed engine performs real-time CNN-based autonomous visual navigation entirely onboard within an approximately 0.2 W nano-UAV power envelope.

III. BACKGROUND

DroNet is a lightweight residual CNN designed to support autonomous quadrotor flight by jointly predicting steering and collision probability. Its shared visual processing is converted directly into velocity and yaw commands.

A. DroNet: DroNet is a lightweight residual CNN that predicts steering angle and collision probability for autonomous quadrotor flight.
A. DroNet: The topology shares residual layers between steering and collision prediction, then uses separate fully connected layers for the two outputs.
A. DroNet: Mean-squared error trains steering prediction, while binary cross-entropy trains collision-probability prediction.
A. DroNet: Low-pass filtered collision probability modulates forward velocity, while filtered steering angle is converted into yaw control.
A. DroNet: The shallow network processes visual input end-to-end and directly produces flight-control commands without requiring an environment map.

B. GAP8 Architecture

GAP8 combines a RISC-V control domain with an eight-core cluster to run DroNet under tight memory, real-time, and power constraints. The deployment modifies and retrains the network for fixed-point execution and the nano-drone camera while preserving navigation quality.

Architecture: GAP8 combines a RISC-V microcontroller with an octa-core accelerator for embedded deep inference.The processor uses separate fabric-control and cluster power and clock domains.
Deployment constraints: 512 kB of L2 SRAM and 64 kB of shared L1 scratchpad constrain the operating set and require ad-hoc memory management.These limits motivate reducing memory footprint and computational load while exploiting parallelism.
Network adaptation: 32-bit floating-point weights and activations were reduced to 16-bit Fixed16 Q4.12 representation for processors without floating-point units.The representation uses 4 integer bits, including sign, and 12 fractional bits.
Network adaptation: The collision dataset added approximately 1300 HiMax-camera images, including 1122 for training and 228 for test or validation.The extension addresses differences between the original high-resolution color camera and the nano-drone’s low-resolution grayscale camera.
Evaluation: Fine-tuning improved paired configurations by up to 8% in accuracy, while the final DroNet configuration was selected for GAP8 deployment.The reported evaluation covers steering-angle regression and collision-probability classification on Original and HiMax datasets.

B. AutoTiler

AutoTiler maps CNN computation onto GAP8 by partitioning tensors into tiles that fit its shared L1 memory. It explores tiling choices and generates pipelined code for computation and memory transfers.

Tiling methodology: Tiling partitions CNN input and output data into portions that fit within GAP8’s shared L1 memory.Each iteration loads an input tile, produces an output tile, and stores the result in higher memory.
Automation: AutoTiler relieves developers from manual coding of tiling loops and data movement mechanisms.This targets tedious and error-prone implementation work in resource-constrained CNN deployment.
Automation: AutoTiler explores tiling configurations and generates C wrapper code for pipelined computation and double-buffered L2-to-L1 transfers.The tool uses the cluster DMA controller and represents computation through basic and node kernels.
Execution schemes: The generated schemes support spatial and feature-wise convolution execution through tiled DMA transfers and basic convolution, bias, and ReLU kernels.The examples iterate over spatial or feature dimensions while moving tiles between L2 and L1.

C. Tiling, Parallelization & Optimization

DroNet is mapped onto GAP8 through tiled tensor processing, scheme-specific parallelization, and specialized signal-processing instructions. The mapping assigns spatial parallelization to the first node kernel and feature-wise parallelization to deeper layers after performance exploration.

Tiling and parallelization: The AutoTiler tiles inputs along Hin and Kin and outputs along Hout and Kout, with partial overlap along Hin for convolutional receptive fields.Node kernels execute using spatial or feature-wise schemes.
Tiling and parallelization: Parallelization schemes allocate DroNet-layer work across GAP8 cores.Figure 5 visualizes core allocation using different colors.
Tiling and parallelization: The first convolution-plus-max-pooling kernel uses spatial parallelization, whereas subsequent kernels use feature-wise parallelization.This choice followed an exploration of parallelization efficiency because deeper feature maps shrink rapidly.
Tiling and parallelization: Feature-wise execution splits work across cores along Kout when the full feature map has one Hout tile.This scheme is applied only to full feature maps.
Instruction optimization: GAP8 signal-processing extensions provide packed-SIMD, bit-manipulation, and shuffling operations for optimized DroNet execution.The implementation accesses these operations through compiler intrinsics.

D. L2 Memory Management Strategy

DroNet requires an ad-hoc L2 memory strategy because residual bypasses extend tensor lifetimes while available memory is insufficient for all weights and feature maps. The design uses two allocation stacks and staged transfers between L3 and L2.

Memory constraints: More than 1 MB would be required for weights and temporary feature maps, exceeding the available 512 kB L2 memory.Residual bypasses also require some output tensors to remain alive longer.
Allocation strategy: Two L2 allocation stacks reserve and release intermediate buffers linearly to manage memory before computation begins.The runtime allocator creates the stacks at boot time.
Execution flow: The memory sequence includes convolution and max-pooling operations with explicit allocation entries for intermediate outputs and weights.The listed operations use stride and filter or pool-size parameters.
Allocation strategy: Weights are allocated immediately before their layers and deallocated afterward, while residual feature maps often remain alive through two following node kernels.Each RES block can require three tensors to coexist.
Execution flow: The execution flow combines node kernels with L3/L2 memory-management blocks and uses allocation-buffer identifiers for the two stacks.The µDMA transfers weights from L3 to L2 during execution.

V. THE PULP-SHIELD

The PULP-Shield integrates GAP8, external memories, and a low-power camera in a drone-compatible board. It acts as a host-accelerator interface that offloads visual navigation while limiting communication overhead and supporting broader IoT-node use.

Hardware platform: The PULP-Shield combines a GAP8 SoC, two HyperBus memories, and a HiMax grayscale camera in a lightweight modular PCB.The camera supports up to 60 fps at 320 × 240 resolution using 4.5 mW.
Hardware platform: The board uses HyperRAM for L3 DRAM and HyperFlash for external storage, with configurable memory-package options.The mounted configuration includes 64 Mbit HyperRAM and 128 Mbit HyperFlash.
Drone integration: The 30×28 mm board weighs approximately 5 g, including components, remaining below the drone payload limit.It connects mechanically and electrically through headers, GPIOs, and SPI.
Host-accelerator operation: The accelerator protocol begins with a host GPIO wake-up, followed by kernel loading, camera configuration, frame transfer, and loading of CNN data.The host can select a kernel or sequence of pre-loaded kernels.
System architecture: The PULP-Shield architecture minimizes data transfers and communication overhead between the main MCU and accelerator through data locality.Its functionality is presented as portable to SPI-equipped drones and generic IoT nodes requiring visual processing.

VI. EXPERIMENTAL RESULTS

The evaluation measures whether the visual navigation engine meets real-time, power, and closed-loop control requirements. These metrics assess deployment feasibility from timing, energy, and flight-response perspectives.

Evaluation criteria: The evaluation uses real-time deadline compliance, power-budget compliance, and closed-loop accuracy measured by reaction time to an unexpected obstacle.All results use the PULP-Shield configuration described for the system.

A. Performance & Power Consumption

GAP8 executes DroNet across configurable operating points, reaching real-time performance within the nano-drone’s power envelope. The most energy-efficient configuration consumes 64 mW, while added payload weight reduces flight lifetime.

Power consumption: 47 mW is the peak GAP8 power associated with the first convolutional layer, while the two final fully connected layers consume 13 mW each.The weighted average is 39 mW for the layers and 45 mW after internal DC/DC losses.
Power consumption: 64 mW is the full-board average power for DroNet at the most energy-efficient configuration, including L3 memory access and the onboard camera.The configuration uses 1.0 V, 50 MHz FC, and 100 MHz CL; CL cores are clock-gated during approximately 7% of execution time.
System power envelope: 284 mW is the peak PULP-Shield power increase in the fastest setting, compared with 64 mW in the most energy-efficient setting.These increases represent 3.5% and 0.8% of the nano-aircraft’s total power envelope, respectively.
System power envelope: Onboard computation accounts for roughly 5% of overall system power, allowing DroNet execution across all tested configurations within the power envelope.The visual-navigation module includes the computational unit, external DRAM, and ultra-low-power camera.
Flight lifetime: Approximately 22% of the original flight lifetime is lost when the drone carries the PULP-Shield and executes DroNet.Lifetime decreases from approximately 440 s for the original drone to approximately 340 s during DroNet execution.

B. State-of-the-Art Comparison & Discussion

The parallel GAP8 implementation compares favorably with sequential microcontroller processing in throughput and energy efficiency. Quantized DroNet also preserves obstacle-avoidance behavior sufficiently for closed-loop flight at the tested speeds and rates.

Execution analysis: 13.47 Mcycles are measured per frame versus approximately 10 Mcycles in the ideal peak-throughput case for the original CNN.Approximately 1.14 Mcycles arise from non-overlapped memory transfers; additional overhead comes from non-ideal parallel efficiency and other effects described in the execution breakdown.
State-of-the-art comparison: 281 MMAC/s at 45 mW matches the STM32H717’s 276 MMAC/s at 346 mW in the most power-efficient configuration.This comparison uses average performance for the proposed system and peak performance for the microcontroller.
Control accuracy: The experiment uses a 20 m path at 4 m/s, with an obstacle appearing at T = 4 s and 4 m available for braking and stopping.The analytical model derives a minimum stopping time of 400 ms and braking distance of 0.7 m.
Control accuracy: Quantized DroNet outperforms the original implementation in collision detection and reacts more quickly to sudden obstacles, including at low operational frequencies.The authors attribute this mainly to fine-tuning for HiMax camera images.
Control accuracy: At 10 Hz, inference allows the drone to brake in time and avoid collision, while both architectures fail at 5 Hz.Increasing frequency does not always improve performance: the quantized system performs better at 20 Hz than at 25 Hz in one comparison.

VII. CONCLUSION

The paper presents a vertically integrated, fully onboard visual-navigation engine for nano-UAVs using GAP8 and a CrazyFlie 2.0. It demonstrates real-time DroNet execution from 6 fps at 64 mW up to 18 fps at 284 mW without loss of flight-control quality relative to the baseline.

Conclusion: The system achieves these operating points without quality-of-results loss relative to the standard-sized UAV baseline running DroNet at 20 fps on a remote PC.The reported flight results include obstacle detection fast enough for the CrazyFlie 2.0 to fly at 4 m/s.