PaperlayerView exact revision on arXiv

Source-linked AI summary

Navion: A 2mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones

Amr Suleiman, Zhengdong Zhang, Luca Carlone, Sertac Karaman, Vivienne Sze

arXiv:1809.05780v1cs.RO

TL;DR

Navion targets the difficulty of running real-time VIO within the severe power and form-factor constraints of miniature drones. It fully integrates the VIO pipeline on chip using algorithmic and architectural optimizations, achieving real-time operation at low power while retaining configurable performance trade-offs.

  • Problem

    Real-time VIO requires powerful CPUs or GPUs, but their power and form-factor demands exceed the constraints of nano and pico drones.

  • Method

    Navion fully integrates VIO on chip and combines memory hierarchy, compression, sparsity exploitation, rescheduling, parallelism, and configurable operation.

  • Results

    At 20 fps on EuRoC stereo images, Navion consumes 2mW average power; at peak, it reaches 171 fps and 52 kHz while consuming 24mW average power.

  • Takeaways & Limitations

    Navion provides a configurable, fully integrated VIO accelerator for real-time autonomous navigation, mapping, and portable AR/VR applications.

Abstract

from arXiv · show

This paper presents Navion, an energy-efficient accelerator for visual-inertial odometry (VIO) that enables autonomous navigation of miniaturized robots (e.g., nano drones), and virtual/augmented reality on portable devices. The chip uses inertial measurements and mono/stereo images to estimate the drone's trajectory and a 3D map of the environment. This estimate is obtained by running a state-of-the-art VIO algorithm based on non-linear factor graph optimization, which requires large irregularly structured memories and heterogeneous computation flow. To reduce the energy consumption and footprint, the entire VIO system is fully integrated on chip to eliminate costly off-chip processing and storage. This work uses compression and exploits both structured and unstructured sparsity to reduce on-chip memory size by 4.1$\times$. Parallelism is used under tight area constraints to increase throughput by 43%. The chip is fabricated in 65nm CMOS, and can process 752$\times$480 stereo images from EuRoC dataset in real-time at 20 frames per second (fps) consuming only an average power of 2mW. At its peak performance, Navion can process stereo images at up to 171 fps and inertial measurements at up to 52 kHz, while consuming an average of 24mW. The chip is configurable to maximize accuracy, throughput and energy-efficiency trade-offs and to adapt to different environments. To the best of our knowledge, this is the first fully integrated VIO system in an ASIC.

I. INTRODUCTION

Navion addresses the power and form-factor limits that prevent conventional real-time VIO implementations on nano and pico drones. It integrates the VIO pipeline on chip and combines co-design, memory optimization, parallelism, and adaptability for efficient operation.

  • Around 100mW is the stable-flight power budget for nano/pico drones, roughly an order of magnitude below embedded CPU power dissipation.
  • A co-design strategy links hardware and algorithmic choices to improve efficiency in the ASIC implementation.The work builds on an FPGA prototype that demonstrated benefits from co-design.
  • Navion integrates a real-time VIO implementation on chip for autonomous navigation in miniaturized robots and UAVs.The paper describes this as the first fully integrated VIO system in an ASIC.
  • 9× lower external DRAM bandwidth, 4.4× smaller memory from compression, 5.2× and 5.4× smaller memories from fixed and dynamic sparsity, and 43% faster processing result from architectural optimizations.The optimizations include an efficient memory hierarchy, data movement, compression, sparsity exploitation, rescheduling, and parallelism.
  • Adaptability improves accuracy and throughput across environments and camera movements, with sequence-specific adaptation adding 2.5× energy-efficiency improvement.

II. OVERVIEW OF VISUAL-INERTIAL ODOMETRY (VIO)

Navion’s VIO pipeline combines camera and IMU processing to estimate motion and reconstruct a sparse map. Its frontend extracts and tracks visual features, while the backend fuses them with inertial measurements through nonlinear optimization.

  • VIO estimates a sensing device’s trajectory while reconstructing a map of its environment from sensor measurements.The trajectory consists of the device’s position and orientation over time.
  • The pipeline contains a visual frontend, an IMU frontend, and a backend that performs state estimation.
  • The visual frontend extracts 2D features corresponding to 3D landmarks and tracks them across camera frames.It supports mono and stereo modes and includes feature tracking, detection, undistortion and rectification, stereo matching, and geometric verification.
  • The IMU frontend summarizes inertial data between camera keyframes into one measurement.
  • The backend fuses feature tracks and summarized IMU data through nonlinear graph optimization to output the drone’s position, orientation, and sparse 3D landmark map.It optimizes multiple states within a sliding time horizon using a fixed-lag smoother.

B. IMU Frontend (IFE)

The IMU frontend summarizes inertial measurements between consecutive keyframes into preintegrated motion measurements for backend state estimation. These measurements contribute to factor-graph optimization over a dynamically maintained horizon.

  • The IFE preintegrates all IMU measurements between consecutive keyframes into one measurement.The output includes relative rotation, velocity, and position between keyframes.
  • The backend fuses IFE measurements with visual feature tracks to estimate keyframe states and a sparse 3D map.
  • The backend represents relationships among horizon states with a continuously updated factor graph.Marginalization removes states and factors outside the current time window.
  • BE solves the linearized system H∆x=ε using Cholesky factorization and back-substitution, then applies the state update to the remaining horizon states.
  • The backend supports up to 20 keyframes and 4000 vision factors, creating substantial on-chip storage and maintenance demands.

D. Design Considerations

Navion exposes design parameters that trade memory, throughput, and accuracy, while selecting a nonlinear factor-graph optimizer and stereo or mono sensing for different application needs.

  • Parameters: The horizon size controls backend complexity: reducing it lowers memory and increases throughput but hurts trajectory-estimation accuracy.The linear-system size is O(N^2), solving costs O(N^3), and Navion uses a horizon size of 20.
  • Parameters: Navion tracks up to 200 features per frame, affecting major VFE components linearly in power and throughput.
  • Parameters: A feature track age of 10 scales backend processing linearly, while larger ages are generally preferred for higher accuracy.
  • Optimization Method: The nonlinear factor-graph optimizer is preferred over EKF because EKF can have 3× higher average motion-estimation error on EuRoC.
  • Stereo vs Mono: Navion supports mono and stereo modes: stereo avoids scale ambiguity but consumes more power than mono.

III. NAVION: ARCHITECTURE

Navion integrates the VIO frontend, IMU frontend, and backend into a standalone accelerator with no external storage or computation. Its architecture streams sensor data on chip while using parallel image processing and reduced external bandwidth.

  • Navion accepts mono or stereo images and inertial measurements, then outputs the trajectory and sparse 3D map without external computation or storage.
  • More than 250 programmable parameters configure sensor calibration, pipeline complexity, and accuracy-throughput trade-offs.
  • IMU Frontend: The IFE processes streamed inertial measurements using double-precision arithmetic and stores intermediate and preintegrated data in local registers and SRAM.
  • Vision Frontend: 9× lower external bandwidth reaches 0.34 MB/frame by storing input pixels in on-chip line or frame buffers.
  • Vision Frontend: Parallel image processing handles feature tracking, detection, undistortion, and stereo matching, while geometric validation uses serial FSM-based RANSAC.
  • Vision Frontend: The VFE detects 1824 features per frame and tracks up to 200 features per frame.

C. Backend

Navion’s backend uses hierarchical control, local storage, and keyframe-dependent operating modes to manage factor-graph computation efficiently. Double precision preserves numerical robustness, while non-keyframe processing gates inactive hardware to reduce power.

  • Hierarchical FSMs divide backend matrix operations, Cholesky factorization, and back-substitution into smaller reusable control units.
  • The backend stores horizon factors and states in shared registers and SRAM, and outputs the trajectory and sparse 3D map at the keyframe rate.
  • Double precision in BE and IFE preserves numerical precision for the open-loop VIO pipeline but doubles their memory size.
  • Processing Modes: During keyframe processing, all components operate and BE updates the trajectory and sparse map after factor-graph optimization.
  • Processing Modes: During non-keyframe processing, only feature tracking remains active while other VFE and BE components are clock-gated and right frames are not streamed.
  • 3.8× faster non-keyframe processing reduces work between keyframes, while compression, rescheduling, and parallelism target memory and throughput.

A. Image Compression

Navion compresses image frames and reorganizes feature-track storage to reduce on-chip memory while controlling VIO accuracy loss. The design combines lossy quantization with a two-stage memory for variable-length tracks.

  • Image Compression: 5-bit LSB truncation increases VIO error by 6% while reducing memory size by 38%.At 1 bit per pixel, the error exceeds twice the 8-bit baseline.
  • Image Compression: 4×4 block quantization provides lower error than 3-bit truncation at 1-bit-per-pixel representation.The method quantizes each block’s dynamic range into two levels, with line-buffer and range-computation overhead.
  • Image Compression: Navion combines 5-bit truncation with 4×4 block quantization to encode each block using 26 bits.Compressed frames are used in stereo matching and feature tracking, but not feature detection because of blocking and quantization sensitivity.
  • Image Compression: 4.4× frame-memory reduction and 4.9× power reduction are achieved with 4 kgates of logic and a 1.4kB line buffer.The compression hardware accounts for 0.8% of total gate count.
  • Feature Tracks Unstructured Sparsity: Feature tracks occupy 88% of backend factor-graph memory and vary in length with the image sequence.A worst-case single memory must accommodate up to 20 keyframes, 200 features per frame, and 10 observations per landmark.
  • Feature Tracks Unstructured Sparsity: A pointer-based two-stage memory stores sparse entries separately from dense 3D observations, reducing graph memory size by 5.4×.The architecture incurs only one additional access-latency cycle.

C. Linear Solver Structured Sparsity

Navion exploits fixed structure in the backend Hessian matrix and reschedules frontend and backend work to reduce memory and processing time. These optimizations increase throughput without changing overall system accuracy.

  • Linear Solver Structured Sparsity: A 300×300 Hessian matrix represents up to 20 keyframes, with 15 state variables per keyframe.The backend linearizes factors into H∆x = ε and updates H at every keyframe.
  • Linear Solver Structured Sparsity: Storing only the symmetric upper or lower triangle and its fixed nonzero entries reduces linear-solver memory size by 5.2×.Only 38% of each triangle contains nonzero values under the described horizon and feature-track configuration.
  • Linear Solver Structured Sparsity: 7.2× linear-solver speed-up is achieved at a 20-keyframe horizon by skipping processing of zero locations.The resulting backend processing-time speed-up reaches 2.5× at the same horizon size.
  • Pipeline Rescheduling and Parallelism: VFE rescheduling saves 43% to 55% of processing time depending on the environment.The parallel design adds 77kB of line-buffer memory, or 10% overhead, without affecting overall accuracy.
  • Pipeline Rescheduling and Parallelism: BE rescheduling saves 4% to 19% of processing time depending on the scene, with no overhead.Initialization and IMU-factor linearization begin after IFE output becomes available, overlapping backend and frontend work.

V. IMPLEMENTATION AND RESULTS

Navion integrates the VIO pipeline in a 65nm chip using on-chip SRAM, compression, sparsity exploitation, and clock-gated memory banks. It processes EuRoC stereo inputs in real time across configurable performance settings.

  • Chip Implementation: 65nm CMOS implementation uses 2 million NAND2-equivalent gates and 854kB of on-chip SRAM.The chip has separate clock domains for VFE and for IFE plus BE.
  • Memory Optimization: 4.1× overall memory saving enables full VIO pipeline integration on-chip.The saving combines image compression with structured and unstructured sparsity exploitation.
  • Throughput and Power: 28–171 fps stereo tracking is supported at up to 752×480 resolution across EuRoC sequences.The chip also processes inertial measurements at up to 52kHz and updates backend states at 16–90 fps.
  • Throughput and Power: 24mW average power is consumed at 1V when programmable parameters are set to their maximum values.The reported throughput and power correspond to the evaluated maximum-configuration setting.
  • Area and Power: On-chip SRAM occupies 77% of chip area but consumes only 21% of total power.Banking and clock-gating inactive memories provide 5–9× power savings depending on memory size.
  • Timing Breakdown: Backend linearization consumes 65% of backend processing time, while the sparse linear solver consumes 25%.Frontend modules run in parallel at keyframes, and IMU-factor linearization takes less than 2ms.

B. Demonstration System

Navion is validated through a hardware demonstration system and EuRoC evaluation against desktop and embedded CPU implementations. The chip maintains low trajectory error while substantially reducing energy consumption.

  • Demonstration System: A ZC-706 FPGA board streams images and IMU measurements to Navion and reads outputs for verification and visualization.The demonstration system combines the custom test-chip board with an FPGA-based host and control interface.
  • Evaluation Setup: EuRoC evaluation covers 11 indoor UAV sequences across Machine Hall and two Vicon rooms.The dataset includes fast, unstable motion and brightness changes that produce dark or blurred images.
  • Accuracy: Navion’s normalized trajectory error increases from 0.22% to 0.28%, a 0.06-percentage-point increase over an average 83m flight.The comparison uses identical configuration parameters and averages five runs per sequence.
  • Energy Efficiency: Three orders of magnitude less energy consumption is achieved than with both Xeon and ARM CPU implementations.The reported error increase is mainly attributed to lossy image compression and fixed-point arithmetic in VFE.
  • Accuracy Across Sequences: Low trajectory error occurs on easy sequences, while error increases on blurry, dark, and fast-moving sequences.All 11 sequences use maximum VIO parameter settings for both Navion and software implementations.

D. Adapting to the environment

Navion adapts programmable parameters to each environment and movement, trading throughput, accuracy, and power. This adaptation preserves a 0.35% trajectory-error target while reducing energy consumption across EuRoC sequences.

  • Targeted adaptation: A 0.35% normalized trajectory-error target is used to select each sequence’s optimum configuration independently.The adapted parameters include feature count and horizon size.
  • Targeted adaptation: Easy sequences use fewer features and smaller horizons because their features are easier to track, reducing workload.
  • Accuracy: Although average trajectory error remains unchanged, adaptation slightly increases easy-sequence errors while significantly decreasing errors on difficult sequences.Easy sequences include MH 1, MH 2, V1 1, and V2 1; difficult sequences include MH 4, MH 5, and V2 3.
  • Energy efficiency: 2.5× lower energy consumption is achieved on average across sequences with adaptation.
  • Energy efficiency: At the EuRoC capture rate of 20 fps, adapted Navion consumes an average power of 2mW.The experiment uses a 1V supply and adapts throughput for each sequence.

VI. CONCLUSIONS

Navion is a fully integrated ASIC VIO accelerator that combines compression, specialized memory organization, sparsity exploitation, and programmable operation for miniature-robot localization and mapping. Fabricated in 65nm CMOS, it supports real-time operation while offering configurable throughput, accuracy, and energy trade-offs.

  • Contribution: Hardware–algorithm co-design is presented as necessary for scaling localization and mapping to nano and pico drones or robots.
  • Architecture: On-chip image compression and specialized feature-track memory reduce storage demands, while structured and unstructured sparsity increase memory efficiency and throughput.
  • Configurability: Programmable keyframe rate, horizon size, and feature-track count let Navion prioritize throughput, accuracy, or energy efficiency across scenarios.
  • Results: 171 fps stereo processing and 52 kHz inertial processing are reached at peak performance with 24mW average power at 1V.
  • Results: At the EuRoC sensor rate of 20 fps, Navion processes stereo images using only 2mW average power at 1V.
Loading 1809.05780v1…