PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of FPGA-Based Robotic Computing

Zishen Wan, Bo Yu, Thomas Yuang Li, Jie Tang, Yuhao Zhu, Yu Wang, Arijit Raychowdhury, Shaoshan Liu

arXiv:2009.06034v3cs.ROcs.AIcs.AR

TL;DR

Robotic computing must meet real-time demands despite high data and computation complexity and constrained onboard resources. This paper surveys FPGA-based accelerators, their optimization techniques, and commercial and space applications. It reports that FPGA co-design can achieve more than 10× better performance and energy efficiency than CPU and GPU implementations, while identifying gaps in HLS, IP, and software support.

  • Problem

    Robotic systems must process large, heterogeneous sensor data in real time despite limited onboard memory, bandwidth, compute, and power.

    Source: §I · p. 1

  • Method

    The paper surveys FPGA robotic accelerators, software and hardware optimizations, partial reconfiguration, and commercial and space applications.

    Source: §IX · p. 17

  • Results

    Co-designing software and hardware on FPGAs achieves more than 10× better performance and energy efficiency than CPU and GPU implementations, according to the survey.

    Source: §IX · p. 17

  • Takeaways & Limitations

    The survey concludes that FPGA is a promising candidate for robotic computing acceleration because of its reliability, adaptability, and power efficiency.

    Source: §IX · p. 17

    Robotic FPGA support remains limited: HLS does not yet produce optimized code, workload IP support is extremely limited, and commercial platforms lack official ROS support.

    Source: §IX · p. 17

Abstract

from arXiv · show

Recent researches on robotics have shown significant improvement, spanning from algorithms, mechanics to hardware architectures. Robotics, including manipulators, legged robots, drones, and autonomous vehicles, are now widely applied in diverse scenarios. However, the high computation and data complexity of robotic algorithms pose great challenges to its applications. On the one hand, CPU platform is flexible to handle multiple robotic tasks. GPU platform has higher computational capacities and easy-touse development frameworks, so they have been widely adopted in several applications. On the other hand, FPGA-based robotic accelerators are becoming increasingly competitive alternatives, especially in latency-critical and power-limited scenarios. With specialized designed hardware logic and algorithm kernels, FPGA-based accelerators can surpass CPU and GPU in performance and energy efficiency. In this paper, we give an overview of previous work on FPGA-based robotic accelerators covering different stages of the robotic system pipeline. An analysis of software and hardware optimization techniques and main technical issues is presented, along with some commercial and space applications, to serve as a guide for future work.

I. INTRODUCTION

Robotic systems face high data and computation demands alongside real-time, onboard-resource, and power constraints. This survey reviews FPGA acceleration across the robotic pipeline, including optimization techniques and commercial and space applications.

I. INTRODUCTION

  • Robotic systems must process large, heterogeneous sensor data in real time despite limited onboard memory, bandwidth, compute, and power.These constraints hinder deployment in latency-critical or power-limited scenarios.Source: §I · p. 1
  • CPUs offer flexible task handling and GPUs offer massive parallelism, but both typically consume 10W to 100W.The paper notes that GPU adoption has also benefited from CUDA/OpenCL accessibility, while this power range exceeds resource-limited robotic-system budgets by orders of magnitude.Source: §I · p. 1
  • FPGAs offer low-power, massively parallel implementations that can exploit hardware-friendly perception, localization, and planning kernels.The survey presents them as possible platforms for meeting real-time requirements with high energy efficiency compared with CPUs and GPUs.Source: §I · pp. 1–2
  • Partial Reconfiguration lets part of an operating FPGA be reconfigured at runtime, enabling robotic applications to time-share FPGA resources.The passage associates this flexibility with energy and performance efficiency for dynamic, complex workloads.Source: §I · p. 2
  • The paper reviews FPGA computing systems used in commercial autonomous vehicles and discusses space-grade FPGAs for robotics.The commercial vehicle systems use FPGAs for heterogeneous sensor synchronization and acceleration of software components on the critical path; space use is associated with flexibility and reduced development cycle and cost.Source: §I · p. 2
  • The paper is organized around robotic workloads, FPGA implementations for perception, localization, and motion planning, partial reconfiguration, and commercial and space applications.It concludes with a synthesis of the survey.Source: §I · p. 2

A. Overview

The robotic-system stack has application workloads, a software edge subsystem, and computing hardware. Its application algorithms turn raw sensor data into information for understanding the environment and choosing actions.

A. Overview

  • The robotic-system stack comprises application workloads, a software edge subsystem, and computing hardware.The software subsystem includes an operating system and runtime; the hardware includes microcontrollers and companion computers.Source: A. Overview · p. 2
  • Application workloads include sensing, perception, localization, motion planning, and control.Source: A. Overview · p. 2
  • Application algorithms extract meaningful information from raw sensor data to understand the environment and dynamically decide actions.Source: A. Overview · p. 2

B. Sensing

Sensor choices and rates vary with the workload and onboard computing capability. The section describes cameras, GNSS/IMU, LiDAR, radar, and sonar and their differing roles and trade-offs.

B. Sensing

  • Sensor number and type depend on workload specifications and onboard compute capability.Source: B. Sensing · p. 2
  • Autonomous vehicles may use eight or more 1080p cameras at 60 Hz, producing multiple gigabytes of raw data per second.Cameras also support object recognition and tracking, including lane detection and obstacle detection.Source: B. Sensing · p. 2
  • GNSS provides fairly accurate localization but runs at 10Hz, while IMU sensors run at 100-200 Hz but can suffer bias drift and noise.Combining GNSS and IMU can provide accurate, real-time updates.Source: B. Sensing · p. 2
  • LiDAR supplies three-dimensional distance information for localization and obstacle detection, while sensor choice affects algorithm and hardware design.For low-speed autonomous driving, PerceptIn data indicated LiDAR processing was slower than camera-based vision processing, with greater power consumption and cost.Source: B. Sensing · p. 2
  • Radar and sonar are cheaper and smaller than LiDAR, and their data can go directly to the control processor for urgent actions such as braking or swerving.Source: B. Sensing · pp. 2–3

C. Perception

Perception turns sensor data into a representation of the environment and handles detection, segmentation, and tracking. The section describes conventional formulations and the use of deep neural networks in recent perception algorithms.

C. Perception

  • The perception layer uses computer vision, including deep learning, to represent static and dynamic objects in the robot’s environment.Source: C. Perception · p. 3
  • Perception handles object detection, segmentation, and tracking through distinct formulations.Detection pipelines combine preprocessing, region-of-interest detection, and classification; segmentation can be graph labeling with CRF inference, while tracking can use Bayesian filtering or tracking-by-detection with an MDP.Source: C. Perception · p. 3
  • Recent robot-perception methods use convolution-based deep neural networks, including object detectors such as Fast R-CNN, Faster R-CNN, SSD, YOLO, and YOLO9000.The passage also identifies FCN and related networks for semantic segmentation.Source: C. Perception · p. 3
  • A stacked autoencoder can learn generic image features offline from auxiliary natural images for online object tracking.Source: C. Perception · p. 3

D. Localization

Localization combines sensor measurements to estimate a robot’s position in its environment. The surveyed approaches use GNSS/IMU, LiDAR with HD maps, cameras, and sensor fusion.

D. Localization

  • GNSS provides accurate but slow updates, while IMUs provide faster, less accurate rotation and acceleration estimates; Kalman filtering can combine them to reduce localization error and latency.GNSS signals can bounce off obstacles, become noisier, or fail in closed environments.Source: D. Localization · p. 3
  • LiDAR supplies environmental point clouds that can be fused with HD maps, after which particle filtering can localize the robot in real time.LiDAR performance may be impaired by rain or snow, introducing localization error.Source: D. Localization · p. 3
  • Vision-based localization derives depth from stereo disparity, estimates motion by matching features across frames, and compares features with a known map to infer position.The described pipeline cites.Source: D. Localization · p. 3
  • Sensor fusion combines multiple sensors and is used to improve localization reliability and robustness,.Source: D. Localization · p. 3

E. Planning and Control

Planning and control turn a robot’s origin and destination into trajectory plans and control commands, with prediction and routing providing inputs. The section covers behavioral decisions, motion planning, feedback control, and learning-based alternatives.

E. Planning and Control

  • The planning and control layer generates trajectory plans and control commands, while prediction and routing outputs feed downstream modules.Prediction estimates future behavior of objects identified by perception, and routing can provide lane-level paths based on HD-map segmentation.Source: E. Planning and Control · p. 3
  • Behavioral decisions use input data to support effective and safe choices, with MDP and POMDP among the widely applied Bayesian models,.Source: E. Planning and Control · p. 4
  • Motion planning generates a trajectory for execution, using grid-based or geometric methods for low-dimensional problems and sampling-based methods such as RRT or PRM for high-dimensional problems –.Reward-based MDP algorithms can also generate paths by maximizing cumulative future rewards.Source: E. Planning and Control · p. 4
  • Feedback control tracks the difference between actual pose and the predefined trajectory through continuous feedback, with PID described as the typical widely used algorithm.Source: E. Planning and Control · p. 4
  • Learning-based approaches, including reinforcement learning, are increasingly popular and can address behavioral decisions, trajectory outputs, or direct feedback-control signals [39]–.Named algorithms include Q-learning, Actor-Critic learning, and policy gradient,,.Source: E. Planning and Control · p. 4

A. Overview

The overview focuses on FPGA implementations of stereo vision as a key robotic-perception component. Stereo matching is its time-consuming bottleneck, and local and global methods trade computational cost against accuracy and resource demands.

A. Overview

  • The section surveys recent stereo-vision algorithms and FPGA implementations, identifying stereo vision as a key component of robotic perception.Stereo vision is used in applications including navigation, obstacle avoidance, and scene reconstruction –.Source: A. Overview · p. 4
  • Stereo vision obtains 3D scene structure from two camera views by searching disparities between corresponding pixels and calculating depth from inverse disparity.Source: A. Overview · p. 4
  • Stereo matching is the time-consuming bottleneck in the stereo-vision pipeline, and local methods are parallel-friendly but can misestimate disparities in textureless or occluded regions.Local methods process pixels around points of interest within windows, whereas global methods match all other pixels and minimize a global cost function.Source: A. Overview · p. 4
  • Global methods can achieve higher accuracy than local methods, but their computational cost, irregular memory access, and sequential nature make them unsuitable for real-time, low-power applications.Source: A. Overview · p. 4

B. Local Stereo Matching on FPGA

The section surveys FPGA implementations of local stereo matching, including census-based, cross-based, block-matching, and dense-stereo approaches with reported real-time or high-frame-rate performance.

B. Local Stereo Matching on FPGA

C. Global Stereo Matching on FPGA

Global stereo matching can provide state-of-the-art accuracy and disparity-map quality but is computationally intensive. FPGA implementations report real-time processing rates, and the comparison table tracks speed, image size, disparity levels, power, resources, and hardware platforms.

C. Global Stereo Matching on FPGA

  • Park et al. presented a trellis-based stereo-matching system on FPGA, one effort to implement global algorithms for improved performance, achieving 30 fps with a low error rate.Source: C. Global Stereo Matching on FPGA · p. 4
  • Global algorithms can provide state-of-the-art accuracy and disparity-map quality, but their computationally intensive optimization or convolutional networks challenge real-time deployment on resource-limited embedded systems.Source: C. Global Stereo Matching on FPGA · p. 4
  • The stereo-system comparison reports frame rate, resolution, disparity levels, MDE/s, power, resource utilization, and hardware platform.MDE/s is defined as width × height × fps × disparity.Source: Table I · p. 5
  • 63.54 fps at 640×480 resolution with 128 disparity levels was reported for an FPGA dynamic programming maximum likelihood architecture.The tree-structured dynamic programming design reported 58.7 fps at 640×480 resolution and a low error rate.Source: C. Global Stereo Matching on FPGA · p. 5
  • Cross-trees, dynamic programming for DNA sequence alignment, and graph cuts have also been adapted for FPGA implementation, with real-time processing reported,,.Source: C. Global Stereo Matching on FPGA · p. 5

D. Semi-Global Matching on FPGA

This section surveys FPGA-based acceleration for SGM and other robotic tasks, including roadmap construction, collision detection, and graph search. It covers SGM architectures and implementations, their computational, storage, and quality–speed tradeoffs, as well as real-time and flexibility tradeoffs and parallel shortest-path architectures.

D. Semi-Global Matching on FPGA

  • SGM combines local pixel matching with image-wide smoothness optimization to produce more robust disparity maps, but hardware implementation faces data dependence, high complexity, and large storage demands.These challenges have motivated FPGA-friendly variants,, –.Source: D. Semi-Global Matching on FPGA · p. 5
  • 30 fps at 640×480 with 128 disparities, 67 fps at 1024×768 with 96 disparities, and 42 fps at 1600×1200 with 128 disparities demonstrate real-time FPGA SGM implementations.[74] uses a systolic array and two-dimensional parallelization; implements cost initialization, cross-based aggregation, and semi-global optimization.Source: D. Semi-Global Matching on FPGA · p. 5
  • The Cyclone IV FPGA-based SGM design achieves 127 fps at 1024×768 resolution with 128 disparity levels, combining disparity and multi-level parallelism, such as image-line processing, to address data dependence and irregular data-access problems.Source: D. Semi-Global Matching on FPGA · p. 5
  • 25 fps at 1024×768 with 256 disparities combines sampling-insensitive absolute-difference preprocessing with streaming post-processing that detects noisy and occluded regions.The design was evaluated in a full stereo vision system using DE2i-150 and DE4 platforms.Source: D. Semi-Global Matching on FPGA · p. 5
  • 72 fps at 1242×375 with 128 disparities demonstrates an HLS SGM variation, while FP-Stereo [79] matches GPU accuracy at competitive speed with much lower energy consumption.FP-Stereo automates high-performance SGM pipeline construction using optimizations that exploit parallelism and reduce resource consumption.Source: D. Semi-Global Matching on FPGA · p. 5

B. Roadmap Construction

  • Roadmap acceleration ranges from faster nearest-neighbor search and edge trimming to design-time precomputation that removes runtime construction latency.The precomputed, more general roadmap supports fast successive queries in complex environments without runtime accelerator reprogramming.Source: B. Roadmap Construction · p. 11

C. Collision Detection

  • Collision detection can consume 90% of motion-planning processing time, and GPU parallelization still struggles to meet real-time requirements.Bialkowski et al. – use data parallelization, but the passage attributes the remaining limitation to GPUs’ core constraints.Source: C. Collision Detection · p. 11
  • Murray et al. report submillisecond motion-planning queries and more than one order of magnitude lower power consumption with specialized FPGA circuits.The design creates a specialized circuit for each motion in the roadmap.Source: C. Collision Detection · p. 11
  • Dadu-P solves dynamic-environment plans in around 300 microseconds, while retargetability to different robots and scenarios incurs a 25× latency increase.Its roadmap-edge data structure supports flexibility, while batched processing and priority rating improve efficiency; Murray et al. report a fully retargetable design below 3 ms at 35 W.Source: C. Collision Detection · p. 11

D. Graph Search

  • FPGA graph-search designs use blocked APSP for a 15× speedup over an optimized CPU, architecture-efficient Dijkstra, and highly parallel large-scale search.Murray et al. use Bellman-Ford with a precomputed roadmap and bounded robot quantities to enable compact storage, dataflows, and a low-cost interconnection network.Source: D. Graph Search · p. 11

VI. PARTIAL RECONFIGURATION

The section reviews partial reconfiguration’s performance and energy trade-offs, then describes PerceptIn’s commercial vehicle computing system and FPGA-based sensing. Its examples cover runtime algorithm swapping, task allocation across hardware, and sensor synchronization.

VI. PARTIAL RECONFIGURATION

  • 1.2 Gbytes/s effective configuration transfer throughput exceeds the 400 Mbytes/s theoretical upper bound; streaming DMA cuts configuration time from seconds to milliseconds (>1000-fold).The bitstream compression scheme reduces bitstream size by up to 75% with negligible hardware overhead.Source: §VI · p. 11
  • A study of PR versus clock gating identifies conditions for reducing total energy and proposes ways to minimize configuration energy overhead.The study applies the techniques to an embedded microprocessor design and reports accelerated application performance alongside reduced overall energy consumption.Source: §VI · p. 12
  • PerceptIn’s runtime PR engine swaps FPGA algorithms for scenario-specific localization tasks, with bitstreams under 4 MB and reconfiguration delay under 1 ms.Feature extraction runs in key frames; tracking runs in non-key frames and takes 10 ms, 50% less time than feature extraction.Source: §VI · pp. 11–12

VII. COMMERCIAL APPLICATIONS OF FPGAS IN AUTONOMOUS VEHICLES

  • PerceptIn’s commercial vehicles are deployed in China, the US, Japan, and Switzerland; their custom computing system uses FPGA to synchronize sensors and accelerate critical-path components.The design accounts for task-level parallelism, cost, safety, and programmability.Source: §VII · p. 12

A. Computing system

  • The vehicle pipeline serializes sensing, perception, and planning, while localization and scene understanding can run in parallel; sensing and localization use FPGA, scene understanding GPU, and planning CPU.The system uses a Zynq Ultrascale+ FPGA with an on-vehicle PC containing an Intel Coffee Lake CPU and Nvidia GTX 1060 GPU.Source: A. Computing system · pp. 12–13

B. Sensing on FPGA

  • FPGA sensing supports in-situ sensor processing and synchronization; the hardware synchronizer uses a GPS-initialized common timer to timestamp camera and IMU samples.Software timestamping can be inaccurate because variable, nondeterministic processing latency precedes timestamp assignment.Source: B. Sensing on FPGA · p. 13

C. Perception on FPGA

The survey reports FPGA-based acceleration across robotic perception and space missions, alongside radiation-mitigation methods and practical barriers to wider adoption.

C. Perception on FPGA

  • On Zynq, FPGA localization was faster than GPU localization; FPGA SLAM achieved 2.2× speedup at 44 fps, 83% energy savings, and 4×/27× Schur/Jacobian speedups.The FPGA was faster than GPU only for localization; the Schur and Jacobian results compare against CPU implementations and save 76% energy.Source: C. Perception on FPGA · pp. 13–14

VIII. APPLICATION OF FPGAS IN SPACE ROBOTICS

  • Space-grade FPGA choices trade off volatility, routing delay, power, reprogrammability, and radiation susceptibility across SRAM, antifuse, and flash technologies.SRAM FPGAs are common in space missions but susceptible to configuration upsets; in-flight reconfiguration is not recommended for flash devices.Source: §VIII · p. 14

A. Radiation Tolerance for Space Computing

  • Radiation can cause cumulative TID damage and single-event errors; TMR tolerates at most one fault per voter stage, while scrubbing restores configuration memory.Blind scrubbing avoids error-detection complexity but may write errors into frames; readback scrubbing checks for errors before repair.Source: A. Radiation Tolerance for Space… · pp. 14–15

B. FPGAs in Space Missions

  • Long transmission delays make onboard processing and compression increasingly important for space robots, and FPGA reconfiguration can support adaptation to changing environments and data.The survey notes that ground processing may become inefficient or unviable as mission travel distance increases.Source: B. FPGAs in Space Missions · p. 15

1) Mars Exploration Rover Missions

  • Mars Exploration Rovers designed for 90 Martian days operated until 2019, with observed FPGA mitigation error rates very similar to predictions.The cited mitigation measures included fabrication processes to prevent SEL, TMR to reduce SEU frequency, and scrubbing to recover from SEFI.Source: 1) Mars Exploration Rover Missions · p. 16

2) Mars Science Laboratory Mission

  • On Curiosity, FPGA-based camera electronics handle image processing and compression, while Microblaze flight software coordinates hardware and implements acquisition and fault-protection functions.The flight software comprises 10,000 lines of ANSI C code implemented on the FPGA.Source: 2) Mars Science Laboratory Mission · p. 16

3) Mars 2020 Mission

  • Perseverance repurposes its Vision Compute Element from landing-site estimation to GESTALT driving acceleration after landing.The element includes a Virtex 5QV Vision Processor and an RTAX 2000 Housekeeping FPGA.Source: 3) Mars 2020 Mission · pp. 16–17

IX. CONCLUSION

  • The survey reports more than 10× better performance and energy efficiency from FPGA software–hardware co-design than CPU and GPU implementations.It also reviews partial reconfiguration and commercial and space applications.Source: §IX · p. 17

Related papers

Loading 2009.06034v3…