PaperlayerView exact revision on arXiv

Source-linked AI summary

Synergy: A HW/SW Framework for High Throughput CNNs on Embedded Heterogeneous SoC

Guanwen Zhong, Akshat Dubey, Tan Cheng, Tulika Mitra

arXiv:1804.00706v1cs.DCcs.ARcs.LG

TL;DR

Resource-constrained mobile and IoT devices need real-time, energy-efficient CNN inference, while embedded implementations remain challenging. Synergy combines CPU, NEON, and FPGA resources with tiled work distribution and pipelined execution. It reports a 7.3x average speedup across seven CNN models over a well-optimized software-only solution.

  • Problem

    Real-time, energy-efficient CNN inference is needed on resource-constrained mobile and IoT devices, but embedded implementations remain challenging and network-specific DSE and HLS burden developers.

    Source: §1 · pp. 2–4

  • Method

    Synergy maps CNNs onto a heterogeneous SoC, using FPGA and NEON accelerators, tiled matrix multiplication, work-stealing, and pipelining to distribute work.

    Source: §1 · pp. 2, 4, §3 · p. 8

  • Results

    7.3x speedup, averaged across seven representative CNN models, is reported over a well-optimized software-only solution.

    Source: §5 · p. 30

  • Takeaways & Limitations

    Network-agnostic accelerators and runtime workload balancing let Synergy adapt to different networks without changing the underlying accelerator architecture.

    Source: §1 · p. 4, §5 · p. 30

Abstract

from arXiv · show

Convolutional Neural Networks (CNN) have been widely deployed in diverse application domains. There has been significant progress in accelerating both their training and inference using high-performance GPUs, FPGAs, and custom ASICs for datacenter-scale environments. The recent proliferation of mobile and IoT devices have necessitated real-time, energy-efficient deep neural network inference on embedded-class, resource-constrained platforms. In this context, we present {\em Synergy}, an automated, hardware-software co-designed, pipelined, high-throughput CNN inference framework on embedded heterogeneous system-on-chip (SoC) architectures (Xilinx Zynq). {\em Synergy} leverages, through multi-threading, all the available on-chip resources, which includes the dual-core ARM processor along with the FPGA and the NEON SIMD engines as accelerators. Moreover, {\em Synergy} provides a unified abstraction of the heterogeneous accelerators (FPGA and NEON) and can adapt to different network configurations at runtime without changing the underlying hardware accelerator architecture by balancing workload across accelerators through work-stealing. {\em Synergy} achieves 7.3X speedup, averaged across seven CNN models, over a well-optimized software-only solution. {\em Synergy} demonstrates substantially better throughput and energy-efficiency compared to the contemporary CNN implementations on the same SoC architecture.

1 introduction

Synergy targets real-time, energy-efficient CNN inference on resource-constrained embedded devices by combining the compute resources of a heterogeneous SoC. It maps CNN work across FPGA and NEON accelerators and uses runtime scheduling and pipelining to improve performance while avoiding network-specific accelerator design.

Heterogeneous HW/SW Acceleration

  • Using NEON and FPGA together rather than FPGA-only solutions improves overall latency by 12% and throughput by 15%, averaged across multiple CNN models.Synergy transforms CONV-layer computation into independently processed tiled matrix multiplication and feeds the tiles to both hardware and software accelerators.Source: §1 · p. 3

Transparent Consolidation of Accelerators

  • Network-specific DSE and HLS burden application developers, while a single time-multiplexed accelerator can compromise across CONV layers with diverse compute requirements.These constraints are especially challenging for developers with limited FPGA experience.Source: §1 · pp. 3–4
  • Network-agnostic FPGA and NEON accelerators process tiled matrix multiplications through work-stealing, achieving performance comparable to network-specific designs returned through DSE.This avoids DSE and HLS for each CNN model and provides an out-of-the-box solution.Source: §1 · p. 4

High-Throughput HW/SW Multi-Threaded Pipeline

  • 39.5–136.4 frames/second throughput and 14.4–55.8 mJ/frame energy are reported across CNN models for Synergy’s pipelined design.Consecutive frames are processed concurrently, allowing CONV layers from different frames to run simultaneously on the FPGA.Source: §1 · p. 4

1 introduction

  • 99.8% average accelerator utilization is reported for Synergy’s pipelined design with multiple frames in flight.Concurrent CONV-layer execution helps keep FPGA accelerators busy while software handles non-compute-intensive layers.Source: §1 · p. 5

Automated Customized Accelerator Design

  • The architecture generator can synthesize designer-parameterized accelerators and their hardware-software interface for a CNN model or a new SoC device.The default plug-and-play solution avoids per-model DSE and HLS for application developers.Source: §1 · p. 5

2 related works

Prior FPGA-based CNN frameworks are divided between network-dependent designs that require model-specific exploration and network-independent designs that reuse fixed hardware. Synergy combines a unified heterogeneous-compute abstraction with runtime work-stealing to adapt across networks without changing the hardware or software implementations.

Network-dependent Frameworks

  • Network-dependent frameworks require designers to explore and synthesize a new architecture for each network, trading flexibility for potentially high performance.Some approaches automate exploration, while others require manual design work [6][8][13][16][17][18][19][20][21][23][25][24].Source: §2 · p. 6

Network-independent Frameworks

  • Network-independent frameworks reuse fixed hardware across networks, but static execution can limit concurrent layer processing and accelerator utilization.The surveyed approach executes layers in sequence and cannot run multiple layers concurrently on different input frames.Source: §2 · pp. 6–7

Synergy

  • Synergy unifies NEON and FPGA compute and uses runtime work-stealing to balance accelerator workloads across networks without changing hardware or software implementations.Its automated toolchain also supports accelerator exploration and migration to other embedded FPGA devices.Source: §2 · p. 7

3 The Synergy Framework

Synergy maps CNN inference onto embedded heterogeneous SoCs, using CPU cores, NEON engines, and FPGA accelerators. Its threaded pipeline and runtime work-stealing support concurrent processing and workload balancing across networks.

3 The Synergy Framework

  • Synergy maps CNN inference to embedded FPGA-based heterogeneous SoCs and uses CPUs, NEON engines, and FPGA resources.Its templates can generate software/hardware applications from a network configuration file, while the FPGA bitstream can remain unchanged across models.Source: §3 · pp. 8–9
  • CONV layers become independent tiled matrix-multiplication jobs executed in parallel by FPGA PEs and NEON accelerators.CPU cores handle other layers and preprocessing functions.Source: §3 · p. 8
  • A producer-consumer pipeline lets CPU cores and accelerators process different layers of different frames concurrently.Threads communicate between layers through synchronized FIFO mailboxes.Source: §3 · p. 8
  • Work-stealing redistributes CONV jobs from busy clusters to idle ones, balancing workload without requiring a network-specific cluster configuration.Within each cluster, jobs are dispatched to available NEONs and FPGA PEs in round-robin fashion.Source: §3 · p. 9

3.1 Software Architecture

Synergy divides CNN work across software threads and heterogeneous accelerators, using tiled matrix-multiplication jobs for CONV layers and work-stealing to balance cluster workloads. CPU cores process other layers and preprocessing, with some sequential overhead partially hidden by pipelining.

3.1 Software Architecture

  • The software architecture assigns CONV computation to heterogeneous accelerators while CPU cores handle other CNN layers and preprocessing.The framework supports multiple processing engines and a multi-threaded pipeline.Source: §3.1 · p. 10, §3.1.4 · p. 15

3.1.1 CONV Layers

  • Each CONV job computes one output tile of a matrix multiplication; loop tiling keeps working data within FPGA BRAM and exposes independent tile computations.Synergy accelerates the tiled computation with FPGA PEs and NEON cores.Source: §3.1.1 · pp. 10–11

3.1.2 Delegate Threads

  • A delegate thread wraps each FPGA PE, enabling it to interact with software threads and perform operating-system calls on the accelerator's behalf.The thread starts its PE, provides job addresses on request, and waits for completion acknowledgments.Source: §3.1.2 · p. 13

3.1.3 Self-balancing: Work Stealing Scheduler

  • Work-stealing lets idle clusters take jobs from busy clusters, balancing workload without requiring an optimal static cluster configuration.The technique uses job-level granularity and avoids designer effort spent finding the best cluster configuration.Source: §3.1.3 · pp. 14–15

3.1.4 Other Layers and Preprocessing functions

  • CPU cores execute fully connected, pooling, and activation layers, while pipelining partially hides sequential preprocessing overheads such as im2col and normalization.Normalization scales input data to values between 0 and 1 during inference.Source: §3.1.4 · p. 15

3.2 Hardware Architecture

Synergy's hardware architecture combines FPGA processing engines with a memory subsystem and software delegate threads. Tiled matrix multiplication, HLS optimizations, and a multi-MMU design address computation, data movement, and memory contention.

3.2 Hardware Architecture

  • Each FPGA PE computes tiled matrix multiplication, fetching input tiles from DDR to local storage and writing completed output tiles back to memory.The PE obtains jobs through its delegate thread and communicates with the memory subsystem through memory FIFOs.Source: §3.2 · p. 16, §3.2.1 · p. 17

3.2.1 Accelerator Design

  • Loop pipelining and complete unrolling expose parallelism in the tiled-MM kernel, while array partitioning can reduce the initiation interval constrained by BRAM read ports.Greater per-PE parallelism consumes more compute resources and can reduce the number of PEs that fit on the FPGA; double buffering overlaps data transfer and computation.Source: §3.2.1 · p. 18

3.2.2 Memory Subsystem

  • The memory subsystem translates virtual addresses and services PE memory requests; Synergy's multiple-MMU design assigns at most two PEs to each MMU and memory controller.The MMUs share a Proc unit, and the reported performance speedup increases linearly as more PEs are instantiated.Source: §3.2.2 · pp. 19–20

3.3 Hardware Architecture Generator

Synergy’s architecture generator automates FPGA accelerator and software-library generation from a configuration file.

3.3 Hardware Architecture Generator

  • The generator takes network-independent hardware architecture parameters from a configuration file as input.The simplified configuration format creates the hardware architecture shown in Figure 5.Source: §3.3 · p. 21
  • It automates HLS processing-element generation, hardware architecture creation, and final FPGA bitstream generation.Source: §3.3 · p. 21
  • The configuration also drives compilation of the Synergy Library, which provides APIs for the HW/SW multithreading framework.Source: §3.3 · p. 21

4 Experimental Evaluation

Evaluation on representative CNN models reports higher throughput and lower energy than the software baseline, with further comparisons of heterogeneous execution and work-stealing load balance.

4 Experimental Evaluation

  • The evaluation uses seven Darknet CNN models, 32-bit floating-point implementations, and a shared two-cluster configuration on the Xilinx Zynq XC7Z020.The cluster configuration is 2 NEONs + 2 S-PE in Cluster-0 and 6 F-PE in Cluster-1; the FPGA-based PEs run at 100 MHz.Source: §4 · pp. 21–23

4.1 Synergy Throughput and Energy-Efficiency

  • 7.3x higher throughput and 80.13% lower energy consumption averaged across CNN models versus the original single-threaded Darknet on ARM cores.Source: §4.1 · pp. 23–24, Figure 9 · p. 24, Table 3 · p. 26
  • Synergy reports 36.63% more power consumption than CPU+NEON-only implementations, while FPGA logic accounts for 27% of total power, about 2.08 W on average.Source: §4.1 · p. 23, Figure 10 · p. 25
  • Synergy’s CIFAR full and MNIST designs outperform [21] in throughput, GOPS, and energy consumption; MNIST and MPCNN achieve 4.5x and 1.8x GOPS speedup versus.The comparison notes that Synergy uses 32-bit floating point, while some compared FPGA designs use fixed-point operations.Source: §4.1 · pp. 24–25, Table 4 · p. 26

4.2 Advantage of Heterogeneity

4.3 Transparent Accelerators: Work Stealing

  • 24% higher throughput than static-mapping+fixed-architecture designs and 6% higher than static-mapping+custom-architecture designs.For CIFAR Alex, work-stealing brings cluster runtimes to 22.2 and 20.9 ms per frame, compared with 24.3 and 12.3 ms under fixed static mapping.Source: §4.3 · pp. 27–28, Figure 13 · p. 28
  • 99.8% average cluster utilization with Synergy, compared with 92.5% for fixed static mapping and 96.5% for custom static mapping.Non-pipelined designs average 56.1% utilization; pipelining raises utilization above 90%.Source: §4.3 · pp. 29–30, Table 6 · p. 30

5 conclusion

Synergy combines CPU, NEON, and FPGA resources with work-stealing to balance accelerator workloads across networks at runtime. Across seven representative CNN models, it achieves 7.3x speedup over optimized software and better throughput and energy-efficiency than contemporary implementations on the same SoC.

5 conclusion

  • Synergy uses CPUs, NEONs, and FPGA resources with a software work-stealing scheduler that balances accelerator workloads and adapts to different networks at runtime without changing hardware or software implementations.Source: §5 · p. 30
  • 7.3x speedup averaged across seven representative CNN models over a well-optimized software-only solution, with better throughput and energy-efficiency than contemporary CNN implementations on the same SoC platform.Source: §5 · p. 30
Loading 1804.00706v1…