PaperlayerView exact revision on arXiv

Source-linked AI summary

XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference

Francesco Conti, Pasquale Davide Schiavone, Luca Benini

arXiv:1807.03010v1cs.NEcs.ARcs.LG

TL;DR

Embedded inference must fit deep-learning workloads within low-power device constraints, while software BNNs require efficient popcount instructions and temporary storage of nonbinary partial results. The paper integrates a configurable XNOR Neural Engine for BNN layers into an MCU, with a microcode processor implementing outer loops. With SRAM power-gated, minimum energy reaches 21.6 fJ per operation at 0.4 V; real-time execution is reported at 8.9 fps and 2.2 mJ per frame.

  • Problem

    Embedded inference must fit deep-learning workloads within low-power device constraints, while software BNNs require efficient popcount instructions and temporary storage of nonbinary partial results.

    Source: §I · p. 1

  • Method

    The paper integrates a configurable XNOR Neural Engine for BNN layers into an MCU, with a microcode processor implementing outer loops.

    Source: §I · p. 1, 2) Microcode processor · p. 5

  • Results

    With SRAM power-gated, minimum energy reaches 21.6 fJ per operation at 0.4 V; real-time execution is reported at 8.9 fps and 2.2 mJ per frame.

    Source: 2) Energy efficiency evaluation · p. 8, §V · p. 10

  • Takeaways & Limitations

    Even with BNNs, memory accesses and data transfers remain a significant part of real-world inference energy, and hybrid memory can empower accelerator use.

    Source: §I · p. 1, §V · p. 10

Abstract

from arXiv · show

Binary Neural Networks (BNNs) are promising to deliver accuracy comparable to conventional deep neural networks at a fraction of the cost in terms of memory and energy. In this paper, we introduce the XNOR Neural Engine (XNE), a fully digital configurable hardware accelerator IP for BNNs, integrated within a microcontroller unit (MCU) equipped with an autonomous I/O subsystem and hybrid SRAM / standard cell memory. The XNE is able to fully compute convolutional and dense layers in autonomy or in cooperation with the core in the MCU to realize more complex behaviors. We show post-synthesis results in 65nm and 22nm technology for the XNE IP and post-layout results in 22nm for the full MCU indicating that this system can drop the energy cost per binary operation to 21.6fJ per operation at 0.4V, and at the same time is flexible and performant enough to execute state-of-the-art BNN topologies such as ResNet-34 in less than 2.2mJ per frame at 8.9 fps.

I. INTRODUCTION

The introduction frames embedded deep learning as constrained by data-center dependence, latency, privacy, and power, then presents binary neural networks and a tightly integrated MCU accelerator as a response. The related-work discussion distinguishes this system-level integration and memory focus from prior efficiency-oriented approaches.

I. INTRODUCTION

  • Edge deployment must fit deep-learning workloads within a few-milliwatt power budgets and sub-dollar device costs, while avoiding data-center latency and radio dependence.The introduction identifies biomedical devices, autonomous vehicles, and embedded controllers as potential applications.Source: §I · p. 1
  • Binarizing weights and activations enables XNOR operations instead of full products and reduces memory footprint, while XNE buffers partial results internally for convolutional and dense layers.The paper reports peak efficiency of 21.6 fJ per operation and execution of ResNet-18 and ResNet-34 above 8 fps in less than 2.2 mJ per frame.Source: §I · p. 1

II. RELATED WORKS

  • Prior CNN accelerators reduce costs through pruning and compression or reduced arithmetic precision, with BNN methods trading retraining and some accuracy loss for lower precision.The related-work discussion also describes in-memory, ASIC, and mixed-signal approaches to binary or ternary networks.Source: §II · p. 2
  • This work targets BNN acceleration tightly integrated with a microcontroller so software and hardware can cooperate while accounting for memory effects in real-world networks such as ResNet and Inception.Its design uses a tightly coupled shared-memory paradigm.Source: §II · p. 2

III. ARCHITECTURE

The primer expresses BNN inference as convolutional or dense layers whose binarized weights and inputs produce outputs through a thresholded batch-normalization and binarization step. It then recasts the binary arithmetic using XNOR and popcount operations.

A. Binary Neural Networks primer

  • BNN inference maps to convolutional and densely connected layers with binarized weights, inputs, and outputs, plus a real-valued bias.The binarization function combines inference-time batch normalization with binarization of the integer-valued sum.Source: A. Binary Neural Networks primer · p. 2
  • Mapping +1 to bit 1 and −1 to bit 0, and moving bias inside binarization, reorganizes the layer computation around a threshold.The threshold is defined using the batch-normalization parameters.Source: A. Binary Neural Networks primer · pp. 2–3
  • XNOR operations replace multiplications, while popcounting—the count of bits set to 1—replaces sums.The thresholding rule depends on whether λ is positive or negative.Source: A. Binary Neural Networks primer · p. 3

B. XNE operating principles

XNE supports binarized convolutional and dense layers by hardwiring feature loops and organizing computation around tiled parallel operations. Its loop ordering and stationary-data choice avoid storing intermediate popcount values in memory.

B. XNE operating principles

  • Convolutional layers use six nested loops, while dense layers are the 1 × 1, single-pixel limit case of convolution.XNE is designed to execute both layer types.Source: B. XNE operating principles · p. 3
  • Deeper layers have more input and output feature maps than spatial positions, favoring pixel- or feature-map-level parallelism over filter-level parallelism.BNNs need tens or hundreds of binary pixels processed in parallel for energy efficiency, which deeper-layer filter-level parallelism cannot provide.Source: B. XNE operating principles · p. 3
  • Because popcount intermediates are integers, XNE avoids storing them in memory and focuses on input- and output-stationary execution.The paper notes that swapping input and weight roles can support weight- and output-stationary operation as well.Source: B. XNE operating principles · p. 3
  • A design-time throughput parameter TP sets tile size and the number of simultaneous XNOR operations per cycle; every TP cycles, the accelerator consumes TP input pixels and TP weight sets to produce TP output pixels.The inner feature loops are hardwired, while the remaining loops are reorganized around tiled feature maps and filter loops.Source: B. XNE operating principles · p. 3
  • The loop reordering places spatial loops outside, feature-map loops inside, and filter loops in the middle, with a tiny microcode processor implementing outer loops.Remainder iterations are needed when input or output feature counts are not multiples of TP.Source: B. XNE operating principles · p. 3

C. Accelerator architecture

The XNE architecture separates control, streaming computation, and memory transduction. Its controller programs the layer and microcode, then coordinates feature loading, accumulation, and thresholding/binarization.

C. Accelerator architecture

  • The architecture comprises a controller, an engine with a streaming datapath, and a streamer that transacts between the engine stream and memory.The controller uses a memory-mapped APB target port; the streamer converts stream widths that are multiples of 32 bits to byte-aligned shared-memory accesses and back.Source: C. Accelerator architecture · pp. 3–4, Figure 1 · p. 4
  • The controller is programmed with layer parameters and microcode, then its FSM orchestrates feature loading, accumulation, and thresholding/binarization.During accumulation, TP weight vectors are loaded over TP iterations and accumulated; threshold values are then loaded and binarized outputs are streamed out.Source: C. Accelerator architecture · p. 4, Figure 2 · p. 4

1) Interface modules:

The XNE uses memory-mapped control and shared-memory master ports, with a controller coordinating computation and streamer modules moving data between memory and the engine.

1) Interface modules:

  • The XNE exposes an APB control port, TP/32 32-bit shared-memory master ports, and an event wire signaling computation completion.Source: 1) Interface modules · p. 4
  • The controller combines a memory-mapped register file, phase-control FSM, and microcode processor for the six loops shown in the listings.Source: 1) Interface modules · p. 4
  • Generic parameters persist across jobs, while duplicated job-dependent registers allow a new job to be offloaded during current execution.Source: 1) Interface modules · p. 4
  • Streamer sources and sink generate addresses and realign non-word-aligned accesses into coherent streams, with mux/demux blocks sharing memory transactions.Source: 1) Interface modules · p. 4

2) Microcode processor:

An internal microcode processor executes the six outer loops, reducing the need for the external processor to maintain iteration state and memory offsets.

2) Microcode processor:

  • The microcode processor implements six nested outer loops using four read/write registers for indices and sixteen read-only registers for loop ranges and values.Source: 2) Microcode processor · p. 4
  • The controller and streamer IPs are available as open source.Source: 2) Microcode processor · p. 4
  • Its tiny ISA uses ADD and MV imperative instructions plus a LOOP instruction that declares iteration ranges and micro-instruction spans.Source: 2) Microcode processor · p. 5
  • A single-stage ISA pipeline uses four lockstep finite-state machines to track the next instruction, its loop index, the next iteration, and the next loop.Source: 2) Microcode processor · p. 5
  • The six-loop microcode occupies 28B—22B imperative and 6B declarative—and is stored in the generic register-file section between jobs.Source: 2) Microcode processor · p. 5

3) Datapath micro-architecture:

The datapath performs binary products, accumulation, and threshold-based binarization, with feature vectors reused across output computations.

3) Datapath micro-architecture:

4) Impact of accumulator and threshold truncation:

The authors report small effects from accumulator and threshold truncation, and introduce experimental results covering standalone XNE area and power alongside the integrated MCU.

4) Impact of accumulator and threshold truncation:

IV. EXPERIMENTAL RESULTS

  • The evaluation covers standalone XNE area and active-accumulation power in nominal conditions for two target technologies.Source: §IV · p. 5, Figure 4 · p. 6
  • The MCU architecture and its 22nm floorplan are shown, including synchronous core, accelerator, and memory blocks.Source: Figure 5 · p. 6

A. Standalone XNE

The XNE’s throughput parameter sets accelerator parallelism and memory-port requirements, while area and power scale with engine size. The selected TP=128 design was integrated into an MCU with hybrid SRAM and standard-cell memory.

A. Standalone XNE

  • TP was swept from 32 to 512 to evaluate its effects on accelerator area and power.TP sets the amount of hardware parallelism and the related number of memory-side master ports.Source: A. Standalone XNE · p. 6
  • After place and route, engine and streamer size grew near-linearly with TP, progressively absorbing fixed costs; these modules also accounted for most active-accumulation power.The standalone physical-design run used a conservative 65% utilization target, but did not model all effects of coupling the XNE to a multi-banked memory.Source: A. Standalone XNE · p. 6

B. XNE in a MCU System

  • The MCU integrated a TP=128 XNE with RISCY, autonomous uDMA, and 520 kB of hybrid memory, including SCM banks.The 520 kB comprises 64 kB of core-coupled memory and 456 kB shared by RISCY, uDMA, and XNE; SCMs support operation below SRAM’s rated voltage but use more area.Source: B. XNE in a MCU System · pp. 6–7

1) Performance evaluation:

The integrated XNE sustains substantially higher binary-layer throughput than optimized software, while SCM operation improves energy efficiency, especially at low voltage. The mVGG-D evaluation shows that memory placement and model size shape the energy–accuracy trade-off.

1) Performance evaluation:

  • 220 op/cycle gives the integrated XNE 71× the throughput of optimized software for binary convolutions and dense layers.This is 86% of theoretical peak at TP=128, with the reduction attributed to memory contention and small control bubbles.Source: 1) Performance evaluation · p. 7

2) Energy efficiency evaluation:

  • With SRAM-backed execution, memory dynamic power is 7.1× computation power; SCMs consume approximately 3× less than SRAMs.In both memory configurations, accesses are largely due to weights, which the XNE loads many times and uses only once.Source: 2) Energy efficiency evaluation · p. 8
  • 21.6 fJ per operation at 0.4 V is the minimum energy point when SRAMs are power-gated.Without fully switching down SRAMs, the minimum is 40.2 fJ per operation at 0.5 V, where the MCU delivers 28 Gop/s.Source: 2) Energy efficiency evaluation · p. 8

3) Energy-accuracy tradeoff in BNNs:

  • mVGG-1 is approximately 6% from CIFAR-10 BNN state of the art but consumes roughly 10× the energy of mVGG-2 because it requires external HyperRAM.As d increases, SRAM-to-SCM marshaling becomes less costly than operating directly on SRAM by mVGG-8; the mVGG-D evaluation plots the energy–error trade-off across memory modes.Source: 3) Energy-accuracy tradeoff in BNNs · p. 8, Figure 8 · p. 8
  • mVGG-F consumes 100× less than mVGG-d but incurs a significant accuracy penalty.The fully depthwise-separable model is small enough to run entirely on SCMs.Source: 3) Energy-accuracy tradeoff in BNNs · p. 9

4) Real-world sized BNN execution:

The MCU runs binarized ResNet-18 and ResNet-34 using external memory for weights while keeping partial results on-chip. ResNet-34 inference reaches 8.9 fps at 2.17 mJ per frame, with memory traffic remaining a major energy cost.

4) Real-world sized BNN execution:

  • The MCU can execute both binarized ResNet topologies without energy cost for moving partial results.The design accommodates external memory for weights while buffering inputs, outputs, and partial results on-chip; the selected ResNets have ImageNet top-5 accuracies of 67.6% and 76.5%, respectively.Source: 4) Real-world sized BNN execution · p. 9
  • ResNet-34 inference runs at 8.9 fps and spends 2.17 mJ per frame on a standard 224×224 input.ResNet-18 runs at approximately 14.7 fps and spends 1.45 mJ per frame under the same SRAM-based mode with asynchronous weight transfers and double buffering.Source: 4) Real-world sized BNN execution · p. 9
  • Memory traffic dominates energy in both networks, especially in the final layers; it is 2.5× more dominant in ResNet-18 than ResNet-34.The fully connected layer is memory-bound, while the system loads weights directly into shared SRAM and keeps partial results there.Source: 4) Real-world sized BNN execution · p. 9

5) Comparison with the state-of-the-art and discussion:

The discussion positions XNE as an MCU-integrated BNN solution designed around system-level memory and integration constraints rather than minimum accelerator energy. The conclusion reports high efficiency and real-time ResNet-34 execution while identifying silicon measurements and further integration as future work.

5) Comparison with the state-of-the-art and discussion:

  • Bankman et al. report more than 700 Top/s/W, but their mixed-signal IP is difficult to port between technologies and hardwires convolution size to 2×2.The fixed convolution size limits flexibility across convolution shapes.Source: 5) Comparison with the… · p. 9
  • XNORBIN [34] reports almost 100 Top/s/W but cannot fit BNNs beyond AlexNet or exchange external data; UNPU is roughly 16× larger and half as energy efficient.XNORBIN's custom memory hierarchy accounts for most of its raw energy-efficiency advantage over the MCU design.Source: 5) Comparison with the… · p. 9
  • XNE prioritizes SoC integration and memory-access costs over the lowest energy solution, occupying ∼1.5% of the MCU with a critical path 30% shorter than the overall MCU system.Interfaces are active ∼95% of execution time in many cases.Source: 5) Comparison with the… · p. 9
  • XNORBIN's separate storage for weights, feature maps, and lines improves effective memory bandwidth and efficiency at the expense of flexibility and area.The MCU design instead targets limited memory bandwidth in an MCU-like SoC.Source: 5) Comparison with the… · p. 9
  • The XNE-accelerated MCU efficiently executes software-defined BNNs whose convolutional and fully connected linear parts are binary matrix-vector products, provided external memory stores all weights.The authors say this class contains most known neural-network topologies.Source: 5) Comparison with the… · p. 9

V. CONCLUSION

  • The MCU reaches up to 46 Top/s/W and runs ResNet-34 at 2.2 mJ per frame in real time (8.9 fps).The authors also report that a hybrid memory scheme can significantly empower accelerator use.Source: §V · pp. 9–10
  • The 22nm MCU prototype was taped out, with silicon measurements, support for advanced binary-network approaches, and deeper SRAM integration left as future work.The proposed SRAM integration aims to reduce power in high-performance modes and enable more parallel accelerator access while retaining shared memory.Source: §V · p. 10
Loading 1807.03010v1…