FINN: A Framework for Fast, Scalable Binarized Neural Network Inference

Yaman Umuroglu; Nicholas J. Fraser; Giulio Gambardella; Michaela Blott; Philip Leong; Magnus Jahre; Kees Vissers

Source-linked AI summary

FINN: A Framework for Fast, Scalable Binarized Neural Network Inference

Yaman Umuroglu, Nicholas J. Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus Jahre, Kees Vissers

arXiv:1612.07119v1Dec 1, 2016cs.CVcs.ARcs.LG

TL;DR

CNN inference is computationally demanding, while binary weights and activations offer a compact, FPGA-suitable alternative. FINN builds scalable BNN accelerators with optimized mapping and per-layer throughput customization, achieving leading classification rates across MNIST, CIFAR-10, and SVHN benchmarks. The results support FPGA-based BNNs for fast, low-latency embedded inference, within the paper’s evaluated settings and assumptions.

Problem
Modern CNNs require millions of floating-point parameters and billions of operations per image, motivating efficient inference despite their computational demands.
Method
FINN uses a parameterizable FPGA BNN architecture with novel mapping optimizations and configurable per-layer resources to meet throughput requirements.
Results
FINN surpassed prior classification rates by over 48× on MNIST, 2.2× on CIFAR-10, and 8× on SVHN.
Takeaways & Limitations
BNNs are well-suited to FPGA inference because compact models fit on-chip and simplified arithmetic enables high performance, low latency, and low power.
Takeaways & Limitations
Accuracy implications of padding bipolar activations with 1 or -1 remain an open question, although early experiments suggest little difference.

Abstract

from arXiv · show

Research has shown that convolutional neural networks contain significant redundancy, and high classification accuracy can be obtained even when weights and activations are reduced from floating point to binary values. In this paper, we present FINN, a framework for building fast and flexible FPGA accelerators using a flexible heterogeneous streaming architecture. By utilizing a novel set of optimizations that enable efficient mapping of binarized neural networks to hardware, we implement fully connected, convolutional and pooling layers, with per-layer compute resources being tailored to user-provided throughput requirements. On a ZC706 embedded FPGA platform drawing less than 25 W total system power, we demonstrate up to 12.3 million image classifications per second with 0.31 μs latency on the MNIST dataset with 95.8% accuracy, and 21906 image classifications per second with 283 μs latency on the CIFAR-10 and SVHN datasets with respectively 80.1% and 94.9% accuracy. To the best of our knowledge, ours are the fastest classification rates reported to date on these benchmarks.

1. INTRODUCTION

CNNs achieve strong visual-recognition performance but demand substantial computation and memory, motivating binary neural networks and FPGA acceleration. FINN addresses this need with scalable BNN hardware, novel mapping optimizations, and customizable throughput.

Millions of floating-point parameters and billions of operations per image make modern CNN inference computationally demanding.AlexNet required 244 MB and 1.4 GFLOP per image, while VGG-16 required 552 MB and 30.8 GFLOP per image.
One- or two-bit weights and activations can preserve accurate classification while reducing memory and arithmetic requirements.Binary operations also offer higher theoretical FPGA performance and can keep parameters on-chip.
FINN is a framework for scalable, fast BNN inference accelerators on FPGAs, with throughput scaled to user requirements.It targets MNIST, SVHN, and CIFAR-10 classification using customizable accelerator resources.
FINN prototypes surpassed prior classification-rate results by over 48× on MNIST, 2.2× on CIFAR-10, and 8× on SVHN.The paper reports these as the fastest classification rates on the stated datasets to the authors’ knowledge.
FINN quantifies BNN FPGA peak performance with a roofline model and introduces optimizations for more efficient hardware mapping.These contributions accompany the framework and accelerator construction tool.

2. BACKGROUND

The paper introduces neural-network and binarization concepts, then situates BNN inference among FPGA hardware architectures. Binarization reduces arithmetic precision and can enable efficient FPGA implementations, though accuracy and network size remain trade-offs.

A multilayer perceptron arranges neurons in layers, with each neuron receiving outputs from all neurons in the preceding layer.
CNN layers restrict each neuron to a local receptive field and use convolutional windows to produce output images.CNN topologies commonly combine convolutional, pooling, and fully connected layers.
BNNs constrain some or all neural-network arithmetic to single-bit values, with full binarization using binary inputs, weights, and outputs.Partial binarization uses one or two of these binary components.
Binary Neural Networks: Prior work reports competitive accuracy for binarized networks, including MNIST, SVHN, CIFAR-10, and ImageNet results.Reported examples include 98.7% MNIST accuracy for a fully connected binary network and up to 65.5% ImageNet top-1 accuracy for partial binarization.
Neural Networks in Hardware: FPGA neural-network hardware includes single-engine, streaming, vector-processor, and neurosynaptic architectures.The paper positions its BNN mapping work within these categories and notes that fully binarized neural networks had not previously been investigated on FPGAs.
Neural Networks in Hardware: The paper claims binary inference can increase classification rates, reduce power, and minimize latency, while larger networks may incur a small accuracy drop.The authors suggest smaller networks or increased network size as possible ways to address accuracy requirements.

3. BNN PERFORMANCE AND ACCURACY

The paper uses roofline modeling and experiments to compare BNN performance with fixed-point CNNs and to examine accuracy–computation trade-offs. The analysis indicates that BNNs benefit from higher binary-operation throughput and lower parameter-storage requirements, while accuracy and network size must be balanced.

Estimating Performance Using Rooflines: A roofline model compares BNN and fixed-point CNN performance using memory bandwidth, peak computation, and arithmetic intensity.The model identifies whether a design is compute-bound or memory-bound.
Estimating Performance Using Rooflines: 66 TOPS is the modeled binary-operation compute-bound performance for the ZU19EG, approximately 16× the 8-bit and 53× the 16-bit fixed-point performance.The model assumes the binarized AlexNet fits in on-chip memory, whereas the fixed-point variant is memory-bandwidth-bound.
Estimating Performance Using Rooflines: 35000 images per second is the estimated throughput for a binarized AlexNet requiring 1.4 GOPS, under the stated 75%-of-peak design assumption.The passage also estimates 16666 images per second for binarized AlexNet on a Tesla K40.
Accuracy–Computation Tradeoffs: The experiments compare floating-point and binary precision using the same MNIST network topology.Binary networks are formed by replacing regular layers with binary equivalents.
Accuracy–Computation Tradeoffs: A trade-off exists among network size, precision, and accuracy when selecting an efficient solution for a target classification accuracy.The comparison considers either floating-point precision or a larger binary network.

4.1 Architecture

FINN uses a heterogeneous streaming architecture that maps each network layer to a dedicated compute engine and keeps BNN parameters on-chip. Its parameterizable implementation lets users tailor throughput and hardware resources to the target platform and topology.

Separate compute engines are dedicated to each layer and communicate through on-chip data streams.Each engine begins computing as soon as the preceding engine starts producing output.
Keeping all BNN parameters in on-chip memory avoids most off-chip accesses and minimizes classification latency.The compact BNN model size makes this storage strategy practical.
FINN exposes parameterizable building blocks and controls classification throughput according to FPGA, platform, throughput, clock-frequency, and topology constraints.The topology determines how compute resources are allocated across the heterogeneous streaming architecture.
Vivado HLS supports portability, faster development cycles, and automated pipelining to meet a clock-frequency target.

4.2 BNN-specific Operator Optimizations

FINN exploits binary neural-network structure to replace conventional arithmetic and batch-normalization activation with hardware-efficient operations. It also reformulates pooling to preserve outputs while keeping inference in a binary datapath.

Binary neural-network assumptions: 1-bit inputs, weights, and outputs define the full-binarization assumptions used throughout FINN.Unset and set bits represent -1 and +1, respectively; batch normalization precedes the sign activation.
Binary dot products: Binary dot products infer the full signed sum by counting synapses of only one value.Because each synapse is either -1 or +1, Y = Y0 + Y1 enables the remaining count and total response to be inferred.
Binary dot products: Popcount-accumulate uses approximately half the LUT and FF resources of signed-accumulate for binary dot products.At 200 MHz, a 128-bit popcount-accumulate uses 376 LUTs and 29 FFs; the passage contrasts this with signed accumulation.
Threshold activation: Batch-normalization followed by sign activation can be replaced by a compile-time threshold and unsigned comparison.The threshold is τk = µk − (Bk/(γk · ik)); sign-flipping weights handles neurons with opposite threshold direction.
Threshold activation: Threshold activation reduces a 16-bit activation block from 2 DSPs, 55 FFs, and 40 LUTs to 6 LUTs.The comparison is performed without computing the batch-normalized value during inference.
Pooling: Max-pooling can be moved after binarization to derive the same layer outputs without retraining.This reformulation supports binary pooling operations in the accelerator.

4.3 FINN Design Flow and Hardware Library

FINN converts a trained BNN and user-specified throughput target into a synthesizable heterogeneous streaming accelerator. Its hardware library centers on configurable matrix–vector–threshold units, streaming convolution support, and binary or partially binarized datapaths.

Design flow: The FINN synthesizer accepts a Theano-trained BNN and FPS target, selects folding parameters, applies optimizations, and emits synthesizable C++.The generated architecture is heterogeneous and streaming, using components from the FINN hardware library.
MVTU: The MVTU computes BNN matrix–vector products followed by thresholding for fully connected and convolutional layers.Convolutions are represented as matrix–vector products within the MVTU-based implementation.
MVTU: Configurable PE and SIMD-lane counts control MVTU throughput while weights remain distributed in on-chip memory.Input images stream through the unit, and each PE processes the shared input vector against part of the weight matrix.
MVTU: The MVTU PE datapath computes a dot product between streamed inputs and a weight-matrix row before comparing the result with a threshold.This datapath is the processing-element-level implementation of matrix–vector–threshold computation.
Partial binarization: The MVTU template supports partial binarization by removing thresholding for non-binarized outputs or using multiply-add for non-binarized inputs.These features support first and last layers that process non-binary images or produce non-binary outputs.
Convolution: Convolution is lowered to matrix–matrix multiplication using a sliding-window image matrix and a packed filter matrix.The SWU generates the image matrix, while the MVTU computes the matrix–matrix product.
Convolution: Interleaving feature-map channels enables a single wide on-chip memory and direct feeding of the next layer without transposition.The interleaved layout matches MVTU SIMD parallelism and can be formed on-the-fly in the FPGA.
Convolution: The SWU pads binary images with either +1 or -1 because bipolar representation has no zero value.Early experiments suggest little accuracy difference, but the impact on trained networks remains future work.

4.4 Folding

FINN folds matrix–vector computation by configuring processing elements and SIMD lanes, trading parallel hardware for cycles. It then rate-balances heterogeneous streaming layers so resources match user throughput requirements and bottlenecks.

Throughput targeting: FINN uses a simpler folding strategy that controls matrix–vector products to meet a user-specified FPS requirement.The approach focuses on workload mapping rather than the broader formal folding and design-space exploration of prior work.
Matrix–vector folding: Folding is controlled by the MVTU’s number of PEs, P, and SIMD lanes per PE, S.Rows map to PEs and columns map to SIMD lanes within each processed matrix tile.
Matrix–vector folding: For an X × Y matrix, neuron fold Fn = X/P and synapse fold Fs = Y/S, with total fold F = Fn · Fs cycles per matrix–vector multiply.The folds should be integers to avoid padding the weight matrix; a 6 × 4 example with three PEs and two lanes takes 4 cycles.
Convolutional folding: Convolutional layers have total fold F = Fm · Fn · Fs, where Fm reflects the number of output pixels.The additional factor arises from implementing matrix–matrix products as multiple matrix–vector products.
Throughput targeting: Rate-balancing assigns each layer roughly equal image-processing cycles because the slowest layer determines streaming throughput.For fully connected layers, total fold equals the initiation interval; other layers are matched to memory or sliding-window bottlenecks when necessary.
Throughput targeting: Tailoring each MVTU’s compute resources to layer requirements avoids the inefficiencies of a one-size-fits-all architecture.The architecture balances heterogeneous streaming stages instead of dimensioning every layer for full parallelism.

5. EVALUATION

FINN prototypes evaluate BNN inference across MNIST, CIFAR-10, and SVHN under maximum-throughput and fixed-rate scenarios. The results show high throughput, low latency, strong energy and runtime efficiency, and bottlenecks shaped by architecture and FPGA resources.

Experimental Setup: The evaluation uses SFC and LFC fully connected MNIST topologies and CNV convolutional topologies for CIFAR-10 and SVHN.SFC and LFC differ in layer width, while CNV uses repeated convolution, pooling, and fully connected layers.
Experimental Setup: The max scenario targets peak platform-limited performance, whereas fix targets a prescribed 9000 FPS rate for tiled 640 × 480 video at 30 FPS.Prototype folding factors were selected to meet each scenario’s throughput requirement.
Results: 12.3 million classifications per second at 0.31 µs latency is achieved by SFC-max while drawing less than 22 W total power.All fix prototypes exceed the 9000 FPS requirement by 30%.
Maximum Throughput and Bottlenecks: 9.1 TOPS is achieved by LFC-max, while SFC-max reaches 8.2 TOPS and CNV-max reaches 2.5 TOPS under different architecture, memory, and resource bottlenecks.SFC-max is memory-bound, CNV-max is architecture-bound, and LFC-max is resource-bound.
Energy Efficiency: 583066 FPS per W is achieved by SFC-max, which outperforms the other prototypes by at least an order of magnitude in total-power energy efficiency.The board’s approximately 7 W idle consumption sets a lower bound for wall-power measurements.
Resource Efficiency: Runtime efficiency is approximately 70% for CNV, 80% for SFC, and 90% for LFC, with further gains possible by tuning folding factors across layers.FPGA resource utilization also depends on BRAM organization and the LUT-to-BRAM ratio.
Comparison to Prior Work: FINN’s designs exceed prior reported throughput on MNIST, CIFAR-10, and SVHN while maintaining classification accuracy within 3% of other low-precision approaches.The comparisons report dataset-specific throughput advantages over prior FPGA and specialized hardware designs.

6. CONCLUSION

FINN demonstrates that binarized neural networks can support high-performance FPGA image classification. Its parameterizable dataflow architecture combines high classification rates, low power and latency, and scalable resource allocation, while assuming all BNN parameters fit in one FPGA’s on-chip memory.

FINN demonstrates the promise of BNNs for high-performance image-classification implementation on FPGAs.The conclusion attributes this suitability to compact parameters and simplified arithmetic.
Its parameterizable dataflow architecture and optimizations enable high classification rates, low power and latency, and flexible scaling to larger and more complex networks.The paper identifies embedded real-time applications such as surveillance, robotics, and augmented reality as suitable use cases.
FINN assumes that all BNN parameters fit into the available on-chip memory of a single FPGA.The conclusion identifies external memory, multi-FPGA implementations, and reconfiguration as directions that could improve utility.