Source-linked AI summary
Espresso: Efficient Forward Propagation for BCNNs
Fabrizio Pedersoli, George Tzanetakis, Andrea Tagliasacchi
TL;DR
DNN prediction can demand too much memory and computation for embedded or low-power devices. Espresso applies binary representations and bit-packed computation in a self-contained forward-propagation framework for dense and convolutional networks; it reports faster execution than existing optimized binary implementations.
Problem
DNN prediction’s memory and computational demands make effective use on embedded and low-power devices difficult.
Source: §1 · p. 1Method
Results
Espresso reports a consistent ≈68× speed-up over BinaryNet on MNIST and supports CNNs while outperforming state-of-the-art MLP implementations.
Source: §6.2 · p. 7, §7 · p. 8Takeaways & Limitations
Espresso combines binary dense and convolutional layers in a lightweight framework designed to simplify deployment on mobile or embedded devices.
Source: §1 · p. 2BinaryNet requires parameters to be binarized and packed on every forward call, with packing cost closely related to multiplication cost.
Source: §6.2 · p. 7
Abstract
from arXiv · showhide
There are many applications scenarios for which the computational performance and memory footprint of the prediction phase of Deep Neural Networks (DNNs) needs to be optimized. Binary Neural Networks (BDNNs) have been shown to be an effective way of achieving this objective. In this paper, we show how Convolutional Neural Networks (CNNs) can be implemented using binary representations. Espresso is a compact, yet powerful library written in C/CUDA that features all the functionalities required for the forward propagation of CNNs, in a binary file less than 400KB, without any external dependencies. Although it is mainly designed to take advantage of massive GPU parallelism, Espresso also provides an equivalent CPU implementation for CNNs. Espresso provides special convolutional and dense layers for BCNNs, leveraging bit-packing and bit-wise computations for efficient execution. These techniques provide a speed-up of matrix-multiplication routines, and at the same time, reduce memory usage when storing parameters and activations. We experimentally show that Espresso is significantly faster than existing implementations of optimized binary neural networks ($\approx$ 2 orders of magnitude). Espresso is released under the Apache 2.0 license and is available at http://github.com/fpeder/espresso.
1 INTRODUCTION
DNNs’ memory and computational demands make them difficult to use on embedded and low-power devices. Espresso addresses this with an optimized binary forward-propagation framework for dense and convolutional networks, focusing on testing rather than training.
Motivation
- Reducing DNN computation can improve energy efficiency and free mobile-device resources for tasks such as speech recognition and analysis.The introduction also identifies memory as a constrained resource on mobile platforms.Source: §1 · p. 1
Efficient deep neural networks
- Espresso follows a software-based efficiency strategy by reducing network memory footprint and computation through quantized parameters stored as small integers.The passage contrasts this approach with relying on specialized hardware.Source: §1 · p. 1
- Binary DNNs encode parameters and activations as 1-bit values, reducing memory use and enabling faster execution at a relatively small accuracy cost.The paper identifies the availability of a flexible, end-to-end framework emphasizing computational efficiency as a need for further research and practical application.Source: §1 · p. 2
Contributions
- Espresso provides an optimized BDNN framework for dense and convolutional layers, targeting forward propagation rather than back-propagation.The framework is self-contained, with no external dependencies, to simplify practical deployment.Source: §1 · p. 2
2 RELATED WORK
Related work addresses DNN efficiency through hardware, simpler or pruned models, weight sharing, and quantization. Binary networks extend quantization to weights and activations, while prior work reports optimized binary multiplication and training results.
Improving DNN performance
- DNN efficiency approaches include dedicated hardware, simpler architectures, pruning weights or filters, weight sharing, and quantized weights (Han et al., 2016; Iandola et al., 2016; Guo et al., 2016; Li et al., 2016; Han et al., 2015; Courbariaux et al., 2014).Source: §2 · p. 2
Quantized networks
- Quantization research ranges from 10-bit Maxout weights to ternary schemes, including fixed-point multiplication and ternary networks (Courbariaux et al., 2014; Zhu et al., 2016).Source: §2 · p. 2
Binary Deep Neural Networks (BDNN)
- Prior work on BinaryNet constrains weights and activations to {−1, +1}, reporting near-state-of-the-art accuracy for MLPs on MNIST and CNNs on CIFAR-10 (Hubara et al., 2016).Its binary matrix multiplication was reported as 7× faster than the non-optimized baseline and almost 2× faster than Theano.Source: §2 · p. 2
3 THE ESPRESSO FRAMEWORK
Espresso supports forward propagation across CPU, GPU, and GPU-optimized implementations, including hybrid networks. Its GPU performance approach combines bit-packed layers, memory management, and custom CUDA kernels.
The Espresso framework
- Espresso provides forward-propagation tools for DNNs, with particular emphasis on convolutional networks and GPU-optimized BDNN architectures.The framework also supports floating-point CPU and GPU counterparts on heterogeneous architectures.Source: §3 · p. 3
Hybrid DNNs
- Espresso offers CPU, GPU, and GPU-optimized variants, which can be converted between and interconnected to form hybrid networks.The CPU and GPU variants use single-precision floating-point data and do not apply binary optimization; the GPU-optimized variant targets BDNNs.Source: §3 · p. 3
The computational bottleneck: dot products
- Dense linear algebra relies on dot products implemented with floating-point multiply-and-add operations, while BDNNs replace these with bitwise operations.Source: §3 · p. 3
Technical highlights
- Espresso’s technical contributions are bit-packing in network layers, improved memory layout and management, and custom optimized CUDA kernels.Bit-packed layers avoid expensive memory rearrangements; a custom allocator pre-allocates GPU memory to replace malloc and free calls.Source: §3 · p. 3
4 BINARY DEEP NEURAL NETWORKS (BDNNS)
BDNNs replace floating-point operations with bitwise computation and use binary weights and activations. Espresso’s design relies on bit-packing, input bit-planes, and training procedures that retain floating-point precision for updates.
4 BINARY DEEP NEURAL NETWORKS (BDNNS)
- BDNNs replace computationally intensive floating-point multiply-and-add operations with XNOR multiplication and bit-count addition.The paper describes XNOR as a simpler instruction and notes that one operation can process multiple 64-bit blocks of dot products.Source: §4 · p. 3
4.1 NETWORK BINARIZATION
- BDNN layers binarize weights and activations to {−1, +1}, encoded in hardware as {0, 1} with −1 mapped to 0 and +1 to 1.The paper selects an activation function for efficient implementation.Source: §4.1 · pp. 3–4
4.2 BIT-PACKING
- Storing binary weights in 64-bit words reduces memory use by 32× and lets a 64-element dot product use one XNOR and one bit-count.The resulting computation builds binary matrix-matrix and matrix-vector operations.Source: §4.2 · p. 4
4.3 INPUT DATA BINARIZATION
- Fixed-precision input dot products can use binary optimization by splitting inputs into bit-planes and summing their contributions with corresponding weights.Source: §4.3 · p. 4
4.4 TRAINING
- During training, gradients use binary weights but accumulate in floating-point precision, with a straight-through estimator and weight clipping to [−1, 1].The estimator back-propagates 1 when the floating-point argument has |x| ≤ 1 and 0 otherwise.Source: §4.4 · p. 4
5 ESPRESSO ARCHITECTURE
Espresso organizes networks as layers that transform tensors, with tensor layout and bit-packing chosen to support efficient access. Its layer implementations use unrolling, matrix multiplication, and specialized handling for binary data and zero-padding.
5 ESPRESSO ARCHITECTURE
- Espresso networks concatenate layers, each of which processes an input tensor and produces an output tensor.Tensors store inputs, weights, and activations.Source: §5 · p. 4
5.1 TENSORS
- Bit-packing follows the channel dimension when L > 1 and the column dimension when L = 1, supporting efficient convolutional access.The layout permits retrieval of pixel neighborhoods without changing the tensor layout.Source: §5.1 · p. 5
5.2 LAYERS
- Convolution is computed as matrix multiplication after unrolling sliding-volume data into rows, then lifting the result back to a tensor automatically at zero cost.Figure 1 illustrates the unrolling and lifting operations.Source: Figure 1 · p. 5, §5.2 · p. 6
- CPU matrix-vector multiplication uses OpenBLAS, while GPU kernels build on MAGMA and use register blocking to improve throughput.The GPU opt variant replaces blocks of 64, or 32 for GPU opt 32, single-precision multiply-add operations with XNOR and bit-count on packed tensors.Source: §5.2 · p. 6
- GPU opt convolution treats zero-padded inputs as binary by considering zero a minus one, then corrects corner cases in post-processing.The correction matrix is computed once when the layer is loaded.Source: §5.2 · p. 6
- A network is loaded from a parameters file specifying layer formats and weights; a utility script converts BinaryNet-trained parameters to Espresso format.Source: §5.2 · p. 6
6 EVALUATION
Espresso is evaluated on dense matrix multiplication, MNIST MLP prediction, and CIFAR-10 CNN prediction, with batch size one for image-level testing. Results report faster execution and lower memory use, with different comparisons for each workload.
6 EVALUATION
- The evaluation averages 100 runs on GTX 960 and dual-Xeon X5660 hardware; MNIST and CIFAR-10 image tests use batch size one and average per-image forward time.The workloads are 8192 × 8192 matrix multiplication, MNIST MLP prediction, and CIFAR-10 CNN prediction; comparisons include BinaryNet, neon, and Espresso CPU/GPU variants.Source: §6 · pp. 6–7
6.1 BINARY DENSE MATRIX MULTIPLICATION
- ≈8× faster dense matrix multiplication than BinaryNet is reported for Espresso.The discussion attributes much of the gain to optimized kernels and register blocking; 64-bit packing gives an additional performance improvement over 32-bit packing.Source: §6.1 · p. 7, Table 1 · p. 7
6.2 MULTI-LAYER PERCEPTRON ON MNIST
- ≈68× speed-up over BinaryNet is reported for MNIST MLP classification execution time.Espresso’s GPU opt implementation also uses 4.57 MB versus 140.6 MB for the non-binary-optimized implementation, a ≈31× memory saving.Source: §6.2 · pp. 7–8
- Espresso’s specific layers pack parameters once during network loading, avoiding BinaryNet’s repeated parameter binarization and packing on every forward call.The passage notes that packing cost is closely related to multiplication cost.Source: §6.2 · pp. 7–8
- Binary-optimizing the first layer uses input bit-planes and weighted recombination, with an overall ≈3× boost over a network without that optimization.Source: §6.2 · p. 8
6.3 CONVOLUTIONAL NEURAL NETWORK ON CIFAR-10
- The GPU opt implementation reports an ≈5× performance gain relative to GPU, with gains slightly smaller than those discussed for MLP in Section 6.2.Convolutional-layer outputs are significantly larger than MLP dense-layer outputs, so computing bit-packing sign-activation requires more computational effort.Source: §6.3 · p. 8
- The GPU opt implementation requires 1.73 MB of memory, versus 53.54 MB for the non-binary-optimized implementation, resulting in an approximately 31× memory saving.Source: §6.3 · p. 8
7 CONCLUSIONS
Espresso is a self-contained forward-propagation framework for traditional DNNs and BCNNs, supporting CPU and GPU deployment. It supports CNNs and outperforms state-of-the-art MLP implementations.
7 CONCLUSIONS
- Espresso is a highly optimized, self-contained forward-propagation framework for traditional DNNs and BCNNs, supporting deployment on CPU and GPU.The authors describe it as highly efficient and light-weight.Source: §7 · p. 8
- Unlike BinaryNet and Nervana/neon implementations limited to MLPs, Espresso supports CNNs while outperforming state-of-the-art MLP implementations.Source: §7 · p. 8
- Espresso's GPU-specific CUDA kernels, careful memory allocation, and bit-packing yield considerable performance improvements.Source: §7 · p. 8
- Future work includes adding training capabilities and performing additional performance comparisons on larger standard datasets.Source: §7 · p. 8