Source-linked AI summary
Loom: Exploiting Weight and Activation Precisions to Accelerate Convolutional Neural Networks
Sayeh Sharify, Alberto Delmas Lascorz, Kevin Siu, Patrick Judd, Andreas Moshovos
TL;DR
Bandwidth-constrained CNN inference motivates reducing data movement while improving performance and energy efficiency. Loom exploits weight and activation precision variability, achieving 4.38× speedup and 3.54× energy efficiency over DPNN in its LM1b configuration.
Problem
CNN inference has high computation and memory demands, motivating acceleration that reduces data transfers under a memory-bandwidth budget.
Source: §1 · p. 1Method
Loom exploits weight and activation precision variability, with convolutional execution scaling with both precisions and fully-connected performance with weight precision.
Source: §1 · p. 1, §3.2 · p. 4Results
4.38× speedup and 3.54× energy efficiency over DPNN for the LM1b configuration.
Source: §4.6 · p. 7Takeaways & Limitations
LM can trade off accuracy for additional improvements in performance and energy efficiency.
Source: §1 · p. 1
Abstract
from arXiv · showhide
Loom (LM), a hardware inference accelerator for Convolutional Neural Networks (CNNs) is presented. In LM every bit of data precision that can be saved translates to proportional performance gains. Specifically, for convolutional layers LM's execution time scales inversely proportionally with the precisions of both weights and activations. For fully-connected layers LM's performance scales inversely proportionally with the precision of the weights. LM targets area- and bandwidth-constrained System-on-a-Chip designs such as those found on mobile devices that cannot afford the multi-megabyte buffers that would be needed to store each layer on-chip. Accordingly, given a data bandwidth budget, LM boosts energy efficiency and performance over an equivalent bit-parallel accelerator. For both weights and activations LM can exploit profile-derived perlayer precisions. However, at runtime LM further trims activation precisions at a much smaller than a layer granularity. Moreover, it can naturally exploit weight precision variability at a smaller granularity than a layer. On average, across several image classification CNNs and for a configuration that can perform the equivalent of 128 16b x 16b multiply-accumulate operations per cycle LM outperforms a state-of-the-art bit-parallel accelerator [1] by 4.38x without any loss in accuracy while being 3.54x more energy efficient. LM can trade-off accuracy for additional improvements in execution performance and energy efficiency and compares favorably to an accelerator that targeted only activation precisions. We also study 2- and 4-bit LM variants and find the the 2-bit per cycle variant is the most energy efficient.
1 INTRODUCTION
Loom targets CNN inference on embedded systems where memory bandwidth is constrained. It exploits weight and activation precision to improve performance and energy efficiency, and the evaluation reports gains over a fixed-precision bit-parallel accelerator.
1 INTRODUCTION
- Loom targets embedded CNN inference where reducing data transfer per memory connection is paramount.Its goal is to improve performance and energy efficiency over a state-of-the-art data-parallel accelerator given a memory-bandwidth budget.Source: §1 · p. 1
- For convolutional and fully-connected layers, Loom's ideal speedup over fixed 16-bit precision is 256×.The introduction states that execution performance scales inversely with precision for both layer types.Source: §1 · p. 1
- Loom processes weights and activations bit-serially, using parallelism and data reuse to recover computation bandwidth without a wider memory interface.It uses profile-derived per-layer precisions and further trims activation precision at runtime.Source: §1 · p. 1
- 3.19× average speedup and 2.59× energy efficiency over DPNN across all layers are reported for the matched-compute configuration.With a 1% relative accuracy loss, the reported gains are 3.57× performance and 2.87× energy efficiency over DPNN.Source: §1 · p. 1
- The paper evaluates Loom on an SoC against DPNN across image-classification CNNs and studies compute-bandwidth and bits-per-cycle variants.The reported sensitivity study finds that a 2-bit-per-cycle design has the best energy efficiency, but not the best performance.Source: §1 · p. 1
2 LOOM: A SIMPLIFIED EXAMPLE
The example illustrates how Loom matches a bit-parallel engine's throughput by decomposing products into bit-serial operations and reusing inputs. Its performance advantage depends on the precisions that the layer requires.
2 LOOM: A SIMPLIFIED EXAMPLE
- The section demonstrates Loom processing a fully-connected layer with 2-bit activations and weights.Its bit-parallel comparison engine accepts two new 2-bit weights and/or activations per cycle.Source: §2 · pp. 1–2
- For the 2-bit example, Loom produces eight 1b × 1b products per cycle, matching the bit-parallel engine's two 2b × 2b products per cycle.The Loom engine uses four bit-serial subunits that share activation inputs by column and weight inputs by row.Source: §2 · p. 2
- The example processes 32 1b × 1b products in five cycles, then produces eight such terms per cycle in steady state.The left subunits can load another set of weights while the right subunits finish the current outputs.Source: §2 · p. 2
- For fully-connected layers, Loom's performance advantage over bit-parallel hardware scales with the ratio of baseline weight precision to required weight precision.The example explains that activation precision below 16 bits does not improve performance in this setting because each column needs 16 cycles to load weights, though it can improve energy efficiency.Source: §2 · p. 2
- For convolutional layers, weight reuse across windows lets Loom exploit reductions in both weight and activation precision.The weights are shared across subunits in a row, and successive activation bits are processed over the activation-precision cycles.Source: §2 · p. 2
3 LOOM ARCHITECTURE
The architecture pairs a fixed-precision bit-parallel baseline with Loom's bit-serial SIP array. Loom's data supply and processing organization exploit layer precision, while cascading and auxiliary units address layers and operations that would otherwise leave resources underused.
3.1 Data Supply and Baseline System
- DPNN is a 16-bit fixed-point data-parallel baseline with k inner-product units, each processing a different filter and producing a partial output activation per cycle.In the described configuration, N = 16 and k = 8.Source: §3.1 · p. 2
3.2 Loom
- Loom uses 2K SIPs in a 128 × 16 grid, sharing weight buses across rows and activation buses across columns.The array processes weights and activations bit-serially and is supplied by a 2Kb weight bus and a 256b activation input bus.Source: §3.2 · p. 3
- For convolutional layers, Loom processes layer weight and activation bits over PLa × PLw cycles; for fully-connected layers, it takes PLw × 16 cycles.Against DPNN, the stated performance ratios are 256/(PLa × PLw) for convolutional layers and 16/PLw for fully-connected layers when the relevant precisions are below 16 bits.Source: §3.2 · p. 4
- Runtime activation precision reduction examines groups of 256 concurrent activations and uses a leading-one detector to find sufficient precision.Per-bit OR trees mark positions where any activation has a 1.Source: §3.2 · p. 4
- SIP cascading chains row units so layers with fewer outputs can split an output computation across bit slices and reduce partial outputs.The text identifies layers with at least 2K outputs as the condition for assigning an output activation to every SIP without cascading.Source: §3.2 · p. 4
- Loom includes max-pooling units and an activation functional unit, while processing multiple activation bits per cycle trades some performance benefit for fewer SIPs and lower area overhead.The evaluated LM2b and LM4b configurations process multiple bits per cycle.Source: §3.2 · p. 4
4 EVALUATION
The evaluation measures Loom’s performance, energy, and area against DPNN and Stripes using profile-derived precisions and practical memory settings. It finds gains from finer-grained weight precision, while performance and area trade-offs vary by configuration.
4.1 Methodology
- The evaluation uses cycle-accurate simulation for execution time and layout-based energy and area measurements, with a 1GHz clock and a final single-channel LPDDR4-4267 setting.Initial tests assume activations fit on-chip and impose no off-chip weight-bandwidth constraint.Source: §4.1 · p. 5
4.2 Weight and Activation Precisions:
- Profile-derived precisions without top-1 accuracy loss range from 5–13 bits for CVL activations, 10–12 bits for CVL weights, and 7–10 bits for FCL weights.At 99% relative top-1 accuracy, CVL activation and weight precision can be as low as 4 and 10 bits, respectively.Source: §4.2 · p. 5
4.3 Performance and Energy Efficiency
- On average, LM1b outperforms DPNN by more than 3× while being more than 2.5× energy efficient; LM1b also consistently outperforms Stripes and DStripes in performance.LM1b is more energy efficient than DStripes except for GoogleNet, where it is within 2%.Source: §4.3 · p. 5
4.4 Area Overhead
- LM1b needs 1.34× DPNN’s area for a 3.19× speedup, while LM2b and LM4b reduce area overhead to 1.25× and 1.16× with 3.05× and 2.74× execution-time improvements.The area measurements use post-layout results.Source: §4.4 · p. 5
4.5 Scaling
- LM’s 1MB activation memory is smaller than DPNN’s 2MB, and the scaling comparison distinguishes compute-bound convolutional layers from off-chip-bound fully-connected layers.The scaling study sizes memory so most layers fit on-chip; VGG19’s 10MB activation requirement is described as impractical for embedded systems.Source: §4.5 · pp. 6–7
- LM outperforms DPNN at every shown design point and reaches real-time processing at the 32 configuration, but its relative advantage falls as larger configurations increase underutilization.LM beats DStripes through the 128 configuration; they are nearly tied at 256, while DStripes is faster at 512. At 256, LM energy efficiency suffers, although off-chip traffic energy is excluded and traffic is on average 0.61× less with LM.Source: §4.5 · pp. 6–7, Figure 5 · p. 7
4.6 Per Group Weight Precisions
- Finer-grained weight precisions can be detected at runtime or statically conveyed through per-group metadata, extending the per-layer precision approach.Table 3 reports average effective weight precision per layer for groups of 16 weights.Source: §4.6 · p. 7, Table 3 · p. 7
- 4.38×, 4.20×, and 3.76× speedups over DPNN, with 3.54×, 3.95×, and 3.94× energy efficiency, for LM1b, LM2b, and LM4b using effective weight precisions.These estimates assume performance scales linearly with weight precision.Source: Table 4 · p. 7, §4.6 · p. 7
5 RELATED WORK
The related-work discussion contrasts Loom with accelerators that exploit activation precision, weight precision, or ineffectual data. It emphasizes Loom’s coverage of both convolutional and fully-connected layers without requiring the whole network on-chip.
5 RELATED WORK
- Loom is compared with Stripes extended by dynamic prediction reduction, while the discussion notes that combining Pragmatic with Loom may be prohibitively expensive due to per-SIP costs.Loom’s activation-precision trimming at runtime is distinguished from hardwired implementations.Source: §5 · p. 7
- Proteus reduces memory footprint and bandwidth through per-layer precisions but requires crossbars per input weight, whereas Loom does not need crossbars.Loom also does not require the whole network to fit on-chip or hardwire precisions, and it trims activation precision at runtime.Source: §5 · p. 7
- Unlike Pragmatic, which does not improve fully-connected-layer performance, Loom targets both fully-connected and convolutional layers and benefits from all inputs, whether ineffectual or not.Pragmatic’s convolutional-layer performance depends only on the number of activation bits that are 1.Source: §5 · pp. 7–8
6 CONCLUSION
Loom’s conclusion is that input precision governs execution time in both convolutional and fully-connected layers, and accuracy can be traded for performance and energy efficiency.
6 CONCLUSION
- Loom’s execution time for convolutional and fully-connected layers scales inversely with the precision used to represent input data.The paper notes that Loom can trade off accuracy against performance and energy efficiency on the fly, and identifies exploiting weight sparsity as future work.Source: §6 · p. 8