Source-linked AI summary
Analyzing and Mitigating the Impact of Permanent Faults on a Systolic Array Based Neural Network Accelerator
Jeff Zhang, Tianyu Gu, Kanad Basu, Siddharth Garg
TL;DR
The paper addresses the vulnerability of systolic-array DNN accelerators to permanent faults in high-defect-rate technologies. It proposes fault-aware pruning with optional retraining, showing operation at up to 50% faulty MACs with small accuracy losses and no run-time overhead, while reducing retraining to 12 minutes.
Problem
Permanent faults can significantly reduce TPU classification accuracy, making fault tolerance important as technology scaling increases defect rates.
Method
FAP prunes DNN connections mapped to faulty MACs, while FAP+T additionally retrains the pruned network to recover accuracy.
Results
Up to 50% faulty MACs are supported with classification-accuracy drops ranging from 0.1% for TIMIT to 8% for AlexNet, without run-time performance overhead.
Takeaways & Limitations
FAP and FAP+T enable fault-tolerant TPU operation, with FAP+T’s one-time retraining reduced to about 12 minutes and amortized over the TPU lifetime.
Takeaways & Limitations
The proposed design addresses manufacturing-defect or process-variation-induced permanent faults, while aging-related faults and larger neural networks remain future work.
Abstract
from arXiv · showhide
Due to their growing popularity and computational cost, deep neural networks (DNNs) are being targeted for hardware acceleration. A popular architecture for DNN acceleration, adopted by the Google Tensor Processing Unit (TPU), utilizes a systolic array based matrix multiplication unit at its core. This paper deals with the design of fault-tolerant, systolic array based DNN accelerators for high defect rate technologies. To this end, we empirically show that the classification accuracy of a baseline TPU drops significantly even at extremely low fault rates (as low as $0.006\%$). We then propose two novel strategies, fault-aware pruning (FAP) and fault-aware pruning+retraining (FAP+T), that enable the TPU to operate at fault rates of up to $50\%$, with negligible drop in classification accuracy (as low as $0.1\%$) and no run-time performance overhead. The FAP+T does introduce a one-time retraining penalty per TPU chip before it is deployed, but we propose optimizations that reduce this one-time penalty to under 12 minutes. The penalty is then amortized over the entire lifetime of the TPU's operation.
1. Introduction
DNNs deliver strong application performance but require specialized acceleration, while permanent faults can sharply reduce TPU accuracy even at very low rates. The paper proposes fault-aware pruning strategies to preserve accuracy at high fault rates without run-time overhead.
- DNN acceleration: DNNs are computationally expensive because their layers repeatedly perform matrix multiplication or convolution followed by nonlinear activation.Systolic arrays implement these operations with neighboring processing elements and synchronized data streaming.
- DNN acceleration: The Google TPU uses a 256 × 256 systolic grid of MAC units and provides 30× to 80× greater performance than CPU- or GPU-based servers.The paper uses this TPU architecture as its baseline while targeting the broader class of systolic-array DNN accelerators.
- Motivation: Four faulty MAC units among approximately 64K sharply reduce TPU classification accuracy, challenging the assumption that DNNs inherently tolerate permanent faults.The result comes from detailed gate-level simulations of stuck-at permanent faults.
- Contributions: Fault-aware pruning (FAP) and fault-aware pruning plus retraining (FAP+T) adapt DNN connections to faulty MACs while retaining the TPU’s run-time performance.FAP uses bypass circuitry, whereas FAP+T additionally retrains the pruned network to restore accuracy.
2. Related Work
Prior fault-tolerance techniques for memories, logic, routing, and systolic arrays generally target hardware-level recovery but can require substantial complexity or performance overhead. FAP and FAP+T instead adapt the DNN itself, avoiding performance penalties with negligible area overhead.
- Prior fault-tolerance techniques: ECC and redundancy-based techniques for memory, logic, and routing faults do not directly apply to systolic arrays.The paper places its approach within fault-tolerant digital-system design while addressing systolic-array-specific constraints.
- Systolic-array fault tolerance: Earlier systolic-array solutions eliminate rows or columns around faulty processing elements, causing performance penalties at high defect rates or requiring complex hardware.More sophisticated designs reduce the penalty at the cost of greater design complexity.
- Application-aware adaptation: FAP and FAP+T prune and retrain the DNN to adapt to TPU faults, achieving no performance penalty and negligible area overhead.This application-aware approach differs from methods that preserve the original application unchanged in faulty hardware.
3. Background and Preliminaries
DNNs stack parameterized computation layers, while TPU acceleration maps matrix operations and convolutions onto a systolic array of MAC units. The TPU processes batches through streamed partial sums across this array.
- 3.1. Deep Neural Networks: A DNN layer multiplies prior activations by a weight matrix, adds biases, and applies an element-wise activation function.Convolutional layers provide a commonly used specialized case of fully connected computation.
- 3.2. DNN Acceleration on TPU: A TPU uses an N × N systolic array of MAC units to perform the matrix multiplication and convolution operations central to DNN execution.Each weight is loaded into a specific MAC according to its row and column mapping.
- 3.2. DNN Acceleration on TPU: Each TPU column accumulates products from streamed inputs, with successive columns receiving delayed copies and producing successive output elements.For a batch of B inputs, multiplication by an N × N matrix takes 2N + B clock cycles.
4. Motivational Analysis: Impact of Permanent Faults on TPU
The paper’s gate-level analysis shows that permanent stuck-at faults can sharply reduce TPU classification accuracy even at very low fault counts. Bypassing faulty columns is possible but becomes impractical as faults increase.
- 4. Motivational Analysis: Impact of Permanent Faults on TPU: With four faulty MACs, approximately 0.005% of the array, TIMIT accuracy falls from 74.13% to 39.69%.The study synthesized the TPU to a 45 nm gate-level netlist, inserted data-path stuck-at faults, and evaluated MNIST and TIMIT.
- 4. Motivational Analysis: Impact of Permanent Faults on TPU: Stuck-at faults often affect higher-order MAC-output bits, creating large matrix-vector-product errors and faulty activations with much higher magnitudes than golden outputs.The activation comparison is reported for the final layer of the TIMIT network.
- 4. Motivational Analysis: Impact of Permanent Faults on TPU: Bypassing entire columns containing faulty MACs would incur an unacceptable performance penalty as the number of permanent faults increases.This motivates a fault-aware design rather than simply removing affected columns.
5. Proposed Fault-Tolerant TPU Design
The proposed design exploits the static mapping between DNN weights and TPU MAC units to prune weights associated with faulty MACs. FAP+T further retrains the remaining weights while preserving the pruning pattern.
- 5. Proposed Fault-Tolerant TPU Design: The weight-to-MAC mapping is formalized for fully connected layers as r(i, j) = j%N and c(i, j) = i%N, with oversized matrices blocked into N × N submatrices.For convolutional layers, rows sum along input channels and columns compute different output channels.
- 5.1. Fault-Aware Pruning (FAP): Fault-aware pruning sets every weight mapped to a faulty MAC to zero, using the TPU fault map to identify affected weights.Multiple weights may map to one MAC, so one faulty MAC can cause multiple weights to be pruned.
- 5.1. Fault-Aware Pruning (FAP): Hardware pruning uses a bypass path that skips a faulty MAC’s contribution to the column sum, adding approximately 9% area overhead.Loading a zero weight into a faulty MAC is not equivalent to bypassing its contribution.
- 5.2. FAP+T Solution: FAP+T retrains only the unpruned weights while forcing pruned weights to remain zero, producing optimized weights intended to improve accuracy over FAP.The retraining time and accuracy trade off through MAX EPOCHS; setting it to zero is equivalent to FAP.
6. Empirical Evaluation
The evaluation uses three DNN benchmarks and gate-level fault injection to measure permanent-fault effects on a systolic-array TPU. FAP and FAP+T preserve accuracy at high fault rates, while FAP+T reduces its retraining cost through an epoch limit.
- Benchmarks: The study evaluates MNIST, TIMIT, and AlexNet, covering fully connected and convolutional-plus-fully-connected DNN architectures.All networks were trained from scratch and reached state-of-the-art classification accuracy.
- Methodology: Permanent-fault effects are measured by scheduling the benchmarks on a 256 × 256 systolic array and performing ten randomized gate-level fault-injection experiments.The synthesized array runs at 658 MHz and consumes 19.7 W of dynamic power.
- Results: At 50% faulty MACs, FAP+T stays close to baseline accuracy for MNIST and TIMIT, whereas FAP accuracy decreases; for AlexNet, FAP+T incurs an 8% accuracy drop.For MNIST and TIMIT, both methods tolerate up to 25% faulty MACs with negligible accuracy reduction.
- Results: A 5-epoch AlexNet retraining run with 25% faulty MACs is marginally below the 25-epoch result, reducing retraining time fivefold from 1 hour to 12 minutes.The epoch limit controls FAP+T’s one-time retraining overhead per TPU chip.
7. Conclusion
The paper presents FAP and FAP+T as fault-tolerance strategies for systolic-array DNN accelerators, achieving operation at fault rates up to 50%. They avoid run-time performance overhead, while FAP+T requires a one-time retraining cost and the study identifies aging-related faults and larger networks as future work.
- Conclusion: FAP and FAP+T support TPU operation at fault rates as high as 50%, with accuracy drops ranging from 0.1% for TIMIT to 8% for AlexNet.The reported techniques avoid run-time performance overhead.
- Conclusion: FAP+T adds a one-time retraining overhead of about 12 minutes per TPU chip for AlexNet, amortized over the TPU’s lifetime.The conclusion presents this cost as a deployment-time penalty rather than a run-time overhead.
- Future work: Future work will address aging-related faults and apply the methodology to larger neural networks.These directions define the study’s stated scope boundary.