PaperlayerView exact revision on arXiv

Source-linked AI summary

FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation

Le-Ha Hoang, Muhammad Abdullah Hanif, Muhammad Shafique

arXiv:1912.00941v1cs.LGstat.ML

TL;DR

Hardware faults can cause significant DNN accuracy degradation, despite assumptions that DNNs tolerate such errors, creating a need for improved resilience. The paper analyzes fault effects and replaces unbounded activations with clipped versions whose thresholds are systematically defined without training data or weight and bias changes. 69.49% improvement for VGG-16 and 18.19% for AlexNet classification accuracy is reported at 5 × 10−7 fault rate versus baseline variants.

  • Problem

    Hardware faults can cause significant DNN accuracy degradation, despite assumptions that DNNs tolerate such errors, creating a need for improved resilience.

  • Method

    The paper analyzes fault effects and replaces unbounded activations with clipped versions whose thresholds are systematically defined without training data or weight and bias changes.

  • Results

    69.49% improvement for VGG-16 and 18.19% for AlexNet classification accuracy is reported at 5 × 10−7 fault rate versus baseline variants.

  • Takeaways & Limitations

    The proposed clipped-activation technique significantly improves DNN resilience to hardware faults, including 68.92% improvement for VGG-16 on CIFAR-10 at 10−5 fault rate.

Abstract

from arXiv · show

Deep Neural Networks (DNNs) are widely being adopted for safety-critical applications, e.g., healthcare and autonomous driving. Inherently, they are considered to be highly error-tolerant. However, recent studies have shown that hardware faults that impact the parameters of a DNN (e.g., weights) can have drastic impacts on its classification accuracy. In this paper, we perform a comprehensive error resilience analysis of DNNs subjected to hardware faults (e.g., permanent faults) in the weight memory. The outcome of this analysis is leveraged to propose a novel error mitigation technique which squashes the high-intensity faulty activation values to alleviate their impact. We achieve this by replacing the unbounded activation functions with their clipped versions. We also present a method to systematically define the clipping values of the activation functions that result in increased resilience of the networks against faults. We evaluate our technique on the AlexNet and the VGG-16 DNNs trained for the CIFAR-10 dataset. The experimental results show that our mitigation technique significantly improves the resilience of the DNNs to faults. For example, the proposed technique offers on average 68.92% improvement in the classification accuracy of resilience-optimized VGG-16 model at 1e-5 fault rate, when compared to the base network without any fault mitigation.

I. INTRODUCTION

DNN hardware faults can sharply reduce accuracy in safety-critical applications, motivating fault tolerance without training data, redundancy, or costly reliability features. The paper analyzes fault impacts and introduces clipped activations with a systematic threshold methodology, improving AlexNet and VGG-16 resilience.

  • I. INTRODUCTION: Hardware faults in DNN accelerators can cause catastrophic effects in safety-critical applications, while accuracy drops significantly as fault rates grow.The analysis concerns faults such as soft errors, aging, and manufacturing-induced defects affecting DNN reliability.
  • I. INTRODUCTION: Fault impact depends strongly on whether corrupted weights belong to sensitive neurons, contradicting the assumption that DNNs broadly forgive hardware errors.The paper identifies a need for resilience at nominal fault rates.
  • I. INTRODUCTION: The paper performs comprehensive analysis of hardware-fault effects on DNN accuracy and intermediate outputs to guide an efficient reliability mechanism.The evaluation covers AlexNet and VGG-16.
  • I. INTRODUCTION: The proposed clipped activation bounds intermediate outputs, while its threshold-selection method requires neither training data nor weight and bias modification.These contributions target resilience with minimal overhead and without redundancy or costly reliability features.
  • I. INTRODUCTION: 69.49% improvement in VGG-16 and 18.19% in AlexNet classification accuracy at 5 × 10−7 fault rate versus baseline networks.The comparison uses variants without error mitigation as baselines.

II. BACKGROUND: AN OVERVIEW OF DNNS

CNNs combine convolutional and fully connected layers whose activations support nonlinear computation, but memory bit flips can produce layer-dependent and intensity-dependent accuracy degradation. Per-layer fault injection exposes monotonic accuracy loss, low-rate masking, and high-intensity faulty activations.

  • II. BACKGROUND: AN OVERVIEW OF DNNS: A CNN uses convolutional layers for feature extraction and fully connected layers toward the end for classification, with activation layers following computation.Convolutional layers may also be followed by pooling layers.
  • III. ERROR RESILIENCE ANALYSIS OF DEEP NEURAL NETWORKS: The study uses per-layer random bit-flip injection in parameter memories to measure layer sensitivity and effects on output activations.The framework analyzes memory faults in the stored DNN parameters.
  • III. ERROR RESILIENCE ANALYSIS OF DEEP NEURAL NETWORKS: Accuracy decreases monotonically with increasing fault rate because critical locations become more likely to experience faults.The resilience analysis examines CONV-1, CONV-5, and FC-1 layers of AlexNet on CIFAR-10.
  • III. ERROR RESILIENCE ANALYSIS OF DEEP NEURAL NETWORKS: At low fault rates, accuracy remains near baseline before dropping sharply because faults may miss critical bits or be masked within the network.The fault-rate boundary differs across layers due to parameter counts and intervening layers.
  • III. ERROR RESILIENCE ANALYSIS OF DEEP NEURAL NETWORKS: Higher fault rates produce higher-intensity output activations across layers, because MSB bit flips can increase near-zero weights to large magnitudes.These activation distributions motivate suppressing unusually large intermediate outputs.

IV. OUR MITIGATION TECHNIQUE FOR IMPROVING FAULT TOLERANCE OF DEEP NEURAL NETWORKS

The mitigation methodology profiles activations, replaces unbounded functions with clipped variants, and fine-tunes layer-specific thresholds. It uses a small validation subset rather than the training dataset and produces a fault-tolerant DNN with optimized thresholds.

  • IV. OUR MITIGATION TECHNIQUE FOR IMPROVING FAULT TOLERANCE OF DEEP NEURAL NETWORKS: The methodology is independent of the training dataset and requires only a small validation subset to tune clipping thresholds.It is designed around the observation that high-magnitude faulty activations can dominate results and cause misclassification.
  • IV. OUR MITIGATION TECHNIQUE FOR IMPROVING FAULT TOLERANCE OF DEEP NEURAL NETWORKS: The method profiles each layer’s maximum activation value using a validation subset to initialize clipping thresholds.The profiling step extracts ACTmax at every layer output.
  • IV. OUR MITIGATION TECHNIQUE FOR IMPROVING FAULT TOLERANCE OF DEEP NEURAL NETWORKS: The approach replaces unbounded activation functions with clipped variants and then fine-tunes their thresholds using an efficient method.Threshold tuning is performed layer by layer.
  • IV. OUR MITIGATION TECHNIQUE FOR IMPROVING FAULT TOLERANCE OF DEEP NEURAL NETWORKS: The final methodology repeatedly fine-tunes clipping thresholds for each layer, producing a fault-tolerant DNN with optimized activation thresholds.The resilience metric is used during threshold tuning.

A. The Clipped Activation Function

The paper introduces a clipped ReLU that maps high-intensity activations to zero and uses resilience evaluation to select clipping thresholds.

  • A. The Clipped Activation Function: Clipped ReLU maps high-intensity, potentially faulty activation values to zero using a layer-specific threshold.The input is retained within the allowed range, while values beyond the threshold are treated as faulty.
  • A. The Clipped Activation Function: The approach can be extended to other activation functions, including Leaky-ReLU.
  • A. The Clipped Activation Function: The methodology evaluates resilience across fault rates with AUC and examines how threshold choices affect the network.AUC summarizes accuracy versus normalized fault rate, while the threshold sweep is illustrated for AlexNet CONV-4 on CIFAR-10.
  • A. The Clipped Activation Function: AUC peaks at an intermediate threshold before decreasing, so the peak identifies the threshold offering the highest resilience within the predefined fault range.The unbounded-activation case has significantly lower AUC than the clipped-activation curve.

C. Threshold Fine-Tuning Algorithm

The threshold fine-tuning algorithm searches below ACTmax for the threshold region with the highest AUC and iteratively narrows that region.

  • C. Threshold Fine-Tuning Algorithm: The algorithm initializes the search interval as [0, ACTmax] because the AUC peak lies below ACTmax.It divides the interval into three equally sized sub-intervals and evaluates four boundary thresholds.
  • C. Threshold Fine-Tuning Algorithm: At each iteration, the algorithm retains the region around the boundary with maximum AUC and discards the remaining regions.The updated interval is repeatedly subdivided for further search.
  • C. Threshold Fine-Tuning Algorithm: Search stops after a defined iteration count or when adjacent AUC differences fall below δ after at least M iterations.The detailed procedure is identified as Algorithm 1.
  • C. Threshold Fine-Tuning Algorithm: The procedure takes the modified pre-trained DNN from Step 2 as its input.

1 BEGIN ALGORITHM

The algorithm evaluates candidate thresholds and their AUC values before selecting an updated search interval.

  • 1 BEGIN ALGORITHM: AUC_Calculation evaluates the candidate thresholds in the current search interval.
  • 1 BEGIN ALGORITHM: The initial branch handles the first candidate separately.

16 END ALGORITHM

The interval-search routine returns the narrowed search region and threshold, while AUC calculation evaluates the model at each candidate threshold.

  • 16 END ALGORITHM: Interval_Search receives four candidate thresholds and their AUC values, then returns the selected region and threshold.
  • 16 END ALGORITHM: AUC_Calculation evaluates the model using each candidate threshold and produces the corresponding AUC values.

A. Experimental setup

The evaluation uses modified AlexNet and VGG-16 models trained on CIFAR-10, with fault-injection experiments implemented in PyTorch.

  • The study evaluates AlexNet and VGG-16 models modified for CIFAR-10 image inputs, with baseline accuracies of 72.8% and 82.8%.
  • The fault-injection framework is implemented in Python using PyTorch and runs experiments on an Intel Core i7 processor with two NVIDIA GeForce GTX 1080 Ti GPUs.

B. Comparison with the unprotected DNNs

Clipped activations improve fault resilience over unprotected networks for both AlexNet and VGG-16 on CIFAR-10, with gains reported across fault rates and aggregate resilience.

  • At 5 × 10^-7 fault rate, resilient AlexNet reaches 69.36% accuracy versus 51.16% for the unprotected network, with 173.32% higher AUC over 0 to 1×10^-5.
  • At 1 × 10^-8 and 5 × 10^-8 fault rates, resilient AlexNet worst-case accuracy stays close to its 72.8% baseline, while the unprotected network falls to 41.93% and 13.66%.
  • 654.91% improvement in VGG-16 AUC is reported at a 5 × 10^-7 fault rate, exceeding the corresponding AlexNet improvement.
  • The comparison uses CIFAR-10 test images to prevent overlap with data used to compute activation thresholds.
  • 68.92% improvement is achieved for resilience-optimized VGG-16 at 10^-5 fault rate compared with the unprotected network.
Loading 1912.00941v1…