Source-linked AI summary
SHE: A Fast and Accurate Deep Neural Network for Encrypted Data
Qian Lou, Lei Jiang
TL;DR
Prior encrypted CNN inference methods trade substantial computing overhead for reduced accuracy. SHE uses LTFHE-based ReLU and pooling with shift-based quantization and mixed-width accumulation, achieving state-of-the-art accuracy and 76.21%–94.23% lower latency than prior LHECNNs across datasets.
Problem
Prior LHECNNs incur high computing overhead and reduced inference accuracy because polynomial approximations replace ReLU activations and pooling.
Method
SHE combines LTFHE implementations of ReLU and max pooling with logarithmic quantization and mixed bitwidth accumulation for encrypted deep neural inference.
Results
SHE achieves state-of-the-art inference accuracy and reduces inference latency by 76.21%–94.23% versus prior LHECNNs across various datasets.
Takeaways & Limitations
SHE supports deeper encrypted CNN architectures on ImageNet and LSTM architectures on Penn Treebank while improving latency and accuracy over prior LHECNNs.
Abstract
from arXiv · showhide
Homomorphic Encryption (HE) is one of the most promising security solutions to emerging Machine Learning as a Service (MLaaS). Leveled-HE (LHE)-enabled Convolutional Neural Networks (LHECNNs) are proposed to implement MLaaS to avoid large bootstrapping overhead. However, prior LHECNNs have to pay significant computing overhead but achieve only low inference accuracy, due to their polynomial approximation activations and poolings. Stacking many polynomial approximation activation layers in a network greatly reduces inference accuracy, since the polynomial approximation activation errors lead to a low distortion of the output distribution of the next batch normalization layer. So the polynomial approximation activations and poolings have become the obstacle to a fast and accurate LHECNN model. In this paper, we propose a Shift-accumulation-based LHE-enabled deep neural network (SHE) for fast and accurate inferences on encrypted data. We use the binary-operation-friendly Leveled Fast Homomorphic Encryption over Torus (LTFHE) encryption scheme to implement ReLU activations and max poolings. We also adopt the logarithmic quantization to accelerate inferences by replacing expensive LTFHE multiplications with cheap LTFHE shifts. We propose a mixed bitwidth accumulator to accelerate accumulations. Since the LTFHE ReLU activations, max poolings, shifts and accumulations have small multiplicative depth overhead, SHE can implement much deeper network architectures with more convolutional and activation layers. Our experimental results show SHE achieves the state-of-the-art inference accuracy and reduces the inference latency by 76.21% ~ 94.23% over prior LHECNNs on MNIST and CIFAR-10. The source code of SHE is available at https://github.com/qianlou/SHE.
1 Introduction
MLaaS exposes clients’ raw data to cloud servers, creating an urgent need for confidentiality. Existing homomorphic-encryption CNNs replace ReLU with low-degree polynomial activations, causing accuracy losses and preventing state-of-the-art accuracy.
- Motivation: MLaaS cloud servers can access clients’ raw data, creating privacy risks for healthcare, financial, and other sensitive information.This motivates protecting the confidentiality of client data uploaded to cloud servers.
- Limitations of prior work: HE-based CNNs use low-degree polynomial activations instead of ReLU because HE supports only polynomial computations.The cited approaches include CryptoNet, NED, n-GraphHE, Lola, and Faster Cryptonet.
- Limitations of prior work: Polynomial activations degrade inference accuracy by 3% ∼6.2% even on the MNIST handwritten-digit dataset.These accuracy losses contribute to prior methods failing to achieve state-of-the-art inference accuracy.
- Limitations of prior work: Faster Cryptonet achieves 76.72% inference accuracy on CIFAR-10.The passage presents this as an example of the accuracy limitations of prior homomorphic-encryption CNNs.
2 Background and Motivation
Encrypted inference is motivated by MLaaS data-leakage risks, while HE enables CNN computation over ciphertexts. TFHE is particularly suitable because its fast binary operations can implement ReLU activations and max pooling, addressing accuracy and latency problems caused by polynomial approximations in prior LHECNNs.
- Threat model: HE enables servers to perform CNN inference over encrypted client data, reducing exposure to leakage risks during MLaaS data transmission and result delivery.Clients send encrypted inputs to servers, while encryption and decryption map between plaintexts and ciphertexts using public and private keys.
- TFHE: TFHE supports fast binary operations over encrypted bits, enabling natural implementation of ReLU activations and max pooling instead of expensive polynomial approximations.TFHE expresses ciphertexts over the torus modulo 1 and supports both fully and leveled homomorphic encryption schemes.
- Motivation: Prior LHECNNs suffer low accuracy and high latency because polynomial approximations replace ReLU activations and max pooling.These approaches use square activations and mean pooling; on MNIST, square activation reduces accuracy by 0.82% versus unencrypted models, while higher-degree approximations increase computing overhead exponentially.
3 SHE
SHE implements accurate homomorphic ReLU and max-pooling operations with TFHE Boolean circuits, while reducing encrypted inference overhead through logarithmic shifts and mixed-bitwidth accumulation. These operations lower per-layer multiplicative-depth costs, enabling deeper LHECNN architectures under a fixed depth budget.
- Homomorphic activations and pooling: TFHE Boolean gates construct homomorphic ReLU and max-pooling units directly over encrypted binary data.The construction supports arbitrary 2-input binary operations, while larger gates are decomposed into multiple 2-input gates.
- Shift-based computation: Logarithmic quantization replaces expensive homomorphic multiplications with binary shifts using power-of-two quantized weights.Each quantized weight is obtained by rounding its logarithm to the nearest integer, then shifting the input by that amount.
- Shift-based computation: TFHE arithmetic shifts are computationally cheap because they copy encrypted sign and data bits to shifted positions, taking approximately 100ns per CPU core.This shift mechanism addresses the convolutional and fully connected layers that become latency bottlenecks in TFHE-based CNNs.
- Mixed-bitwidth accumulation: The mixed-bitwidth accumulator uses progressively wider TFHE adders in an adder tree to avoid unnecessary accumulation overhead.With 5-bit bottom-level adders, the next level uses 6-bit adders because summing two 5-bit integers requires at most 6 bits.
- Multiplicative-depth reduction: TCN reduces total multiplicative-depth overhead to 9.0K on MNIST while implementing the FCN architecture with LTFHE operations.Reducing each layer’s depth overhead allows deeper networks with more layers under a fixed multiplicative-depth budget.
4 Experimental Methodology
The experiments implement SHE with TFHE in leveled homomorphic-encryption mode, evaluate it across four datasets, and use dataset-specific network architectures. The methodology also specifies the computing platform, security settings, and topology references.
- TFHE setting and security analysis: SHE uses all three TFHE levels in leveled mode, with security degrees λ = 194, 152, and 289 for levels 0, 1, and 3, respectively.The reported overall security degree of SHE is λ = 152, and the configuration supports 32K depth of homomorphic AND primitives.
- Simulation, benchmark and dataset: Experiments ran on an Intel Xeon E7-4850 CPU with 1TB DRAM using MNIST, CIFAR-10, ImageNet, and diabetic retinopathy datasets.The diabetic retinopathy dataset is denoted as medicare.
- Network architecture: Network architectures were estimated per dataset, matching CNT for MNIST, using SHE and DSHE configurations for CIFAR-10, and adapting an LSTM architecture for Penn Treebank.For Penn Treebank, activations of all LSTM gates were converted to ReLU; the architecture summary is provided in Table 3.
5 Results and Analysis
SHE improves encrypted inference accuracy and efficiency across MNIST, CIFAR-10, ImageNet, and Penn Treebank by using LTFHE-supported operations with low multiplicative-depth overhead. Deeper SHE networks address accuracy and feasibility limitations that prevent prior LHECNNs from supporting several tasks.
- MNIST: On MNIST, CNT reaches 98.95% accuracy, while FCN shortens latency by 84.3% with only a 0.24% accuracy decrease; TCN improves accuracy by 0.6% over FCN but slows inference by 126.6%.CNT uses degree-2 polynomial approximation activations, whereas TCN uses LTFHE-based ReLU activations, max poolings, and matrix multiplications.
- CIFAR-10: On CIFAR-10, GHE and Lola lose more than 20% accuracy versus a full-precision unencrypted model, while FCN is 69.1% faster than NED but 16.2% less accurate.The cited accuracy losses are attributed to polynomial approximation activation layers.
- CIFAR-10: 96.62% accuracy is achieved by the deeper DSHE network on CIFAR-10, with 12041s inference latency under the 32K LTFHE multiplicative-depth budget.DSHE uses the available depth budget to increase inference accuracy.
- ImageNet and Medical Datasets: SHE enables full ImageNet inference with AlexNet, ResNet-18, and ShuffleNet, whereas prior LHECNNs cannot infer an entire ImageNet image.SHE requires 1 day and 2.5 days to test an ImageNet image with two of these architectures, while TCN exceeds the 32K depth budget for AlexNet and ResNet-18.
- Penn Treebank: SHE-based LSTM inference on Penn Treebank is feasible despite prior LHECNNs lacking LSTM support because SHE replaces high-depth computation with TFHE shifts and accumulations.Prior matrix multiplication across 25 timesteps requires 117.5K multiplicative-depth overhead, exceeding the 32K LTFHE budget.
6 Conclusion
The paper concludes that SHE combines LTFHE-compatible ReLU, max pooling, and mixed-bitwidth accumulation to enable fast, accurate encrypted inference. It achieves state-of-the-art accuracy, substantially reduces latency, and supports deep CNNs on ImageNet and LSTMs on Penn Treebank.
- 6 Conclusion: SHE combines LTFHE-enabled ReLU and max pooling units with a mixed bitwidth accumulator for fast and accurate encrypted inference.The conclusion identifies these components as the core of the proposed LTFHE-enabled deep neural network.
- 6 Conclusion: 76.21% ∼94.23% lower inference latency and state-of-the-art accuracy were achieved over prior LHECNNs across various datasets.SHE is also the first LHE-enabled model supporting deep CNN architectures on ImageNet and LSTM architectures on Penn Treebank.