PaperlayerView exact revision on arXiv

Source-linked AI summary

CryptoDL: Deep Neural Networks over Encrypted Data

Ehsan Hesamifard, Hassan Takabi, Mehdi Ghasemi

arXiv:1711.05189v1cs.CRcs.LG

TL;DR

Deep neural networks often require access to privacy-sensitive raw data, motivating CryptoDL’s encrypted CNN classification approach. The paper replaces activation functions with low-degree polynomial approximations and evaluates encrypted inference, reporting strong MNIST performance and broader results on CIFAR-10.

  • Problem

    Deep neural-network services require access to raw data that may be privacy sensitive, while practical homomorphic encryption restricts encrypted computation to limited operations and depths.

  • Method

    CryptoDL combines CNN classification with homomorphic encryption, replacing activation functions with low-degree polynomial approximations and running inference over encrypted data.

  • Results

    CryptoDL reports accurate encrypted predictions across MNIST and CIFAR-10, including 99.52% MNIST accuracy and close to 164000 predictions per hour.

  • Takeaways & Limitations

    The results support CryptoDL as an efficient, accurate, and scalable approach to privacy-preserving predictions within homomorphic-encryption limitations.

  • Takeaways & Limitations

    FHE bootstrapping is computationally expensive, while HE generally permits only limited additions and multiplications, constraining encrypted neural-network computation.

Abstract

from arXiv · show

Machine learning algorithms based on deep neural networks have achieved remarkable results and are being extensively used in different domains. However, the machine learning algorithms requires access to raw data which is often privacy sensitive. To address this issue, we develop new techniques to provide solutions for running deep neural networks over encrypted data. In this paper, we develop new techniques to adopt deep neural networks within the practical limitation of current homomorphic encryption schemes. More specifically, we focus on classification of the well-known convolutional neural networks (CNN). First, we design methods for approximation of the activation functions commonly used in CNNs (i.e. ReLU, Sigmoid, and Tanh) with low degree polynomials which is essential for efficient homomorphic encryption schemes. Then, we train convolutional neural networks with the approximation polynomials instead of original activation functions and analyze the performance of the models. Finally, we implement convolutional neural networks over encrypted data and measure performance of the models. Our experimental results validate the soundness of our approach with several convolutional neural networks with varying number of layers and structures. When applied to the MNIST optical character recognition tasks, our approach achieves 99.52\% accuracy which significantly outperforms the state-of-the-art solutions and is very close to the accuracy of the best non-private version, 99.77\%. Also, it can make close to 164000 predictions per hour. We also applied our approach to CIFAR-10, which is much more complex compared to MNIST, and were able to achieve 91.5\% accuracy with approximation polynomials used as activation functions. These results show that CryptoDL provides efficient, accurate and scalable privacy-preserving predictions.

1 Introduction

CryptoDL adapts CNN classification to encrypted data by replacing incompatible activation functions with low-degree polynomial approximations, then evaluates accuracy and performance on encrypted datasets.

  • 1 Introduction: CryptoDL runs deep neural network classification on encrypted data without requiring parties to reveal sensitive data.The system combines CNNs with homomorphic encryption for encrypted inference.
  • 1 Introduction: Homomorphic encryption permits addition and multiplication but restricts practical encrypted neural-network computation, especially for activation functions.Leveled schemes further constrain practical solutions to low-degree polynomials.
  • 1 Introduction: The paper develops theoretical and empirical methods to approximate ReLU, Sigmoid, and Tanh with low-degree polynomials and uses them to train modified CNNs.The approximations replace original activation functions during model training and evaluation.
  • 1 Introduction: CryptoDL was implemented and evaluated on encrypted MNIST and CIFAR-10 datasets using pretrained CNN models.The training phase is assumed to occur on plaintext data before encrypted classification.
  • 1 Introduction: 99.52% accuracy was achieved on MNIST, close to the original model’s 99.56% accuracy.The reported result is paired with close to 164000 predictions per hour.
  • 1 Introduction: The paper reports efficient, accurate, and scalable privacy-preserving predictions as the resulting system goal.The introduction frames practical solutions as requiring both prediction accuracy and low running time.

2 Overview and Background Information

This section introduces homomorphic encryption and deep CNNs as the background needed to adapt neural-network computation to encrypted data.

  • 2 Overview and Background Information: The overview covers homomorphic encryption schemes, their strengths and weaknesses, deep CNNs, and required modifications for encrypted computation.It positions these topics as foundational background for the paper’s adaptation approach.

2.1 Homomorphic Encryption

Homomorphic encryption evaluates some computations on ciphertexts, but noise growth, ciphertext expansion, limited operations, and circuit-depth restrictions constrain practical use.

  • 2.1 Homomorphic Encryption: Homomorphic encryption provides Gen, Enc, Dec, and Eval functions, allowing circuits to be evaluated on ciphertexts while preserving decrypted results.The correctness relation is Dec(sk, Eval(pk, C, c1, · · · , cn)) = C(m1, · · · , mn).
  • 2.1 Homomorphic Encryption: Somewhat homomorphic encryption schemes support only one operation, such as addition.The Paillier cryptosystem is given as an example.
  • 2.1 Homomorphic Encryption: Ciphertext noise increases during computation, with multiplication increasing it more than addition, and decryption works only below a threshold.The threshold bounds the number of computations that can be performed over encrypted data.
  • 2.1 Homomorphic Encryption: Fully homomorphic encryption supports arbitrary-depth circuits through bootstrapping, but bootstrapping has high computational cost.The paper states that this limitation makes FHE impractical for actual use.
  • 2.1 Homomorphic Encryption: Leveled homomorphic encryption omits bootstrapping and permits circuits only below a specific depth threshold.Its performance can be improved using SIMD techniques and ciphertext batching.
  • 2.1 Homomorphic Encryption: HE schemes commonly require integer messages, produce larger ciphertexts, accumulate noise, and lack division.These constraints leave only limited additions and multiplications for encrypted computation.
  • 2.1 Homomorphic Encryption: Figure 1 depicts different layers in a convolutional neural network.The supplied caption identifies the figure’s subject but does not specify an additional encoding.

2.2 Deep Learning: Convolutional Neural Networks (CNNs / ConvNets)

CNNs are feed-forward networks organized into specialized layers that progressively process data for recognition and classification. Their encrypted-data implementation retains addition and multiplication operations but replaces incompatible nonlinear or max operations with alternatives.

  • CNNs are feed-forward networks whose connectivity is inspired by the animal visual cortex and which are effective for image recognition and classification.
  • Convolutional Layer: A convolutional layer applies filters to neighboring input values to learn features, using dot products composed of addition and multiplication.
  • Convolutional Layer: A stride specifies how far a convolutional filter moves horizontally or vertically at each step.
  • Activation Layer: Activation layers apply nonlinear functions such as ReLU, Sigmoid, and Tanh after convolution, but encrypted computation requires addition-and-multiplication replacements.
  • Pooling Layer: Pooling reduces data size; because max pooling requires an unavailable max operation, the approach uses scaled average pooling based on summation without division.
  • Pooling Layer: Average pooling can be implemented with addition only, without increasing computational depth.
  • Fully Connected Layer: Fully connected layers connect every neuron to all neurons in the previous layer, producing outputs through weighted dot products.
  • Training: Dropout randomly removes connections during training to reduce over-fitting and can be removed during classification.

3 The Proposed Privacy-preserving Classification for Deep Convolutional Neural Networks

CryptoDL adapts CNN classification to homomorphic encryption by replacing incompatible activation functions with low-degree polynomial approximations. The proposed derivative-based ReLU approximation supports accurate CNN training and encrypted inference, reaching 99.52% accuracy in the reported larger CNN.

  • Problem: Homomorphic encryption supports addition and multiplication but not CNN activation functions, requiring compatible replacement functions.Practical leveled HE further restricts computation to low-degree polynomials.
  • Approach: CryptoDL approximates ReLU, Sigmoid, and Tanh with polynomials so CNNs can operate over encrypted data.The approach replaces non-compatible activation functions with forms implementable using HE-supported operations.
  • Polynomial approximation: Higher-degree polynomials improve approximation accuracy but reduce practicality for homomorphic encryption.The paper therefore investigates approximation methods while attempting to keep polynomial degree low.
  • Proposed approximation: The derivative-based method approximates Sigmoid, integrates the resulting polynomial, and uses that integral as the activation function.The paper reports that this method achieves the best ReLU approximation among the evaluated methods.
  • Experimental results: The evaluation used 50,000 MNIST training images and 10,000 testing images, while increasing model size brought accuracy closer to the 99.77% state-of-the-art result.The smaller CNN remained below that benchmark because of its small size and simple architecture.

4 Experimental Results: Deep Convolutional Neural Network over Encrypted Data

CryptoDL evaluates CNN classification over encrypted data using polynomial activation functions and reports accuracy, runtime, and communication results on MNIST and CIFAR-10.

  • Implementation: The encrypted-data implementation trains CNNs on plaintext, then evaluates encrypted inputs using polynomial-approximated activations.HELib is used for implementation, and SIMD enables batch classification of encrypted images.
  • MNIST evaluation: 8192 ciphertexts form the classification batch for the reported MNIST timing breakdown, including encryption, transfer, classification, and decryption.The experiments separately measure encryption, data transfer, encrypted classification, and decryption costs.
  • Comparison with prior work: CryptoDL significantly outperforms CryptoNets across the reported comparison aspects, using similarly configured machines.The comparison covers performance measures presented in Table 5.
  • Comparison with prior work: 336.7MB is CryptoDL’s communication cost for CNN Model 1, compared with 595.5MB for CryptoNets and 722GB for DeepSecure.The passage attributes the larger SMC cost to repeated client-server interactions.
  • CIFAR-10 evaluation: 91.5% accuracy is achieved on CIFAR-10 with polynomial activations, versus 94.2% with the original activation function.CIFAR-10 contains 50,000 training and 10,000 test images across 10 classes.
  • CIFAR-10 evaluation: CIFAR-10 is reported as much slower than MNIST, consistent with its more complex dataset and CNN setting.The supplied passage states that this slowdown was expected.
  • Scope: The study assumes that CNN training is completed on plaintext data and that the trained model is already available.The encrypted-data experiments therefore focus on classification rather than private training.

5 Related Work

Related work spans homomorphic-encryption and secure-multi-party-computation approaches, while CryptoDL emphasizes deep learning over encrypted data using homomorphic encryption alone.

  • Positioning: CryptoDL is based only on homomorphic encryption and targets deep learning, contrasting with earlier simple-classifier or SMC-based approaches.The paper states that its approach is efficient for large datasets.
  • Homomorphic-encryption approaches: CryptoNets and Chabanne et al. are identified as the closest prior approaches to encrypted neural-network classification.CryptoNets uses polynomial replacement choices, while Chabanne et al. use Taylor approximation and batch normalization.
  • Secure multi-party computation: SMC-based works such as DeepSecure and SecureML use interactive protocols for privacy-preserving neural-network computation.DeepSecure uses Yao’s Garbled Circuit protocol for distributed private evaluation.
  • Privacy-preserving training: Prior privacy-preserving training work includes BGV-based back-propagation and secure computation of higher-order operations.These works address training-phase privacy rather than only encrypted classification.

6 Conclusion and Future Work

The paper presents polynomially activated CNNs for encrypted prediction and reports results on MNIST and CIFAR-10, with more complex models and privacy-preserving training left for future work.

  • Conclusion: 99.52% accuracy and close to 164000 predictions per hour are reported for encrypted MNIST classification.The paper also reports 91.5% accuracy on CIFAR-10 using polynomial activation functions.
  • Conclusion: The authors conclude that carefully chosen polynomials can replace activation functions within homomorphic-encryption limitations.They characterize the resulting predictions as efficient, accurate, and scalable.
  • Future work: Future work includes implementing more complex models on GPUs and studying privacy-preserving neural-network training.The planned work extends beyond the paper’s classification focus.
Loading 1711.05189v1…