Source-linked AI summary
Crypto-Nets: Neural Networks over Encrypted Data
Pengtao Xie, Misha Bilenko, Tom Finley, Ran Gilad-Bachrach, Kristin Lauter, Michael Naehrig
TL;DR
The paper addresses how users can obtain predictions from third-party neural networks without exposing private data or prediction results. It uses homomorphic encryption with polynomial approximations of neural networks, concluding that encrypted inference may be feasible while encrypted learning is harder to scale.
Problem
MLaaS can require raw user data, creating privacy risks and regulatory problems when users want predictions from third-party models.
Method
The paper approximates neural-network transformations with polynomials and evaluates the resulting model on encrypted data using homomorphic encryption.
Results
The paper presents crypto-nets as a way to apply neural networks to encrypted data while keeping user data and prediction results confidential.
Takeaways & Limitations
Crypto-nets may be feasible for inference in medical and financial applications, with implementation requiring further work in machine learning and cryptology.
Takeaways & Limitations
Encrypted learning is more difficult to scale because polynomial degree grows across layers and homomorphic multiplications increase computational cost.
Abstract
from arXiv · showhide
The problem we address is the following: how can a user employ a predictive model that is held by a third party, without compromising private information. For example, a hospital may wish to use a cloud service to predict the readmission risk of a patient. However, due to regulations, the patient's medical files cannot be revealed. The goal is to make an inference using the model, without jeopardizing the accuracy of the prediction or the privacy of the data. To achieve high accuracy, we use neural networks, which have been shown to outperform other learning models for many tasks. To achieve the privacy requirements, we use homomorphic encryption in the following protocol: the data owner encrypts the data and sends the ciphertexts to the third party to obtain a prediction from a trained model. The model operates on these ciphertexts and sends back the encrypted prediction. In this protocol, not only the data remains private, even the values predicted are available only to the data owner. Using homomorphic encryption and modifications to the activation functions and training algorithms of neural networks, we show that it is protocol is possible and may be feasible. This method paves the way to build a secure cloud-based neural network prediction services without invading users' privacy.
1 INTRODUCTION
Cloud machine learning separates data owners, model owners, and compute providers, but traditional services expose raw user data. The paper proposes neural-network prediction over encrypted inputs, replacing non-polynomial activations with polynomial approximations.
- Cloud machine learning: Cloud machine learning lets users access deployed models without maintaining the service, while separating data owners, model owners, and compute providers.This separation supports Machine Learning as a Service and allows model owners to charge per prediction.
- Encrypted prediction: The proposed protocol encrypts user data, evaluates a trained neural network on ciphertexts, and returns an encrypted prediction for local decryption.The cloud does not learn the user’s data or prediction result, while the result remains correct.
- Privacy problem: Traditional machine-learning services require raw-data access, creating privacy risks and potentially violating regulations for medical information.The paper’s goal is to let data owners use MLaaS without exposing their data.
- Core challenge: Crypto-nets address non-polynomial activations by approximating continuous functions with polynomials and minimizing polynomial degree for feasible computation.The paper identifies sigmoid and rectified-linear functions as examples of activations requiring this modification.
2 RELATED WORK
Prior work applied homomorphic encryption to several machine-learning models and used interaction or differential privacy for related privacy goals. The paper positions crypto-nets as the first neural-network approach for encrypted data without interactive nonlinearity evaluation.
- Homomorphic-encryption learning: Earlier homomorphic-encryption studies addressed privacy-preserving training or prediction for linear regression, linear classifiers, decision trees, and matrix factorization.The paper contrasts these models with neural networks, which can provide highly accurate predictions.
- Interactive neural networks: Orlandi et al. proposed having the data owner decrypt, apply, and re-encrypt each non-linear transformation, but this interaction increases latency and complexity.The model owner sends encrypted inputs to the data owner for every nonlinearity.
- Differential privacy: Differential privacy is unsuitable for privacy-preserving prediction here because inference uses a single record, which is fully exposed.The paper distinguishes this setting from collecting aggregate database statistics.
3 HOMOMORPHIC ENCRYPTION
The homomorphic-encryption scheme supports encrypted addition and multiplication, preserving bounded-degree polynomial computations without decryption. Fully homomorphic encryption removes the degree bound theoretically, but practical efficiency requires restricting computation depth, especially multiplications.
- Scheme operations: The assumed homomorphic-encryption scheme provides encryption, decryption, ciphertext addition, and ciphertext multiplication over integer messages.Ciphertext operations do not require the secret key.
- Correctness and privacy: Encrypted addition and multiplication decrypt to the corresponding plaintext sum and product, while encryption remains computationally hiding without the private key.The scheme’s correctness follows from the stated homomorphic identities.
- Polynomial evaluation: Repeated homomorphic operations evaluate any polynomial whose degree remains within the scheme’s supported bound, without decrypting intermediate values.The evaluated ciphertext polynomial decrypts to the plaintext polynomial result.
- Practical limits: Fully homomorphic encryption permits unrestricted polynomial degree and theoretically arbitrary computation, but current schemes remain impractical for practical applications.Restricting schemes to bounded operations and integer messages improves efficiency, although ciphertext complexity grows with operations.
4 POLYNOMIAL APPROXIMATION TO NEURAL NETWORKS
The paper converts neural networks into polynomial networks so their inference and, under stronger conditions, learning procedures can operate on encrypted data. Continuity on compact domains provides the approximation basis, while polynomial degree and encrypted weights constrain practicality.
- Polynomial approximation: On a compact input domain, a neural network with continuous non-linear transformations can be uniformly approximated by a polynomial to arbitrary error.This follows from the continuity of the composed network and polynomial approximation on compact sets.
- Encrypted inference: An encrypted neural network is constructed by replacing polynomial additions and multiplications with homomorphic operations and encrypting polynomial constants.The construction requires access only to the public encryption function.
- Encrypted inference: Existing neural-network inference therefore follows two stages: approximate the network by a polynomial, then encrypt that polynomial representation.This permits the polynomial network to be evaluated on encrypted inputs.
- Encrypted learning: When activations and the loss function are polynomial, back-propagation can learn encrypted versions of the coefficients learned from plaintext data.Polynomial derivatives allow the gradient computations to be performed over encrypted data.
- Encrypted learning: Polynomial approximations also support encrypted learning algorithms by approximating back-propagation under continuity and compact-domain assumptions.The paper describes a polynomial learning algorithm that approximates the plaintext algorithm’s learned weights.
5 PRACTICAL CONSIDERATION
Crypto-net feasibility depends on controlling polynomial complexity: inference can be evaluated over encrypted data, but both depth and training introduce substantial computational costs.
- Computational complexity: Homomorphic evaluation is slower than plain-data computation, and higher-degree polynomials require larger parameters and slower execution.Ciphertext noise grows with additions and especially multiplications, motivating low-degree polynomial approximations.
- Computational complexity: A network with l layers and degree-d approximations has polynomial degree d^l, so both approximation degree and network depth must remain small.Polynomial degrees multiply when activation and pooling approximations are composed across layers.
- Learning complexity: Inference may be feasible, but encrypted-data learning scales poorly because gradient computations increase polynomial degree with network depth and gradient steps.Under L2 loss, the gradient degree is d^2l before accounting for additional growth from learned polynomial constants and repeated updates.
- Learning complexity: The proposed encrypted-data learning approach is feasible only for small approximation degree and a limited number of gradient steps.Training neural networks already requires substantial computing resources without encryption.
6 DISCUSSION
The discussion distinguishes promising privacy-preserving inference from more constrained encrypted-data learning, outlining scenarios where network depth, data size, or cryptographic support determine feasibility.
- Inference: Encrypted inference can protect both user data and prediction results, but its slower speed limits use to settings where latency and throughput are not major concerns.The model is trained on plain data, while users encrypt inputs and receive predictions that only they can decrypt.
- Encrypted-data learning: Learning a model from one encrypted sample is theoretically possible but practically feasible only when the sample is small or the network is shallow.This scenario assumes the encrypted sample is used directly to learn the model.
- Encrypted-data learning: Aggregating multiple datasets encrypted under different keys is not supported by the discussed homomorphic encryption, but secure multi-party computation could address this case.A fully homomorphic scheme has been proposed in which all contributing data owners jointly control decryption.
- Encrypted-data learning: Fine-tuning a model trained on plain data may be practical when the data are small and the network has a not-too-high polynomial degree.The scenario involves performing only a few gradient steps to adapt the pretrained model.
7 CONCLUSION
The paper presents crypto-nets as a framework for learning and applying neural networks to encrypted data, while emphasizing that practical deployment remains an ongoing research challenge.
- Conclusion: Crypto-nets provide a way to learn and apply neural networks to encrypted data, combining theoretical analysis with discussion of practical implications.The paper conjectures feasibility for inference in medical and financial applications and possibly for limited learning.
- Conclusion: Implementing crypto-nets requires careful work in both machine learning and cryptology and remains ongoing research.