PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network

Joon-Woo Lee, HyungChul Kang, Yongwoo Lee, Woosuk Choi, Jieun Eom, Maxim Deryabin, Eunsang Lee, Junghyun Lee, Donghoon Yoo, Young-Sik Kim, Jong-Seon No

arXiv:2106.07229v1cs.LGcs.CR

TL;DR

Prior FHE-based PPML models were limited in architecture depth, standard activations, and demonstrated performance on advanced datasets. This paper implements standard ResNet-20 with RNS-CKKS, polynomial activation approximation, bootstrapping, and encrypted softmax. The model achieves results close to plaintext inference on CIFAR-10, while retaining practical limitations in runtime, security level, and bootstrapping reliability.

  • Problem

    Previous FHE-based PPML models were limited to non-standard models, simple arithmetic substitutes for non-arithmetic activations, and shallow networks, with limited evidence on advanced datasets.

  • Method

    The paper implements standard ResNet-20 over RNS-CKKS using approximate ReLU, bootstrapping for deep encrypted evaluation, and homomorphic softmax.

  • Results

    90.67% classification accuracy was achieved, with 98.67% identical inference results to the original ResNet-20 and 91.89% plaintext accuracy.

  • Takeaways & Limitations

    The results show that word-wise FHE with advanced approximation and bootstrapping can implement a standard deep learning model without retraining it.

  • Takeaways & Limitations

    The model takes about 4 hours to run, processes only one image, uses 98-bit security, and requires 1,149 bootstrapping procedures susceptible to failure.

Abstract

from arXiv · show

Fully homomorphic encryption (FHE) is one of the prospective tools for privacypreserving machine learning (PPML), and several PPML models have been proposed based on various FHE schemes and approaches. Although the FHE schemes are known as suitable tools to implement PPML models, previous PPML models on FHE encrypted data are limited to only simple and non-standard types of machine learning models. These non-standard machine learning models are not proven efficient and accurate with more practical and advanced datasets. Previous PPML schemes replace non-arithmetic activation functions with simple arithmetic functions instead of adopting approximation methods and do not use bootstrapping, which enables continuous homomorphic evaluations. Thus, they could not use standard activation functions and could not employ a large number of layers. The maximum classification accuracy of the existing PPML model with the FHE for the CIFAR-10 dataset was only 77% until now. In this work, we firstly implement the standard ResNet-20 model with the RNS-CKKS FHE with bootstrapping and verify the implemented model with the CIFAR-10 dataset and the plaintext model parameters. Instead of replacing the non-arithmetic functions with the simple arithmetic function, we use state-of-the-art approximation methods to evaluate these non-arithmetic functions, such as the ReLU, with sufficient precision [1]. Further, for the first time, we use the bootstrapping technique of the RNS-CKKS scheme in the proposed model, which enables us to evaluate a deep learning model on the encrypted data. We numerically verify that the proposed model with the CIFAR-10 dataset shows 98.67% identical results to the original ResNet-20 model with non-encrypted data. The classification accuracy of the proposed model is 90.67%, which is pretty close to that of the original ResNet-20 CNN model...

1 Introduction

FHE offers encrypted computation for PPML, but prior FHE models were limited by non-standard architectures, non-arithmetic activations, shallow depth, and incomplete privacy protections. This work implements standard ResNet-20 with RNS-CKKS, approximation, bootstrapping, and encrypted softmax, achieving results close to plaintext inference.

  • Motivation: FHE enables PPML servers to compute on encrypted data without accessing the underlying information.The scheme supports processing encrypted data and returning encrypted outputs to the client.
  • Prior limitations: Prior FHE-based PPML models used limited, non-standard architectures whose efficiency and accuracy were not established on advanced datasets.Earlier approaches included HE-friendly networks and hybrid FHE-MPC models with practical limitations.
  • Prior limitations: Non-arithmetic activations such as ReLU are not directly supported by CKKS arithmetic, while the lack of bootstrapping restricted previous models to few layers.Bootstrapping extends ciphertext levels so deeper homomorphic computations can continue.
  • Contribution: The paper implements standard ResNet-20 over RNS-CKKS using minimax polynomial approximation for ReLU and bootstrapping for encrypted deep inference.The implementation also evaluates softmax homomorphically to address model extraction concerns.
  • Results: 90.67% classification accuracy and 98.67% identical inference results were obtained, compared with 91.89% plaintext accuracy and a previous 77% word-wise-FHE result.The authors report the proposed model as the highest-accuracy CNN implementation using FHE at the time.

2 Preliminaries

CKKS supports arithmetic on encrypted real or complex data, and RNS-CKKS reduces large-integer complexity. Bootstrapping refreshes ciphertexts to support deeper arithmetic circuits, but remains time-consuming and error-prone.

  • 2.1 RNS-CKKS scheme: CKKS performs homomorphic addition, multiplication, rotation, and conjugation on encrypted real or complex data.Its security is based on the Ring-LWE hardness assumption.
  • 2.1 RNS-CKKS scheme: RNS-CKKS splits large integers into smaller residue components, reducing the computational complexity of the original CKKS scheme.Component-wise residue operations correspond to addition and multiplication of the original integers.
  • 2.2 Bootstrapping of CKKS scheme: Bootstrapping transforms a ciphertext with insufficient modulus into a fresh ciphertext with large modulus, enabling arithmetic circuits with large multiplicative depth.Rescaling reduces the modulus after homomorphic multiplication, eventually necessitating refresh.
  • 2.2 Bootstrapping of CKKS scheme: Applying state-of-the-art CKKS bootstrapping to privacy-preserving deep neural networks had not previously been demonstrated in the cited work.The operation is identified as both the most time-consuming and the most error-prone CKKS homomorphic operation.
  • 2.2 Bootstrapping of CKKS scheme: CKKS bootstrapping performs modulus raising, coefficient-to-slot conversion, homomorphic modular reduction, and reverse transformation.These stages implement the ciphertext refresh procedure described in the paper.

3 ResNet-20 on RNS-CKKS scheme

The encrypted ResNet-20 structure follows the original network while adding bootstrapping and implementing arithmetic-compatible approximations and packed convolution operations. Parameter choices and bounded-input analysis support feasible encrypted evaluation.

  • 3.1 Structure: The implemented structure contains convolution, batch normalization, ReLU, bootstrapping, average pooling, fully connected, and softmax stages.It is virtually identical to the original ResNet-20 except for the added bootstrapping procedure.
  • 3.2.1 Parameters: The system uses a 216-degree ciphertext polynomial, 98-bit security, 11 evaluation levels, and 18 bootstrapping levels.The secret-key Hamming weight is 64 and the maximum modulus bit length is 1750.
  • 3.2.1 Parameters: One 32 × 32 CIFAR-10 image is processed at a time, with each channel packed into one ciphertext using 2^10 sparse slots.Sparse packing is used because sparsely packed ciphertext bootstrapping takes less time than fully packed bootstrapping.
  • 3.2.2 Approximation region: Polynomial approximations require procedure inputs to remain within a bounded approximation region, because values outside it can cause classification failure.The observed maximum absolute input for ReLU, bootstrapping, and softmax was 37.1, motivating a region below 40 with high probability.
  • 3.2.3 Precision and efficiency: Lazy rescaling, lazy relinearization, and scaling-factor management reduce precision errors and computation time.These optimizations reduce work associated with number-theoretic transformation and gadget decomposition.
  • 3.3 Convolution and batch normalization: The implementation uses packed SISO convolution for stride 1 and a modified stride-2 convolution that avoids slot rearrangement.Avoiding rearrangement reduces rotation operations and their time-consuming key-switching procedures.
  • 3.3 Convolution and batch normalization: Batch normalization is implemented with homomorphic addition and scalar multiplication because it is linear with constant coefficients.This keeps the batch-normalization operation within the arithmetic capabilities of the scheme.

3.4 ReLU

The implementation approximates ReLU with composed minimax polynomials and relies on bootstrapping to evaluate the deep ResNet-20 homomorphically. Because bootstrapping can fail outside its approximation region, the design analyzes input bounds and trades approximation cost against failure probability.

  • ReLU is approximated with arithmetic functions rather than replaced by a simple non-linear polynomial, preserving the standard ResNet-20 activation.
  • The RNS-CKKS implementation composes minimax polynomial approximations for the ReLU sign function, using precision parameter α = 13.
  • Three minimax polynomials of degrees 15, 15, and 27 provide about 16-bit average approximation precision.
  • ReLU polynomial evaluation consumes many multiplicative depths, so bootstrapping is performed twice during its evaluation.
  • 1,149 bootstrapping procedures are required, and one failure can largely distort the whole neural-network result.
  • The approximation-region parameters create a trade-off: larger input bounds require higher-degree polynomials and more computation, while reducing failure probability.

3.6 Average pooling and fully connected layer

The implementation preserves ciphertext channels through average pooling and the fully connected layer, avoiding some rotations to improve speed at the cost of memory. Its softmax uses scaled polynomial approximations to keep intermediate values within encryption capacity.

  • Average pooling and fully connected layer: After stride-2 convolutions and average pooling, the ciphertext retains valid message slots used by subsequent computation.
  • Average pooling and fully connected layer: Combining 64 ciphertexts would require 63 additional rotations in the fully connected layer, so the implementation keeps them separate to favor operation speed over memory use.
  • Softmax: The softmax avoids directly evaluating e^x because exponential outputs may exceed the RNS-CKKS encryption capacity.
  • Softmax: The implementation instead evaluates a scaled softmax expression whose output is nearly unchanged while keeping exponential outputs within encryption capacity.
  • Softmax: A degree-12 least-squares polynomial approximates e^x on [-1, 1], with scaling and repeated squaring used to evaluate e^x/4.
  • Softmax: The inverse of the summed exponential outputs is also approximated after scaling inputs by 10^-4, and the softmax procedures require 22 bootstrappings.

4 Simulation result

The encrypted ResNet-20 simulation uses plaintext-trained parameters and encrypted CIFAR-10 images. It achieves high agreement with plaintext inference and comparable classification accuracy, but requires substantial computation time.

  • Simulation setup: The simulation uses the SEAL library on dual Intel Xeon Platinum 8280 CPUs with 512 GB memory and one OpenMP thread per channel.
  • Simulation setup: The plaintext ResNet-20 is trained on augmented 32 × 32 color images using mean subtraction, horizontal mirroring, shifting, He initialization, and cross-entropy loss.
  • Performance: 98.67%± 2.59% agreement is obtained between encrypted and plaintext classification results across 75 encrypted images.
  • Performance: 90.67%± 6.58% encrypted-data classification accuracy is reported, compared with 89.33%± 6.99% for corresponding plaintext images.
  • Comparison: The previous word-wise FHE model’s highest reported CIFAR-10 classification accuracy was 77%.
  • Performance: The proposed model takes about 4 hours to infer one image, with convolution, ReLU, and bootstrapping consuming the most time.

5 Limitation

The implementation demonstrates encrypted ResNet-20 inference but remains constrained by practical runtime, security-level, and evaluation limitations.

  • Running time: About 4 hours of running time makes the proposed model somewhat impractical for current use.The authors identify accelerator-based implementation and packing across many images as future optimization directions.
  • Security level: 98-bit security is the proposed model’s minimum considered secure level, below the 128-bit standard used in most applications.Higher security can be reached by changing RNS-CKKS parameters, at the cost of longer running time.
  • Classification accuracy: The classification result is based on one training run, although random weight initialization can produce different model performances.The authors therefore do not report an average over several training runs.

6 Conclusion

The paper applies state-of-the-art RNS-CKKS techniques to standard ResNet-20 without retraining and reports results close to the original model. It presents this as evidence that advanced word-wise FHE can support standard deep learning models.

  • 6 Conclusion: RNS-CKKS was applied to the standard ResNet-20 using precise ReLU approximation, bootstrapping, and softmax evaluation.These techniques were fine-tuned with various parameters for PPML implementation.
  • 6 Conclusion: The implemented ResNet-20 achieves almost the same result as the original ResNet-20 and the highest classification accuracy among cited word-wise-FHE PPML models.The work applies the model to encrypted data without retraining it.

A.1 Minimax Composition of ReLU

The ReLU is evaluated through a composed polynomial approximation of sign rather than a single high-degree minimax polynomial. This reduces homomorphic evaluation time while retaining the precision needed for pretrained ResNet-20 parameters.

  • A.1 Minimax Composition of ReLU: ReLU requires sufficiently high-precision approximation when pretrained original ResNet-20 parameters are used.A single minimax polynomial would require quite a large degree and substantial homomorphic evaluation time.
  • A.1 Minimax Composition of ReLU: The method rewrites ReLU using sign(x) and approximates sign(x) with a minimax composition of small polynomials.This composition replaces the slower single-polynomial approach.
  • A.1 Minimax Composition of ReLU: The approximation reduces homomorphic ReLU evaluation time and makes non-arithmetic functions more practical to evaluate homomorphically.ReLU is given as the principal example of the supported non-arithmetic functions.

A.2 Analysis of Bootstrapping Failure

Bootstrapping failure probability accumulates across ciphertext slots and repeated network bootstrappings. The analysis motivates reducing this probability because large slot counts and thousands of bootstrappings make prior approximations unreliable for the whole network.

  • A.2 Analysis of Bootstrapping Failure: The proposed analysis describes how CKKS bootstrapping failure affects whole-ResNet evaluation and motivates reducing its probability.CKKS bootstrapping depends on secret-key sparsity and therefore has a failure probability.
  • A.2 Analysis of Bootstrapping Failure: A modulus-reduction failure occurs when a coefficient of I is at least K, causing the modulus reduction to return a useless value.The approximation domain is bounded by K and epsilon.
  • A.2 Analysis of Bootstrapping Failure: For n slots, single-bootstrapping failure is approximately 2n · p, where p = Pr(|I_i|≥K).There are 2n coefficients undergoing modulus reduction.
  • A.2 Analysis of Bootstrapping Failure: Across N_b bootstrappings, whole-network failure is approximately 2N_b · n · p and becomes very high with many slots and thousands of bootstrappings.The stated accumulation explains why previous approximate polynomials are insufficient for the full network.
Loading 2106.07229v1…