PaperlayerView exact revision on arXiv

Source-linked AI summary

Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference

Edward Chou, Josh Beal, Daniel Levy, Serena Yeung, Albert Haque, Li Fei-Fei

arXiv:1811.09953v1cs.CR

TL;DR

Homomorphic-encrypted inference protects sensitive inputs and outputs but remains too computationally expensive for many neural-network services. Faster CryptoNets combines pruning, power-of-2 quantization, sparse encodings, and activation approximation, with feature extraction and differential privacy for larger datasets. The method reports faster inference while retaining competitive accuracy, although encrypted inference does not address private training or all output-based attacks.

  • Problem

    Encrypted inference offers privacy for machine learning but faces substantial computational cost, network-depth limits, and practical deployment challenges for real-world neural networks.

  • Method

    Faster CryptoNets combines encryption-aware pruning and quantization, sparse polynomial encodings, optimized activation approximations, and delegated feature extraction with differentially private training.

  • Results

    The method achieves 6.4× faster network-layer evaluation than CryptoNets on MNIST while reaching 98.71% test accuracy versus CryptoNets’ 98.95%.

  • Takeaways & Limitations

    The results support the viability of sparse encrypted inference for practical tasks, including a reported 7.8× speedup on a real-world medical-imaging setting.

  • Takeaways & Limitations

    The approach remains constrained by multiplicative depth and does not cover private training or defenses against output-based attacks such as membership inference and model stealing.

Abstract

from arXiv · show

Homomorphic encryption enables arbitrary computation over data while it remains encrypted. This privacy-preserving feature is attractive for machine learning, but requires significant computational time due to the large overhead of the encryption scheme. We present Faster CryptoNets, a method for efficient encrypted inference using neural networks. We develop a pruning and quantization approach that leverages sparse representations in the underlying cryptosystem to accelerate inference. We derive an optimal approximation for popular activation functions that achieves maximally-sparse encodings and minimizes approximation error. We also show how privacy-safe training techniques can be used to reduce the overhead of encrypted inference for real-world datasets by leveraging transfer learning and differential privacy. Our experiments show that our method maintains competitive accuracy and achieves a significant speedup over previous methods. This work increases the viability of deep learning systems that use homomorphic encryption to protect user privacy.

I. INTRODUCTION

Encrypted inference protects data processed by third-party machine-learning services, but its computational cost and restricted arithmetic make practical deployment difficult. Faster CryptoNets addresses these barriers with sparse representations, activation approximations, and privacy-safe training techniques.

  • Motivation: Homomorphic encryption lets models compute on encrypted data while keeping inputs and output predictions inaccessible to the host.The end-user retains the private key, so the third-party provider performs oblivious inference without decrypting data.
  • Motivation: Encrypted neural-network inference can take minutes or hours per example, compared with milliseconds for plain networks, while HE limits modern activation functions.These constraints create a major computational bottleneck and require simpler, potentially lower-performance activations.
  • Contributions: Faster CryptoNets accelerates encrypted inference by combining pruning, power-of-2 quantization, and sparse polynomial representations throughout the network.The method targets multiplication costs by removing parameters and making plaintext encodings sparse.
  • Contributions: Experiments extend beyond MNIST by using feature extraction and differential privacy to reduce encrypted computation for real-world datasets.The approach combines transfer-learning-style feature extraction with privacy-preserving training techniques.

III. THREAT MODEL

The threat model considers cloud-hosted inference in which users send sensitive data to a provider running a pre-trained model. Homomorphic encryption prevents the provider from accessing plaintext inputs while supporting computation over encrypted representations.

  • Threat model: The MLaaS scenario assumes a cloud provider hosts a pre-trained proprietary model and returns predictions for user-submitted data.The provider already has the trained model; the focus is privacy-preserving inference rather than private training.
  • Threat model: A naive cloud-inference scheme requires the host to access plaintext data, creating risks of unauthorized retention, sale, theft, or misuse.The threat includes both transmission risks and the provider’s ability to collect and use submitted data.
  • Threat model: Homomorphic encryption enables oblivious inference by allowing the provider to compute on encrypted inputs and produce encrypted outputs without decrypting them.The provider never receives the private key or plaintext data during the computation.
  • Cryptosystem: The underlying scheme represents plaintexts and ciphertexts in polynomial rings and supports additive and multiplicative homomorphisms.The leveled scheme supports a depth-bounded arithmetic circuit, while FV-RNS decomposes coefficient moduli into smaller moduli for efficiency.
  • Cryptosystem: Ciphertext-ciphertext operations combine encrypted values, while ciphertext-plaintext addition and multiplication apply plaintext values directly to ciphertext components.These operations form the arithmetic interface used to evaluate the neural network homomorphically.

D. Integer Encoder

The integer encoder represents scaled values as base-2 polynomials, enabling powers-of-2 parameters to become sparse monomial multipliers. Pruning and quantization then reduce the number and cost of homomorphic multiplications.

  • Integer encoding: Real-valued computation uses fixed 15-bit precision, power-of-2 scaling, integer encoding, and post-decryption rescaling.The accumulated scaling factor is removed after decryption to recover a real-valued prediction.
  • Integer encoding: The base-2 encoder maps an integer’s binary coefficients into a plaintext polynomial, with negative coefficients represented modulo the plaintext modulus.This construction exposes sparsity in the polynomial representation.
  • Sparse multiplication: Plaintext-ciphertext multiplication is expensive because neural layers perform many polynomial products, but sparse multiplication can reduce the operation count.The paper contrasts brute-force O(n^2) multiplication with O(n log n) methods under suitable conditions.
  • Sparse multiplication: When a plaintext multiplier equals ±2^k, its encoding is a monomial, allowing sparse multiplication with O(n) coefficient multiplications and modular reductions.This is the central cryptosystem-aware efficiency opportunity exploited by the method.
  • Network pruning and quantization: Pruning removes weights and quantization converts remaining weights to powers of 2, producing sparse plaintext representations and reducing inference time.Dynamic Network Surgery and Incremental Network Quantization are used to preserve accuracy while creating the desired representations.
  • Network pruning and quantization: The pruning mask is updated during gradient descent using a parameter-importance measure, while quantization progressively covers more weights under bounded power-of-2 ranges.An example schedule quantizes 50%, 75%, 87.5%, and finally 100% of the weights, retraining the rest at each step.

C. Approximating the Activation Function

The paper formulates activation approximation as a minimax problem over polynomials whose coefficients are powers of 2. It searches a finite bounded candidate set to obtain an approximation that balances error and sparse encrypted computation.

  • Polynomial formulation: The method seeks a polynomial approximation p* for an activation function f, with every coefficient constrained to be a power of 2.The resulting polynomial belongs to the set P_n^(2) of degree-bounded base-2-coefficient polynomials.
  • Polynomial formulation: The approximation rounds coefficients of an unconstrained minimax polynomial to nearby powers of 2, yielding a feasible sparse candidate.The paper bounds the constrained approximation error between the unconstrained minimax error and the rounded candidate’s error.
  • Minimax optimization: The optimization minimizes maximum error over a chosen interval while enforcing the power-of-2 coefficient constraint.This formulation balances activation fidelity against practical usability in encrypted inference.
  • Finite search: A bounded polyhedron contains finitely many feasible coefficient tuples, making exhaustive scanning possible for the optimal approximation.Each tuple corresponds to a polynomial in the constrained candidate set, and the scanning method identifies p*.

VI. EXPERIMENTS

The experiments evaluate encrypted inference with hardware-independent operation counts, MNIST, and a batch-normalized CryptoNets variant. They emphasize that multiplicative homomorphic operations dominate cost and that wall-clock runtime is hardware- and implementation-dependent.

  • Evaluation Metrics: Wall-clock runtime depends strongly on hardware, memory, computational power, and encryption-library implementation.The paper introduces hardware-independent metrics to address these limitations.
  • Evaluation Metrics: HOPs count ciphertext-involving additions and multiplications across four operation classes.The classes are plaintext-ciphertext addition, ciphertext-ciphertext addition, plaintext-ciphertext multiplication, and ciphertext-ciphertext multiplication.
  • Evaluation Metrics: Ciphertext-ciphertext multiplications are generally the most costly neural-network operations, so reducing multiplicative HOPs is prioritized over adding additive HOPs.The analysis separates HOP classes because their costs differ substantially.
  • Experimental Setup: MNIST evaluation uses 28 × 28 grayscale digit images in a 10-class task with 50,000 training and 10,000 test examples.MNIST remains a standard benchmark for homomorphic inference despite its simplicity.
  • Experimental Setup: The MNIST network is a CryptoNets variant with batch normalization, while preserving the original multiplicative depth.Batch normalization supports a wider range of activation functions and reduces approximation error by narrowing activation inputs.
  • Experimental Setup: The architecture combines convolution, batch normalization, approximate activation, scaled average pooling, and fully connected layers.The listed network includes two convolutional layers, two approximate activation layers, two batch-normalization layers, two pooling layers, and two fully connected layers.

E. Encryption Scheme

The paper specifies the encryption, hardware, training, and sparsification settings used to evaluate Faster CryptoNets. Its activation approach uses quantized polynomial approximations, while avoiding square activations because of gradient-related stability concerns.

  • Encryption and Training Setup: The FV-RNS encryption setup uses n = 8192, plaintext moduli t1 = 1099511922689 and t2 = 1099512004609, and log q = 219 for 128-bit security.The coefficient modulus choice meets the cited Homomorphic Encryption Standardization Workshop security standards.
  • Encryption and Training Setup: MNIST experiments run on an Intel Core i7-5930K at 3.5 GHz with 48 GB RAM using a modified SEAL v2.3.0-4 library.The modified library supports the proposed method.
  • Encryption and Training Setup: Training uses batch size 64, 30 epochs, stochastic gradient descent with momentum 0.9, and a learning-rate schedule initialized at λ = 0.008.The schedule uses a 10-epoch step size and γ = 0.1; square-function gradients are clipped at 0.25.
  • Sparsification: Dynamic network surgery reports final non-pruned connection fractions and compression rates for each layer after 10,000 iterations.Reported sparsities are 0.1440, 0.0701, 0.0568, and 0.1480 for conv1, conv2, dense-fc1, and dense-fc2, respectively.
  • Activation Approximation: Faster CryptoNets constructs optimal quantized polynomial approximations of arbitrary functions and evaluates ReLU, Softplus, and Swish.Quadratic activations are used to approximate modern activations with different complexity and expressivity.
  • Activation Approximation: The method excludes square activations because their large gradients can require clipping and may indicate instability in deeper variants.The paper states that clipping may work for recurrent networks but may not work for deeper variants of this network.

K. Error Minimization

The experiments assess quantized activation approximations, their value distributions, homomorphic operation counts, and end-to-end encrypted inference. Results show accurate Swish approximations, reduced HOPs, and faster MNIST inference, while batching remains outside the evaluation scope.

  • Error Minimization: The error-minimization experiment compares approximation schemes to identify the lowest error under quantization constraints.It evaluates multiple schemes, including the proposed method.
  • Error Minimization: Accuracy trials compare original, approximated, and quantized activation functions across ReLU, square, Swish, and Softplus.Three models per activation use different random seeds, with mean accuracy and standard deviation reported.
  • Error Minimization: Within [−1, 1], most approximations fit the original activation, and Swish has lower error than ReLU and Softplus.Batch normalization helps constrain pre-activation values toward the region where approximation error is lower.
  • Activation Distributions: Theoretical Swish analysis gives a minimum near −0.278465 at x ≈ −1.27846, matching the empirical minimum in the activation distributions.The approximated Swish function retains this minimum, supporting the correctness of the approximation.
  • Homomorphic Operation Counts: The Faster CryptoNets HOP breakdown shows fewer homomorphic operations at each layer and across the overall network than CryptoNets.Tables II and III provide granular layer-level HOP counts using plaintext and ciphertext operation categories.
  • Evaluation Scope: The evaluation targets inference on one encrypted image and does not implement ciphertext batching.The authors state that batching analysis is beyond scope, while their method does not preclude batching techniques.
  • End-to-End Results: 39.1 seconds versus 249.6 seconds yields a 6.4× wall-clock improvement, while Faster CryptoNets uses 9.1× fewer HOPs than CryptoNets.Test accuracy is 98.71% after pruning and quantization versus 98.95% for CryptoNets.

O. Ablation Studies

The ablations examine activation choices, approximation accuracy, pruning and quantization, encrypted correctness, and evaluation settings for Faster CryptoNets.

  • Activation and layer comparisons: Swish-AQ reached 99.10% MNIST test accuracy, while ReLU-AQ achieved 99.09%.The approximations used the proposed polynomial approximation and quantization method.
  • Pruning and quantization: Combining pruning and quantization produced 98.71% accuracy with 45.7-second inference, versus 104.7 seconds for pruning alone and 162.5 seconds for quantization alone.Pruning alone achieved 98.73% accuracy; quantization alone achieved 99.06%.
  • Experimental correctness: Encrypted MNIST inference preserved the 98.71% plaintext accuracy, with approximately 0.05% precision error from floating-point-to-fixed-point conversion.The conversion error did not affect model accuracy.
  • Evaluation settings: The experiments evaluate increasingly practical settings, including MNIST, CIFAR-10, and diabetic retinopathy, using encrypted-network metrics and approximation results.CIFAR-10 is evaluated with the eight-convolutional-layer CNN-8 model.

A. Activation Comparisons

Activation comparisons show that square activations perform poorly in deeper networks, whereas quantized ReLU and Swish provide comparable and substantially better accuracy while supporting practical encrypted inference.

  • Activation comparisons: 59.88% square-activation test accuracy was substantially below ReLU’s 86.76% on CIFAR-10.The paper attributes the gap to instability of the square function in deeper neural networks.
  • Activation comparisons: ReLU-AQ and Swish-AQ achieved comparable CIFAR-10 test accuracies of 75.99% and 75.66%, respectively.Their training accuracies were 77.95% and 78.20%, respectively.
  • Efficiency: Pruning and quantization produced 76.72% accuracy with 6.12 × 10^8 HOPs, compared with 6.47 × 10^9 HOPs for the baseline.The resulting inference time was 22,372 seconds.
  • Medical imaging: The retinal dataset contains approximately 35,126 images graded into five diabetic-retinopathy severity categories before binary relabeling and an 80-20 train-test split.The binary task groups none/mild versus moderate/severe/proliferative diagnoses.
  • Medical imaging: Delegated feature extraction evaluates pretrained layers on the client and encrypted task-specific layers on the server for retinal-image diagnosis.The encrypted output is returned to the client for decryption and prediction.

B. Transfer Learning

Transfer learning and delegated feature extraction reduce the depth and encrypted workload of large neural networks, while private training further addresses feature-extractor privacy concerns.

  • Delegated feature extraction: Delegated feature extraction evaluates generic base layers on the client and only fine-tuned layers under encryption on the server.The client encrypts the final base-layer activation volume, and the server returns encrypted outputs.
  • Transfer learning: Fully encrypted ResNet-152 or Inception-v3 models have at least an order-of-magnitude greater multiplicative depth and prohibitively large encrypted runtimes.DFE-ResNet-152 limits encrypted evaluation to fine-tuned layers.
  • Encrypted retraining: The Swish approximation p*(x) = 2^-3x^2 + 2^-1x + 2^-4 is required only in the retrained block.Restricting the approximation to that block reduces convergence difficulties during training.
  • Evaluation: The comparison fixes multiplicative depth and uses HOPs as a hardware-independent measure of encrypted evaluation complexity.This supports a fair comparison between transfer-learning and fully trained models.
  • Private training: The paper combines DFE with private training to improve accuracy while preserving end-to-end privacy.The private-training approach uses modified DP-SGD with warm-starting and weight clustering.

H. Message Sizes

Delegated feature extraction reduces encrypted-input message size by transforming images into lower-dimensional representations before transmission, although encrypted messages remain large.

  • Message-size drivers: Message size is proportional to the number of plaintext moduli used for evaluation; the experiments use fifteen moduli.Both encrypted inputs and outputs are transferred between client and server.
  • Encrypted message sizes: 789.2 GB is the DFE-ResNet-152 encrypted-input message size, compared with 1183.8 GB for the CNN-8 baseline.The DFE input encrypts a 2048 × 7 × 7 activation volume, while CNN-8 encrypts the 224 × 224 × 3 image.
  • Transfer-learning benefit: The transfer-learning representation reduces data-transfer cost by 1.5× relative to transmitting the encrypted input image.Ciphertext batching can amortize encrypted-inference costs across multiple image requests.

I. Experimental Correctness

The experiments report correctness on CIFAR-10 and Retina images, broad performance gains from transfer learning, and a 7.8x speedup, while identifying remaining privacy and efficiency limitations.

  • Correctness: Around 0.05% error arose from floating/fixed point conversion when validating decrypted CIFAR-10 and Retina outputs on 100 images.The validation assessed correctness of decrypted outputs.
  • Overall Comparison: DFE-RN-50 improved accuracy, HOP counts, runtime, and message sizes compared with the baseline CNN-8 model.The comparison used a transfer learning feature-extraction model against the baseline.
  • Overall Comparison: 7.8x speedup demonstrated the effectiveness of sparsity-based optimization techniques for reducing computation time.The reported comparison also included smaller message sizes and fewer HOPs for DFE-RN-50.
  • Limitations: Encrypted inference remains constrained by computational cost, network depth, private training coverage, and vulnerability to black box attacks.The paper explicitly states that encrypted inference does not address private training or defenses against machine learning attacks.
  • Privacy Scope: The approach combines sparsity-based acceleration with private training techniques in a proposed end-to-end privacy-safe training and inference pipeline.The paper presents this combination as a possible direction for future work rather than a completed solution to all privacy problems.
  • Future Work: Structured sparsity, filter-level pruning, batching, hardware acceleration, and improved encoding schemes could further accelerate deeper-network evaluation.The proposed developments could also reduce encrypted-message sizes and improve encryption parameters.
Loading 1811.09953v1…