Source-linked AI summary
nGraph-HE2: A High-Throughput Framework for Neural Network Inference on Encrypted Data
Fabian Boemer, Anamaria Costache, Rosario Cammarota, Casimir Wierzynski
TL;DR
Encrypted deep-learning inference is limited by non-polynomial activations, framework support, and substantial computational overhead. nGraph-HE2 combines CKKS optimizations, graph-level techniques, and a client-aided model to support standard pre-trained networks, achieving high CryptoNets throughput and MobileNetV2 evaluation on ImageNet.
Problem
Encrypted inference must handle standard pre-trained networks despite SHE limitations on non-linear activations, missing framework support, and substantial computational overhead.
Method
nGraph-HE2 combines CKKS encoding and arithmetic optimizations, lazy rescaling, depth-aware encoding, and client-aided two-party computation for non-polynomial activations.
Results
The optimizations achieve 1,998 images/s on CryptoNets and enable MobileNetV2 evaluation on ImageNet with 60.4%/82.7% top-1/top-5 accuracy and 381 ms/image.
Takeaways & Limitations
nGraph-HE2 enables inference on pre-trained DL models without modification, including a full production-level network and encrypted ImageNet data.
Takeaways & Limitations
The client-aided approach currently has the client decrypt, compute, and re-encrypt non-polynomial activations, while communication accounts for roughly 27% of LAN runtime.
Abstract
from arXiv · showhide
In previous work, Boemer et al. introduced nGraph-HE, an extension to the Intel nGraph deep learning (DL) compiler, that enables data scientists to deploy models with popular frameworks such as TensorFlow and PyTorch with minimal code changes. However, the class of supported models was limited to relatively shallow networks with polynomial activations. Here, we introduce nGraph-HE2, which extends nGraph-HE to enable privacy-preserving inference on standard, pre-trained models using their native activation functions and number fields (typically real numbers). The proposed framework leverages the CKKS scheme, whose support for real numbers is friendly to data science, and a client-aided model using a two-party approach to compute activation functions. We first present CKKS-specific optimizations, enabling a 3x-88x runtime speedup for scalar encoding, and doubling the throughput through a novel use of CKKS plaintext packing into complex numbers. Second, we optimize ciphertext-plaintext addition and multiplication, yielding 2.6x-4.2x runtime speedup. Third, we exploit two graph-level optimizations: lazy rescaling and depth-aware encoding, which allow us to significantly improve performance. Together, these optimizations enable state-of-the-art throughput of 1,998 images/s on the CryptoNets network. Using the client-aided model, we also present homomorphic evaluation of (to our knowledge) the largest network to date, namely, pre-trained MobileNetV2 models on the ImageNet dataset, with 60.4\percent/82.7\percent\ top-1/top-5 accuracy and an amortized runtime of 381 ms/image.
1 INTRODUCTION
nGraph-HE2 extends encrypted deep-learning inference to standard pre-trained models, addressing activation-function and framework limitations through CKKS, client-aided computation, and graph-level optimizations.
- Motivation and approach: nGraph-HE2 uses two-party computation to execute a wider class of pre-trained networks, including models with non-polynomial activations.The framework focuses on inference over encrypted data with a plaintext model and uses batch-axis packing for Reshape operations.
- Motivation and approach: CKKS supports floating-point numbers and plaintext packing, making it suitable for data-scientist-friendly encrypted inference.The setting supports standard DL models without HE-specific training or cryptographic expertise.
- Contributions: 3x-88x faster scalar encoding and doubled throughput from complex packing are achieved through CKKS encoding optimizations.Complex packing doubles inference throughput in networks without ciphertext-ciphertext multiplication.
- Contributions: Ciphertext-plaintext addition and multiplication are optimized alongside lazy rescaling and depth-aware encoding.Lazy rescaling delays expensive rescaling, while depth-aware encoding reduces encoded-model memory usage.
- Results: 1,998 images/s is achieved on CryptoNets, while MobileNetV2 inference reaches 60.4%/82.7% top-1/top-5 accuracy and 381 ms/image.The MobileNetV2 result is presented as homomorphic evaluation of a full production-level network.
2 BACKGROUND
The background describes CKKS-based leveled homomorphic computation, its arithmetic and representation constraints, and the challenges of applying it to neural networks and software frameworks.
- CKKS background: CKKS supports approximate addition and multiplication on encrypted data, with ciphertext-plaintext operations typically faster than ciphertext-ciphertext operations.Arithmetic imprecision comes from computation noise controlled through encryption parameters.
- CKKS background: CKKS uses rescaling to manage noise growth, limiting computation to a predetermined multiplicative depth L.Rescaling lowers ciphertext levels, and L is typically set at least as large as the function’s multiplicative depth L_f.
- CKKS background: The double-CRT representation combines residue-number-system arithmetic with the number-theoretic transform, but rescaling requires costly NTT conversions.These conversions motivate optimizations that avoid NTT and inverse-NTT operations where possible.
- HE for deep learning: Non-polynomial activations such as ReLU and MaxPool require comparisons unsupported by relevant SHE schemes.This creates a central obstacle for applying encrypted computation to neural networks.
- HE for deep learning: HE-friendly networks replace native activations with low-degree polynomials, but reported CIFAR10 accuracy falls from 84% to 81.5% and from 94.2% to 91.5%.High-degree polynomials also require prohibitively large encryption parameters because of increased multiplicative depth.
- Frameworks and deployment: Popular DL frameworks lack HE support, while HE libraries operate below DL primitives, creating significant engineering overhead.Graph compilers address this by representing models in a common intermediate representation and enabling framework-agnostic optimizations.
- Frameworks and deployment: HE inference has computational and memory overhead that can reach several orders of magnitude over unencrypted computation.Encryption-parameter selection strongly affects both overhead and security and remains largely hand-tuned.
3 CONTRIBUTIONS
nGraph-HE2 introduces batch-axis-packing contributions spanning CKKS encoding, arithmetic, and graph-level optimization, including complex packing for higher-throughput inference.
- The framework contributes CKKS encoding, CKKS arithmetic, and graph-level optimizations for batch-axis packing.
- Scalar encoding is optimized for plaintexts containing identical real-valued scalars throughout their lifetime, a property maintained by batch-axis packing.The optimization is not maintained by inter-axis packing and therefore applies only in specific use cases.
- Complex packing doubles throughput without ciphertext-ciphertext multiplication by using the complex components of CKKS plaintexts.It represents the same computation as real packing on twice as many slots for supported operations.
- Complex packing supports compositions of addition, subtraction, and broadcast multiplication, but not the relevant ciphertext-ciphertext multiplication.Real packing supports multiplication but requires twice as many slots, represented by two plaintexts or doubled N.
- Batch-axis packing stores S batch values in each ciphertext while model weights use identical scalar values across slots.This supports fully connected and convolution operations through element-wise ciphertext-plaintext arithmetic.
3.2 CKKS Arithmetic Optimizations
nGraph-HE2 optimizes ciphertext-plaintext arithmetic for batch-axis packing, including scalar operations and a specialized 32-bit multiplication path.
- The arithmetic optimizations target ciphertext-plaintext addition and multiplication in the batch-axis-packing setting.The multiplication optimization additionally applies when the coefficient modulus is less than 32 bits.
- Scalar encoding lets ciphertext-plaintext addition and multiplication apply one scalar across polynomial elements instead of operating on two full polynomials.The same scalar-application idea is used for both arithmetic operations.
- Ciphertext-plaintext arithmetic uses Barrett reduction to reduce products or sums with respect to coefficient moduli.
- 32-bit coefficient moduli simplify Barrett reduction to two multiplications and two subtractions, compared with the more expensive 64-bit path.
- The optimized 32-bit multiplication stores only L plaintext entries rather than N · L and remains O(LN) runtime while improving cache friendliness.
3.3 Graph-level Optimizations
nGraph-HE2 adds lazy rescaling and depth-aware encoding to reduce graph execution overhead in CKKS inference.
- Lazy rescaling reduces rescaling operations by postponing them until after fully connected or convolution layers.It can also skip rescaling when no later multiplication occurs before decryption.
- Lazy rescaling allows addition after multiplication before rescaling and can omit rescaling for terminal FC-Add, Convolution-Add, or Multiply-Add patterns.
- Depth-aware encoding encodes plaintexts at the ciphertext’s current level rather than always at level L.Lower-level encoding reduces encoding runtime and memory usage as rescaling decreases the ciphertext level.
- Depth-aware encoding can use compile-time encoding for fastest runtime or lazy runtime encoding for simpler implementation.With scalar encoding, lazy encoding incurs little slowdown because scalar encoding is cheaper than general encoding.
- Together with CKKS-specific optimizations, the graph-level contributions include complex packing and depth-aware encoding.
4 EVALUATION
The evaluation measures nGraph-HE2’s component optimizations and end-to-end performance on CryptoNets and MobileNetV2. Results show substantial speedups, high CryptoNets throughput, and practical MobileNetV2 inference with minimal accuracy degradation.
- Single-operation tests: 3x-88x faster scalar encoding is achieved through the CKKS encoding optimization.The optimization targets scalar encoding and is evaluated across parameter choices satisfying 128-bit security.
- Single-operation tests: 2.6x-4.2x speedup is achieved for ciphertext-plaintext scalar addition through register-resident scalar operands.Both optimized algorithms have O(LN) runtime; the reported gain comes from reduced cache-placement and retrieval competition.
- Single-operation tests: 2.6x speedup is achieved for ciphertext-plaintext scalar multiplication with 30-bit coefficient moduli.The improvement uses the Barrett reduction optimization under those modulus parameters.
- CryptoNets: ≈8x lower CryptoNets runtime results from lazy rescaling while preserving 98.95% accuracy.The CryptoNets model uses x^2 activations and multiplicative depth 5.
- CryptoNets: Complex packing roughly halves amortized CryptoNets runtime by doubling capacity, despite slightly increasing runtime relative to real packing.This evaluation uses ReLU activations, N = 2^11, L = 1, and 98.62% accuracy.
- End-to-end neural network workloads: 1,998 images/s is the highest reported CryptoNets throughput, while the client-aided model reaches 2,959 images/s.For MobileNetV2, encryption reduces top-1 accuracy by approximately 0.01%, and LAN communication substantially increases runtime relative to localhost.
5 CONCLUSION AND FUTURE WORK
nGraph-HE2 combines CKKS and graph-level optimizations to improve encrypted inference, achieving high CryptoNets throughput and enabling MobileNetV2 evaluation on encrypted ImageNet data. Its client-aided activation protocol remains a security limitation addressed as future work.
- Optimizations: 3x-88x improvement in scalar encoding, 2.6x-4.2x speedup in ciphertext-plaintext scalar addition, and 2.6x speedup in ciphertext-plaintext multiplication were achieved for batch-axis plaintext packing.The optimizations target SEAL’s CKKS implementation.
- Optimizations: 8x latency reduction on CryptoNets was achieved through lazy rescaling, while complex packing doubled throughput with minimal runtime effect.Both are CKKS-specific or graph-level optimizations described in the conclusion.
- Results: 1,998 images/s was achieved for CryptoNets on MNIST, while MobileNetV2 reached 60.4%/82.7% top-1/top-5 accuracy with 381 ms/image amortized runtime on encrypted ImageNet data.The MobileNetV2 evaluation used pre-trained models without modification and was presented as the first evaluation of a model with encrypted ImageNet data.
- Future work: Non-polynomial activations remain a future-work target because the current approach has the client decrypt, compute, and re-encrypt activations.The proposed future direction is a provably privacy-preserving primitive such as Yao’s Garbled Circuit.
A.1 Network Architectures
The evaluated architectures include CryptoNets and CryptoNets-ReLU, with convolutional and fully connected layers operating on batches whose size is denoted by n.
- Notation: n denotes the batch size for each architecture.The notation is used in the listed layer input and output dimensions.
- CryptoNets-ReLU: CryptoNets-ReLU replaces the squared activation with ReLU and uses bias in its convolution and fully connected layers.Its listed dimensions match the CryptoNets architecture: n × 28 × 28, n × 845, n × 100, and n × 10.
A.2 Parallel Scaling
CryptoNets runtime decreases with more OpenMP threads, scaling linearly up to 16 threads and reaching best performance at 88 threads. Using 24 threads offers a close runtime with fewer resources.
- Scaling behavior: Latency decreases linearly with thread count up to 16 threads for CryptoNets with N = 213 and L = 6.The runtime depends heavily on the number of threads because high-level operations are parallelized with OpenMP.
- Scaling behavior: 1.87s vs. 2.05s makes 24 threads just 9% slower than 88 threads, providing a better runtime-resource tradeoff.Best performance is achieved with 88 threads.
- Measurement: Figure 2 reports runtimes averaged across 10 trials for different thread counts.The figure evaluates the CryptoNets network.
A.3 Scalar Encoding
The scalar CKKS encoding proof shows that a vector containing the same real value in every slot produces a plaintext containing that value in every slot.
- Encoding property: A constant real input vector c yields a CKKS plaintext p containing the same real number in every slot.This is the encoding property established by Lemma 1.
- Proof: The inverse DFT maps the constant vector (r, r, ..., r) to (r, 0, 0, ..., 0).The proof uses the DFT^-1 matrix representation and cancellation from the roots-of-unity structure.
- Proof: The roots-of-unity identity makes every non-first row sum to zero, leaving only the first component after the inverse DFT.The first row of the transformation matrix consists of ones.
- Proof: Scaling by s and reducing modulo q produce ([rs]q, 0, ..., 0), which the negacyclic NTT maps to [rs]q in every slot.This completes the argument that CKKS encoding preserves a constant scalar across slots.
A.4 SEAL Performance Test
The SEAL CKKS performance test reports runtimes averaged over 1,000 trials at 128-bit security. Runtime increases with N and L, while larger L supports more multiplications but requires larger N to maintain security.
- Runtime increases as the CKKS parameters N and L grow.
- Larger L supports more multiplications, but maintaining the same security level requires increasing N accordingly.
- The test uses parameters satisfying λ = 128-bit security and averages runtimes across 1,000 trials.