PaperlayerView exact revision on arXiv

Source-linked AI summary

CryptGPU: Fast Privacy-Preserving Machine Learning on the GPU

Sijun Tan, Brian Knott, Yuan Tian, David J. Wu

arXiv:2104.10949v1cs.CRcs.LG

TL;DR

Privacy-preserving machine learning needs GPU acceleration to scale, but GPU libraries target floating-point operations while cryptographic protocols use discrete spaces. CRYPTGPU implements cryptographic operations on the GPU and substantially accelerates private inference and training for modern deep-learning models.

  • Problem

    GPU acceleration is essential for scaling privacy-preserving deep learning, yet CUDA libraries target floating-point operations while cryptographic protocols typically use discrete spaces.

  • Method

    CRYPTGPU implements linear and non-linear cryptographic operations on the GPU through an MPC framework built on PyTorch and CRYPTEN.

  • Results

    CRYPTGPU achieves 2.3× faster private inference than CRYPTFLOW for ResNet-152 on ImageNet and reduces AlexNet private-training iterations on Tiny ImageNet from 6.9 minutes to 11.30s versus FALCON.

  • Takeaways & Limitations

    GPU-based MPC makes privacy-preserving deep learning practical at ImageNet scale and supports complex networks, while motivating GPU-friendly cryptographic protocol design.

  • Takeaways & Limitations

    CRYPTGPU does not support batch normalization during private training, so it reports no ResNet-family private-training benchmarks.

Abstract

from arXiv · show

We introduce CryptGPU, a system for privacy-preserving machine learning that implements all operations on the GPU (graphics processing unit). Just as GPUs played a pivotal role in the success of modern deep learning, they are also essential for realizing scalable privacy-preserving deep learning. In this work, we start by introducing a new interface to losslessly embed cryptographic operations over secret-shared values (in a discrete domain) into floating-point operations that can be processed by highly-optimized CUDA kernels for linear algebra. We then identify a sequence of "GPU-friendly" cryptographic protocols to enable privacy-preserving evaluation of both linear and non-linear operations on the GPU. Our microbenchmarks indicate that our private GPU-based convolution protocol is over 150x faster than the analogous CPU-based protocol; for non-linear operations like the ReLU activation function, our GPU-based protocol is around 10x faster than its CPU analog. With CryptGPU, we support private inference and private training on convolutional neural networks with over 60 million parameters as well as handle large datasets like ImageNet. Compared to the previous state-of-the-art, when considering large models and datasets, our protocols achieve a 2x to 8x improvement in private inference and a 6x to 36x improvement for private training. Our work not only showcases the viability of performing secure multiparty computation (MPC) entirely on the GPU to enable fast privacy-preserving machine learning, but also highlights the importance of designing new MPC primitives that can take full advantage of the GPU's computing capabilities.

I. INTRODUCTION

Privacy-preserving deep learning must address sensitive data while overcoming MPC overhead that limits scalability. CRYPTGPU uses GPU-based cryptographic computation to support larger models and datasets with substantially improved performance.

  • Motivation: Sensitive training and inference data create privacy challenges for widely deployed deep-learning applications.Applications include voice assistants, surveillance, and disease-related systems.
  • Motivation: MPC hides parties’ inputs while enabling joint computation, but existing approaches remain expensive and scale poorly to modern neural networks.The practical gap is especially pronounced for datasets such as ImageNet, which is over 1000× larger than CIFAR/MNIST.
  • Motivation: GPU acceleration is essential because MPC adds overhead to plaintext computation, while CPU-bound MPC cannot scale to deep learning workloads.Modern deep learning already relies heavily on GPUs and related accelerators.
  • Approach: CRYPTGPU implements cryptographic operations on GPUs and supports private inference for models including ResNet-152 on ImageNet.Its 3-party system uses replicated secret sharing and assumes security against one semi-honest corruption.
  • Results: 2.3× faster private inference is achieved for ResNet-152 on ImageNet compared with CRYPTFLOW, with single-image inference taking just over 25s.For AlexNet, private inference over ImageNet requires 1.5s.
  • Results: 36× speed-up is demonstrated for private AlexNet training on Tiny ImageNet compared with FALCON.CRYPTGPU reduces the estimated training time from over a year to just over a week.
  • Results: GPU execution accelerates secret-shared convolutions by over 150× and ReLU evaluation by 10× relative to corresponding CPU protocols.Fine-grained measurements attribute much of the advantage to 25× to 72× faster linear layers.

II. SYSTEM OVERVIEW

The system extends PyTorch and CRYPTEN with GPU-oriented, three-party MPC abstractions for private machine learning. Its central challenge is mapping discrete cryptographic computation onto GPU libraries designed for floating-point operations.

  • Framework: CRYPTGPU adapts CRYPTEN’s PyTorch-based architecture to support three-party protocols using replicated secret sharing.CRYPTEN preserves PyTorch APIs while providing secure computation over secret-shared tensors.
  • GPU execution: GPU parallelism suits convolutions, pooling, matrix multiplication, and point-wise neural-network functions.CUDA libraries provide optimized implementations of these primitives, but they are designed for floating-point computation.
  • GPU execution: A core system challenge is that MPC operates over discrete spaces whereas CUDA deep-learning kernels lack integer-valued analogs.The design therefore requires embedding cryptographic computation into floating-point operations without directly translating CPU protocols.
  • Framework: The system represents secret-shared values through tensor-like abstractions while using arithmetic sharing for bilinear operations and Boolean sharing for non-linear operations.CRYPTEN uses Beaver triples, GMW circuits, and polynomial approximations for its cryptographic building blocks.

B. System Design and Architecture

CryptGPU keeps cryptographic computation on the GPU by embedding secret-shared integer operations into floating-point kernels and selecting GPU-friendly protocols for linear and nonlinear operations. Its block decomposition preserves exactness while exploiting CUDA parallelism, although it increases memory use and can constrain training batch sizes.

  • GPU-friendly cryptography: CryptGPU keeps nonlinear protocols on the GPU by favoring secret-sharing schemes and component-wise operations over protocols with conditionals or extensive finite-field arithmetic.The design explicitly targets GPU-friendly cryptography for both linear and nonlinear computation.
  • Floating-point embedding: 64-bit floating-point arithmetic exactly represents sufficiently small integers, enabling lossless embedding of discrete cryptographic operations.The approach relies on 52 bits of floating-point precision and exact products for appropriately bounded operands.
  • Floating-point embedding: Bilinear operations such as matrix multiplication and convolution are decomposed into blockwise pairwise products, then recombined using elementwise additions and scalar multiplications.With k = 4 blocks of w = 16 bits, each operation expands into 10 pairwise products.
  • Correctness and overhead: Using 16-bit blocks avoids precision overflow for large bilinear computations, supporting operations involving up to 2^20 intermediate products.Three 22-bit blocks can exceed available floating-point precision when outputs sum many products.
  • Correctness and overhead: Blockwise decomposition incurs roughly 2× overhead for 64 × 64 inputs and 9× for 224 × 224 inputs, while increasing memory requirements and limiting training batch size.Larger training batches may require distributing computation across multiple GPUs.
  • System interface: CryptGPU introduces CUDALongTensor to represent secret-shared 64-bit integers while dispatching supported operations to CUDA kernels.Integer operations use direct kernels when available; bilinear operations use the decomposition-and-recombination method.

III. THREAT MODEL AND CRYPTOGRAPHIC DESIGN

The system formalizes three-party secure computation for private inference and training under an honest-majority, semi-honest threat model. Secret sharing and simulation-based security specify how inputs, outputs, and intermediate views remain protected.

  • Threat model: CryptGPU assumes three computing parties in an honest-majority model with at most one semi-honest corruption.Parties follow the protocol but may try to learn information about others’ inputs.
  • Threat model: Security is defined through simulation: a corrupted party’s execution view should be indistinguishable from a simulator’s output given the allowed information.The formal definition covers randomized functionalities and a single semi-honest corruption.
  • Secret sharing: The framework models additive and replicated secret sharing through Share and Reconstruct algorithms with threshold correctness and perfect security properties.Shares can reconstruct the value when enough shares are available, while smaller subsets reveal no information under the security definition.
  • Functionalities: Private inference maps secret shares of a model and input to secret shares of Eval(M, x), without reconstructing those values for the parties.The ideal functionality reconstructs internally and returns only a share of the model output.
  • Functionalities: Private training maps secret shares of a dataset to secret shares of Train(D), while parties learn nothing about the dataset or resulting model.The ideal functionality reconstructs the dataset internally before applying the training algorithm.

B. Cryptographic Building Blocks for Private Inference

Private inference is built by composing secure protocols for linear, bilinear, and ReLU operations on secret-shared values. Arithmetic sharing, fixed-point encoding, bit protocols, and GPU kernels provide the main building blocks.

  • Protocol composition: Inference decomposes a neural network into linear, pooling, convolution, and ReLU operations, then sequentially composes secure protocols for those operations.Correctness and semi-honest security follow from composing the elementary protocols.
  • Representation: The implementation uses replicated secret sharing over a 64-bit power-of-two ring and fixed-point encoding to represent machine-learning values in a discrete domain.Each party holds a pair of shares in the default 2-out-of-3 scheme.
  • Arithmetic protocols: Linear operations require only local computation on each party’s shares, while multiplication adds communication to produce a shared product.The multiplication protocol locally forms share products and then redistributes blinded shares.
  • Arithmetic protocols: Share truncation is inexact and can introduce at most 1 bit of error, bounded by 2^-t for t bits of fixed-point precision.The resulting numerical error and model-accuracy impact are assessed empirically.
  • Bilinear operations: Convolutions and matrix multiplications use vectorized secret-sharing protocols, with truncation applied only after the tensor product to preserve CUDA-kernel efficiency and reduce communication.This avoids modifying elementary multiplications inside optimized matrix and convolution kernels.
  • Nonlinear operations: ReLU evaluation reduces to computing the most significant bit of a fixed-point value and injecting that bit back into arithmetic shares.The binary addition circuit primarily runs on the GPU using bitwise operations and communication for AND gates.

C. Additional Building Blocks for Private Training

Private training extends inference with secure protocols for softmax, cross-entropy, division, maximum, and ReLU derivatives within minibatch SGD. The design uses normalized inputs, an iterative reciprocal approximation, and GPU-compatible elementary operations.

  • Training pipeline: Private training composes secure protocols for backpropagation with softmax/cross-entropy loss and minibatch stochastic gradient descent.The algorithm is decomposed into elementary operations, as in private inference.
  • Softmax: Softmax is evaluated on x − max_i x_i, preserving its value while keeping exponential inputs at most 0 and the denominator within [1, d].The protocol combines secure maximum, exponentiation, and division.
  • Exponentiation: The degree-m approximation f_m computes exponentials with logarithmic multiplication depth and a smaller operation count than a degree-m Taylor approximation.For fixed degree m, f_m requires log m multiplications versus m multiplications for the Taylor method.
  • Exponentiation: The f_m approximation is accurate over an interval of size O(√m) centered at zero, while Taylor approximation offers an interval of size O(m) but can diverge for large negative inputs.The choice trades interval width against computational cost and behavior on the negative half-line.
  • Division: Secret-shared division uses Newton-Raphson reciprocal iteration initialized at z_0 = 1/Y, achieving a highly accurate estimate over y ∈ [1, Y] in O(log Y) iterations.The error decreases quadratically across iterations.
  • Maximum and derivatives: The maximum over a secret-shared vector is computed through a comparison tree, reducing the round complexity to log m.Each comparison is implemented through the most significant bit of a difference.
  • Maximum and derivatives: Backpropagation computes ReLU derivatives through the same most-significant-bit protocol used for ReLU evaluation, while linear and convolution gradients remain linear operations.The derivative is 0 for negative inputs and 1 for positive inputs.

IV. SYSTEM IMPLEMENTATION AND EVALUATION

CRYPTGPU integrates secret-shared cryptographic operations with GPU-oriented tensor computation and communication primitives. Its implementation supports bilinear operations, pseudorandom generation, and point-to-point communication on GPU-backed infrastructure.

  • GPU computation: CUDALongTensor represents 64-bit integer-valued PyTorch tensors for secret-shared computation.It enables CUDA kernels to process bilinear operations such as convolutions and matrix multiplications.
  • GPU computation: Optimized CUDA kernels evaluate convolutions and matrix multiplications on secret-shared tensors.These operations provide the elementary building blocks for arithmetic circuits.
  • Communication: PyTorch distributed communication is configured as separate pairwise backends to emulate point-to-point channels between parties.This matches the communication structure required by the protocol.
  • GPU cryptography: AES serves as the protocol’s pseudorandom function for share rerandomization during truncation.A PyTorch C++/CUDA extension enables AES evaluation on the GPU.

A. Experimental Setup for System Evaluation

The evaluation covers standard object-recognition datasets and representative CNN architectures, using protocol parameters and architecture adjustments tailored to private GPU computation. The setup includes fixed-point arithmetic, approximate exponentiation and division, and average-pooling substitutions whose accuracy and performance effects are assessed.

  • Datasets: CRYPTGPU is evaluated on MNIST, CIFAR-10, Tiny ImageNet, and ImageNet object-recognition datasets.The datasets range from small handwritten-digit and 10-class image benchmarks to ImageNet with more than 1,000,000 training images and 1000 classes.
  • Models: The benchmark models include LeNet, AlexNet, VGG-16, and ResNet-50, ResNet-101, and ResNet-152.These architectures combine standard convolutional, pooling, activation, normalization, softmax, and fully connected layers.
  • Architecture adjustments: AlexNet and VGG-16 are modified for smaller inputs by changing pooling and fully connected classification layers.The adjustments make ImageNet-designed architectures compatible with CIFAR-10 and Tiny ImageNet.
  • Activation functions: All evaluated networks except LeNet use ReLU, while LeNet’s tanh is replaced with ReLU because CRYPTGPU does not support private tanh evaluation.This change aligns the experiment with the activation supported by the system.
  • Pooling: Average pooling replaces max pooling in CRYPTGPU’s networks, but pooling is not the existing-system bottleneck and the substitution does not significantly affect measured accuracy.The evaluation separately examines whether this adjustment explains performance differences.
  • Protocol parameters: Inputs use 64-bit-ring secret sharing with 20 bits of fixed-point precision; exponentiation and division use approximate protocols with measured errors.The exponential approximation uses m = 512 and has maximum error at most 6 · 10^-4 on x ≤ 0; division error is approximately 10^-4 over the evaluated interval.

B. Benchmarks for Private Training and Inference

CRYPTGPU is evaluated for private inference and training against FALCON and CRYPTFLOW in LAN settings. GPU acceleration provides the strongest advantages for larger models and datasets, while memory, communication, unsupported layers, and remaining plaintext gaps constrain scalability.

  • Private inference: CRYPTGPU outperforms FALCON for larger private-inference workloads, reaching 3.7× faster on VGG-16 with Tiny ImageNet.FALCON remains faster on shallow networks and small datasets, where GPU parallelism provides less benefit.
  • Private inference: 25 seconds enables private inference for ResNet-152 on ImageNet, about 2.2× faster than CRYPTFLOW.CRYPTGPU’s ResNet-family running time scales linearly with network depth.
  • Batch inference: 53× and 12× reductions lower amortized per-image inference cost for AlexNet and VGG-16, respectively, when processing CIFAR-10 batches of 64 images.AlexNet drops from 0.91s to 0.017s per image, while VGG-16 drops from 2.14s to 0.18s.
  • Private training: 36× faster private training reduces AlexNet on Tiny ImageNet from over a year with FALCON to just over a week with CRYPTGPU.A single private backpropagation iteration takes 11.30s with CRYPTGPU versus 6.9 minutes with FALCON.
  • Private training: CRYPTGPU remains 30× faster on CIFAR-10 and 26× faster on Tiny ImageNet when training VGG-16 under matched batch-size adjustments.The comparison uses the same batch-size adjustment for both systems.
  • Training breakdown: 25×–70× faster linear-layer evaluation is the main training advantage over FALCON, whose linear layers account for 86%–99% of computational cost.CRYPTGPU’s computation is more evenly divided between linear and non-linear layers.
  • Limitations: The system is more communication-intensive than FALCON, and its VGG-16 training is constrained by available GPU memory.Supporting multiple GPUs and reducing protocol memory overhead are identified as future directions.
  • Limitations: CRYPTGPU does not support batch normalization during private training, so the evaluation reports no private-training benchmarks for the ResNet family.A GPU-friendly batch-normalization protocol is identified as necessary for broader ResNet training support.

C. Microbenchmarks

CryptGPU’s GPU-based MPC protocols substantially accelerate private convolutions and ReLU evaluation, especially for large inputs and batches. The benchmarks compare CPU and GPU execution under LAN conditions across representative CNN workloads.

  • Private convolution: GPU vs. CPU: 174× faster private convolution on a 512 × 512 × 3 input, reducing runtime from 23.9s on the CPU to 0.14s on the GPU.
  • Private convolution: GPU vs. CPU: 168× faster private convolution on a 64 × 64 × 512 input, reducing runtime from 543s on the CPU to 3.2s on the GPU.
  • Private convolution: GPU vs. CPU: GPU runtime remains essentially constant for small convolution instances, whereas CPU runtime scales with input size.
  • Private convolution: GPU vs. CPU: 40× to 60× speed-ups occur for batched private convolutions, compared with a 10× speed-up for a single input.
  • Private ReLU: GPU vs. CPU: 16× faster private ReLU evaluation on 256,000 inputs, decreasing runtime from 2s on the CPU to 0.12s on the GPU.
  • Private ReLU: GPU vs. CPU: 9× faster private ReLU evaluation on 32 million inputs, reducing runtime from 149s on the CPU to 16.3s on the GPU.

D. Accuracy of Privacy-Preserving Protocols

CRYPTGPU evaluates private inference and training with small output discrepancies relative to plaintext, while fixed-point precision becomes more important for deeper models and larger datasets. The evaluation also examines protocol approximation and pooling choices.

  • Protocol accuracy: CRYPTGPU uses fixed-point encodings, share truncation, and approximations that can introduce a small amount of error.
  • Fixed point precision: Under 0.02% average relative error for private inference on ResNet-50 over ImageNet when using t = 20 bits of fixed-point precision.
  • Fixed point precision: Around 1% relative error is observed for AlexNet on CIFAR-10 with 12 to 14 bits of fixed-point precision, while ResNet-50 on ImageNet reaches almost 5%.
  • Privacy-preserving inference: At most 0.021% average relative error, with Top-1 and Top-5 accuracies exactly matching plaintext across tested ResNet models on ImageNet.
  • Privacy-preserving training: CRYPTGPU’s trained models closely match plaintext model accuracy across the evaluated model and dataset configurations.
  • Average pooling vs. max pooling: Replacing max pooling with average pooling causes a 3% accuracy drop for AlexNet on CIFAR-10, from 76% to 73%.

V. RELATED WORK

Prior privacy-preserving machine-learning systems addressed increasingly larger models and datasets, but substantial computational overhead and limited GPU use remained. CryptGPU targets scalable inference and training by keeping cryptographic computation on the GPU.

  • General cryptographic approaches support privacy-preserving machine learning but incur significant overhead.
  • Existing privacy-preserving inference systems span different party models and neural-network architectures.
  • Privacy-preserving inference: Earlier approaches largely focused on shallow networks and datasets at the scale of MNIST or CIFAR, whereas this work targets modern deep models and ImageNet-scale data.
  • Privacy-preserving training: Privacy-preserving training is more challenging and computationally intensive than inference, and FALCON was previously the only system scaling beyond MNIST/CIFAR.
  • Privacy-preserving machine learning using GPUs: Most privacy-preserving machine-learning systems are CPU-based; CryptGPU instead keeps linear and non-linear cryptographic operations on the GPU.
  • Model stealing and inversion attacks: MPC hides inputs only up to information inferable from outputs, leaving model-stealing and inversion attacks as relevant concerns.

VI. CONCLUSION

CryptGPU is an MPC framework that runs cryptographic operations for privacy-preserving machine learning on GPUs. The system accelerates private training and inference for modern deep learning at ImageNet scale while motivating GPU-friendly MPC protocol design.

  • CryptGPU implements both linear and non-linear cryptographic operations on the GPU.
  • The framework accelerates private training and inference for modern deep-learning models and makes ImageNet-scale privacy-preserving deep learning practical.
  • CryptGPU provides insights for designing GPU-friendly cryptographic protocols and aims to narrow the roughly 1000× gap between private and plaintext GPU machine learning.

APPENDIX A NETWORK ARCHITECTURE

The evaluated AlexNet and VGG-16 models retain dataset-independent base architectures while adapting their classification heads to input dimensions and output classes. The appendix documents these base and head configurations across CIFAR-10, Tiny ImageNet, and ImageNet.

  • Base architectures: AlexNet and VGG-16 keep the same base architecture across datasets while using different input dimensions for CIFAR-10, Tiny ImageNet, and ImageNet.
  • Head architectures: Classification heads vary across datasets as a function of input size and number of output classes.
  • AlexNet heads: The AlexNet appendix includes flattening, fully connected, and ReLU layers in its head architecture.
  • Head architectures: Figures 6 and 8 show the dataset-specific AlexNet and VGG-16 classification heads.
  • VGG-16 heads: The VGG-16 appendix includes flattening, fully connected, and ReLU layers with a 512-dimensional flattened input.
Loading 2104.10949v1…