PaperlayerView exact revision on arXiv

Source-linked AI summary

CrypTen: Secure Multi-Party Computation Meets Machine Learning

Brian Knott, Shobha Venkataraman, Awni Hannun, Shubho Sengupta, Mark Ibrahim, Laurens van der Maaten

arXiv:2109.00984v2cs.LGcs.CR

TL;DR

Secure MPC could support machine learning on private data, but adoption is limited by frameworks that do not match machine-learning workflows. CrypTen addresses this gap with a PyTorch-like secure-MPC framework and benchmarks efficient private evaluation across text, speech, and image models, including faster-than-real-time speech prediction.

  • Problem

    Secure MPC enables collaborative computation without revealing parties’ data, but machine-learning adoption is limited by the lack of flexible frameworks aligned with researchers’ and developers’ workflows.

  • Method

    CrypTen exposes secure MPC through tensor computations, reverse-mode automatic differentiation, modular neural networks, and GPU support modeled on PyTorch.

  • Results

    Benchmarks across text classification, speech recognition, and image classification demonstrate efficient private model evaluation, including faster-than-real-time phoneme prediction and 2.49-second two-party ResNet-18 evaluation.

  • Takeaways & Limitations

    CrypTen’s flexible machine-learning-first API and performance may help foster adoption of secure MPC in machine learning.

  • Takeaways & Limitations

    Encrypted computation makes intermediate activations and model outputs difficult to inspect, complicating quality control, explanation, and detection of data-poisoning attacks.

Abstract

from arXiv · show

Secure multi-party computation (MPC) allows parties to perform computations on data while keeping that data private. This capability has great potential for machine-learning applications: it facilitates training of machine-learning models on private data sets owned by different parties, evaluation of one party's private model using another party's private data, etc. Although a range of studies implement machine-learning models via secure MPC, such implementations are not yet mainstream. Adoption of secure MPC is hampered by the absence of flexible software frameworks that "speak the language" of machine-learning researchers and engineers. To foster adoption of secure MPC in machine learning, we present CrypTen: a software framework that exposes popular secure MPC primitives via abstractions that are common in modern machine-learning frameworks, such as tensor computations, automatic differentiation, and modular neural networks. This paper describes the design of CrypTen and measure its performance on state-of-the-art models for text classification, speech recognition, and image classification. Our benchmarks show that CrypTen's GPU support and high-performance communication between (an arbitrary number of) parties allows it to perform efficient private evaluation of modern machine-learning models under a semi-honest threat model. For example, two parties using CrypTen can securely predict phonemes in speech recordings using Wav2Letter faster than real-time. We hope that CrypTen will spur adoption of secure MPC in the machine-learning community.

1 Introduction

Secure MPC enables collaborative computation on private data, but machine-learning adoption is limited by a lack of flexible, ML-oriented frameworks. CrypTen addresses this gap with a PyTorch-like framework and benchmarks efficient private inference and training.

  • Secure MPC lets parties compute on combined data without revealing their individual data.Potential applications include joint medical-model training and privacy-preserving wage-gap analysis.
  • CrypTen makes secure MPC accessible to machine-learning researchers and developers without cryptography backgrounds.It is presented specifically to foster adoption of secure MPC in machine learning.
  • CrypTen provides secure tensor computations, a PyTorch-like API, automatic differentiation, modular neural networks, arbitrary-party support, GPU execution, and high-performance communication.These features target efficient private training and inference under a semi-honest threat model.
  • CrypTen benchmarks private text classification, speech recognition, and image classification using state-of-the-art models.The paper also discusses open problems and a roadmap for further development.
  • Two parties can privately classify an image in 2-3 seconds, while phoneme prediction for 16kHz speech recordings runs faster than real-time.

2 Related Work

CrypTen is positioned among secure MPC systems as a flexible, machine-learning-focused framework supporting more parties and broader ML functionality than several related systems.

  • CrypTen is related to systems implementing secure MPC protocols for machine learning, including CryptGPU and Falcon.CryptGPU uses 2-out-of-3 replicated secret sharing and is limited to three parties, while Falcon provides malicious security in a three-party setting.
  • Compared with several two-party semi-honest systems, CrypTen offers a more flexible ML-focused API, reverse-mode automatic differentiation, rich functions, and native GPU execution.
  • CrypTen supports a wider range of use cases by working with an arbitrary number of parties.

3 Design Principles

CrypTen follows two design principles: a machine-learning-first API and eager execution. These choices align secure computation with familiar ML development workflows while retaining performance-oriented implementations.

  • Machine-learning first API: CrypTen’s machine-learning-first API provides PyTorch tensor operations, reverse-mode automatic differentiation, modular neural networks, learning routines, and GPU computation.The design aims to let developers transition code from familiar machine-learning frameworks.
  • Eager execution: CrypTen adopts an imperative programming model instead of compiling computations into a domain-specific language.
  • Eager execution: Eager execution is intended to improve the development experience by reducing development-cycle delays, debugging difficulty, and restrictions on host-language constructs.
  • Eager execution: CrypTen combines eager execution with state-of-the-art secure MPC protocols and PyTorch’s optimized tensor library to remain performant.

4 Design Overview

CrypTen presents secure computation through PyTorch-like tensor abstractions, secret sharing, automatic differentiation, neural-network APIs, and GPU-supported execution. Its implementation assumes synchronized execution and, for efficient Beaver triples, a trusted third party.

  • CrypTen maps PyTorch tensor operations onto arithmetic and binary secret shares, converting between share types when required.Integer secure computations are mapped to floating-point GPU computations because GPUs do not natively support the required integer operations.
  • All secure computations use CrypTensor objects that implement the PyTorch tensor API and reverse-mode automatic differentiation.Each party executes the same code, with communication synchronizing parties when required.
  • Private multiplication uses Beaver triples generated by a trusted third party for efficiency.The paper plans alternatives based on additive homomorphic encryption or oblivious transfer to remove the trusted-third-party requirement.
  • CrypTen relies on MPI communication primitives, with each party knowing its rank and the total number of parties.
  • CrypTen supports encrypted neural networks, automatic differentiation, PyTorch-style training loops, ONNX import, and private ResNet-18 inference.All parties must use the same device type, either CPU or GPU.

5 Secure Computations

CRYPTEN implements secure computations with arithmetic and binary secret sharing, conversions between them, and private addition and multiplication supporting linear, nonlinear, and comparison functions. It accelerates fixed-point secure computation on GPUs through floating-point representations and parallelized component products, while assuming a semi-honest threat model.

  • Secret sharing: Arithmetic and binary secret sharing, plus conversions between them, provide CRYPTEN’s foundation for secure computation.Arithmetic sharing supports operations such as matrix multiplication and convolution, while binary sharing supports functions such as rectified linear units.
  • Private computation: Private addition and Beaver-triple-based multiplication enable secure linear operations including dot products, matrix products, and convolutions.Linear functions are composed from private addition and multiplication using secret shares.
  • Nonlinear functions: Standard iterative approximations implement exponentials, logarithms, reciprocals, and machine-learning functions such as sigmoid, softmax, and logistic loss.Comparators use arithmetic-to-binary conversion, sign-bit extraction, and conversion back to arithmetic sharing.
  • Security model: Semi-honest security enables broad secure-machine-learning use cases while remaining more efficient than maliciously secure protocols.The protocol supports the semi-honest threat model described by the paper.
  • GPU acceleration: CRYPTEN off-loads secure computations to GPUs by splitting variables into fixed-width components whose pairwise products use optimized floating-point CUDA kernels.The same strategy is applied to matrix multiplications and convolutions, with a three-component optimization reducing pairwise products to 6 when possible.

6 Benchmarks

CRYPTEN benchmarks private inference across text, speech, and image classification, comparing secure computation with PyTorch on CPUs and GPUs. The results show substantial overhead versus PyTorch, but GPU acceleration and batching make several private-inference workloads practical.

  • Benchmark scope: CRYPTEN benchmarks private inference for text classification, Wav2Letter speech recognition, and ResNet-18 and ViT-B/16 image classification.The experiments use secret-shared models and secret-shared data, with CPU and GPU evaluations in separate processes on one machine.
  • Text classification: 2.5–3 orders of magnitude slower than PyTorch, CRYPTEN’s text-sentiment inference still takes 0.03 seconds per sample for two parties with batch size 32.The main text-classification overhead comes from replacing sparse embedding lookups with a full matrix multiplication on encrypted inputs.
  • Text classification: Larger batch sizes reduce inference time and communication per sample, while communication rounds remain independent of batch size.Weight tensors can be amortized across larger batches; moving from two to three parties adds rounds because of the public-division protocol, after which the count remains constant.
  • Speech recognition: 2.5–3 orders of magnitude slower than PyTorch, Wav2Letter’s CRYPTEN inference incurs overhead largely from ReLU layers and their arithmetic-to-binary share conversions.Communication rounds increase beyond four parties because the comparator protocol uses tree reduction.
  • Speech recognition: 1–2 orders of magnitude faster on GPUs than CPUs, Wav2Letter evaluation is compute-bound on CPUs but communication-bound on GPUs.For eight-party computation on GPUs, 63% of wall-clock time is spent communicating.
  • Image classification: 2.49 seconds for ResNet-18 and 8.47 seconds for ViT-B/16, two parties can securely evaluate both image-classification models.Image-model communication is dominated by activation-sized tensors, so increasing batch size does not reduce bytes per sample; ViT communication is especially high because of repeated softmax evaluation.

7 Conclusion and Future Work

CRYPTEN’s future development centers on adoption, integration, stronger privacy guarantees, broader threat-model support, and architectures optimized for secure MPC.

  • CRYPTEN aims to foster adoption of secure MPC in machine learning through its flexible, machine-learning-first API and performance.
  • End-to-end privacy requires integrating private data-processing frameworks with CRYPTEN to support complete machine-learning pipelines.
  • Differential privacy may be required to provide rigorous guarantees against information leakage when private computation results are publicly revealed.
  • Threat-model requirements vary by use case, motivating extensions for malicious security and settings without a trusted third party.
  • Existing model architectures may be suboptimal for secure MPC because communication costs differ from plaintext GPU or TPU computation.

8 Broader Impact

The paper highlights practical limitations of private machine-learning systems, including reduced observability, implementation vulnerabilities, and the complexity of their secure-computation foundations.

  • Encrypted computation makes quality control harder because intermediate activations and model outputs cannot be inspected.
  • Practical implementations may remain susceptible to side-channel attacks despite rigorous cryptographic guarantees, so data stewardship remains essential.
  • CRYPTEN’s security guarantee relies on secure arithmetic and binary sharing protocols, secure conversions, non-interactive indexing, and composition.
  • Arithmetic and binary secret sharing support different operations, with arithmetic sharing used for many computations and binary sharing for bit-oriented functions.

C.1.1 Arithmetic Secret Sharing

CRYPTEN combines arithmetic and binary secret sharing to implement private numerical and bitwise computation, while addressing wrap-around and truncation errors in shared arithmetic.

  • Arithmetic secret sharing distributes ring-valued scalars across parties so their shares reconstruct the original value.
  • Fixed-point encoding represents real values by scaling and rounding, then requires downscaling after multiplication, introducing precision loss.
  • Beaver triples enable private multiplication, matrix multiplication, and convolution by masking operands and combining secret-shared products.The resulting product is [c] + ϵ[b] + [a]δ + ϵδ.
  • Simple share-wise division can fail when shares wrap around the ring, so CRYPTEN privately computes wrap counts to correct truncation.
  • Skipping a difficult auxiliary wrap computation can produce incorrect results with probability ⌊B x̂⌉/Q, reducible by increasing Q or lowering B.
  • Binary secret sharing operates over Z/2Z and is used in CRYPTEN primarily for comparators because sequential AND gates require communication rounds.

C.1.3 Converting Between Secret-Sharing Types

CRYPTEN combines arithmetic and binary secret sharing, converting between them to support both numerical operations and logic-based functions. It implements comparisons, multiplexing, extrema, and numerical approximations for common nonlinear operations.

  • Secret-sharing conversions: CRYPTEN uses arithmetic shares for operations such as matrix multiplication and binary shares for logic-based operations such as argmax, converting between them as needed.The conversion protocols are designed for multi-party computation using both sharing types.
  • Secret-sharing conversions: The arithmetic-to-binary conversion constructs binary shares of every arithmetic share and sums them with a carry-lookahead or lower-memory adder.The carry-lookahead circuit uses log2(|P|) log2(Q) rounds, while the alternative uses |P| log2(L) communication rounds.
  • Comparisons and branching: Comparisons convert an arithmetic difference to a binary share, extract its sign bit, and convert that bit back to an arithmetic share.This mechanism supports arbitrary comparators and derived functions such as sign, absolute value, and ReLU.
  • Comparisons and branching: Multiplexing evaluates conditional expressions as [c][x] + (1 −[c])[y], allowing piece-wise functions while requiring both branches to be evaluated.The same technique supports sign, absolute value, and ReLU computations.
  • Extrema: CRYPTEN computes argmax using tree reduction by default and can use pairwise comparisons when network bandwidth or latency favors that configuration.Tree reduction requires O(log2 N) rounds, O(N^2) communication bits, and O(N) computation complexity.
  • Numerical approximations: CRYPTEN uses numerical approximations and iterative methods for expensive functions, with configurable domains, initializations, and convergence behavior.Examples include repeated squaring for exponentials, Newton-Raphson methods for reciprocals and square roots, and domain-dependent approximations.

C.2.6 Gaussian Error Function

CRYPTEN implements secret-shared sampling for several probability distributions using uniform randomness, bit conversions, inverse transforms, and numerical approximations. Quantization and finite-precision domains shape the resulting distributions and approximation quality.

  • Randomness generation: CRYPTEN generates secret-shared random values so that no party can learn the sampled realizations.The framework provides methods for several popular distributions.
  • Uniform and Bernoulli sampling: Uniform samples are constructed from locally generated secret-shared Rademacher bits, with quantization limiting the output to 2^L possible values.The XOR sum of independent Rademacher variates remains uncorrelated with any individual input.
  • Uniform and Bernoulli sampling: Bernoulli samples are generated by comparing a uniform sample with the private probability parameter, whose effective value is quantized to the nearest multiple of 2^-L.This quantization follows from the fixed-point representation.
  • Continuous distributions: Gaussian samples use the Box-Muller transform on two independent uniform samples, followed by scaling and shifting to obtain N(µ, σ^2).Approximations are optimized for the uniform-input domain [0, 1].
  • Continuous distributions: Exponential samples use the inverse CDF method, while Laplace samples combine an exponential sample with a Rademacher variate.The logarithm is optimized for inputs in [0, 1].
  • Weighted sampling: Weighted sampling computes a uniform draw over the total weight, compares it with cumulative weights, and converts the resulting mask into a one-hot vector.Finite fixed-point precision can introduce precision issues when generating the uniform draw.

D Comparison with Secure MPC Frameworks for Machine Learning

The comparison organizes secure MPC machine-learning frameworks by security, preprocessing, hardware, training, general-purpose evaluation, and automatic differentiation capabilities. It defines general-purpose support through five neural-network layers and notes a future limitation in CRYPTEN’s Beaver-triple generation.

  • Comparison dimensions: The framework comparison records support for malicious security, trusted-third-party-free Beaver triples, GPU computation, model training, general-purpose evaluation, and automatic differentiation.These properties are presented as the table’s comparison dimensions.
  • Comparison scope: The comparison focuses on five functions because they are described as the main building blocks of many deep-network architectures.CRYPTEN supports additional functions beyond those five.
  • Comparison dimensions: A framework is classified as general-purpose when it supports linear, convolution, ReLU, max-pooling, and batch-normalization layers.The definition is given in the table note.
  • Comparison scope: CryptGPU is built on CRYPTEN and therefore inherits many CRYPTEN features.The table note explicitly identifies this relationship.
  • Comparison scope: Future CRYPTEN versions are expected to generate Beaver triples without requiring a trusted third party.The table note presents this as planned future support rather than current support.
Loading 2109.00984v2…