Source-linked AI summary
QUOTIENT: Two-Party Secure Neural Network Training and Prediction
Nitin Agrawal, Ali Shahin Shamsabadi, Matt J. Kusner, Adrià Gascón
TL;DR
Secure DNN training remains difficult because generic secure computation does not scale well, while prior work often separates ML and cryptographic optimization. QUOTIENT jointly designs a discretized training algorithm and customized secure two-party protocol, achieving near-floating-point accuracy on most datasets and ∼6% absolute accuracy gains with >50× WAN speedup over prior 2PC training.
Problem
Generic secure computation does not scale to real-world ML, and prior secure training work largely optimizes models or cryptographic techniques separately.
Method
QUOTIENT co-designs discretized DNN training with a customized semi-honest secure two-party protocol using ternary weights and MPC-aware operations.
Results
∼6% absolute accuracy gains and >50× WAN speedup are achieved over prior 2PC secure DNN training for both training and prediction.
Takeaways & Limitations
QUOTIENT improves secure DNN training and prediction by jointly optimizing the learning procedure and secure-computation protocol.
Takeaways & Limitations
The protocols consider total time rather than an assumed offline preprocessing phase and convolutional-network training remains slow with high communication load.
Abstract
from arXiv · showhide
Recently, there has been a wealth of effort devoted to the design of secure protocols for machine learning tasks. Much of this is aimed at enabling secure prediction from highly-accurate Deep Neural Networks (DNNs). However, as DNNs are trained on data, a key question is how such models can be also trained securely. The few prior works on secure DNN training have focused either on designing custom protocols for existing training algorithms, or on developing tailored training algorithms and then applying generic secure protocols. In this work, we investigate the advantages of designing training algorithms alongside a novel secure protocol, incorporating optimizations on both fronts. We present QUOTIENT, a new method for discretized training of DNNs, along with a customized secure two-party protocol for it. QUOTIENT incorporates key components of state-of-the-art DNN training such as layer normalization and adaptive gradient methods, and improves upon the state-of-the-art in DNN training in two-party computation. Compared to prior work, we obtain an improvement of 50X in WAN time and 6% in absolute accuracy.
1 Introduction
QUOTIENT jointly designs a discretized DNN training algorithm and a customized secure two-party protocol, addressing the limited scalability of generic secure computation for ML. It reports near-floating-point accuracy on most datasets, with substantial gains over prior secure DNN training.
- Motivation: Generic secure computation protocols do not scale well to real-world machine-learning applications, motivating task-specific protocols.Prior work customized cryptographic techniques for regressors, neural networks, matrix factorization, constrained optimization, and nearest-neighbor classification.
- Motivation: Recent secure ML work often optimizes either the machine-learning model or cryptographic techniques in isolation rather than designing them together.QUOTIENT instead targets a holistic co-design of the optimization algorithm and secure computation protocol.
- Contributions: QUOTIENT combines a new secure two-party DNN training and evaluation method with a semi-honest secure-computation implementation called 2PC-QUOTIENT.The method leverages repeated quantization to low fixed-point precisions, while the protocol is tailored to secure training.
- Contributions: Ternary weights W ∈ {−1, 0, 1} replace general weights during forward and backward passes, enabling a specialized Boolean/additive-sharing protocol based on Correlated Oblivious Transfer.Ternary matrix-vector multiplication becomes a central training primitive.
- Contributions: MPC-aware backward-pass alternatives replace costly quantization and normalization operations, and the approach extends to residual layers without affecting accuracy empirically.The paper identifies residual layers as an important DNN building block.
- Evaluation: ∼6% absolute accuracy gains and >50× WAN speedup are achieved over state-of-the-art 2PC secure DNN training for both training and prediction.Accuracy nearly matches state-of-the-art floating-point accuracy on 4 out of 5 datasets.
2 Overview and Problem Description
The paper reviews DNN layer operations, standard training, and the normalization, adaptive-step-size, quantization, and MPC primitives relevant to secure learning. Its overview connects common DNN computation with the secure two-party setting and efficient cryptographic operations.
- Deep Neural Networks: DNNs repeatedly apply layers that combine linear and nonlinear operations to map input a0 to output aL across network depth L.The overview covers fully-connected, convolutional, and residual layers.
- Deep Neural Networks: Fully-connected layers compute al = ReLU(Wl al−1), while convolutional layers apply structured local connectivity over tensor-valued inputs and outputs.Convolution uses moving square regions, weights, summation, and a nonlinear operation.
- Deep Neural Networks: Residual layers add an earlier activation to a later layer before applying the nonlinear function, supporting networks trained beyond 50 layers.The intermediate layers are usually convolutional but may be other layer types.
- DNN Training: SGD iterates over randomized inputs, forward prediction, backward gradient computation, and weight updates, with batches and epochs organizing training.The update uses learning rate η, and a pass over the training set is an epoch.
- Optimization: Normalization and adaptive step-sizes are presented as critical additions because normalization supports larger learning rates and adaptive methods adjust rates using prior gradients.Without normalization, convergence can be slower and lower minima may be unreachable; poorly chosen fixed rates can diverge or converge impractically slowly.
- MPC for Machine Learning: Correlated Oblivious Transfer is used as an efficient MPC primitive, reducing communication relative to general OT extension and supporting garbled circuits and OT-based triplet generation.In COT, the sender chooses a function relating the two messages rather than choosing both messages independently.
3 Deep Learning for MPC
The paper adapts fixed-point DNN training for MPC by co-designing quantization, optimization, and secure-computation-friendly operations. Its key changes include ternary weights, MPC-friendly gradient quantization, efficient normalization, and a fixed-point adaptive gradient method.
- Crypto-Friendly Modifications: QUOTIENT ternarizes weights to W ∈{−1, 0, 1} during forward and backward passes, enabling matrix multiplication through repeated 1-out-of-2 oblivious transfers.A specialized ternary matrix-vector multiplication protocol combines Boolean-sharing and additive-sharing.
- Fixed-Point Training: Fixed-point DNN training quantizes weights, activations, and gradients to finite sets of fixed-point numbers, with frequent quantization intended to stabilize optimization.The framework uses quantization functions Qw, Qa, Qд, and Qe for these quantities.
- Crypto-Friendly Modifications: Activation-gradient quantization normalizes by the next power of two of the maximum magnitude, avoiding the extra computation required to identify the closest power of two.The next-power-of-two operation is faster than the closest-power-of-two alternative.
- Crypto-Friendly Modifications: QUOTIENT replaces prior weight-gradient quantization with a saturation-free function that is significantly easier to implement using secure computation.The original function is described as unnecessarily removing information and adding secure-implementation overhead.
- Training Procedure: The training procedure maintains ternary low-precision weights for secure passes and higher-precision weights that receive gradient-based updates.The ternary weights are obtained by quantizing the higher-precision weights.
- Adaptive Optimization: The method redesigns AMSgrad for fixed-point secure computation by replacing square roots with absolute values, division with next-power-of-two division, and continuously quantizing moving sums.These changes target operations that may be unstable or costly in fixed-point computation.
4 Oblivious Transfer for Secure Learning
The section develops OT-based MPC protocols for secure DNN training and prediction by exploiting ternary weights, fixed-point values, and mixed sharing. These designs compose into efficient forward and backward passes with reported performance gains over prior work.
- Secure Ternary Matrix-Vector Multiplication: Ternary weights replace costly multiplications with selections, enabling faster secure matrix-vector products.The ternary product computes additions or subtractions of vector entries according to each weight.
- Secure Ternary Matrix-Vector Multiplication: The protocol represents ternary W using Boolean matrices W+ and W−, then combines Boolean and additive sharing without secure Boolean additions.The product is rewritten as W+a − W−a, reducing ternary multiplication to two binary-weight computations.
- Oblivious Transfer for Secure Learning: The core two-party subprotocol computes additive shares of Wa from Boolean-shared W and additively shared a using an inner-product protocol based on oblivious transfer.Protocol 6 takes arithmetic shares of an integer vector and Boolean shares of a binary vector; Protocol 7 provides a semi-honest secure COT-based variant.
- Secure Ternary Matrix-Vector Multiplication: The matrix-vector product protocol reduces communication to nm(τ + ℓ) bits per party, versus at least nmℓ(τ + ℓ) for prior arithmetic-sharing OT approaches.The comparison assumes optimizations such as packing and vectorization for the prior approach.
- Secure Forward Pass: The forward pass sequentially composes ternary matrix-vector multiplication with garbled-circuit ReLU, normalization, and quantization components.The backward pass uses Boolean representations for ReLU and saturation derivatives, alternating between Boolean and arithmetic shares through Protocol 7.
- Secure Backward Pass: The backward pass uses three small garbled circuits, two parallelizable, and relies heavily on oblivious transfer computations.Infinite-norm normalization approximates the maximum with bitwise OR and uses leading-zero counting and arithmetic shifts to reduce circuit cost.
5 Experiments
QUOTIENT is evaluated through component, layer, and end-to-end benchmarks across LAN and WAN settings, measuring accuracy, training time, and prediction time. The experiments report near-floating-point accuracy on most datasets and substantial speedups over SecureML, especially over WAN.
- Experimental settings: Experiments use two Azure machines with 0.3 ms latency and 1.82 GB/s bandwidth in LAN, versus 42 ms latency and 24.3 MB/s bandwidth in WAN.The machines were chosen to enable direct running-time comparisons with SecureML.
- Data-independent benchmarking: QUOTIENT’s COT-based component-wise multiplication outperforms competing approaches after base-OT setup overhead, particularly for layers with more than 10^4 multiplications.The comparison includes garbled circuits and SecureML’s OT and LHE protocols.
- Data-independent benchmarking: QUOTIENT’s matrix-vector multiplication is at least 5x faster than SecureML’s vectorized LHE on LAN and roughly 10x faster than its OT protocol on WAN for n ≥500.The speedup generally increases with the number of computations.
- Real-world data: QUOTIENT reaches near state-of-the-art floating-point accuracy on MNIST and Breast Cancer, while differing by at most ∼2% on German Credit, MotionSense, and Thyroid.Secure AMSgrad converges faster than secure SGD, especially for convolutional networks.
6 Conclusion
QUOTIENT securely trains DNNs using oblivious transfer while jointly optimizing the learning method and secure protocols. It improves accuracy and speed, but convolutional training remains slow and communication-heavy.
- QUOTIENT is a secure DNN training method that leverages oblivious transfer and improves state-of-the-art accuracy and speed.
- The work enables secure training of convolutional and residual layers, which are key building blocks of modern deep learning.
- Convolutional-network training remains slow and incurs a large communication load, motivating dedicated MPC protocols for faster convolutional evaluation.
A Standard AMSgrad Optimizer
The standard AMSgrad training procedure samples data, computes predictions and weight gradients, then updates weights using past gradient moments. The supplied tables describe prediction and training evaluations on Skin cancer MNIST.
- A Standard AMSgrad Optimizer: AMSgrad samples an input-target pair from dataset D before computing the network prediction and loss gradients.
- A Standard AMSgrad Optimizer: Table 7 reports prediction time for residual and fully-connected networks on Skin cancer MNIST over LAN with batch size 128.
- A Standard AMSgrad Optimizer: Table 8 reports training time and accuracy after 1, 5, and 10 epochs for fully-connected networks over LAN and WAN.
- A Standard AMSgrad Optimizer: The procedure computes each layer’s gradient with respect to its network weights.
- A Standard AMSgrad Optimizer: AMSgrad updates weights using weighted averages of past gradients represented by first and second moments M_l and V_l.
B Experiments on Residual Layers
2PC-QUOTIENT is evaluated on residual neural networks as well as fully-connected and convolutional layers. Prediction is measured on Skin cancer MNIST, while training timing is reported for practical fully-connected networks.
- B Experiments on Residual Layers: 2PC-QUOTIENT evaluates residual neural networks in addition to fully-connected and convolutional layers.
- B Experiments on Residual Layers: Table 7 measures prediction time on Skin cancer MNIST for fully-connected and residual neural networks.
- B Experiments on Residual Layers: Training timing is reported for practical fully-connected neural networks in Table 8.
C Proof of Protocol 6
Protocol 6 securely computes an additive share of the inner product between a Boolean vector and an integer vector under a secure OT protocol. Correctness follows from additive decomposition, while security uses fresh OT masking randomness.
- C Proof of Protocol 6: Given secure OT, Protocol 6 computes an additive share of b⊤a for Boolean vector b and integer vector a shared between two parties.
- C Proof of Protocol 6: Correctness follows because each product term b_j a_j is decomposed across the parties’ additive shares and accumulated through OT outputs.
- C Proof of Protocol 6: Security follows because OT messages are masked with fresh randomness, allowing straightforward simulators based on secure OT.
D Proof of Protocol 7
The proof establishes that Protocol 7 computes the correct value by analyzing all possible share configurations of each w_j. In the examined cases, the parties’ accumulated shares sum to w_ja_j, while privacy follows from correctness of the COT subprotocol.
- Privacy follows directly from the correctness of the COT subprotocol.
- The correctness proof considers each j ∈[m] and distinguishes cases based on all possible shares of w_j.
- When w_j = 0 with both shares equal to 1, the accumulated shares sum to 0, matching w_ja_j.
- When w_j = 1 with shares ([[w_j]]_1, [[w_j]]_2) = (0,1), the accumulated shares sum to a_j, matching w_ja_j.
- When w_j = 1 with shares ([[w_j]]_1, [[w_j]]_2) = (1,0), the protocol accumulates shares whose sum equals a_j.