PaperlayerView exact revision on arXiv

Source-linked AI summary

FALCON: Honest-Majority Maliciously Secure Framework for Private Deep Learning

Sameer Wagh, Shruti Tople, Fabrice Benhamouda, Eyal Kushilevitz, Prateek Mittal, Tal Rabin

arXiv:2004.02229v2cs.CRcs.LG

TL;DR

Private deep learning needs expressive protocols that support large-network training and inference while handling malicious behavior under realistic trust assumptions. Falcon addresses this with an honest-majority 3-party framework combining malicious security, batch normalization, and more efficient protocols. It reports substantial speed and communication improvements over prior systems, while leaving model-inversion, membership-inference, attribute-inference, and model-poisoning defenses out of scope.

  • Problem

    Existing private deep learning frameworks require broader expressiveness, scalability to large models, and stronger security guarantees for practical deployment.

  • Method

    Falcon is an end-to-end honest-majority 3-party framework using arithmetic secret sharing and optimized protocols for nonlinear functions, batch normalization, private training, and inference.

  • Results

    Falcon reports substantial performance and communication improvements over prior work for private training and inference, including WAN evaluations where compute dominates latency for large networks.

  • Takeaways & Limitations

    Falcon supports maliciously secure private training and inference for large networks and datasets, including batch normalization and models such as AlexNet and VGG16.

  • Takeaways & Limitations

    Falcon assumes an honest majority and does not protect training data from model inversion, membership inference, attribute inference, or model poisoning attacks.

Abstract

from arXiv · show

We propose Falcon, an end-to-end 3-party protocol for efficient private training and inference of large machine learning models. Falcon presents four main advantages - (i) It is highly expressive with support for high capacity networks such as VGG16 (ii) it supports batch normalization which is important for training complex networks such as AlexNet (iii) Falcon guarantees security with abort against malicious adversaries, assuming an honest majority (iv) Lastly, Falcon presents new theoretical insights for protocol design that make it highly efficient and allow it to outperform existing secure deep learning solutions. Compared to prior art for private inference, we are about 8x faster than SecureNN (PETS'19) on average and comparable to ABY3 (CCS'18). We are about 16-200x more communication efficient than either of these. For private training, we are about 6x faster than SecureNN, 4.4x faster than ABY3 and about 2-60x more communication efficient. Our experiments in the WAN setting show that over large networks and datasets, compute operations dominate the overall latency of MPC, as opposed to the communication.

1 Introduction

Falcon is an expressive, maliciously secure 3-party framework for private deep learning that supports training and inference on large networks. It combines protocol improvements with broad evaluations, achieving substantial performance and communication gains over prior work.

  • Motivation and framework: Falcon provides secure analytics without revealing individual secret inputs, while supporting private training and inference in an honest-majority 3-party setting.Its malicious variant ensures computation either correctly completes or aborts when malicious activity is detected.
  • Protocol improvements: 2× lower theoretical complexity for the derivative of ReLU and at least 2× lower communication complexity result from simplified fixed-point algebra and smaller ring sizes.Falcon uses exact truncation to enable a smaller datatype and avoids interconversion protocols between arithmetic, Boolean, and garbled circuits.
  • Expressiveness: Falcon supports batch normalization in both forward and backward passes, enabling private training and inference for large networks such as VGG16 and AlexNet.The framework is evaluated on MNIST, CIFAR-10, and Tiny ImageNet in LAN and WAN settings.
  • Evaluation: Six networks spanning roughly 118,000 to 138 million parameters were evaluated across three datasets, covering both private training and inference.The experiments include simple 3-layer MLPs and large networks with about 16 layers.
  • Evaluation: In WAN experiments over large networks and datasets, compute operations dominate Falcon’s overall MPC latency rather than communication rounds.This supports Falcon’s communication-optimized 3-party design.

2 Falcon Overview

Falcon is a 3-party private machine-learning service for training and inference over secret-shared data, designed for honest-majority security, efficient protocols, and broad neural-network support. Its contributions combine malicious security with abort, lower communication and computation overhead, batch normalization, and extensive evaluation across networks, datasets, and deployment settings.

  • Application Setting: Three computing servers train and serve models on replicated secret shares, keeping data holders’ inputs private from each server.The service has separate training and inference phases, with query users receiving answers from the jointly held model.
  • Threat Model: Falcon assumes an honest majority and tolerates one semi-honest or malicious party, either completing correctly or aborting when malicious activity is detected.Denial-of-service attacks are not prevented; Falcon aborts when parties refuse to cooperate.
  • Technical Contributions: Falcon reduces nonlinear-operation overhead by over 2× for DReLU and cuts communication complexity by at least 2× through smaller rings and datatypes.The smaller-ring approach uses an exact but expensive truncation protocol, while communication also benefits from superlinear dependence on ring size.
  • Technical Contributions: Falcon supports batch normalization in both forward and backward passes, addressing a functionality considered important for stable and efficient private neural-network training.Batch normalization can speed training, prevent extreme activations, reduce over-fitting, and improve training stability.
  • Broader Applicability: Falcon’s protocol techniques may also support private decision trees, searching, thresholding, and sorting because ReLU is essentially a comparison function.This applicability follows from the comparison structure underlying ReLU.

3 Protocol Constructions

Falcon builds private machine-learning protocols from arithmetic secret sharing, supporting core neural-network operations while addressing malicious security and efficiency. Its constructions include private comparison, ReLU, division, batch normalization, and optimized matrix and convolution operations.

  • 3.1 Notation: Falcon uses 2-out-of-3 replicated secret sharing over multiple moduli, including L = 2^ℓ, a small prime p, and 2.The implementation uses ℓ = 25, p = 37, and fixed-point encoding with 13 bits of precision.
  • 3.2 Basic Operations: Linear operations are local, while multiplication combines shares and then reshapes 3-out-of-3 sharing into 2-out-of-3 sharing.Convolutions are expanded into matrix multiplications, and fixed-point multiplication is followed by truncation.
  • 3.3 Private Compare: Private Compare outputs shares of whether a secret-shared value x is at least public threshold r, using blinding to protect the comparison result.The protocol computes bitwise intermediate values, multiplies them with a mask, and returns a blinded comparison bit.
  • 3.4–3.5 Wrap Function and ReLU: Falcon connects the wrap3 function on shares with the most significant bit, enabling derivative-of-ReLU computation through local operations and ReLU through selective sharing.The construction avoids interconversion protocols and uses arithmetic secret sharing for nonlinear operations.
  • 3.7 Division and Batch Normalization: Division uses numerical methods after securely estimating a secret’s bounding power of 2, revealing only that bound and nothing else.Batch normalization uses Newton’s method; four rounds suffice for a close approximation at the stated fixed-point precision.

4 Theoretical Analysis

Falcon’s theoretical analysis models its protocols in the real-world/ideal-world simulation paradigm and proves security with abort against one malicious party. The analysis also establishes perfect stand-alone security, straight-line black-box simulation, and concrete protocol overheads.

  • 4.1 Security Framework: Falcon proves security by constructing simulators that make real protocol transcripts indistinguishable from ideal interactions, using hybrid arguments and input extraction.The framework models protocols in the real-world/ideal-world simulation paradigm.
  • 4.1 Security Framework: Falcon achieves perfect stand-alone security with straight-line black-box simulators and extends security to concurrent general composition using a prior result.Perfect security means the joint distributions of inputs, outputs, and transcripts are exactly equal rather than merely statistically close.
  • 4.2 Security Theorems: Falcon’s protocols for private comparison, weighted averaging, ReLU, maxpool, exponentiation, division, and batch normalization securely realize their functionalities with abort against one malicious party.The stated theorems use hybrid models involving multiplication, reconstruction, preprocessing, private comparison, weighted averaging, ReLU, and exponentiation functionalities.
  • 4.3 Complexity Analysis: Private comparison’s dominant round cost comes from string multiplication, while wrap3 adds one round and derivative-of-ReLU is local over wrap3.ReLU adds two rounds and one ring element, or two in malicious security, according to the overhead analysis.

5 Experimental Evaluation

Falcon is evaluated end-to-end for private training and inference across six networks, three datasets, LAN and WAN settings, and semi-honest and malicious adversaries. The experiments show strong efficiency, support for batch normalization and fixed-point computation, and a shift toward computation dominating latency on larger networks.

  • Private Inference: 3×-120× faster inference than existing work is achieved depending on network architecture, including 8× faster than SecureNN and comparable performance to ABY3.Falcon is also reported as up to 18× faster than XONN and 32× faster than Gazelle.
  • Private Inference: 40× more communication efficiency than ABY3 and 200× more than SecureNN are reported for inference, with over 760× efficiency over XONN.The paper cautions that comparisons depend on protocol scaling and network architecture.
  • Private Training: 6× faster training than SecureNN, 4.4× faster than ABY3 and 70× faster than SecureML are reported, while communication overhead is 10× to 100× lower than other solutions.The authors attribute these speedups to improved protocol round complexity and communication.
  • Compute vs. Communication: Computation becomes the dominant performance cost for larger networks because matrix multiplication has super-quadratic complexity, despite linear communication for its protocol.The paper therefore motivates faster computation through GPUs, parallelization, efficient matrix multiplication and caching.
  • Batch Normalization: Batch normalization helps train AlexNet faster and achieve better networks, while its high round complexity creates a significant overhead gap mainly in WAN settings.Falcon supports batch normalization in both forward and backward passes.
  • Accuracy and Representation: Less than 1% final-layer output differences are observed between Falcon’s 32-bit fixed-point computation and 64-bit PyTorch computation, with most networks showing no or low accuracy loss.This evaluates the accuracy impact of approximations and the smaller datatype.

6 Related Work

Related work applies secret sharing, garbled circuits, homomorphic encryption and GMW to private machine learning, often combining techniques for different operations. Prior frameworks vary in security model, party count, network scope and supported tasks, while Falcon is evaluated against these alternatives using comparable benchmarks.

  • Privacy-Preserving Training: ABY3 generalizes and optimizes switching between computation representations for private machine learning, following earlier two-party protocols for learning algorithms.The supplied related-work passage introduces ABY3 in the context of privacy-preserving training.
  • Privacy-Preserving Inference: Private inference frameworks use secret sharing, garbled circuits, homomorphic encryption or GMW, each with different advantages and disadvantages.The related work includes CryptoNets, DeepSecure, Chameleon, Gazelle, EzPC and XONN.
  • Mixed Protocols: Chameleon combines GMW for low-depth nonlinear functions, garbled circuits for high-depth functions and secret sharing for linear operations.This mixed-protocol design is presented as a route to high performance gains.
  • Security Models: FLASH provides malicious security with guaranteed output delivery in a 4PC model, while QuantizedNN covers combinations of security and corruption assumptions.The comparison also includes Astra, BLAZE and Trident.

7 Conclusion

Falcon provides private training and inference in an honest-majority 3-party setting, combining malicious security, improved complexity, large-network support, and efficient batch normalization.

  • Falcon supports private training and inference in an honest-majority 3-party setting.
  • Falcon provides security against malicious adversaries while allowing the computation to abort when malicious activity is detected.
  • Falcon improves round and communication complexity through novel protocols.
  • Falcon evaluates secure deep learning on large-scale networks including AlexNet and VGG16 and datasets including Tiny ImageNet.
  • Falcon demonstrates efficient protocols for batch normalization, a critical component of present-day machine learning.

A Recent Related Work

The related-work comparison examines theoretical protocol complexity and end-to-end ReLU overhead across several privacy-preserving frameworks, while noting limited evaluation of training approaches.

  • The comparison covers Astra, BLAZE, FLASH, Trident, and Falcon using theoretical complexities and end-to-end ReLU overhead.
  • Astra and BLAZE are 3PC protocols, whereas FLASH and Trident are 4PC protocols.
  • Most compared works evaluate their approaches only over DNNs.
  • None of the compared frameworks evaluate their approaches for training of neural networks.

B Theoretical Complexity

Falcon’s theoretical overhead analysis identifies the dominant costs of private comparison and derives the round and communication requirements of compound nonlinear protocols.

  • String multiplication in Step 6 dominates the round complexity of private comparison.
  • wrap3 adds one round and one ring element over private comparison, with two elements under malicious security.
  • Computing the derivative of ReLU is a local computation over wrap3.
  • ReLU requires two additional rounds and one ring element, or two under malicious security.
  • Maxpool and its derivative require rounds proportional to the vectorized operation.

C Datasets

The experiments use image-classification datasets spanning handwritten digits, natural images, and 200-class imagery, with theoretical-overhead tables defining protocol parameters and security costs.

  • The evaluation selects MNIST, CIFAR-10, and Tiny ImageNet for image-classification training experiments.
  • MNIST contains 60,000 training images and 10,000 test images of 28 × 28 handwritten digits labeled 0 through 9.
  • Table 8 compares theoretical multiplication and ReLU complexities using datatype bit-size ℓ and security parameter κ, while Table 9 reports overheads using ring-size logarithm ℓ, element size k, and vector size n.
  • Malicious protocols have higher communication complexity than semi-honest protocols, resulting in poorer concrete efficiency when implemented.
  • CIFAR-10 contains 50,000 training and 10,000 test 32 × 32 color images across 10 classes, supporting AlexNet and VGG16 experiments.
  • Tiny ImageNet contains 100,000 training samples and 10,000 test samples across 200 classes.

D Networks

Falcon is evaluated on networks spanning varied architectures and model sizes, including high-capacity AlexNet and VGG16. Its reported accuracies are similar to those obtained by prior private-learning systems.

  • Reported accuracy: 97.42% accuracy is achieved on Network-A, 97.81% on Network-B, 98.64% on Network-C, and 99.15% on LeNet.These accuracies are reported as similar to SecureNN, SecureML, and ABY3.
  • Network architectures: Network-A is a 3-layer fully connected ReLU network with around 118K parameters.It is the smallest evaluated network and was also evaluated in SecureML.
  • Network architectures: Network-B combines one convolutional layer with two fully connected layers and ReLU activations, using approximately 100K parameters.Its architecture was selected from Chameleon.
  • Network architectures: Network-C has two convolutional and two fully connected layers, with Max Pooling and ReLU, and around 10,500 parameters.The network was selected from prior work on MiniONN.
  • High-capacity networks: AlexNet has five convolutional and three fully connected layers, uses batch normalization, and has about 60 million parameters.Falcon evaluates AlexNet because its system supports batch normalization.
  • High-capacity networks: VGG16 has 16 layers and approximately 138 million parameters.It was the runner-up of the ILSVRC-2014 competition.

E Functionality Descriptions

The section presents ideal functionalities and preprocessing procedures for Falcon’s protocol components. It covers common randomness, truncation, private comparison, wraparound, ReLU, Maxpool, division, power, and batch normalization.

  • Functionality descriptions: Figures define ideal functionalities for private comparison, weighted averaging, ReLU, Maxpool, power, division, and batch normalization.These are labeled ΠPC, ΠWA, ΠReLU, ΠMaxpool, ΠPow, ΠDiv, and ΠBN.
  • Common randomness: Common randomness can be generated from party-specific random seeds using a seeded PNRG and counter updates.The construction supports both 3-out-of-3 and 2-out-of-3 common randomness.
  • Preprocessing material: Truncation pairs are generated as ⟨r⟩, ⟨r′⟩ = ⟨r/2^d⟩ using protocol Πtrunc2.The cited procedure runs protocol Πtrunc2 from prior work.
  • Private comparison: Private Compare preprocessing samples random values, opens their powers, removes zero openings, and queues one openings.The computation takes ⌈log2 p⌉ rounds and can be amortized by choosing a large k.
  • Correlated randomness: Maxpool and Division require no additional correlated randomness beyond the randomness used by their subroutines.ReLU and Wrap3 instead have dedicated correlated-randomness generation procedures involving random bits and bit composition.
  • Preprocessing: Preprocessing material required by the online protocol is generated in a setup step performed only once.The preprocessing material is used by the online protocol.
Loading 2004.02229v2…