PaperlayerView exact revision on arXiv

Source-linked AI summary

POSEIDON: Privacy-Preserving Federated Neural Network Learning

Sinem Sav, Apostolos Pyrgelis, Juan R. Troncoso-Pastoriza, David Froelicher, Jean-Philippe Bossuat, Joao Sa Sousa, Jean-Pierre Hubaux

arXiv:2009.00349v3cs.CRcs.LG

TL;DR

POSEIDON addresses privacy-preserving neural-network training and evaluation across N federated parties without exposing their data or model state. It combines multiparty lattice-based homomorphic encryption with packing, distributed bootstrapping, and parameter optimization, achieving accuracy comparable to non-private approaches and training a 3-layer MNIST network across 10 parties in less than 2 hours.

  • Problem

    Collaborative neural-network learning needs diverse data, but sensitive data and models cannot always be shared or outsourced securely across parties.

  • Method

    POSEIDON uses multiparty lattice-based homomorphic encryption, encrypted SIMD packing, distributed bootstrapping, and optimized cryptographic parameters for federated neural-network training and inference.

  • Results

    POSEIDON achieves accuracy on par with centralized or decentralized non-private approaches, with computation and communication overhead scaling linearly with the number of parties.

  • Takeaways & Limitations

    POSEIDON demonstrates practical secure federated neural-network learning while protecting training data, model information, and evaluation data under collusions of up to N−1 parties.

Abstract

from arXiv · show

In this paper, we address the problem of privacy-preserving training and evaluation of neural networks in an $N$-party, federated learning setting. We propose a novel system, POSEIDON, the first of its kind in the regime of privacy-preserving neural network training. It employs multiparty lattice-based cryptography to preserve the confidentiality of the training data, the model, and the evaluation data, under a passive-adversary model and collusions between up to $N-1$ parties. To efficiently execute the secure backpropagation algorithm for training neural networks, we provide a generic packing approach that enables Single Instruction, Multiple Data (SIMD) operations on encrypted data. We also introduce arbitrary linear transformations within the cryptographic bootstrapping operation, optimizing the costly cryptographic computations over the parties, and we define a constrained optimization problem for choosing the cryptographic parameters. Our experimental results show that POSEIDON achieves accuracy similar to centralized or decentralized non-private approaches and that its computation and communication overhead scales linearly with the number of parties. POSEIDON trains a 3-layer neural network on the MNIST dataset with 784 features and 60K samples distributed among 10 parties in less than 2 hours.

I. INTRODUCTION

POSEIDON addresses the difficulty of collaboratively training neural networks without sharing sensitive data or relying on narrowly scoped MPC settings. It provides federated, quantum-resistant privacy protection for training and inference while retaining competitive accuracy and scalability.

  • Motivation: Sensitive and heterogeneous data motivates collaborative neural-network training, but privacy regulations and confidentiality concerns often prevent direct sharing.Existing trusted-party and MPC approaches introduce costly outsourcing, single points of failure, limited party counts, honest-majority assumptions, or off-premises data sharing.
  • System: POSEIDON protects training data, intermediate updates, final model weights, and evaluation data during federated neural-network training and inference.It uses multiparty homomorphic encryption while supporting fully connected, convolutional, and pooling layers.
  • Methods: POSEIDON uses alternating packing and a generic encrypted-neural-network protocol to enable SIMD operations during secure backpropagation.The system also introduces arbitrary linear transformations in distributed bootstrapping and optimizes cryptographic parameters for training and evaluation.
  • Security and scalability: POSEIDON provides federated flexibility, protection against collusions of up to N−1 parties, and communication linear in the number of parties.The paper contrasts these properties with MPC systems that typically use two to four servers, assume an honest majority, and scale quadratically in communication.
  • Evaluation: POSEIDON reports negligible accuracy impact relative to non-private approaches while avoiding the utility degradation associated with differential privacy-based federated learning.The evaluation covers real-world datasets and multiple network architectures.

III. PRELIMINARIES

This section introduces neural-network computation and its distributed training formulation. Neural networks use forward and backpropagation through layered linear and nonlinear transformations, while parties compute and aggregate local gradients with parallel SGD.

  • Neural networks: Neural networks extract complex nonlinear relationships through layered forward and backpropagation computations.The layers perform linear transformations followed by activation functions.
  • MLPs: MLPs are fully connected networks with input, hidden, and output layers, where each neuron connects to all neurons in the next layer.The weights between layers are represented by matrices.
  • Backpropagation: Backpropagation updates layer weights by minimizing a cost function over random mini-batches using gradient descent.The formulation uses batch size B and can employ mean squared error as the cost function.
  • CNNs: CNNs use convolutional, pooling, and fully connected layers, with convolution operations represented as matrix multiplications in the protocols.Pooling layers perform downsampling.
  • Distributed training: Distributed training uses a MapReduce-style parallel SGD procedure in which parties compute partial gradients locally and a reducer averages them for model updates.Each party performs b local iterations before gradient aggregation.

C. Multiparty Homomorphic Encryption (MHE)

POSEIDON relies on CKKS-based multiparty homomorphic encryption to compute on packed encrypted data while distributing decryption authority across the parties. Its operations manage ciphertext arithmetic, levels, scales, rotations, key switching, and bootstrapping.

  • MHE scheme: POSEIDON uses CKKS multiparty homomorphic encryption, whose collective secret key is distributed so decryption requires all parties.The scheme supports floating-point arithmetic and is based on the RLWE problem.
  • Encoding: Packed plaintexts encode vectors of values in one ciphertext, allowing encrypted values to be processed in parallel.The encoding, ciphertext levels, and scales determine precision and the available homomorphic computation depth.
  • Operations: Homomorphic addition, subtraction, plaintext and ciphertext multiplication, rotations, rescaling, and scale setting provide the core encrypted arithmetic operations.Rotations move ciphertext slots, while rescaling reduces the level after multiplication.
  • Distributed functionality: Distributed key generation, decryption, key switching, and bootstrapping require coordinated participation by the secret-key holders.Bootstrapping restores a ciphertext to its initial level and scale so deeper circuits can continue.

L. This is done by the bootstrapping functionality (DBootstrap(·)). Encode(·) enables us to pack several values into one ciphertext and operate on them in parallel.

POSEIDON combines distributed key-switching and bootstrapping with a federated system model in which parties retain encrypted local data and model state. Its protocols prepare encrypted inputs, propagate model weights, compute local gradients, and aggregate updates.

  • Cryptographic functionality: Local key switching supports noninteractive slot rotations and relinearization, whereas collective key switching supports result decryption or changing a ciphertext’s encryption key.The two functions differ in whether they require interaction among all parties.
  • System model: POSEIDON’s system model has N parties that retain local datasets and labels, collectively train a neural network, and support queries on evaluation data.The querier may be a participating party or an external entity.
  • Objectives: The protocol aims to protect local data, intermediate model updates, and final model weights during training and to protect the querier’s data and prediction results during inference.The confidentiality objectives cover both data and model information.
  • Collective training: Collective training begins with shared cryptographic parameters and encoded local data, then repeatedly maps model weights, computes local gradients, combines them, and reduces the aggregate.The protocol is organized into PREPARE, MAP, COMBINE, and REDUCE phases.

C. Overview of POSEIDON

POSEIDON keeps model weights encrypted while parties perform federated neural-network training and inference through privacy-preserving distributed protocols. Its workflow uses collective cryptography, encrypted local computation, and synchronized model updates.

  • Protocol overview: Encrypted model weights remain protected by the parties’ collective public key throughout training.The cryptomorphic operations allow parties to compute between local data and encrypted weights.
  • Protocol overview: The protocol uses four phases: PREPARE, MAP, COMBINE, and REDUCE.These phases are expressed through an extended MapReduce abstraction for privacy-preserving machine learning.
  • PREPARE: During PREPARE, parties agree on learning parameters and use Xavier or He initialization to control gradient behavior.Xavier initialization is used for sigmoid or tanh layers, while He initialization is used for ReLU layers.
  • MAP: During MAP, the root party communicates current encrypted weights to other parties for local gradient-descent computation.Each party performs local forward and backward passes over its batch to compute gradients.
  • Local Gradient Descent: Local gradient descent performs forward propagation, computes output errors, and backpropagates derivatives using encrypted operations.The protocol uses stochastic gradient descent with element-wise products and activation derivatives.
  • Prediction: At inference time, the model remains encrypted and the querier encrypts evaluation data with the parties’ collective key.Oblivious inference is equivalent to one encrypted forward pass.

V. CRYPTOGRAPHIC OPERATIONS AND OPTIMIZATIONS

POSEIDON optimizes encrypted neural-network computation through packing, alignment, and specialized bootstrapping operations. Its alternating packing approach reduces rotation costs while supporting layer-to-layer matrix operations.

  • Packing: Packing encodes vectors in ciphertext slots so SIMD operations parallelize encrypted computations.This improves the forward pass and backpropagation performance.
  • Alternating Packing: Existing row-based and diagonal packing strategies require many rotations, which substantially affect encrypted neural-network training time.The limitation applies to matrix-matrix multiplication and transpose operations in local gradient descent.
  • Alternating Packing: Alternating packing combines row-based and column-based packing, using the opposite scheme for each consecutive fully connected layer.It reduces the cost of transforming packing between consecutive layers.
  • Alternating Packing: Alternating packing makes rotations scale logarithmically with the matrix dimension by padding matrices to power-of-two dimensions.The relevant dimensions include the number of features and hidden neurons.
  • Packing protocol: The packing protocol prepares encoded inputs and labels, initializes encrypted weights, and alternates row and column packing across layers.Gaps are added when dimensions require alignment between adjacent layers.
  • Kernel packing: High-depth kernels use alternating row and column packing with replicated decomposed matrices, trading MAP multiplications for reduced communication overhead.The approach introduces k multiplications in MAP, where k is the number of kernels in that layer.

B. Approximated Activation Functions

POSEIDON evaluates nonlinear activation functions under encryption using polynomial approximations and optimized encrypted evaluation. Initialization and collective normalization help keep activation inputs within approximation-friendly ranges.

  • Activation approximations: Sigmoid and Softmax are evaluated with least-squares polynomial approximations under encryption.Optimized polynomial evaluation consumes ⌈log(da+1)⌉ levels for approximation degree da.
  • Activation approximations: ReLU is approximated by fitting the smooth softplus function, SmoothReLU, with least squares.The paper uses derivatives of the approximated functions during training.
  • Range control: Different weight initializations and privacy-preserving collective normalization keep activation inputs in smaller ranges.This supports better approximation with the lowest possible polynomial degree.
  • AP pipeline: The AP representation begins with local data preparation, encrypted weight initialization, and mask generation before forward computation.Inputs and labels are prepared by each party, while P1 initializes the weights.
  • Forward pass: The execution pipeline applies encrypted matrix operations, rotations, masking, bootstrapping, and activation evaluation during the forward pass.The listed steps include computation of U1 and U2 before applying the activation function.

3. U2=L1×W2,· 4. L2=ϕ(U2)

The backpropagation stage computes output error, applies activation derivatives, and reshapes encrypted error values for subsequent gradient calculations. The pipeline explicitly combines derivative evaluation with encrypted tensor operations.

  • Backpropagation: The output error is computed by subtracting the network output from the encoded labels.This is the first backpropagation operation for the displayed 2-layer pipeline.
  • Backpropagation: The output error is multiplied element-wise by the derivative of the activation function.The derivative is denoted by ϕ′(U2), and ⊙ denotes element-wise multiplication.
  • Backpropagation: Encrypted error values are multiplied, reshaped, and rotated to align them for earlier-layer computations.The pipeline then computes the derivative for U1 and applies the corresponding mask and encrypted multiplication.
  • Pipeline representation: Table I presents the execution pipeline for a 2-layer MLP using alternating packing.Orange steps identify operations introduced into DBootstrapALT(·).

C. Cryptographic Building Blocks

POSEIDON’s cryptographic building blocks combine packed encrypted computation with distributed bootstrapping and rotation optimizations for privacy-preserving neural-network learning.

  • Rotations: POSEIDON uses packed ciphertexts to support encrypted vector operations, including inner sums and transpose-related transformations.The design relies on slot rotations for restructuring packed vectors.
  • Distributed bootstrapping: Distributed bootstrapping switches encrypted computation between domains while preserving security under the CKKS-based protocol design.The paper contrasts this approach with a BFV secret-sharing protocol that cannot be directly applied to CKKS.
  • Rotation optimization: A naïve packed matrix-vector multiplication requires √g×k rotations, motivating approaches that reduce rotations through alternative packing and processing choices.For mini-batch gradient descent, samples are processed in parallel rather than operating directly on a packed batch matrix.
  • Bootstrapping optimization: POSEIDON embeds masking and rotations after bootstrapping into distributed bootstrapping to reduce their computational cost.The protocol is designed around linear transformations and encrypted-domain operations.

D. Execution Pipeline

POSEIDON executes neural-network forward and backward passes through packed ciphertext operations, while its cost model captures cryptographic complexity and guides parameter selection.

  • Execution Pipeline: The pipeline encodes and encrypts weights, packs intermediate values, and embeds selected masking or rotation operations within DBootstrapALT(·).The same execution structure extends across multiple layers and represents convolutional layers through the corresponding packed operations.
  • Execution Pipeline: Each forward and backward pass for a layer consumes one RIS(·) and one RR(·) operation, except for the last layer.Convolutional layers require fewer RIS(·) operations than fully connected layers because kernels are typically smaller than neuron counts.
  • Complexity: KS(·) and DBootstrap(·) are 2 orders of magnitude slower than an addition operation, making cryptographic primitives dominant cost components.The complexity analysis reports communication and worst-case computation for POSEIDON’s building blocks.
  • Complexity: Communication complexity depends solely on the number of parties N, total ciphertexts sent per global iteration z, and ciphertext size |c|.The distributed bootstrapping takes one communication round, with communication size scaling with N and ciphertext size.
  • Parameter Selection: The parameter optimization accounts for network structure, multiplicative depth, security level, scale, and ciphertext-level constraints.Bootstrapping is scheduled before the current level approaches the threshold L−τ, where τ depends on security parameters.
  • Parameter Selection: POSEIDON reduces level consumption by rescaling after r consecutive multiplications when the initial scale satisfies Q/S=r.This optimization is integrated into the cost function used for cryptographic parameter selection.

VI. SECURITY ANALYSIS

POSEIDON’s security analysis targets confidentiality during training and prediction against passive adversaries colluding with up to N−1 parties. The evaluation section describes empirical testing across datasets and architectures.

  • Security Guarantees: POSEIDON achieves Data and Model Confidentiality during training against a passive adversary corrupting at most N−1 parties.The claim assumes CKKS parameters providing a post-quantum security level λ.
  • Security Guarantees: The security argument relies on CKKS semantic security and simulation of the adversary’s view in the real/ideal world paradigm.The proof sketch treats exchanged model parameters as encrypted during training.
  • Security Guarantees: Intermediate and final model weights remain encrypted, supporting confidentiality of the trained model and addressing federated-learning extraction attacks.The paper presents this as a consequence of the model being computed from parties’ inputs while encrypted.
  • Security Guarantees: POSEIDON achieves Data and Model Confidentiality during prediction under the same passive-adversary threshold of at most N−1 corrupted parties.The prediction protocol is analyzed as equivalent to a forward pass during training.
  • Experimental Evaluation: The experiments evaluate POSEIDON on real-world datasets and multiple fully connected and convolutional network architectures.The implementation uses Go, Lattigo, Onet, and TLS-secured TCP communication in a decentralized system.
  • Experimental Evaluation: Table III compares POSEIDON’s accuracy and execution times for N=10 parties against several non-private approaches.The cited table caption identifies both the performance measures and the comparison baseline.

D. Neural Network Configuration

POSEIDON is evaluated across neural-network architectures, datasets, accuracy, runtime, communication, and scalability. The experiments compare privacy-preserving collaboration with local and prior approaches while measuring how system costs vary with federation size and dataset dimensions.

  • Experimental setup: POSEIDON evaluates fully connected and convolutional neural networks across multiple real-world and synthetic datasets, measuring accuracy, execution time, communication, and scalability.The evaluation varies the number of parties N, samples n, and features d, with additional microbenchmarks for fully connected, convolutional, and pooling layers.
  • Accuracy: 1−3% higher accuracy is achieved than one-party local training through privacy-preserving collaboration, despite approximation and encryption losses.The comparison uses local datasets containing 1/10 or 1/50 of the overall data and averages the corresponding locally trained models.
  • Execution time: 1.4 hours is the reported MNIST training time when its data is evenly distributed among 10 parties.BCW, ESR, and CREDIT each train in less than 15 minutes under the same 10-party distribution; MNIST takes less than an hour with 20 parties using the same local batch size.
  • Inference: 864,000 predictions per hour are achieved on MNIST with parallel query execution and multithreading.Inference timing includes the forward pass, DKeySwitch operations, and communication among parties while the model remains encrypted.
  • Scalability: Increasing parties has little effect on execution time with fixed per-party samples, while fixed total samples yield lower runtime because parties process samples in parallel.With N ranging from 3 to 24, communication increases as parties are added; runtime scales linearly with total samples at N =10.
  • Security and comparison: POSEIDON protects training data, the model, and evaluation data against passive adversaries colluding with up to N−1 parties while supporting federated neural-network training.The system uses multiparty lattice-based homomorphic encryption and keeps parties’ data local, distinguishing it from outsourced MPC approaches.

APPENDIX A SYMBOLS AND NOTATIONS

The supplied passages describe notation references, security-comparison criteria, and approximation procedures for neural-network operations. They also outline a max-pooling approximation and activation-function choices.

  • Table V collects the paper’s frequently used symbols and notations.
  • The comparison framework distinguishes setup size, adversarial capabilities, collusion limits, and confidentiality across private deep-learning systems.The table defines A/P as active/passive capabilities and GC/SS/HE as cryptographic approaches.
  • SmoothReLU is used in evaluations because it provides an efficient degree da=3 approximation, while higher-degree approximations improve exact ReLU accuracy.The alternatives include square-root-based ReLU approximation and least-squares approximation of softplus.
  • The max-pooling procedure repeatedly sharpens relative values, renormalizes them, and extracts the largest value, but average-pooling is proposed as more efficient and precise.The direct max computation uses polynomial square-root approximation and repeated bootstrapping, whereas average-pooling avoids that cost.

APPENDIX E TECHNICAL DETAILS OF DISTRIBUTED BOOTSTRAPPING WITH ARBITRARY LINEAR TRANSFORMATIONS (DBOOTSTRAPALT(·))

The passages explain how arbitrary linear transformations can be integrated into distributed bootstrapping and how masks support statistical hiding. They also describe experimental timing measurements for POSEIDON.

  • Linear transformations can be applied locally to secret-shared plaintext during bootstrapping because φ(msg+M)=φ(msg)+φ(M).This avoids directly performing expensive encrypted matrix-vector transformations in suitable parts of the protocol.
  • DBootstrapALT(·) modifies distributed bootstrapping by incorporating a public matrix, with security following the passive-adversary proof structure plus a share-indistinguishability lemma.The CKKS and BFV variants differ in how their shares are hidden.
  • For simple transformations, Encode(·) and Decode(·) can be skipped to perform the transformation directly on the encoded plaintext.This is preferable when the transformation requires few rotations or no complex permutation.
  • Masking requires bounding the encoded message magnitude and choosing a ciphertext modulus large enough for additions of N masks.The stated bound is that the ciphertext modulus should exceed (N+1)·||M|| when bootstrapping is called.
  • The experiments report execution times per global iteration and microbenchmarks for neural-network functionalities with N=10 parties.The measurements cover feed-forward, backpropagation, communication, and preprocessing-related operations.

C. Benchmarks on Various Convolutional Neural Network Topologies

POSEIDON reports execution times for CNN topologies using overall per-global-iteration measurements, with pooling derivatives incorporated into bootstrapping. The section also identifies security, learning, and optimization extensions relevant to interpreting and extending these benchmarks.

  • C. Benchmarks on Various Convolutional Neural Network Topologies: Overall execution time combines forward-pass and backpropagation costs because pooling derivatives are incorporated into the bootstrapping function.This measurement convention applies to the extrapolated CNN topology timings.
  • C. Benchmarks on Various Convolutional Neural Network Topologies: Table IX compares per-global-iteration execution times across CNN architectures with batch size B=120 and N=10 parties.Topology encodes padded feature count, convolution and pooling kernel sizes, and the final fully connected layer structure.
  • A. Security Extensions: POSEIDON can incorporate active-adversary protections through verifiable computation, but this would increase computational complexity and remains future work.The current security model is passive-adversary protection with up to N−1 colluding parties.
  • B. Learning Extensions: Uneven data distribution and asynchronous learning reduce accuracy in the reported MNIST analysis, by 6% and between 1 and 4%, respectively.The evaluation itself evenly distributes datasets across parties; non-iid data can also cause parameter divergence.
  • B. Learning Extensions: POSEIDON’s packing protocols target MLPs and CNNs, while other neural-network structures require modifications to computation and packing.For ResNet, skip connections require ciphertext-slot alignment, which DBootstrapALT(·) can provide by rearranging slots.
  • C. Optimization Extensions: CNN convolution can use polynomial multiplication after switching between slots and coefficient domains, but the required domain switching is expensive.Embedding domain switching into distributed bootstrapping is proposed as a future efficiency direction for multiple convolutional layers.
Loading 2009.00349v3…