PaperlayerView exact revision on arXiv

Source-linked AI summary

VFL: A Verifiable Federated Learning with Privacy-Preserving for Big Data in Industrial IoT

Anmin Fu, Xianglong Zhang, Naixue Xiong, Yansong Gao, Huaqun Wang

arXiv:2007.13585v2cs.CR

TL;DR

Industrial IoT requires collaborative deep learning without exposing sensitive training data, yet federated gradients can leak information and aggregation servers can forge results. VFL combines Lagrange-interpolation verification with blinding-based gradient protection, reporting high accuracy and efficiency while keeping verification overhead independent of participant count.

  • Problem

    Industrial IoT training needs data collaboration under privacy constraints, but shared federated gradients can leak sensitive training-set information and aggregation servers may return forged results.

  • Method

    VFL combines Lagrange interpolation for verifying aggregated gradients with blinding technology for privacy-preserving gradient aggregation.

  • Results

    VFL achieves high accuracy and efficiency, with verification overhead that does not increase with participant count.

  • Takeaways & Limitations

    VFL provides verifiable and privacy-preserving federated learning for industrial intelligent applications under the stated collusion condition.

Abstract

from arXiv · show

Due to the strong analytical ability of big data, deep learning has been widely applied to train the collected data in industrial IoT. However, for privacy issues, traditional data-gathering centralized learning is not applicable to industrial scenarios sensitive to training sets. Recently, federated learning has received widespread attention, since it trains a model by only relying on gradient aggregation without accessing training sets. But existing researches reveal that the shared gradient still retains the sensitive information of the training set. Even worse, a malicious aggregation server may return forged aggregated gradients. In this paper, we propose the VFL, verifiable federated learning with privacy-preserving for big data in industrial IoT. Specifically, we use Lagrange interpolation to elaborately set interpolation points for verifying the correctness of the aggregated gradients. Compared with existing schemes, the verification overhead of VFL remains constant regardless of the number of participants. Moreover, we employ the blinding technology to protect the privacy of the gradients submitted by the participants. If no more than n-2 of n participants collude with the aggregation server, VFL could guarantee the encrypted gradients of other participants not being inverted. Experimental evaluations corroborate the practical performance of the presented VFL framework with high accuracy and efficiency.

I. INTRODUCTION

Industrial IoT needs collaborative deep learning that uses distributed data without exposing sensitive training sets. VFL combines privacy-preserving gradient aggregation with verification against forged results.

  • Motivation: Industrial IoT data contain sensitive information, while privacy laws and data-island constraints hinder sharing and collaborative model training.Examples include account information, case history, and medical diagnoses; isolated local datasets may produce inaccurate or poorly generalizing models.
  • Motivation: Secure centralized learning is difficult to apply at industrial scale because encrypted or noise-added training introduces computational cost or accuracy trade-offs.The paper states that nonlinear neural-network computation is difficult on encrypted datasets and that noise-added training usually reduces accuracy.
  • Motivation: Federated learning shares gradients rather than localized data, but plaintext gradients can still leak training information and shared-key encryption remains vulnerable to collusion.Prior work recovered local data from gradients, while a colluding aggregation server and participant could obtain the common secret key in homomorphic-encryption schemes.
  • VFL Contributions: VFL uses Lagrange interpolation to let participants independently verify aggregated results with computational overhead that does not increase with participant count.The mechanism is also described as requiring less overhead for each participant.
  • VFL Contributions: VFL combines Lagrange interpolation with blinding to secure gradient aggregation and protect the trained model and private gradients under a bounded collusion condition.The stated guarantee applies when no more than n-2 of n participants collude with the aggregation server.
  • Evaluation: MNIST experiments with an MLP report high accuracy and efficiency, with acceptable verification overhead for participants.The introduction identifies security analysis and experimental evaluation as part of the framework assessment.

B. Secure Distributed Training

Secure distributed training assigns model training to participants and aggregates their gradients, reducing direct data sharing but leaving privacy and verifiability challenges. The section introduces federated learning and the need for efficient verification.

  • Secure Distributed Training: Secure distributed training assigns training work to participants, with federated learning requiring synchronous gradient uploads for aggregation.This contrasts with collaborative deep learning, where participants share gradients and update the model asynchronously.
  • Secure Distributed Training: Existing distributed-training research shows that shared gradients can still cause privacy breaches.The cited concern follows federated learning’s gradient-sharing design.
  • Prior Schemes: Prior secure-distributed schemes use additive homomorphic encryption or one-time masks to protect gradients, while other work deploys federated learning for IoT.These approaches may involve ciphertext training and multiple interactions between trainers and server-aid components.
  • Open Challenge: Existing verifiable federated-learning schemes can impose high verification cost or support limited model types, leaving efficient verification for nonlinear models unresolved.One cited approach requires all participants to join verification, increasing cost as participant numbers grow.
  • Federated Learning: In federated learning, participants share gradients with an aggregation server, which returns the aggregate for model-parameter updates.The framework assumes n participants using an agreed model architecture and repeating rounds until termination conditions are met.
  • Federated Learning: The neural-network model is represented as f(x, M), where x is the input and M contains model parameters such as biases and neuron connections.Participants hold local training sets and compute gradients using stochastic gradient descent on random subsets.

B. Lagrange Interpolation

Lagrange interpolation uniquely reconstructs an n-degree polynomial from n+1 distinct interpolation points, enabling function recovery from selected evaluations.

  • n+1 distinct interpolation points with corresponding values determine a unique n-degree polynomial.
  • Lagrange basis polynomials provide the interpolation representation of the recovered polynomial.
  • Any n+1 points of an n-degree polynomial are sufficient to recover its function expression.

IV. OUR PROPOSED VFL SCHEME

VFL coordinates initialization, local encrypted training, server aggregation, and participant verification through PKG, participants, and an aggregation server.

  • The VFL architecture contains a Public Key Generator, participants, and an aggregation server.
  • Initialization Phase: The PKG initializes the model and distributes keys, parameters, and related values to participants.
  • Model Training Phase: Participants train locally, encrypt gradients, upload ciphertexts, verify returned aggregates, and update their models.
  • Aggregation Phase: The aggregation server combines uploaded ciphertexts and distributes the aggregated result to all participants.
  • Initialization Phase: VFL uses Lagrange interpolation and blinding to support secure aggregation and verification while keeping most parameters secret from the server.

C. Model Training Phase

During model training, participants compute local gradients, blind and interpolate them, then package the resulting evaluations into fixed-size ciphertexts for aggregation.

  • Each participant trains on a local dataset subset and computes a private gradient using back propagation and SGD.
  • The PRG generates parameter sequences used to blind each participant’s gradient before encryption.
  • Participants split the blinded gradient, apply Lagrange interpolation using points a_i and A_i, and evaluate the polynomial at points b_i.
  • CRT packages the m polynomial evaluations into a gradient ciphertext whose size matches the original gradient.

D. Aggregation Phase

The aggregation phase sums encrypted gradients, while participants unpack and interpolate the result, verify it against A, and decrypt only if it is correct.

  • The aggregation server sums the participants’ uploaded ciphertexts into an aggregate ciphertext.
  • Participants unpack the aggregate ciphertext and use Lagrange interpolation on evaluations at b_i to obtain F(x).
  • Participants verify correctness by checking whether F(a_m) equals A before decrypting and updating the model.
  • The server cannot determine F(x) because the interpolation points b_i are hidden, despite observing function values at unknown points.

E. Update Phase

Participants verify the received aggregation value through Lagrange interpolation before decrypting it and updating their local model. Correctness is checked by evaluating the interpolated function at a designated point and comparing the result with A.

  • Participants unpack the aggregation ciphertext, interpolate F(x) from the pairs {(b_i, F(b_i))}, and calculate the resulting function.The interpolation uses the received values at the secret points b_i.
  • If F(a_m) = A, the aggregation value is accepted; otherwise, participants classify it as forged and terminate federated learning.Because F(x) contains one variable function per model parameter, the procedure verifies multiple equations.
  • After a correct verification, participants input the constant sequence {a_i | i = 1, 2, ..., m − 1} into F(x) and sum the outputs to obtain w.The resulting w is the aggregated value of participants’ original gradients.
  • Each participant updates the local model parameter using M = M − η·w, then proceeds to the next federated-learning round until termination.

V. EFFECTIVENESS ANALYSIS FOR VFL

The effectiveness analysis establishes correctness under honest protocol execution. In that setting, participants obtain the correct aggregated gradients and can use them to update the model.

  • Under honest execution by every entity, VFL participants obtain correct aggregated gradients for model updates.
  • The correctness proof reduces valid gradient recovery to showing that equation (6) holds.
  • The preceding equations establish equation (6), confirming that honest participants obtain correct aggregated gradients for local model updates.

B. Data Privacy

VFL targets leakage of private gradients and model parameters to the aggregation server. Its security analysis states protection against server collusion with up to n−2 participants for the remaining participants’ gradients.

  • The scheme aims to protect both participant gradients and the model parameter because gradients retain sensitive training-set information and the final model parameter is private.
  • Theorem 2 states that each participant’s private gradient w_i and model parameter M are not leaked to the aggregation server.
  • The server cannot calculate the relevant interpolation functions because the constant sequence {b_i | i = 1, 2, ..., m} is kept secret.
  • In the finite field F_q, the analysis links security of w_i and w to the server’s probability of obtaining the two constant sequences, with q described as very large, such as 2^64.
  • If the aggregation server colludes with k ≤ n−2 participants, the private gradients of the other participants will not be leaked.

C. Verifiability

VFL enables participants to verify aggregated results independently, addressing the risk that an aggregation server may reduce or forge aggregation. The verification relies on confidential interpolation parameters and is evaluated alongside accuracy and overhead.

  • The framework gives participants a mechanism to verify aggregated-result correctness, countering server reduction or forgery of aggregation operations.
  • Theorem 4 states that each participant can verify the result independently and detect forged results with overwhelming probability.
  • Participants locally check whether equation (11) holds after receiving the correct aggregated result, enabling independent verification.
  • A forged result perturbs the interpolation values by Δx_i, with nonzero total squared perturbation distinguishing it from the valid result.
  • Because l_i(a_m) depends on confidential interpolation points and a_m, the server cannot feasibly forge a result satisfying the verification equation.
  • The experiments evaluate model accuracy, computational overhead, and communication overhead on a representative dataset.

A. Experimental Setup

The experiments evaluate VFL on MNIST using a 784-512-1024-256-10 MLP with 20 participants, comparing accuracy and computational overhead against existing schemes. VFL achieves similar accuracy while maintaining lower encryption, decryption, and verification costs in the reported settings.

  • Experimental Setup: MNIST experiments use a 784-512-1024-256-10 MLP with 1,192,202 gradient parameters, 64-bit precision, and 20 participants.The learning rate is 10^-2 with decay 10^-5.
  • Model Accuracy: After 400 rounds, VFL reaches about 94% accuracy versus approximately 95% for federated learning scheme [7].The authors report that privacy protection hardly sacrifices model accuracy.
  • Encryption Overhead: For 1,192,202 gradients, VFL encryption overhead is 1.383s at m=4 and 4.916s at m=8, compared with 8.178s for scheme and 19.741s for scheme.Larger m requires more interpolation data and increases overhead.
  • Decryption Overhead: VFL decryption overhead is 0.961s at m=4 and 4.379s at m=8, compared with 8.007s for scheme and 10.577s for scheme.VFL decryption uses m−1 interpolation points and the addition of m equations.
  • Verification Overhead: VFL verification overhead remains constant as participant count increases, reaching 0.325s at m=4 and 0.623s at m=8 for 20 participants.Scheme reaches 14.624s at 20 participants, while scheme does not support verification.

D. Communication Overhead

The paper discusses communication and total per-round overhead, then outlines industrial applications for VFL. It reports lower total overhead for VFL than the compared schemes and identifies enterprise risk assessment, anti-money laundering, and medical systems as application areas.

  • D. Communication Overhead: Communication overhead depends on uploaded gradient-ciphertext and downloaded aggregated-ciphertext sizes, with communication time increasing linearly with communication amount.Table I reports communication amount and time for each scheme per training round.
  • E. Total Overhead: For 20 participants, VFL total overhead is 3.053s at m=4 and 10.304s at m=8, compared with 16.713s for scheme and 45.528s for scheme.The total comprises participant, server, and communication overheads.
  • Industrial Applications: VFL is described as applicable to industrial scenarios involving secure training on multi-source data.The paper states that the framework can be applied to many industrial scenarios.
  • Enterprise Risk Assessment: In enterprise risk assessment, banks can use invoice amounts and credit data held across institutions while modeling enterprise risk labels.The described variables include quarterly invoice amounts, credit score, and a binary risk label.
  • Anti-Money Laundering: For anti-money laundering, VFL can combine samples from several banks without disclosing those samples.Inputs may include large-sum transactions and fund sources inconsistent with business scope.
  • Conclusion and Future Work: The conclusion reports that MNIST experiments demonstrate advantages for VFL in verification and total overhead, while future work targets more complex networks and richer labels.The conclusion also states that blinding protects trained models and private participant gradients.
Loading 2007.13585v2…