PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy Preserving Distributed Machine Learning with Federated Learning

M. A. P. Chamikara, P. Bertok, I. Khalil, D. Liu, S. Camtepe

arXiv:2004.12108v2cs.DB

TL;DR

Sensitive data distributed across organizations remain exposed to privacy attacks even when federated learning keeps raw data at their owners. DISTPAB perturbs data locally using centrally generated parameters, and the paper reports strong accuracy, efficiency, scalability, and attack resistance. The study considers horizontal federated learning and notes that theoretical analysis of model convergence remains future work.

  • Problem

    Federated learning keeps raw data on owners’ devices, but model and server vulnerabilities can still expose private data, creating a need for additional privacy measures in distributed learning.

    Source: §2 · p. 7

  • Method

    DISTPAB centrally generates global perturbation parameters and distributes them for local data perturbation before data are shared.

    Source: §1 · p. 4

  • Results

    DISTPAB is reported to provide high accuracy, efficiency, scalability, and attack resistance, with accuracy similar to the centralized algorithm and better than RP and GP.

    Source: §1 · p. 5, §6 · p. 31

  • Takeaways & Limitations

    The reported approach applies privacy-preserving perturbation within distributed machine learning while keeping original data from being communicated before perturbation.

    Source: §1 · p. 4, §7 · p. 32

    The study considers only horizontal federated learning, and theoretical proof of model convergence is beyond its scope.

    Source: §3.9.1 · p. 11, §5.4.1 · p. 28

Abstract

from arXiv · show

Edge computing and distributed machine learning have advanced to a level that can revolutionize a particular organization. Distributed devices such as the Internet of Things (IoT) often produce a large amount of data, eventually resulting in big data that can be vital in uncovering hidden patterns, and other insights in numerous fields such as healthcare, banking, and policing. Data related to areas such as healthcare and banking can contain potentially sensitive data that can become public if they are not appropriately sanitized. Federated learning (FedML) is a recently developed distributed machine learning (DML) approach that tries to preserve privacy by bringing the learning of an ML model to data owners'. However, literature shows different attack methods such as membership inference that exploit the vulnerabilities of ML models as well as the coordinating servers to retrieve private data. Hence, FedML needs additional measures to guarantee data privacy. Furthermore, big data often requires more resources than available in a standard computer. This paper addresses these issues by proposing a distributed perturbation algorithm named as DISTPAB, for privacy preservation of horizontally partitioned data. DISTPAB alleviates computational bottlenecks by distributing the task of privacy preservation utilizing the asymmetry of resources of a distributed environment, which can have resource-constrained devices as well as high-performance computers. Experiments show that DISTPAB provides high accuracy, high efficiency, high scalability, and high attack resistance. Further experiments on privacy-preserving FedML show that DISTPAB is an excellent solution to stop privacy leaks in DML while preserving high data utility.

1. Introduction

Distributed data creates a privacy challenge because sensitive information must be analyzed across complex systems, while federated learning still faces model and server attacks. DISTPAB applies perturbation locally before data are shared and is reported to provide strong accuracy, attack resistance, and efficiency.

  • Healthcare, banking, and policing data can contain sensitive information, making privacy protection important during distributed analysis.Source: §1 · p. 2
  • Homomorphic encryption faces efficiency, data-volume, and distribution challenges in big-data DML, while perturbation offers a lightweight alternative with predetermined error.Source: §1 · p. 3
  • Federated learning keeps raw data on owners’ devices, but trained models and coordinating servers remain vulnerable to privacy attacks.The cited attacks include membership inference, model memorizing, and model inversion.Source: §1 · p. 3
  • DISTPAB generates global perturbation parameters centrally and applies perturbation locally at distributed entities before data are shared.The approach is intended to keep original data from being communicated before perturbation.Source: §1 · p. 4
  • DISTPAB is reported to provide excellent classification accuracy, attack resistance, and efficiency for distributed machine learning under big-data settings.Source: §1 · p. 5

2. Literature Review

Privacy-preserving distributed learning must balance utility, privacy, and computational cost across large, distributed datasets. Prior perturbation and learning approaches face scalability or central-server risks, while PABIDOT is not distributed.

  • Cryptographic approaches can impose high computational and storage costs, and encryption can increase data size; perturbation is presented as a lighter option for distributed big data.Source: §2 · pp. 5–6
  • Perturbation can reduce utility, while several privacy models and perturbation methods face attack vulnerabilities or poor scalability on high-dimensional data.The review also notes that differential privacy can impose high noise levels.Source: §2 · p. 6
  • FedML and SplitML retain a central point of failure, and attacks on their servers can expose training data; encrypted partitioned-data methods also face efficiency and trust concerns.Source: §2 · pp. 6–7
  • Matrix-multiplicative perturbations preserve tuple distances and can support classification and clustering, but prior parameter-generation methods scale poorly; PABIDOT is efficient yet not distributed.Distributed perturbation before data leave local networks is identified as needed for geographically dispersed settings.Source: §2 · pp. 7–8

3. Fundamentals

DISTPAB treats numeric records as vectors and uses multidimensional transformations, normalization, and randomized expansion to perturb data. The fundamentals also describe variance-based privacy quantification and Φ-separation for selecting perturbation parameters.

3. Fundamentals

  • DISTPAB applies multidimensional transformations followed by randomized expansion to perturb the input data.Source: §3 · p. 8

3.1. Data matrix (D)

  • The data matrix contains numeric data, with attributes as columns and data-owner records as rows transformed in an n-dimensional coordinate system.Source: §3.1 · p. 8

3.2. Multidimensional isometric transformations

  • Reflection, translation, and rotation are examples of multidimensional isometric transformations.Source: §3.2 · p. 8

3.3. Homogeneous coordinate form and composite operations

  • Homogeneous coordinates represent transformations as matrix multiplications, allowing sequential operations to be expressed as a composite transformation.A column of ones can convert the data matrix to homogeneous form.Source: §3.3 · pp. 8–9

3.4. z-score normalization of the input dataset, D

  • Z-score normalization gives attributes a standard deviation of 1 and mean of 0 so they receive equal weights during transformations.Source: §3.4 · p. 9

3.5. Generating transformation matrices for perturbation

  • The perturbation matrices use uniform random translation coefficients, an axis-reflection matrix, and rotation generated through concatenated subplane rotation.Source: §3.5 · p. 9

3.6. Randomized expansion

  • Randomized expansion adds noise to improve the randomness of perturbed data while improving its positiveness or negativeness for utility.Source: §3.6 · p. 9

3.7. Quantification of privacy; 3.8. Φ −separation

  • Perturbation variance is used to quantify privacy, with higher variance associated with greater difficulty and privacy; Φ-separation selects perturbation parameters for an optimal empirical privacy guarantee.Source: §3.7 · p. 10, §3.8 · p. 10

to generate the perturbed instances of D as Dp

The fundamentals describe generating perturbed datasets by maximizing minimum privacy guarantees, then outline federated learning and the horizontal data-partitioning setting considered in this work.

to generate the perturbed instances of D as Dp

3.9. Federated Learning

  • In federated learning, owners train models locally and send model parameters to a server, which combines them into a global representation.The federated model’s accuracy should be close to that of a centrally trained model.Source: §3.9 · pp. 10–11

3.9.1. Horizontal and vertical federated learning

  • Horizontal federated learning shares features but partitions samples across clients; this work considers only the horizontal setting.Vertical federated learning instead shares sample IDs while clients have different feature spaces.Source: §3.9.1 · p. 11

4. Our Approach: DISTPAB

DISTPAB moves data perturbation to distributed branches while the central entity generates global perturbation parameters, aiming to keep original data local and preserve utility. The paper describes its centralized perturbation procedure, its distributed parameter exchange, and federated learning over the perturbed data.

4. Our Approach: DISTPAB

  • DISTPAB shifts data maintenance and perturbation to distributed entities, leaving global perturbation parameter generation to the central entity.The paper presents this as a way to avoid sharing original data with an untrusted third party while maintaining high utility.Source: §4 · p. 12

4.1. Centralized perturbation paradigm

  • Centralized DISTPAB selects Φ as the largest global minimum privacy guarantee, then applies reflection, noise translation, rotation, randomized expansion, and tuple shuffling.The covariance-based calculation simplifies parameter selection by avoiding repeated searches through the full dataset.Source: §4.1 · pp. 13–14

4.2. Distributing the perturbation

  • Branches send local means and covariance matrices for global parameter generation, then receive the resulting parameters for local perturbation.The central entity merges partition covariance information using an adapted pairwise covariance update formula.Source: §4.2 · pp. 16–17

4.3. Workload of a distributed entity

  • Each distributed entity sends local dataset parameters, receives global parameters, and perturbs its own data without transmitting the data to the central entity.Randomized expansion is computed locally for individual instances and increases randomization over the centralized approach.Source: §4.3 · p. 18

4.4. Workload of the central node

  • The central entity combines partition statistics and calculates global perturbation parameters, assuming distributed nodes were validated through a prior handshake.The paper states that parameter communication does not leak privacy because original data are not shared before perturbation.Source: §4.4 · p. 19

4.5. Federated learning over perturbed data

  • After perturbation, entities train local models on perturbed data and send model parameters for server aggregation and return in each federation round.The required number of local epochs and federation rounds depends on organizational requirements, model architecture, and input-dataset properties.Source: §4.5 · p. 21

5. Results

The experiments evaluate DISTPAB across datasets and classifiers, then analyze its computational and communication costs. With a fixed number of attributes, DISTPAB and centralized PABIDOT both have linear complexity in the number of instances, though communication delays add time to DISTPAB.

5. Results

  • Experiments compare DISTPAB with RP, GP, and PABIDOT on six datasets, using five classifiers to assess attack resistance and classification accuracy.The default σ is 0.3; the datasets span varying dimensions.Source: §5 · p. 21, Table 2 · p. 22

5.1. Distributed computing setup

  • The distributed setup equally divides each input dataset among four MATLAB parallel workers unless otherwise specified.The workers are created with parpool(N).Source: §5.1 · p. 22

5.2. Time Complexity

  • DISTPAB’s central node has worst-case complexity O(n^4), while distributed nodes have O(n^3 × m) when m >>> n.The algorithm moves steps 7–11 to the central node and step 19 to distributed nodes; communication delays can increase perturbation time.Source: §5.2 · pp. 22–23
  • Figures 4 and 5 report central-entity perturbation time for LRDS, excluding communication and I/O delays; Figure 5 averages results for four nodes.For attribute-count analyses, the LRDS data were equally divided among nodes, each processing 5,000 tuples.Source: Figure 4 · p. 23, §5.2 · p. 23, Figure 5 · p. 24

5.3. Complexity of distributed data perturbation vs. centralized data perturbation

  • For fixed attributes, DISTPAB and centralized PABIDOT both have O(m) complexity; DISTPAB takes more time due to communication delays, whose plotted effect is described as reasonable.The reported communication cost is independent of the number of instances when node and attribute counts are fixed.Source: §5.3 · pp. 24–25, Figure 6 · p. 25

5.4. Classification Accuracy

DISTPAB produces classification accuracy similar to PABIDOT and higher than RP and GP. In federated learning, more clients require more convergence rounds, while DISTPAB’s perturbed data yields accuracy close to the original data.

5.4. Classification Accuracy

  • Separate randomized expansion makes DISTPAB’s data slightly more randomized, reducing its utility relative to the centralized version.DISTPAB generates global perturbation parameters like the centralized algorithm, but each distributed entity performs randomized expansion separately.Source: §5.4 · p. 25
  • DISTPAB and PABIDOT produce similar average accuracies across five classifiers, and both outperform RP and GP.Figure 7 compares the original data with data perturbed by four approaches; training and testing used the same perturbed dataset under a strict privacy setting.Source: Figure 7 · p. 26, §5.4 · p. 26

5.4.1. Federated learning setup

  • More federated clients require more rounds for convergence, and the second-round accuracy drop becomes larger as client count increases.The data is divided equally among clients; the authors attribute the drop to extensive parameter modification during the second round.Source: Figure 8 · p. 27, §5.4.1 · p. 27
  • DISTPAB’s perturbed data has accuracy close to the original data, but convergence on perturbed data is assumed to require suitable hyperparameter tuning.Theoretical convergence analysis is beyond the paper’s scope.Source: §5.4.1 · pp. 27–28

5.5. Scalability

DISTPAB’s scalability depends on instance, attribute, and distributed-entity growth; unequal instance allocations are reported not to impair its complexity or accuracy. It also provides better attack resistance than RP and GP, with a slight advantage over PABIDOT.

5.5. Scalability

  • With attributes and distributed entities fixed, DISTPAB’s complexity is linear in instances; increasing attributes raises worst-case complexity to O(n^4).The paper argues that attributes and distributed entities increase less frequently or substantially than instances in the settings considered.Source: §5.5 · p. 28

5.6. Effect of the distribution of data instances (tuples) among distributed entities on the performance of perturbation

  • Unequal instance counts across entities do not adversely affect overall complexity or classification accuracy, according to the authors.For fixed attributes, each entity and the overall computation-plus-communication complexity are O(m); global parameters make perturbation similar to centralized PABIDOT.Source: §5.6 · pp. 28–29

5.7. Attack Resistance

  • Figure 9 compares minimum resistance measures for NI, ICA, and IO attacks; DISTPAB and PABIDOT outperform RP and GP, with DISTPAB slightly ahead of PABIDOT.The authors attribute DISTPAB’s slight increase over PABIDOT to greater randomization from separate randomized expansion at each entity.Source: §5.7 · p. 29, Figure 9 · p. 30

6. Discussion

DISTPAB combines randomized geometric transformations, randomized expansion, and Φ-separation for distributed privacy preservation. The discussion reports similar classification accuracy to a centralized method, linear per-node time growth in instances for a fixed setting, and a possible role in geographically distributed systems handling extensive data.

6. Discussion

  • For fixed attributes and a fixed distributed setup, central processing and communication costs are constant, while distributed-node time grows linearly with tuples.The analysis gives central complexity O(n^4), treated as O(1) when attributes are fixed, and per-node complexity O(n^3 × m), treated as O(m).Source: §6 · pp. 30–31
  • DISTPAB provides classification accuracy similar to the centralized approach and better than RP and GP; more federated clients increase convergence time without noticeably affecting accuracy.The paper also reports slightly lower accuracy than the centralized approach, attributing this to increased randomization in distributed expansion.Source: §6 · p. 31
  • DISTPAB uses geometric transformations, randomized expansion, and Φ-separation to perturb data, while reverse z-score normalization restores original attribute ranges.The paper describes these mechanisms as reducing reconstruction-attack success and reports better attack resistance than RP and GP.Source: §6 · pp. 30–31
  • The authors present DISTPAB as a potential privacy-preservation solution for extensive data in geographically distributed systems.The discussion names healthcare and open banking as examples of geographically distributed systems with privacy-preserving data-sharing needs.Source: §6 · p. 31

7. Conclusions

DISTPAB distributes perturbation work between a coordinating entity and distributed entities, with low stated computational complexity at each. The paper reports accuracy close to original-data classification, stronger attack resistance than two perturbation methods, and no utility degradation as entity count increases; improving efficiency as attributes grow remains future work.

7. Conclusions

  • DISTPAB achieves classification accuracy close to original-data classification, outperforms rotation and geometric perturbation in attack resistance, and avoids utility degradation as distributed-entity count increases.The paper presents these as overall conclusions, without supplying numerical values in this passage.Source: §7 · p. 32
  • The coordinating entity’s algorithm runs in O(1) time with respect to instances, while each distributed entity’s algorithm runs in O(m) time for m tuples when attribute count is constant.The paper links the distributed entities’ low computational complexity to excellent efficiency.Source: §7 · p. 32
  • Future work will investigate vertical federated learning to improve efficiency with respect to attribute count, with clients holding different feature spaces.The stated motivation is that clients could work on fewer attributes.Source: §7 · p. 32

Appendix .1. n-Dimensional translation matrix generation

The n-dimensional homogeneous translation matrix uses translational coefficients sampled from a uniform distribution bounded within (0, 1). The stated bounds are tied to z-score normalization.

Appendix .1. n-Dimensional translation matrix generation

Appendix .2. n-Dimensional reflection matrix

The appendix describes a homogeneous reflection matrix for reflection across axis one and a separate reflection across the (n+1) axis. The latter is stated to provide a low level of bias in the perturbation.

Appendix .2. n-Dimensional reflection matrix

Appendix .3. Generating the rotational matrix

The rotational matrix is generated by concatenating rotations on coordinate-axis pairs in a particular order to form a composite n-dimensional orthonormal matrix. The procedure uses selected angles and specifies entries for each rotation block.

Appendix .3. Generating the rotational matrix

  • Concatenating N(N−1) rotations on coordinate-axis pairs in a particular order generates the composite n-dimensional orthonormal matrix.The rotations act on planes represented by pairs of coordinate axes.Source: Appendix .3. Generating the rotational… · p. 34
  • Algorithm 4 generates the n-dimensional concatenated subplane rotation matrix M for angles 0 < θi < π, excluding π/6, π/4, π/3, π/2, 2π/3, 3π/4, and 5π/6.The passage identifies Algorithm 4 as the rotation-matrix generation procedure.Source: Appendix .3. Generating the rotational… · pp. 33–35
  • For each selected coordinate-pair block, the rotation matrix assigns cos(θ) and sin(θ) to specified entries, with the paired sine entry negated.The specified block entries are indexed through the four element positions in A.Source: Appendix .3. Generating the rotational… · pp. 34–35
Loading 2004.12108v2…