PaperlayerView exact revision on arXiv

Source-linked AI summary

Privacy-Preserving Machine Learning: Methods, Challenges and Directions

Runhua Xu, Nathalie Baracaldo, James Joshi

arXiv:2108.04417v2cs.LGcs.AIcs.CR

TL;DR

ML systems need substantial data and computation, yet data collection and model use expose privacy-sensitive information to inference attacks. This paper systematically reviews PPML methods and proposes the PGU framework to organize their phases, privacy guarantees, and utility. It concludes that stronger privacy can impair model utility, while important limitations and open questions remain around practical security, robustness, and deployability.

  • Problem

    Large-scale ML data collection and trained models can expose private information through inference attacks, creating a need to systematize PPML methods and evaluate their privacy protection.

  • Method

    The paper reviews PPML methodologies by pipeline phase and design principle, then evaluates them through the Phase, Guarantee, and technical Utility triad.

  • Results

    Privacy-preserving training is more challenging than privacy-preserving serving, and stronger differential-privacy guarantees reduce model accuracy.

  • Takeaways & Limitations

    PPML evaluation requires considering privacy guarantees together with utility, pipeline boundaries, trust assumptions, and interdisciplinary research needs.

  • Takeaways & Limitations

    TEE-based PPML can remain vulnerable to side-channel attacks, often requiring oblivious execution or ORAM-style protections.

Abstract

from arXiv · show

Machine learning (ML) is increasingly being adopted in a wide variety of application domains. Usually, a well-performing ML model relies on a large volume of training data and high-powered computational resources. Such a need for and the use of huge volumes of data raise serious privacy concerns because of the potential risks of leakage of highly privacy-sensitive information; further, the evolving regulatory environments that increasingly restrict access to and use of privacy-sensitive data add significant challenges to fully benefiting from the power of ML for data-driven applications. A trained ML model may also be vulnerable to adversarial attacks such as membership, attribute, or property inference attacks and model inversion attacks. Hence, well-designed privacy-preserving ML (PPML) solutions are critically needed for many emerging applications. Increasingly, significant research efforts from both academia and industry can be seen in PPML areas that aim toward integrating privacy-preserving techniques into ML pipeline or specific algorithms, or designing various PPML architectures. In particular, existing PPML research cross-cut ML, systems and applications design, as well as security and privacy areas; hence, there is a critical need to understand state-of-the-art research, related challenges and a research roadmap for future research in PPML area. In this paper, we systematically review and summarize existing privacy-preserving approaches and propose a Phase, Guarantee, and Utility (PGU) triad based model to understand and guide the evaluation of various PPML solutions by decomposing their privacy-preserving functionalities. We discuss the unique characteristics and challenges of PPML and outline possible research directions that leverage as well as benefit multiple research communities such as ML, distributed systems, security and privacy.

1 Introduction

ML systems depend on extensive data and computation, but data collection and model use create privacy risks that motivate PPML. The paper organizes PPML evaluation around phases, privacy guarantees, and technical utility, then reviews challenges and future directions.

  • Motivation: ML performance generally depends on large training datasets and high-powered computational resources.These requirements affect both training and inference, while resource-limited clients may rely on ML infrastructure services.
  • Motivation: Large-scale data collection and model use can leak private information through membership, model inversion, property, and gradient-based inference attacks.The paper gives patient membership inference as an example of sensitive information exposure.
  • Motivation: The paper motivates PPML solutions that integrate anonymization mechanisms into ML pipelines or introduce new privacy-preserving methods and architectures.The motivation includes privacy-sensitive health records and location information.
  • Paper framework: The PGU triad evaluates PPML by decomposing privacy-preserving functionality into Phase, Guarantee, and technical Utility.Phase concerns where protection occurs in the ML pipeline; Guarantee concerns protection strength or scope under threats and trust assumptions; Utility concerns computational-result usefulness.
  • Paper framework: The review examines privacy-preserving data preparation, model training and evaluation, deployment, and inference across PPML pipelines.It also analyzes object-oriented and pipeline-oriented privacy guarantees and classifies solutions into data publishing, data processing, architecture-based, and hybrid approaches.
  • Research directions: The paper discusses PPML design challenges and future research directions spanning machine learning, distributed systems, security, and privacy.Its stated aim is to support systematic understanding and evaluation of the PPML solution space.

2 Machine Learning Pipeline in a Nutshell

The ML pipeline spans training and serving, with training learning model parameters from samples and serving applying the trained model to predict outputs. Its stages, participants, facilities, trust domains, and computational tasks define the privacy-preserving design space.

  • The ML pipeline comprises data preparation, model training and evaluation, model deployment, and model inference, broadly grouped into training and serving.
  • Training learns model parameters by minimizing regularized error over labeled samples, while stochastic gradient descent updates parameters using the learning rate.
  • Serving applies the trained parameters to a target sample to produce a prediction without loss computation, regularization, derivatives, or weight updates.
  • Privacy-preserving training is more challenging than privacy-preserving serving because inference is a simplified subset of the training computation.
  • The pipeline involves data producers, model consumers, and computational facilities that may be locally owned or provided by third parties.
  • PPML research distinguishes privacy-preserving training from privacy-preserving serving and evaluates solutions across trust domains and underlying techniques.

3 Privacy-Preserving Phases in PPML

PPML approaches protect different phases of the ML pipeline, from data preparation and model generation to deployment and inference. The review organizes these approaches by computational protection, privacy-preserving serving strategies, and their compatibility across the pipeline.

  • The PPML pipeline covers privacy-preserving data preparation, model training and evaluation, model deployment, and inference.
  • Privacy-preserving model generation addresses how to minimize privacy leakage from training data and how to process that data privately.
  • Data-oriented approaches include anonymization, surrogate datasets, sketching, differential privacy, and related mechanisms applied before or during training.
  • Cryptosystem-based training performs more complex computation over protected data, relying especially on homomorphic or functional encryption rather than conventional encryption.
  • Encrypted-data training may require encoding floating-point values as integers and decoding results afterward, with conversion precision affecting accuracy and efficiency.
  • Privacy-preserving serving is generally simpler computationally than training, while model querying and publication aim to limit inference of private training information.
  • Serving-oriented methods include private aggregation of teacher ensembles, oblivious model transformations, and model compression through knowledge distillation.
  • Data-preparation techniques such as anonymization, sketching, and differential privacy are typically incompatible with privacy-preserving inference because perturbation can reduce prediction usefulness.

4 Privacy Guarantee in PPML

PPML privacy guarantees are evaluated across protected objects and the processing pipeline, with explicit trust boundaries and threat assumptions. The paper distinguishes data-oriented, model-oriented, and pipeline-wide guarantees while recognizing utility and computational trade-offs.

  • Privacy perspectives: PPML privacy analysis addresses both specific objects—data samples, model weights, and exchanged gradients—and the entire processing pipeline.Object-oriented analysis focuses on privacy leakage from protected artifacts, whereas pipeline-oriented analysis evaluates assurance across processing stages.
  • Object-oriented guarantees: Data-oriented guarantees prevent adversaries from learning private information directly from training or inference samples or linking it to an individual.Such protections include anonymization and differential privacy mechanisms applied to input data.
  • Privacy–utility trade-offs: Privacy protection can reduce utility because anonymization removes or aggregates features, differential privacy adds noise, and encryption increases training burden.The cited consequences include detrimental effects on trained-model accuracy and additional processing overhead.
  • Object-oriented guarantees: Model-oriented guarantees require that adversaries cannot derive private information from a model through repeated queries.Approaches include perturbing trained parameters with differentially private training and limiting model access patterns.
  • Pipeline-oriented guarantees: Pipeline-wide privacy assessment depends on defining processing boundaries and trust assumptions for each domain.The paper illustrates boundaries among data owners, local and third-party computational facilities, and model consumers.
  • Pipeline-oriented guarantees: The paper defines privacy levels ranging from no guarantee to full guarantee, which combines local protection during model generation with global protection during serving.Intermediate levels vary according to whether raw data, local updates, third-party facilities, or model serving are protected.

5 Technical Utility in PPML

The paper classifies PPML approaches by how they release data, use data for training, and structure the ML architecture. It then evaluates their effects through multiple utility dimensions.

  • PPML approaches are examined according to how privacy-sensitive data is released or published.
  • The classification also considers how privacy-sensitive data is used during model training.
  • Architecture-based analysis asks whether the ML system design prevents disclosure of private information.
  • The paper groups solutions into data publishing-based, data processing-based, architecture-based, and hybrid approaches.Hybrid approaches integrate two or three of the other categories.
  • Utility impacts are analyzed across computation, communication, model, and scalability costs.The broader utility analysis also includes scenario utility and other dimensions.

5.1 Type I: Data Publishing Approaches

Data publishing approaches protect privacy by eliminating or concealing identifiers, perturbing statistical results, transforming raw data, or using cryptographic techniques. These methods differ in privacy strength, utility effects, and processing requirements.

  • Data publishing approaches eliminate identifiers, conceal quasi-identifiers, perturb statistical results, or transform raw data through confusion and diffusion.
  • Elimination-based approaches: Anonymization mechanisms such as k-anonymity, l-diversity, and t-closeness remove or obscure private information in raw data.k-anonymity makes an individual indistinguishable from at least k-1 others, while l-diversity preserves multiple sensitive values within equivalence classes.
  • Perturbation-based approaches: Differential privacy publishes perturbed data or trains models with noise to provide an epsilon-differential privacy guarantee.The paper discusses both (epsilon, delta)-differential privacy and Rényi differential privacy, along with Laplace and Gaussian noise mechanisms.
  • Perturbation-based approaches: Sketching summarizes data streams with probabilistic structures and can combine communication compression with provable privacy benefits in distributed learning.Its randomness can implicitly provide differential privacy guarantees in collaborative filtering and related settings.
  • Confusion-based approaches: Cryptographic approaches use symmetric or advanced encryption schemes to compute over protected data and achieve stronger confidential-level privacy than anonymization or perturbation.Examples include secure multi-party computation, homomorphic encryption, and functional encryption.
  • Confusion-based approaches: Cryptographic data publishing generally requires subsequent secure processing because receivers should learn processing results rather than raw data.Key sharing and computation over encrypted data are central concerns for these systems.

5.2 Type II: Data Processing Approaches

Data processing approaches protect PPML computations through ordinary or secure computation, with secure methods organized by underlying cryptographic principles. These methods offer different privacy, implementation, precision, and efficiency trade-offs.

  • Data processing approaches are classified as ordinary computation or secure computation according to their data publication methodologies.
  • Secure computation approaches include additive masking, garbled circuits with oblivious transfer, advanced cryptography, mixed protocols, and trusted execution environments with oblivious methods.
  • Additive Mask based Approaches: Additive masking protects inputs with randomized values that cancel in the final output, including perturbation, DC-nets, and secret sharing.
  • Advanced Cryptographic Approaches: Functional-encryption implementations are scarce, while practical constructions often emphasize theoretical feasibility or limited inner-product functionality.
  • Advanced Cryptographic Approaches: Cryptographic PPML requires integer encoding, creating a precision trade-off: higher encoding precision can improve model accuracy but typically increases secure-computation time.
  • Trusted Execution Environment Approach: TEE-based approaches can expose information through hardware side channels, motivating oblivious data structures or ORAM to conceal secret-dependent memory access.

5.3 Type III: Architectural Approaches

Architectural approaches embed privacy into how models, participants, and computation are organized. The paper discusses delegation, distributed learning, federated learning, and knowledge-transfer architectures, while noting that no universal design principle applies.

  • Architecture-based PPML solutions have no universally applicable design principle, and representative designs illustrate privacy through system architecture.
  • Delegation-based Architecture: Delegation-based architectures let computation-limited parties create and use ML models while adding secure techniques or trust assumptions for privacy.
  • Distributed Learning: Distributed selective SGD enables parties to jointly train an accurate neural network without sharing their input datasets, using a parameter server.
  • Federated Learning: Federated learning keeps each participant’s training data local while a coordinator aggregates local model updates into a shared global model.
  • Knowledge Transfer: Knowledge-transfer architectures target distillation, compression, and transfer learning, with some systems also providing privacy guarantees.
  • Knowledge Transfer: PATE transfers knowledge from teachers trained on disjoint data to a student model, using noisy aggregation of teacher responses for stronger privacy.

5.4 Type IV: Hybrid Approaches

Hybrid approaches combine multiple privacy-preserving methods because a single technique may not provide sufficient protection against demonstrated attacks. Examples combine architecture-based methods with differential privacy, secure computation, or trusted execution environments.

  • Hybrid PPML systems combine more than one privacy-preserving methodology to address increased privacy requirements and threats such as membership inference, model inversion, and gradient leakage.
  • Architecture and Differential Privacy: Federated learning alone provides only a rudimentary privacy guarantee because its global model remains vulnerable to membership-inference and gradient-inference attacks.
  • Architecture and Differential Privacy: One hybrid strategy combines architecture-based approaches with differential privacy to strengthen protection for federated learning systems.
  • Architecture and Secure Computation: Another hybrid strategy combines architecture-based techniques with secure computation to protect privacy during distributed model training.
  • Federated Learning and TEE: Privacy-preserving federated learning also combines federated architectures with trusted execution environments, decrypting gradients inside secure enclaves to avoid gradient-related privacy violations.

5.5 Technical Approaches and Utility Cost

Privacy-preserving techniques impose utility costs relative to conventional ML systems. The paper evaluates these costs across model, computation, communication, scalability, scenario, and privacy-strength dimensions.

  • Privacy-preserving ML cannot be implemented without sacrificing utility compared with conventional machine-learning solutions.
  • The paper summarizes utility costs across model, computation, communication, scalability, scenario, and privacy-strength utility.
  • Anonymization can reduce model accuracy by removing identifiers or quasi-identifiers, while differential privacy and approximation techniques can also reduce model utility.
  • Homomorphic and functional encryption provide confidential-level privacy and enable computation over encrypted data, but their adoption carries technical utility costs.

6 Challenges and Potential Directions

PPML remains an active research area because privacy techniques, ML models, and privacy-related policies continue to evolve. The paper highlights privacy-focused systematization and evaluation as ongoing needs.

  • PPML remains active because privacy mechanisms, machine learning models, and privacy-related policies are rapidly evolving.

6.1 Open Problems and Challenges

PPML must balance privacy protection with model quality and system efficiency. The paper frames these goals as mutually challenging, with privacy-preserving mechanisms potentially reducing accuracy, fairness, or efficiency.

  • PPML evaluation must address privacy protection, model performance, and system efficiency together.Performance includes accuracy, robustness, and fairness; efficiency includes training or inference time.
  • Privacy-preserving capabilities can conflict with model performance and efficiency rather than preserving all vanilla-system qualities.The paper describes these open problems as generally mutually incompatible.
  • Communication efficiency requires fewer secure interactions and lower transmission overhead.
  • Computation efficiency requires low computational overhead or acceptable computation time during complex ML training.
  • PPML still lacks systematic definitions of privacy guarantees across threat models and trust settings.

6.2 Research Directions

The paper identifies research directions spanning privacy-guarantee evaluation, attack-aware defenses, efficient secure computation, and practical cryptographic support. These directions address unresolved limitations in current PPML systems.

  • PPML needs a formal, commonly accepted framework for evaluating the degree of privacy protection provided by systems.
  • Attack-aware PPML research should account for poisoning, inference, and evasion or exploratory attacks.Examples include clean-label and dirty-label poisoning, membership inference, model inversion, and gradient leakage.
  • Secure multiparty computation remains computationally and communicationally costly because inputs require encoding and garbled-table construction.
  • Research can improve secure computation by reducing ML complexity, optimizing neural-network architectures, and improving garbled-circuit MPC protocols.
  • Crypto-based PPML needs simpler practical schemes and solutions for encoding precision when converting between integers and floating-point values.Most secure-computation cryptography schemes operate on integer groups, whereas CKKS supports approximate-number arithmetic.

6.2.5 Privacy Perturbation Budget and Model Utility

Privacy perturbation creates a direct tension between privacy and model utility, while its effects on fairness and robustness remain active research questions. The paper also points to benchmarking and interpretability as complementary directions.

  • Tighter differential-privacy guarantees reduce model accuracy, whereas weaker privacy increases the likelihood of successful privacy-inference attacks.The paper describes this relationship through the privacy budget 𝜖.
  • Future work should determine privacy budgets that balance inference-attack likelihood with model accuracy, including dynamic approaches for different models.
  • Differential privacy can reduce accuracy more for underrepresented classes and subgroups, creating a fairness compatibility problem.
  • The relationship between privacy and model robustness remains insufficiently studied, including whether DP can defend against poisoning and backdoor attacks.Some studies suggest LDP and CDP may provide stronger practical than theoretical defenses against these attacks.
  • Interpretability research may clarify how deep models disclose private information and support the design of privacy-preserving methods.
  • PPML benchmarking should compare privacy protection, accuracy, fairness, and robustness across tools, tasks, methods, and datasets.The paper suggests a framework similar to LEAF for federated environments.

7 Conclusion

The paper synthesizes privacy-preserving methodologies for machine learning and evaluates them across multiple perspectives. It also identifies privacy-guarantee levels, challenges, open problems, and future research directions requiring interdisciplinary effort.

  • The paper summarizes privacy-preserving machine-learning methodologies across system phases and underlying design principles.
  • It evaluates privacy guarantees by defining several privacy-guarantee levels for PPML systems.
  • It outlines challenges, open problems, and future directions requiring collaboration across machine learning, distributed systems, and security-privacy communities.
Loading 2108.04417v2…