PaperlayerView exact revision on arXiv

Source-linked AI summary

When Machine Learning Meets Privacy: A Survey and Outlook

Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, Zihuai Lin

arXiv:2011.11819v1cs.LGcs.AIcs.CR

TL;DR

Privacy concerns have intensified as powerful ML methods spread, while existing research has often concentrated on privacy during the ML process. This paper surveys privacy and ML through three categories, reviews their techniques and challenges, and identifies future research directions, concluding that privacy metrics remain incomplete and protection against ML-based attacks is still in its infancy.

  • Problem

    Powerful ML creates privacy challenges because ML can act as both a privacy threat and a privacy-protection tool, while prior surveys often focused on particular models or methods.

  • Method

    The paper conducts a comprehensive survey organized into private ML, ML-aided privacy protection, and protection against ML-based privacy attacks.

  • Results

    The survey finds that differential privacy does not provide comprehensive evaluation across complex data and targets, ML-aided privacy protection is gaining momentum, and protection against ML-based attacks remains in its infancy.

  • Takeaways & Limitations

    The paper identifies new privacy metrics as an open question and points to synthetic-data generation with GNNs as a direction for protecting unstructured data.

  • Takeaways & Limitations

    Protection against rapidly evolving ML techniques remains the most challenging category, with many research problems requiring further investigation.

Abstract

from arXiv · show

The newly emerged machine learning (e.g. deep learning) methods have become a strong driving force to revolutionize a wide range of industries, such as smart healthcare, financial technology, and surveillance systems. Meanwhile, privacy has emerged as a big concern in this machine learning-based artificial intelligence era. It is important to note that the problem of privacy preservation in the context of machine learning is quite different from that in traditional data privacy protection, as machine learning can act as both friend and foe. Currently, the work on the preservation of privacy and machine learning (ML) is still in an infancy stage, as most existing solutions only focus on privacy problems during the machine learning process. Therefore, a comprehensive study on the privacy preservation problems and machine learning is required. This paper surveys the state of the art in privacy issues and solutions for machine learning. The survey covers three categories of interactions between privacy and machine learning: (i) private machine learning, (ii) machine learning aided privacy protection, and (iii) machine learning-based privacy attack and corresponding protection schemes. The current research progress in each category is reviewed and the key challenges are identified. Finally, based on our in-depth analysis of the area of privacy and machine learning, we point out future research directions in this field.

1 INTRODUCTION

The paper frames privacy in ML as a distinct, urgent research problem because ML can both threaten privacy and support its protection. It provides a comprehensive survey organized around ML’s different roles and identifies future research directions.

  • Motivation: The survey addresses the question: What are the privacy challenges and solutions associated with ML?The authors motivate this question by the growing impact of powerful deep learning techniques on privacy preservation.
  • Motivation: Existing work has largely focused on mitigating privacy risks during the machine learning process, including attacks on models and training datasets.These attacks can lead to sensitive information leakage, while protection schemes have also been proposed.
  • Motivation: ML introduces privacy challenges distinct from traditional data privacy because it can serve as a protection target, attack tool, or protection tool.ML systems and models face different privacy risks, and a single problem may involve multiple roles.
  • Contributions: The study presents a comprehensive survey of privacy in ML across different scenarios and applications.It responds to earlier surveys that focused on particular ML models or specific methods.
  • Contributions: The paper divides research into private ML, ML-enhanced privacy protection, and ML-based attacks, then analyzes schemes and future directions in each category.For private ML, it categorizes attacks and protection schemes; for the other categories, it reviews existing work and discusses new privacy-preservation techniques.

2 PRIVACY THREATS AND MACHINE LEARNING

This section introduces ML systems, their training and inference processes, and the three roles ML can play in privacy research. It distinguishes centralized from distributed learning and organizes privacy problems by whether ML is protected, used for protection, or used for attack.

  • The Machine Learning System and Models: Machine learning constructs models from training data to make predictions or decisions without explicit instructions.A supervised model maps input features to labels and is trained to predict labels for unseen data.
  • The Machine Learning System and Models: ML comprises model training, which minimizes prediction loss with regularization, and inference, which applies learned parameters to new inputs.Prediction accuracy on a testing dataset measures model performance.
  • The Machine Learning System and Models: Centralized learning stores data and trains models centrally, whereas distributed learning stores data locally and trains at least part of the model on individual clients.Centralization can provide high efficiency and accuracy but gives the operator direct access to sensitive data.
  • The Machine Learning System and Models: Collaborative frameworks such as federated and split learning enable shared model learning without sharing local raw training data with the server or other users.In split learning, clients train up to a cut layer and send weights representing intermediate feature vectors to the server.
  • Relationship of Privacy and Machine Learning: Privacy and ML research is grouped into private ML, ML-enhanced privacy protection, and ML-based privacy attack according to ML’s role.ML may be the protection target, a tool for protecting data, or an adversary’s attack tool, and one technique may occupy multiple categories.

3 PRIVATE MACHINE LEARNING

Private ML studies privacy threats and protection schemes when the ML system itself is the protection target. The section organizes these schemes alongside relevant attack and threat models.

  • 3 PRIVATE MACHINE LEARNING: Private ML concerns privacy preservation in machine learning systems and models.The section examines challenges, existing solutions, attack and threat models, and comparisons among protection schemes.
  • 3 PRIVATE MACHINE LEARNING: The proposed taxonomy organizes privacy-and-ML research by category and analyzes attack or threat models before protection schemes.The taxonomy is summarized in Fig. 3 and structures the paper’s review of private ML.

3.1 Attack and Threat Models

The survey analyzes ML privacy attacks by their targets, adversary knowledge, and attack methods. It distinguishes model privacy from training-data privacy and reviews extraction, feature-estimation, and membership-inference attacks.

  • Attack and Threat Models: ML privacy attacks are analyzed by attack targets, adversary knowledge, and attack methods.The attack targets include model privacy and training-data privacy; adversaries may have white-box or black-box access and use methods such as inversion or shadow models.
  • Attack and Threat Models: Training-data privacy protects sensitive records, features, statistical properties, and other information used to train ML models.Examples include medical profiles and financial records held by institutions using ML services.
  • Attack and Threat Models: Model privacy concerns protecting model structure, parameters, training algorithms, and commercially valuable ML services.Unauthorized disclosure can expose intellectual property or cause revenue loss for commercial API providers.
  • Model Extraction Attack: Model extraction attacks steal an AI model by producing a function f′ approximately equivalent to the original function f.Shadow-training attacks have extracted popular logistic-regression, decision-tree, and neural-network models with near-perfect fidelity.
  • Feature Estimation Attack: Feature-estimation attacks infer individual features or statistical properties of training data through inversion, shadow-model, or side-channel methods.Reported examples include genomic-information inference, facial-image reconstruction, deblurring, and recovery of input images from power traces.
  • Membership Inference Attack: Membership-inference attacks determine whether a record belongs to a model’s training dataset, using prediction differences or shared training updates.Studies examine black-box models, generative models, and collaborative learning, where gradients can reveal features tied to private data.

3.2 Private Machine Learning Schemes

Private ML schemes protect data or models through encryption, secure multiparty computation, obfuscation, and aggregation. These methods address different privacy settings but face computational, communication, usability, and leakage-control challenges.

  • Private Machine Learning Schemes: Private ML schemes include encryption, obfuscation, and aggregation.The survey presents these as major approaches for protecting ML data, models, or collaborative training processes.
  • Encryption: Homomorphic encryption can protect training data or model updates, but encrypted training is at least an order of magnitude slower.Training over encrypted data is feasible when the algorithm can be expressed as a low-degree polynomial; additive homomorphic encryption can protect gradients from an honest-but-curious server.
  • Encryption: Encrypted neural-network and DNN training remains challenging because computational complexity increases with backpropagation and limits data-scientist access to the process.The survey also identifies difficulty examining data, correcting mislabeled items, adding functionality, and tuning networks.
  • Secure Multi-Party Computation: Secure multiparty computation privately computes model updates without exposing individual data or model components.It has been applied to decision trees, linear and logistic regression, Naive Bayes, and k-means clustering.
  • Secure Multi-Party Computation: SMC-based methods can address both data and model privacy at the cost of communication overhead.Their non-trivial computational overhead and continued difficulty for privacy-preserving neural networks limit broader deep-learning applications.
  • Obfuscation: Obfuscation protects data or models by reducing precision through noise addition, including differential-privacy mechanisms.The survey discusses record-level and user-level privacy, including DP-FedAvg for protecting a user’s multiple data sources.
  • Aggregation: Aggregation supports collaborative learning by combining model parameters or prediction outputs while parties retain private datasets.Federated learning addresses non-IID, unbalanced, and massively distributed data, while PATE aggregates predictions from teacher ensembles trained on disjoint subsets.
  • Aggregation: Federated learning improves aggregation efficiency and robustness but lacks guarantees on the amount of user information leaked during training.Secure aggregation strengthens federated learning by using SMC to compute sums of model-parameter updates.

3.3 Summary on Private ML

Private ML schemes address multiple attack models, but their effectiveness depends on the protection mechanism and setting. Differential privacy does not provide comprehensive evaluation across complex data and privacy targets.

  • Private ML research summarizes attack models and compares protection schemes across different situations.
  • Attack models can be combined because model extraction may convert black-box access into white-box access for subsequent attacks.
  • Encryption preserves an adversary’s black-box knowledge, obfuscation reduces privacy risk at a utility cost, and aggregation commonly supports distributed systems.
  • Membership inference aligns with differential privacy, whereas model inversion can remain effective in collaborative and federated learning under differential privacy.
  • Differential privacy cannot comprehensively evaluate privacy in private ML when data are unstructured or multimedia and protection targets include features as well as membership.
  • Distributed ML still requires careful data-sharing design because parameters, feature vectors, and classification results can reveal user privacy, while aggregation and SMC may not protect against insiders.

4 MACHINE LEARNING AIDED PRIVACY PROTECTION

Machine learning is used to assess privacy risks, manage user preferences, and support private data release as online services and shared multimedia create new exposure. The surveyed approaches include automated risk detection, privacy assistants, and emerging methods for complex data, though unified metrics remain challenging.

  • Privacy risks and threats: Online photo sharing can disclose identities, surroundings, and sensitive family information, while traditional privacy schemes primarily target structured data.
  • ML-aided privacy protection schemes: ML aids privacy protection by assessing risks, managing privacy preferences, and supporting private data release.
  • Privacy Risk Assessment and Prediction: Automated systems evaluate data streams during access and sharing so that privacy protections can be deployed according to detected risk.
  • Privacy Risk Assessment and Prediction: Over 90% accuracy was reported for an SVM classifier that identified privacy risks in mobile applications.
  • Personal Privacy Management Assistant: Privacy assistants use policy evaluation and predictive models to reduce users’ burden in configuring privacy settings.
  • Personal Privacy Management Assistant: Recommendations based on like-minded user clusters and predictive privacy-preference models were reported to satisfy users.
  • ML-aided Privacy Protection: GNN-based privacy research is promising for unstructured data such as images and video, but unified privacy metrics remain unavailable.

5 MACHINE LEARNING-BASED PRIVACY ATTACKS AND CORRESPONDING PROTECTION SCHEMES

Machine learning serves as an attack tool that can automatically extract sensitive information from social, multimedia, and surveillance data. The survey reviews major attack models and emerging protections, especially adversarial examples, while identifying this area as comparatively immature and challenging.

  • Deep learning can automatically process photos and videos to extract private or sensitive information from social networks.
  • Social-network posts, check-ins, photos, and videos expose sensitive information that ML can use for location and preference inference.
  • Re-identification attacks use face recognition, while inference attacks use ML classifiers to infer sensitive attributes from available data.
  • Deep neural networks make re-identification more automatic and accurate, while traditional obfuscation protections become less effective.
  • Adversarial examples exploit carefully designed, often imperceptible inputs that cause deep neural networks to misclassify images.
  • Adversarial perturbations are being explored to defend users against automatic detection and attribute-inference attacks.
  • Protection against ML-based privacy attacks remains the paper’s most challenging category, with initial adversarial-learning solutions but many open problems.

6 OUTLOOK AND FUTURE DIRECTIONS

The outlook highlights unresolved challenges in privacy-preserving ML, including inadequate privacy metrics, perturbation trade-offs, and realistic black-box attack settings. It also identifies synthetic-data methods such as GANs as promising directions for protecting unstructured data while preserving utility.

  • 6.1 Perturbation in Deep Learning: Differential privacy is widely used for private ML, but its notation does not comprehensively evaluate privacy across complex data and protection targets.
  • 6.1 Perturbation in Deep Learning: Noise can perturb inputs, model parameters, or outputs, although most deep-learning work injects noise into gradients.
  • 6.1 Perturbation in Deep Learning: Output perturbation is fast and easy to implement but requires query limits because repeated adversarial queries can degrade its protection.
  • 6.1 Perturbation in Deep Learning: Objective perturbation adds a random linear term to the objective function, but deep neural networks complicate sensitivity calculation and privacy guarantees.
  • 6.1 Perturbation in Deep Learning: PixelDP places differential-privacy noise within neural networks and could be extended to protect training sets and autoencoder representations.
  • 6.2 Adversarial Examples for Privacy Protection: Privacy protection using adversarial examples commonly assumes white-box access, whereas black-box generation is presented as a more realistic practical setting.
  • 6.3 ML-aided Privacy Protection: GAN and VAE: GANs are expected to support privacy protection for unstructured data by preserving high ML utility while protecting sensitive dataset information.

7 CONCLUSION

The survey organizes privacy and ML research into private ML, ML-aided privacy protection, and protection against ML-based attacks. It concludes that differential privacy remains limited, synthetic-data approaches are emerging, and attack-protection research is still in its infancy.

  • The survey classifies research into private ML, ML-aided privacy protection, and protection against ML-based privacy attacks.
  • Private ML has received the most attention, but differential privacy does not provide comprehensive evaluation for complex data and privacy targets.
  • GNN-generated synthetic datasets open a direction for protecting unstructured data such as images and videos.
  • Protection against ML-based privacy attacks remains in its infancy, with adversarial examples and perturbations currently serving as mainstream techniques.
  • The authors expect increasing research activity as attention to privacy and machine learning grows.
Loading 2011.11819v1…