Source-linked AI summary
Algorithms that Remember: Model Inversion Attacks and Data Protection Law
Michael Veale, Reuben Binns, Lilian Edwards
TL;DR
The paper asks whether data-protection law adequately governs machine-learning models when models have traditionally been treated as intellectual property. It reviews model inversion and membership inference research, finding that some models can reveal training data and therefore may qualify as personal data, before examining the resulting rights, obligations, and regulatory implications.
Problem
Machine-learning governance is limited because the GDPR mainly regulates personal-data processing, while models themselves are generally treated as intellectual property.
Method
The paper reviews and explains model inversion and membership inference attacks, then analyzes how data-protection law could apply to vulnerable models and their participants.
Results
Vulnerable models may be classified as personal data because attacks can recover training-set variables, private information, or membership information from them.
Takeaways & Limitations
Treating vulnerable models as personal data could provide data subjects with information, erasure, and objection rights while imposing security, design, and storage obligations on model actors.
Takeaways & Limitations
Applying objection rights to an entire model could give each individual a potentially disproportionate veto over model consultation.
Abstract
from arXiv · showhide
Many individuals are concerned about the governance of machine learning systems and the prevention of algorithmic harms. The EU's recent General Data Protection Regulation (GDPR) has been seen as a core tool for achieving better governance of this area. While the GDPR does apply to the use of models in some limited situations, most of its provisions relate to the governance of personal data, while models have traditionally been seen as intellectual property. We present recent work from the information security literature around `model inversion' and `membership inference' attacks, which indicate that the process of turning training data into machine learned systems is not one-way, and demonstrate how this could lead some models to be legally classified as personal data. Taking this as a probing experiment, we explore the different rights and obligations this would trigger and their utility, and posit future directions for algorithmic governance and regulation.
1. Introduction
The paper examines whether machine-learning systems are adequately governed when they can reproduce discrimination and transform ordinary personal data into sensitive insights. It argues that model inversion and membership inference challenge the assumption that training data are irreversibly converted into models, with significant implications for data-protection regulation.
- Machine-learning systems raise governance concerns because they may reproduce societal discrimination and transform mundane personal data into sensitive, invasive insights.
- Although the GDPR is viewed as a route toward better governance, it primarily regulates personal-data processing rather than aggregated analytic products or models themselves.
- Recent security research indicates that training data, subsets of it, or information about training-set membership can sometimes be reconstructed from machine-learned models.
- The paper introduces data-protection law, model inversion, and membership inference before examining legal consequences for data subjects, modellers, and model recipients.
2. European data protection law and machine learning
The GDPR governs machine learning when personal data are collected, used, or applied in decisions, but it generally does not govern models during the period after construction and before deployment. The paper argues that model trading and model leakage make this boundary consequential for individual control and accountability.
- The GDPR applies when personal data are processed and provides data subjects rights including access, erasure, objection, and portability.Controllers also face confidentiality, breach-notification, risk-assessment, lawful-basis, purpose-limitation, and storage-limitation duties.
- Current data-protection law already constrains training and certain automated decisions through lawful grounds, information duties, objection or consent withdrawal, and appeal rights.
- Limits of the current regime: The GDPR generally leaves models themselves outside the data-protection regime after construction, even though individuals may want control over how models read them.The paper illustrates this concern with speaker-dependent automated lip-reading models that become more accurate after integrating individual training data.
- Models on the move: Models are increasingly traded or rented instead of underlying datasets, including through APIs, packaged models, and pre-trained learnware.The passages describe commercial model access through platforms and deployment on portable devices.
- Models on the move: Model trading may offer privacy and power-balancing benefits by reducing large-scale data accumulation and enabling local personalization.The paper connects this possibility to distributed local processing and privacy-preserving analytical techniques.
- Models as personal data: The paper challenges models’ conventional exclusion from personal-data status because model leakage may make them capable of rendering individuals identifiable.
3. Why might models be personal data?
Model inversion can recover some training-set variables, while membership inference determines whether an individual’s data were included. These attacks support treating vulnerable models as personal data, including when the model and auxiliary dataset are held separately.
- Attack setup: The attack framework considers a controller with auxiliary dataset A and model M(B), where some individuals overlap between A and training set B.The model may be accessed directly or through a query interface.
- Model inversion: Model inversion recovers some variables from B for individuals appearing in both B and A, producing a dataset combining A with recovered training information.Recovered variables may contain error or inexactitude.
- Attack implications: Evidence indicates that small training-algorithm changes can leave nearly indistinguishable models exploitable for leaking large amounts of private data, while some systems memorize sensitive strings.Support vector machines may also bundle training vectors with the model.
- Membership inference: Membership inference instead determines whether a given individual’s data were included in the training set, without recovering the training data’s columns.The attack adds membership information to an existing dataset.
- Legal classification: A model-inversion-vulnerable M(B) may be personal data by analogy to pseudonymized data when A provides a key for partial re-identification.This argument applies where one controller possesses both A and M(B), and legal precedent also supports the possibility when the model is released to another entity.
- Legal classification: Membership information may also qualify as personal data because GDPR’s “any information” formulation can relate to an individual by content, purpose or effect.The paper concludes that both attack types risk models being considered personal data without a maximalist reading of data-protection law.
4. Implications of models as personal data
Treating vulnerable machine-learning models as personal data would extend data-protection rights and obligations beyond training and querying, but their application creates substantial practical and interpretive difficulties.
- Rights and obligations: Classifying models as personal data would trigger new data-subject rights and controller obligations, including access, erasure, objection, provenance disclosure, and security duties.The paper examines these consequences for models after they have been built.
- Information and provenance: Information rights could require controllers to disclose model provenance and recipients, potentially making model flows publicly traceable.Where individual notification requires disproportionate effort, public disclosure of a model’s receipt and data sources may still be required.
- Erasure: Erasure is technically costly because retraining requires substantial time, labour, and energy, while post-training amendment remains difficult and largely undeployed.Retraining is not instantaneous or cost free; machine-unlearning methods are still exploratory and may require redesigned pipelines.
- Objection and use: Objection rights become difficult to interpret because querying a model may consult information from the entire training set rather than a single individual record.A possible reading could give each individual a veto over model consultation, although the paper describes this as problematic and potentially disproportionate.
- Security and model trading: If an unlawfully transferred model is inverted, the transfer could constitute both a data breach and a violation of the GDPR security principle.This risks undermining an advertised benefit of model trading: avoiding constant transmission of large datasets.
- Technical safeguards: Differential privacy and avoiding overfitting offer defenses against inversion, but deployment can be computationally expensive and methodological safeguards do not eliminate all practical challenges.Differential-privacy tools may be undermined by small or arcane implementation choices, while overfitting reflects memorization rather than generalizable learning.
- Storage limitation: Storage limitation could apply to models as well as training data, motivating model retirement or adaptation as the underlying phenomena change.Concept-drift methods are identified as a possible way to limit reliance on older data in machine-learning systems.
5. Discussion and conclusion
The paper argues that vulnerable machine-learning models may also constitute personal data, extending data-protection rights and obligations to traded models. It presents model inversion as a useful regulatory experiment but cautions that relying on security vulnerabilities is arbitrary and insufficient for broader algorithmic governance.
- Models as personal data: Vulnerable models may be both intellectual-property-protected analytic products and personal data, bringing information, erasure, objection, security, and storage-limitation obligations.The paper suggests this classification could rebalance power between model holders and people whose data trained them.
- Limits of the approach: Requiring a security vulnerability to trigger data-protection rights is disconnected and arbitrary.Models resistant to inversion may still raise legitimate demands for provenance, destination transparency, or erasure.
- Regulatory experiment: Model inversion serves as a realistic but inadvisable regulatory experiment for testing whether data-protection rights and obligations should extend to analytic products.The experiment also highlights potentially useful oversight consequences, including mapping trained-model provenance.
- Future governance: Treating data protection as an omnibus regime for all data-driven problems is misguided because existing provisions were not designed for contemporary profiling and inference capabilities.The authors frame boundary-testing as useful for applying existing law and developing regimes for issues it does not address in practice.