Machine Learning Models that Remember Too Much

Congzheng Song; Thomas Ristenpart; Vitaly Shmatikov

Source-linked AI summary

Machine Learning Models that Remember Too Much

Congzheng Song, Thomas Ristenpart, Vitaly Shmatikov

arXiv:1709.07886v1Sep 22, 2017cs.CRcs.LG

TL;DR

The paper asks whether a malicious ML provider can cause models trained on sensitive data to memorize and leak that data without observing training. It develops white- and black-box attacks using parameter encoding, malicious regularization, and synthetic-data-like secondary tasks, and finds that models retain standard predictive quality while exposing training information.

Problem
Sensitive-data holders may use third-party ML code without understanding what models capture, while high-capacity models can memorize information beyond primary-task needs.
Method
The paper modifies training pipelines through parameter encoding, malicious regularization, or augmentation-like synthetic inputs whose labels encode secret training-data bits.
Results
Across image, face, and text tasks, malicious models retain conventional accuracy and generalizability while enabling accurate extraction, including 70% of a 10,000-document corpus and 17 complete face images.
Takeaways & Limitations
Training-code providers can abuse model memorization to leak substantial training information even when released models expose only black-box prediction access.
Takeaways & Limitations
The evaluation focuses on supervised learning, although the techniques may potentially apply to unsupervised learning; membership inference and related objectives are not explored directly.

Abstract

from arXiv · show

Machine learning (ML) is becoming a commodity. Numerous ML frameworks and services are available to data holders who are not ML experts but want to train predictive models on their data. It is important that ML models trained on sensitive inputs (e.g., personal images or documents) not leak too much information about the training data. We consider a malicious ML provider who supplies model-training code to the data holder, does not observe the training, but then obtains white- or black-box access to the resulting model. In this setting, we design and implement practical algorithms, some of them very similar to standard ML techniques such as regularization and data augmentation, that "memorize" information about the training dataset in the model yet the model is as accurate and predictive as a conventionally trained model. We then explain how the adversary can extract memorized information from the model. We evaluate our techniques on standard ML tasks for image classification (CIFAR10), face recognition (LFW and FaceScrub), and text analysis (20 Newsgroups and IMDB). In all cases, we show how our algorithms create models that have high predictive power yet allow accurate extraction of subsets of their training data.

1 INTRODUCTION

The paper shows that malicious training-code providers can make accurate models memorize sensitive training information, then recover it through white- or black-box access. Across image, face, and text tasks, these attacks preserve conventional model-quality metrics while enabling substantial extraction.

Data holders may use third-party ML code without understanding what the resulting model captures about sensitive training data.
Modern ML models have enough capacity to encode information beyond what is strictly necessary for accurate primary-task predictions.
Minor training modifications can produce high-quality models that leak detailed information about their training datasets.
White-box methods encode secrets in parameter bits, correlations, or signs, while black-box methods resemble data augmentation and train a secondary secret-prediction task.
Across CIFAR10, LFW, FaceScrub, 20 Newsgroups, and IMDB, malicious models had virtually identical accuracy and generalizability to conventional models.
70% of a 10,000-document training corpus was leaked by a white-box text classifier without reducing accuracy; a black-box gender classifier reconstructed 17 complete face images.
Black-box attacks on image models required no auxiliary information, whereas text attacks compared exact training vocabularies with publicly compiled vocabularies.
The paper concludes that third-party training code is risky because model memorization can leak training data even through black-box interfaces without significant quality-metric impact.

2 BACKGROUND

The background describes supervised classification pipelines from data splitting and augmentation through regularized training and validation. It also situates these pipelines within widely available ML libraries, cloud services, and algorithm marketplaces.

2.1 Machine Learning Pipelines: Supervised learning represents a classification model as a parameterized function fθ mapping feature space X to a discrete class set Y.
2.1 Machine Learning Pipelines: A dataset is partitioned into training data Dtrain of size n and test data Dtest before model training.
2.1 Machine Learning Pipelines: Data augmentation expands Dtrain with transformed points, such as flipped, noisy, or distorted images, to improve generalizability.
2.1 Machine Learning Pipelines: Training minimizes a loss over Dtrain, while a regularizer penalizes model complexity and helps prevent overfitting.
2.1 Machine Learning Pipelines: SGD and related optimizers iteratively update parameters using negative gradients until reaching a local minimum.
2.1 Machine Learning Pipelines: Validation measures test accuracy and may reject a model when accuracy is too low; the train-test gap measures overfitting.
2.1 Machine Learning Pipelines: Linear models such as SVMs and logistic regression are common for text classification, while deep networks use layered nonlinear transformations and can have huge parameter counts.
2.2 ML Platforms and Algorithm Providers: ML providers supply training code, including through cloud services and marketplaces where users execute third-party programs on their data.

3 THREAT MODEL

The threat model considers a client who uses adversary-controlled ML code on private data, while the adversary cannot observe execution but later accesses the published model. The adversary seeks to infer private training data through memorization that survives client validation.

The client trains a classification model on private data and accepts it after measuring test accuracy and the test-train gap.
The adversary controls the ML pipeline but focuses on modifying augmentation, regularization, or post-training parameters while keeping the core trainer conventional.
The adversary’s main objective is to infer as much of the client’s private training dataset as possible.
The paper does not explore membership inference, partial information, or metadata extraction, although it states that the techniques can directly support these objectives.
The adversary cannot observe the client’s data, pipeline execution, or unpublished model because the training environment is isolated and erased afterward.
The pipeline can leak data only by making the model memorize it while ensuring that the model passes validation.
After publication, white-box access exposes model parameters, whereas black-box access provides only input-output predictions and class labels.

4 WHITE-BOX ATTACKS

White-box attacks encode training data into visible model parameters while preserving the model’s primary-task performance. The paper studies direct bit encoding, correlation-based encoding, and sign encoding, each with corresponding decoding procedures and capacity limits.

White-box attacks directly encode training-dataset information into model parameters, with client validation requiring high test accuracy.
Lower-bit encoding: Lower-bit encoding post-processes a benign model by replacing the lower b bits of each parameter with substrings of a secret extracted from training data.
Lower-bit encoding: The lower-bit method can encode up to ℓb secret bits, where ℓ is the number of model parameters.
Correlated value encoding: Correlation encoding adds a malicious loss term that drives model parameters toward correlation with numerical or pseudorandom vector representations of the secret.
Correlated value encoding: The correlation term resembles conventional regularization but weights parameters according to the secrets being memorized.
Sign encoding: Sign encoding constrains parameter signs to match secret bits and uses a penalty term when signs disagree, though optimization may not satisfy every constraint.
Sign encoding: Sign decoding reads parameter signs as bits, but capacity and imperfect optimization make the method unsuitable for encoding compressed training-data binaries.

5 BLACK-BOX ATTACKS

Black-box attacks abuse model capacity by adding synthetic, randomly labeled inputs whose labels encode training-data secrets. The resulting models can memorize these inputs and reveal their labels through prediction queries without materially reducing primary-task performance.

The capacity-abuse attack augments the benign training set with synthetic inputs whose labels encode secrets about the original training data.
The adversary changes only data generation, so the training pipeline remains identical to benign training apart from augmentation.
Each synthetic input can ideally encode ⌊log2(c)⌋ bits when the model has c output classes.
Image synthesis requires no auxiliary knowledge, while text synthesis can use either the exact training vocabulary or a public vocabulary.
Unknown model vocabularies can produce empty-feature documents and potentially degrade both test accuracy and decoding accuracy.
The adversary reproduces deterministic synthetic inputs and queries the trained model, whose returned labels reveal the encoded secret bits when it is overfitted to those inputs.
If the model has sufficient capacity, accuracy on malicious inputs can be near perfect, yielding low extraction error.
The malicious secondary task uses essentially random labels, yet models retain high quality on original tasks while leaking information through augmented inputs.

6 EXPERIMENTS

Across image and text benchmarks, the attacks extract training information while often preserving primary-task accuracy and generalizability. Extraction quality and capacity vary with attack parameters, model size, task type, and auxiliary knowledge.

Experimental setup: The evaluation covers image and text benchmarks, including CIFAR10, LFW, FaceScrub, 20 Newsgroups, and IMDB, using conventional and malicious models.The experiments use representative publicly available datasets and compare malicious training against conventional training.
LSB encoding attack: For most datasets, using fewer than 20 lower parameter bits does not reduce primary-task test accuracy, whereas CIFAR10 degrades when b exceeds 20.Binary image classifiers tolerate more precision loss than multi-class tasks.
LSB encoding attack: 57 Mb of compressed data, equivalent to 70% of the 20 Newsgroups training dataset, can be perfectly extracted with LSB encoding.Lossless compression and error-free parameter encoding make decoding perfect.
Correlated value encoding attack: Correlated-value encoding reconstructs reasonable images when λc exceeds 1.0, but lower reconstruction error can reduce test accuracy by more than 1% on CIFAR10 and FaceScrub recognition.For binary classification, reasonably low MAPE can be achieved with about a 0.1% accuracy reduction.
Sign encoding attack: Sign encoding can reconstruct FaceScrub gender images with error below 1, while accuracy changes remain below 0.5% for 20 Newsgroups and around 0.8%–1.2% for IMDB.FaceScrub face recognition degrades by 2.6% at large λs, whereas CIFAR10 with λs = 50 still generalizes well.
Capacity abuse attack: Capacity abuse encodes up to 17 FaceScrub gender images and 79 20 Newsgroups documents, while text accuracy drops less than 0.6% under the reported settings.Larger models provide greater encoding capacity and more accurate decoding, despite similar test accuracy on smaller and bigger models.

7 COUNTERMEASURES

The paper discusses why malicious memorization is difficult to detect and describes countermeasures for parameter- and augmentation-based attacks. Some attacks alter parameter distributions, while capacity abuse leaves them visually normal.

Detecting malicious memorization is difficult because information can be encoded directly in parameters, through regularization, or via specially crafted data augmentation.These approaches can resemble standard machine-learning techniques during manual code inspection.
Replacing lower parameter bits with random noise can destroy least-significant-bit encodings without affecting model performance.The attack depends on lower bits contributing little to accuracy.
Malicious correlated-value and sign-encoding attacks produce parameter distributions unlike those of benign models.Sign encoding is more centered at zero, resembling conventional l1-norm regularization.
Capacity-abuse parameters are not visibly different because training remains unchanged apart from adding inputs.This makes distribution-based anomaly detection ineffective for that attack.

8 RELATED WORK

The paper distinguishes its malicious-training threat from prior inference, poisoning, evasion, and secure-training work. Its attacks intentionally create accurate models that leak specific training inputs, including under black-box access.

Existing secure-training research includes secure computation, differential privacy, and trusted hardware, whereas this paper focuses on malicious training algorithms.The related-work discussion identifies no prior work targeting malicious learning algorithms that create models leaking training data.
Unlike membership inference and model inversion, the paper studies training algorithms that intentionally make models leak specific training inputs.The authors characterize this distinction as a covert-channel problem rather than merely exploiting an existing side channel.
Prior membership-inference attacks determine whether a labeled example was in the training set and work best when generalization is poor.The paper instead targets substantially more information extraction while maintaining good generalization.
The capacity-abuse attack resembles malicious data augmentation but targets training-data leakage rather than evasion.It uses randomized augmented data while simultaneously training for good testing accuracy on the primary task.
The paper differs from privacy-preserving classification protocols because those systems prevent white-box attacks but not black-box attacks.
The LSB attack exploits the large number and unnecessarily high precision of model parameters, connecting it to model-compression research.The paper identifies compression techniques as a possible future countermeasure.

9 CONCLUSION

The paper concludes that malicious training algorithms can produce accurate, generalizable models that leak substantial training information, even through black-box access. It argues that sensitive-data ML cannot be used blindly and calls for least-privilege training frameworks.

Malicious training algorithms can preserve accuracy and generalizability while leaking significant information from their training datasets, even with black-box model access.
Data holders should understand third-party training code before making models trained on sensitive data available to others.The recommendation applies even when a service claims not to observe algorithm execution.
The paper proposes a machine-learning version of the principle of least privilege: models should capture only what their designated task requires from the training dataset.Formalization, practical methods, and certification remain open research topics.