PaperlayerView exact revision on arXiv

Source-linked AI summary

Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets

Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong, Nicholas Carlini

arXiv:2204.00032v2cs.CRcs.LGstat.ML

TL;DR

The paper asks whether poisoning training data can make models leak private information from other parties’ examples. It develops targeted and untargeted poisoning attacks against privacy inference, finding large gains across membership inference, attribute inference, and data extraction, while showing that defenses and protocol assumptions constrain the scope of the results.

  • Problem

    Existing research studied training-data integrity attacks and privacy attacks largely as separate threats, leaving their interaction insufficiently addressed.

  • Method

    The paper designs targeted poisoning attacks on known or partially known samples and untargeted attacks that poison data to increase leakage from all honest training points.

  • Results

    Across membership inference, attribute inference, and data extraction, poisoning less than 0.1% of training data increases privacy leakage by 1 to 2 orders of magnitude.

  • Takeaways & Limitations

    The results challenge the privacy relevance of ideal-functionality guarantees in machine-learning MPC when parties can choose their training data.

  • Takeaways & Limitations

    The attacks are not designed to evade specific poisoning defenses, and loss clipping can significantly reduce their effectiveness.

Abstract

from arXiv · show

We introduce a new class of attacks on machine learning models. We show that an adversary who can poison a training dataset can cause models trained on this dataset to leak significant private details of training points belonging to other parties. Our active inference attacks connect two independent lines of work targeting the integrity and privacy of machine learning training data. Our attacks are effective across membership inference, attribute inference, and data extraction. For example, our targeted attacks can poison <0.1% of the training dataset to boost the performance of inference attacks by 1 to 2 orders of magnitude. Further, an adversary who controls a significant fraction of the training data (e.g., 50%) can launch untargeted attacks that enable 8x more precise inference on all other users' otherwise-private data points. Our results cast doubts on the relevance of cryptographic privacy guarantees in multiparty computation protocols for machine learning, if parties can arbitrarily select their share of training data.

1 INTRODUCTION

The paper connects training-data poisoning with privacy attacks, showing that an adversary can write into a training dataset to extract private information from other users’ data. Small targeted poisonings and larger untargeted poisonings substantially amplify membership, attribute, and data-extraction attacks.

  • Threat connection: Poisoning links attacks on training-data integrity to privacy attacks against membership, attributes, and extracted data.The paper frames poisoning as a way to influence privacy leakage from other training examples.
  • Targeted attacks: <0.1% poisoning improves membership inference, attribute inference, and data extraction by 1 to 2 orders of magnitude.The attacks target other training examples rather than merely degrading model performance.
  • Targeted attacks: 8 poison samples, or 0.03% of CIFAR-10, raise membership-inference TPR from 7% to 59% at a 0.1% FPR.At 50% TPR, poisoning reduces FPR to 0.05%, compared with 24% from prior work.
  • Targeted attacks: 64 poisoned WikiText sentences reduce the median guesses needed to extract a 6-digit canary from 9,018 to 230.This is a 39× improvement in the extraction attack.
  • Untargeted attacks: Untargeted poisoning with control of up to 50% of training data reduces membership-inference error across all victim data points by 8×.This setting models collaborative training in which parties pool private datasets.
  • Implications: The results challenge treating trained neural networks as ideal functionalities with bounded privacy leakage in cryptographic protocols.A malicious party can follow the training protocol while choosing inputs that increase leakage.

2 BACKGROUND AND RELATED WORK

The paper situates its contribution at the intersection of training-data privacy and poisoning research. It reviews canonical privacy attacks, poisoning categories, stronger adversary models, defenses, and the notation used for classifiers and language models.

  • Training privacy: Membership inference tests training-set participation, attribute inference recovers an unknown user feature, and data extraction recovers a secret string.These are the three canonical training-data privacy attacks considered in the paper.
  • Poisoning attacks: Poisoning attacks include indiscriminate attacks that reduce performance, targeted attacks that induce specific errors, and backdoors triggered at test time.Backdoor attacks implant a trigger that enables adversarial misclassification of arbitrary perturbed test points.
  • Threat model: The paper differs from stronger prior adversaries because it assumes control of training data rather than training code, model architecture, or the training server.Its attacker poisons data to violate other users’ privacy.
  • Defenses: Poisoning and privacy defenses include robust data sanitization, memorization-reduction heuristics, and differential privacy.Differential privacy is described as providing provable protection even for poisoned datasets.
  • Defenses: The attacks are worst-case and are not explicitly designed to evade specific poisoning defenses; loss clipping can significantly reduce their effectiveness.Robustness to stronger sanitization mechanisms remains an open question.
  • Machine learning notation: A classifier maps an input sample to a probability vector over n classes, and training produces its learned weights from a dataset.The notation defines a training algorithm T and a loss function for labeled samples.
  • Machine learning notation: A causal language model predicts the next token by mapping a token sequence to a probability distribution over the vocabulary.Its loss is defined using cross-entropy over successive token predictions.

3 AMPLIFYING PRIVACY LEAKAGE WITH DATA POISONING

The paper introduces a privacy inference game in which an adversary poisons training data before querying the trained model. It develops targeted and untargeted poisoning strategies that amplify leakage across membership, attribute, and data-extraction attacks.

  • Privacy inference game: The poisoning privacy game lets an adversary add arbitrary poisoned points before a challenger trains and exposes a model for queries.The game generalizes privacy attacks by allowing the adversary to maximize leakage of a targeted point.
  • Adversary capabilities: The adversary may use as few as one targeted poison or as many as the original dataset size, covering both small-scale and MPC-like settings.The framework assumes knowledge of the data distribution, target universe, and training algorithm.
  • Attack goals: Targeted attacks aim to increase leakage for a specific example, whereas untargeted attacks aim to harm the privacy of all training samples.The paper studies both attack types under the poisoning framework.
  • Evaluation: Attack performance for small secret spaces is evaluated with TPR and FPR, especially at FPR=0.1%, which focuses on precisely targeting worst-case users.Membership and attribute inference use these rates over repeated games.
  • Attack constructions: The paper presents a provably perfect kNN attack that labels a target correctly if and only if it was present in the original training set.This yields 100% membership-inference accuracy, and some points cannot be inferred better than chance without poisoning.
  • Empirical attack design: The empirical strategy handcrafts poisons to increase a sample’s influence, extending from membership inference on images to attribute inference on tabular data and canary extraction from language models.The approach builds on the observation that outliers are especially vulnerable because they have greater influence on model behavior.

4 MEMBERSHIP INFERENCE ATTACKS

The paper develops targeted and untargeted poisoning attacks that substantially increase membership-inference leakage, often with very small data poisoning budgets. The attack works by making targeted examples more influential and their member/non-member behavior easier to distinguish, while its effectiveness depends on training choices such as loss clipping.

  • Targeted attacks: 8 poisons, or 0.03% of CIFAR-10 training data, increase membership-inference TPR at 0.1% FPR from 7% to 59%.A single poison increases TPR by 1.75×; on CIFAR-100, poisoning increases TPR from 22% to 69% at 0.1% FPR.
  • Targeted attacks: At fixed 50% TPR, poisoning reduces CIFAR-10 false positives from 24% to 0.05%, a 480× improvement.On CIFAR-100, poisoning reduces the false-positive rate by 100× at the same TPR.
  • Why the attack works: Poisoning increases the targeted example’s relative influence, shifting confidence distributions and making members easier to separate from non-members.For one ship example, 16 poisons change member confidence to 0.4%–28.5% and non-member confidence to 0%–2.4%.
  • Vulnerable points: For originally hardest-to-attack points, poisoning improves TPR at 0.1% FPR by 430×, from 0.1% to 43%, while already-vulnerable outliers improve little.The baseline TPR for initially easiest points is already 73%, leaving little room for improvement.
  • Shadow models: Poisoning makes a global-threshold membership attack perform nearly as well as an attack using 128 shadow models.With 16 targeted poisons, the non-calibrated attack reaches 43% TPR at 0.1% FPR, compared with 0.1% without poisoning.
  • Defenses and limitations: Loss clipping constrains the attack: with C=1, additional poisons can weaken it, while aggressive clipping still permits up to 3× gains but raises test error by 45%.At C=1, one poison improves over baseline, but additional poisons harm performance at low false-positive rates.

5 ATTRIBUTE INFERENCE ATTACKS

The paper strengthens attribute inference by calibrating loss-based attacks and poisoning both versions of a target, making the unknown attribute more distinguishable. On Adult, targeted poisoning substantially improves gender inference over unpoisoned attacks and the imputation baseline.

  • Attack setup: The attack infers a binary attribute by comparing losses for alternate target versions and selecting the value with the lower loss.A stronger variant calibrates these losses with shadow models and a likelihood-ratio test.
  • Attack setup: Poisoning inserts r/2 mislabeled copies of each target version, forcing similarly large losses so the true variant has greater detectable influence.
  • Experimental setup: The Adult evaluation uses 48,842 users, a three-layer income classifier, and models trained on random 50% subsets, achieving 84% test accuracy.Targets include stated gender and relationship status.
  • Results: At FPR 0.1%, gender-inference TPR rises from 1% without poisoning to 30% with 16 poisons.
  • Results: At TPR 50%, poisoning reduces FPR from 39% to 1.2%, a 33× error reduction; without poisoning, the attack underperforms imputation.

6 EXTRACTION IN LANGUAGE MODELS

The paper applies poisoning and shadow-model calibration to extract high-entropy secrets from language models. These techniques substantially increase canary exposure, reducing the guesses needed for recovery.

  • Setup: The extraction task targets well-formatted high-entropy secrets, using six-digit canaries preceded by attacker-known or partially known prefixes.The setup models secrets such as numbers embedded in standardized contexts.
  • Setup: The experiments train small GPT-2 variants on WikiText-2, a roughly 3-million-token corpus, and evaluate recovery by enumerating possible secret values.
  • Poisoning strategies: Prefix poisoning assumes control over the canary’s preceding prefix, while suffix poisoning inserts poisoned copies of that prefix with chosen suffixes.
  • Calibration: Shadow-model calibration increases average exposure from 3.1 bits to 7.1 bits with 2 shadow models, reducing guesswork by 16×.Exposure rises moderately to 7.4 bits with additional shadow models.
  • Calibration: Calibration increases recovery in fewer than 100 guesses from 0.1% to 10%, a 100× improvement.
  • Poisoning results: Combining prefix and suffix poisoning reaches 11.4-bit average exposure with 64 insertions and reduces median guesswork from 9,018 to 230 guesses, a 39× improvement.Recovery within 100 guesses rises from 10% to 42%.
  • Poisoning results: Poisoning alone raises exposure from 3.1 bits to 7.3 bits, while shadow models alone raise it to 7.4 bits; combining both abilities compounds the benefit.

7 DISCUSSION AND CONCLUSION

The paper shows that poisoning training data can increase privacy leakage from machine learning models, linking data integrity and privacy risks. These attacks challenge privacy expectations and cryptographic protocol abstractions in collaborative learning.

  • Poisoning as little as <0.1% of training data increases membership inference, attribute inference, and data extraction leakage by 1 to 2 orders of magnitude.
  • In decentralized settings, untrusted data creates a privacy threat beyond the previously recognized integrity threat.
  • The attacks can transform inlier points into privacy outliers, bringing average-case leakage toward worst-case leakage.
  • Average-case privacy audits may underestimate worst-case leakage under small poisoning attacks, while worst-case auditing may better reflect privacy for most users.
  • The authors frame data privacy and integrity as intimately connected and identify their interplay as a direction for future work.

A.1 Results on CIFAR-100

On CIFAR-100, the targeted membership-inference experiment replicates the CIFAR-10 setup while examining poison-label and target-modification strategies. Consistent incorrect labels and exact-target poisoning perform best among the tested alternatives.

  • Results on CIFAR-100: The CIFAR-100 experiment uses the same setup as CIFAR-10 while varying the number of poisons per target.
  • Mislabeling strategies: Consistently assigning one random incorrect label to all poisoned copies performs better than assigning different incorrect labels or selecting likelihood-based labels.
  • Target-modification strategies: Modifying the target with augmentations, Gaussian noise, or unadversarial examples performs worse than mislabeling the exact target sample.
  • Alternative poisoning strategies: The alternative strategies include mislabeled noisy versions, feature-space surrounding variants, and unadversarial examples intended to increase target influence.

A.3 Attacks with Partial Knowledge of the Target

The membership-inference attack remains effective when the adversary knows only a near neighbor rather than the exact target image. With loss clipping, however, poisoning shifts losses without improving member–non-member separability.

  • Partial target knowledge: The attack remains robust when the adversary knows only a similar image rather than the exact target.
  • Target approximation: Near-neighbor pairs are selected by matching CIFAR-10 examples using CLIP features, often depicting the same object from different views or scales.
  • Evaluation setup: The evaluation separates target models trained on the exact image from shadow models trained on its known neighbor.
  • Loss clipping: With clipped losses, poisoning shifts loss distributions but does not make members and non-members more separable.

A.5 Untargeted Membership Inference Attacks

Untargeted poisoning increases membership-inference vulnerability across datasets and strategies by making member and non-member confidence distributions more separable. Its largest effect is on points that were initially difficult to attack.

  • Datasets and strategies: Untargeted attacks are evaluated on CIFAR-10, CIFAR-100, and the Texas100 SVM dataset.
  • Poisoning strategies: On CIFAR-10 and CIFAR-100, consistently flipping poisoned points into one class is the best-performing strategy, while random flipping performs slightly better on Texas100.
  • Poison distribution: Out-of-distribution poisoning with CIFAR-100, MNIST, or random-noise data does not perform as well as consistently mislabeling in-distribution data on CIFAR-10.
  • Mechanism: Untargeted poisoning decreases confidence and makes member and non-member loss distributions more separable, facilitating membership inference.
  • Disparate impact: 37×: For initially least-vulnerable points, the TPR at a 0.1% FPR rises from 0.1% to 3.7% under untargeted poisoning.

B ADDITIONAL EXPERIMENTS FOR ATTRIBUTE INFERENCE ATTACKS

The relationship-status experiment replicates the attribute-inference setup and finds that poisoning substantially improves low-false-positive inference, while performance degrades at higher false-positive rates.

  • At a 0.1% false-positive rate, 16 poisons raise relationship-status inference TPR from 4% without poisoning to 18%.
  • Above 5% false-positive rates, all attribute-inference attacks, including poisoned attacks, perform worse than trivial imputation.
  • Poisoning makes the attribute-inference attack significantly outperform correlation-based baseline imputation at low false-positive rates.

C ADDITIONAL EXPERIMENTS FOR CANARY EXTRACTION

The canary-extraction experiments compare poisoning strategies and show that repeated, identical padding is especially effective, while poisoning improves recovery for any fixed guessing budget.

  • Random-token and random-6-digit-number padding provide moderate exposure increases to 8.5 bits and 8.3 bits when the same suffix is reused.
  • Zero-padding a canary prefix and repeating the same padding for every poisoned copy outperform random-token and random-number alternatives.
  • Average exposure alone does not establish that canary extraction becomes practically easier across the relevant guessing range.
  • For any fixed guess budget, poisoning increases the attacker’s success rate in recovering a secret canary.

D PROVABLY AMPLIFYING PRIVACY LEAKAGE

The paper proves that poisoning can amplify membership leakage in kNNs and linear SVMs by making otherwise uninformative or non-support-vector points affect model behavior.

  • k-nearest neighbors: Targeted poisoning achieves 100% membership-inference accuracy on a targeted point for kNNs.
  • k-nearest neighbors: Without poisoning, membership inference cannot outperform chance for points whose removal leaves every kNN prediction unchanged.
  • k-nearest neighbors: Unused kNN training points can be removed without affecting the model’s output on any input.
  • Support vector machines: For linear SVMs, poisoning can force a targeted member to become a support vector, enabling membership inference with 100% accuracy through black-box boundary-distance queries.
  • Support vector machines: The SVM analysis applies only to examples on a class’s convex hull, although the paper notes that almost all points lie on the boundary in high dimensions.
  • Support vector machines: For a protruding point, adding an oppositely labeled nearby poison forces the point to become a support vector and yields a larger margin when the point is absent.
  • Assumptions: The theoretical analysis assumes the adversary knows the full training set except for the target’s membership and that the dataset is linearly separable.
Loading 2204.00032v2…