PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Unintended Feature Leakage in Collaborative Learning

Luca Melis, Congzheng Song, Emiliano De Cristofaro, Vitaly Shmatikov

arXiv:1805.04049v3cs.CRcs.AI

TL;DR

Collaborative learning raises the question of what model updates reveal about participants’ training data. The paper develops passive and active membership and property-inference attacks, finding leakage of exact points and unintended subset-specific properties across learning settings. It also finds that tested defenses are ineffective, while attack success depends on whether the model internally separates the target features.

  • Problem

    Collaborative learning exchanges model updates based on private training data, creating the question of what an adversary can infer from those updates.

  • Method

    The paper evaluates passive and active attacks that exploit model-update leakage to infer exact data-point membership and properties of other participants’ training subsets.

  • Results

    The attacks infer exact membership and unintended properties across collaborative and federated learning, including 0.99 precision with perfect recall for a location profile and 0.9 AUC for person presence.

  • Takeaways & Limitations

    Unintended feature leakage exposes collaborative learning to powerful inference attacks, while selective gradient sharing, dimensionality reduction, and dropout do not effectively defend against them.

  • Takeaways & Limitations

    Inference can fail when the model does not internally separate features associated with the target property, and some identities were undetected under multi-party model averaging.

Abstract

from arXiv · show

Collaborative machine learning and related techniques such as federated learning allow multiple participants, each with his own training dataset, to build a joint model by training locally and periodically exchanging model updates. We demonstrate that these updates leak unintended information about participants' training data and develop passive and active inference attacks to exploit this leakage. First, we show that an adversarial participant can infer the presence of exact data points -- for example, specific locations -- in others' training data (i.e., membership inference). Then, we show how this adversary can infer properties that hold only for a subset of the training data and are independent of the properties that the joint model aims to capture. For example, he can infer when a specific person first appears in the photos used to train a binary gender classifier. We evaluate our attacks on a variety of tasks, datasets, and learning configurations, analyze their limitations, and discuss possible defenses.

1 Introduction

Collaborative learning exchanges model information without pooling participants’ data, but those updates can leak unintended training-data features. The paper develops passive and active attacks that infer membership and subset-specific properties, including under federated averaging, while tested defenses remain ineffective.

  • Collaborative learning lets participants train locally and periodically exchange parameters, updates, or partially constructed models to build a joint model.
  • An adversarial participant can perform membership inference, identifying whether an exact location profile trained a gender classifier with 0.99 precision and perfect recall.
  • Passive and active property-inference attacks recover subset-specific or task-independent properties, such as when a person first appears in gender-classifier photos.
  • Model updates leak unintended features because deep-learning models internally represent features independent of the task being learned.
  • On FaceScrub, person presence reached 0.9 AUC, while Yelp attacks inferred doctor specialty perfectly and identified authors whose reviews formed less than one-third of a batch.
  • Attacks also succeed with federated model averaging, and sharing fewer gradients, reducing input dimensionality, or dropout does not effectively thwart them.

2 Background

The paper situates collaborative learning within supervised deep-learning training, where participants optimize local models and exchange updates or averaged models. Its convergence depends heavily on task and hyperparameters, while privacy-oriented designs limit what is exposed but do not eliminate leakage concerns.

  • 2.1 Machine learning (ML): Supervised learning trains models on labeled images or word sequences by optimizing a loss that penalizes incorrect class predictions.
  • 2.1 Machine learning (ML): Stochastic gradient descent iteratively updates parameters from small training batches, although the paper’s inference methodology is not specific to SGD.
  • 2.1 Machine learning (ML): Deep-learning models use layered nonlinear mappings with trainable weight matrices, whose topology depends on the task and affects accuracy.
  • 2.2 Collaborative learning: Collaborative training partitions data across participants, who compute local updates and send them to a parameter server for aggregation into global parameters.
  • 2.2 Collaborative learning: Record-level differential privacy bounds membership inference but not group property inference, whereas participant-level privacy bounds all considered attacks.
  • 2.2 Collaborative learning: Federated model averaging has participants perform local SGD on entire datasets before the server computes a weighted average of their resulting models.
  • 2.2 Collaborative learning: Convergence of both collaborative approaches depends heavily on the learning task and hyperparameters, including participant count and batch size.

3 Reasoning about Privacy in Machine Learning

The paper distinguishes legitimate class-level information revealed by useful models from privacy-relevant leakage about specific training inputs or subsets. It argues that collaborative-learning attacks expose unintended properties beyond class characteristics, while prior attacks and representative-generation methods have narrower scope.

  • Class representatives and privacy: Useful classifiers reveal features characteristic of their classes, but this does not by itself establish leakage of specific training inputs.Model inversion and GAN-generated representatives can resemble training data when all class members are similar, but otherwise need not reconstruct actual inputs.
  • Scope of prior attacks: The active attack in prior work assumes that one participant owns the entire training corpus for a class, unlike the paper’s setting where class data are distributed across participants.The paper presents distributed class data as a more realistic collaborative-learning scenario.
  • Membership inference: Membership inference asks whether an exact data point was included in training, creating an immediate privacy threat when the dataset itself is sensitive.The paper frames this as distinct from merely learning general information about a class.
  • Unintended properties: Prior property-inference work targets properties characterizing an entire class, whereas this paper targets properties true only of subsets of training inputs.The paper emphasizes properties independent of the class’s characteristic features, such as eyewear in photos used for gender classification.
  • Unintended properties: Collaborative-learning participants contribute batches, enabling attacks that detect batch-level properties, identify when properties emerge, and infer properties of an entire participant dataset.Examples include detecting when a person first appears in photos, when a participant begins visiting a type of doctor, or who authored sentiment-analysis texts.

4 Inference Attacks

The paper models an adversarial participant who analyzes periodic collaborative-training updates to infer information about another participant’s data. It develops membership and property inference attacks using update leakage, auxiliary data, and passive or active strategies.

  • Threat model: The adversary analyzes periodic joint-model updates to infer another participant’s training data, with observed updates depending on the number of participants and training procedure.For K = 2 synchronized SGD, the adversary observes a target’s single-batch gradients; for K > 2, updates aggregate other participants’ gradients.
  • Attack requirements: Property inference requires auxiliary data labeled with the property, and active attacks additionally require labels for the model’s main task.The auxiliary examples are used to construct labeled gradient data for training the property classifier.
  • Leakage mechanisms: Embedding-layer gradients reveal which discrete tokens occur in participants’ batches because only embeddings for present words or locations receive nonzero updates.The embedding matrix maps vocabulary items to vectors, and gradients for absent items are zero.
  • Membership inference: Membership inference tests whether a record’s token set appears as a subset of any observed batch vocabulary.The attacker collects vocabularies across training rounds and declares membership when the record’s words are included in at least one vocabulary.
  • Passive property inference: Passive property inference trains a binary batch classifier from gradients generated on auxiliary data with and without the target property.The classifier can score each observed batch, and averaging scores across iterations supports inference about the target’s entire dataset.
  • Active property inference: Active property inference uses multi-task learning to make the collaborative model learn representations that separate data with and without a property, producing separable gradients.The adversary uploads updates based on a joint loss combining the main task and property classification.

5 Datasets and model architectures

The experiments span text, location, essay, and face datasets paired with classification or property-inference tasks. The models use embedding-based recurrent or convolutional architectures for text and location data, and CNNs for image data.

  • Datasets and tasks: The experiments use Yelp, FourSquare, CSI, LFW, FaceScrub, and PIPA datasets across language, location, authorship, and image-based tasks.Table 1 reports the datasets and collaborative-learning tasks used for evaluation.
  • Model architectures: LFW, FaceScrub, and PIPA use convolutional image models, with task-specific datasets and CNN architectures.LFW contains 13,233 images of 5,749 individuals; FaceScrub contributes 18,809 selected images; PIPA contributes 18,000 selected images.
  • Model architectures: Yelp models use recurrent neural networks with 100-dimensional word embeddings, a gated recurrent unit, and a fully connected classification layer.The reported SGD learning rate for both Yelp datasets is 0.05.
  • Datasets and tasks: FourSquare contains 528,878 check-ins from 15,548 New York City users and supports gender classification experiments.The selected users checked in at least 10 different locations and had known gender labels.
  • Datasets and tasks: CSI contains 1,412 student-written reviews labeled with class and author or document attributes, including gender, age, region, timestamp, and sentiment.The reviews are used for classification and inference of properties associated with authors or documents.
  • Model architectures: FourSquare and CSI models embed locations or word tokens into 320-dimensional vectors before applying three spatial convolutional layers and pooling.The architecture uses variable kernel windows and sequence padding for the embedded inputs.

6 Two-Party Experiments

The two-party experiments show that collaborative model updates reveal membership and unintended properties, including dynamic properties, across image and text tasks. Attacks remain effective when properties are unrelated to the main task and when models generalize well.

  • 6.2 Single-batch property inference: Single-batch property inference reaches perfect AUC for race: black during gender classification, despite a Pearson correlation of -0.005 between the labels.It also achieves almost perfect AUC for sunglasses during race classification and performs well for glasses during Asian-race classification.
  • 6.2 Single-batch property inference: Lower-layer features group inputs by unintended properties while higher-layer features group them by class, exposing information beyond class-characteristic features.The t-SNE visualization uses shape for gender and color for the race: black property versus its absence.
  • 6.2 Single-batch property inference: Yelp-health specialty inference achieves almost perfect AUC, while Yelp-author exceeds 0.95 AUC even when the property appears in only 0.3 of a batch.The specialty attack uses embedding-gradient BoWs; author inference succeeds although the main task is generic sentiment analysis and more than two thirds of each batch are from other authors.
  • 6.3 Inferring when a property occurs: Dynamic property inference makes property appearance and disappearance visible across iterations, and the attacks successfully infer dataset changes during collaborative training.For identities appearing in the data, classifier scores peak during their presence; an identity that never appears receives very low scores.
  • 6.4 Inference against well-generalized models: The attack reaches 0.98 AUC after only 2 epochs on CSI while the main sentiment model remains well generalized and improves on test accuracy with training.Increasing the active-attack parameter α also increases AUC and produces tighter separation of property-bearing data.

7 Multi-Party Experiments

Multi-party experiments show that passive attacks can infer properties from aggregated updates across several participants, although performance generally degrades as aggregation increases and depends on the property. Model averaging also reveals whether a target face is present and can detect when that face-bearing participant joins.

  • 7.1 Synchronized SGD: Some Yelp authors remain identifiable with many participants because distinctive word combinations persist in embedding-layer gradients.In multi-party learning, these gradients leak the batch bag-of-words of all honest participants.
  • 7.2 Model averaging: When a target face is present, the adversary assigns much higher scores to aggregated updates than when the face is absent, with three or five participants.The experiment used FaceScrub identities with 80% of one honest participant’s data depicting the target face.
  • 7.2 Model averaging: Attack success depends on the inferred property and data distribution; some FaceScrub identities were not detected.Face IDs 2 and 4 failed under the same training procedure that detected IDs 1 and 3.
  • 7.2 Model averaging: The attack can identify when a participant joins with property-bearing data by detecting a change in inference performance after that participant enters training.The target joined at round 250 with photos depicting Face ID 1; Figure 10 reports the resulting AUC trajectory.

8 Defenses

The evaluated defenses reduce leakage inconsistently: sharing fewer gradients lowers communication but leaves substantial inference accuracy, while dropout can strengthen attacks. Vocabulary restrictions and participant-level differential privacy impose important utility or scalability constraints.

  • 8.1 Sharing fewer gradients: Sharing only 10% of gradients still yields 0.84 AUC for author-region inference, versus 0.93 AUC when all updates are shared.The result comes from single-batch inference against a sentiment classifier trained on the CSI Corpus.
  • 8.2 Restricting the input space: Restricting inputs to frequent locations can eliminate most FourSquare training data, limiting this defense’s general applicability.The top-location profile uniqueness is illustrated in Figure 11.
  • 8.3 Dropout: Dropout increases attack strength while slightly decreasing joint-model accuracy as its probability increases.The evaluation infers the top region, Antwerpen, on the CSI Corpus for different dropout probabilities.
  • 8.4 Differential privacy: Record-level differential privacy bounds membership inference but does not prevent property inference, while participant-level privacy failed to produce a converged LFW gender classifier in tests of up to 30 participants.The authors attribute non-convergence to the noise magnitude required by the moments-accountant bound.
  • 8.4 Differential privacy: Participant-level differential privacy is expected to provide reasonable accuracy mainly in settings with at least thousands of participants.The paper calls for further work on adapting it to smaller deployments while preventing these attacks.

9 Limitations of the attacks

The attacks require suitable auxiliary data and are most effective in relatively small collaborative groups, while some properties remain undetectable or unattributable. Nevertheless, leakage can constitute a privacy breach even without identifying the responsible participant.

  • 9.1 Auxiliary data: Targeted attacks require specialized auxiliary data, such as other photos of a person or enough texts known to be written by an author.Generic properties may require only broadly available labeled datasets.
  • 9.2 Number of participants: Experiments used 2–30 participants, whereas some federated applications involve thousands or millions, and attack performance drops significantly as participation increases.This bounds how directly the reported results transfer to much larger deployments.
  • 9.3 Undetectable properties: Inference can fail when the model does not internally separate features associated with the target property; some face identities were undetected in model-averaging experiments.Thus, detectability depends on the property’s representation in the learned model.
  • 9.4 Attribution of inferred properties: In multi-party settings, aggregated updates may reveal a property without identifying which participant’s data caused the leakage.Attribution generally requires auxiliary information specific to the leakage, such as participants’ social networks.
  • 9.4 Attribution of inferred properties: If the property first appears after a new participant joins, the adversary may attribute the associated photos to that participant.The paper presents this as an attribution example based on auxiliary timing information.
  • 9.4 Attribution of inferred properties: Leaking medical conditions, locations, images, or known authorship is a privacy breach even without participant-level attribution.The paper notes that such leakage can reveal addresses, religion, sexual orientation, relationships, or intimate connections.

10 Related Work

Related work covers privacy-preserving distributed learning, membership inference, class-feature extraction, input reconstruction, memorization, and property inference. This paper distinguishes its attacks by targeting unintended properties in collaborative training rather than only class-wide features or post-training model release.

  • Privacy-preserving distributed learning: Prior privacy-preserving approaches include differentially private transfer learning, trusted aggregation, knowledge transfer, and secure multi-party computation.These methods differ in trust assumptions and whether training occurs over centralized, encrypted, or distributed data.
  • Membership inference: Prior membership-inference studies attack aggregate statistics, black-box models, generative models, and collaborative learning using model outputs or updates.Truex et al. specifically report improved membership inference when an adversary accesses individual updates from honest participants.
  • Other attacks on machine learning models: Other attacks infer features characterizing an entire class or construct class representatives, unlike the paper’s focus on properties specific to subsets of training data.The distinction generally excludes pathological overfitting cases where the training sample is the whole class membership.
  • Other attacks on machine learning models: Gradient-based reconstruction work partially recovers training inputs under a single-input batch assumption, with evaluation limited to visually homogeneous MNIST classes.The cited work does not establish whether individual training images can be distinguished from other images in the same class.
  • Other attacks on machine learning models: Memorization and model-stealing studies show that explicitly incorporated or unintentionally memorized training data can be extracted through black-box access or model stealing.Examples include engineered memorization and sequences artificially inserted into text.
  • Other attacks on machine learning models: Concurrently developed property-inference attacks target post-training white-box models and properties that may correlate with the main task, using simpler datasets and tasks.The cited evaluation includes MNIST, U.S. Census tabular data, and hardware performance counters.

11 Conclusion

The paper evaluates inference attacks that recover both exact data-point membership and unintended properties from collaborative-learning updates. It finds that unintended-feature leakage enables powerful attacks, while several defenses are ineffective and participant-level differential privacy remains unresolved for few-participant settings.

  • The attacks infer exact membership and properties of training-data subsets that are independent of the joint model’s target features.
  • Unintended features can leak through model updates because deep models internally recognize features unrelated to their training tasks.Active attacks can induce the joint model to learn attacker-chosen features without significantly affecting performance.
  • Selective gradient sharing, dimensionality reduction, and dropout do not effectively defend against the demonstrated leakage.The paper suggests task-relevant feature learning and detection of active attacks as directions for stronger defenses.
  • Participant-level differential privacy remains an open question when collaborative learning involves relatively few participants.
Loading 1805.04049v3…