PaperlayerView exact revision on arXiv

Source-linked AI summary

Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning

Ahmed Salem, Apratim Bhattacharya, Michael Backes, Mario Fritz, Yang Zhang

arXiv:1904.01067v2cs.CRcs.LGstat.ML

TL;DR

The paper asks whether prediction changes between two versions of a black-box model leak information about the updating set, a concern for model-owner intellectual property and data privacy. It proposes four encoder-decoder attacks, including CBM-GAN for multi-sample reconstruction, and reports promising performance across the evaluated settings. The study remains limited to classification models updated solely with new data and updating sets of at most 100 samples.

  • Problem

    The paper investigates whether different outputs from two versions of a black-box model queried with the same samples can reveal information about the updating set, potentially compromising intellectual property and data privacy.

  • Method

    The paper proposes four encoder-decoder attacks that use model-output differences to infer labels, label distributions, or samples from the updating set, including CBM-GAN for multi-sample reconstruction.

  • Results

    Quantitative and qualitative results show promising attack performance, with CBM-GAN outperforming all baselines for updating-set reconstruction on MNIST, CIFAR-10, and Insta-NY.

  • Takeaways & Limitations

    Output differences between model versions can be exploited to infer information about an updating set, including reconstructing the set itself.

  • Takeaways & Limitations

    The attacks assume models updated solely with new data and updating sets containing at most 100 samples; larger sets and mixtures of new and old data remain future work.

Abstract

from arXiv · show

Machine learning (ML) has progressed rapidly during the past decade and the major factor that drives such development is the unprecedented large-scale data. As data generation is a continuous process, this leads to ML model owners updating their models frequently with newly-collected data in an online learning scenario. In consequence, if an ML model is queried with the same set of data samples at two different points in time, it will provide different results. In this paper, we investigate whether the change in the output of a black-box ML model before and after being updated can leak information of the dataset used to perform the update, namely the updating set. This constitutes a new attack surface against black-box ML models and such information leakage may compromise the intellectual property and data privacy of the ML model owner. We propose four attacks following an encoder-decoder formulation, which allows inferring diverse information of the updating set. Our new attacks are facilitated by state-of-the-art deep learning techniques. In particular, we propose a hybrid generative model (CBM-GAN) that is based on generative adversarial networks (GANs) but includes a reconstructive loss that allows reconstructing accurate samples. Our experiments show that the proposed attacks achieve strong performance.

1 Introduction

The paper asks whether output changes between two black-box model versions reveal information about the updating set, and develops four encoder-decoder attacks to test this possibility.

  • The study identifies output differences between pre-update and post-update black-box models as a new attack surface that may compromise model-owner intellectual property and data privacy.
  • The four attacks comprise single-sample label inference and reconstruction, plus multi-sample label-distribution estimation and reconstruction.Single-sample attacks address one updating sample, while multi-sample attacks address larger updating sets.
  • The attacks use an encoder-decoder structure in which an MLP encoder processes posterior differences and attack-specific decoders infer updating-set information.
  • Single-Sample Attack Class: 0.96 accuracy was achieved for single-sample label inference on CIFAR-10.
  • Single-Sample Attack Class: 22%, 107.1%, and 114.7% performance gains in mean squared error were reported for single-sample reconstruction on MNIST, CIFAR-10, and Insta-NY, respectively.
  • Multi-Sample Attack Class: The multi-sample attacks estimate label distributions or reconstruct all updating samples, with CBM-GAN outperforming all baselines on MNIST, CIFAR-10, and Insta-NY.CBM-GAN adds a Best Match loss, while clustering selects central generated samples as final reconstructions.

2 Preliminaries

The paper studies online updates for classification models and defines its threat model around black-box querying, local same-distribution data, and model-architecture knowledge.

  • Online learning repeatedly updates trained models with newly generated data rather than training each model from scratch.
  • A classifier maps each input sample to posterior probabilities whose entries sum to one, and an update transforms a trained model M into an updated model M′ using Dupdate.
  • The adversary has black-box access, allowing queries with a probing set and access only to prediction posteriors.The paper characterizes this as the most difficult attack setting for the adversary.
  • The threat model assumes a local dataset from the same distribution as the target training set and knowledge of the target model architecture for constructing a shadow model.
  • The target model is assumed to be updated only with new data, while the paper notes that the architecture and same-distribution-data assumptions can be relaxed.
  • The experiments use MNIST, CIFAR-10, and Insta-NY datasets.

3 General Attack Pipeline

The attack pipeline converts prediction changes caused by model updates into latent representations, then decodes them into information about the updating set using shadow-model-derived training data.

  • The pipeline has three phases: generating posterior differences, encoding them into latent vectors, and decoding those vectors into updating-set information.
  • 3.1 Attack Input: The adversary forms a random probing set from local data, queries both model versions, and computes their posterior-output difference as the attack input.
  • 3.1 Attack Input: The posterior-difference dimension equals the probing-set cardinality multiplied by the number of target-dataset classes.With 100 probes, the dimension is 1,000 for MNIST and CIFAR-10 and 800 for Insta-NY.
  • 3.3 Decoder: Different decoders infer different targets, ranging from sample labels to the updating set itself.
  • 3.4 Shadow Model: Shadow models provide supervised training data by mimicking the target architecture, applying controlled updates, and pairing posterior differences with known updating-set information.

4 Single-sample Attacks

The single-sample attacks use posterior differences between model versions to infer an updating sample’s label or reconstruct the sample itself. Experiments on MNIST, CIFAR-10, and Insta-NY show strong label-inference and reconstruction performance, with autoencoder results serving as an oracle for reconstruction.

  • Single-sample label inference attack: The attack uses posterior differences from two model versions as input to an MLP-based encoder, with attack-specific decoders producing updating-set information.For label inference, the decoder uses fully connected and softmax layers; training data is generated with shadow models and cross-entropy loss.
  • Single-sample label inference attack: 0.97 accuracy on Insta-NY, 0.96 on CIFAR-10, and 0.68 on MNIST demonstrate strong performance for single-sample label inference.The attack significantly outperforms the Random baseline; balanced CIFAR-10 and MNIST baselines are approximately 10%.
  • Single-sample reconstruction attack: The reconstruction attack transfers a pretrained autoencoder decoder into its attack model to reconstruct the single updating sample from posterior differences.An additional fully connected layer links the attack encoder’s latent representation to the autoencoder decoder’s latent dimension, and the decoder is further trained with the attack model.
  • Single-sample reconstruction attack: 0.014 MSE on CIFAR-10 and 0.68 on Insta-NY outperform the corresponding baseline models, while MNIST beats the Label-random baseline by 22%.The autoencoder oracle achieves 0.042, 0.0043, and 0.51 MSE on MNIST, CIFAR-10, and Insta-NY, respectively, and outperforms the attack.
  • Single-sample reconstruction attack: Randomly sampled MNIST reconstructions are visually similar to original updating samples in rotation and shape, while autoencoder outputs represent the upper bound.The visualization compares original images, ASSR reconstructions, and autoencoder reconstructions without cherry-picking samples.

5 Multi-sample Attacks

The multi-sample attacks extend updating-set inference from single samples to label distributions and dataset reconstruction. ALDE substantially outperforms random baselines, while AMSR uses CBM-GAN to generate and cluster samples resembling the updating set, with performance depending on cardinality knowledge and separate clustering.

  • Multi-sample label distribution estimation: ALDE estimates the label distribution of multi-sample updating sets, generalizing single-sample label inference.
  • Multi-sample label distribution estimation: 3 and 2.5 times better accuracy and KL-divergence, respectively, than Random for 100-sample CIFAR-10 updating sets.
  • Multi-sample label distribution estimation: 10-sample shadow updates transfer to 100-sample target updates with significant performance drops, especially for CIFAR-10 KL-divergence.The authors attribute this to insufficient information for generalizing to larger updating sets.
  • Multi-sample reconstruction: AMSR learns the updating-set distribution with GANs and generates multiple samples, while CBM-GAN conditions generation on posterior differences and adds a Best Match reconstruction loss.The approach addresses the difficulty of directly predicting a set of examples.
  • Multi-sample reconstruction: 20%, 22%, and 25% performance gains over Shadow-clustering, and 60.1%, 5.5%, and 14% over Label-average, occur on CIFAR-10, MNIST, and Insta-NY, respectively.The Hungarian algorithm enforces one-to-one matching between reconstructed and ground-truth samples.
  • Limitations and cardinality: Separating CBM-GAN sample generation from clustering remains a limitation, motivating future end-to-end training.The full attack achieves MSEs of 0.036, 0.051, and 0.64 on CIFAR-10, MNIST, and Insta-NY, respectively.

6 Discussion

The discussion evaluates how attack performance changes under different target-model, attacker, and probing conditions, while identifying a scope limitation. Performance is generally robust to model-architecture changes, but depends on updating epochs, data transferability, and probing-set size.

  • Data Transferability: Data transferability reduces performance, yet attacks remain significantly better than baselines across evaluated updating-set sizes.The experiments use Insta-NY as the target dataset and Insta-LA as the shadow dataset.
  • Model Transferability: Model transferability causes less than 2% performance drop across all attacks when shadow and target architectures differ.The shadow model removes one hidden layer and uses half as many neurons in its other hidden layers.
  • Effect of the Probing Set Cardinality: A probing set of 100 samples is suitable because size 10 reduces performance, larger sizes provide at most a 3.5% gain, and computation increases with cardinality.For single-sample label inference and reconstruction, size 10 reduces performance by 9% and 71%, respectively.
  • Target Model’s Training Epochs Before Updating: Less than 2%: attack-performance difference across target models trained for 10, 20, and 50 epochs before updating.The authors attribute this to gradients not decreasing monotonically during training, so information is not necessarily vanishing.
  • Target Model’s Updating Epochs: 25.4%: multi-sample label distribution estimation attack performance improvement when target-model updating epochs increase from 2 to 10.The experiment evaluates updates using 2 through 10 epochs on Insta-NY; Figure 11 uses a lower-is-better metric.
  • Limitations of Our Attacks: The attacks assume updates use only new data and evaluate updating sets with at most 100 samples.Future work will consider larger updating sets containing both new and old data.

7 Possible Defenses

The paper discusses sanitizing posterior differences and differentially private learning as defenses against the proposed attacks, while noting utility and research-scope trade-offs.

  • Adding independent noise to each queried posterior can reduce attack performance by sanitizing the posterior differences used as input.The model owner cannot directly manipulate posterior differences because the adversary controls the probes and their timing.
  • Differentially private learning can reduce attack performance while allowing the model to learn its main tasks and reducing its memory on training data.Applying differential privacy during model updates is presented as a defense against the attacks.
  • Depending on the privacy budget, differentially private learning can significantly reduce model utility.
  • An in-depth exploration of effective defense mechanisms is left for future work.

8 Related Works

The related-work discussion places the attacks alongside membership inference, model inversion, and model stealing, while distinguishing their targets and practical requirements.

  • Membership Inference: Membership inference determines whether a sample belongs to a dataset, including the training set of a black-box ML model.
  • Membership Inference: Dataset reconstruction through membership inference is not scalable because the adversary needs a large-scale dataset containing all samples in the target training set.The paper’s reconstruction attacks are designed specifically for online learning and use a pretrained decoder and CBM-GAN.
  • Model Inversion: Model inversion infers missing input attributes or reconstructs recognizable data representations through interaction with a trained ML model.
  • Model Stealing: Model stealing targets a black-box model’s parameters or functionality, with techniques including equation-solving, decision-tree path-finding, and API replication.Other work also targets model hyperparameters.
  • The paper situates these topics within a broader literature on attacks and defenses for machine learning models.

9 Conclusion

The paper identifies output differences between model versions as an attack surface for inferring update-set information and presents four encoder-decoder attacks. It covers single-sample and multi-sample settings, including reconstruction and label-distribution estimation, with promising quantitative and qualitative results.

  • Output differences between the same model’s pre-update and post-update versions can expose information about the updating set through an encoder-decoder attack structure.The encoder represents output differences, while the decoder generates different types of updating-set information.
  • Two single-sample attacks infer the updating sample’s label and reconstruct the sample using an autoencoder decoder as a pretrained attack decoder.
  • The multi-sample label-distribution attack uses KL-divergence loss to estimate the labels’ distribution in the updating set.
  • CBM-GAN reconstructs multiple updating samples using a Best Match loss that directs its generator toward reconstructing each sample.
  • Quantitative and qualitative results indicate that the proposed attacks achieve promising performance.

A Target Models Architecture

The target-model architectures use specified pooling, fully connected, convolutional, and Softmax layers, with ReLU activations for the MNIST, CIFAR-10, and Location models.

  • The architectures include max-pooling, fully connected, two-dimensional convolutional, and Softmax layers.The passage specifies kernel, hidden-unit, filter, and activation parameters through the corresponding layer notation.
  • ReLU is used as the activation function for all layers in the MNIST, CIFAR-10, and Location models.

B Encoder Architecture

The encoder uses LeakyReLU activation and dropout for regularization, while its latent vector μ serves as the decoder input.

  • μ is the latent vector that serves as the decoder’s input.
  • The encoder uses LeakyReLU as its activation function.
  • Dropout is applied on both encoder layers for regularization.

C Single-sample Label Inference Attack’s Decoder Architecture

The section presents ALI’s decoder and AE encoder architectures for MNIST, CIFAR-10, and Insta-NY, with μAE as the encoder’s latent output and dataset-dependent hyperparameters.

  • The section specifies ALI’s decoder architecture and AE encoder architectures for MNIST, CIFAR-10, and Insta-NY.
  • μAE denotes the latent vector output of the encoder.
  • The encoder hyperparameters include kernel size, filter count, and unit count, with values depending on the target dataset.
  • MNIST and CIFAR-10 encoders use ReLU in all layers, while the Insta-NY decoder uses ELU except in its last layer.
  • Dropout is applied in the described architectures for regularization.

D.2 AE’s Decoder Architecture

The section describes autoencoder and CBM-GAN decoder or generator–discriminator architectures, including transposed convolutions, dataset-specific hyperparameters, normalization, activations, and regularization.

  • AE’s Decoder Architecture: Autoencoder decoder architectures are provided separately for MNIST, CIFAR-10, and Insta-NY.
  • AE’s Decoder Architecture: ConvTranspose2d(k’,s’) denotes a two-dimensional transposed convolution with kernel size k′ × k′ and stride parameter s′.
  • AE’s Decoder Architecture: AE decoder hyperparameters are listed in Table 2, with activation and regularization choices varying by dataset.
  • AE’s Decoder Architecture: MNIST and CIFAR-10 decoders use ReLU except for a final tanh layer, whereas Insta-NY uses ELU except for its final layer.
  • CBM-GAN Architectures: CBM-GAN architectures include separate generators and discriminators for MNIST, CIFAR-10, and Insta-NY.
  • CBM-GAN Architectures: CBM-GAN applies batch normalization after each non-final layer and uses LeakyReLU before a final tanh layer.
Loading 1904.01067v2…