PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Confuse: Generating Training Time Adversarial Data with Auto-Encoder

Ji Feng, Qi-Zhi Cai, Zhi-Hua Zhou

arXiv:1905.09027v1cs.LGstat.ML

TL;DR

The paper asks how bounded perturbations to training data can make a corresponding classifier perform poorly on clean test samples. It trains an auto-encoder-like generator against an imaginary victim classifier and decouples their updates for stability. Experiments on image data report effectiveness, including when a different classifier is used.

  • Problem

    The paper asks how bounded, transferable perturbations to training data can make the corresponding classifier fail on clean test samples.

    Source: §1 · p. 2

  • Method

    An auto-encoder watches an imaginary victim classifier and generates bounded training-data perturbations, with decoupled alternating updates for stability.

    Source: §6 · p. 14

  • Results

    Experiments on image data confirmed the method's effectiveness, including when a different classifier is used.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    The reported image-data results indicate that the perturbations can remain effective beyond the classifier used in the attack setup.

    Source: §6 · p. 14

    Theoretical analysis and further optimization-procedure improvements are left for future work.

    Source: §6 · p. 14

Abstract

from arXiv · show

In this work, we consider one challenging training time attack by modifying training data with bounded perturbation, hoping to manipulate the behavior (both targeted or non-targeted) of any corresponding trained classifier during test time when facing clean samples. To achieve this, we proposed to use an auto-encoder-like network to generate the pertubation on the training data paired with one differentiable system acting as the imaginary victim classifier. The perturbation generator will learn to update its weights by watching the training procedure of the imaginary classifier in order to produce the most harmful and imperceivable noise which in turn will lead the lowest generalization power for the victim classifier. This can be formulated into a non-linear equality constrained optimization problem. Unlike GANs, solving such problem is computationally challenging, we then proposed a simple yet effective procedure to decouple the alternating updates for the two networks for stability. The method proposed in this paper can be easily extended to the label specific setting where the attacker can manipulate the predictions of the victim classifiers according to some predefined rules rather than only making wrong predictions. Experiments on various datasets including CIFAR-10 and a reduced version of ImageNet confirmed the effectiveness of the proposed method and empirical results showed that, such bounded perturbation have good transferability regardless of which classifier the victim is actually using on image data.

1. Introduction

The paper studies bounded perturbations to training data that seek to make the resulting classifier fail on clean test samples. It proposes an encoder-decoder noise generator trained against an imaginary victim classifier, with decoupled updates for stability.

1. Introduction

  • The encoder-decoder takes clean samples as input, generates bounded noises in the same sample space, and adds them to training data while an imaginary classifier acts as the victim.The generator updates its weights while observing the victim's training procedure.Source: §1 · p. 2
  • The method formulates perturbation generation as a nonlinear equality-constrained optimization problem; decoupled pseudo-updates address instability in alternating training and improve practical stability.The decoupling is likened to stabilization techniques used in reinforcement learning.Source: §1 · p. 2
  • Unlike poisoning settings that change only a few examples, the paper assumes full control of training data, which it describes as realistic for internal-data release.The motivating scenario is releasing data for peer assessment or academic research while preventing a receiver from building a strong model on real test data.Source: §1 · p. 2
  • The framework extends to label-specific attacks that direct an input pattern toward a predefined class rather than merely seeking an incorrect prediction.This extension is presented as another contribution of the work.Source: §1 · p. 2
  • Experiments report that learned noises remain effective across classifiers with different structures or types, including Random Forest and Support Vector models.The passage describes the noises as robust to models other than the one used during training.Source: §1 · p. 3

2. Related Works

The related-work discussion distinguishes training-time perturbations from data poisoning and testing-time adversarial examples. It situates prior poisoning work across classical and deep models, and notes hypotheses about adversarial-example effectiveness.

2. Related Works

  • Prior poisoning work covers classical models including SVM, Linear Regression, and Naive Bayes, while deep-model poisoning is more challenging; Kon et al. proposed using influence functions to derive adversarial training examples.Sample-specific poisoning aims to manipulate model behavior on particular test samples [11] [12].Source: §2 · p. 3
  • Unlike poisoning approaches focused on changing as few samples as possible, this work seeks small bounded noises and reports scalability to larger datasets with good transferability.The passage contrasts the two approaches by their modification targets: few samples versus bounded noise magnitude.Source: §2 · p. 3
  • Testing-time adversarial examples target an already trained, fixed classifier and involve no two-party game; the paper characterizes its own work as their training-time analogy.The discussion links adversarial examples to security and safety concerns in high-stakes systems.Source: §2 · p. 3
  • Prior work attributes adversarial-example vulnerability in high-dimensional space to neural-network linearity, while explaining training-time noise effectiveness is described as beyond this paper's scope.The authors say they tested several hypotheses about that effectiveness.Source: §2 · p. 4

3. The proposed method

The method seeks bounded, imperceptible training perturbations that reduce a paired classifier’s performance on clean test data, and addresses the resulting nonlinear equality-constrained optimization with decoupled updates. It also introduces a memory-efficient update procedure and uses an encoder-decoder noise generator.

3. The proposed method

  • The perturbation generator is an encoder-decoder network whose final activation is ε·tanh(·), with ε controlling perturbation strength.The generator maps each training sample to an imperceptible noise pattern in the same input space.Source: §3 · p. 6
  • The objective seeks a noise generator whose paired classifier, trained on modified data, performs worst on the clean test set under a nonlinear equality constraint.The perturbations are intended to be imperceptible and to change classifier behavior on clean test samples.Source: §3 · p. 6
  • To stabilize training, the method decouples alternating updates because direct alternation does not converge in practice.It first trains the classifier on adversarial data and collects its update trajectory, then updates the adversarial data and generator using pseudo-updates.Source: §3 · pp. 6–8
  • For memory efficiency, the procedure updates a copied generator and copies its parameters back, merging loops and avoiding storage of the full classifier gradient trajectory.The full trajectory is described as memory-inefficient to store.Source: §3 · p. 8

4. Label Specific Adversaries

Label-specific adversaries aim to steer a classifier toward attacker-predefined predictions rather than merely induce errors. The paper expresses this with a label transformation and adapts the existing procedure with two update changes.

4. Label Specific Adversaries

  • A label-specific adversary seeks predictions set by predefined rules, such as mapping class A to class B rather than class C.This targets a particular wrong prediction instead of any wrong prediction.Source: §4 · p. 8
  • The attacker-defined function η maps a label index to a different label index.The example maps patterns from class A specifically to class B, not class C.Source: §4 · p. 8
  • The label-specific version changes the existing procedure by replacing gradient ascent with gradient descent and η(y) with y.The paper describes the optimization procedure as nearly identical after these substitutions.Source: §4 · pp. 8–9

5. Experiment

Experiments evaluate training-time adversaries across datasets, perturbation settings, classifier types, and label-specific targets. The results report reduced clean-test accuracy, transfer to other classifiers, and a generalization gap associated with fitting training noise.

5. Experiment

  • Experiments evaluated the method on MNIST and CIFAR-10 for multiclass classification and a two-class ImageNet subset (bulbul vs. jellyfish) comprising 2,600 colored images sized 224×224×3.For MNIST and ImageNet, the noise generator uses an encoder-decoder with four 4×4 convolution layers per encoder/decoder and channel counts 16, 32, 64, and 128. CIFAR-10 uses a U-Net.Source: §5 · p. 9

5.1. Performance Evaluation of Training Time Adversary

  • When trained on adversarial datasets, test accuracy dramatically dropped to 0.25±0.04, 54.2 ± 11.19, and 28.77 ± 2.80, showing low accuracies under this condition.Table 1 reports test accuracy (mean±std) for classifiers trained on original clean and adversarial training sets.Source: §5.1 · p. 10, Table 1 · p. 10
  • Varying ε from 0 to 0.3 on MNIST tested its effects on accuracy and visual appearance; the experiments also varied the fraction of adversarial training data.The supplied passages describe these evaluations and their figures but do not provide the plotted outcomes for either variation.Source: §5.1 · p. 11
  • For MNIST, test performance dropped suddenly when the perturbation bound ε exceeded 0.15.The authors conjecture this result is related or somewhat consistent with a robust-error-bound guarantee for ε = 0.10.Source: §5.1 · p. 11

5.2. Evaluation of Transferability

  • Adversarial noise transferred to random forests and SVMs, which produced poor results, and to varied convolutional architectures on CIFAR-10 and ImageNet.The authors report good transferability across the tested model architectures.Source: §5.2 · p. 12

5.3. The Generalization Gap and Linear Hypothesis

  • Training curves showed a generalization gap, and noise-only inputs achieved 95.15%, 93.00%, and 72.98% test accuracy across the reported datasets.The authors say this supports their conjecture that deep models overfit to the training noises; the supplied passages do not map the values to datasets.Source: §5.3 · pp. 12–13

5.4. Label Specific Adversaries

  • The label-specific MNIST experiment shifted desired predictions by one class, from class 1 to 2 through class 9 to 0.Figure 10 summarizes test-set confusion matrices for clean, non-label-specific, and label-specific training data.Source: §5.4 · p. 13, Figure 10 · p. 14
  • Label-specific targeting shifted MNIST predictions toward desired classes with a 79.7 ± 0.38 success rate, while test accuracy was 1.48 ± 0.21.In the non-label-specific setting, test accuracy was 0.25 ± 0.04; the authors describe label-specific targeting as a side-product of the method.Source: §5.4 · p. 14

6. Conclusion

The paper proposes a framework for generating training-time adversarial data with an auto-encoder-like generator that watches an imaginary victim classifier. It uses decoupled alternating updates for stability and reports effectiveness on image data, including transfer to a different classifier.

6. Conclusion

  • The framework generates training-time adversarial data by having an auto-encoder watch and move against an imaginary victim classifier.The perturbation is added to training data to confuse a corresponding classifier.Source: §6 · p. 14
  • A training scheme decouples the alternating updates of the two networks to improve stability.The conclusion describes the scheme as simple yet effective.Source: §6 · p. 14
  • Experiments on image data confirmed the method's effectiveness, including when a different classifier is used.The conclusion notes that this transferability makes the method more useful in a realistic setting.Source: §6 · p. 14
  • The authors identify theoretical analysis and further optimization improvements as future work.They also propose designing adversarially robust classifiers against these attacks.Source: §6 · p. 14

Related papers

Loading 1905.09027v1…