PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Sampling for Active Learning

Christoph Mayer, Radu Timofte

arXiv:1808.06671v2cs.LGcs.CVstat.ML

TL;DR

Prior GAN-based active-learning approaches were limited to simple binary tasks, while ASAL generates uncertain samples and matches them to real pool examples for labeling. ASAL outperforms random sampling on eight of ten benchmarks and is intended for large-data interactive settings where preprocessing is acceptable.

  • Problem

    Prior GAN-based active-learning methods are designed for small, simple binary tasks and use SVMs rather than CNNs.

    Source: §1 · p. 1

  • Method

    ASAL generates uncertain samples and retrieves similar real samples from the pool for labeling and training.

    Source: §3 · p. 2

  • Results

    ASAL outperforms random sampling on eight out of ten benchmarks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    ASAL is suitable for interactive active learning when preprocessing is acceptable but short sampling times are needed.

    Source: §6 · p. 8

    ASAL's success depends on dataset structure and size, which affect generated-image and matching quality.

    Source: §5.4 · p. 7, §6 · p. 8

Abstract

from arXiv · show

This paper proposes asal, a new GAN based active learning method that generates high entropy samples. Instead of directly annotating the synthetic samples, ASAL searches similar samples from the pool and includes them for training. Hence, the quality of new samples is high and annotations are reliable. To the best of our knowledge, ASAL is the first GAN based AL method applicable to multi-class problems that outperforms random sample selection. Another benefit of ASAL is its small run-time complexity (sub-linear) compared to traditional uncertainty sampling (linear). We present a comprehensive set of experiments on multiple traditional data sets and show that ASAL outperforms similar methods and clearly exceeds the established baseline (random sampling). In the discussion section we analyze in which situations ASAL performs best and why it is sometimes hard to outperform random sample selection.

1. Introduction

Active learning seeks strong models with few labels, using pool-based selection to approach fully supervised performance while exceeding random sampling. The paper introduces ASAL for multi-class GAN-based active learning and evaluates it on full image datasets.

1. Introduction

  • The goal is to approach fully supervised performance with as few labels as possible while exceeding random sampling.Pool-based learning is emphasized because unlabeled pools are often available and can be fully annotated to establish an upper bound.Source: §1 · p. 1
  • Prior GAN-based active-learning methods target small, simple binary-classification tasks and use SVMs rather than CNNs.GAAL also fails to outperform random sample selection.Source: §1 · p. 1
  • ASAL is, to the authors’ knowledge, the first pool-based GAN active-learning method designed for multi-class problems.It is evaluated on full image datasets rather than the simpler classification challenges used by related methods.Source: §1 · p. 1
  • ASAL has sub-linear runtime complexity despite searching the full pool in every active-learning cycle.Source: §1 · p. 1

2. Related Work

Related GAN-based active-learning methods have limitations in task scope, annotation, or computational cost. ASAL reuses sample generation but matches synthetic samples to real pool samples for labeling.

2. Related Work

  • Hashing accelerates minimum-distance sampling only with fixed feature maps; changing CNN features makes the methods linear-time again.These methods search for points near a hyperplane in a fixed feature space.Source: §2 · p. 2
  • GAAL generates uncertain samples but annotates them directly, and it underperforms random sampling on two-class MNIST and CIFAR-10 subsets.The passage attributes this to sampling bias and difficult annotation of sometimes poor-quality synthetic images.Source: §2 · p. 2
  • ADBA labels generated image sequences along a line crossing the decision boundary, but it does not compare with image-space uncertainty sampling.Computing latent representations is costly, and lines with nearby class crossings may be hard and more costly to annotate than one image.Source: §2 · p. 2
  • ASAL reuses GAN sample generation but uses entropy for multi-class uncertainty and retrieves similar real pool samples for annotation.It uses CNN classifiers and precomputed feature maps for sub-linear nearest-neighbor matching.Source: §2 · p. 2

3. Proposed Adversarial Sampling for Active Learning

ASAL generates high-entropy synthetic samples, then retrieves and labels similar real samples from the pool rather than annotating synthetic images. It combines GAN-based sample generation with feature-space matching designed to keep pool search efficient.

3. Proposed Adversarial Sampling for Active Learning

  • ASAL generates high-entropy samples, matches them to real pool samples in feature space, and has an oracle label those matches for the next training cycle.This avoids directly annotating synthetic images.Source: §3 · p. 2, §3.2 · p. 4

3.1. Adversarial Sample Generation using GANs

  • The method generates samples by maximizing classifier entropy over images produced by a trained GAN, optimizing the latent variable with gradient descent.The generator maps latent variables to images, and the resulting generation step has O(1) runtime with respect to pool size, versus O(n) for traditional uncertainty sampling.Source: §3.1 · pp. 2–3

3.2. Sample Matching

  • The matching features include CNN classifier features, autoencoder encoder features, and discriminator features, but classifier features must be recomputed after weight updates.Classifier-based matching therefore requires linear-time recomputation, while the other proposed extractors are independent of current classifier weights and can be computed once in preprocessing.Source: §3.2 · p. 3
  • ASAL’s feature space should represent real and synthetic samples meaningfully, and the pool must be sufficiently dense for close matches to be useful.Nearby feature representations are more likely to have similar entropy, but sparse pools may still yield distant matches.Source: §3.2 · p. 3
  • Sample matching retrieves the pool sample closest to a generated sample under a feature-space distance, using Euclidean distance in the proposed formulation.Pool features can be precomputed, and k-d trees provide sub-linear nearest-neighbor search; the runtime also depends on feature dimension.Source: §3.2 · pp. 3–4
  • PCA reduces feature dimensionality for matching, and the pool feature set can be precomputed before active learning begins.The experiments compress feature representations to a common dimensionality to keep nearest-neighbor runtimes similar across extractors.Source: §3.2 · p. 4

4. Experiments

The experiments cover five datasets and several binary and multi-class benchmarks, including four CelebA attribute tasks. They compare ASAL variants with maximum-entropy sampling, random sampling, and, in selected settings, Core-set AL.

4.1. Datasets

  • Experiments evaluate ASAL on MNIST, CIFAR-10, SVHN, LSUN Scenes, and four CelebA face-attribute benchmarks.The CelebA tasks use Blond Hair, Wearing Hat, Bangs, and Eyeglasses labels; separate binary benchmarks support comparison with GAAL and ADBA.Source: §4.1 · p. 4

4.2. Experimental Settings

  • The evaluation uses maximum-entropy sampling and fully supervised training as upper-bound references, random sampling as the baseline, and Core-set AL in selected MNIST and SVHN experiments.ASAL is tested with Gray/RGB, Autoencoder, and Discriminator features, with classifier-feature matching included in some settings.Source: §4.2 · p. 5
  • Most experiments use five random seeds and report mean accuracy, while CIFAR-10 ten-class, SVHN, and LSUN use three seeds.The reported shaded regions span the maximum and minimum values across seeds.Source: §4.2 · p. 5
  • The experiments reduce feature dimensionality to 50 with PCA and use 100 Adam gradient steps to maximize entropy with respect to latent variables.Models are trained from scratch in each active-learning cycle; data augmentation is used only for LSUN.Source: §4.2 · p. 5

5. Results

ASAL outperforms random sampling on eight of ten benchmarks, though it is less reliable on CIFAR-10 ten-class and CelebA Eyeglasses. The results also show how feature choice, dataset size, and preprocessing affect performance and runtime.

5.1. Linear Models

  • On MNIST two-class, all ASAL strategies reach 98.5% accuracy with 200 labelled samples, while random sampling requires 500; on CIFAR-10 two-class, autoencoder features match maximum-entropy accuracy with 500 samples.Random sampling requires approximately twice as many samples for comparable CIFAR-10 two-class performance.Source: §5.1 · p. 5

5.2. Convolutional Neural Networks

  • ASAL outperforms random sampling on eight of ten benchmarks, performs similarly to random on CIFAR-10 ten classes, and is only marginally better on CelebA Eyeglasses, equaling random beyond 750 labelled samples.Source: §5.2 · pp. 5–6

5.3. Comparison between ASAL, GAAL and ADBA

  • GAAL’s accuracy drops toward random after 350 samples, while ADBA needs many more annotations and is limited to binary classification with linear latent-space models.On CIFAR-10 two-class, GAAL performs worse than random and ASAL; annotating one ADBA line counts as one annotation but labels eleven synthetic images.Source: §5.3 · pp. 6–7

5.4. Discussion

  • ASAL’s success depends on dataset density and feature choice: FCLS helps on Eyeglasses, while small datasets can hinder matching; on LSUN, meaningful features suffice despite lower-quality generated samples.The discussion links Eyeglasses errors to rare examples and a feature extractor that emphasizes many attributes, while LSUN has about 1M samples per class.Source: §5.4 · p. 7

5.5. Timings

  • At 16M samples, ASAL selects in approximately 44s versus maximum-entropy sampling’s approximately 53min, but ASAL requires preprocessing.ASAL stores 50 float features per image, enabling feature storage and nearest-neighbor fitting at large dataset sizes.Source: §5.5 · p. 8, Figure 6 · p. 8
  • Learning-Loss and Core-Set selection have linear runtime, and sample selection can take longer than classifier training when fewer than 15k samples are used.Table 2 reports per-cycle runtime and complexity on LSUN, including I/O time.Source: §5.5 · p. 8, Table 2 · p. 8

6. Conclusion

ASAL combines sample synthesis with matching to select pool samples, achieving sub-linear runtime. It beats random sampling on eight of ten benchmarks, with success depending on dataset structure and size and the quality of generated images and matches.

6. Conclusion

  • ASAL outperforms random sampling on eight of ten benchmarks, with success depending on dataset structure, size, and the quality of generated images and matches.The authors report that sub-linear runtime compensates for preprocessing on large datasets and suggest ASAL for interactive active learning when preprocessing is acceptable but sampling time must be short.Source: §6 · p. 8
  • The proposed pool-based method combines sample synthesis and matching to achieve sub-linear runtime.Source: §6 · p. 8
  • The authors propose evaluating ASAL with other active-learning scores and studying VGG or AlexNet features for natural images.Source: §6 · p. 8

A. Model Architectures and Experimental Details

The appendix details dataset-specific architectures and training settings for the experiments, including GAN and classifier configurations.

A. Model Architectures and Experimental Details

  • The appendix specifies autoencoder and classifier, generator, and discriminator architectures, and lists GAN training iterations by dataset.Where possible, the authors follow the accompanying papers' training parameters, including and for GANs and for the CIFAR-10 ten-class classifier.Source: A. Model Architectures and… · p. 10, Table 3 · p. 10

A.1. MNIST - two classes

  • For binary MNIST, the classifier, WGAN-GP, and autoencoder are trained on digits 5 and 7, with the autoencoder channels reduced threefold.The classifier uses 10 epochs, batch size 10, and learning rate 0.001; the autoencoder uses 40 epochs, batch size 100, and learning rate 0.001.Source: §A.1 · p. 10

A.2. CIFAR-10 - two classes

  • For binary CIFAR-10, the classifier, WGAN-GP, and autoencoder use only the two target classes; DCGAN exhibits mode collapse beyond 100k training iterations.The classifier is trained for 10 epochs and the autoencoder for 100 epochs.Source: §A.2 · p. 10

A.3. MNIST - ten classes

  • For ten-class MNIST, the experiments use LeNet and train the WGAN-GP and autoencoder for 40 epochs.The classifier is trained for 10 epochs; the autoencoder uses batch size 100 and learning rate 0.001.Source: §A.3 · p. 10

A.4. CIFAR-10 - ten classes

  • For ten-class CIFAR-10, the authors use All-CNN, report an 11.8% classification error, and begin with 100 randomly selected images per class.They add 1,000 samples per active-learning cycle to a 30k budget and compare layered and residual GAN structures, with or without a soft consistency term.Source: §A.4 · p. 10

A.5. CelebA

  • For CelebA, classification starts with 100 labeled samples distributed according to the data, and the autoencoder is trained for 100 epochs.The classifier uses Adam with learning rate 0.001 and batch size 50 for 30 epochs; the autoencoder uses batch size 64 and learning rate 0.0001.Source: §A.5 · p. 10

A.6. SVHN

  • For classification, active learning starts with 1,000 labeled samples, with per-class counts matching the data distribution; the autoencoder is trained for 100k iterations using batch size 64 and Adam at learning rate 0.0001.Source: §A.6 · p. 10

A.7. LSUN

  • For LSUN, the classifier uses Conv-Small CNN with a modified first layer, and the autoencoder is trained for 100k iterations.The experiments start with 1,000 labeled samples distributed according to the data; the first convolution uses 5×5 kernels with stride two.Source: §A.7 · p. 11

B. Additional Results: MNIST - two classes

The additional MNIST two-class results examine matched-label agreement and compare ASAL variants. WGAN-GP and gray-value matching show higher agreement, with ASAL-Gray also performing best in related results.

B. Additional Results: MNIST - two classes

B.1. Agreement of Manual Annotations and Matched Labels

  • The agreement experiment uses 1,300 generated samples per GAN, manually annotates them, and retrieves each sample's closest pool match and label.The goal is to compare the matched labels with the generated images' manual class annotations.Source: §B.1 · p. 11
  • To score the generated-sample test set, the authors calculate accuracy independently for each class and then average the class scores.They use this procedure because the generated samples may have uneven label frequencies, unlike an almost evenly distributed test set.Source: §B.1 · p. 11
  • WGAN-GP yields higher agreement between manual annotations and matched labels than DCGAN, while gray-value matching achieves the highest agreement.The results also show best performance for ASAL-Gray in the related additional results.Source: §B.1 · p. 11

C. Additional Results: MNIST - ten classes

Additional MNIST ten-class results compare ASAL variants and classical methods. ASAL combined with uncertainty sampling retrieves higher-entropy samples than random sampling at a fixed subset size, and ASAL achieves the best classification accuracy while approaching uncertainty sampling faster.

C. Additional Results: MNIST - ten classes

D. Additional Results: CIFAR - two classes

The CIFAR-10 additional results report accuracy, label-distribution, and selected-sample entropy figures for both two-class and ten-class settings.

D. Additional Results: CIFAR - two classes

  • For CIFAR-10 two-class experiments, Figure 19 presents test accuracy across GANs, while Figures 14, 17, and 18 show label distributions and Figure 20 shows selected-sample entropy.The validation set contains equal numbers of each class, so the figures do not mark the true label distribution with a tick.Source: D. Additional Results: CIFAR - two… · p. 11

E. Additional Results: CIFAR - ten classes

  • For CIFAR-10 ten-class experiments, Figure 21 presents test accuracy across GANs, while Figures 23 and 24 show label distributions and Figure 22 shows selected-sample entropy.The validation set contains equal numbers of each class, so the figures do not mark the true label distribution with a tick.Source: E. Additional Results: CIFAR - ten… · p. 11

E.1. Discussion of results on CIFAR-10 - ten classes

On CIFAR-10 ten-class data, maximum-entropy sampling favors cats, whereas most auto-encoder ASAL setups include them less often. The authors relate generated class frequencies to entropy near the decision boundary.

E.1. Discussion of results on CIFAR-10 - ten classes

  • Figure 24 shows cats are most frequent under maximum-entropy sampling but among the least frequent in ASAL with auto-encoder features.FCLS produces a similar, less distinctive distribution.Source: §E.1 · pp. 11–12
  • The authors conclude that the generator moves images from various classes near the decision boundary, while producing other classes instead of low-entropy trucks and automobiles.They report that trucks and automobiles are among the least frequent classes across experiments.Source: §E.1 · p. 12

F. Matching Strategy Visualization

The matching-strategy visualizations pair generated images with pool samples. They show visually close matches in CIFAR-10 two-class data, but less reliable image quality and class matching in the ten-class setting.

F. Matching Strategy Visualization

  • Figure 25 compares generated digits with their closest pool samples using different features, and Figure 26 shows generated and matched MNIST ten-class images from WGAN-GP.Figure 25 labels each image with its class and image id.Source: Figure 25 · p. 27, Figure 26 · p. 27
  • For CIFAR-10 two-class data, WGAN-GP generated images have reasonable quality, and all matching strategies retrieve visually close samples or samples from the same class.Figure 27 shows generated and matched image rows.Source: Figure 27 · p. 28
  • For CIFAR-10 ten-class data, generated images are mostly moderate quality, and some closest pool samples show mismatched classes; the generated examples are therefore left without manual annotations.Figure 28 includes visually similar matches labeled as airplane or ship where horse would be appropriate; n.a. marks missing manual annotations.Source: F. Matching Strategy Visualization · p. 12, Figure 28 · p. 28
Loading 1808.06671v2…