PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Generation From Small Datasets via Batch Statistics Adaptation

Atsuhiro Noguchi, Tatsuya Harada

arXiv:1904.01774v4cs.CV

TL;DR

Generative models usually need large datasets, so the paper transfers a pretrained generator to a small, different-domain dataset by training only hidden-layer scale and shift parameters. The method achieves stable, non-collapsing generation and higher quality than prior methods on very small datasets, while also supporting new domains without disturbing the source domain.

  • Problem

    Generative models have many trainable parameters and typically require large datasets to avoid overfitting, making small-dataset image generation difficult.

  • Method

    The method transfers a pretrained generator by updating only scale and shift parameters in its hidden-layer batch statistics while fixing kernel parameters.

  • Results

    The method generates higher-quality images than existing methods from much smaller datasets, with stable training and no collapse reported in the experiments.

  • Takeaways & Limitations

    The approach supports low-shot learning for generative models and can add new classes or domains without disturbing source-domain performance.

  • Takeaways & Limitations

    Performance is limited when the dataset becomes large; the paper suggests combining adversarial approaches or increasing trainable parameters as possible remedies.

Abstract

from arXiv · show

Thanks to the recent development of deep generative models, it is becoming possible to generate high-quality images with both fidelity and diversity. However, the training of such generative models requires a large dataset. To reduce the amount of data required, we propose a new method for transferring prior knowledge of the pre-trained generator, which is trained with a large dataset, to a small dataset in a different domain. Using such prior knowledge, the model can generate images leveraging some common sense that cannot be acquired from a small dataset. In this work, we propose a novel method focusing on the parameters for batch statistics, scale and shift, of the hidden layers in the generator. By training only these parameters in a supervised manner, we achieved stable training of the generator, and our method can generate higher quality images compared to previous methods without collapsing, even when the dataset is small (~100). Our results show that the diversity of the filters acquired in the pre-trained generator is important for the performance on the target domain. Our method makes it possible to add a new class or domain to a pre-trained generator without disturbing the performance on the original domain.

1. Introduction

Deep generative models can produce high-quality images but typically require large datasets because their many parameters make small-data training prone to overfitting. The paper transfers a pretrained generator’s prior knowledge by updating only batch-statistics scale and shift parameters, enabling generation from fewer than 100 images.

  • Motivation: Generative models commonly require datasets of approximately 10,000 examples to train large networks without overfitting.An SNGAN projection model for 128 × 128 images has 90 M trainable parameters.
  • Motivation: Constructing very large datasets is difficult, motivating methods that reduce the data required for generative-model training.Transfer learning is presented as a way to reuse generally useful weights learned from large datasets.
  • Proposed method: The proposed method updates only the generator’s batch-statistics scale and shift parameters while fixing its convolutional kernels.These parameters are treated as controlling active filters useful for the target domain.
  • Results: Experiments on fewer than 100 images produced higher-quality images than previous methods without collapse and captured target-data semantics.The method is trained using prior knowledge from a pretrained generator.

2. Related works

Prior work established generative modeling and transfer learning, but adapting generative models to very small target datasets remained difficult. This paper positions selective scale-and-shift adaptation as a way to transfer pretrained knowledge while reducing trainable parameters.

  • Selective adaptation: The paper proposes transferring a pretrained generative model by introducing scale and shift parameters into hidden activations and updating only those parameters.The approach is described as applicable to CNN-based generators such as GANs and VAEs.
  • Deep generative models: Deep generative models include VAEs, GANs, and autoregressive models, but their large parameter counts create substantial data requirements.SNGAN projection has 90 M trainable parameters for 128 × 128 images, and training often uses more than 10,000 examples.
  • Deep generative models: Reducing dataset-construction costs is important because generative models can support applications such as classification and abnormality detection through semi-supervised learning.The passage connects small-dataset generative modeling with potential use in these fields.
  • Transfer learning: Transfer learning reuses knowledge learned from large datasets for different domains with insufficient labels or data.Prior generative-model transfer methods improved convergence and performance relative to normal training.
  • Related methods: Earlier scale-and-shift transfer methods included adaptive batch normalization and meta-transfer learning, while prior GAN transfer fine-tuned models on target data.These studies provide precedents for parameter-efficient transfer to limited data.

3. Role of Batch Statistics

Scale and shift parameters modulate convolutional filter activity, making them a compact mechanism for selecting filters during domain adaptation. In ImageNet-trained SNGAN, semantically similar classes use similar parameter patterns, suggesting that pretrained filters encode reusable structure.

  • Filter selection: Applying scale and shift after a convolution is equivalent to modifying the convolution’s filters.The analysis interprets this equivalence as a basis for filter selection.
  • Filter selection: Changing γ adjusts each filter’s activation strength, while changing β adjusts its activation threshold.Large γ_i and β_i make the corresponding neuron easier to activate; small values make it less active.
  • Class conditioning: In class-conditional SNGAN, different γ and β values are applied per class to create class-specific activation distributions.These parameters therefore provide a way to inspect which filters are selected for each class.
  • Class conditioning: T-SNE visualizes γ and β for each layer, with points representing ImageNet classes grouped into categories such as dogs, birds, arthropods, and devices.The figure compares parameter patterns across semantically organized classes.
  • Implication: Semantically similar ImageNet classes use similar scale-and-shift patterns, suggesting that the pretrained generator learns class-dependent filter selection.The paper links this diversity of learned filters to generating images in other domains by learning semantic selection patterns.

4. Method

The method adapts a pretrained CNN generator to a small target dataset by updating only hidden-layer scale and shift parameters while estimating latent vectors for supervised reconstruction. Pixel-level and semantic losses guide adaptation, with regularization and truncated latent sampling addressing overfitting and sparse coverage.

  • Learnable parameters: The method adapts pretrained generators by updating only hidden-layer scale and shift parameters while fixing convolutional kernels.These parameters are introduced per channel in each hidden layer, excluding the final layer.
  • Learnable parameters: Scale and shift control filter activation strength and threshold, allowing adaptation to select filters useful for the target domain.Larger values make the corresponding neuron easier to activate, whereas smaller values reduce activity.
  • Learnable parameters: The adapted feature is computed as G(l) · γ(l) + β(l), where γ(l) and β(l) are initialized to 1 and zero, respectively.G(l) denotes the feature representation at layer l; the parameters are used through batch normalization without adding new statistics parameters.
  • Training: Training jointly estimates a latent vector for every training image and updates generator parameters to minimize pixel-level L1 and semantic perceptual losses.The pipeline compares generated images with all training data, while latent vectors are learned by gradient descent.
  • Training: The training objective regularizes latent vectors toward a standard normal distribution and includes a term that prevents overfitting.Perceptual loss uses features from a trained classifier, specified here as VGG16.
  • Inference: Inference samples z from a truncated normal distribution because latent vectors far from training samples produce poor performance.This sampling strategy is identified as the truncation trick.

5. Experiments

Experiments evaluate transfer to human-face, anime-face, and flower domains using very small datasets, comparing the proposed method with alternative adaptation strategies. The proposed method generally trains stably without collapse, supports consistent interpolation, and achieves better FID and KMMD in general, while source-filter diversity and dataset size affect performance.

  • Comparison methods: With 25 training images, competing approaches showed unstable convergence, mode collapse, meaningless or blurred outputs, or overfitting to training samples.GANs trained from scratch converged slowly; Transfer GAN collapsed; updating all layers overfit; and other partial-update strategies failed to transfer either global or texture information.
  • Sampling: Small truncation thresholds increased fidelity, whereas larger thresholds increased diversity in sampled outputs.The reported truncation thresholds were 0.3 or 0.4 in the comparison results.
  • Proposed method: The proposed method converged stably without collapse and produced more consistent images and smoother interpolation than other methods despite the small training set.The method reuses pretrained convolutional kernels while learning the relationship between latent vectors and sparse target samples.
  • Dataset size: Performance was limited as dataset size became large, although the method worked well for small datasets.The authors suggest combining adversarial approaches or increasing the number of trainable parameters as possible remedies.
  • Source domain selection: ImageNet-pretrained generators produced the best adaptation quality, while generators pretrained on other datasets produced blurry or meaningless images, showing that source-filter diversity matters.The comparison included randomly initialized generators and generators pretrained on face, bedroom, and ImageNet datasets.

6. Conclusion

The proposed method transfers prior knowledge from a pre-trained generator while updating only scale and shift parameters, enabling image generation from much smaller datasets. It produced higher-quality images than existing methods and supports low-shot generative learning.

  • Updating only the scale and shift parameters of a pre-trained generator enables image generation from far fewer images than regular generator training.The method transfers prior knowledge while limiting the parameters adapted to the target dataset.
  • The proposed method generated higher-quality images from small training datasets relative to existing methods.
  • The method can be used for a new task: low-shot learning for generative models.
  • The authors identify small-dataset augmentation as an application and propose extending the method to classification and few-shot learning.

A. Experimental detail

The experiments control anime-face texture diversity by selecting stylistically similar images and use established GAN, perceptual-loss, and pretrained-weight configurations.

  • Anime-face samples with similar Gram matrices were selected to limit textural diversity and make the sparse-data problem easier.The Gram matrix is described as controlling style information.
  • The experiments used SNGAN for unconditional generation and SNGAN projection and BigGAN for conditional generation.The implementation used pretrained weights for SNGAN-128 and a BigGAN-256 reimplementation.
  • VGG16 trained on ImageNet supplied features for the perceptual loss across layers from conv1_1 through conv4_3.

A.3. Training settings

Training settings vary across experiments, with optimizer schedules, regularization coefficients, and computational costs specified for different model configurations. An activation analysis relates batch-statistics parameters to filter activity.

  • A.3. Training settings: Some experiments used a different training setting, with implementation details referenced separately.
  • A.3. Training settings: 3,000 to 4,000 iterations used batchsize 25, with Adam learning rates decreasing from 0.1 at datasize 25 to 0.02 at datasize 500.
  • A.3. Training settings: The comparison setup used Adam learning rates of 0.001 for encoders and 0.0001 for other GAN models, with λC = 0.001, λz = 0.2, and 0 ≤λγ,β ≤0.02.
  • A.3. Training settings: For another configuration, models were trained for 6000 to 10000 iterations with batchsize 16 and separate learning rates for class embeddings, statistics parameters, and latent vectors.
  • B. Relationship between the scale and shift and activation rate: The activation analysis found positive correlations between γ, β, and filter activation rates across the examined generator layers.The figure plots each filter and columns corresponding to the first conditional batch normalization in each residual block.
  • B. Relationship between the scale and shift and activation rate: Figure A describes the relationship between active-kernel rate and γ and β.

C. Generated samples from Transfer GAN and “Update all”

The section compares generated samples from Transfer GAN and “Update all” across several training-set sizes and image domains. Transfer GAN training was stopped before generated images collapsed.

  • Generated samples were compared for Transfer GAN and “Update all” using human-face, anime-face, and flower images.
  • The comparisons used training-set sizes of 25, 50, 100, and 500 images, except flowers, which were not tested at 500 because that dataset contained only 251 images.
  • Transfer GAN models were stopped before the generated images collapsed.
  • Figure B presents generated images from both methods trained with 25, 50, 100, and 500 training images.

D. Comparison of interpolation results

With 25 training samples, the comparison examines interpolation layouts across human, anime-face, and flower images. Our method produces more consistent interpolations, though they are somewhat blurry, while other methods show limited quality or collapse.

  • Interpolation setup: With 25 training samples, the figures compare interpolations between randomly sampled images and between generated images corresponding to training samples.The top four rows use randomly sampled images, while the bottom four rows use generated images corresponding to two training samples.
  • Method comparison: Transfer GAN produces more consistent images but collapses to a few modes according to random-generation results and evaluation scores.
  • Method comparison: “Update all” performs almost pixel-wise interpolation, producing hair changes and flower shapes that are inconsistent.
  • Method comparison: Our method performs more consistent interpolation between two images, although the results are a little blurry.
  • Compared domains: The comparison includes generated human-face, anime-face, and flower images from methods trained with 25 training images.
Loading 1904.01774v4…