Source-linked AI summary
When do GANs replicate? On the choice of dataset size
Qianli Feng, Chenqi Guo, Fabian Benitez-Quiroz, Aleix Martinez
TL;DR
Prior work identified several factors related to GAN memorization, but had not examined dataset size and complexity. This paper empirically studies their relationship with replication and image quality, finding exponential decay in replication and a U-shaped quality trend as dataset size changes.
Problem
Previous work explored factors related to GAN replication, but the effect of dataset size and complexity remained unexamined.
Source: §1 · p. 2Method
Results
Replication percentage decays exponentially with dataset size and complexity, while image quality follows a U-shaped trend with dataset size.
Source: §12 · p. 8Takeaways & Limitations
The authors provide a one-shot tool to estimate the training-sample count needed for a given replication percentage, guiding dataset-size choices for image synthesis.
Source: §12 · p. 8The proposed trends are hypotheses, with dataset complexity increasing only before converging to ground-truth complexity and manifold reconstruction framed as conditional on a correct model choice.
Source: §4 · p. 3, §10 · p. 5
Abstract
from arXiv · showhide
Do GANs replicate training images? Previous studies have shown that GANs do not seem to replicate training data without significant change in the training procedure. This leads to a series of research on the exact condition needed for GANs to overfit to the training data. Although a number of factors has been theoretically or empirically identified, the effect of dataset size and complexity on GANs replication is still unknown. With empirical evidence from BigGAN and StyleGAN2, on datasets CelebA, Flower and LSUN-bedroom, we show that dataset size and its complexity play an important role in GANs replication and perceptual quality of the generated images. We further quantify this relationship, discovering that replication percentage decays exponentially with respect to dataset size and complexity, with a shared decaying factor across GAN-dataset combinations. Meanwhile, the perceptual image quality follows a U-shape trend w.r.t dataset size. This finding leads to a practical tool for one-shot estimation on minimal dataset size to prevent GAN replication which can be used to guide datasets construction and selection.
1. Introduction
GAN replication matters because synthetic samples are useful for content creation and dataset augmentation only under the assumption that they are not merely copies of training data. The paper examines the previously unstudied role of dataset size and complexity, reporting replication under unmodified training procedures and offering guidance for dataset sizing.
1. Introduction
- Replication is practically important because content creation and dataset augmentation rely on generators not merely copying training data.The authors also note ethical implications in face-generation applications and urge caution when augmenting small datasets to address class imbalance.Source: §1 · p. 1, §2 · p. 2
2. Related Works
- Earlier work examined factors such as latent codes and discriminator complexity, but had not explored dataset size and complexity as drivers of replication [26] [19] [1].Prior replication studies also changed the GANs’ originally proposed training procedures, which the authors say affects the external validity of their conclusions.Source: §1 · p. 2, §2 · p. 2
1. Introduction
3. Problem Definition
The paper formalizes replication as a probability defined by a distance threshold between generated images and training images. It then studies how that probability varies with a dataset measure, focusing on dataset size and complexity.
3. Problem Definition
- Pα(G, d, pdata) denotes the probability that a generated image is within distance α of its nearest training image.The definition uses the minimum distance from G(z) to any image in the training set.Source: §3 · p. 2
- The authors frame the target as the relationship between dataset size or complexity and GAN replication, rather than the full function across all GANs, metrics, and datasets.They describe that broader joint-domain function as impractical to characterize.Source: §3 · p. 2
- The study focuses on how a dataset measure maps to replication probability while holding the generator and distance metric fixed.The measures of particular interest are counting measure and complexity measurement.Source: §3 · p. 2
4. Hypothesis
The authors hypothesize that increasing dataset size reduces replication probability, while generated-image quality follows a U-shaped trend. Their rationale contrasts fitting a single image with the difficulty of fitting a larger empirical dataset and the later possibility of reconstructing the image manifold.
4. Hypothesis
- With a single-image training set, a sufficiently complex model can fit that image and map all sampled latent codes to it.As nontrivial samples are added, the empirical dataset distribution becomes more complex, making overfitting more difficult.Source: §4 · p. 3
- The proposed manifold-reconstruction analogy places a threshold beyond which sufficient samples can improve image quality and reduce replication, given a correct model choice.The authors compare this threshold to the double maximum frequency threshold for lossless reconstruction of a continuous signal.Source: §4 · p. 3
- The authors hypothesize that replication probability decreases as dataset size increases, while image quality follows a U-shaped trend.Quality is initially high with faithful replication, falls as the dataset grows but remains too small for manifold reconstruction, then rises with further growth while replication stays low.Source: §4 · p. 3
5. Definition of Replication
The paper operationalizes replication using Euclidean distance in pixel space, treating a generated sample as replicated when its nearest training image is within a threshold. This deliberately makes replication correspond to near-exact copying rather than merely semantic similarity.
5. Definition of Replication
- The distance metric matters because it directly defines what counts as replication in this study.The operational metric is Euclidean distance in image space.Source: §5 · p. 3
- Euclidean nearest-neighbor distance can be fooled by unperceivable color shifts, small translations, or dead pixels, motivating semantic-space metrics in prior work [19] [22] [27] [20] [16].The paper treats those weaknesses as compatible with its stricter focus on copies.Source: §5 · p. 3
- The study uses Euclidean distance in image space because its aim is to identify GAN memorization rather than show that a GAN does not memorize.The authors argue that this metric requires replications to be exact copies up to noise level α, which would also count as memorization under semantic metrics.Source: §5 · p. 3
6. Dataset Size and Complexity
The study characterizes training sets by complexity and size, using intrinsic dimensionality and a counting measure, respectively. Intrinsic dimensionality is estimated from pixel-space nearest-neighbor distances.
6. Dataset Size and Complexity
- Intrinsic dimensionality (ID) estimates the degree of freedom of the data manifold in high-dimensional pixel space.The study uses ID as its dataset-complexity measure.Source: §6 · p. 3
- The maximum-likelihood ID estimator uses Euclidean distances to each image’s k-th nearest neighbor, with k1 and k2 setting the locality range.The ID estimate is based on k-NN measurements averaged across the selected k range and training images.Source: §6 · p. 3
- ID is measured in pixel space to match the study’s use of pixel-space distance for GAN replication.The authors use this alignment rather than a neural-network semantic embedding space.Source: §6 · p. 3
- The study measures dataset size with the counting measure, |X|.This measure counts the training-set samples.Source: §6 · p. 3
7. Selection of GAN Architectures
The study uses BigGAN and StyleGAN2 as established GAN architectures to examine dataset size and complexity. It follows each architecture’s original training procedure and uses configurations reported to have the highest performance in the respective original papers.
7. Selection of GAN Architectures
- BigGAN and StyleGAN2 were selected as established architectures suitable for studying dataset size and complexity.The authors cite their state-of-the-art performance as the selection rationale.Source: §7 · p. 3
- BigGAN uses the BigGAN-deep architecture with adversarial hinge loss, following the original procedure and its highest-performing implementation.The passage attributes the performance characterization to [2].Source: §7 · p. 3
- StyleGAN2 uses the original paper’s config-f, with large networks and regularized adversarial logistic non-saturation loss.Config-f is identified as the configuration with the highest reported performance.Source: §7 · p. 3
8. Dataset Setup
The experiments vary training-set size within datasets and compare datasets of differing complexity. The setup includes CelebA, LSUN-bedroom, and Oxford Flower 17, with subset levels selected for each dataset.
8. Dataset Setup
- With small training sets, GANs can generate almost exact replicas, and replication gradually lessens as dataset size increases.Figure 2 shows randomly generated, non-cherry-picked examples for BigGAN-CelebA, BigGAN-Flower, and StyleGAN2-LSUN-bedroom.Source: Figure 2 · p. 4
- The study varies dataset size by increasing subset size and varies complexity by combining datasets of different object types and sizes.This creates training sets spanning different dataset complexities for examining replication.Source: §8 · pp. 3–4
- The dataset collection comprises CelebA, LSUN-bedroom, and Oxford Flower 17, spanning faces to complex bedroom scenes.The authors describe this collection as providing a range of dataset complexity.Source: §8 · p. 4
- Multiple randomly selected subset levels were created for each dataset, with the specific sample counts listed in Table 1.The number of samples at each level depends on the replication trend for that dataset.Source: §8 · p. 4, Table 1 · p. 4
9. Experimental Setup
The experimental setup measures replication by matching generated samples to training images and estimating intrinsic dimensionality for each subset. Perceptual quality is assessed separately through human ratings on sampled generated images.
9. Experimental Setup
- Replication is estimated from 1,024 generated samples per generator as the percentage whose nearest training image is within Euclidean distance α = 8,000.The authors report α = 8,000 in the main paper, describing it as the loosest threshold consistent with human perception of replication; α = 9,000 and 10,000 are in supplemental results.Source: §9 · p. 4
- Subset ID is estimated with a maximum-likelihood estimator using k1 = 10 and k2 = 20, after downscaling images to 32×32.The authors report no significant difference from 128×128 for this downscaling; Faiss supplies exact nearest-neighbor search for both ID and replication calculations.Source: §9 · p. 4
9.1. Measuring perceptual quality
- Perceptual quality is assessed in a behavioral experiment because FID does not necessarily reflect perceived image quality.The study uses Amazon Mechanical Turk for the experiment.Source: §9.1 · pp. 4–5
- Each image is rated by nine subjects on a five-point Excellent-to-Terrible Likert scale, using 100 randomly sampled images per subset level, GAN, and dataset.The rating criteria are described in the Supplemental Material.Source: §9.1 · p. 5
10. Result and Analysis
Across the tested datasets and GAN architectures, replication percentage decays exponentially with dataset complexity and size. Perceptual image quality varies with dataset size, showing a full U-shaped trend in some combinations and partial trends in others.
10.1. Dataset complexity vs. GAN replication
- Replication percentage follows a common exponential decay with dataset complexity across datasets and GAN architectures.The fitted curves share decay factor a in the range 0.96–0.98 and translation c near 100; fit quality is above 0.9 in almost all experiments.Source: §10.1 · pp. 5–6
10.2. Dataset size vs. GAN replication
- Dataset size also shows a common exponential decay relationship with replication percentage across GAN architectures and datasets.The size-based formulation has R2 > 0.85 for all experiments, while the intermediate size-complexity model has R2 close to 1 across all three datasets.Source: Figure 4 · p. 6, §10.2 · pp. 6–7
10.3. Dataset size vs. Perceptual Quality
- BigGAN-Flower, BigGAN-LSUN, and StyleGAN2-LSUN show the full quality trend: high with replication, lower at intermediate sizes, then higher at larger sizes.Figure 5 reports average perceptual quality with 95% confidence intervals and uses the real-image average as a reference.Source: §10.3 · p. 7, Figure 5 · p. 7
- BigGAN-CelebA shows the first half of the quality trend, while StyleGAN2-CelebA and StyleGAN2-Flower show the second half.The authors suggest that larger subset levels may be needed for the former and smaller subset levels for the latter.Source: §10.3 · p. 7
11. One-shot Prediction on GAN Replication
The section tests whether replication curves can be predicted from very early dataset subsets and whether they can estimate replication or dataset size. One-shot prediction performs close to full-subset fitting for replication percentage from dataset ID, while predicting from dataset size is less accurate.
11. One-shot Prediction on GAN Replication
- The prediction tool is intended to estimate the replication curve early, when a dataset is still growing, and guide dataset-size choices for image synthesis.The section identifies predicting from dataset ID or dataset size, and vice versa, as the practical aim.Source: §11 · pp. 7–8
11.1. From dataset ID to replication
- The method estimates shared decay factor a and translation c across combinations, then estimates b for a held-out GAN–dataset combination from one or two smallest subset levels.Leave-one-out cross-validation simulates prediction while a dataset is still being collected.Source: §11.1 · p. 8
- R2 > 0.8 for every one-shot replication-curve prediction, with only minor drops for BigGAN on Flower and CelebA versus full-subset fitting.Two-shot prediction does not improve over one-shot.Source: §11.1 · p. 8
- The median replication-percentage error from dataset ID is 2.42% with one subset level, matching the 2.42% median error from full-subset fitting.The section presents this as evidence that one-shot prediction is effective.Source: §11.1 · p. 8
11.2. From dataset size to replication
- The median error for predicting replication percentage from dataset size is around 4.56%, and this prediction is less accurate than predicting from dataset ID.The passage attributes the loss to accumulated error during the change of variable.Source: §11.2 · p. 8
11.3. From replication to dataset size
- The inverse relationship predicts dataset size from a target replication percentage, with mean error around 1.3K for full subsets and 1.5K for one-shot prediction.BigGAN-CelebA and BigGAN-LSUN perform poorly when replication percentages are near zero; the table caption identifies the prediction error measures reported.Source: §11.3 · p. 8, Table 4 · p. 8
12. Discussion and Conclusion
For a given GAN model and synthesis task, replication percentage decays exponentially with dataset size and complexity, while image quality follows a U-shaped trend with dataset size. The paper presents a one-shot prediction tool to estimate training-sample needs for a given replication percentage or the reverse.
12. Discussion and Conclusion
- Replication percentage decays exponentially with dataset size and complexity for a given GAN model and synthesis task.The paper presents this relationship as deepening understanding of GAN replication and overfitting.Source: §12 · p. 8
- Generated-image quality follows a U-shaped trend with dataset size.Source: §12 · p. 8
- The one-shot tool predicts training-sample needs for a given replication percentage or predicts replication percentage from dataset size, offering guidance for constructing image-synthesis datasets.Source: §12 · p. 8
Related papers
This paper cites
- Progressive Growing of GANs for Improved Quality, Stability, and Variation2017
- Large Scale GAN Training for High Fidelity Natural Image Synthesis2018
- Analyzing and Improving the Image Quality of StyleGAN2019
- Billion-scale similarity search with GPUs2017
- Are GANs Created Equal? A Large-Scale Study2017
- GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification2018