PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Overfitting of Deep Generative Networks via Latent Recovery

Ryan Webster, Julien Rabin, Loic Simon, Frederic Jurie

arXiv:1901.03396v1cs.LGstat.ML

TL;DR

Visualizing training-set nearest neighbors is insufficient to rule out memorization, motivating closer study of deep generators. The paper uses latent recovery to compare reconstruction errors on training and validation images; it reports overfitting as undetectable in GANs but detectable in hybrid adversarial and non-adversarial methods.

  • Problem

    Training-set nearest-neighbor visualizations are insufficient to rule out memorization in realistic deep-generator outputs, despite privacy and copyright concerns.

    Source: §1 · p. 2, §1.1 · p. 2

  • Method

    The paper optimizes latent codes to recover target images and analyzes reconstruction-error statistics for training and validation sets.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Overfitting was undetectable for GANs but detectable for hybrid adversarial methods such as CycleGAN and non-adversarial methods such as GLO.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The paper applies latent recovery to face inpainting and face super-resolution without hybrid adversarial losses.

    Source: §1 · p. 2

    Latent recovery failed to verbatim recover generated images for a few generators trained on LSUN bedroom, suggesting the dataset may affect latent-space organization.

    Source: §5 · p. 8

Abstract

from arXiv · show

State of the art deep generative networks are capable of producing images with such incredible realism that they can be suspected of memorizing training images. It is why it is not uncommon to include visualizations of training set nearest neighbors, to suggest generated images are not simply memorized. We demonstrate this is not sufficient and motivates the need to study memorization/overfitting of deep generators with more scrutiny. This paper addresses this question by i) showing how simple losses are highly effective at reconstructing images for deep generators ii) analyzing the statistics of reconstruction errors when reconstructing training and validation images, which is the standard way to analyze overfitting in machine learning. Using this methodology, this paper shows that overfitting is not detectable in the pure GAN models proposed in the literature, in contrast with those using hybrid adversarial losses, which are amongst the most widely applied generative methods. The paper also shows that standard GAN evaluation metrics fail to capture memorization for some deep generators. Finally, the paper also shows how off-the-shelf GAN generators can be successfully applied to face inpainting and face super-resolution using the proposed reconstruction method, without hybrid adversarial losses.

1. Introduction and related work

Realistic GAN outputs make memorization difficult to rule out: nearest training-set neighbors can be sensitive to perturbations, so visual checks alone are insufficient. The paper proposes latent recovery and compares reconstruction errors across training and validation images to study overfitting, alongside prior work on evaluation, generalization, and privacy.

1. Introduction and related work

  • The paper studies memorization because training data may be private or copyrighted and should not appear in generated images.Source: §1 · p. 2, §1.1 · p. 2
  • Nearest training-set neighbors can recover pose but are sensitive to simple perturbations such as flipping, so they do not guarantee that a GAN avoids memorized outputs.Source: §1 · p. 2
  • Latent recovery searches the generator manifold for nearest reconstructions of training and validation images, making the comparison more robust to perturbations.Source: §1 · p. 2
  • The paper proposes analyzing latent-recovery errors on train and test sets; it reports overfitting as undetectable for GANs but detectable for CycleGAN and GLO.Source: §1 · p. 2

1.1. Related Work

  • Prior work describes GAN generalization in theoretical terms and estimates face-identity support, while classification overfitting is defined by a training-versus-holdout performance discrepancy.Source: §1.1 · p. 2

2. Reconstruction by Latent Code Recovery

The paper reconstructs target images by optimizing latent codes for an existing generator, extending latent recovery to unseen or deliberately distorted images. It reports that Euclidean loss recovers plausible images under substantial distortions, while recovery errors also reveal differences in generator specificity.

2. Reconstruction by Latent Code Recovery

  • Latent codes are recovered by optimization to reconstruct images the generator has not seen, including validation images and deliberately distorted images.Source: §2 · p. 3

2.1. Latent Code Recovery with Euclidean Loss

  • The recovered nearest neighbor is the generator output G(z*) that minimizes discrepancy from target image y, unlike a nearest-neighbor search over dataset D.Source: §2.1 · p. 3
  • Euclidean loss recovers generated images verbatim across analyzed networks, while VGG-feature perceptual loss showed no improvement or degraded visual results.Source: §2.1 · p. 3

2.2. Latent Code Recovery Under Distortion

  • Latent recovery can reconstruct faces near ground truth after heavy distortion, but GLO recovers distorted images with training-like MREs, indicating lower specificity.Source: Figure 2 · p. 3, §2.2 · p. 4

3. Using Latent Recovery to Assess Overfitting

The section tests overfitting by comparing latent-recovery errors for training and validation images. It finds distinct patterns across pure GANs and generators trained with reconstruction losses, and introduces statistics for detecting those differences.

3. Using Latent Recovery to Assess Overfitting

  • The section extends the train/validation comparison used to assess classifier overfitting to generative models by comparing image recovery errors across the two sets.Source: §3 · p. 4

3.1. Training Protocols

  • The protocols compare pure GANs, including PGGAN and MESCH, with reconstruction-loss models such as GLO, AEGAN, and CycleGAN, using varied training-set sizes.For the reconstruction-loss models, the study uses the autoencoder reconstruction losses; GLO-N and AEGAN-N denote training-set size N.Source: §3.1 · pp. 4–5

3.2. Comparison of recovery errors

  • PGGAN and MESCH show barely distinguishable train/validation recovery-error distributions, while GLO-N and AEGAN-N differences shrink as training sets grow and nearly disappear at 8192 images.Pure GAN approaches could not be trained on the small datasets used for GLO-N and AEGAN-N.Source: §3.2 · pp. 5–6, Figure 4 · p. 6

3.3. Statistical Analysis

  • At 26,000 CelebA-HQ images and 32,768 LSUN-bedroom images, overfitting is no longer detectable; KS p-values and normalized MRE-gap make detection more practical than visual inspection.The study uses a 1% KS p-value threshold and highlights MRE-gap values above 10%.Source: §3.3 · p. 6

3.4. FID Does Not Detect Memorization

  • MRE detects memorization in GLO models when FID does not, so the authors propose using MRE to complement—not replace—FID.Source: §3.4 · pp. 6–7

4. Applications to image editing

Latent recovery with an off-the-shelf GAN generator is demonstrated for face inpainting and super-resolution. The inpainting results are artifacted, while super-resolution is reported at least on par with prior work.

4. Applications to image editing

  • A competing approach that optimizes a randomly initialized generator is described as flexible but bound to fail on inpainting large semantic structures such as faces.That approach fixes a random latent code and optimizes the generator parameters.Source: §4 · p. 7
  • Figure 5 reports that FID does not detect memorization in GLO models, whereas median recovery error does.The figure compares FID and median recovery error for training images in green and validation images in red.Source: Figure 5 · p. 7
  • Latent recovery with a vanilla GAN not trained for a specific task can address inverse problems using different operators φ.The paper gives un-distortion and blind inpainting as examples.Source: §4 · p. 8
  • Face inpainting is artifacted, while super-resolution results are at least on par with [9].Both use the progressive GAN generator [16] without post-processing; super-resolution uses 64x pooling.Source: §4 · p. 8, Figure 6 · p. 8
  • The recovered faces retain pose but change identity, even when the inputs come from images used to train the PGGAN generator.The authors describe this identity change as a positive feature because hybrid-loss inpainting methods could potentially overfit faces.Source: §4 · p. 8

5. Discussion and future work

The discussion notes that recovery losses can enable overfitting even with adversarial training, while GLO and CycleGAN can retain decent generation quality with few data points. It also identifies limits of the overfitting test and latent-recovery optimization.

5. Discussion and future work

  • Using a recovery loss like GLO’s can make a training set overfit even when the loss is combined with adversarial training.GLO and CycleGAN can nevertheless train with few data points while retaining decent generation quality.Source: §5 · p. 8
  • Overfitting as measured in Section 3 does not capture every possible privacy concern: object information may leak even when a face GAN generalizes across individual faces.The authors cite recognizable Eiffel Tower images from a network trained on LSUN tower and call for closer examination of generative-model privacy.Source: §5 · p. 8
  • For a few generators trained on LSUN bedroom, Eq. (NNG) optimization could not verbatim recover generated images, suggesting dataset-dependent latent-space complexity and local minima.Other networks could verbatim recover generated images for LSUN tower and bedroom; the authors say this phenomenon needs more investigation.Source: §5 · pp. 8–9
  • The paper concludes that latent recovery finds plausible images after significant transformations and that overfitting was undetectable for GANs but detectable for CycleGAN and GLO, even with moderate training-set sizes.The authors present the methodology as a step toward analyzing overfitting amid privacy and copyright concerns.Source: §5 · p. 9

A.1. Optimization Failures

Most networks could recover their generated images, but some could not; the PGGAN trained on LSUN Bedroom did not verbatim recover any image. Networks that could never recover generated images were excluded from the analysis.

A.1. Optimization Failures

  • The PGGAN trained on LSUN Bedroom did not verbatim recover any image, while some LSUN networks, including DCGAN, could recover generated images.The authors suggest that some LSUN networks may have more complex latent spaces with many local minima, and excluded networks that could never recover generated images.Source: §A.1 · p. 11, Figure 7 · p. 11
  • Figure 7 shows inconsistent recovery for MESCH, which the authors report can be alleviated by restarting optimization.The figure compares generated images in the first row with their recoveries in the second row.Source: Figure 7 · p. 11

A.2. Recovery Success Rate

Recovery success varied across networks, and repeated optimization improved MESCH recovery rates. The authors treat MSE < .1 as a plausible recovery for real images.

A.2. Recovery Success Rate

  • MESCH recovery success rose from 68% to 98% with 10 optimization restarts per image.The reported success-rate threshold is MSE < .1, corresponding to plausible synthesis.Source: §A.2 · p. 11, Table 2 · p. 11
  • For generated images, the authors note that MSE < .025 can be used as a stricter threshold corresponding to verbatim recovery.The table caption reports success rates for real and generated images using MSE < .1 as the plausible-synthesis threshold.Source: §A.2 · p. 11, Table 2 · p. 11
  • MESCH resnets were less consistent than other architectures, and the authors attribute failures likely to bad initialization.They report that restarting greatly increases success rates.Source: §A.2 · p. 11, Table 2 · p. 11

A.3. Local vs Global overfitting

The authors tested local overfitting by recovering eye and mouth patches, finding no significant training-validation differences at their 1% threshold. Eye recovery was consistent across most random initializations.

A.3. Local vs Global overfitting

  • The local-overfitting test used a masking operator on eye or mouth regions to compare training and validation recovery-error distributions.The authors describe this test as investigating whether GANs overfit image patches despite appearing not to overfit entire images.Source: §A.3 · pp. 11–12
  • Eye and mouth patch recovery showed no significant local overfitting, with KS p-values of 0.0545 and 0.6918, respectively.Both p-values are above the 1% threshold used for global overfitting detection.Source: §A.3 · p. 12, Figure 9 · p. 12
  • For eye-patch recovery, optimization was consistent across most random initializations, as illustrated for PGGAN with LBFGS and Euclidean loss.The figure compares targets, initialization, and optimization after 100 iterations; recovery is reported as more consistent than global optimization.Source: §A.3 · p. 11, Figure 8 · p. 12

B. Comparison with Other Loss Functions

The authors compared Euclidean recovery loss with pooling and VGG-19 losses. Pooling appeared to improve texture recovery while preserving facial similarity, whereas VGG-19 hindered PGGAN recovery.

B. Comparison with Other Loss Functions

C. Convergence analysis of latent recovery

The section evaluates latent-recovery convergence across optimizers, loss functions, operators, and generators. LBFGS converges faster than SGD and Adam, while recovery with MESCH is less consistent but can be made more reliable with multiple initializations.

C.1. Protocol

  • The protocol compares recovery of 20 generated and 20 real images over 100 iterations, summarizing 400 error curves with medians and quartiles.The same 20 random latent codes initialize recovery across the compared algorithms.Source: §C.1 · p. 14

C.2. Comparison of optimization algorithm

  • LBFGS converges faster than SGD and Adam, with 50 iterations enough to reach the recovery threshold in half the cases.The threshold is MRE = 0.024.Source: §C.2 · p. 14

C.3. Comparison of objective loss functions

  • The loss comparison evaluates Euclidean distance (L2), Manhattan distance (L1), and VGG-based perceptual loss.The text identifies L2 as the objective used throughout the paper and L1 as an alternative described as more robust to outliers.Source: §C.3 · p. 14

C.4. Convergence with operator φ

  • Convergence is examined under different operators φ for latent recovery.The section points to Figure 14 for this comparison but does not state its outcome in the supplied passage.Source: §C.4 · p. 14

C.5. Recovery with other generators

  • MESCH recovery is less consistent, but 10 random initializations give a 96% chance of recovering a generated or real image.The section also presents recovery-error comparisons for DCGAN and MESCH generators.Source: §C.5 · p. 14

C.6. Convergence on real images

The section presents real-image recovery with PGGAN and visual comparisons of optimization methods and generators. Recovery is highly consistent for PGGAN, while MESCH recovery has a lower success rate that can be improved by restarting or using Adam.

C.6. Convergence on real images

  • Recovery on real images is highly consistent for the PGGAN network.Figure 16 presents real-image recovery with LBFGS and Euclidean loss.Source: §C.6 · p. 14, Figure 16 · p. 17
  • For Euclidean-loss recovery, LBFGS gives much better visual results than SGD, though it may still need a restart.The illustrated comparison is for PGGAN-generated target images.Source: Figure 11 · p. 15
  • LBFGS converges faster than SGD and Adam in recovery-error comparisons, and changing φ has no effect on convergence for the stated applications.The applications named are super-resolution and inpainting.Source: Figure 12 · p. 16, Figure 14 · p. 16
  • MESCH recovery succeeds in about 67% of cases, but restarting with a new initialization or using Adam can circumvent failures.Figure 18 shows visual results for recovery of generated MESCH images.Source: Figure 18 · p. 18

Related papers

Loading 1901.03396v1…