Source-linked AI summary
SinGAN: Learning a Generative Model from a Single Natural Image
Tamar Rott Shaham, Tali Dekel, Tomer Michaeli
TL;DR
Existing unconditional GANs struggle with diverse natural images, while prior single-image methods are task-specific or conditional. SinGAN learns multi-scale patch statistics from one image and generates diverse realistic samples that preserve its visual content, while supporting varied image manipulations; its semantic diversity remains limited by the single training image.
Problem
Unconditional generation for diverse natural images remains challenging, and prior single-image methods are often task-specific or conditional.
Method
SinGAN trains a pyramid of GANs against multi-scale patch distributions from one natural image, generating samples through coarse-to-fine noise-driven synthesis.
Results
SinGAN generates diverse realistic samples that preserve global structures and fine textures, and supports multiple image manipulation tasks from the same training image.
Takeaways & Limitations
Single-image internal patch statistics can support natural-image generation beyond textures and a wide range of image manipulation tasks.
Takeaways & Limitations
SinGAN has less semantic diversity than externally trained generators and cannot generate different object categories or breeds absent from the training image.
Abstract
from arXiv · showhide
We introduce SinGAN, an unconditional generative model that can be learned from a single natural image. Our model is trained to capture the internal distribution of patches within the image, and is then able to generate high quality, diverse samples that carry the same visual content as the image. SinGAN contains a pyramid of fully convolutional GANs, each responsible for learning the patch distribution at a different scale of the image. This allows generating new samples of arbitrary size and aspect ratio, that have significant variability, yet maintain both the global structure and the fine textures of the training image. In contrast to previous single image GAN schemes, our approach is not limited to texture images, and is not conditional (i.e. it generates samples from noise). User studies confirm that the generated samples are commonly confused to be real images. We illustrate the utility of SinGAN in a wide range of image manipulation tasks.
1. Introduction
SinGAN learns an unconditional generative model from one natural image by modeling its multi-scale internal patch statistics. Unlike prior single-image and texture-focused methods, it generates diverse natural-image samples and supports multiple manipulation tasks without additional training.
- SinGAN addresses the challenge of unconditional generation for diverse natural images without requiring a database of same-class images.
- A pyramid of fully convolutional GANs captures patch distributions at different image scales, using a coarse-to-fine multi-scale pipeline.
- SinGAN supports paint-to-image, editing, harmonization, super-resolution, and animation from one image using the same generative network without further training.
- Unlike earlier internal GAN work, SinGAN is purely generative and maps noise to image samples rather than conditioning generation on an input image.
- The model generates realistic samples with complex textures and non-repetitive global structures, extending single-image generation beyond texture images.
2. Method
SinGAN learns an unconditional generative model from one natural image by matching patch statistics across scales. Its multi-scale generators combine global structure, fine details, and noise-driven variability while supporting arbitrary output dimensions.
- Multi-scale architecture: SinGAN trains a hierarchy of patch-GANs, each modeling the patch distribution at a different scale of the training image.The generators are trained adversarially against discriminators that distinguish generated patches from patches in the corresponding image scale.
- Multi-scale architecture: The coarsest generator maps spatial white Gaussian noise to an image sample and establishes the image’s global layout and object structure.Its effective receptive field is typically about half the image height.
- Multi-scale architecture: Finer generators add missing details to an upsampled coarser image while receiving independent spatial noise at each scale.Noise is added before convolutional processing so the network does not disregard randomness in conditional generation.
- Multi-scale architecture: Fully convolutional generators can produce images with arbitrary size and aspect ratio by changing the dimensions of the noise maps at test time.Each generator uses five convolutional blocks, with the number of kernels increasing across scales.
- Training: Training proceeds sequentially from coarse to fine scales, combining adversarial and reconstruction losses while keeping trained GANs fixed.The adversarial loss matches patch distributions, whereas reconstruction ensures that specific noise maps can reproduce the training image.
- Training: Each Markovian discriminator classifies overlapping patches across the whole generated image, enabling the model to learn boundary conditions.The discriminator uses a receptive-field patch size of 11 × 11 and WGAN-GP training for stability.
- Training: The reconstruction pathway uses fixed noise maps to generate the original image and sets each scale’s noise standard deviation from the upsampling reconstruction error.This error indicates how much detail needs to be added at that scale.
3. Results
SinGAN generates diverse, realistic samples from single images while preserving visual content across scales, and its controllable generation trades global variability against structural fidelity. User studies and SIFID evaluate realism and internal-statistics matching, while super-resolution results compare favorably with established methods.
- Random image samples: SinGAN generated realistic samples with new object configurations while preserving global structures, fine textures, reflections, and shadows across varied image sizes.The architecture is resolution agnostic and can generate samples with expanded or decreased dimensions in each axis.
- Random image samples: Arbitrary image sizes and aspect ratios are supported because the fully convolutional model generates random samples optimized for patch statistics rather than salient-object preservation.This objective distinguishes the samples from image retargeting.
- Effect of scales at test time: Starting generation at finer scale N−1 preserves global structure while changing finer features, whereas starting at the coarsest scale produces greater global variability.The Zebra example preserves shape and pose while altering stripe texture when generation starts at N−1.
- Effect of scales during training: Increasing the number of training scales enables SinGAN to capture larger structures and better preserve global object arrangement instead of only fine textures.With fewer scales, the smaller effective receptive field captures only fine textures.
- Quantitative evaluation: 50% confusion would indicate perfect real/fake confusion, and generated images remained difficult to distinguish from real images even when large structures changed.Confusion rates decreased as generated-image diversity increased.
- Quantitative evaluation: SIFID measures the distance between internal deep-feature statistics of a single real image and a generated sample, while its correlation with AMT results supports human-ranking agreement.The metric uses convolutional-layer features at multiple spatial locations rather than a single pooled image vector.
- Super-resolution: SinGAN super-resolves low-resolution inputs through iterative upsampling and its finest-scale generator, achieving visual quality better than ZSSR, DIP, and EDSR and comparable to SRGAN.The figure reports PSNR and NIQE in parentheses.
4. Applications
SinGAN reuses the same trained generative model for multiple image-manipulation tasks by injecting an image or its downsampled version into an intermediate generation scale. The resulting applications preserve supplied structure while synthesizing or matching textures and produce favorable qualitative comparisons in the reported examples.
- Unified framework: The same trained model handles multiple applications without architectural changes, further tuning, additional information, or task-specific retraining.Manipulation injects a possibly downsampled image into the generation pyramid and feed-forwards it to match the training image’s patch distribution.
- Super-Resolution: For super-resolution, SinGAN repeatedly upsamples a low-resolution image and injects it into the finest generator to produce the final high-resolution output.The model is trained on the low-resolution image with reconstruction-loss weight α = 100 and pyramid factor r = k√s.
- Super-Resolution: On BSD100, SinGAN’s super-resolution performance is similar to SRGAN, with slightly inferior NIQE and slightly better RMSE.NIQE measures perceptual quality and RMSE measures distortion in the reported comparison.
- Paint-to-Image: Paint-to-image transfer preserves the clipart’s global structure while generating realistic texture and high-frequency information matching the target image.The reported visual quality exceeds the compared style-transfer methods.
- Harmonization: Harmonization adjusts a pasted object’s texture to match the background while often preserving its structure better than the dedicated comparison method.The object and background are combined after generation from a downsampled pasted composite.
- Editing: Editing regenerates fine textures and seamlessly stitches copied regions, producing nicer results than Photoshop’s Content-Aware-Move in the reported examples.The edited composite is injected at a coarse scale before combining generated edited regions with the original image.
- Single Image Animation: Single-image animation synthesizes realistic object motion by traversing appearances inferred from repeated visual snapshots in the input image.The approach uses repetitions such as different wing postures within an image of a flock of birds.
5. Conclusion
SinGAN is an unconditional generative scheme learned from a single natural image, producing diverse realistic samples beyond textures. Its internal learning limits semantic diversity, yet it remains useful for many image-manipulation tasks.
- SinGAN is an unconditional generative scheme learned from a single natural image.
- The model generates diverse, realistic samples for complex natural images beyond texture synthesis.
- Internal learning limits semantic diversity compared with externally trained methods; a single-dog training image will not produce different dog breeds.
- Despite this limitation, SinGAN provides a powerful tool for a wide range of image-manipulation tasks.