Source-linked AI summary
Improved Techniques for Training Single-Image GANs
Tobias Hinz, Matthew Fisher, Oliver Wang, Stefan Wermter
TL;DR
Training realistic generative models from a single image is difficult but useful when large datasets are unavailable. The paper studies these challenges and proposes concurrently trained stages, improved rescaling, and fine-tuning, yielding faster training, fewer stages, and better global-structure capture than SinGAN.
Problem
Training realistic generative models from a single image is difficult, despite its usefulness when collecting large datasets is infeasible.
Method
ConSinGAN sequentially increases image resolution while concurrently training several stages, propagating features directly, improving rescaling, and supporting task-specific fine-tuning.
Results
ConSinGAN generates more believable images with better global-structure capture, fewer stages, fewer parameters, and less training time than SinGAN.
Takeaways & Limitations
The approach provides control over generated-image diversity and conformity while supporting single-image generation and task-specific applications.
Abstract
from arXiv · showhide
Recently there has been an interest in the potential of learning generative models from a single image, as opposed to from a large dataset. This task is of practical significance, as it means that generative models can be used in domains where collecting a large dataset is not feasible. However, training a model capable of generating realistic images from only a single sample is a difficult problem. In this work, we conduct a number of experiments to understand the challenges of training these methods and propose some best practices that we found allowed us to generate improved results over previous work in this space. One key piece is that unlike prior single image generation methods, we concurrently train several stages in a sequential multi-stage manner, allowing us to learn models with fewer stages of increasing image resolution. Compared to a recent state of the art baseline, our model is up to six times faster to train, has fewer parameters, and can better capture the global structure of images.
1. Introduction
ConSinGAN addresses single-image GAN training by concurrently training stages while increasing resolution and generator capacity. The resulting modifications improve efficiency, reduce required stages, and support task-specific fine-tuning.
- 500 additional fine-tuning iterations can improve results in less than two minutes on the authors’ hardware.
- A sequential overview begins with a small generator and low resolution, then increases both generator capacity and image resolution across stages.
- 20-25 minutes versus 120-150 minutes in the original SinGAN work reduces overall training time while using fewer stages.
- ConSinGAN trains several stages in parallel with different learning rates, trading off generated-image variance against conformity to the training image.
- Features, rather than generated images, are propagated directly between stages.
- Improved rescaling enables training on fewer stages while preserving low-resolution stages for global layout and using higher-resolution stages for texture and color.
2. Related Work
Single-image GAN research remains limited, with most approaches focusing on textures or relying on specialized methods. SinGAN is identified as the most relevant prior model for unconditional generation from one natural image.
- Single-image GAN approaches are relatively rare, and only a few models can train on one natural image.
- Some single-image GAN methods train only on texture images, while others target applications such as image-to-image translation with two training images.
- SinGAN is described as the only model capable of unconditional image generation after training on a single natural image.
- SinGAN learns image-patch statistics across scales using multiple resolutions, training each stage individually while freezing previous stages.
3. Methodology
ConSinGAN modifies single-image GAN training by jointly training selected stages, passing feature maps between them, and using progressive learning-rate and resolution schedules. These changes support fewer stages while preserving global layout and improving training behavior.
- Multi-stage Training: ConSinGAN trains multiple, but not all, generator stages concurrently while progressively reducing learning rates at lower stages.Training all stages overfits by generating only the original image without variation.
- Multi-stage Training: Each newly added stage receives raw features from the previous stage, previous layers remain trainable, and residual connections link original features to the new convolutional layers.The default setting jointly trains the last three stages.
- Task-Specific Training: The training objective combines WGAN-GP adversarial loss with reconstruction loss, using a downsampled original image for reconstruction at each stage.The reconstruction-loss weight is α = 10 in the reported experiments.
- Task-Specific Training: Task-specific generator inputs can replace random noise when the desired task is known beforehand, while the original image remains the reconstruction target.The paper reports this variation for tasks such as image harmonization.
- Image Rescaling: Aggressive downsampling such as r = 0.5 reduces the number of stages but causes generated images to lose global coherence.SinGAN typically uses eight to ten stages at 250-pixel width or height with r = 0.75.
- Image Rescaling: The modified rescaling pyramid concentrates stages at low resolution rather than using a strictly geometric progression.At least three low-resolution stages are usually needed for a target global layout, while relatively few high-resolution stages are needed.
4. Results
Experiments show that ConSinGAN captures global structure while training faster and with fewer stages than SinGAN, across unconditional generation and image harmonization. Its results also depend on training-stage configuration, learning-rate scaling, and task-specific fine-tuning.
- Unconditional Image Generation: ConSinGAN captures global structures in challenging images, including when test-time image resolution is modified.In the Stonehenge example, increasing image width adds stones and layers.
- Ablation: A learning-rate scale of δ = 0.1 produces more diverse images than δ = 0.5, while decreasing δ increases generated-image variance.For Marina Bay Sands, δ = 0.1 can add towers and change their appearance more drastically than δ = 0.5.
- Unconditional Image Generation: ConSinGAN retains better global coherence than SinGAN across the compared rescaling methods and numbers of trained stages.The rescaling technique benefits both models, regardless of the number of trained stages.
- Unconditional Image Generation: A diversity score of 0.54 indicates that ConSinGAN can learn the global structure of complex images.
- Unconditional Image Generation: ConSinGAN achieves a better SIFID of 0.06 than SinGAN’s 0.09 while using fewer stages and parameters.User-study performance was similar, but generated images often retained better global structure.
- Image Harmonization: ConSinGAN performs similarly to or better than SinGAN on image harmonization while training only three stages.It generally introduces fewer artifacts, and fine-tuning can further absorb inserted objects into the background colors.
5. Conclusion
ConSinGAN applies best practices for training single-image GANs, enabling efficient generation and multiple image-based tasks from one image.
- ConSinGAN trains at sequentially increasing resolutions, learning global image structure before later texture and stylistic details.
- The approach controls how closely generated images match the training image’s internal patch distribution by adjusting concurrent stages and lower-stage learning-rate scaling.
- A new image-rescaling approach enables training with fewer image scales than previous models.
- ConSinGAN can be trained on a single image for unconditional generation, harmonization, editing, and animation.
- The model is smaller and more efficient to train than previous models.