Source-linked AI summary
An Acceleration Framework for High Resolution Image Synthesis
Jinlin Liu, Yuan Yao, Jianqiang Ren
TL;DR
High-resolution GAN training demands substantial time and computational resources. The paper trains a generator to model compact latent codes rather than large images, reporting faster training and improved sample quality, including a fivefold speedup at 1024×1024.
Problem
Training high-resolution GANs is time-consuming and computationally intensive, often requiring high-end graphics hardware and long runs.
Source: §1 · p. 1Method
The framework trains encoder and decoder networks to map images to compact latent codes, trains a code generator, then decodes generated codes into images.
Source: §1 · pp. 1–2Results
At 1024×1024, training was 5 times faster than before, and generated-sample quality improved.
Source: §1 · p. 2, §4.2 · p. 5Takeaways & Limitations
The framework makes high-resolution synthesis possible with less training time and limited hardware resources.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Synthesis of high resolution images using Generative Adversarial Networks (GANs) is challenging, which usually requires numbers of high-end graphic cards with large memory and long time of training. In this paper, we propose a two-stage framework to accelerate the training process of synthesizing high resolution images. High resolution images are first transformed to small codes via the trained encoder and decoder networks. The code in latent space is times smaller than the original high resolution images. Then, we train a code generation network to learn the distribution of the latent codes. In this way, the generator only learns to generate small latent codes instead of large images. Finally, we decode the generated latent codes to image space via the decoder networks so as to output the synthesized high resolution images. Experimental results show that the proposed method accelerates the training process significantly and increases the quality of the generated samples. The proposed acceleration framework makes it possible to generate high resolution images using less training time with limited hardware resource. After using the proposed acceleration method, it takes only 3 days to train a 1024 *1024 image generator on Celeba-HQ dataset using just one NVIDIA P100 graphic card.
1. Introduction
High-resolution GAN training is time- and computation-intensive. The proposed framework shifts generation to small latent codes, reporting faster training and improved sample quality.
1. Introduction
- The reported 1024 × 1024 outputs were generated after three days on one P100.Source: Figure 1 · p. 1
- At 1024×1024, prior single-card training took 14 days for [6] and 41 days for [7].The paper also reports that [6] used eight Tesla V100 GPUs for four days at this resolution.Source: §1 · p. 1
- The framework encodes high-resolution images into small latent codes, trains a code generator, and decodes generated codes into images.The encoder and decoder are trained before the code generator.Source: §1 · pp. 1–2
- Smaller codes reduce generator and discriminator layers and feature-map sizes, while fully convolutional encoder–decoder networks can be trained quickly on low-resolution images.For different resolutions, the encoder and decoder need to be trained only once; GAN training avoids high-resolution images at every step.Source: §1 · p. 2
- Across resolutions, the proposed framework reports faster training and improved generated-sample quality.The introduction does not quantify the speedup.Source: §1 · p. 2
2. Related Work
Prior work addresses high-resolution GAN instability through several training and architecture techniques. This framework instead uses an encoder–decoder and latent-code generator, with a training setup distinct from VAEs.
2. Related Work
- Prior high-resolution GAN work addresses instability through gradient penalties [4], spectral normalization [10], self-attention [18], and progressive growth [6] and [7].Source: §2 · p. 2
- The method accelerates GANs without changing network structure or loss, adding encoder and decoder networks to shift generation into latent codes.Source: §2 · p. 2, Figure 2 · p. 2
- Unlike VAEs, this framework trains encoder and decoder networks independently on low-resolution images, fixes their parameters, then trains a generator on encoded codes.The passage contrasts this with VAEs, which use full-resolution images in every training step and constrain latent codes to a distribution such as the standard normal.Source: §2 · p. 2
3. The Acceleration Framework
The framework replaces direct high-resolution image generation with latent-code generation: an encoder/decoder pair maps between images and codes, and a GAN learns the code distribution. The decoder converts generated codes back to images.
3. The Acceleration Framework
- The framework trains an encoder/decoder to map images and codes, then trains a GAN on codes and decodes generated codes into images.This shifts the generation network from large images to smaller latent codes.Source: §3 · pp. 2–3, §3.2 · p. 3
3.1. Encoder and Decoder Networks
- Encoder F downsamples images to codes and decoder H upsamples codes back; tanh constrains codes to [-1,1], while training uses L1 reconstruction and adversarial losses.The image discriminator predicts whether images are real.Source: §3.1 · p. 3
3.2. Code Generation Networks
- The code GAN uses a near-standard image generator/discriminator architecture, outputs latent-code dimensions, and is trained with WGAN-GP.The code values share the common generator/discriminator range of [-1,1].Source: §3.2 · pp. 3–4
3.3. Implementation Details
- The implementation uses h/4 × w/4 × 16 codes as a balance and trains fully convolutional encoder/decoder networks once on 128 × 128 images.Normalization is omitted in the encoder and decoder; pixel normalization is used after generator convolutions to prevent possible signal-magnitude escalation.Source: §3.3 · p. 4
4. Experiments
The experiments compare the framework with conventional image generation across resolutions, assessing training speed and sample quality. They report faster training, improved quality at higher resolutions, and results on additional datasets.
4.1. Experimental Settings
- The 256 × 256 network comparison uses latent-code outputs instead of image outputs; experiments use CelebA-HQ at three resolutions and evaluate quality with FID.The dataset contains 30,000 faces, and all experiments run on one NVIDIA P100 with 16 GB memory.Source: §4.1 · p. 5
4.2. Training Speed
- At 1024 × 1024, acceleration makes training 5 times faster, enabling a generator to train in 3 days on one NVIDIA P100; all three resolutions exceed 2× speedup.The comparison measures generative-network training speed; Table 3 reports one-epoch running time for 30,000 images.Source: §4.2 · p. 5, Table 3 · p. 6
4.3. Qualitative Evaluation
- After acceleration, samples are reported as comparable at 256 × 256, better with fewer artifacts at 512 × 512, and significantly better at 1024 × 1024.The evaluation describes the accelerated samples as good at all three resolutions and generally more natural with fewer artifacts.Source: §4.3 · p. 6, Figure 5 · p. 6, Figure 6 · p. 6
4.4. Quantitative Evaluation
- FID decreases at all three resolutions, with significant improvements at 512 × 512 and 1024 × 1024.The results identify baseline failures at higher resolutions while reporting improvements at all tested resolutions.Source: Table 4 · p. 6, §4.4 · p. 7
4.5. Other Datasets
- On LSUN at 256 × 256, the framework reuses encoder and decoder networks trained on CelebA-HQ and improves sample quality across the tested datasets.Figure 8 shows LSUN bedroom samples after acceleration.Source: §4.5 · p. 8, Figure 8 · p. 8
5. Limitations and Discussions
The framework improves the shown generated samples, but its simple generator remains below recent advanced image-generation structures. The discussion also examines code-size choices and latent-space interpolation behavior.
5. Limitations and Discussions
- The relatively simple generative network produces images below recent advanced structures such as [7]; reaching their quality requires a more complicated network.Source: §5 · p. 8
- Figure 9's 256 × 256 LSUN church samples show improved quality after acceleration, without manual selection.Source: Figure 9 · p. 8
- The experiments use h/4 × w/4 × 16 codes for best image quality, while h/16 × w/16 × 16 is fastest and still has much less FID after acceleration.Source: §5 · p. 8
- Figure 10's input-noise latent interpolations change smoothly, which the authors say shows the framework does not merely memorize training samples.Source: §5 · p. 8, Figure 10 · p. 8
6. Conclusion
The framework first transforms images into small latent codes, then trains a code generator to produce them. The authors report faster training and improved image quality, with less training time and limited hardware resources.
6. Conclusion
- The framework trains encoder and decoder networks to transform images into small latent codes, then trains code-generation networks to generate codes; experiments report faster training and improved image quality.Source: §6 · p. 9