Source-linked AI summary
TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Network
Ayushman Dash, John Cristian Borges Gamboa, Sheraz Ahmed, Marcus Liwicki, Muhammad Zeshan Afzal
TL;DR
Text-to-image synthesis must produce realistic, semantically aligned, and diverse images, but combining text conditioning with class information had not been explored. TAC-GAN combines text-conditioned generation with auxiliary class information and, on Oxford-102 flowers, reports a 7.8% inception-score improvement over StackGAN alongside diverse samples.
Problem
Text-to-image synthesis must produce realistic, semantically faithful, and diverse images, while combining textual conditioning with class information had not been explored.
Method
TAC-GAN extends AC-GAN by conditioning generated images on text embeddings while retaining auxiliary class information in the model.
Results
7.8% improvement in inception score over state-of-the-art models is reported for TAC-GAN, with diverse samples and smoothly varying style under fixed content.
Takeaways & Limitations
TAC-GAN generates images whose content follows text descriptions while separating image content from style and producing diverse samples.
Takeaways & Limitations
The effects of conditioning on other information types and their impact on training stability and image quality remain to be examined.
Abstract
from arXiv · showhide
In this work, we present the Text Conditioned Auxiliary Classifier Generative Adversarial Network, (TAC-GAN) a text to image Generative Adversarial Network (GAN) for synthesizing images from their text descriptions. Former approaches have tried to condition the generative process on the textual data; but allying it to the usage of class information, known to diversify the generated samples and improve their structural coherence, has not been explored. We trained the presented TAC-GAN model on the Oxford-102 dataset of flowers, and evaluated the discriminability of the generated images with Inception-Score, as well as their diversity using the Multi-Scale Structural Similarity Index (MS-SSIM). Our approach outperforms the state-of-the-art models, i.e., its inception score is 3.45, corresponding to a relative increase of 7.8% compared to the recently introduced StackGan. A comparison of the mean MS-SSIM scores of the training and generated samples per class shows that our approach is able to generate highly diverse images with an average MS-SSIM of 0.14 over all generated classes.
1. Introduction
Text-to-image synthesis must produce images that are both realistic and semantically faithful, while also remaining diverse. TAC-GAN extends AC-GAN by conditioning generation on text and reports improved discriminability and diversity.
- Realistic, semantically aligned, and diverse image synthesis remains difficult because generated images must capture text meaning while preserving visual quality.
- AC-GAN generates images conditioned on class labels and trains its discriminator to classify both source and class.
- TAC-GAN conditions image generation on text descriptions while retaining auxiliary class information from AC-GAN.The generator uses noise and embedded text, while the discriminator also receives text information before classification.
- 7.8% improvement in inception score over state-of-the-art models is reported for TAC-GAN.The evaluation uses the Oxford-102 flower dataset; Figure 1 presents generated examples.
2. Related Work
Prior work developed generative models, adversarial architectures, and text-conditioned image synthesis, including multi-stage methods for higher-resolution outputs. These approaches motivate TAC-GAN’s combination of text conditioning with class-aware generation.
- Image synthesis from text differs from inpainting and texture generation because it does not rely on an existing image.
- Earlier generative approaches include VAEs and DRAW, which approximate training distributions or use recurrent attention for sequential image construction.
- GANs have been applied to image synthesis tasks, but basic architectures can produce blurry images whose quality decreases as resolution increases.
- Multi-stage architectures iteratively refine generated images, separating or progressively improving image structure and style.
- Text-conditioned GANs made synthesized images correspond to input descriptions, while StackGAN used multiple stages to generate 256 × 256 images.
3. Background
GANs train a generator and discriminator with conflicting objectives, while AC-GAN adds class prediction to source discrimination. Their objectives separately represent source realism and class correctness.
- Generative Adversarial Networks: GAN training jointly optimizes a generator that models the data distribution and a discriminator that distinguishes real from generated samples.
- Auxiliary Classifier Generative Adversarial Networks: AC-GAN conditions generation on class labels and makes the discriminator predict both whether images are real and which class they represent.
- Auxiliary Classifier Generative Adversarial Networks: AC-GAN separates its objective into correct-source likelihood LS and correct-class likelihood LC; the discriminator maximizes both while the generator minimizes LC − LS.
4. Text Conditioned Auxiliary Classifier Generative Adversarial Network (TAC-GAN)
TAC-GAN is an AC-GAN variant that synthesizes 128 × 128 images from text embeddings rather than class labels. Its implementation uses the Oxford-102 flower dataset, DCGAN components, and Skip-Thought text embeddings.
- TAC-GAN synthesizes 128 × 128 images that comply with the content of input text.
- Training uses Oxford-102 flowers, with class labels and at least five text descriptions per image.
- The implementation uses a DCGAN generator and discriminator together with Skip-Thought vectors for text embeddings.
4.1. Model Architecture
TAC-GAN extends AC-GAN by conditioning image generation on text embeddings rather than class labels, while retaining auxiliary class discrimination. Its generator combines text information with noise, and its discriminator evaluates image source, class, and text-image correspondence.
- 4.1. Model Architecture: TAC-GAN replaces AC-GAN's class-label conditioning with text embeddings while retaining auxiliary class classification.The model is introduced as a text-conditioned variant of AC-GAN.
- 4.1.2 Generator Network: The generator concatenates a text representation with noise, transforms the result into a convolutional representation, and produces a 128 × 128 × 3 image.The combined representation is passed through a fully connected layer before transposed convolutions generate the fake image.
- 4.1.3 Discriminator Network: The discriminator receives real, fake, and mismatched images paired with class labels and text embeddings to distinguish sources and classify image classes.The mismatched image provides a negative example for text-image correspondence.
- 4.1.3 Discriminator Network: The discriminator spatially replicates the text embedding, concatenates it with image features, and uses separate outputs for source and class distributions.The source output distinguishes real from generated inputs, while the class output predicts class labels.
- 4.1. Model Architecture: The architecture is designed to support text-conditioned image synthesis while using class information to structure discriminator training.The discriminator design is inspired by GAN-CLS, and its text and image-processing components are trained with TAC-GAN.
4.2. Implementation Details
The implementation uses convolutional generator and discriminator networks with fixed filter configurations and Adam optimization. An interpolation experiment varies text embeddings while holding noise vectors fixed to examine content changes across generated images.
- 4.2. Implementation Details: The generator uses three transposed convolutional layers with 256, 128, and 64 filter maps, while the discriminator uses three convolutional layers with 128, 256, and 384 filter maps.The convolutional layers use 5 × 5 kernels until the discriminator feature map is generated.
- 4.2. Implementation Details: With fixed noise vectors, interpolating between text embeddings produces image sequences that vary in content while using the same underlying noise conditions.Figure 5 uses one noise vector for the first two rows and another for the third and fourth rows.
- 4.2. Implementation Details: Adam optimization is used, and the discriminator's final convolutional layer has 512 1 × 1 filter maps with stride 1.The model uses same convolutions throughout training.
4.3. Training
TAC-GAN training combines source, class, and generator objectives, with losses summed across available information types. The framework can be extended by adding corresponding discriminator and generator losses for additional information.
- Training objectives: TAC-GAN trains the discriminator with source and class objectives, minimizing the combined loss LDC + LDS.Source loss distinguishes real, fake, and wrong inputs; class loss addresses the expected class.
- Training objectives: The generator minimizes LGC + LGS using the discriminator’s generated-image output and the expected class.The generator receives no real or wrong images during training.
- Training objectives: LDS drives the discriminator to classify generated images as fake, whereas LGS drives the generator toward images judged as real.The two losses use opposite target values for the discriminator’s source output.
- Extending the current model: Training losses are sums of terms for different information types, allowing new dataset-specific losses to be added to both networks.For an additional dataset Y, the related losses LDY and LGY can be incorporated into the discriminator and generator objectives.
5. Evaluation
The evaluation compares TAC-GAN with prior text-to-image approaches using Inception Score for discriminability and MS-SSIM for diversity. TAC-GAN is reported to score better than other state-of-the-art approaches and to generate more diverse images than the training data.
- Qualitative comparison: Figure 6 compares TAC-GAN and StackGAN outputs across Phase-I and Phase-II for the same text captions.The figure presents synthesized images organized by caption and generation phase.
- Evaluation metrics: The evaluation uses Inception Score to assess discriminability and MS-SSIM to assess sample diversity.The two metrics are applied to complementary properties of the generated images.
- Discriminability: TAC-GAN produces better Inception Scores than other state-of-the-art text-to-image approaches on Oxford-102.The comparison is reported in Table 2.
- Diversity: TAC-GAN produces more diverse images than the Oxford-102 training data according to class-wise mean MS-SSIM comparisons.Figure 7 compares mean MS-SSIM values for each training and generated class.
6. Analysis and Discussion
TAC-GAN separates content and style representations in generated images. Holding text or noise fixed while interpolating the other input produces smooth changes in the corresponding content or style.
- Style-content separation: Interpolating between noise vectors while holding text fixed changes style smoothly while leaving image content roughly unchanged.This behavior supports separate representations for style and content.
- Style-content separation: Interpolating between text embeddings while holding the noise vector fixed preserves style while smoothly transitioning image content.The text embeddings are produced from the descriptions and interpolated directly.
7. Conclusion and Future Work
TAC-GAN generates images from textual descriptions and achieves slightly better results than other state-of-the-art approaches. Future work includes testing other conditioning information and multi-stage refinement pipelines.
- TAC-GAN generates images from textual descriptions and produces slightly better results than other state-of-the-art approaches.
- The effect of conditioning on other types of information remains unexamined, including its influence on training stability and image quality at higher resolutions.
- A multi-staged refinement pipeline may further improve TAC-GAN's reported results.