PaperlayerView exact revision on arXiv

Source-linked AI summary

LAFITE: Towards Language-Free Training for Text-to-Image Generation

Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, Tong Sun

arXiv:2111.13792v3cs.CVcs.LG

TL;DR

High-quality image-text pairs are costly to collect and caption, motivating text-to-image training without text data. LAFITE generates pseudo text features from image features using CLIP's aligned multimodal space, and reports strong results across settings, including zero-shot COCO performance with less than 1% of the trainable parameter size of DALL-E and CogView.

  • Problem

    High-quality image-text pairs require costly human captioning and filtering, limiting affordable text-to-image model training for customized domains.

  • Method

    LAFITE generates pseudo text features from image features through CLIP's aligned multimodal space and feeds them to a text-to-image generator.

  • Results

    LAFITE reports state-of-the-art performance across text-to-image settings, including zero-shot COCO performance against DALL-E and CogView with less than 1% of their trainable model parameter size.

  • Takeaways & Limitations

    Language-free training is presented as a promising direction for broader text-to-image applications and can be used when only image data is available.

  • Takeaways & Limitations

    LAFITE relies on the off-the-shelf CLIP model, which was trained on 400 million image-text pairs.

Abstract

from arXiv · show

One of the major challenges in training text-to-image generation models is the need of a large number of high-quality image-text pairs. While image samples are often easily accessible, the associated text descriptions typically require careful human captioning, which is particularly time- and cost-consuming. In this paper, we propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model: the requirement of text-conditioning is seamlessly alleviated via generating text features from image features. Extensive experiments are conducted to illustrate the effectiveness of the proposed method. We obtain state-of-the-art results in the standard text-to-image generation tasks. Importantly, the proposed language-free model outperforms most existing models trained with full image-text pairs. Furthermore, our method can be applied in fine-tuning pre-trained models, which saves both training time and cost in training text-to-image generation models. Our pre-trained model obtains competitive results in zero-shot text-to-image generation on the MS-COCO dataset, yet with around only 1% of the model size and training data size relative to the recently proposed large DALL-E model.

1 State University of New York at Buffalo 2 Adobe Research 3 Microsoft Research, Redmond

The passage provides contact information for authors associated with the University at Buffalo.

  • The listed contact email uses the buffal​o.edu domain.
  • The email address lists three author-name identifiers.
  • The passage contains no research content beyond contact information.

1. Introduction

Text-to-image generation commonly depends on costly image-text datasets, motivating LAFITE's language-free approach and its reported efficiency and performance advantages.

  • Motivation: Over 70,000 workers gathered and annotated MS-COCO captions, illustrating the labor required for high-quality image-text pairs.
  • Motivation: DALL-E and CogView require billions of parameters, tens or hundreds of millions of image-text pairs, and extensive GPU training.
  • Motivation: Language-free training uses only image data, which is easier to collect than high-quality image-text pairs for domain-specific applications.
  • LAFITE: LAFITE uses CLIP image-text feature alignment to construct pseudo image-text feature pairs for a text-to-image GAN.
  • Contributions: LAFITE is presented as the first language-free text-to-image training system, evaluated across language-free, zero-shot, and fully supervised settings.
  • Results: Less than 1% trainable model parameter size accompanies LAFITE's reported zero-shot performance advantage over DALL-E and CogView on COCO.
  • Results: LAFITE outperforms several state-of-the-art methods by a large margin in fully supervised settings.

2. Related Work

Prior work covers supervised and zero-shot generation plus multimodal representation learning, while LAFITE applies aligned feature spaces to language-free text-to-image training.

  • Text-to-image generation: Text-to-image generation methods are categorized into fully supervised and zero-shot settings.
  • Text-to-image generation: DALL-E and CogView achieve zero-shot generation by training huge autoregressive Transformers on web-scale paired image-text data.
  • Text-to-image generation: LAFITE is described as the first generative adversarial approach achieving state-of-the-art zero-shot generation.
  • Multi-modal feature learning: CLIP and ALIGN scale multimodal learning toward universal image-text representation spaces with aligned visual and semantic features.
  • CLIP for generation/manipulation: Unlike prior CLIP-based generation or manipulation work, LAFITE targets caption-free text-to-image training and generates pseudo feature pairs in the joint multimodal space.

3. LAFITE: A Language-Free Paradigm

LAFITE replaces paired captions with pseudo text features generated from image features in CLIP’s aligned multimodal space, then conditions a StyleGAN2-based model on them. It supports fixed or trainable perturbations, layer-wise conditional generation, and discriminator objectives combining realism with semantic alignment.

  • Pseudo Text-Feature Generation: LAFITE generates pseudo text features from image features to avoid requiring image-caption pairs during language-free training.The alternative is motivated by the difficulty and cost of obtaining high-quality captions or training a universal captioning model.
  • Pseudo Text-Feature Generation: Pseudo features h′ are constructed near the corresponding image feature on a cosine-similarity hypersphere, using either fixed or trainable perturbations.The fixed scheme adds adaptive Gaussian noise; the trainable scheme learns noise-distribution parameters from image features.
  • Pseudo Text-Feature Generation: Adaptive Gaussian noise is normalized and rescaled by the image-feature norm, and the resulting generator can produce H(x) with high probability depending on ξ, c, and d.The perturbation level ξ, similarity threshold c, and feature dimension d determine the stated probability.
  • Pseudo Text-Feature Generation: Trainable perturbations generally perform better, whereas fixed perturbations are easier to use because they require no separately trained inference model with annotated image-text pairs.Trainable-perturbation performance is also influenced by the dataset gap between inference-model training and generative-model training.
  • Network Architectures: The generator transforms latent noise and pseudo text features into style and condition codes, concatenating them at each layer to modulate StyleGAN2 image generation.The conditional vector is mapped through a separate two-layer fully connected network for each generator layer.
  • Training Objectives: The discriminator combines a real-or-fake score with semantic alignment between the pseudo text feature h′ and an image embedding from its shared backbone.Additional contrastive regularization encourages discriminator image features to match corresponding pseudo text features.

4. Experiments

Experiments evaluate LAFITE across language-free, zero-shot, fully supervised, fine-tuning, semi-supervised, and ablation settings. Results show strong generation quality, training efficiency, and sensitivity to pretraining domain and objective design.

  • Language-free text-to-image generation: Both LAFITE variants significantly outperform the VinVL captioning baseline in language-free MS-COCO training, with LAFITEG performing best.The VinVL comparison is biased toward the baseline because its captioner was trained using MS-COCO image-text pairs.
  • Multi-modal conditions: LAFITE also supports image-conditioned generation, producing images similar to a given image while allowing simultaneous image and text conditioning.The method generates an image-conditioned pseudo text feature and passes it to the generator.
  • Zero-shot text-to-image generation: LAFITE achieves better quantitative results than DALL-E and CogView in most zero-shot MS-COCO cases with 75 million trainable parameters.DALL-E has over 12 billion parameters, while LAFITE uses a much smaller pretrained model and CC3M pretraining dataset.
  • Standard text-to-image generation: LAFITE consistently outperforms other methods and establishes new state-of-the-art results in standard fully supervised text-to-image generation.The comparison reports FID, IS, and Semantic Object Accuracy across multiple datasets.
  • Adaptation of pre-trained models: Fine-tuning generally outperforms training from scratch, but LAFITENN can underperform LAFITEG when fine-tuning datasets differ from CC3M.The authors attribute this pattern to biases in the inference model used to generate text features, which may overfit CC3M.
  • Adaptation of pre-trained models: Fine-tuning reaches FID 18 on MS-COCO in 3 hours, compared with around 4 days for training from scratch.The authors identify this efficiency as especially useful when training models across several datasets.
  • Ablation studies: Removing either LConG or LConD causes significant performance drops, supporting the importance of both training-loss components.The ablation compares both LAFITE variants on MS-COCO.

5. Conclusion

The paper presents LAFITE for text-to-image generation without domain-specific image-text pairs. It reports strong results across language-free, zero-shot, fully supervised, and fine-tuning settings, while identifying open-set, long-tail generation as future work.

  • Conclusion: LAFITE generates pseudo text features from images to build text-to-image systems without domain-specific image-text pairs.The approach targets language-free, zero-shot, and fully supervised generation settings.
  • Conclusion: LAFITE creates new state-of-the-art zero-shot results with only 1% of the trainable parameter counts of recent systems such as DALL-E and CogView.The conclusion also reports superior performance to prior art in fully supervised settings.
  • Future direction: The authors identify image synthesis in the wild under long-tail and open-set conditions as an interesting future direction.This setting extends beyond the evaluated domains and conditions described in the conclusion.

A.1. Theoretical Results

Theorem 1 derives a lower bound on the probability that LAFITEG’s generated text feature remains sufficiently similar to the corresponding image feature. The proof reduces this event to the inner product of normalized random vectors and evaluates it using the sphere inner-product distribution.

  • Theorem 1: Theorem 1 bounds the probability that Sim(fimg(xi), h′i) is at least c for a given threshold c > 0.The stated result concerns the text feature generated by LAFITEG.
  • Proof: The similarity is lower-bounded by (1 + ξa^T b)/(1 + ξ) using the triangle inequality.The bound follows after rewriting the perturbed feature in normalized coordinates.
  • Theorem 1: The probability bound becomes 1 − ∫ from −1 to (c−1)/ξ+c of the sphere inner-product density.The density uses Gamma-function terms and depends on the feature dimension d.
  • Proof: The derivation concludes by applying the cumulative distribution function of inner products between random vectors on the sphere.The proof explicitly invokes this CDF to complete the probability calculation.

A.2. Experiment Details

The experiment details describe dataset-statistics reporting, image-feature extraction with random cropping, selected hyper-parameters, and an implementation sharpening applied before softmax. Additional materials include generated examples and discriminator ablations.

  • Image feature extraction: Algorithm 2 extracts image features from an image dataset using a pre-trained image encoder and optional random-crop augmentation.The algorithm samples crop sizes from [a, w] and averages augmented features before generating pseudo text features.
  • Image feature extraction: The implementation sets k = 1 and a = 256 for features used to generate h′, but a = 128 in contrastive loss (7).These settings apply to the two stated feature-extraction uses.
  • Hyper-parameter: Hyper-parameters τ, λ, and γ are selected from the specified candidate sets using MS-COCO performance.τ is selected from [0.1, 0.2, 0.5, 1.0, 2.0], while λ and γ use [0, 1, 2, 5, 10, 20, 50].
  • Implementation: An extra exponential operation is applied before softmax in contrastive-loss equations (6) and (7) to make reproduction easier.The authors provide an implementation repository.

A.3. More Results

The additional results cover generated examples, multimodal conditioning, and a discriminator ablation. Multimodal generation combines image-derived and text-derived conditions, while image conditioning preserves semantic characteristics without performing reconstruction.

  • Generated examples: Generated examples are provided for CUB, MS-COCO, MM CelebA-HQ, and LN-COCO under the language-free setting.Figures 8–11 show examples for these four datasets.
  • Generation with multi-modal conditions: Multimodal generation extracts text and pseudo text features, converts them into conditional style codes, and randomly combines corresponding code elements.The combined style code is then fed into the generator.
  • Generation with multi-modal conditions: Image-conditioned generation is not reconstruction, so generated images may differ from the reference while sharing semantically meaningful visible characteristics.The stated boundary applies when only a reference image is provided.
  • Ablation study on discriminator: Both discriminator terms are important in the language-free setting, with the “real or fake” term appearing more important.The comparison is reported from the discriminator ablation in Table 8.
  • Generation with multi-modal conditions: Figures 12–15 present multimodal generations conditioned on both image and text for MS-COCO, CUB, MM CelebA-HQ, and LN-COCO.The figures cover the same four datasets used for the generated examples.
Loading 2111.13792v3…