PaperlayerView exact revision on arXiv

Source-linked AI summary

Return of Unconditional Generation: A Self-supervised Representation Generation Method

Tianhong Li, Dina Katabi, Kaiming He

arXiv:2312.03701v4cs.CV

TL;DR

Unconditional generation seeks to model data without human labels, but it has underperformed conditional generation because labels provide semantic information. RCG generates self-supervised representations and uses them to condition image generation. It achieves 2.15 FID on ImageNet 256×256, reaching the tier of leading class-conditional methods.

  • Problem

    Unconditional generation aims to learn data distributions without human annotations, but its quality has lagged behind conditional generation on complex data.

  • Method

    RCG first generates representations in the space of a pre-trained self-supervised encoder, then conditions an image generator on those representations.

  • Results

    2.15 FID on ImageNet 256×256 makes RCG a state-of-the-art unconditional method whose results reach the tier of leading class-conditional methods.

  • Takeaways & Limitations

    RCG shows that self-supervised representations can provide semantic conditioning while preserving unconditional generation without human labels.

  • Takeaways & Limitations

    Adapting RCG to new conditioning requires retraining the lightweight representation generator, and the proposed downstream adaptation remains future work.

Abstract

from arXiv · show

Unconditional generation -- the problem of modeling data distribution without relying on human-annotated labels -- is a long-standing and fundamental challenge in generative models, creating a potential of learning from large-scale unlabeled data. In the literature, the generation quality of an unconditional method has been much worse than that of its conditional counterpart. This gap can be attributed to the lack of semantic information provided by labels. In this work, we show that one can close this gap by generating semantic representations in the representation space produced by a self-supervised encoder. These representations can be used to condition the image generator. This framework, called Representation-Conditioned Generation (RCG), provides an effective solution to the unconditional generation problem without using labels. Through comprehensive experiments, we observe that RCG significantly improves unconditional generation quality: e.g., it achieves a new state-of-the-art FID of 2.15 on ImageNet 256x256, largely reducing the previous best of 5.91 by a relative 64%. Our unconditional results are situated in the same tier as the leading class-conditional ones. We hope these encouraging observations will attract the community's attention to the fundamental problem of unconditional generation. Code is available at https://github.com/LTH14/rcg.

1 Introduction

Unconditional generation aims to learn data distributions without human annotations, but it has lagged behind conditional generation because labels provide semantic information. RCG addresses this gap by generating self-supervised representations and using them to condition image generation.

  • Unconditional generation has stagnated relative to conditional generation, whose labels or text descriptions provide semantic information and improve image-generation quality.Closing this gap matters for learning from large-scale unannotated data.
  • The framework treats self-supervised representations as semantic conditioning while preserving the paper’s definition of unconditional generation as not using human labels.The paper explicitly defines self-supervised learning as a form of unsupervised learning in this context.
  • RCG first generates a self-supervised representation, then conditions an image generator on that representation without human labels.A pre-trained encoder maps images into a compact representation space, where a lightweight unconditional generator models the representation distribution.
  • RCG improves unconditional quality across LDM-8, ADM, DiT-XL/2, and MAGE-L, reducing FID by 71%, 76%, 82%, and 51%, respectively.These comparisons are made across different image-generator architectures under the RCG framework.
  • 2.15 FID on ImageNet 256×256 lets RCG approach leading class-conditional generation while outperforming previous unconditional methods.The authors present this as an unprecedented unconditional-generation result.

2 Related Work

Prior work spans generative-model architectures, unconditional generation, representation-based image generation, and clustering-based conditioning. RCG differs by directly generating self-supervised image representations to enable representation-conditioned unconditional generation.

  • Generative-model families include GANs, two-stage token-space methods, and diffusion models, while conditioning design is described as mostly orthogonal to model design.The literature nevertheless reports a substantial unconditional-versus-conditional gap under the same design.
  • Unconditional generation models data distributions without human annotations but has recently lagged conditional generation on complex data distributions.The gap has limited recent progress despite applications including molecular design, medical imaging, and audio generation.
  • Prior gap-narrowing methods use representation-space clusters as class-like conditioning, but assume clusterability and cannot generate diverse representations within a cluster.Their assumptions include an optimal cluster count near the number of underlying classes.
  • RCG directly generates image representations from scratch, unlike approaches that convert text prompts into embeddings or learn representations jointly with an image decoder.This generated representation distribution enables conditioning without human labels.

3 Method

RCG decomposes image generation into unconditional generation of compact self-supervised representations followed by representation-conditioned image generation. Its representation diffusion model and flexible image generator improve unconditional generation across multiple architectures, while supporting label-free guidance and class-conditional extensions.

  • 3 Method: RCG decomposes complex image modeling into unconditional generation of compact representations followed by image generation conditioned on those representations.The framework uses a pre-trained self-supervised encoder, a representation generator, and an image generator.
  • 3 Method: The representation distribution is obtained by mapping images with an off-the-shelf self-supervised encoder, producing a simpler space for unconditional modeling.The supplied passage describes the encoder as trained with self-supervised contrastive learning and the representation distribution as sufficiently simple for generation.
  • 3.1 The RCG Framework: RDM unconditionally generates representations with a diffusion model built from fully connected residual blocks and timestep embeddings.Its backbone uses LayerNorm, SiLU, and linear layers, with block count N and hidden dimension C controlling the architecture.
  • 3.1 The RCG Framework: RDM corrupts representations with Gaussian noise, learns to denoise them, and samples representations from Gaussian noise using DDIM with marginal computation overhead.The generated representations provide semantic information to the subsequent image generator.
  • 3.1 The RCG Framework: The image generator replaces class or text conditioning with self-supervised representations, improving four representative generative models in unconditional generation.The experiments cover ADM, LDM, DiT, and MAGE; the supplied table caption specifies ImageNet 256×256 and the unconditional setting.
  • 3.2 Extensions: RCG also enables label-free guidance and class-conditional generation through representation conditioning and a task-specific conditional RDM.For guidance, MAGE combines conditioned and unconditioned logits; for class conditioning, a class embedding is added to each RDM block.

4 Experiments

RCG substantially improves unconditional ImageNet generation across generators, achieving state-of-the-art quality, lower training cost, and performance comparable to or better than class-conditional systems. Its representation-based conditioning also supports diverse outputs, semantic interpolation, and simple adaptation to class-conditional generation.

  • 4.1 Observations: RCG reduces unconditional FID by 71%, 76%, 82%, 54%, and 51% for LDM-8, ADM, DiT-XL/2, MAGE-B, and MAGE-L, respectively.The improvement is also reported across CIFAR-10 and iNaturalist.
  • 4.1 Observations: RCG achieves lower FID with less training cost, including MAGE-B reaching FID 4.87 in less than one day on 64 V100 GPUs.The authors attribute this efficiency to decomposing unconditional generation into simpler sub-tasks.
  • 4.1 Observations: FID reaches 2.15 with RCG-G and Inception Score reaches 253.4, significantly surpassing previous unconditional-generation methods on ImageNet 256×256.Unguided RCG reaches FID 3.44 and Inception Score 186.9.
  • 4.1 Observations: RCG’s unconditional performance rivals leading class-conditional methods, effectively bridging the historical gap on complex data distributions.The system-level comparison is reported in Table 3.
  • 4.1 Observations: With RCG, LDM-8, ADM, and DiT-XL/2 consistently outperform their class-conditional counterparts, indicating that generated representations can guide generation more effectively than class labels.This is reported as an apple-to-apple comparison in Table 4.
  • 4.1 Observations: RCG also supports class-conditional generation by retraining only a lightweight representation generator rather than the representation-conditioned image generator.This adaptation is presented as a simple extension, with potential downstream-training benefits.
  • 4.2 Qualitative Insights: Given one reference representation, RCG generates diverse images with aligned semantics, while interpolated representations yield realistic images whose semantics transition smoothly between endpoints.These properties support semantic control without sacrificing low-level diversity.

5 Discussion

The paper positions RCG as a bridge between unconditional and conditional generation by modeling self-supervised representations rather than relying on human annotations. It argues that this approach could extend image generation toward large unlabeled datasets and modalities beyond human annotation.

  • 5 Discussion: RCG bridges unconditional and conditional generation by generating images conditioned on representations sampled from a self-supervised representation space.The representation generator models and samples that space without human labels.
  • 5 Discussion: The authors suggest that RCG could help image generation exploit extensive unlabeled data and generalize to modalities beyond the scope of human annotation.This is presented as a potential implication rather than an experimentally established result.

A Implementation Details

The implementation uses a MoCo v3 ViT encoder, a fully connected representation diffusion model, and several established image generators. Training and preprocessing choices are specified for representation extraction, diffusion modeling, image generation, and guidance.

  • Image Encoder: The default encoder is a MoCo v3-pretrained ViT, using normalized 256-dimensional post-projection representations.ViT-S, ViT-B, and ViT-L variants are evaluated, each pretrained on ImageNet for 300 epochs.
  • Representation Diffusion Model (RDM): The representation diffusion model uses 12 fully connected blocks with hidden dimension 1536 and 1000 discretized timesteps.Timestep embeddings are 256-dimensional, while class-label embeddings for conditional RDM are 512-dimensional.
  • Image Generator: RCG experiments use ADM, LDM, DiT, and MAGE as image generators, replacing class embeddings with image representations where applicable.Training follows the corresponding generator recipes with stated changes for batch size, learning rate, and duration.
  • Image Generator: Images are resized, randomly flipped, and cropped to 256×256, while encoder inputs are resized to 224×224; guidance uses a linear schedule.Additional MAGE-specific hyperparameters are reported separately.

B.1 Ablations

Ablations examine the self-supervised encoder, representation generator, and image generator on ImageNet, while additional results report consistent improvements across CIFAR-10 and iNaturalist. The study defines default architectures and evaluation measures for these component analyses.

  • B.1 Ablations: The ablation study evaluates three core RCG components using a MoCo v3 ViT-B encoder, a 12-block 1536-dimensional RDM, and a MAGE-B generator as defaults.The default RDM is trained for 100 epochs and the default image generator for 200 epochs.
  • Distribution Mapping: Distribution-mapping ablations compare self-supervised encoders trained with MoCo v3, DINO, and iBOT.Table 7 is the designated comparison for encoder choices.
  • Representation Generation: Representation-generation ablations evaluate RDM architecture and training choices using representation Fréchet Distance between generated and ImageNet-training representations.The default configuration uses 12 blocks, 1536 hidden dimensions, and 250 sampling steps.
  • Image Generation: Image-generation ablations evaluate generator settings, including guidance scale, with FID measured using the ADM evaluation suite.Table 9 uses MAGE-B by default and evaluates guidance-scale variants with MAGE-L in one subsection.
  • B.1 Ablations: RCG consistently improves unconditional generation on CIFAR-10 and iNaturalist in addition to ImageNet.These cross-dataset results are summarized in Table 10.

B.2 Other Datasets

RCG improves unconditional image generation across datasets and achieves strong performance with relatively low computational cost. Its gains are especially pronounced on complex data distributions, while guidance and larger models improve recall at a slight precision cost.

  • B.2 Other Datasets: RCG consistently enhances unconditional image generation on CIFAR-10 and iNaturalist 2021, with larger gains on complex distributions such as iNaturalist than on CIFAR-10.The authors attribute this pattern to decomposing a complex distribution into representation and representation-conditioned image distributions.
  • Computational Cost: RCG achieves a much smaller FID with similar or lower computational cost than baseline methods.Table 11 reports the representation and image generators separately for parameters, training cost, and training epochs.
  • Computational Cost: 4.87 FID is achieved in less than one day of training, while representation-generator training and inference costs remain marginal relative to the image generator.The reported computational-cost analysis compares parameters, training costs, and generation throughput.
  • ImageNet Analysis: Larger models and guidance improve recall while slightly decreasing precision in unconditional ImageNet generation.The evaluation follows the ADM suite and reports FID, IS, precision, and recall.

C Additional Qualitative Results

Additional qualitative results show strong RCG image generation across unconditional and class-conditional settings, while also documenting failure cases.

  • C Additional Qualitative Results: RCG qualitative results cover class-unconditional generation, class-conditional generation, and comparisons with and without guidance.The accompanying examples are presented in Figures 9–13.
  • C Additional Qualitative Results: Figure 12 documents failure cases in which RCG, like other generative models, struggles with text, regular shapes, and realistic humans.The listed examples include keyboards and wheels as regular shapes.

D Limitations and Negative Impact

The paper acknowledges that RCG can produce unrealistic or low-quality outputs and identifies recurring difficulties with text, regular shapes, and realistic humans. It also discusses potential bias-related societal impacts of generative models.

  • D Limitations and Negative Impact: RCG can produce unrealistic or low-quality results, as documented in the paper’s limitations discussion.The authors refer readers to Appendix C for examples.
  • D Limitations and Negative Impact: RCG may still amplify biases present in internet data, although its unconditional framework does not depend on human-provided labels.The paper suggests this characteristic might mitigate some human-label bias, while presenting that possibility cautiously.
  • D Limitations and Negative Impact: RCG faces difficulty generating text, regular shapes such as keyboards and wheels, and realistic humans.These are presented as failure cases alongside otherwise strong qualitative results.

NeurIPS Paper Checklist

The checklist reports that the paper’s claims match its contributions, discusses limitations and societal impacts, and provides reproducibility information through appendices and released code. It also notes that error bars are omitted because of computational overhead.

  • Claims and Scope: The checklist finds that the abstract and introduction accurately reflect the paper’s contribution of unconditional generation rivaling state-of-the-art class-conditional methods.The justification explicitly characterizes the method as an unconditional image generation approach.
  • Limitations: The paper discusses limitations and reflects on assumptions, robustness, scope, and factors that can influence empirical performance.The checklist points to the limitations discussion and recommends articulating practical violations and implications.
  • Theory: The paper does not include theoretical contributions or theoretical results requiring assumptions and proofs.The theory checklist marks the relevant question as not applicable.
  • Reproducibility: The paper reports reproducibility support through Appendix A, experimental details, compute-resource information, and code availability.The checklist cites Appendix A and subsection B.3, while the code is available at the project repository.
  • Statistical Significance: Error bars are not reported because of the heavy computational overhead associated with statistical evaluation in generative modeling.This justification appears in the experiment statistical-significance checklist.
  • Societal Impact: The paper acknowledges potential negative societal impacts, including bias amplification and possible harms from generative-model use.The checklist guidance highlights harms from intended use, incorrect outputs, and misuse, as well as possible mitigation strategies.
Loading 2312.03701v4…