PaperlayerView exact revision on arXiv

Source-linked AI summary

Fractal Generative Models

Tianhong Li, Qinyi Sun, Lijie Fan, Kaiming He

arXiv:2502.17437v2cs.LGcs.CV

TL;DR

Pixel-by-pixel image generation is challenging, and existing methods have struggled to achieve satisfactory likelihood estimation and generation quality. The paper recursively composes generative models as modules, using autoregressive models as its example. It reports strong likelihood and generation quality on this task, including a likelihood result that exceeds the previous best autoregressive model cited.

  • Problem

    Existing pixel-sequence methods do not achieve satisfactory likelihood estimation and generation quality, leaving pixel-by-pixel image generation a challenging test of modeling structured data.

    Source: §1 · p. 2, §4.1 · p. 4

  • Method

    The framework recursively composes generative models as modules, with autoregressive models serving as generators that divide large distributions into conditional subsets.

    Source: §1 · pp. 1–2, §3.2 · p. 4

  • Results

    The framework achieves strong likelihood and generation quality on pixel-by-pixel images; its negative log-likelihood is 3.14 bits per dim versus 3.40 bits per dim for the previous best autoregressive model.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    The results suggest fractal generative models may suit data with intrinsic structure beyond one-dimensional sequential orders.

    Source: §6 · p. 8

    The autoregressive construction assumes each model handles a manageable sequence length k, with the total variable count organized as N = k^n.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

Modularization is a cornerstone of computer science, abstracting complex functions into atomic building blocks. In this paper, we introduce a new level of modularization by abstracting generative models into atomic generative modules. Analogous to fractals in mathematics, our method constructs a new type of generative model by recursively invoking atomic generative modules, resulting in self-similar fractal architectures that we call fractal generative models. As a running example, we instantiate our fractal framework using autoregressive models as the atomic generative modules and examine it on the challenging task of pixel-by-pixel image generation, demonstrating strong performance in both likelihood estimation and generation quality. We hope this work could open a new paradigm in generative modeling and provide a fertile ground for future research. Code is available at https://github.com/LTH14/fractalgen.

1. Introduction

The paper introduces fractal generative models, which recursively invoke generative models as modules to form self-similar architectures. Using autoregressive models as modules, it applies this framework to pixel-by-pixel image generation, where it reports strong likelihood estimation and generation quality.

1. Introduction

  • The framework recursively invokes generative models as modules, creating architectures with self-similarity across levels.The paper calls this framework a fractal generative model, by analogy with fractals formed through recursive rules.Source: §1 · p. 1
  • The paper connects modularization to natural structure: images can be viewed as sub-images that are themselves images, while brain development recursively builds larger networks from smaller ones.It also notes evidence of fractal or scale-invariant small-world organization in brain networks.Source: §1 · pp. 1–2
  • Autoregressive models instantiate the framework: parent blocks spawn child autoregressive blocks that further spawn blocks, producing self-similarity across levels.The design is inspired by fractal properties observed in biological neural networks and natural data.Source: §1 · p. 2
  • Existing methods that directly model pixel sequences do not achieve satisfactory likelihood estimation and generation quality, motivating a test on structured non-sequential data.The paper identifies images, molecular structures, proteins, and biological neural networks as examples of such data.Source: §1 · p. 2
  • The framework generates raw images pixel by pixel while achieving accurate likelihood estimation and high generation quality.The paper presents this as evidence of the framework's performance on a challenging image-generation task.Source: §1 · p. 2

2. Related Work

The related work situates fractal generative models among natural fractal patterns, hierarchical vision representations, and hierarchical generative approaches. It distinguishes the framework by recursively composing whole generative models and using this structure to generate large outputs.

Related work

  • A fractal is characterized by self-similarity across scales and is often constructed by a recursive generation rule.The paper relates this structure to natural examples such as clouds, tree branches, crystals, chromatin, and proteins.Source: §2 · p. 2
  • Natural data range from familiar fractal structures to near-fractal patterns, including images and biological neural networks.The paper describes near-fractal data as having multi-scale representations or patterns without strict self-similarity.Source: §2 · p. 2
  • Hierarchical vision methods build multi-scale representations, but the paper distinguishes fractal construction by its combination of recursion, divide-and-conquer, and self-similarity.Examples of hierarchical representations include feature pyramids and Swin Transformer's multi-scale windows.Source: §2 · pp. 2–3
  • Hierarchical generative methods include tokenizer-based two-stage models, MegaByte's global and local modules, cascaded diffusion, and scale-space autoregressive generation.The paper notes that tokenizer-free scale-space autoregressive methods can be prohibitively expensive because attention within each scale has quadratic computational cost.Source: §2 · p. 3
  • FractalNet recursively applies a small convolutional block for classification, whereas this framework recursively composes whole generative models to generate large outputs such as millions of image pixels.The paper presents these as different levels of modularization and contrasts FractalNet's low-dimensional logits with the framework's larger output sets.Source: §2 · p. 3

3. Fractal Generative Models

The paper develops fractal generative models by treating generative modules as recursive generators. Its autoregressive instantiation uses divide-and-conquer to model large joint distributions with manageable sequence lengths at each level.

3.1. Motivation and Intuition

  • A fractal generator maps an output at one level to multiple outputs at the next, allowing exponential output growth with a linear number of recursive levels.This structure is presented as suitable for high-dimensional data with relatively few generator levels.Source: §3.1 · p. 3
  • Fractal generative models use atomic generative modules as parametric generators, so a neural network learns the recursive rule from data.Combining recursive exponential output growth with neural generative modules is presented as a way to model high-dimensional non-sequential data.Source: §3.1 · p. 4

3.2. Autoregressive Model as Fractal Generator

  • The autoregressive instantiation recursively partitions a joint distribution into conditional subsets, with each model handling a manageable sequence length k across n levels.Under the stated setup, N = k^n; recursive decomposition reduces computational costs relative to one large autoregressive model and captures hierarchical structure.Source: §3.2 · p. 4

4. An Image Generation Instantiation

This section instantiates fractal generative models for pixel-by-pixel image generation, a challenging testbed for modeling high-dimensional, structured data without an intrinsic sequence. Its architecture recursively applies autoregressive generators to image patches and pixels.

4.1. Pixel-by-pixel Image Generation

  • Pixel-by-pixel generation is difficult because raw images are high-dimensional and existing methods often fail to generate satisfactory images, while structured non-sequential data extends beyond images.The section presents molecular configurations, proteins, and biological neural networks as examples of such data.Source: §4.1 · p. 4

4.2. Architecture

  • Each level conditions a transformer on the previous generator’s output and image-patch tokens, recursively subdividing patches and reducing model width and depth toward lightweight autoregressive RGB prediction.The first generator uses a sequence length of 256.Source: §4.2 · pp. 4–5, Figure 3 · p. 5
  • FractalAR uses a raster-order, GPT-like causal transformer, whereas FractalMAR uses a random-order, BERT-like bidirectional transformer; both follow next-token prediction.The paper discusses their respective advantages and disadvantages in Appendix B.Source: §4.2 · p. 5

4.3. Relation to Scale-space Autoregressive Models

  • At 256×256, fractal attention is 4000× more computationally efficient than scale-space autoregression at the finest resolution, enabling high-resolution pixel-by-pixel modeling.Scale-space models attend over the full sequence of next-scale tokens; the fractal model attends to small 4×4 patches.Source: §4.3 · p. 5

4.4. Relation to Long-Sequence Modeling

  • Unlike long-sequence methods, the framework treats intrinsically structured data as sets and recursively models smaller subsets with generative submodules.The motivation is that images, molecules, and biological neural networks can be viewed as compositions of smaller structures.Source: §4.4 · p. 6

4.5. Implementation

  • Training proceeds breadth-first end-to-end on raw pixels, while generation traverses the architecture depth-first from 16×16 patches through 4×4 patches to individual pixels and RGB values.The final RGB values are sampled from autoregressively predicted logits.Source: §4.5 · p. 6
  • More fractal levels improve likelihood estimation while lowering computational costs on unconditional ImageNet 64×64.The table reports NLL in bits/dim.Source: Table 2 · p. 6
  • Both FractalAR and FractalMAR achieve competitive likelihood-based performance on unconditional ImageNet 64×64.The comparison reports NLL in bits/dim.Source: Table 3 · p. 6

5. Experiments

The experiments evaluate fractal generation on ImageNet at 64×64 and 256×256, using likelihood and image-quality measures. Results include strong likelihood performance, high-quality pixel-by-pixel generation, and conditional prediction across image-editing tasks.

5. Experiments

  • The evaluation covers unconditional and class-conditional ImageNet generation at 64×64 and 256×256, assessing likelihood, fidelity, diversity, and generation quality.Reported measures include NLL, FID, Inception Score, Precision, and Recall.Source: §5 · p. 6

5.1. Likelihood Estimation

  • More fractal levels improve likelihood estimation while reducing computation: a two-level model requires over ten times the computation of the three-level model, and a single autoregressive model is infeasible to train.The authors suggest that more levels better capture images’ intrinsic hierarchical structure.Source: §5.1 · p. 7
  • 3.14 bits per dim, versus 3.40 bits per dim for the previous best autoregressive model; fractal variants also remain competitive with advanced diffusion methods.The comparison evaluates causal and masked autoregressive fractal generators on pixel-by-pixel image generation.Source: §5.1 · p. 7

5.2. Generation Quality

  • FID 6.15 and Inception Score 348.9: FractalMAR-H generates class-conditional 256×256 images at 1.29 seconds per image, with strong fidelity but relatively weaker FID and Recall.The authors suggest the weaker FID and Recall may reflect the challenge of modeling nearly 200,000 pixels pixel-by-pixel; throughput was measured at batch size 1,024 on one Nvidia H100 PCIe GPU.Source: §5.2 · pp. 7–8, Figure 4 · p. 7

5.3. Conditional Pixel-by-pixel Prediction

  • Inpainting, outpainting, uncropping, and class-conditional editing show that the method predicts masked pixels from unmasked regions and reflects class-label semantics in predictions.In the class-conditional editing example, a cat’s face is replaced with a dog’s face when conditioned on the dog class.Source: §5.3 · p. 8, Figure 5 · p. 8

6. Discussion and Conclusion

The discussion presents fractal generative models as a way to divide complex data distributions into manageable subproblems by modularizing existing generative models. It identifies possible misuse to produce disinformation or amplify biases as outside this paper’s scope.

6. Discussion and Conclusion

  • Fractal generative models suggest dividing complex data distributions into manageable subproblems by abstracting existing generative models into modular units.The authors consider them particularly well-suited to data with intrinsic structures beyond one-dimensional sequential orders.Source: §6 · p. 8
  • Misuse to produce disinformation or amplify biases is a potential negative societal consequence, but lies outside the paper’s scope.The paper does not discuss these considerations in detail.Source: §6 · p. 9

A. Implementation Details

The implementation details describe training and generation choices for fractal models, including high-resolution context inputs, optimization settings, and class-conditioning guidance.

Training

  • For ImageNet 256×256, a guiding pixel conditions each generator on the average pixel value predicted from the previous generator, providing global context before fine-grained prediction.The guiding pixel is used only in the 256×256 experiments.Source: A. Implementation Details · p. 11
  • Providing the next-level generator with the current patch and its four surrounding patches slightly increases sequence length but significantly reduces patch-boundary artifacts.The issue arises because the autoregressive model divides images into patches.Source: A. Implementation Details · p. 11
  • Training uses AdamW for 800 epochs, except FractalMAR-H for 600, with a batch size of 2,048 at 64×64 and 1,024 at 256×256.The stated weight decay and momenta are 0.05 and (0.9, 0.95); the learning rate uses 40 warmup epochs followed by cosine decay.Source: A. Implementation Details · p. 11

Generation

  • Classifier-free guidance combines conditional and dummy-class logits using a guidance scale; training replaces the class condition with a dummy token in 10% of samples.At inference, a linear guidance schedule is used for the first-level autoregressive model, and guidance-scale and temperature settings are swept for each conditional model.Source: A. Implementation Details · p. 11
  • Top-p sampling with threshold 0.0001 is applied to conditional logits before guidance to mitigate numerical instability when a pixel value has very small predicted probability.The two variants shown use raster-scan causal attention for AR and random-order bidirectional attention for MAR; both are valid fractal generators.Source: A. Implementation Details · p. 11, Figure 6 · p. 11

B. Additional Results

Additional results cover class-conditional ImageNet 64×64 generation, comparisons between AR and MAR variants, and the effect of pixel-channel modeling order. Further figures show qualitative generation and conditional prediction examples.

Class-conditional ImageNet 64×64

  • Class conditioning has negligible impact on NLL but substantially improves visual quality and FID on ImageNet 64×64; the model achieves performance competitive with classical generative models.The authors report this pattern as consistent with PixelCNN.Source: B. Additional Results · p. 11

AR and MAR variants

  • FractalMAR outperforms FractalAR overall; MAR’s bidirectional attention enables parallel prediction of multiple patches and is chosen for conditional 256×256 synthesis.AR uses key-value caching to accelerate generation, while MAR’s attention aligns more naturally with image modeling.Source: B. Additional Results · p. 11

Modeling Pixels

  • Different autoregressive channel orders yield similar NLL values and only slight FID variation, with no significant overall performance differences.The NLL in YCbCr space is not comparable with NLL in RGB space; the reported order comparison includes RGB, GRB, BGR, and YCbCr.Source: B. Additional Results · pp. 11–12, Table 6 · p. 12

Additional qualitative results

Loading 2502.17437v2…