Source-linked AI summary
FARMER: Flow AutoRegressive Transformer over Pixels
Guangting Zheng, Qinyu Zhao, Tao Yang, Fei Xiao, Zhijie Lin, Jie Wu, Jiajun Deng, Yanyong Zhang, Rui Zhu
TL;DR
Modeling explicit likelihoods directly on raw pixels is challenging because continuous autoregressive models face very long sequences. FARMER combines invertible autoregressive flows with autoregressive latent modeling and adds channel reduction, distillation, and resampling-based guidance; on class-conditional ImageNet 256×256, it reduces FID by 3.04 compared with JetFormer while remaining competitive with other model families.
Problem
Continuous autoregressive modeling over high-dimensional pixels is challenging because it requires extremely long sequences, while explicit likelihood modeling remains difficult.
Source: §1 · pp. 1–2Method
FARMER jointly trains an invertible autoregressive flow that maps images to latent sequences and an autoregressive model of those latents, with channel reduction, one-step distillation, and resampling-based guidance.
Source: §1 · pp. 2–3Results
On class-conditional ImageNet 256×256, FARMER reduces FID by 3.04 compared with JetFormer and achieves competitive performance against GANs, diffusion models, and autoregressive models.
Source: §4.2 · p. 10Takeaways & Limitations
FARMER demonstrates competitive image generation while supporting explicit likelihood estimation directly from raw pixels.
Source: §5 · p. 14The prior strategy of assigning redundant latent dimensions an independent standard Gaussian can discard information when those dimensions remain correlated with informative dimensions, and can restrict multimodal interactions.
Source: §3.3 · p. 7
Abstract
from arXiv · showhide
Directly modeling the explicit likelihood of the raw data distribution is key topic in the machine learning area, which achieves the scaling successes in Large Language Models by autoregressive modeling. However, continuous AR modeling over visual pixel data suffer from extremely long sequences and high-dimensional spaces. In this paper, we present FARMER, a novel end-to-end generative framework that unifies Normalizing Flows (NF) and Autoregressive (AR) models for tractable likelihood estimation and high-quality image synthesis directly from raw pixels. FARMER employs an invertible autoregressive flow to transform images into latent sequences, whose distribution is modeled implicitly by an autoregressive model. To address the redundancy and complexity in pixel-level modeling, we propose a self-supervised dimension reduction scheme that partitions NF latent channels into informative and redundant groups, enabling more effective and efficient AR modeling. Furthermore, we design a one-step distillation scheme to significantly accelerate inference speed and introduce a resampling-based classifier-free guidance algorithm to boost image generation quality. Extensive experiments demonstrate that FARMER achieves competitive performance compared to existing pixel-based generative models while providing exact likelihoods and scalable training.
1 Introduction
Explicit likelihood modeling on raw pixels is difficult because continuous autoregressive models face extremely long sequences, while standard-Gaussian flow targets can complicate sampling. FARMER combines invertible flows and autoregressive modeling, with channel reduction, one-step distillation, and resampling-based guidance addressing modeling complexity, inference speed, and image quality.
1 Introduction
- Continuous autoregressive pixel modeling faces extremely long sequences, while mapping dispersed image distributions to an isotropic Gaussian can complicate sampling.The paper frames these as complementary challenges in scaling likelihood modeling and image generation.Source: §1 · pp. 1–2
- FARMER jointly trains an autoregressive flow and an autoregressive model, mapping images to latent sequences whose distribution is modeled autoregressively.The framework retains tractable, exact likelihoods while using an autoregressive target rather than a fixed Gaussian.Source: §1 · p. 2
- Self-supervised dimension reduction partitions flow latents into informative and redundant channels, concentrating structure in the former and detail and color in the latter.The redundant channels across tokens are modeled jointly, effectively turning N high-dimensional tokens into N + 1 lower-dimensional tokens.Source: §1 · pp. 2–3
- One-step distillation learns a single-step student reverse path from the teacher’s forward path, avoiding the flow’s causal reverse process.The paper reports a 22× inference speedup with 60 additional training epochs while maintaining comparable generation quality.Source: §1 · p. 3
- A resampling-based classifier-free guidance algorithm is introduced to improve generation quality within FARMER.The introduction presents it as a separate generation-quality technique alongside one-step distillation.Source: §1 · p. 3
2 Preliminary
The preliminaries describe normalizing flows as invertible transformations that support exact density evaluation, and autoregressive models as products of next-token conditional probabilities. These sections introduce the likelihood foundations used by the paper’s modeling approach.
2.1 Normalizing Flows
- A normalizing flow maps data through a sequence of invertible functions to a latent distribution, enabling exact density calculation by change of variables.The flow is written as a forward map z = F(x) and its inverse x = F^-1(z).Source: §2.1 · p. 3
- Flow training commonly uses maximum likelihood, formulated as minimizing negative log-likelihood.The change-of-variables density includes the determinant of the transformation’s Jacobian.Source: §2.1 · p. 3
3 Approach
FARMER jointly trains an invertible autoregressive flow and an autoregressive model to estimate image likelihoods directly from pixels. It reduces latent-channel modeling complexity and adds resampling-based guidance and one-step distillation for sampling.
3.1 Mapping Image to AR Distributions via Invertible Flows
- FARMER maps images to latent sequences with an invertible flow whose distribution is modeled by an autoregressive model, trained jointly for exact likelihood estimation.The flow uses an autoregressive architecture, and the latent model represents token conditionals with Gaussian mixture models.Source: §3.1 · p. 4
3.2 Flow AutoRegressive Transformer
- The flow transforms image patches into latents, while a causal transformer models their distribution; its blocks use invertible next-token transformations.Autoregressive flow structure makes each block’s Jacobian lower triangular, allowing its determinant to be computed from diagonal scaling terms.Source: §3.2 · pp. 4–6, Figure 2 · p. 5
3.3 Self-supervised Dimension Reduction
- Channel-wise splitting models informative tokens individually and redundant tokens with a shared GMM conditioned on the informative sequence and context.The factorization retains dependence between the channel groups rather than assigning redundant channels an independent standard Gaussian prior.Source: §3.3 · pp. 6–7
3.4 Resampling-based Classifier-Free Guidance
- Resampling-based guidance approximates sampling from the intractable guided distribution by proposing conditional and unconditional candidates, weighting them, and resampling.Direct sampling is infeasible because the guided distribution combines products and sums of GMMs that do not form a known tractable distribution.Source: §3.4 · p. 7
3.5 Fast Inferring via One-Step Distillation
- One-step distillation trains a student’s forward path against the reversed teacher path, yielding 22× faster AF reversal and 4× overall inference speed-up.The student is trained end-to-end from the teacher’s forward path and requires 60 additional training epochs on the AF.Source: Figure 3 · p. 8, §3.5 · p. 8
4 Experiments
Experiments evaluate FARMER on class-conditional ImageNet 256×256 generation, where it outperforms JetFormer and compares favorably with several other model families. Ablations examine its channel partitioning, flow architecture, distillation, sampling guidance, and generation behavior.
4.1 Experimental Settings
- The experiments use ImageNet’s 1,281,167 training images at 256 × 256 resolution and evaluate image quality with FID, IS, precision, and recall on 50K generated samples.Two model scales, FARMER-1.1B and FARMER-1.9B, are tested.Source: §4.1 · pp. 9–10
4.2 Results
- FARMER reduces FID by 3.04 versus JetFormer on class-conditional ImageNet 256×256, and is competitive with GAN, diffusion, and AR models.The comparison applies resampling-based CFG to FARMER; the authors also report stronger quality than TARFlow and STARFlow.Source: §4.2 · p. 10, Table 2 · p. 10
- FARMER-1.9B generates diverse ImageNet images with fine-grained details, and its samples can reconstruct intricate features such as faces.The reported qualitative examples use resampling-based classifier-free guidance and compare class-0 images from FARMER, MAR, and DiT.Source: §4.2 · p. 11, Figure 4 · p. 11, Figure 5 · p. 11
4.3 Experimental Analysis
- The self-supervised dimension-reduction ablation improves FID from 61.17 to 49.29 and IS from 22.10 to 30.61; the complete method also reduces FID from 7.81 to 5.67 versus JetFormer’s channel approach.The ablation further reports FID 45.34 after repeating the class embedding, 44.56 after adding token permutation, and 5.67 after resampling-based CFG.Source: §4.3 · pp. 11–12, Table 3 · p. 12
- AF offers greater representational capacity than Jet but has a slow sequential reverse pass; one-step distillation reduces that pass from 0.1689 to 0.0076 seconds per image after 60 additional NF training epochs.Jet’s simpler transformations run quickly but limit its ability to separate image information into channel groups.Source: §4.3 · p. 12, Table 4 · p. 12
- FID is optimal at 64 GMM mixtures and informative dimension 128; reducing mixtures to 32 impairs dimension reduction, while larger informative dimensions make AR modeling and sampling more challenging.The authors select 64 mixtures and informative dimension 128 to balance quality and modeling or training cost.Source: §4.3 · p. 13, Figure 6 · p. 13
- Changing redundant-channel variance tunes sample diversity and color smoothness while leaving global structure largely unchanged; larger variance raises the risk of color artifacts or incoherent samples.Figure 7 illustrates the channel-group separation; abnormal log-determinant values are also associated with blurred images and missing fine details.Source: §4.3 · pp. 13–14, Figure 7 · p. 13, Figure 8 · p. 14
- One-step distillation cuts NF reverse time from 0.1689 to 0.0076 seconds per image and total inference time from 0.2189 to 0.0567 seconds, with comparable FID and IS.The reported FID values are 5.63 vs. 5.55 and IS values are 193.49 vs. 194.63; the NF reverse component is accelerated 22×.Source: §4.3 · p. 14, Table 5 · p. 14
5 Conclusion
FARMER combines invertible autoregressive flow and autoregressive modeling for end-to-end raw-pixel likelihood estimation and image synthesis. The paper reports competitive generation performance, while identifying dequantization and log-determinant complications as challenges for future work.
5 Conclusion
- FARMER integrates invertible AF with an AR model for end-to-end training on raw image pixels, enabling image synthesis and explicit likelihood estimation.Its supporting techniques include self-supervised dimension reduction, resampling-based CFG, and one-step distillation.Source: §5 · p. 14
- FARMER demonstrates competitive image-generation performance relative to pixel-based and latent generative models.Source: §5 · p. 14
- Dequantization relying on noise injection and complications from the log-determinant loss remain NF–AR challenges for future work.Source: §5 · p. 14
6 Related Work
Related work covers continuous autoregressive image modeling and autoregressive normalizing flows. These approaches differ in representation, generation pipeline, and how they model continuous tokens or transform them.
6.1 Continuous AR
- Discrete-token AR methods such as LlamaGen, Janus-Pro, and SimpleAR quantize images, while continuous-AR methods use VAE latents, Gaussian models, or pixel-space NF–AR.The cited discussion notes that quantization introduces information loss that limits generated-image fidelity.Source: §6.1 · p. 15
- Continuous-token methods combine AR with diffusion or flow-matching heads, but diffusion-based approaches require iterative denoising; FARMER directly fits the token distribution without noise sampling.Source: §6.1 · p. 15
6.2 Autoregressive Normalizing Flow
- Autoregressive normalizing flows transform tokens conditioned on preceding tokens; cited examples include IAF, MAF, neural autoregressive flows, T-NAF, TARFlow, STARFlow, and JetFormer.The passage describes TARFlow’s simplified log-determinant term and STARFlow’s extension into VAE latent space.Source: §6.2 · p. 15
A.1 FARMER reduces to Autoregressive Flow when K = 1
When FARMER’s GMM has one component, each sequence token is modeled by a conditional Gaussian based on preceding tokens. The resulting objective includes both token likelihood and a Jacobian volume correction.
A.1 FARMER reduces to Autoregressive Flow when K = 1
- With K = 1, FARMER models each token zi as a conditional Gaussian whose mean and variance depend on preceding tokens z<i.Source: §A.1 · p. 20
- The token objective combines its likelihood under a standard Gaussian with the log-determinant of the affine transformation’s Jacobian.The Jacobian term is the change-of-variables volume correction.Source: §A.1 · p. 20
- This K = 1 construction reduces FARMER to an Autoregressive Flow, with the AR model functioning as the final AF block.Source: §A.1 · p. 20
A.2 Resample-based CFG
The resampling-based CFG method exposes tunable temperatures and sample counts across its Propose, Weigh, and Resample stages. These controls adjust proposal diversity, scoring sharpness, and resampling selection.
A.2 Resample-based CFG
- Proposal diversity is controlled by component-weight and variance temperatures, while separate conditional and unconditional sample counts balance conditioned proposals with broader exploration.The proposal step draws candidates from conditional and unconditional GMMs.Source: §A.2 · p. 20
- Evaluation-distribution temperatures in the Weigh stage control the sharpness or smoothness of candidate scoring in log-probability space.These temperatures need not match the temperatures used during Propose.Source: §A.2 · p. 20
- A resampling temperature applied to normalized log-weights modulates selection sharpness; higher Ts emphasizes high-weight proposals, while lower Ts encourages diversity.The normalized weights define a categorical distribution for resampling.Source: §A.2 · p. 21
- Table 6 summarizes temperature and sampling configurations for the different model variants evaluated in this work.Source: Table 6 · p. 21, §A.2 · p. 21