PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

Wenda Chu, Bingliang Zhang, Jiaqi Han, Yizhuo Li, Linjie Yang, Yisong Yue, Qiushan Guo

arXiv:2605.00503v2cs.CVcs.LG

TL;DR

Autoregressive image generation needs tokenizers whose representations support generation, but next-token prediction does not determine final pixel-space quality. The paper jointly trains a 1D tokenizer and autoregressive model with reconstruction and semantic supervision, achieving a state-of-the-art gFID of 1.48 without guidance.

  • Problem

    Separate tokenizer and generator training, together with next-token prediction's mismatch with final generation quality, leaves the tokenizer poorly optimized for autoregressive generation.

  • Method

    The paper jointly optimizes a 1D tokenizer and autoregressive model using autoregressive prediction reconstruction and semantic vision-foundation-model alignment.

  • Results

    1.48 gFID without guidance is achieved on ImageNet 256x256 generation by the scaled EOSTok-H model.

  • Takeaways & Limitations

    EOSTok learns a 1D sequential token space that facilitates autoregressive modeling and improves overall generation quality.

  • Takeaways & Limitations

    The next-token-prediction loss cannot determine final generation quality because it is defined on discrete latent sequences that change during training.

Abstract

from arXiv · show

Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.

1 Introduction

The paper develops an end-to-end autoregressive image-generation pipeline centered on an Autoregressive Prediction Reconstruction loss. It also investigates vision foundation models for regulating latent spaces and improving autoregressive visual generation.

  • Motivation: Recent work explores autoregressive visual generation with discrete visual tokens as an alternative to diffusion-based approaches.
  • End-to-end training: The Autoregressive Prediction Reconstruction loss decodes teacher-forcing predictions from the autoregressive model into pixel space and computes reconstruction loss.This directly connects autoregressive generation predictions to tokenizer reconstruction.
  • End-to-end training: End-to-end training improves final generation quality and makes the latent space more autoregressive-predictable.
  • Vision foundation models: The work investigates vision foundation models to regulate tokenizer latent spaces and improve autoregressive visual generation.This direction is motivated by prior use of vision foundation models for regulating 2D tokenizer latent spaces and improving diffusion models.

2 Related Work

Related work spans discrete image tokenizers, autoregressive visual generation, and vision-foundation-model-based representations. Image generation has progressed from spatially aligned 2D token prediction toward masked, next-scale, and learnable-query-token approaches, while pretrained semantic features have been used to improve learned representations.

  • Image tokenizers: VAEs learn continuous latent spaces through reconstruction and KL losses, whereas VQ-VAEs map images to discrete representations using vector quantization.
  • Autoregressive visual generation: Early autoregressive image models predict spatially aligned tokens in raster-scan order, but bidirectional token dependencies make this strategy suboptimal.
  • Autoregressive visual generation: MaskGIT and MAR use masked autoregressive modeling with bidirectional attention, while VAR performs autoregressive next-scale prediction.
  • Autoregressive visual generation: TiTok and SEED extract 1D image representations with learnable query tokens appended to 2D image patches in a vision transformer encoder.
  • Representation from vision foundation models: Vision foundation model features from DINO and CLIP have improved diffusion models through latent-space or intermediate-layer alignment.
  • Representation from vision foundation models: Another line of work replaces the VAE encoder with a frozen pretrained vision encoder, optionally adding a lightweight learnable adaptation module.

3 Method

The method develops a discrete 1D ViT tokenizer for autoregressive image generation and jointly trains it with the generator using end-to-end pixel-space supervision. It further investigates vision foundation model representations to improve semantic tokenization and autoregressive generation.

  • 1D ViT tokenizer: The tokenizer flattens image patches, appends learnable query tokens, and uses a causal ViT encoder to retain a discrete 1D latent sequence compatible with autoregressive modeling.The encoder discards hidden patch embeddings after producing the sequential latent representation, and vector quantization yields discrete codes.
  • Joint training: End-to-end training jointly optimizes reconstruction and autoregressive generation instead of freezing a reconstruction-trained tokenizer before training the generator.The next-token-prediction loss depends on both the autoregressive model and tokenizer encoder, motivating a single-stage pipeline.
  • Joint training: NTP-only joint training can collapse the latent space by exploiting a small subset of codebook tokens, sharply reducing codebook usage, rFID, and gFID.The resulting code distribution is uneven and highly skewed toward a small subset of tokens; APR is introduced to alleviate this failure.
  • Joint training: The APR loss decodes teacher-forced autoregressive predictions into pixels and matches them to ground-truth images, providing direct generative supervision to the tokenizer.APR regulates next-token prediction so that it remains meaningful for pixel-space generation quality.
  • Semantic VFM representation: The method evaluates direct, substitution, and implicit VFM-representation injection, with implicit alignment matching hidden patch embeddings rather than forcing latent codes to equal semantic features.On ImageNet generation, decoder alignment improves reconstruction metrics, while encoder-side approaches slightly improve autoregressive generation quality and decoder alignment only marginally raises prediction accuracy.

4 Experiments

Experiments show that end-to-end training yields strong ImageNet 256×256 generation and reconstruction, while making token sequences more suitable for autoregressive modeling. Scaling improves generation, but sequence length and codebook size expose reconstruction–generation trade-offs.

  • ImageNet generation: 1.74 gFID without guidance is achieved by EOSTok-L with 312M parameters, outperforming competitive 1D tokenization baselines even with classifier guidance.Its tokenizer also reaches 0.73 rFID with 165M parameters, attributed especially to semantic vision foundation model representation alignment.
  • ImageNet generation: EOSTok achieves state-of-the-art ImageNet 256 generation performance without classifier guidance.The comparison covers models using 2D continuous, 2D discrete, and 1D visual tokenizers.
  • Scaling behavior: Scaling the autoregressive model consistently improves FID while lowering the converged cross-entropy loss.The ViT tokenizer and AR model are jointly scaled to 388M and 644M parameters, respectively, totaling 1B trainable parameters.
  • Facilitating AR modeling: Original token ordering produces much better generation FID than reversed or randomly ordered sequences, indicating that end-to-end losses learn sequentially modelable latent representations.The comparison trains AR models on original, reversed, and randomly fixed-ordered token sequences from a frozen EOSTok-L tokenizer.
  • Comparing to two-stage separate training: End-to-end training achieves better generation FID than separate training despite its next-token prediction loss plateauing earlier at a larger value.This shows that next-token cross-entropy alone does not determine generative quality.
  • Sequence length: 192 is the sequence length at which gFID peaks, while longer sequences consistently improve reconstruction quality but make autoregressive modeling more difficult.This establishes a reconstruction–generation trade-off, and nested dropout expands the attainable frontier.
  • Codebook size: 1.74 gFID for K = 4096 and 1.92 for K = 16384 on EOSTok-L narrow to 1.48 and 1.51 on EOSTok-H, respectively.Larger models close the gap between codebook sizes, indicating that capacity alleviates the reconstruction–generation dilemma.

5 Conclusion · Appendix

The paper presents an end-to-end pipeline that jointly optimizes a 1D vision tokenizer and autoregressive model for reconstruction, generation, and semantic alignment. It also introduces autoregressive prediction reconstruction loss and incorporates global semantic information from vision foundation models.

  • 5 Conclusion: The pipeline jointly optimizes a 1D vision tokenizer and an autoregressive generative model.The joint objectives cover reconstruction, generation, and semantic alignment.
  • 5 Conclusion: Next-token prediction loss on discrete latent sequences cannot determine final generation quality.
  • 5 Conclusion: The autoregressive prediction reconstruction loss bridges the gap between token prediction loss and generation quality.
  • 5 Conclusion: The method incorporates global semantic information from vision foundation models into the 1D vision tokenizer.
  • 5 Conclusion: The tokenizer does not enforce alignment between its sequential latent space and the 2D spatial structure of vision foundation model representations.
  • 5 Conclusion: EOSTok learns a 1D sequential token space that facilitates autoregressive modeling.

A Implementation Details … A.1.2 Autoregressive Generative Model.

The implementation uses a 1D ViT tokenizer with structured encoder–decoder attention and IBQ quantization, followed by a modified LlamaGen autoregressive model tailored to 1D image sequences. The generative model adds shared AdaLN modulation, RMSNorm, SwiGLU, and learnable positional embeddings.

  • A.1.1 Tokenizer.: The 1D ViT tokenizer compresses images into discrete one-dimensional sequences without a 2D structural prior.Its architecture is similar to TiTok.
  • A.1.1 Tokenizer.: Images are patchified, flattened, concatenated with L learnable query tokens, and processed by transformer blocks with learnable positional embeddings.The encoder produces a hidden 2D embedding alongside the query-token pathway.
  • A.1.1 Tokenizer.: The decoder symmetrically combines sequential 1D latent codes with 2D mask tokens before unpatchifying and applying a convolutional output layer.It outputs the unmasked tokens after multiple transformer blocks.
  • A.1.1 Tokenizer.: Hybrid attention is bidirectional among 2D patch tokens but causal along 1D query tokens in both encoder and decoder.Query tokens attend to patch tokens, while patch tokens cannot attend to query tokens.
  • A.1.1 Tokenizer.: IBQ quantizes latent embeddings, with ℓ2 normalization applied to both codebook C and continuous latent embeddings when computing similarity.This normalization is used to stabilize codebook training.
  • A.1.1 Tokenizer.: The tokenizer quantization represents sequential latent codes z1, …, zT using codebook entries C1, …, CK.The provided passage contains the corresponding codebook-index notation.
  • A.1.2 Autoregressive Generative Model.: The autoregressive generator slightly modifies LlamaGen for image generation and adds shared global AdaLN modulation with per-block learnable biases.The model configurations are provided in table 9.

A.2 Training Details … A.3 Sampling Details

EOSTok jointly trains its tokenizer and autoregressive model using reconstruction, prediction, and semantic-alignment objectives. Training uses Adam-based optimization, while sampling combines cached autoregression with model-dependent guidance strategies.

  • A.2.1 Loss functions: EOSTok jointly trains encoder Eϕ, decoder Dψ, and autoregressive model Gθ with reconstruction, next-token prediction, autoregressive prediction reconstruction, and semantic-alignment losses.The combined objective includes LVQVAE, λNTPLNTP, and λAPRLAPR terms.
  • A.2.1 Loss functions: LVQVAE combines L2, LPIPS, GAN, and regularization losses, while LAPR combines L2 and LPIPS losses.The GAN loss uses the StyleGAN-T discriminator with LeCAM divergence, and LPIPS uses a VGG backbone.
  • A.2.1 Loss functions: λNTP, λAPR, and λsem are predetermined loss weights, while ω1 and ω2 are learnable MLP layers for representation alignment.The loss-weight details are provided in table 9.
  • A.2.2 Optimizer: 1e-4 is the initial learning rate for both tokenizer and autoregressive-model Adam optimizers.The discriminator also uses Adam with a fixed learning rate of 1e-4.
  • A.2.2 Optimizer: 2M iterations is the cosine scheduler endpoint, where the learning rate decays to 1e-6 during training on 8 H100 GPUs.Training uses batch size 256 for 400 epochs, approximately 2M iterations.
  • A.3 Sampling Details: 1.0 is the sampling temperature, with KV-cache autoregressive sampling and no top-k or top-p strategy.Classifier guidance is used for EOSTok-S and EOSTok-B.
  • A.3 Sampling Details: EOSTok-L and EOSTok-H use AutoGuidance because classifier-guidance benefits diminish as unconditional generation quality improves with scale.AutoGuidance replaces unconditional logits using a lightweight autoregressive model trained on the tokenizer, with the same search budget for all models.

A.4 Computation Cost

EOSTok’s computation cost is analyzed across module GFLOPs and training memory, while KV-cached sampling enables high throughput and substantial speedups over diffusion sampling.

  • Cost analysis: Table 7 analyzes EOSTok’s computation cost through the GFLOPs of each module and peak training memory usage.The passage identifies both module-level computation and training memory as components of the cost analysis.
  • Sampling efficiency: 10.5 images per second: EOSTok-H generates about this many batched images on a single H100.The reported throughput uses KV cache during sampling.
  • Cost analysis: Table 8 lists the sampling GFLOPs of EOSTok-H, its largest model.The sampling analysis specifically covers EOSTok-H.
  • Sampling efficiency: 20 to 100 times faster: EOSTok-H outpaces DiT-XL/2 depending on the diffusion sampling algorithm.The comparison is reported for sampling speed, with KV caching contributing to faster sampling than diffusion.

B Additional Experimental Results

Additional experiments show that EOSTok’s performance is robust to training and representation choices, extends to higher resolution, and preserves reconstruction quality despite compact 1D compression. An APR weight of 1.0 gives the best reconstruction and generation quality, while higher-resolution EOSTok-L reaches a gFID of 1.98 without guidance.

  • APR weights: An APR weight of 1.0 achieves the best reconstruction and generation quality when λNTP is fixed at 0.1.The experiment trains EOSTok-L for 50 epochs.
  • The choice of VFM in representation alignment: SigLIP2 slightly improves generative results over DINOv2, demonstrating robustness to the choice of pretrained vision foundation model.The comparison uses EOSTok-L in the ablation setting.
  • Scalability to higher resolution: 1.98 gFID is achieved by EOSTok-L without guidance at higher resolution, outperforming diffusion models with 2D tokenizers and mask-based models with 1D tokenizers.The higher-resolution setup keeps patch size 16 and AR sequence length 256 using the EOSTok-L architecture.
  • Reconstruction evaluations: EOSTok achieves comparable PSNR, SSIM, and LPIPS while delivering better rFID distributional performance despite compact 1D compression.The reconstruction comparison includes recent discrete tokenizers for reference.
Loading 2605.00503v2…