PaperlayerView exact revision on arXiv

Source-linked AI summary

iFSQ: Improving FSQ for Image Generation with 1 Line of Code

Bin Lin, Zongjian Li, Yuwei Niu, Kaixiong Gong, Yunyang Ge, Yunlong Lin, Mingzhe Zheng, JianWei Zhang, Miles Yang, Zhao Zhong, Liefeng Bo, Li Yuan

arXiv:2601.17124v2cs.CV

TL;DR

Image generation uses incompatible discrete AR and continuous diffusion tokenizers, making unified modeling and fair comparison difficult. The paper introduces iFSQ by replacing FSQ’s activation with a distribution-matching mapping, then uses the shared tokenizer to compare paradigms and extend REPA to AR models. It reports a roughly 4-bit equilibrium, faster early AR convergence, and a higher diffusion performance ceiling.

  • Problem

    Discrete VQ-VAE tokenizers for AR models and continuous VAE latents for diffusion models hinder unified modeling and fair benchmarking.

  • Method

    iFSQ replaces FSQ’s tanh with a distribution-matching activation that maps Gaussian latents to a bounded uniform prior, enabling a shared tokenizer for AR–diffusion benchmarking.

  • Results

    Under identical reconstruction constraints, AR models converge faster initially, whereas diffusion models achieve a superior performance ceiling; the best discrete–continuous equilibrium is approximately 4 bits per dimension.

  • Takeaways & Limitations

    A shared iFSQ tokenizer separates generative-paradigm comparisons from tokenizer differences and supports extending REPA to AR models through LlamaGen-REPA.

  • Takeaways & Limitations

    Autoregressive models must scale sufficiently to provide capacity for predicting large codebooks.

Abstract

from arXiv · show

The field of image generation is currently bifurcated into autoregressive (AR) models operating on discrete tokens and diffusion models utilizing continuous latents. This divide, rooted in the distinction between VQ-VAEs and VAEs, hinders unified modeling and fair benchmarking. Finite Scalar Quantization (FSQ) offers a theoretical bridge, yet vanilla FSQ suffers from a critical flaw: its equal-interval quantization can cause activation collapse. This mismatch forces a trade-off between reconstruction fidelity and information efficiency. In this work, we resolve this dilemma by simply replacing the activation function in original FSQ with a distribution-matching mapping to enforce a uniform prior. Termed iFSQ, this simple strategy requires just one line of code yet mathematically guarantees both optimal bin utilization and reconstruction precision. Leveraging iFSQ as a controlled benchmark, we uncover two key insights: (1) The optimal equilibrium between discrete and continuous representations lies at approximately 4 bits per dimension. (2) Under identical reconstruction constraints, AR models exhibit rapid initial convergence, whereas diffusion models achieve a superior performance ceiling, suggesting that strict sequential ordering may limit the upper bounds of generation quality. Finally, we extend our analysis by adapting Representation Alignment (REPA) to AR models, yielding LlamaGen-REPA. Codes is available at https://github.com/Tencent-Hunyuan/iFSQ

1 Introduction

The paper introduces iFSQ, a one-line distribution-aware modification that aligns activation distributions with equal-interval quantization, improving both bin utilization and reconstruction fidelity. Using iFSQ for controlled benchmarking, it identifies a roughly 4-bit equilibrium and distinct AR–diffusion scaling behaviors.

  • Motivation: Vanilla FSQ’s equal-interval bins mismatch Gaussian-like activations, creating a trade-off between reconstruction fidelity and information efficiency.Central bins become crowded while edge bins are underutilized.
  • Motivation: MSE: 0.1678 pairs with 83.3% utilization under vanilla FSQ, yielding high fidelity but activation collapse and low effective capacity.Most samples occupy central bins, leaving edge bins severely underutilized.
  • Motivation: MSE: 0.1812 pairs with 100% utilization under equal-probability quantization, improving entropy but reducing edge precision.The outer bins must be wide enough to accommodate Gaussian tails.
  • iFSQ: iFSQ replaces FSQ’s tanh with y = 2.0 · σ(1.6x) − 1, mapping Gaussian latents to a bounded uniform distribution in one line of code.The mapping preserves equal intervals while matching the latent distribution to the quantizer.
  • iFSQ: MSE: 0.1669 and 100% utilization demonstrate simultaneous reconstruction fidelity and information efficiency for iFSQ.The result is reported for the uniform-distribution setting in Figure 1(c).
  • Benchmarking: Approximately 4 bits per dimension is the reported equilibrium between discrete and continuous representations.With the same pretrained tokenizer, AR models converge faster initially, while diffusion models reach a higher performance ceiling under identical reconstruction constraints.
  • Analysis & Extension: LlamaGen-REPA extends Representation Alignment to AR models, with best results at layer 8 and stronger regularization of λ = 2.0 than diffusion’s λ = 0.5.These settings are reported for the Large model and the comparison with diffusion models.

2 Related Work

Related work frames tokenizer design as the central distinction between continuous diffusion representations and discrete autoregressive tokens. FSQ provides a simpler discrete alternative to codebook-based quantization, while broader quantization methods target efficiency and rate–distortion trade-offs.

  • Open problem: Tokenizer design remains unconverged because image generation is divided between diffusion’s continuous representations and autoregression’s discrete tokens.This divergence also affects attempts to unify multimodal architectures.
  • Continuous representations: VAEs impose Gaussian latent priors for diffusion, while reconstruction-focused systems add adversarial, perceptual, or discriminative supervision.These methods address fidelity within continuous latent representations.
  • Discrete codebooks: VQ-VAE supplies discrete AR tokens through nearest-neighbor lookup in a learnable codebook, but suffers codebook collapse and memory-intensive lookup.Entropy regularization can mitigate collapse, while gradient training uses the straight-through estimator.
  • Scalar quantization: FSQ projects latents onto a fixed bounded grid through element-wise rounding, avoiding codebook lookup and codebook collapse.This makes FSQ a discrete but implicit-codebook alternative.
  • Quantization: Weight-only quantization and activation-aware methods reduce memory and computation in large models, motivating broader rate–distortion optimization.The passage situates image-tokenizer quantization within efficiency-oriented model compression.

3 iFSQ

iFSQ replaces FSQ’s tanh-based bounding function with a distribution-matching sigmoid mapping, aligning Gaussian-like activations with uniform quantization bins. The resulting pipeline supports continuous diffusion latents and discrete autoregressive indices while targeting efficient, faithful quantization.

  • 3.1 Background: Latent Quantization via FSQ: FSQ bounds latent representations to [-1, 1], quantizes each dimension onto L = 2K + 1 levels, and maps values to integer indices by rounding.The +1 term provides an exact zero-center; the quantized vector can be used directly by diffusion models or converted into a scalar autoregressive index.
  • 3.1 Background: Latent Quantization via FSQ: Gaussian-like activations passed through tanh produce a non-uniform, bimodal distribution that misaligns with FSQ’s fixed quantization intervals.This mismatch concentrates values unevenly across bins and motivates distribution-aware activation design.
  • 3.2 Distribution Analysis and Optimization for iFSQ: α = 1.6 most closely approximates a uniform distribution, while larger α values produce concave or bimodal shapes and α = 2.0 recovers original FSQ.The figure reports distribution similarity using RMSE and KS, with lower values indicating closer agreement with the uniform target.
  • 3.2 Distribution Analysis and Optimization for iFSQ: iFSQ replaces tanh with 2σ(1.6x) − 1, a one-line, parameter-free modification that minimizes KS and RMSE while preserving fixed intervals.The method is described as plug-and-play, with no additional inference latency; its implementation scales, rounds, and applies a straight-through estimator.
  • 3.2 Distribution Analysis and Optimization for iFSQ: The optimized α = 1.6 setting is highlighted against tanh at α = 2.0 using PSNR, SSIM, LPIPS, RMSE, and KS.The plotted primary axes mark reconstruction metrics, while secondary axes report distribution metrics.

4 Experiments

The paper validates iFSQ experimentally and extends Representation Alignment to the LlamaGen autoregressive model.

  • 4 Experiments: iFSQ’s effectiveness is validated through extensive experiments in section 4.1.
  • 4 Experiments: Representation Alignment is extended to LlamaGen in section 4.2.
  • 4 Experiments: The extension applies REPA to an autoregressive visual-generation model.

4.1 Experiments for iFSQ

The experiments evaluate iFSQ against AE, VQ, and FSQ for reconstruction and generation, then compare scaling across quantization levels and AR–diffusion training efficiency. Results consistently identify approximately 4 bits as a strong operating point, while iFSQ supports fair cross-paradigm comparison through a shared tokenizer.

  • Image reconstruction: iFSQ consistently outperforms original FSQ across PSNR, SSIM, and LPIPS, with α = 1.6 selected as the optimal setting.At α = 1.6, KS and RMSE reach minima; similar trends appear on COCO validation data.
  • Diffusion image generation: 12.76 gFID gives iFSQ an advantage over AE’s 13.78 while providing a 3× higher compression rate, 96 versus 24.The comparison is reported for DiT-Large without CFG on ImageNet validation.
  • Quantization scaling: At 4 bits, iFSQ generation becomes comparable to AE, whereas 2-bit iFSQ remains substantially worse and levels above 4 bits fluctuate without consistent improvement.Reported gFID values at 2 bits are 18.52 versus AE’s 13.78 without REPA and 14.97 versus 10.67 with REPA.
  • Autoregressive image generation: At equal latent dimension, autoregressive generation trained on VQ underperforms iFSQ, while iFSQ operates at a lower bit rate.The comparison uses LlamaGen-REPA with 256× spatial compression.
  • Training efficiency: Using the same iFSQ tokenizer, DiT-Large and LlamaGen-L require approximately 161.04G and 169.65G FLOPs, respectively, for 256-resolution training-efficiency comparison.Both models use optimal configurations derived from ablation studies.
  • Scaling behavior: Across quantization levels, iFSQ approaches AE around 4 bits and can exceed AE in PSNR and SSIM at 16-dimensional latent representations.Each point uses 64× spatial compression, and larger markers denote higher latent dimensionality.

4.2 Experiments for LlamaGen-REPA

The LlamaGen-REPA experiments analyze how autoregressive features evolve from self-encoding toward next-token prediction and test representation alignment across depths, architectures, and scales. They find that semantic alignment tracks this transition, with optimal alignment near one-third of network depth and stronger regularization benefiting AR models.

  • Layer-wise quantitative analysis: STS decreases with depth, while NTS and CKNNA rise sharply in middle-to-late layers, indicating a transition toward prediction and semantic representation.The NTS surge synchronizes with the rise in CKNNA across model scales and resolutions.
  • Layer-wise quantitative analysis: NTS and CKNNA show positive correlations of r = 0.47, 0.79, and 0.72 across Large@256, Large@384, and XXLarge@384.The correlation becomes stronger with increasing model scale and resolution.
  • Semantic acceleration: REPA shifts the layer with maximal DINOv2 similarity to the enforced alignment depth, including when alignment is applied at Layer 8.This effect is observed across tested loss-coefficient values.
  • Ablation studies: LlamaGen performance peaks when aligning its 8-th layer with final-layer DINOv2-Base features, transferring the optimal diffusion-model configuration to AR generation.The authors then examine which alignment depth is best for autoregressive prediction.
  • Ablation studies: Optimal alignment occurs near one-third of total depth: 8/24 for Large, 12/36 for XLarge, and 16/48 for XXLarge, across AR and diffusion models.The proportional rule does not transfer as a fixed absolute layer index to deeper networks.
  • Ablation studies: λ = 2.0 benefits LlamaGen-REPA and achieves the best performance at the optimal eighth-layer depth, unlike the smaller coefficient used in DiT-REPA.The paper attributes this difference to teacher-forcing inductive bias in autoregressive training.

5 Conclusion

The paper unifies diffusion and autoregressive image generation through activation quantization. Its iFSQ tokenizer uses a tailored activation function to map latent distributions toward a uniform prior while addressing the reconstruction–efficiency trade-off.

  • iFSQ unifies diffusion and autoregressive models by transforming latent distributions into a uniform prior through a tailored activation function.

A Background: Tokenizer for Generation

Generation tokenizers encode images into latent representations and reconstruct pixels through an encoder–decoder pair. Diffusion models operate on continuous noisy latents, whereas autoregressive models quantize latents into discrete token sequences and predict subsequent tokens.

  • A tokenizer encoder maps an image x to latent representation z, and a decoder reconstructs the image from z.
  • Diffusion models: Diffusion models interpolate latent data with Gaussian noise and train a network to predict velocity during denoising.
  • Autoregressive models: Autoregressive models quantize latent representations into indices I and predict each subsequent token conditioned on preceding tokens.

B Background: Compression Ratio Analysis

Compression ratio compares raw image bit-rate with latent representation bit-rate. VAEs use high-bit-depth continuous latents, VQ-VAEs use codebook indices, and iFSQ uses scalar levels to provide adjustable bit rates while retaining continuous-space structure before rounding.

  • Compression ratio is defined as the ratio of raw image bit-rate to latent representation bit-rate.For 8-bit RGB images, the raw input size is S_input = 24HW bits.
  • With spatial downsampling factor f and channel dimension d, the latent resolution is h = H/f and w = W/f.
  • Continuous VAE: VAEs use 16 bits per latent value, yielding CR_VAE = 24f^2/(16d) and prioritizing reconstruction fidelity over storage efficiency.
  • VQ-VAE: VQ-VAEs represent each spatial location with log2(|C|) bits from a learnable codebook and achieve higher compression ratios than VAEs.
  • iFSQ: iFSQ requires d · log2(L) bits for d dimensions and L levels per dimension, allowing flexible compression while preserving continuous-space structure before rounding.For L = 5, this is approximately 2.3 bits per channel.
  • Scalability: The plotted scalability analysis uses a logarithmic compression-ratio axis, with VQ compression ratio around 438 under a 256× spatial compression factor.

C Scaling of Compression Ratio with iFSQ

Across compression ratios, models show approximately linear performance changes on a log scale, with an optimal knee around 48× compression, corresponding to 4 bits.

  • 48× compression, or 4 bits, marks a clear optimal knee point in the performance trend.The result comes from retraining iFSQ and AE under a 256× spatial compression setting matched to the standard VQ configuration.
  • On a logarithmic compression-ratio scale, all models exhibit approximately linear performance growth or decay as compression changes.
  • VQ data points lie almost exactly on the same scaling trend as the other models.

D Tokenizer Setup

The tokenizer study uses matched training and evaluation settings, reports multiple reconstruction metrics, and compares continuous and discrete tokenizer configurations under controlled conditions.

  • Implementation details: All tokenizers are trained for 25 epochs on ImageNet 256×256 using Adam with a constant learning rate of 0.001.The LPIPS loss coefficient is set to 0.1, and model configurations follow DiT and LlamaGen setups.
  • Tokenizer Baselines: Table 4 compares tokenizer baselines on the ImageNet validation set, with arrows indicating whether higher or lower metric values are better.
  • Evaluation: Tokenizer reconstruction is evaluated with PSNR, SSIM, LPIPS, and reconstruction rFID, including additional COCO2017 evaluation to check potential ImageNet overfitting.
  • Evaluation: The study ablates whether continuous tokenizers should impose distributional constraints on their latent spaces.Discrete tokenizers typically impose no additional distributional constraints, whereas continuous tokenizers usually apply KL-divergence regularization.
  • Tokenizer Baselines: Under matched settings, continuous VAE reconstruction performs worse than AE reconstruction.The authors therefore use a standard continuous AE without a KL-loss constraint as the continuous baseline.
Loading 2601.17124v2…