GIVT: Generative Infinite-Vocabulary Transformers

1 Introduction

• VQ-VAE-based generation can require differentiable approximations, trade latent detail against modeling ease, suffer low vocabulary utilization, and consume substantial embedding memory.Small vocabularies can limit control of low-level image details, while large vocabularies can reduce utilization and increase memory use.Source: §1 · p. 2
• GIVT replaces discrete-token lookup with linear projections of real-valued vectors and predicts multivariate Gaussian-mixture parameters instead of categorical logits.The compared models retain the same decoder-only transformer architecture.Source: Figure 2 · p. 3, §1 · p. 3
• GIVT models unquantized latent sequences from a Gaussian-prior β-VAE, using standard training techniques rather than specialized VQ-VAE training methods.The authors also explore MaskGIT-like masked-bidirectional modeling.Source: §1 · p. 3, Figure 3 · p. 3
• GIVT outperforms VQ-GAN variants and MaskGIT in class-conditional image generation, often with lower computational cost, and is competitive with latent diffusion models.At high resolution, it is particularly competitive with latent diffusion baselines.Source: §1 · p. 3
• GIVT matches or outperforms prior sequential image-generation models in representation learning at lower computational cost and performs comparably to VQ-based UViM in dense prediction.The cited dense-prediction tasks include semantic segmentation and monocular depth estimation.Source: §1 · p. 4
• Transformer-based visual generation can benefit from advances in language-model scaling and inference efficiency, while GIVT advances can transfer to interleaved multimodal modeling [1].The paper contrasts this transfer with diffusion models.Source: §1 · p. 4

2 Related work

• VQ-VAE latents support efficient image generation with causal and masked transformers, interleaved image-text modeling [1], dense-prediction tokenization, and self-supervised learning.GANs and perceptual losses for VQ-VAE training have also contributed to quality improvements.Source: §2 · p. 4
• Discretized mixture models replace categorical-logit prediction with a continuous mixture that is subsequently discretized, an approach used in pixel-space autoregression and neural compression.The cited neural-compression works are.Source: §2 · p. 4
• Continuous-output translation models predict word embeddings with continuous distributions, but decode through finite embedding lookup and usually do not produce diverse samples [38] [65].The predicted embeddings still come from a fixed, finite set.Source: §2 · p. 4
• Prior VAE work improves learned priors with inverse autoregressive or normalizing flows, pseudo-input mixtures, and Restricted Boltzmann Machines for discrete non-VQ latents.The cited approaches include,,, and [57].Source: §2 · p. 4
• Time-series transformers use regression, quantile forecasting, or discretization, while continuous speech-feature regression from discrete tokens is related but does not predict continuous distributions for autoregressive generation.The cited time-series approaches are [11] [22] [36], [19], and [53]; speech-feature approaches are [47] [74].Source: §2 · pp. 4–5

3.1 VAE training

• The β-VAE encodes images into Gaussian latent sequences, with KL regularization weighted by β to control how strongly those latents are constrained.Image-generation reconstruction uses MSE, perceptual, and GAN losses; dense prediction uses categorical cross-entropy.Source: §3.1 · p. 5

3 Generative infinite-vocabulary transformers

• GIVT replaces discrete token embeddings with linear projections of real-valued latent vectors and predicts a continuous Gaussian mixture instead of categorical logits.It models unquantized β-VAE latents rather than VQ-VAE code indices.Source: §3.2 · p. 5, §3 · p. 5

3.2 GIVT training

• GIVT predicts a k-mixture continuous distribution over latent vectors, trained by negative log-likelihood, with causal teacher forcing or randomly masked bidirectional modeling.The causal model conditions on previous vectors; the masked variant zeros masked inputs and adds learned mask-status vectors.Source: §3.2 · pp. 5–7

3.3 Towards end-to-end training: Adapters

• An invertible, volume-preserving adapter transforms VAE latents to better match GIVT predictions, adding less than 0.1% compute and parameter overhead.The adapter and GIVT are trained jointly; generated samples are mapped back through the inverse adapter before VAE decoding.Source: §3.3 · p. 7

3.4 Inference

• For continuous sampling, variance scaling adjusts predicted Gaussian covariances, while beam search retains sequences with the highest cumulative log probability.Per-dimension truncation is considered similar to variance scaling and is not pursued; top-k has no direct analogue for continuous distributions.Source: §3.4 · p. 8
• DB-CFG combines conditional and unconditional GIVT densities into a guided target distribution, sampled with rejection sampling after two model evaluations per step.The proposal strategy uses conditional Gaussian components with increased variance; 1,000 proposals find a valid sample 99.9% of the time.Source: §3.4 · p. 9

4.1 Image generation

• ImageNet experiments cover class-conditional generation at 256px and 512px and unconditional generation at 256px, evaluating sampling FID, precision, recall, and representation learning.The β-VAE is trained at 256×256 and reused for 512×512 experiments without retraining.Source: §4.1 · pp. 9–11
• DB-CFG values from 0.3 to 0.8 and variance scaling t from 0.9 to 1.0 lead to low FID; conditional predictions have lower standard deviation, which decreases later in sampling.The standard-deviation visualization is for class 130, averaged over 128 samples.Source: Figure 7 · p. 10

4.2 Panoptic segmentation and depth estimation

• For panoptic segmentation and depth estimation, the UViM variant replaces its VQ-VAE with a β-VAE and models continuous latents using an encoder-decoder GIVT.The setup uses cross-attention, a pretrained ViT-L/16 image encoder, and task-specific KL weights.Source: §4.2 · p. 11

4.1 Image generation

• With 16 inference steps, GIVT-MaskGIT is competitive on 512×512 ImageNet and outperforms its VQ counterpart, while GIVT-Causal-L+A outperforms the best DiT variant.These comparisons are reported in Table 2.Source: Table 2 · p. 12

5.1 Image generation

• Increasing β worsens VAE reconstruction FID but improves sampling FID for GIVT; increasing mixture count improves sampling FID initially, then plateaus at k = 16.The authors suggest the sampling improvement may arise because the more regularized latent sequence more closely follows the Gaussian prior and is easier for GIVT to model.Source: §5.1 · p. 12
• FID 3.35 for GIVT-Causal versus 5.20 for VQGAN on class-conditional 256 × 256 ImageNet, with a model more than 4.5× smaller.GIVT-Causal also outperforms the 32%-larger LDM-4-G; GIVT-Causal-L+A reduces FID by 16% and 17% without and with guidance, respectively, versus ViT-VQGAN.Source: §5.1 · pp. 12–13
• At 512 × 512, GIVT-MaskGIT has 38% lower FID than MaskGIT at comparable model size and inference cost, while GIVT-Causal-L+A outperforms DiT-XL/2 with a larger model.The compared DiT-XL/2 is described as the best available DiT model.Source: §5.1 · p. 13
• DB-CFG is the most effective tested sampling algorithm across configurations, while increasing k from 1 to 16 generally improves FID more than adding an adapter alone.Combining an adapter with k = 16 compounds improvements across sampling algorithms.Source: §5.1 · p. 13
• A full-covariance single Gaussian yields only about 3% gains over a diagonal-covariance Gaussian for GIVT-MaskGIT, while making DB-CFG less tractable.The authors identify a GMM with factorized component densities as the more effective alternative.Source: §5.1 · p. 13
• GIVT-Causal-L+A produces high-fidelity, coherent 512 × 512 samples, and the reported VAE prior samples show a soup of image textures.The paper also presents multiple samples for a fixed class to study diversity and shows GIVT-MaskGIT inference steps with behavior similar to the VQ-based model [7, Fig. 2].Source: §5.1 · pp. 13–14
• GIVT-Causal matches VIM+ViT (ViT-VQGAN) in ImageNet linear probing despite that model having more than 2× the parameters and 4× the sequence length, and is outperformed only by MAGE.The authors note that MAGE's latent encoder-decoder architecture is better suited to representation learning than decoder-only models.Source: §5.1 · p. 14, Table 4 · p. 14

5.2 Panoptic segmentation and depth estimation

• GIVT-based UViM outperforms the VQ-VAE baseline on COCO Panoptic 2017 panoptic segmentation and performs slightly worse on NYU Depth v2 depth estimation.The comparison reports PQ for panoptic segmentation and RMSE for depth estimation; Table 3 covers reconstruction and inference metrics.Source: Table 3 · p. 13, §5.2 · p. 14

• GIVT modifies decoder-only transformers to generate sequences of real-valued vectors, avoiding quantization.The approach is presented as the first decoder-only model amenable to generating such sequences.Source: §6 · p. 14
• In image generation, GIVT avoids VQ-VAE codebook-usage difficulties and their mitigation techniques by using standard VAEs that are easier to train.The paper also notes that GIVT avoids large embedding matrices by directly consuming and predicting feature representations.Source: §6 · p. 14
• GIVT outperforms VQ-based counterparts in class-conditional image generation and representation learning, often by significant margins, and performs strongly in dense prediction with UViM.Source: §6 · p. 14

arXiv version history

• The version history says version 2 added loss details, while version 3 made multiple model updates.Version 1 was the initial version.Source: arXiv version history · p. 19
• Version 3 changed the GMM to a d-variate mixture with factorized components, introduced adapters, added 1.67B-parameter models and 512px results, and updated the optimizer to Adam for model sharding.The optimizer change is reported as performance-neutral; version 4 was the ECCV 2024 camera-ready version with minor changes.Source: arXiv version history · p. 19

A Additional discussions

• The discussion argues that continuous latents suit continuous-valued image representations and avoid VQ compression loss and discrete-representation optimization difficulties.It contrasts the β-VAE approach with VQ methods that require advanced mitigation techniques for issues such as low vocabulary utilization.Source: A Additional discussions · p. 19
• Exactly mapping continuous latents to discrete codes implies an infinite vocabulary; even float32 codes imply about 4.3B possibilities per latent dimension.GIVT directly models continuous latents instead of materializing that vocabulary.Source: A Additional discussions · p. 19

• The image-generation architectures use a standard pre-LN setup.Source: B Architecture details · p. 19
• For UViM, GIVT-Causal uses the Default configuration with cross-attention after every self-attention layer to fuse features extracted from the input RGB image.The cross-attention design follows.Source: B Architecture details · p. 20
• The adapter stacks eight bijective iRevNet blocks with hidden dimension 4d and 112k additional parameters when d = 16.Each block preserves the w×h×d shape and uses GroupNorm and ReLU.Source: B Architecture details · p. 20
• Image-generation variants use β = 5 · 10^-5 by default and β = 10^-5 for Large models; the 512px Large variant stacks pairs of 16-dimensional features to halve sequence length to 512.The Base architecture is used to explore feature dimension, mixture count, and β.Source: Table 5 · p. 20

C.1 Loss function and implementation details

• GIVT predicts conditional feature-vector distributions as k-component Gaussian mixtures with diagonal covariance, using means, scales, and mixture weights as output parameters.The predicted output for a batch has shape B × L × (2dk + k); the scale entries are lower-bounded by 10^-5.Source: §C.1 · pp. 20, 22
• Increasing mixture count k or reducing β lowers training loss, but lower loss does not always yield lower sampling FID.The reported curves use Base-sized GIVT-causal models with latent dimension d = 16.Source: Figure 8 · p. 21, §C.1 · p. 22
• Loss computation and sampling can be implemented with deep-learning packages such as JAX’s distrax.The implementation constructs a mixture distribution with categorical mixture weights and diagonal multivariate Gaussian components.Source: §C.1 · p. 22

C.2 Image generation

• For ImageNet generation, the CNN-based VQ-GAN tokenizer’s VQ layer and related losses are replaced with a Gaussian reparametrization layer.The tokenizer’s 32 embedding features are split into 16 means and 16 scales, yielding d = 16 channels after reparametrization.Source: §C.2 · p. 22, Table 6 · p. 23

C.3 Panoptic Segmentation and Depth Estimation

• For UViM panoptic segmentation and depth estimation, the experiments replace the stage-I VQ layer, adapt the stage-II transformer decoder, and modify the losses.Source: §C.3 · p. 23

D DB-CFG implementation

• The DB-CFG rejection sampler is presented as a JAX implementation.The section directs readers to Fig. 11 for the implementation.Source: D DB-CFG implementation · p. 23

E Unconditional image generation

• DB-CFG is the most effective tested sampling strategy for increasing precision across all GIVT-Causal-L model variants.The figure complements FID comparisons and reports compounding effects from increasing mixture count k and adding an adapter.Source: Figure 10 · p. 24

F Probing intermediate representations

• Layers 10–20 yield high 25-shot linear-probing accuracy across the tested datasets for representations from unconditional 256 × 256 ImageNet GIVT-Causal.Layer 18 is selected for full-ImageNet linear-probing experiments.Source: F Probing intermediate representations · pp. 24–25

G Comparison of methods in terms of FLOPs

• MaskGIT and GIVT-MaskGIT require virtually the same inference FLOPs, while the approximate comparison uses GIVT-Causal-L+A as its reference.The estimate assumes the same autoregressive optimizations, such as caching, and VAE inference cost; Table 8 presents the approximate FLOPs comparison.Source: G Comparison of methods in terms of… · p. 25, Table 8 · p. 25

H Broader impact

• The ImageNet class-conditional models provide basic content control but not fine-grained control or manipulation of existing images.The discussion also notes that the models could reflect documented ImageNet biases and advises careful use and deployment.Source: H Broader impact · pp. 25–26

I Additional visual examples

• Additional examples show VAE samples preserving input semantic layout while varying low-level textures, and MaskGIT samples becoming more coherent over more inference steps.The examples also include GIVT-Causal samples with labels changed after the top half is generated.Source: Figure 12 · p. 28, Figure 15 · p. 29, Figure 19 · p. 31