Image Tokenizer Needs Post-Training

1 INTRODUCTION

• Tokenizers with strong reconstruction ability do not necessarily produce high-quality generations, and some with weaker reconstruction perform better generatively.Prior tokenizer improvements were commonly validated with reconstruction metrics.Source: §1 · p. 1
• Reconstruction uses true tokens, whereas generation sampling uses predicted tokens that can be out-of-distribution, creating a latent-token distribution difference.The mismatch reflects different tokenizer use during reconstruction and generation.Source: Figure 1 · p. 2, §1 · p. 2
• Robustness to random perturbations alone cannot resolve systematic generator sampling errors, while direct post-training supervision is difficult without latent-to-image correspondence.The paper identifies generated-token reconstruction as an open challenge.Source: §1 · p. 2
• 1.36 gFID with a ∼400M generator establishes the paper’s reported state-of-the-art result under that parameter budget.The two-stage approach is evaluated with autoregressive and diffusion-based generators.Source: §1 · p. 2
• RobusTok combines latent-perturbation main-training for robust latent construction with post-training that aligns the decoder to generated latents.The post-training adapts the decoder to latents produced by a well-trained generator.Source: §1 · p. 2

2 RELATED WORKS

• Image tokenizers produce continuous or discrete image representations for generation and semantic tasks; VQGAN-family discrete codes support autoregressive generation (Esser et al., 2021; Razavi et al., 2019b).The literature also includes semantic representations used in classification, detection, segmentation, and multimodal applications.Source: §2 · p. 3
• Generative image tokenizers use VAEs to map images to probabilistic distributions, while VQGAN and variants introduce discrete latent spaces that enhance compression and facilitate autoregressive image generation.Source: §2 · p. 3
• This paper examines image tokenizers through the lens of perturbation robustness (Chen et al., 2024a; Li et al., 2024f; Xu et al.; Li et al., 2024g; 2023b;a).Source: §2 · p. 3
• Autoregressive visual generation models discrete codes sequentially, while recent directions include global-information fusion in VAR and RAR and continuous generation in MAR and GIVIT (Esser et al., 2021; Pang et al., 2024a; Chang et al., 2022; Wang et al., 2024; Tian et al., 2024; Li et al., 2024e; Han et al., 2024; Ren et al., 2024; Qiu et al., 2024; Yu et al., 2024b; Pang et al., 2024b; Li et al., 2024b; Fan et al., 2024; Tschannen et al., 2024).Source: §2 · p. 3

3 PRELIMINARY - TOKENIZER ROBUSTNESS

• VQ maps each continuous feature vector to its closest codeword in a learnable codebook, producing discrete features for reconstruction.The encoder first extracts latent representations, and the decoder reconstructs the input from the quantized representation.Source: §3 · p. 3
• RobusTok’s overview applies latent perturbation to a portion of the batch by replacing tokens with top-δ codebook neighbors, with probability α, and uses frozen DINO supervision.Source: Figure 2 · p. 4
• During inference, sampling error perturbs the decoder’s latent input, so robustness concerns whether decoding noisy latents can still reconstruct the ground-truth image.RobusTok targets synthetic noisy latents during main-training and generator-produced noisy latents during post-training.Source: §3 · p. 4
• pFID measures tokenizer robustness and reconstruction quality by computing FID between the image reconstructed from a perturbed latent and the ground-truth image.The paper introduces the metric because the potential perturbations are constrained and this image discrepancy is better reflected by FID.Source: §3 · p. 4

4.1 ARCHITECTURE

• RobusTok uses a ViT encoder and decoder with learnable tokens, perturbed quantized latents, and pretrained DINOv2 to retain visual semantics and structural coherence.In reconstruction with latent perturbations, the figure reports distorted outputs for non-semantic tokenizers and promising robustness for RobusTok.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.2 MAIN-TRAINING

• Main-training simulates sampling errors by randomly replacing selected discrete tokens with top-δ nearest neighbors from the learned codebook.Perturbations are applied after semantic regularization; the method randomly selects a proportion of images and tokens for replacement.Source: §4.2 · p. 5
• The perturbation rate α controls the share of perturbed tokens per image, while β controls the share of images perturbed within a batch.The passages define α using the number of perturbed tokens and the spatial token count, and β using the perturbed and clean image counts.Source: §4.2 · p. 5

4.3 POST-TRAINING

• Post-training freezes the encoder and quantizer and fine-tunes only the decoder, using generated images re-encoded and paired with corresponding real images.The pretrained discriminator is reused, and optimization continues with the same loss combination.Source: §4.3 · p. 6
• The preservation ratio σ controls replacement of generated tokens with ground-truth counterparts, smoothly connecting fully reconstructed and fully generated latents.A similar mechanism is applied to diffusion-based generation through SDEdit.Source: §4.3 · p. 6

5.2 MAIN EXPERIMENTS ANALYSIS

• Benchmarking finds that smaller codebooks can generate better at similar reconstruction quality, semantic tokenizers tend to do well on both tasks, and rFID does not align with generation quality.The authors relate semantic tokenizers’ results to structured, clustered latents and identify latent robustness as a possible factor in the rFID–generation mismatch.Source: §5.2 · p. 7

5.1 EXPERIMENTAL SETTING

• pFID correlates with gFID within codebook sizes, unlike rFID, and permits tokenizer assessment without training downstream generators.It is computed by averaging FID for perturbed reconstructions across α and δ settings; δ is scaled by codebook size.Source: §5.2 · pp. 7–8, §5.1 · p. 7, Figure 5 · p. 7

5.2 MAIN EXPERIMENTS ANALYSIS

• 1.36 gFID is the post-trained RobusTok result, a new state of the art among generators under 500M parameters; RobusTok gains 0.10 gFID on RAR.The comparison is on class-conditional ImageNet 256×256; the table caption identifies the system-level comparison.Source: Table 2 · p. 8, §5.2 · p. 8
• Across four representative continuous and discrete tokenizers with diffusion and autoregressive generators, post-training consistently improves performance.The study also evaluates 11 open-source tokenizers across four codebook sizes on ImageNet 256×256.Source: §5.1 · p. 6, §5.2 · p. 8, Table 3 · p. 8

5.3 MORE ANALYSIS

• Latent perturbation produces reusable key tokens alongside supportive tokens for finer detail, whereas unperturbed training spreads token use more uniformly.The comparison visualizes token-use frequency during inference.Source: §5.3 · pp. 8–9, Figure 7 · p. 8
• Post-training visualizations show improvements in out-of-distribution artifacts, color fidelity, and detail.The comparison is of 256 × 256 generated images before and after post-training.Source: Figure 8 · p. 9, §5.3 · p. 9
• Annealing perturbation to half balances robustness and adaptability; no annealing leads to mode collapse, while annealing to zero makes the tokenizer overly deterministic.A large perturbation such as β = 0.5 degrades reconstruction and generation performance.Source: §5.3 · p. 9, Table 5 · p. 9
• Post-training effectiveness depends more on training stability than on simply increasing data or adjusting σ; the authors link RobusTok’s stronger response to its more robust decoder.The ablation is reported in Table 4.Source: §5.3 · p. 9, Table 4 · p. 9

6 CONCLUSION

• The proposed scheme combines plug-and-play latent-perturbation main training with lightweight decoder post-training to address differences between generated and reconstructed latent distributions.The paper reports effectiveness across autoregressive and diffusion-based generators.Source: §6 · p. 9

A.1 CODEBOOK SIZE SELECTION

• XQGAN-1024 has a more fragmented latent space and greater robustness discrepancies than XQGAN-8192 and XQGAN-16384.The appendix connects this comparison to selecting a more compact codebook while preserving reconstruction fidelity and generation quality.Source: §A.1 · p. 10, Figure 9 · p. 10

A.2 LOSS FUNCTION.

• RobusTok combines reconstruction, vector-quantization, adversarial, perceptual, and semantic losses in its composite objective.The semantic loss regularizes semantic tokens against pretrained DINOv2 features.Source: §A.2 · p. 11
• Latent perturbation is added after the VQ/commitment loss, exposing the decoder to perturbed latents without directly modifying the encoder loss.The authors say this indirectly improves encoder representation quality by enforcing stable reconstruction under noisy conditions.Source: §A.2 · p. 11
• DINO class features appear more structured than pixel-level features, making them the more suitable supervision signal in this model.The paper uses DINO features to provide supervision.Source: §A.2 · p. 11

• RobusTok converges faster and achieves better final generative quality than both XQGAN and vanilla RAR.The benchmark uses the same LlamaGen-B generator.Source: Table 7 · p. 12, §A.3 · p. 12
• XQGAN's final gFID of 2.22 under classifier-free guidance remains suboptimal compared with RAR.Source: §A.3 · p. 12
• The annealing strategy balances generation diversity and quality: no annealing compromises diversity, while annealing to zero offers limited improvement over baseline.Source: §A.3 · p. 12

A.4 PFID RESULTS IN LLAMAGEN-L

• Despite some outliers, pFID shows a stronger correlation with gFID than with rFID in the LlamaGen-L setting.Source: §A.4 · p. 12, Figure 12 · p. 13

A.5 SDEDIT FOR POST-TRAINING

• SDEdit bridges reconstruction and generation by adding controlled noise to a real image before diffusion denoising, with noise level interpolating between the two.The preservation ratio σ is the fraction of the original latent space preserved after noise is added.Source: §A.5 · pp. 12–13

A.6 POST-TRAINING RESULTS

• RobusTok is relatively insensitive to preservation ratio σ, while other tokenizers show noticeable performance variation across σ values.Source: §A.6 · p. 13, Table 9 · p. 13, Table 10 · p. 14

A.7 LATENT PERTURBATION V.S. OTHER NOISES

• Latent perturbation adds cluster-based random noise modeled on observed sampling errors, unlike scheduled diffusion noise that systematically disrupts the latent space.The perturbation enlarges the decision boundary and zero-shot generalization during inference.Source: §A.7 · p. 15

A.8 VISUALIZATION

• In failed cases, tokenizer post-training recovers structural consistency, improves color fidelity, and sharpens local textures.Source: §A.8 · p. 15, Figure 16 · p. 16