Source-linked AI summary
AffineTok: Semantic Affine Consistency for Diffusion-Friendly Visual Tokenizer
Junqiu Yu, Pandeng Li, Yikai Wang, Jiaxing Zhao, Yujie Wei, Kaixun Jiang, Quanhao Li, Hongtao Yu, Zhihang Liu, Zhaohe Liao, Junjie Zhou, Yun Zheng, Yu Liu, Yanwei Fu
TL;DR
Existing tokenizer supervision does not ensure that semantics decoded from predicted clean latents align with clean-image semantics, leaving semantic organization for diffusion underexplored. AffineTok addresses this with SAC-guided GSCT and PMSA training, and reaches gFID 1.21 without classifier-free guidance and 1.10 with guidance on ImageNet 256 × 256.
Problem
Existing methods supervise semantics in clean or noisy latents but do not constrain semantics decoded from diffusion’s predicted clean latent, leaving their required organization for denoising underexplored.
Method
AffineTok defines SAC, proxies it with M_SAC using interpolated neighboring clean latents, and promotes it during tokenizer training with GSCT and PMSA.
Results
1.21 gFID without classifier-free guidance and 1.10 with guidance are achieved on ImageNet 256 × 256, while M_SAC reaches a Pearson correlation of 0.960 with SiT-XL gFID.
Takeaways & Limitations
M_SAC closely tracks generation quality, motivating SAC-guided tokenizer training that leaves downstream diffusion training unchanged.
Takeaways & Limitations
The formulation assumes a tokenizer encoder and decoder together with a frozen visual foundation model ϕ.
Abstract
from arXiv · showhide
Visual tokenizers increasingly inject semantic supervision into latent spaces to make downstream diffusion easier. Yet how these semantics should be organized to facilitate denoising remains underexplored. In this paper, we define the semantic recovery objective: the denoising process should recover the semantic content of the clean image from noisy latent, and a good tokenizer should make it easier. Existing approaches train a projector to predict the semantics directly from the noisy latent. We argue that this predicts the average of clean-image semantics, whereas what really needs to be aligned is the semantics of averaged clean latents. More importantly, we demonstrate that the semantic recovery error orthogonally decomposes into the error of the optimal semantic prediction directly from the noisy latent and the error between these two predictions. We therefore identify their consistency as the missing requirement and call it Semantic Affine Consistency (SAC). To examine whether this overlooked requirement is closely related to downstream generation, we introduce M_SAC, a tokenizer-side proxy for SAC. Across the evaluated tokenizers and diffusion model scales, M_SAC closely tracks generation quality, reaching a Pearson correlation of 0.960 with SiT-XL gFID, thereby motivating SAC-guided tokenizer training. We then introduce AffineTok, which promotes SAC through two complementary, training-only components. Global Semantic Coordination Token (GSCT) coordinates the semantic organization of clean latents, keeping semantic averaging meaningful, while Posterior-Mean Semantic Alignment (PMSA) predicts posterior-mean latents from noisy inputs and supervises their semantics. On ImageNet 256, compared with the baseline, AffineTok reduces gFID by 26% at 20 epochs and, with continued training, achieves a new state-of-the-art gFID of 1.21 without classifier-free guidance and 1.10 with guidance.
1 INTRODUCTION
AffineTok identifies Semantic Affine Consistency as the missing requirement for organizing tokenizer semantics so diffusion can recover clean-image content from noisy latents. It measures this requirement with M_SAC and trains it through GSCT and PMSA, improving downstream generation.
- Motivation: Semantic recovery requires aligning clean-image semantics with the semantics decoded from the predicted clean latent, not only predicting semantics from the noisy latent.Under squared loss, direct noisy-latent supervision predicts average clean-image semantics, while nonlinear semantic mapping may not commute with averaging.
- Semantic Affine Consistency: Semantic Affine Consistency (SAC) formalizes agreement between semantics predicted directly from noisy latents and semantics decoded from posterior-mean clean-latent predictions.The semantic recovery error orthogonally decomposes into direct semantic-prediction error and the gap between these two semantic predictions.
- Evaluation: M_SAC closely tracks generation quality across 14 visual tokenizers and diffusion-model scales, reaching a Pearson correlation of 0.960 with SiT-XL gFID.The metric uses interpolated neighboring clean latents as local proxies for posterior averaging, avoiding downstream diffusion training for every tokenizer.
- AffineTok: GSCT coordinates clean patch-latent semantics across images, while PMSA estimates posterior-mean latents from noisy inputs and supervises their semantics.Both components are used only during tokenizer training, leaving downstream diffusion training unchanged.
2 METHOD
The method identifies Semantic Affine Consistency (SAC) as the missing alignment between semantics predicted from noisy latents and semantics decoded from posterior-mean clean latents. It probes SAC with MSAC and promotes it during tokenizer training through GSCT and PMSA.
- 2.1 SEMANTIC AFFINE CONSISTENCY: Semantic recovery requires reconstructing both the clean latent and its decoded semantic content from noisy observations.The paper names this latter requirement semantic recovery.
- 2.1 SEMANTIC AFFINE CONSISTENCY: Existing semantic supervision predicts clean-image semantics directly from noisy latents, reducing semantic uncertainty but not enforcing agreement with posterior-mean latent semantics.The optimal direct semantic predictor minimizes the uncertainty component of semantic recovery error.
- 2.1 SEMANTIC AFFINE CONSISTENCY: Semantic Affine Consistency (SAC) requires the average of clean-image semantics to equal the semantics decoded from the averaged clean latent.This requirement arises because the semantic decoding map is generally nonlinear, so averaging and decoding need not commute.
- 2.2 SAC ALIGNS WITH DOWNSTREAM GENERATION CAPACITY: MSAC probes SAC by interpolating neighboring clean latents and comparing interpolated endpoint semantics with semantics decoded from the interpolated latent.The interpolation serves as a local proxy for posterior averaging; lower MSAC indicates closer preservation of semantic interpolation.
- 2.2 SAC ALIGNS WITH DOWNSTREAM GENERATION CAPACITY: 0.960 Pearson correlation links MSAC with SiT-XL gFID across 14 visual tokenizers.The reported association motivates using MSAC as a tokenizer-side proxy for downstream generation capacity.
- 2.3 SAC-GUIDED TOKENIZER TRAINING: GSCT coordinates clean-latent semantic organization, while PMSA predicts posterior-mean latents and supervises their semantics during tokenizer training.Both components are training-only; PMSA uses a posterior predictor and a semantic projector calibrated on clean latents.
3 EXPERIMENTS
The experiments evaluate MSAC as a predictor of downstream generation, AffineTok’s diffusability, the roles of GSCT and PMSA, and reconstruction–representation trade-offs. Across these tests, SAC-guided components improve semantic organization, semantic recovery, and generation while preserving comparable reconstruction.
- SAC metric predicts downstream generation: MSAC has the strongest correlation with downstream gFID across tokenizer-side metrics and model scales, reaching a Pearson correlation of 0.960 for SiT-XL.Correlations are reported across 14 tokenizers for SiT-XL and 13 for SiT-B.
- Improved tokenizer diffusability: 26% lower gFID at 20 epochs improves RecTok’s 4.44 to 3.28 while increasing IS from 149.0 to 181.3.After 600 epochs, AffineTok reaches gFID 1.21 without guidance and 1.10 with guidance.
- GSCT semantic consistency: GSCT reduces cross-split semantic-axis inconsistency by 69% on average across three angular thresholds.The evaluation uses repeated partitions of ImageNet class subsets to compare centroid-difference directions.
- PMSA semantic recovery: GSCT and PMSA reduce top-10 semantic retrieval miss rates by 56% on average across three severe noise levels.The probe applies a projector trained on clean latents to predicted clean-latent outputs from identically trained denoisers.
- Reconstruction and representation quality: AffineTok gains about 10 percentage points in pooled and flattened linear-probing accuracy while maintaining comparable reconstruction quality.The results indicate improved latent semantics without materially compromising reconstruction.
- Component ablations: GSCT and PMSA each improve over RecTok alone, and their combination further reduces gFID to 3.28 under the shared ablation protocol.GSCT coordinates clean patch-latent semantics across images, whereas PMSA supervises recovery from predicted clean latents.
4 RELATED WORK
Related work shapes visual tokenizer latents for generation, develops metrics for latent diffusability, and builds scalable latent diffusion backbones. Existing diffusability metrics do not address semantic recovery.
- Semantically and structurally organized latent spaces: Recent visual tokenizers align latent representations with visual foundation-model features and impose structural or noisy-state semantic supervision.These approaches move beyond reconstruction-focused compression toward latents shaped for generation.
- Metrics for latent diffusability: iFID measures local manifold continuity and VIV measures irreducible velocity ambiguity, but neither metric considers semantic recovery.The paper positions SAC-related evaluation as addressing this omitted dimension.
- Latent diffusion generation: Latent diffusion reduces high-resolution synthesis cost through denoising in compact continuous autoencoder spaces, with DiT and SiT providing scalable transformer-based frameworks.The related systems include diffusion- and flow-based objectives over latent patches.
5 CONCLUSION
The paper identifies SAC as a missing requirement for organizing tokenizer semantics around downstream diffusion and introduces MSAC, GSCT, and PMSA to operationalize and promote it. AffineTok achieves strong ImageNet generation results without changing downstream diffusion training.
- Conclusion: AffineTok achieves gFID 1.21 without classifier-free guidance and 1.10 with guidance on ImageNet 256 × 256.GSCT coordinates semantics across images, while PMSA supervises recovery from posterior-mean latent predictions.
- Conclusion: MSAC is a tokenizer-side SAC proxy based on interpolation between neighboring clean latents and closely tracks generation quality.SAC is defined as agreement between semantics directly predicted from noisy latents and semantics decoded from posterior-mean clean latents.