Source-linked AI summary
IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder
Yitong Chen, Zijie Diao, Junke Wang, Lingyu Kong, Yixuan Ren, Bo He, Yu-Gang Jiang, Zuxuan Wu
TL;DR
Deep VFM features support semantics but lose fine-grained detail, limiting discrete representation autoencoder reconstruction. Ideal jointly aligns quantized tokens with shallow and deep features, achieving 0.61 rFID on ImageNet and state-of-the-art autoregressive generation with 1.89 gFID.
Problem
Deep VFM features emphasize high-level semantics while losing fine-grained visual attributes, leaving existing discrete representation autoencoders suboptimal for faithful reconstruction.
Method
Ideal combines shallow appearance-rich and deep semantic VFM features before quantization, supervising tokens to retain both for reconstruction and generation.
Results
Ideal improves reconstruction fidelity and semantic consistency while preserving strong VFM semantics and achieving state-of-the-art autoregressive generation, including 0.61 rFID on ImageNet.
Takeaways & Limitations
Ideal shows that combining shallow detail with deep semantics can produce discrete visual tokens that retain both high-fidelity reconstructability and semantic richness.
Takeaways & Limitations
Ideal reports lower gIS than VFMTok at similar parameter counts, despite strong fidelity-oriented gFID and sFID metrics.
Abstract
from arXiv · showhide
Built on pretrained vision foundation models (VFMs), representation autoencoders (RAEs) have recently emerged as a promising approach for constructing semantically rich latent spaces for image generation. However, their reconstruction quality often remains suboptimal, largely because deep VFM representations do not preserve sufficient fine-grained visual detail. This limitation becomes even more severe after discretization, where missing low-level information is difficult to recover. In fact, we observe that shallow VFM features retain considerably richer local appearance and structural detail, which complements the high-level semantics carried by deep features used in existing RAEs. Motivated by this complementary property, we propose Ideal, an In-depth Alignment framework for discrete representation autoencoding. By jointly aligning quantized tokens with both shallow and deep VFM features, Ideal enables the resulting discrete visual tokens to preserve both visual fidelity and rich semantics. Extensive experiments demonstrate that Ideal yields superior reconstruction performance, achieving 0.61 rFID on ImageNet and outperforming the previous best method by 0.28. When used for autoregressive image generation, Ideal further produces a gFID of 1.89, establishing a new state of the art for autoregressive image generation.
1 Introduction
Ideal addresses the reconstruction bottleneck in discrete representation autoencoding by aligning tokens with complementary shallow and deep VFM features. This design preserves fine-grained visual detail alongside high-level semantics and improves ImageNet reconstruction and semantic preservation.
- Problem: Deep VFM features emphasize high-level semantics but lose fine-grained attributes such as color, texture, and local structure, limiting faithful reconstruction.Existing RAEs therefore remain suboptimal for reconstruction despite using semantically rich pretrained representations.
- Motivation: A depth-wise study reveals a trade-off: shallow VFM representations reconstruct better, whereas deeper representations preserve stronger semantics.The study evaluates discretized intermediate representations from both semantic-preservation and reconstruction-fidelity perspectives.
- Method: Ideal combines appearance-rich shallow features with semantically informative deep features before vector quantization, then supervises tokens to recover both feature types.This unified representation is designed to preserve visual details and high-level semantics simultaneously.
- Results: 0.61 rFID is achieved on ImageNet, outperforming previous tokenizers by 0.28 while maintaining 80.89% zero-shot ImageNet classification accuracy.These results assess reconstruction fidelity and semantic preservation, respectively.
2 Related Work
Related work spans conventional continuous and discrete visual tokenizers, VFM-based tokenizers that incorporate pretrained semantic representations, and autoregressive models for sequence-based visual generation.
- Conventional Tokenizers: Continuous tokenizers typically use VAEs with encoders parameterizing continuous latent distributions and decoders reconstructing images.
- Conventional Tokenizers: Discrete tokenizers such as VQ-VAE learn finite codebooks and quantize encoder features through nearest-neighbor lookup to produce token indices.VQGAN adds perceptual and adversarial losses, while ViT-VQGAN uses Transformer-based architectures.
- VFM-based Tokenizers: VFM-based tokenizers incorporate pretrained vision foundation model semantics during tokenization or generation to improve generation quality and training efficiency.RAE directly applies tokenization to VFM features as a continuous semantic tokenizer.
- Autoregressive Visual Generation: Strong discrete visual tokenizers enable autoregressive models to generate diverse, high-quality images and videos by modeling compressed visual token sequences.VAR reframes autoregressive learning from raster-scan next-token prediction to coarse-to-fine next-scale prediction.
3 Method
Ideal builds a discrete semantic tokenizer by fusing shallow and deep features from a frozen VFM before quantization. It reconstructs both feature levels and pixels, using alignment objectives to preserve fine-grained detail and semantics for autoregressive modeling.
- Motivation: Shallow VFM features favor pixel reconstruction, whereas deep features better preserve semantics after quantization, motivating their complementary use.The depth-wise probe finds that shallow features are more reconstruction-friendly while deep features are more semantic.
- Feature fusion: Ideal extracts matched-shape shallow and deep features from a frozen VFM and fuses them with a lightweight cross-attention block before quantization.Deep features provide queries and shallow features provide complementary information, without additional resizing or projection.
- Vector quantization: The fused representation is quantized with standard vector quantization into discrete token indices, using normalized codebook vectors and down-factorization for stable, lower-dimensional lookup.De-quantization recovers embeddings in the original dimension for subsequent decoding.
- Two-step decoding: A ViT feature decoder reconstructs the unified representation, while separate linear heads recover deep semantic and shallow spatial features; the deep output also drives pixel reconstruction.The reconstructed deep feature serves as the interface for semantic preservation evaluation and is passed to a lightweight pixel decoder.
- Objectives: Training combines VQ and auto-encoding losses with deep and shallow feature-alignment terms that preserve semantic structure and fine-grained detail.A frozen DINOv1-s discriminator replaces PatchGAN to provide semantically meaningful adversarial guidance.
- Autoregressive modeling: After training, flattened discrete codes are modeled by an autoregressive Transformer with next-token prediction, and 2D RoPE captures spatial locality during generation.Generated token sequences are decoded back into images by the tokenizer decoder.
4 Experiments
Experiments show that Ideal preserves reconstruction fidelity and VFM semantics while enabling strong class-conditional autoregressive generation. Its performance remains robust across controlled settings, fusion choices, auxiliary supervision, and VFM backbones.
- Experimental setup: Ideal is trained with SigLIP2-Large-384 features from the 8th and 24th Transformer blocks, using a VQ codebook of size K=16384 and vector dimension d=64.Reconstructed images are resized to 256×256 for metric computation.
- Semantic preservation: 80.89% Top-1 and 96.40% Top-5 zero-shot accuracy show that Ideal retains near-original SigLIPv2 semantic structure after quantization and decoding.SigLIPv2’s deepest feature achieves 83.23% Top-1 and 97.11% Top-5, and Ideal’s decoded features remain compatible with SigLIPv2 text embeddings without additional contrastive training.
- Class-conditional image generation: 1.89 gFID at 3B parameters establishes Ideal as a state-of-the-art autoregressive image generator, improving from 1.95 gFID at XXL scale.Ideal-B reaches 3.38 gFID, while Ideal-L reaches 2.26 gFID; Ideal-XXL also achieves the best sFID of 4.81.
- Controlled generation and backbone robustness: Ideal improves reconstruction quality and generation fidelity over LlamaGen and VFMTok under controlled 256×256 training, while SigLIP2 better supports semantic retention than DINO-style features.DINO-style features tend to favor reconstruction, whereas SigLIP2 provides vision–language-aligned representations that interact directly with text.
- Ablation studies: Removing feature fusion reduces reconstruction quality, attention better preserves semantics, and auxiliary shallow-feature reconstruction improves reconstruction consistently.These ablations support integrating shallow spatial cues into deep semantic features and explicitly supervising reconstruction-friendly signals.
5 Conclusion
Ideal is a discrete representation autoencoder that converts VFM features into discrete codes for autoregressive image generation while preserving semantic richness and high-fidelity reconstructability. Its design exploits the complementary roles of shallow and deep VFM layers: spatial detail for reconstruction and stronger semantics.
- 5 Conclusion: Ideal converts VFM features into discrete codes for autoregressive image generation while preserving semantic richness and high-fidelity reconstructability.The method is presented as a discrete representation autoencoder.
- 5 Conclusion: Shallow VFM layers retain spatial detail useful for reconstruction, whereas deeper layers encode stronger semantics.This depth-dependent trade-off motivates combining information across the VFM feature hierarchy.
- 5 Conclusion: Ideal exploits the complementarity between shallow-layer detail and deep-layer semantics.
A Ideal Implementation Details · A.1 Tokenizer Training Details
Ideal’s tokenizer training recipe follows VFMTok while adapting patch size and input resolution to preserve feature-map consistency. It is trained on ImageNet-1K with standard augmentations at 384 × 384 and evaluated at 256 × 256.
- A Ideal Implementation Details: Ideal’s tokenizer training recipe closely follows prior work VFMTok.
- A Ideal Implementation Details: VFMTok’s 14-pixel patch size and 336-pixel input resolution motivate Ideal’s use of a 16-pixel patch size and 384-pixel input resolution.
- A Ideal Implementation Details: These settings maintain consistency in the feature-map size.
- A.1 Tokenizer Training Details: Ideal is trained on the ImageNet-1K training set.
- A.1 Tokenizer Training Details: Training uses random resized crops and horizontal flips.
- A.1 Tokenizer Training Details: Training inputs use a resolution of 384 × 384.
- A.1 Tokenizer Training Details: Reconstructions are evaluated at 256 × 256 following the common protocol.
- A.1 Tokenizer Training Details: Table 8 summarizes the tokenizer implementation details.
A.2 Autoregressive Training Details · B Additional Qualitative Results
The paper trains an autoregressive generator on discrete tokenizer sequences, improving throughput by pre-extracting tokens offline after ten-crop preprocessing. Additional qualitative results examine reconstruction, generation, and representative tokenizer and autoregressive-model failures.
- A.2 Autoregressive Training Details: The autoregressive generator models discrete token sequences produced by the tokenizer.
- A.2 Autoregressive Training Details: Unlike VFMTok, the pipeline does not extract token sequences on-the-fly at every training epoch.
- A.2 Autoregressive Training Details: The training images undergo ten-crop preprocessing before autoregressive training.
- A.2 Autoregressive Training Details: All token sequences are pre-extracted offline before autoregressive training.This follows the original LlamaGen training pipeline.
- A.2 Autoregressive Training Details: Offline token pre-extraction significantly improves training throughput.
- B Additional Qualitative Results: Additional qualitative results cover image reconstruction and generation.
- B Additional Qualitative Results: The qualitative analysis further examines representative failures of the tokenizer and autoregressive model.
B.1 Reconstruction Results
Ideal’s tokenizer produces fine-grained reconstructions across diverse scenes and objects.
- Ideal’s tokenizer produces fine-grained reconstructions across diverse scenes and objects.
B.2 Generation Results · B.3 Failure Cases
Ideal generates images spanning varied styles, subjects, and compositions, but reconstruction degrades on faces and text and generation can produce artifacts in fine-structure regions. These failures are attributed to limited domain coverage in tokenizer training data.
- B.2 Generation Results: Ideal synthesizes images with varied styles, subjects, and compositions.
- B.2 Generation Results: The generation results include samples demonstrating diversity across visual styles.
- B.2 Generation Results: The generated samples also vary in their depicted subjects.
- B.2 Generation Results: Their compositions likewise show substantial variation.
- B.3 Failure Cases: Reconstruction quality degrades on faces and text.This behavior is illustrated in Figure 7.
- B.3 Failure Cases: The authors attribute these failures to limited domain coverage in tokenizer training data.
- B.3 Failure Cases: The tokenizer is trained only on ImageNet, which sparsely covers close-up faces and rich-text images.
- B.3 Failure Cases: Generation can still produce artifacts in fine-structure regions such as hands.Figure 8 shows this failure case.
C Limitation and Future Work
The tokenizer’s ImageNet-focused training limits reconstruction and semantic evaluation coverage, motivating broader-data adaptation, expanded semantic benchmarks, and extension to video.
- Limitations: ImageNet-focused tokenizer training limits domain coverage, degrading reconstruction on faces, text, and other long-tail visual patterns.The semantic-preservation evaluation also centers on ImageNet zero-shot classification, emphasizing category-level semantics without fully covering broader capabilities.
- Future Work: Pretraining or adapting the tokenizer on larger, more diverse datasets is a direct step toward improving coverage of faces, text, and long-tail domains.The paper also plans to evaluate decoded interface features on broader semantic benchmarks.
- Future Work: Broader semantic benchmarks are planned to better characterize semantic preservation.This addresses the limited scope of ImageNet zero-shot classification evaluation.
- Future Work: The discrete-token formulation may extend naturally to videos through temporal consistency for semantic tokenization and generation.