Source-linked AI summary
NesTok: Nested Self-Aligned 1D Tokenizer for Autoregressive Image Generation
Jiawei Zhang, Shuhao Liu, Rong Huang, Yuancheng Li, Zhihui Li, Xiaojun Chang, Changlin Li
TL;DR
Variable-length tokenizers offer adaptive compression, but existing approaches can underuse their representational capacity and show suboptimal reconstruction or generation. NesTok jointly trains across token lengths and aligns shorter sequences with full-length representations. On ImageNet-1K 256×256, it reports rFID 0.98 and gFID 1.46.
Problem
Variable-length tokenizers offer adaptive compression, but nested-dropout approaches may underuse representational capacity, while longer sequences can bring diminishing or degraded AR quality.
Source: §1 · pp. 1–2Method
NesTok jointly optimizes reconstruction across token lengths and aligns shorter-sequence representations with the full-length sequence, without external distillation.
Source: §1 · pp. 2–3, §3.2 · p. 6Results
On ImageNet-1K 256×256, NesTok-XL achieves gFID 1.46 versus One-D-Piece's 2.35, and NesTok reports rFID 0.98.
Source: §1 · p. 3, §4.2 · p. 8Takeaways & Limitations
A single trained AR generator supports multiple token lengths, with generation quality improving as more tokens are generated.
Source: §1 · p. 3, §5 · p. 10
Abstract
from arXiv · showhide
One-dimensional (1D) variable-length visual tokenizers enable adaptive compression by varying the number of tokens, allowing downstream autoregressive (AR) models to flexibly trade off generation quality against computational cost using a single tokenizer. However, existing approaches based on nested dropout often fail to fully exploit the representational capacity of the tokenizer, resulting in suboptimal performance in both image reconstruction and generation. In this work, we introduce NesTok, a nested self-alignment framework tailored to dynamic visual tokenizers. NesTok introduces cross-length training, which jointly optimizes reconstruction across token lengths while using the full-length sequence to guide shorter counterparts, enabling shorter token sequences to approach the reconstruction quality of full-length sequences. On ImageNet, NesTok improves substantially over standard training and achieves an rFID score of 0.98. On downstream image generation, it achieves the state-of-the-art gFID score of 1.46 on ImageNet 256$\times$256 among existing variable-length autoregressive image generation methods. Code will be available at https://github.com/Jiawei804/NesTok.
1 INTRODUCTION
Variable-length visual tokenizers offer adaptive compression, but flexible token counts alone do not ensure useful information allocation or strong autoregressive generation. NesTok combines cross-length reconstruction training with nested self-alignment and reports improved ImageNet reconstruction and generation results.
1 INTRODUCTION
- Dynamic tokenizers let one model adjust token budgets to computational constraints or image complexity, trading generation quality against computational cost.Source: §1 · p. 1
- Longer sequences may yield diminishing or degraded AR quality when prefixes are weak and later tokens add little complementary information.Source: §1 · p. 2
- NesTok jointly optimizes reconstruction across token lengths and aligns shorter sequences' semantic representations with their full-length counterparts, without external distillation.Source: §1 · pp. 2–3
- On ImageNet-1K 256×256, NesTok reaches rFID 0.98 with 256 tokens and gFID 1.46 using vanilla AR generation.Source: §1 · p. 3
2 RELATED WORK
Related work spans image tokenizers, variable-length visual tokenizers, and autoregressive generators. NesTok focuses on reshaping visual representations for compatibility with vanilla autoregressive modeling rather than changing the generator's mechanism.
2 RELATED WORK
- Image tokenizers range from continuous VAE latents and discrete 2D grids to compact 1D sequences that reduce spatial redundancy.Source: §2 · p. 3
- Variable-length methods use ViT decoders or diffusion/flow decoders, but unbalanced information allocation remains a challenge for downstream AR generation.Source: §2 · p. 3
- Unlike methods that alter generation order or attention, NesTok reshapes visual representations to improve balanced allocation and compatibility with vanilla AR modeling.Source: §2 · pp. 3–4
3 METHOD
NesTok combines a variable-length 1D ViT tokenizer with cross-length joint training and nested self-alignment. The method organizes token information from broad prefix representations toward later refinements and supports autoregressive generation at variable token budgets.
3.1 1D VISION TOKENIZER AND AUTOREGRESSIVE MODEL
- The ViT tokenizer quantizes 1D latent tokens and decodes them with mask tokens, while reconstruction-only training does not explicitly order tokens for autoregressive prediction.Source: §3.1 · p. 4
3.2 NESTED SELF-ALIGNMENT LEARNING
- Cross-length joint training decodes both the full sequence and a sampled shorter prefix, giving every latent position reconstruction supervision while training variable-length reconstruction.The two sequences share tokenizer parameters.Source: §3.2 · p. 5
- Nested self-alignment uses full-length decoder features as an online target for shorter prefixes, transferring feature-level guidance without an additional teacher network.Alignment is applied at the first decoder layer by default.Source: §3.2 · pp. 5–6
- Full-length decoding makes tail tokens useful as residual refinements, while alignment favors early tokens that approximate global layout and semantics.Source: §3.2 · p. 6
3.3 VARIABLE-LENGTH AUTOREGRESSIVE MODEL
- A single generator trained with next-token prediction can sample a chosen-length prefix at inference, supporting variable-length generation and quality–compute trade-offs.The sampled prefix is decoded into an image by the tokenizer decoder.Source: §3.3 · p. 6
4 EXPERIMENT
NesTok combines a ViT-based variable-length tokenizer with cross-length training and self-alignment, then evaluates its reconstruction, autoregressive generation, and sampling trade-offs on ImageNet 256×256. Its strongest reported results include 0.98 rFID and 1.46 gFID among compared 1D variable-length methods.
4.1 IMPLEMENTATION DETAILS
- Tokenizer training uses 200K steps with reconstruction, perceptual, and quantization losses, 200K with LPIPS disabled, then 150K adding GAN loss.The tokenizer uses a ViT-B encoder and ViT-L decoder.Source: §4.1 · pp. 6–7
4.2 MAIN RESULTS
- 0.98 rFID beats One-D-Piece’s 1.08 with 390M vs. 641M tokenizer parameters; NesTok-XL’s 1.46 gFID is best among compared 1D variable-length methods.The rFID result is reported without external distillation.Source: Table 1 · p. 7, §4.2 · p. 8
- ReTok-L improves by only 0.01 gFID from 128 to 256 tokens (3.79 to 3.78), while NesTok-L and NesTok-XL improve by 0.20 and 0.25, respectively.NesTok-L substantially outperforms One-D-Piece-L and ReTok-L across evaluated token lengths.Source: §4.2 · p. 8, Figure 6 · p. 8
- Training loss decreases and accuracy rises, while larger autoregressive generators consistently improve unguided gFID at 256 tokens.These curves support the autoregressive compatibility of NesTok’s learned token sequences.Source: Figure 3 · p. 7, §4.2 · pp. 8–9
4.3 ABLATION STUDIES.
- 1.92 gFID with both components beats the 1D Tokenizer’s 2.33; joint training engages all token positions, while alignment chiefly improves shorter-length generation.The ablation reports LlamaGen-Base trained for 200 epochs.Source: §4.3 · p. 9, Table 2 · p. 9
- NesTok achieves the best gFID among compared methods with competitive throughput; Semanticist reaches only 1.03 images per second even with 32 tokens.One-D-Piece has high throughput but comparatively limited generation quality.Source: §4.3 · p. 9, Table 3 · p. 9
5 CONCLUSION
NesTok combines cross-length joint training and feature alignment without external distillation to improve variable-length tokenization. Its representations support coarse-to-fine generation across token lengths, with nearly 100% average codebook utilization.
5 CONCLUSION
- NesTok uses nested self-alignment without external distillation to mitigate tail-token collapse and raise average codebook utilization to nearly 100%.The framework combines cross-length joint training with feature alignment.Source: §5 · p. 10
- NesTok representations support coarse-to-fine refinement, and generation quality improves consistently as more tokens are generated.The AR generator supports multiple token lengths while retaining standard next-token prediction.Source: §5 · p. 10
- The results show that tokenizer training can improve variable-length image generation while retaining standard next-token prediction.Source: §5 · p. 10
A.1 TRAINING PSEUDOCODE
The training procedure initializes the tokenizer, samples data, and encodes each batch into a full-length sequence. It then trains across sampled token lengths using joint reconstruction and alignment losses.
A.1 TRAINING PSEUDOCODE
- The algorithm takes a dataset, encoder, decoder, vector quantizer, maximum token length, iteration count, alignment layer, and token-length sampling distribution, and outputs a trained tokenizer.Source: §A.1 · p. 13
- Each iteration samples a batch and encodes and quantizes it to obtain the full-length sequence.Source: §A.1 · p. 13
- For sampled lengths, the procedure reconstructs images and extracts features, then combines joint reconstruction and alignment losses using full-length features as the stop-gradient target.The resulting total objective trains the tokenizer.Source: §A.1 · p. 13
A.2 MORE RESULTS
Across evaluated token lengths, NesTok-L achieves the lowest gFID among compared ViT-decoder methods, while its reconstruction results vary against other tokenizers by length. Visualizations show reconstruction improvements over nested dropout and coarse-to-fine refinement during generation.
A.2 MORE RESULTS
- Compared with nested dropout, NesTok reconstructions avoid pronounced artifacts below 32 tokens and retain a visual-fidelity advantage at longer lengths.Source: §A.2 · p. 13
- Generated images progressively refine coarse content into sharper details as token count increases, including object shape, texture, and lighting.The visualizations cover 8–256 tokens without classifier-free guidance.Source: §A.2 · p. 13, Figure 8 · p. 14, Figure 9 · p. 14
- NesTok rFID improves from 3.49 at 32 tokens to 0.98 at 256 tokens and consistently outperforms ReTok-S-B.It matches One-D-Piece at 64 tokens and is better at 128 and 256, while DetailFlow is better at 256 tokens.Source: §A.2 · pp. 13–14, Table 4 · p. 14
- NesTok-L has the lowest gFID among compared methods at every evaluated token length, with gFID falling from 4.82 to 2.08 without cfg and from 3.78 to 1.50 with cfg as length rises from 32 to 256 tokens.The comparison uses a single trained AR model.Source: §A.2 · pp. 14–15, Table 5 · p. 15