PaperlayerView exact revision on arXiv

Source-linked AI summary

ResTok: Learning Hierarchical Residuals in 1D Visual Tokenizers for Autoregressive Image Generation

Xu Zhang, Cheng Da, Huan Yang, Kun Gai, Ming Lu, Zhan Ma

arXiv:2601.03955v1cs.CV

TL;DR

Existing 1D visual tokenizers often treat visual data as flat sequences, lacking cross-level fusion and producing redundant codebooks. ResTok builds hierarchies and semantic residuals into image and latent tokens, with a hierarchical AR generator that predicts a level at a time. It achieves 2.34 gFID on ImageNet 256×256 with 9 sampling steps.

  • Problem

    Existing 1D tokenizers lack cross-level fusion and often produce high-entropy codebooks with redundant latent embeddings, which are unfriendly to AR modeling.

    Source: §1 · p. 2

  • Method

    ResTok progressively merges image tokens and learns semantic residuals across hierarchies, alongside a hierarchical AR generator that predicts latent-token levels.

    Source: §1 · p. 2

  • Results

    2.34 gFID on ImageNet 256×256 with only 9 sampling steps; ResTok achieves state-of-the-art AR generation performance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments verify that hierarchical residuals and implicit alignments enhance reconstruction and generation efficiencies.

    Source: §6 · p. 8

Abstract

from arXiv · show

Existing 1D visual tokenizers for autoregressive (AR) generation largely follow the design principles of language modeling, as they are built directly upon transformers whose priors originate in language, yielding single-hierarchy latent tokens and treating visual data as flat sequential token streams. However, this language-like formulation overlooks key properties of vision, particularly the hierarchical and residual network designs that have long been essential for convergence and efficiency in visual models. To bring "vision" back to vision, we propose the Residual Tokenizer (ResTok), a 1D visual tokenizer that builds hierarchical residuals for both image tokens and latent tokens. The hierarchical representations obtained through progressively merging enable cross-level feature fusion at each layer, substantially enhancing representational capacity. Meanwhile, the semantic residuals between hierarchies prevent information overlap, yielding more concentrated latent distributions that are easier for AR modeling. Cross-level bindings consequently emerge without any explicit constraints. To accelerate the generation process, we further introduce a hierarchical AR generator that substantially reduces sampling steps by predicting an entire level of latent tokens at once rather than generating them strictly token-by-token. Extensive experiments demonstrate that restoring hierarchical residual priors in visual tokenization significantly improves AR image generation, achieving a gFID of 2.34 on ImageNet-256 with only 9 sampling steps. Code is available at https://github.com/Kwai-Kolors/ResTok.

1. Introduction

ResTok restores hierarchical residual structure to 1D visual tokenization, addressing limited cross-level fusion and high-entropy codebooks in existing methods. Its hierarchical AR generator reaches 2.34 gFID on ImageNet 256×256 with 9 sampling steps.

1. Introduction

  • Existing 1D tokenizers lack cross-level feature fusion and often produce redundant, high-entropy codebooks that are unfriendly to AR modeling.They extract features along network depth without fusing multiple levels at a given layer.Source: §1 · p. 2
  • ResTok progressively merges image tokens into coarser features so latent tokens can fuse in-context features across hierarchies.This hierarchical representation increases representational capacity, while semantic residuals reduce information overlap and concentrate latent distributions.Source: §1 · p. 2
  • 2.34 gFID on ImageNet 256×256 with only 9 sampling steps; ResTok achieves state-of-the-art autoregressive generation performance.The hierarchical AR generator predicts an entire level of latent tokens at once, reducing sampling steps with acceptable generation-performance degradation.Source: §1 · p. 2
  • ResTok’s cross-level bindings emerge without explicit constraints, aligning coarser latent tokens with high-level image tokens and finer latents with low-level residual details.Figure 1 contrasts depth-only feature querying in existing 1D tokenizers [16, 22, 25, 52] with ResTok’s querying across depth and hierarchy.Source: Figure 1 · p. 1, §1 · p. 2

2. Related Work

Related work traces a shift from flattened spatial latents toward query-based 1D tokenizers and increasingly varied autoregressive generation strategies. ResTok uses LlamaGen as its testbed and proposes a hierarchical variant for acceleration.

2.1. Visual Tokenization

  • Raster-scanned 2D latents can disrupt autoregressive causality at scan turns, motivating query-based 1D tokenizers that learn naturally sequential tokens.SPAE explicitly aligns token hierarchies with semantic structures, while SEED and TiTok learn latent sequences directly from image patches.Source: §2.1 · p. 2
  • SpectralAR and DetailFlow link token length to frequency bands or spatial resolutions, while ImageFolder, GigaTok, and VFMTok explore distinct latent-hierarchy designs.These approaches include hand-crafted constraints, additive residual quantization, progressive latent initialization, and queries over multiple foundation-model levels.Source: §2.1 · p. 2

2.2. Autoregressive Image Generation

  • AR generation research spans discrete-latent transformers, coarse-to-fine generation, flow matching, long-range refinement, random prediction order, and VQ-free generation.The cited examples include VAR, FlowAR, Infinity, MaskGIT, and MAR.Source: §2.2 · pp. 2–3
  • ResTok builds its accelerated hierarchical generator on LlamaGen, chosen for its simplicity and integration capability with unified multimodal models.LlamaGen is described as a foundation for many subsequent works.Source: §2.2 · p. 3

3. Residual Tokenizer

ResTok encodes images into quantized 1D latent hierarchies, progressively merging image features and forming semantic residuals across levels. Its optimization aligns representations with a vision foundation model while allowing latent hierarchies to self-align.

3.1. Pipeline Overview

  • The ViT encoder processes image tokens with latent tokens, quantizes the resulting latent hierarchies, and a decoder uses them to restore image tokens for image reconstruction.A CNN encoder creates the initial image tokens, and a CNN decoder produces the reconstructed image.Source: §3.1 · p. 3

3.2. Hierarchical Representations in ViT

  • Progressive merging creates coarser image-token scales within the ViT, making multi-level features accessible to hierarchical latent tokens.Residual merging blocks are inserted at intervals to build the scales.Source: §3.2 · pp. 3–4

3.3. Semantic Residuals

  • Semantic residuals for image and latent tokens address overlap between levels, with residual initialization guiding training and reducing redundant latent information.For latent tokens, pooled features are upsampled, differenced against the original features, and pooled to form residual tokens.Source: §3.3 · p. 4

3.4. Optimization Strategies

  • Vision-foundation alignment guides both the encoder’s coarsest image tokens and the decoder’s output features, supporting alignment at the quantization bottleneck.Encoder tokens are aligned to the model’s [CLS] feature, while decoder outputs are aligned to its visual patch tokens.Source: §3.4 · p. 4
  • Latent hierarchies are not explicitly tied to image hierarchies; nested dropout and shared reconstruction objectives let the tokenizer learn features suited to each token length.The objectives combine MSE, perceptual, GAN, and vision-foundation losses.Source: §3.4 · pp. 4–5

4. Hierarchical Autoregressive Generation

The hierarchical autoregressive (HAR) generator speeds sampling by predicting latent tokens a hierarchy at a time. A vanilla autoregressive prefix initializes HAR generation, while special mask tokens accommodate differing hierarchy sizes.

4. Hierarchical Autoregressive Generation

  • HAR predicts a whole latent hierarchy at once, substantially reducing sampling steps compared with strict next-token generation.The first HAR group combines its predicted token with mask tokens so the group reaches the next hierarchy’s token count.Source: §4 · p. 5
  • A vanilla autoregressive next-token prefix initializes HAR prediction and reduces accumulated sampling error at the start of generation.The number of prefix tokens matches the minimal remaining tokens used in nested token dropout training.Source: §4 · p. 5
  • HAR uses mask tokens to match each predicted group to the next hierarchy’s token count, which varies across ResTok hierarchies.Training applies a hierarchical grouped attention mask while retaining LlamaGen’s optimization objective.Source: §4 · p. 5

5. Experiments

Experiments compare ResTok with representative continuous and discrete tokenizers, assess its coarse-to-fine representations, and ablate its design choices. Results favor hierarchical residuals and show that HAR sharply reduces sampling steps, with a trade-off in generation quality.

5.1. Experimental Settings

  • The experiments train ResTok on ImageNet at 256×256 for 200 epochs and train LlamaGen-L under HAR for 300 epochs; ablations use shorter 30- and 50-epoch runs, respectively.The tokenizer uses 128 latent tokens and an 8,192-entry codebook; evaluation includes FID, IS, precision, recall, and codebook entropy for ablations.Source: §5.1 · p. 6

5.2. Quantitative Results

  • Query-based visual tokenizers generally reach below 3.0 gFID with ∼300M generators, while around 128 latent tokens often yield rFID near 1.0 as model capacity and sequence length scale.The passage describes these trends as evidence that query-based tokenizers align more naturally with AR image generation.Source: §5.2 · p. 6
  • 2.34 gFID with 9-step sampling makes ResTok’s HAR generator state of the art among query-based tokenizers, outperforming prior methods with stronger rFID and accelerated AR models using longer sequences.ResTok uses 128 latent tokens versus ImageFolder’s 286, with better gFID and sampling efficiency.Source: §5.2 · p. 6

5.3. Qualitative Results

  • ResTok’s latent groups refine images coarse-to-fine, progressing from semantic attributes such as identity and layout to texture and boundary details, unlike frequency- or texture-based refinement in SpectralAR and DetailFlow.Attention maps associate the first 16 latent tokens with coarse image scales and later tokens with finer scales; the coarsest scale acts as a global semantic source.Source: §5.3 · pp. 6–7

5.4. Ablation Study

  • Hierarchies enhance representation and reconstruction, while applying residuals to both image and latent tokens gives the lowest codebook entropy and gFID among tested residual settings.Four hierarchies balance rFID and complexity in the ablations.Source: §5.4 · pp. 7–8
  • Pooling image tokens by a factor of 2 gives the best generation performance among tested settings, balancing abstraction against semantic loss at 4× pooling.Source: §5.4 · p. 8
  • Higher codebook entropy indicates a more dispersed latent distribution and poorer gFID; representation alignment works better when applied on both encoder and decoder sides.Encoder alignment guides feature extraction, while decoder alignment preserves semantics through the quantization bottleneck.Source: §5.4 · p. 8
  • HAR cuts sampling from 128 steps to 8 or 9 with acceptable gFID degradation; a vanilla AR prefix further reduces sampling errors, and tokenizer gFID peaks around 750k training steps.Reconstruction rFID keeps improving with longer tokenizer training, while generation quality degrades after the gFID optimum.Source: §5.4 · p. 8

6. Conclusion

The conclusion presents ResTok as a 1D visual tokenizer that combines hierarchical image-token merging with semantic residuals across levels. The appendix figures illustrate its attention masks and 2D positional encoding.

6. Conclusion

  • ResTok progressively merges image tokens and accumulates semantic residuals, organizing latent tokens coarse-to-fine with natural alignment between image and latent hierarchies.The conclusion reports that experiments support improved reconstruction and generation efficiencies, and identifies fidelity and unified understanding-generation models as future directions.Source: §6 · p. 8

Appendix

  • Figure 8 illustrates tokenizer and generator attention masks, using three image-token scales and four latent hierarchies for the tokenizer example.Its generator example shows four vanilla AR tokens and two HAR groups.Source: Figure 8 · p. 12
  • Figure 9 illustrates ResTok’s 2D RoPE implementation with three image-token scales and three latent tokens.Source: Figure 9 · p. 12

A.1. Architecture

The architecture combines CNN and ViT encoder-decoder configurations with attention masks and modified 2D M-RoPE. It also specifies residual latent-token initialization and merging procedures.

A.1. Architecture

  • The ViT encoder and decoder each use 24 transformer layers, 1024 dimensions, and 16 heads; a linear layer bridges their dimensions to the CNN encoder-decoder.Encoder attention masks enforce causality, and modified 2D M-RoPE replaces TiTok’s learnable positional embeddings.Source: §A.1 · p. 12
  • Residual latent-token initialization starts from image tokens and the number of hierarchical levels, pools an initial token, then alternates width and height growth before returning latent tokens.The residual merging process is also formalized as an algorithm.Source: §A.1 · p. 12
  • M-RoPE orders encoder positions as coarse-to-fine 2D image tokens followed by 1D latent tokens, while the decoder places latent tokens before masked 2D image tokens.The generator’s attention mask enables next-hierarchy prediction.Source: §A.1 · p. 12

A.2. Training

ResTok and LlamaGen-L are trained from scratch on ImageNet-1K, with random resizing and cropping used to prepare images. Training uses nested token dropout, while CFG settings specify alternative sampling schedules.

A.2. Training

  • Both ResTok and LlamaGen-L are trained from scratch on ImageNet-1K’s 1,281,167 images across 1,000 classes.Images are randomly resized by a factor between 0.8 and 1.0, then randomly cropped to 256×256; generator codes use LlamaGen’s scripts and augmentations.Source: §A.2 · p. 12
  • CFG step schedules activate at a specified sampling-trajectory ratio, while linear schedules ramp from 1.0 to the maximum guidance value; Top-K filtering precedes Top-P filtering.A K or P value of 0 bypasses the corresponding filtering stage.Source: Table 6 · p. 13
  • Nested token dropout preserves at least 4 tokens; there is an 80% chance of no dropout, and dropout probability decreases exponentially for shorter target lengths.Keeping probabilities for each token length are listed in Table 9.Source: §A.2 · p. 13

A.3. Evaluation

Evaluation follows established protocols for reconstruction and generation, selecting a final generator checkpoint after searching CFG schedules. Codebook entropy is measured as an indicator of codebook-usage uniformity.

A.3. Evaluation

  • Reconstruction follows TiTok’s protocol, while generation uses GigaTok scripts and selects the 750K-step checkpoint after searching CFG schedules for each checkpoint.Ablations use the 150K-step tokenizer and 250K-step generator checkpoints without CFG.Source: §A.3 · p. 13
  • Codebook entropy is the empirical Shannon entropy of selected codebook entries, measured in bits; higher entropy indicates more uniform usage, while lower entropy indicates concentration.A small constant is added for numerical stability, set to 1 × 10^-8 as in TiTok.Source: §A.3 · p. 13

B. Additional Results

The vanilla AR variant improves gFID over HAR at substantially higher sampling cost. The authors report using released projects under their licenses and adhering to their requirements.

B. Additional Results

  • gFID improves from 2.34 with HAR’s 9 sampling steps to 2.18 with vanilla AR, which requires more than 10× as many steps.The vanilla AR variant is evaluated as an upper bound on ResTok’s AR generation performance.Source: B. Additional Results · p. 14

C. Licenses for Released Assets

  • The released projects listed in Table 11 were used under their licenses, and the authors state that they adhered to the license requirements.The original projects’ copyright notices and license texts are available in their official repositories.Source: C. Licenses for Released Assets · p. 14

Related papers

Loading 2601.03955v1…