PaperlayerView exact revision on arXiv

Source-linked AI summary

NativeTok: Native Visual Tokenization for Improved Image Generation

Bin Wu, Mengqi Huang, Weinan Jia, Zhendong Mao

arXiv:2601.22837v1cs.CV

TL;DR

VQ-based image generation separates reconstruction-oriented tokenization from second-stage dependency modeling, leaving token dependencies insufficiently constrained. NativeTok introduces causal visual tokenization through latent context modeling and position-specific experts, and reports improved generation despite slightly worse reconstruction in one comparison. Its main scope includes the assumption that each token depends only on the image and preceding tokens, while naive causal masking is ineffective in practice.

  • Problem

    Existing tokenization methods do not constrain intrinsic token dependencies during tokenization, creating a mismatch with the structured dependency modeling required during generation.

  • Method

    NativeTok combines MIT latent image modeling with MoCET, whose position-specific expert blocks generate ordered tokens conditioned on image features and previously generated tokens.

  • Results

    NativeTok bridges tokenization and generation with ordered token sequences and improves generation quality; with the same AR generator and 32 tokens, it reduces gFID from 7.45 to 5.23 versus TiTok-L-32.

  • Takeaways & Limitations

    Embedding relational constraints in tokenization can align the first and second stages and make visual dependencies easier for generation models to capture.

  • Takeaways & Limitations

    The tokenization assumes each token depends only on the image and preceding tokens, while naive causal masking is ineffective and capacity-dependent in practice.

Abstract

from arXiv · show

VQ-based image generation typically follows a two-stage pipeline: a tokenizer encodes images into discrete tokens, and a generative model learns their dependencies for reconstruction. However, improved tokenization in the first stage does not necessarily enhance the second-stage generation, as existing methods fail to constrain token dependencies. This mismatch forces the generative model to learn from unordered distributions, leading to bias and weak coherence. To address this, we propose native visual tokenization, which enforces causal dependencies during tokenization. Building on this idea, we introduce NativeTok, a framework that achieves efficient reconstruction while embedding relational constraints within token sequences. NativeTok consists of: (1) a Meta Image Transformer (MIT) for latent image modeling, and (2) a Mixture of Causal Expert Transformer (MoCET), where each lightweight expert block generates a single token conditioned on prior tokens and latent features. We further design a Hierarchical Native Training strategy that updates only new expert blocks, ensuring training efficiency. Extensive experiments demonstrate the effectiveness of NativeTok.

1 Introduction

Existing VQ-based visual generation separates image tokenization from dependency modeling, creating a mismatch between disordered token outputs and structured generation. NativeTok addresses this by imposing causal, visually ordered dependencies during tokenization.

  • Existing Paradigm: Most large visual models tokenize images by reconstruction, then train a second-stage generator with next-token prediction over the resulting discrete tokens.This two-stage design compresses pixels into tokens before modeling their distribution.
  • Existing Paradigm: Existing tokenizers primarily optimize compression or reconstruction independently from the dependency structure required by generation.Examples include variable-length, multi-scale, and one-dimensional tokenization strategies.
  • Motivation: This misalignment can produce biased, incomplete token distributions and means better first-stage reconstruction does not necessarily improve second-stage generation.More complex token interrelationships may further violate the autoregressive principle.
  • Native Visual Tokenization: Native visual tokenization aligns tokenization with generation by imposing relational constraints and modeling visual information in a native causal order.The approach jointly considers reconstruction quality and token ordering rather than treating the stages as independent.
  • NativeTok: NativeTok separates visual context modeling from dependency modeling to produce ordered token sequences that make subsequent dependency learning easier.Its framework uses MIT for latent image modeling and MoCET for position-specific causal token generation.

2 Related Work

Image tokenization compresses images into discrete or low-dimensional representations for reconstruction and efficient generation. Related work improves compression, reconstruction, quantization, and serialization, while generation methods model tokens from prior context.

  • Image Tokenization: Image tokenization uses autoencoders to compress high-dimensional images into low-dimensional latent representations and decode them for reconstruction.CNN- and transformer-based tokenizers produce discrete or serialized image representations.
  • Image Tokenization: Prior tokenization methods improve representation efficiency through information-density coding, high compression ratios, flexible lengths, and vector-quantization variants.Examples include DQ-VAE, TiTok, MAETok, FlexTok, SoftVQ, MoVQ, RQ-VAE, and FSQ/LFQ methods.
  • Image Generation: Image generation methods include GAN, diffusion, and autoregressive paradigms that predict tokens or token groups from conditions and previously generated tokens.These methods operate on the image latent space produced during tokenization to improve generation efficiency.

3 Methodology

NativeTok aligns tokenization with generation by producing an ordered, causally dependent token sequence while separating global image-context modeling from token-dependency modeling. Its MIT, MoCET, and hierarchical training strategy support efficient reconstruction and reduced training cost.

  • NativeTok framework: NativeTok produces an ordered, causally dependent token sequence so first-stage tokenization aligns with subsequent generation.The framework explicitly models ordered relationships among tokens during tokenization.
  • NativeTok framework: A naive causal mask is ineffective because one transformer must perform image self-modeling and sequential token generation simultaneously.Its performance also depends heavily on the original model’s capacity.
  • Meta Image Transformer (MIT): MIT applies transformer-based image modeling and an FNN dimension switcher to obtain latent image context for efficient visual dependency modeling.The resulting Xlatent represents rich contextual image information in latent space.
  • Mixture of Causal Expert Transformer (MoCET): MoCET assigns each token position a lightweight expert that receives locked latent image information, prior tokens, and a padding token, retaining one generated vector.Each expert is solely responsible for generating its corresponding token, and generated tokens are incorporated into later steps.
  • Hierarchical Native Training strategy: Hierarchical Native Training reuses earlier expert and MIT weights, freezes reused modules, and trains only newly added experts and a decoder for longer token sequences.For the 64-token model, trainable parameters are reduced to 56%; full fine-tuning is applied during the final 10% of steps.

4 Experiments

Experiments evaluate NativeTok across autoregressive and MaskGIT-style generation, ablations, encoding efficiency, and token-dependency visualizations. Results show improved generation quality and structured token dependencies, with only a moderate encoding-speed reduction.

  • Main Results: NativeTok reduces AR gFID from 7.45 to 5.23 versus TiTok-L-32 at the same 32-token sequence length.This occurs despite NativeTok’s slightly worse reconstruction metric, rFID = 2.57.
  • Main Results: NativeTok achieves a gFID of 2.16 with 287M parameters in the MaskGIT framework.The result uses NativeTok128 with MaskGIT-UVit-L as the generator.
  • Main Results: NativeTok’s ordered token sequences improve generation quality consistently in both AR and MaskGIT-style settings.Token-level constraints produce more structured and learnable dependencies for the second-stage generator.
  • Ablations: Under equal training steps, NativeTok32 lowers rFID to 11.19 from 12.99 for TiTok-L-32 and 12.95 for TiTok-L(mask)-32.The comparison evaluates causal masking and NativeTok attention mechanisms.
  • Ablations: NativeTok’s longer MoCET sequence causes a moderate encoding-speed drop because transformer attention has O(n^2) complexity.The reported impact does not significantly affect overall efficiency.
  • Visualization: Perturbing preceding tokens yields a smaller overlap among high-probability positions in NativeTok’s next-token distributions.The visualization and quantitative analysis indicate greater sensitivity to token-level variations.

5 Conclusion

The paper proposes NativeTok as native visual tokenization with ordered token sequences aligned to image generation. Its hierarchical training strategy is presented as bridging tokenization and generation to improve image generation.

  • Conclusion: NativeTok encodes images into ordered token sequences aligned with the generation stage.The framework incorporates native visual order and a hierarchical training strategy.
  • Conclusion: NativeTok bridges the gap between tokenization and generation, enabling improved image generation.This conclusion follows the proposed native visual tokenization framework.

F.1 Training and Testing Protocols

NativeTok variants are trained on ImageNet-1K at 256 × 256 resolution with specified model configurations and optimization settings. Larger variants use Hierarchical Native Training to expand trainable parameter subsets gradually.

  • Training and Testing Protocols: All NativeTok models use ImageNet-1K images at 256 × 256 resolution with center cropping and horizontal flipping.These are the only reported data augmentations.
  • Training and Testing Protocols: NativeTok32 combines an 18-layer MIT, a 32-layer MoCET, and a 24-layer decoder in a 616M-parameter model.Its reported training throughput is 63.44 samples per second per GPU.
  • Training and Testing Protocols: NativeTok64 uses 666M parameters and HNT updates 56% of parameters during initial training.The model uses 64 MoCET layers and trains for 450K HNT steps.
  • Training and Testing Protocols: NativeTok128 uses 766M parameters and HNT with 55% of parameters trainable.Its training throughput is 39.44 samples/s/GPU before full fine-tuning.
  • Training and Testing Protocols: Training uses AdamW with β1 = 0.9, β2 = 0.99, learning-rate decay from 1 × 10^-4 to 1 × 10^-5, and 500K total steps.The setup uses four NVIDIA A800 GPUs with global batch size 256.

F.2 Detailed Results of Preliminary Experiments

NativeTok’s reconstruction quality improves with model capacity, while adding decoder fine-tuning yields further gains. Increasing token counts expands MoCET parameters sublinearly.

  • Reconstruction Quality: rFID decreases from 5.10 to 2.86 under Stage 1 alone as model scale increases from NativeTok32 to NativeTok128.Lower rFID indicates better reconstruction quality.
  • Reconstruction Quality: With decoder fine-tuning, rFID reaches 2.57, 1.89, and 1.19 for NativeTok32, NativeTok64, and NativeTok128, respectively.The reported values correspond to increasing model capacity.
  • Parameter Scaling: Every additional 32 tokens increase parameters by only about 8% relative to the base model.The sublinear growth reflects adding more experts in MoCET.

F.3 Qualitative Visualization

NativeTok’s token-level behavior differs qualitatively from TiTok’s during reconstruction, and token perturbation tests show distinct generator sensitivity. Visual examples compare reconstructions across NativeTok variants and generated images.

  • Token Sensitivity: NativeTok consistently exhibits a lower top-100 overlap rate than TiTok when short token subsequences are modified.The comparison averages overlap rates across 1,000 ImageNet classes for subsequence lengths from 1 to 4.
  • Reconstruction Examples: Figure 7 compares the original image with NativeTok128, NativeTok64, and NativeTok32 reconstructions.The variants are shown from left to right after the original image.
  • Reconstruction Dynamics: NativeTok32 progressively incorporates novel visual information as token count increases, unlike TiTok32’s bidirectional reconstructions.The visualization covers token counts from 0 to 32.
  • Generation Examples: Figure 8 presents qualitative examples of generated images.The passage identifies the figure as a generation-results visualization without reporting a quantitative comparison.

F.4 Limitations

The authors identify two unresolved limitations: NativeTok remains a two-stage system rather than a unified end-to-end pipeline, and autoregressive training was limited to NativeTok32.

  • Two-stage Training Pipeline: A fully unified end-to-end training pipeline remains unexplored despite hierarchical causal training and decoder fine-tuning.The authors suggest unifying the stages could further improve token quality and model coherence in downstream autoregressive tasks.
  • Limited Training Scale: Autoregressive generation was trained only with NativeTok32 because of computational constraints and limited resources.Consequently, reported generation results may not reflect the potential of higher-capacity tokenizers such as NativeTok128.
Loading 2601.22837v1…