PaperlayerView exact revision on arXiv

Source-linked AI summary

MixTok: Adaptive Visual Tokenization with Global 1D Tokens and Sparse 2D Refinement

Chenge Li, Yefei He, Xinyu Liu, Feng Chen, Hong Zhou, Bohan Zhuang

arXiv:2610.13181v1cs.CV

TL;DR

Visual tokenizers must balance compact representations with preservation of fine image detail, while existing grid and 1D approaches have complementary limitations. MixTok combines global 1D tokens with sparse 2D refinement, selecting locations and per-image token counts adaptively. On ImageNet at 256 × 256, it reports rFID 1.195 and gFID 1.95 at average budgets of about 128 and 131 tokens per image.

  • Problem

    Uniform 2D token allocation and compact 1D sequences have complementary limitations in allocating capacity and preserving fine-grained local detail across images of varying complexity.

    Source: §1 · p. 2

  • Method

    MixTok combines a compact 1D global representation with selected 2D local tokens, using an adaptive router to choose refinement locations and per-image token counts.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    On ImageNet at 256 × 256, MixTok reports an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 tokens per image.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The results demonstrate the value of combining global compression with content-adaptive local refinement for compact, high-quality visual representations.

    Source: §6 · p. 10

    Evaluation is limited to ImageNet at 256 × 256 and a narrow range of average token budgets; reconstruction still requires dense 2D encoding, and the second-stage generator uses the full spatial grid.

    Source: §6 · p. 11

Abstract

from arXiv · show

Efficient visual tokenization requires balancing representation compactness with the preservation of fine-grained image content. Conventional grid-based tokenizers allocate tokens uniformly across spatial locations despite non-uniform visual complexity, whereas compact 1D tokenizers aggregate global information but can struggle to preserve local details. We introduce MixTok, a hybrid visual tokenizer that combines a compact 1D global representation with sparse 2D tokens for local refinement. A content-adaptive router ranks candidate locations by refinement priority and determines the number of 2D tokens for each image under a target average token budget. At the highest-scoring locations, 2D token features replace the spatial features recovered from the 1D representation, and a shared decoder reconstructs the image from the resulting mixed features. On ImageNet 256 x 256, MixTok achieves highly competitive reconstruction and generation performance, with an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 content tokens per image.

1 Introduction

MixTok addresses the tension between compact global representations and detailed local reconstruction by combining 1D tokens with selectively allocated 2D refinement. Its router chooses both refinement locations and per-image token counts, and the approach supports reconstruction and two-stage image generation.

1 Introduction

  • Visual tokenizers must preserve reconstruction and generation information while allocating capacity across images whose visual complexity varies within and between them.The introduction frames content-adaptive token allocation as a way to meet this challenge.Source: §1 · p. 2
  • Uniform 2D grids can spend tokens on simple regions, while compact 1D sequences can struggle to retain fine local detail.A fixed per-image budget also fails to adapt token counts to image complexity.Source: §1 · p. 2
  • On ImageNet at 256 × 256, MixTok reports an rFID of 1.195 and a gFID of 1.95 at respective average budgets of approximately 128 and 131 tokens per image.The introduction presents these as reconstruction and generation results.Source: §1 · p. 2
  • For image generation, a two-stage MaskGIT pipeline generates 1D base tokens first, then selected 2D tokens, before shared-decoder reconstruction.The base tokens guide the refinement locations and number of 2D tokens.Source: §1 · p. 2
  • MixTok combines compact 1D global representations with sparse 2D local refinement through spatial feature replacement and a shared decoder.The shared decoder reconstructs from the combined representation.Source: §1 · p. 2
  • A content-adaptive router learns spatial refinement priorities and selects per-image token counts under a target average token budget.It uses grid scoring and top-p selection.Source: §1 · p. 2

2 Related Work

Prior visual tokenizers use spatial grids or compact sequences, while adaptive methods vary token counts or spatial allocation. MixTok combines both adaptations by selecting where to refine and how many tokens to allocate per image.

2 Related Work

  • Spatial-grid tokenizers preserve location correspondence but impose uniform token density, even though image regions differ in detail.VQGAN [1] and related grid methods improve spatial representations, but retain fixed latent grids.Source: §2 · p. 3
  • Compact 1D methods aggregate image information into short sequences, while SelfBootTok separates global and local groups but predicts local detail from global tokens.MixTok instead retains selected 2D tokens to complement its compact 1D representation.Source: §2 · p. 3
  • Adaptive methods vary sequence length or spatial support, but MixTok jointly selects refinement locations and per-image token counts under a target average budget.FlexTok [26] supports chosen sequence lengths, while DQ-VAE [28] adapts regional coding granularity.Source: §2 · p. 3

3 Preliminary

The preliminary section establishes notation for visual tokenizers and distinguishes spatial 2D grids from compact 1D sequences. Both are quantized into discrete tokens and corresponding features used in reconstruction.

3 Preliminary

  • A visual tokenizer encodes an image into continuous latent features, then quantizes them into discrete tokens for downstream modeling and features for reconstruction.The image is denoted x, its encoder E, and the continuous latent features Z.Source: §3 · p. 3
  • A 2D tokenizer quantizes an encoder-produced latent grid into spatial tokens and corresponding features, then projects the features into decoder input space.The resulting feature grid retains spatial organization for local refinement and decoding.Source: §3 · p. 4
  • A 1D tokenizer compresses an image into a compact latent sequence without explicit token-to-location correspondence, then quantizes it into tokens and corresponding features.T1D denotes the token sequence and F1D its quantized features.Source: §3 · p. 4

4 Method

MixTok combines a compact 1D base representation with selected 2D refinements, replacing base features at chosen locations before shared decoding. A content-adaptive router scores refinement priorities and uses top-p selection to vary per-image token counts under a target average budget; the method extends this representation to two-stage image generation.

4.1 Hybrid Visual Representation

  • MixTok retains a compact 1D sequence and selected 2D tokens, replacing corresponding base-grid features before a shared decoder reconstructs the image.Source: §4.1 · pp. 4–5, Figure 2 · p. 5
  • Because 1D tokens lack explicit spatial correspondence, MixTok expands their quantized features into a spatial grid and projects them into the decoder feature space.Source: §4.1 · pp. 4–5

4.2 Content-Adaptive Token Allocation

  • The Reconstruction Grid Scorer ranks candidate locations by refinement priority, using local reconstruction-error reductions as auxiliary supervision and a differentiable selection surrogate in mixed-loss training.Source: §4.2 · p. 5
  • Top-p selection retains the shortest prefix of ranked locations reaching threshold p, so token counts vary with score concentration while p is calibrated to a target average budget.Source: §4.2 · pp. 5–6
  • MixTok trains with a weighted combination of reconstruction, perceptual, and adversarial losses, with separate weights controlling the loss terms.Source: §4.2 · p. 6

4.3 Image Generation with MixTok

  • For generation, a pretrained 1D MaskGIT supplies global tokens and a base grid; a Generation Grid Scorer distilled from the reconstruction scorer selects refinement locations using only that grid.Source: §4.3 · pp. 6–7
  • An Embedding Adapter maps base features to spatial context embeddings, which condition the 2D generator at locations not selected for token generation.Source: §4.3 · p. 7
  • A fine-tuned Halton-MaskGIT generates 2D tokens only at selected locations, then their features combine with the base grid for decoding; training applies cross-entropy at masked selected positions.Source: §4.3 · p. 7

5 Experiment

MixTok is evaluated for reconstruction, generation, transfer across tokenizer backbones, and adaptive-allocation ablations on ImageNet 256 × 256. Results show strong quality at compact token budgets, while ablations support learned spatial selection and content-matched budgets.

5.1 Experimental Setup

  • Evaluation uses ImageNet 256 × 256, reconstruction metrics on all 50,000 validation images, and gFID on 50,000 class-balanced samples per seed.The main configuration combines TiTok-L-32 with LlamaGen VQ-16; the top-p variant averages approximately 96.1 2D tokens, and the 1D encoder and quantizer stay frozen during training.Source: §5.1 · pp. 7–8

5.2 Image Reconstruction / 5.3 Image Generation

  • At approximately 128 tokens, MixTok achieves an rFID of 1.1950 and PSNR of 19.51 dB; at approximately 131 tokens, generation achieves a gFID of 1.95.The reconstruction comparison reports competitive performance against other tokenizers, while generation improves gFID over TiTok-S-128 at a similar budget.Source: Table 1 · p. 8, §5.2 · p. 8, Table 3 · p. 9

5.2 Image Reconstruction

  • Selective 2D refinement improves stripe patterns and curved boundaries over 1D base reconstructions.The qualitative results describe recovery of fine details and sharpening of structures blurred or missing in the base reconstruction.Source: Figure 4 · p. 8, §5.2 · p. 8
  • At the matched 128-token ALIT budget, MixTok-ALIT improves rFID from 2.9453 to 2.4720 and PSNR from 18.71 to 18.76 dB over ALIT-128.The configuration uses 64 1D and 64 2D tokens, supporting transfer beyond TiTok–LlamaGen.Source: §5.2 · pp. 8–9, Table 2 · p. 9

5.3 Image Generation

  • With similar token budgets, MixTok improves gFID over TiTok-S-128 from 2.00 to 1.95 and reduces sampling rounds from 64 to 16.Against DetailFlow-32, it reports gFID of 1.95 versus 2.63 with approximately half as many tokens and 16 versus 39 rounds.Source: Table 3 · p. 9, §5.3 · p. 9

5.4 Ablation Studies

  • The Reconstruction Grid Scorer achieves the best rFID of 1.1950 under identical per-image budgets; the Generation Grid Scorer using only 1D features reaches rFID 1.3479 and PSNR 18.4776.Both scorers outperform the tested random, uniform, and gradient-entropy rankings.Source: Table 4 · p. 10, §5.4 · p. 10
  • Top-p allocation achieves the best rFID and PSNR at a comparable average budget, while shuffling budgets across images degrades both metrics.The shuffled control preserves the budget distribution and mean, isolating correspondence between each image and its budget.Source: Table 5 · p. 10, §5.4 · p. 10

6 Conclusion and Future Work

MixTok combines compact global 1D tokens with sparse 2D refinement and an adaptive router to support reconstruction and two-stage masked generation. The paper reports ImageNet results at compact average token budgets, while noting computational and evaluation boundaries.

6 Conclusion and Future Work

  • MixTok combines compact 1D global tokens with sparse 2D tokens for local detail, with a content-adaptive router choosing locations and per-image token counts under a target average budget.On ImageNet, it reports rFID 1.195 and gFID 1.95 at approximately 128 and 131 content tokens, respectively.Source: §6 · p. 10
  • Reconstruction still requires dense 2D encoding, and the second-stage generator operates on the full spatial grid; evaluation is limited to ImageNet 256 × 256 and a narrow budget range.Future work targets sparse computation, broader budgets, higher resolutions, and more diverse visual data.Source: §6 · p. 11

AI use statement

  • Generative AI tools assisted implementation, experiments, data processing, manuscript work, and reference checking, while the authors state that the main methodological advances were theirs.The authors report reviewing and revising AI-assisted code and checking experimental logs and metrics.Source: AI use statement · p. 11

Ethics statement

  • The study uses ImageNet and publicly available pretrained models, with no new data collection or human-participant studies.The authors note potential inherited training-data biases and misuse to produce misleading imagery.Source: Ethics statement · p. 11

Reproducibility statement

  • Source code and configuration files for reconstruction and generation are publicly available, with the repository documenting required datasets, pretrained checkpoints, and external assets.The project page is also provided.Source: Reproducibility statement · p. 11

A.1 Implementation Details

The implementation uses separate modules to expand 1D features spatially, align feature spaces, score refinement locations, and condition the generation model.

A.2.1 Implementation Details

  • Spatial expansion G1D, feature alignment P1D, and generation Embedding Adapter A are distinct modules with different output spaces.G1D produces a 16 × 16 grid; P1D projects features into the shared 256-channel decoder space.Source: §A.1 · p. 15
  • The tokenizer combines TiTok-L-32 and LlamaGen VQ-16 codebooks, with the 2D post-quantization projection mapping codebook vectors into the shared 256-channel decoder space.The reported codebooks contain 4,096 12-dimensional vectors and 16,384 8-dimensional vectors, respectively.Source: §A.1 · p. 15
  • The Reconstruction Grid Scorer combines projected 1D and 2D features, their difference, pooled base reconstruction, and learned spatial embeddings before predicting scores.Its three-block convolutional trunk has 128 hidden channels; the Generation Grid Scorer instead uses projected 1D features and a learned spatial embedding.Source: §A.1 · p. 15
  • The Embedding Adapter adds a convolutional residual to the base grid and projects it into spatial context embeddings for the generator.The 2D generator is initialized from pretrained Halton-MaskGIT; it retains the full spatial sequence while restricting prediction targets and token updates to R.Source: §A.1 · p. 15

A.2.1 Training Schedule

Training proceeds from feature alignment to fixed-budget refinement and then top-p fine-tuning, with the 1D encoder and quantizer frozen throughout.

A.2.1 Training Schedule

  • The 1D encoder and quantizer remain frozen; alignment pretraining optimizes G1D and P1D while the 2D tokenizer is frozen.Fixed-budget training then jointly optimizes the spatial expansion, feature alignment, 2D encoding path, post-quantization projection, decoder, and scorer.Source: §A.2.1 · p. 15, Table 6 · p. 16
  • Top-p fine-tuning starts from the fixed-budget EMA model, retains the discriminator, resets the optimizer, and enables joint and adversarial training from the start.The schedule begins with scorer optimization and starts adversarial training at epoch 1.5 after a 0.01-epoch discriminator warm-up.Source: §A.2.1 · p. 15, Table 6 · p. 16
  • Training uses AdamW with cosine learning-rate decay, BF16 mixed precision, and gradient clipping at norm 1; preprocessing resizes the shorter image side to 256 pixels and center-crops.The stated setup uses no random cropping or flipping.Source: §A.2.1 · p. 16

A.2.2 Scorer Supervision and Differentiable Selection

The appendix details scorer supervision, generation training, and evaluation protocols, then reports adaptive token allocation, bitrate, sampling-round, and efficiency analyses.

A.2.2 Scorer Supervision and Differentiable Selection

  • The scorer targets the 37.5% of cells with largest local error-reduction values; top-p selection uses a hard prefix mask with a differentiable training surrogate.The error map proxies the benefit of replacing an individual grid feature; fixed-budget training instead uses a hard top-96 mask and sigmoid surrogate.Source: §A.2.2 · pp. 16–17

A.2.3 Loss

  • Alignment pretraining combines image and feature losses, while joint reconstruction training adds feature and routing losses alongside the reconstruction objective.Alignment weights are 1, 0.5, 0.05, and 0.05 for image L1, LPIPS, feature L1, and feature-moment matching, respectively.Source: §A.2.3 · p. 17, Table 7 · p. 17

A.3.1 Generation Grid Scorer; A.3.2 Embedding Adapter and 2D Generator

  • The generation scorer learns location priorities by distilling reconstruction-model teacher logits, and the adapter and masked generator are trained on paired 1D and original-image 2D targets.Generator training masks selected positions and computes cross-entropy only on masked selected positions.Source: §A.3.1 · p. 17, §A.3.2 · pp. 17–18, Table 8 · p. 18

A.3.3 Sampling Configuration; A.4 Evaluation Details

  • Main generation uses eight rounds for each stage, while reconstruction calibrates its top-p threshold to target approximately 96 2D tokens.Generation scorer settings are T = 2 and p = 0.75; reconstruction uses T = 1 and p = 0.994897.Source: §A.3.3 · p. 18, §A.4 · p. 19

A.4 Evaluation Details

  • Mean gFID falls from 4.3346 to 1.9511 with sparse 2D refinement, using eight additional sampling rounds and approximately 131 total content tokens.The comparison starts from the same generated 1D tokens.Source: Table 9 · p. 19, §A.4 · p. 19

A.5 Distribution of 2D Token Counts

  • Generation refinement budgets vary by image; across seeds 2–4, mean K ranges from 98.53 to 98.75 with similar distributions.Figure 6 plots the percentage of images assigned each integer K and excludes the fixed 32 1D tokens.Source: §A.5 · p. 19, Figure 6 · p. 20

A.6 Bitrate Accounting

  • 0.03030 bpp is the measured mean bitrate for MixTok at E[K(x)] = 96.1093, including 0.00390625 bpp for spatial side information.The payload estimate assumes no entropy coding and excludes headers, byte padding, and shared model weights.Source: §A.6 · p. 20, Table 10 · p. 21

A.7 Generation Sampling-Round Ablation; A.8 Generation Efficiency

  • Increasing the 2D stage from 4 to 8 rounds substantially improves gFID, with diminishing returns beyond that and saturation around 24–40 total rounds.Efficiency measurements time end-to-end generation on one RTX 5090 after warm-up; FLOPs include sampling rounds and final decoding but exclude elementwise operations and data movement.Source: §A.7 · p. 21, Table 11 · p. 21, §A.8 · pp. 21–22

Related papers

Loading 2610.13181v1…