PaperlayerView exact revision on arXiv

Source-linked AI summary

QuadTok: Quadtree Visual Tokenizer for Autoregressive Image Generation

Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, Divyansh Srivastava, Bingnan Li, Zhuowen Tu

arXiv:2610.10497v1cs.CV

TL;DR

Fixed grids allocate tokens uniformly despite uneven image complexity, while variable-length 1D representations offer flexibility but weak spatial grounding. QuadTok uses a quadtree tokenizer and autoregressive generator; on ImageNet it reduces tokenizer length relative to a fixed grid and achieves strong generation quality.

  • Problem

    Fixed-grid tokenizers allocate tokens uniformly despite uneven image complexity, while variable-length 1D tokenizers offer flexibility but often lack explicit correspondence between tokens and image patches.

    Source: §1 · pp. 1–2

  • Method

    QuadTok uses a hierarchical quadtree to adapt token allocation to local complexity while preserving spatial correspondence, with refinements selected by reconstruction benefit.

    Source: §1 · p. 2, §3.1.1 · p. 3

  • Results

    On ImageNet, QuadTok averages 230 tokens (rFID 1.46; PSNR 20.37), saving approximately 10% versus a fixed 256-token grid; its 947M generator achieves gFID 2.08.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Prescribed quadtree layouts improve all three localization hit rates across four directions, supporting zero-shot spatial control under a fixed token budget.

    Source: §1 · p. 2, §4.3 · p. 8

    Evaluation is confined to class-conditional ImageNet, whose object-centric, single-subject bias restricts the upper bound of zero-shot spatial layout control.

    Source: H LIMITATIONS AND FUTURE WORK · pp. 18–19

Abstract

from arXiv · show

We introduce QuadTok, a novel framework for visual tokenization and autoregressive image generation. Compared to traditional approaches using 2D grids or 1D token sequences, we propose a hierarchical quadtree structure, bridging the gap between 2D spatial binding and 1D sequence-level flexibility. The QuadTok tokenizer dynamically allocates representational capacity to visually intricate areas while leaving homogeneous regions at a coarse resolution. Compared with a fixed 256-token grid, our ImageNet-trained tokenizer saves approximately 10% of tokens on ImageNet and 9% when transferred zero-shot to the COCO dataset, while maintaining comparable reconstruction fidelity. Furthermore, the natural causality introduced by the tree structure seamlessly enables autoregressive image generation. Conditioned on a quadtree topology supplied before generation, our 947M GPT-style generative model achieves a 2.08 gFID on the ImageNet $256 \times 256$ benchmark. Additionally, leveraging the strong spatial correlation preserved by the quadtree structure, the QuadTok generator enables zero-shot spatially controlled image generation capabilities. Code: https://github.com/myc634/QuadTok.

1 INTRODUCTION

QuadTok addresses fixed grids’ uniform allocation and 1D tokenizers’ weak spatial grounding with a quadtree that adapts resolution to local complexity. Region-Wise Complexity Guidance selects refinements by reconstruction benefit, while the Kinship Causal Mask enforces hierarchical dependencies.

1 INTRODUCTION

  • Fixed-grid tokenizers distribute tokens uniformly despite uneven image complexity, while variable-length 1D tokenizers offer flexibility but lack explicit spatial correspondence.This can under-represent complex regions, redundantly allocate tokens to simple regions, and impair pixel-level fidelity or precise localization.Source: §1 · p. 2
  • QuadTok represents images as hierarchical quadtree regions, adapting token allocation to local complexity while retaining explicit spatial correspondence.The design bridges the spatial grounding of 2D tokenizers and the sequence-level flexibility of 1D representations.Source: §1 · p. 2
  • Region-Wise Complexity Guidance selects regions to refine according to reconstruction benefit, and the Kinship Causal Mask enforces hierarchical dependencies among tree nodes.The mask supports coarse-to-fine information flow, establishing global structure before localized refinement in spatially bounded regions.Source: §1 · p. 2
  • On ImageNet, QuadTok averages 230 tokens (rFID 1.46; PSNR 20.37), saving approximately 10% versus a fixed 256-token grid; its 947M generator achieves gFID 2.08 versus 2.15 for the 1.4B RandAR-XXL model.Transferred zero-shot to COCO, the tokenizer averages 232 tokens, saving approximately 9%.Source: §1 · p. 2
  • QuadTok’s hierarchical causality supports competitive image generation and zero-shot spatially controllable synthesis conditioned on quadtree layouts supplied before generation.The tokenizer and generator use the same quadtree structure to preserve spatial relationships.Source: §1 · p. 2

2 RELATED WORK

Prior work explores fixed-grid and variable-length visual tokenization, multiple autoregressive prediction units, and quadtree structures for vision. QuadTok’s related-work discussion distinguishes its region-level discrete tokens and supplied spatial topology from these approaches.

2.1 VISUAL TOKENIZATION

  • Visual tokenizers progress from discrete codebooks and fixed grids to compact or variable-length 1D sequences, while QuadTok assigns discrete tokens to image regions with explicit spatial correspondence.TiTok represents images as compact 1D sequences; FlexTok and One-D-Piece support variable sequence lengths, while GigaTok uses semantic regularization for reconstruction and downstream generation.Source: §2.1 · pp. 2–3
  • CAT selects an image-wide compression ratio using caption-derived scores, whereas QuadTok allocates discrete tokens at different spatial resolutions within each image.QuadTok uses image content to select reconstruction refinements and an autoregressive model to predict visual codes under a topology supplied before sampling.Source: §2.1 · p. 3

2.2 AUTOREGRESSIVE IMAGE GENERATION

  • Autoregressive image models vary in representation and prediction unit, including sequential grid tokens, reordered tokens, resolution scales, variable-length patches, and continuous entities.QuadTok generates discrete regional codes in breadth-first order conditioned on a given tree.Source: §2.2 · p. 3

2.3 QUADTREE REPRESENTATIONS IN VISION

  • QuadTok incorporates recursive quadtree partitions into a generative visual tokenizer, using topology-aware attention and discrete representations for reconstruction and autoregressive synthesis under supplied spatial topology.Prior quadtree applications include image processing, compression, and adaptive visual computation (Tang et al., 2022; Ronen et al., 2023; Zhao et al., 2025).Source: §2.3 · p. 3

3 METHOD

QuadTok represents images as spatially adaptive quadtree tokens, preserving region correspondence while supporting coarse-to-fine token sequences. Its tokenizer, attention mask, and generator use the hierarchy to guide image encoding, reconstruction, and autoregressive synthesis.

3.1.1 QUADTREE REPRESENTATION

  • QuadTok assigns each quadtree node to a spatial region and serializes nodes breadth-first, placing coarse tokens before finer subdivisions.The default uses two levels; a three-level extension is also evaluated.Source: §3.1.1 · p. 3, §3.1 · p. 3

3.1.2 IMAGE ENCODING; 3.1.3 VECTOR QUANTIZATION

  • ViT features are aggregated into region-indexed node embeddings, then quantized through a learned codebook into discrete image tokens.Node instructions encode tree level and spatial position.Source: §3.1.2 · p. 4, Figure 2 · p. 4, §3.1.3 · p. 4

3.1.4 HIERARCHICAL DECODING

  • Hierarchical decoding places level-specific patches at their spatial locations and aggregates them coarse-to-fine to reconstruct the image.Coarse tokens recover global structure, while higher-resolution tokens refine local details.Source: §3.1.4 · pp. 4–5

3.1.5 TREE-STRUCTURE SAMPLING

  • Random subtree sampling retains all level-1 nodes and independently expands each parent with probability γ, exposing the tokenizer to varied tree layouts.This trains consistent representations across coarse and refined token layouts.Source: §3.1.5 · p. 5

3.2 KINSHIP CAUSAL ATTENTION

  • Kinship Causal Attention enforces coarse-to-fine dependencies: coarse nodes attend bidirectionally, while finer nodes attend to coarser nodes and same-parent siblings.The generator instead uses a standard causal mask over the breadth-first sequence.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3 REGION-WISE COMPLEXITY GUIDED TOKENIZATION

  • Complementary probing trees compare each region in expanded and collapsed states, estimating reconstruction benefit to select regions for refinement.Regions with benefit above threshold τ expand into four children; others remain coarse.Source: §3.3 · pp. 5–6

3.4 QUADTREE-BASED AUTOREGRESSIVE IMAGE GENERATION

  • Given a topology supplied before generation, the autoregressive model predicts visual codes in breadth-first order from prior codes, conditioning information, and available tree instructions.Coarse tokens establish global structure before finer tokens refine local details and textures.Source: §3.4 · p. 6

4 EXPERIMENTS

QuadTok’s experiments test adaptive reconstruction, autoregressive generation, spatial layout control, and the effects of its causal mask and complexity guidance. Results show competitive generation and reconstruction, with the mask and guidance shaping spatial fidelity and token allocation.

4.1 QUADTOK IMAGE RECONSTRUCTION

  • Compared with LlamaGen at downsampling factor 16, content-adaptive allocation achieves lower rFID with fewer tokens but lower PSNR, including under zero-shot COCO transfer.Table 1 reports ImageNet-1K and COCO reconstruction at 256 × 256; its token counts are dataset-specific means for QuadTok.Source: §4.1 · p. 7, Table 1 · p. 7
  • Adding a 32 × 32 grid to the default 8 × 8 → 16 × 16 hierarchy improves reconstruction on both datasets, while increasing depth and token budget together.This evaluates a three-level tokenizer with the shared codebook and Kinship Mask rules.Source: §4.1 · p. 7, Table 1 · p. 7

4.2.1 IMAGENET GENERATION BENCHMARK.

  • QuadTok improves gFID over native-256 LlamaGen at comparable scales, remains competitive with larger discrete-token generators, and has lower latency at reduced token budgets.These are cross-setting comparisons; high-resolution results are reported separately.Source: §4.2.1 · p. 7
  • Under supplied three-level topologies, increasing expansion initially improves gFID, but further expansion reverses the trend, so the best evaluated quality occurs before maximal expansion.The comparison holds the CFG schedule and other sampling settings fixed within each sweep.Source: §4.2.1 · p. 7

4.3 ZERO-SHOT SPATIAL LAYOUT CONTROL WITH QUADTOK

  • Prescribed layouts improve all three localization hit rates across four directions versus random topologies at the same token budget, while FID-50cls changes range from −0.15 to +0.12 points.The frozen ImageNet-trained model uses the input topology to refine selected regions.Source: §4.3 · p. 8, Figure 5 · p. 8, Table 3 · p. 9

4.4.1 EFFECTIVENESS OF THE KINSHIP CAUSAL MASK

  • With the Kinship Causal Mask, perturbing a fine-level token stays localized to its patch; without it, the change spreads across the image and removing the mask raises gFID from 2.23 to 3.17.Without the mask, rFID is slightly lower (1.46→1.41), while PSNR falls from 20.37 to 19.16; the generation comparison uses the same 697M generator and CFG.Source: §4.4.1 · p. 9

4.4.2 REGION-WISE COMPLEXITY GUIDANCE STRATEGY.

  • Guided refinement matches Full-tree reconstruction fidelity (rFID 1.46, PSNR 20.37) with 230 tokens, 28% fewer than Full’s 320 tokens; a similar-sized Random tree has rFID 1.60 at 253 tokens.The default uses LPIPS-based guidance with τ=0.05.Source: §4.4.2 · p. 9, Table 5 · p. 9

5 CONCLUSION

The conclusion presents QuadTok as a spatially adaptive, variable-length tokenizer that preserves token-to-region correspondence and supports strong reconstruction, generation, and zero-shot layout control. The paper focuses on class-conditional generation and leaves multimodal and text-to-image pipelines for future work.

5 CONCLUSION

  • QuadTok organizes variable-length visual representations in a quadtree, allocating tokens by local complexity while preserving spatial correspondence, and experiments demonstrate reconstruction, generation, and zero-shot layout control.The authors identify multimodal pipelines and text-to-image generation as future work beyond the paper’s class-conditional focus.Source: §5 · p. 9

A ADDITIONAL QUALITATIVE EXAMPLES

B QUADTOK TOKENIZER TRAINING DETAILS

The tokenizer is trained on 256 × 256 ImageNet-1K images using Transformer encoder, aggregator, and decoder modules with a shared codebook across quadtree levels. Training combines reconstruction, quantization, and adversarial objectives, with adversarial loss delayed for stability.

B QUADTOK TOKENIZER TRAINING DETAILS

  • The ImageNet-1K tokenizer uses a ViT encoder and Transformer aggregator and decoder, each with eight blocks, 512-dimensional hidden states, and structural instruction embeddings.The encoder uses 16 × 16 patches; tokenizer training uses 256 × 256 images.Source: B QUADTOK TOKENIZER TRAINING DETAILS · p. 14
  • A shared codebook has vocabulary size V = 16384 and latent dimension d = 8; the default two-level grids are 8 × 8 and 16 × 16, with child expansion probability γ = 0.5.The tokenizer has approximately 120M parameters and uses probabilistic tree sampling.Source: B QUADTOK TOKENIZER TRAINING DETAILS · pp. 14–15
  • The training objective combines MSE, LPIPS, vector-quantization commitment, and GAN losses, with the adversarial weight set to zero initially and activated only after 200,000 training steps.MSE targets pixel fidelity, LPIPS preserves high-frequency details and structure, vector quantization aligns representations with the codebook, and GAN loss enhances perceptual realism and sharpness.Source: B QUADTOK TOKENIZER TRAINING DETAILS · p. 15

C QUADTOK GENERATOR TRAINING DETAILS

The generator predicts visual codes autoregressively under a quadtree topology supplied before generation. High-resolution evaluations report reconstruction with fewer tokens than grid baselines and competitive class-conditional generation, while noting that the comparisons use differing systems and settings.

C QUADTOK GENERATOR TRAINING DETAILS

  • QuadTok uses a decoder-only autoregressive generator with two-level sizes from 111M to 947M parameters and a 344M three-level model; its topology is specified before sampling.For unconditional generation, random trees are sampled independently of image content; inference uses KV caching and a dynamic power-cosine CFG scheduler.Source: C QUADTOK GENERATOR TRAINING DETAILS · p. 15

D HIGH-RESOLUTION RECONSTRUCTION AND GENERATION

  • At 512 × 512, the evaluation retains two quadtree levels with 16 × 16 and 32 × 32 grids, separating higher resolution from the deeper-tree experiment.The high-resolution study evaluates ImageNet reconstruction and topology-conditioned generation.Source: D HIGH-RESOLUTION RECONSTRUCTION AND… · p. 15
  • At 512 × 512, Region-wise Complexity Guidance demonstrates reconstruction with fewer tokens than the 1,024-token grid baselines.Table 6 reports ImageNet-1K reconstruction; LlamaGen reconstructions are resized to 256 × 256 for evaluation.Source: D HIGH-RESOLUTION RECONSTRUCTION AND… · p. 15, Table 6 · p. 16
  • The 344M generator achieves lower reported gFID than DiT-XL/2 and CAT on 50,000 native-512 × 512 class-conditional samples, in system-level comparisons with different tokenizer, training, and sampling settings.The evaluation uses the ImageNet-512 ADM reference and random topologies supplied before generation.Source: D HIGH-RESOLUTION RECONSTRUCTION AND… · p. 15, Table 7 · p. 16

D.1 EXPERIMENTAL DETAILS

The appendix specifies tokenizer and generator configurations, evaluation protocols, generation-runtime comparisons, and pretokenization costs. It distinguishes runtime measurement from image-quality evaluation and notes that adaptive encoding adds preprocessing cost.

D.1 EXPERIMENTAL DETAILS

  • The tokenizer uses ViT-small and ViT-large encoder/decoder models with a shared 16,384-entry codebook; training uses 512 × 512 crops and a 350,000-step budget.The patch size is 16, codes are 8-dimensional, and the configured global batch size is 512.Source: §D.1 · p. 16
  • The 344M generator checkpoint is evaluated at 400,000 steps with a supplied 16×16 coarse topology and expansion probability 0.75.Sampling uses linear classifier-free guidance at scale 16; evaluation generates 50,000 native-512 × 512 images against the ImageNet-512 ADM reference.Source: §D.1 · p. 16
  • Runtime comparisons measure autoregressive sampling separately from decoding, using matched batch sizes and CFG modes on an A100-80GB GPU.The LlamaGen-XL training settings were unstable in this setup, so quality evaluation uses a 697M backbone while timing uses a 775M QuadTok model to match LlamaGen-XL; image quality is evaluated separately.Source: §D.1 · p. 17, Table 8 · p. 17
  • At batch size 128, QuadTok pretokenization processes 161.66 images/s versus 346.58 for LlamaGen VQ-16.Adaptive tree search and LPIPS-based selection add image-dependent preprocessing cost, which can be amortized by caching codes; generation with a supplied topology requires no such search.Source: §D.1 · pp. 17–18, Table 10 · p. 18

F SPATIAL LAYOUT EVALUATION DETAILS

The spatial-layout evaluation compares four prescribed half-plane topologies with a matched random-expansion baseline using localization measures and FID. Each topology condition is evaluated on 1,000 generated images.

F SPATIAL LAYOUT EVALUATION DETAILS

  • Each prescribed layout expands 32 cells in one half-plane, yielding 192 visual tokens; the random baseline expands 32 randomly selected coarse cells.The evaluation uses 20 images for each of 50 fixed ImageNet classes per condition.Source: F SPATIAL LAYOUT EVALUATION DETAILS · p. 18
  • Localization is evaluated by whether Grad-CAM centers, detected-box centers, or mask centroids fall in the requested half-plane.Box and mask scores count missed detections as failures and use all generated images as the denominator.Source: F SPATIAL LAYOUT EVALUATION DETAILS · p. 18
  • FID-50cls is 35.03 for the random-topology baseline and 35.15, 34.88, 34.97, and 35.05 for prescribed left, right, top, and bottom layouts, respectively.Each condition's 1,000 generated images are compared with the same 2,500 ImageNet validation images from the evaluated classes.Source: F SPATIAL LAYOUT EVALUATION DETAILS · p. 18

G COMPLEXITY GUIDANCE: METRIC AND THRESHOLD

The ablation compares complexity-probing signals and expansion thresholds at approximately matched token budgets. LPIPS at τ=0.05 is selected as the default guidance setting.

G COMPLEXITY GUIDANCE: METRIC AND THRESHOLD

H LIMITATIONS AND FUTURE WORK

The study's empirical evaluation is limited to class-conditional ImageNet generation. The authors identify open-domain and complex text-to-image generation, as well as complex multi-object layout control, as future extensions.

H LIMITATIONS AND FUTURE WORK

  • Empirical evaluations are currently confined to class-conditional image generation on ImageNet because of computational resource limitations.Source: H LIMITATIONS AND FUTURE WORK · p. 18
  • QuadTok has not been scaled to massive open-domain datasets or extended to complex text-to-image synthesis.The authors state that scaling model capacity and extending cross-attention for open-vocabulary conditioning require substantially more training compute.Source: H LIMITATIONS AND FUTURE WORK · p. 19
  • ImageNet's object-centric, single-subject bias restricts the upper bound of zero-shot layout control, leaving complex multi-object relational scenes challenging.The authors attribute this boundary to the compositional diversity of the training data.Source: H LIMITATIONS AND FUTURE WORK · p. 19
Loading 2610.10497v1…