PaperlayerView exact revision on arXiv

Source-linked AI summary

Patch-Derived or Register Tokens? Comparing Variable-Length Image Tokenizers

Philipp Schenkel, Tobias Schürmann, Omer Sahin Tas

arXiv:2610.33540v1cs.CVcs.LG

TL;DR

Grid-based image tokenizers tie sequence length to resolution, and prior comparisons make the effect of bottleneck design difficult to isolate. The paper compares patch-derived and register-token representations under a shared setup, finding generative trade-offs that vary with token budget and model capacity.

  • Problem

    Most image tokenizers place latents on a downsampled grid, tying token count to image resolution, while differing experimental setups confound comparisons of bottleneck design.

    Source: §1 · p. 2

  • Method

    The paper compares patch-derived and learned register tokens in variable-length representations, training them with nested dropout and optional alignment to pretrained visual features.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Generative gains largely saturate beyond approximately 128–192 tokens; REPA particularly benefits patch-derived tokens, while register-token performance with REPA is less robust for limited-capacity generators.

    Source: §5 · p. 12

  • Takeaways & Limitations

    Both structured and unconstrained tokenization can learn compact image representations, with different spatial-organization and downstream-generation trade-offs.

    Source: §5 · p. 12

    The comparison is not compute-matched: register tokens require more computation, leaving training-compute effects on the no-REPA performance gap unresolved.

    Source: §5 · p. 12

Abstract

from arXiv · show

Image tokenizers convert pixels into compact latent sequences for efficient generative modeling. Most approaches arrange latents on a downsampled two-dimensional grid, tying sequence length to image resolution. We compare two variable-length strategies: computationally efficient patch-derived tokens, which inherit explicit spatial organization, and more demanding register tokens, which have no predefined spatial layout. Our reconstruction results reveal that patch-derived tokens perform substantially better under extreme compression, whereas register tokens achieve slightly better reconstruction quality at intermediate budgets. At high token budgets, both approaches achieve similar reconstruction quality. For downstream generative tasks, register tokens outperform patch-derived tokens. Adding representation alignment with external pretrained visual encoders substantially improves the generative performance of patch-derived tokens, allowing them to match or outperform register tokens depending on the token budget. We further find that both approaches learn spatially localized representations. Patch-derived tokens largely retain their explicitly imposed spatial organization, while register tokens develop an implicit two-dimensional spatial structure despite having no predefined layout. Representation alignment reduces this spatial organizational constraint, enabling individual tokens to exert a stronger non-local influence.

1 Introduction

Grid-based tokenizers tie sequence length to image resolution, while flexible-length tokenizers support multiple token budgets. This work controls the comparison of patch-derived and register bottlenecks and examines their quality, cost, and spatial trade-offs.

1 Introduction

  • Grid-based tokenizers tie token count to image resolution, creating prohibitively long sequences for high-resolution image, video, and world generation.This motivates compact token representations for reducing the computational costs of subsequent generative models.Source: §1 · p. 2
  • Patch-derived tokens come from spatially arranged encoder features, while register tokens are learned embeddings; nested dropout supports variable-length representations.Nested dropout randomly truncates the latent sequence during training so the decoder learns to reconstruct from different lengths.Source: §1 · p. 2
  • The comparison holds the backbone, quantizer, objective, and training regime constant to isolate the bottleneck design.The authors describe it as the first controlled comparison of these flexible-length tokenization strategies.Source: §1 · p. 2
  • The study characterizes trade-offs in reconstruction quality, generation quality, and computational cost across token budgets.It also analyzes how individual latent tokens influence spatial regions.Source: §1 · p. 3
  • Patch-derived representations show substantially stronger spatial locality, while learned representations include some spatially diffuse tokens.The paper also analyzes the spatial influence of individual latent tokens.Source: §1 · p. 3

2 Related Work

Related work covers discrete vector-quantized image representations and two families of compact, variable-length tokenizers. Differences among existing methods make it difficult to isolate the effect of bottleneck construction.

2 Related Work

  • Vector-quantized autoencoders map continuous encoder outputs to learned codebook entries, producing discrete codes for decoder reconstruction.They are a common basis for generative image models.Source: §2 · p. 3
  • Register-token approaches include fixed-size TiTok, ordered variable-length One-D-Piece and FlexTok, and continuous-token SEMANTICIST.Their decoder types, training objectives, and token-ordering strategies differ.Source: §2 · p. 3
  • Patch-derived approaches include ElasticTok, which uses spatial encoder tokens, and ReTok, which initializes queries through pooled encoder features.ReTok further refines its queries with a Q-Former.Source: §2 · p. 3
  • Existing methods vary in token initialization, interaction with spatial features, quantization, decoders, and semantic alignment, confounding bottleneck comparisons.This work compares patch-derived and independently learned register tokens under shared architectures, quantization, training, and downstream modeling setups.Source: §2 · pp. 3–4

3 Method

The method trains variable-length, ordered latent sequences with a shared Vision Transformer encoder-decoder backbone. It contrasts patch-derived and register-token information flow, and uses nested dropout and optional representation alignment during training.

3 Method

  • The autoencoder encodes images into ordered latent sequences, discretizes them with vector quantization, and uses nested dropout to train variable-length representations.The architecture is based on a standard Vision Transformer encoder-decoder backbone.Source: §3 · p. 4

3.1 Information Flow

  • Patch-derived tokens pass the truncated encoder sequence to the decoder, whereas register tokens act as a learned memory bank attended to by decoder queries.Only the register tokens are passed to the decoder; patch-derived tokens retain an underlying two-dimensional grid structure disrupted by nested dropout.Source: §3.1 · p. 4

3.2 Training

  • Training minimizes a VQGAN loss plus weighted REPA alignment, with REPA aligning decoder-layer features to DINOv2-L features.Nested dropout uniformly samples a retained length and passes only the sequence prefix to the decoder; the authors use all possible lengths to assess each additional token independently.Source: §3.2 · pp. 4–5
  • Both architectures use nested dropout; patch-derived tokens transform image patches, while register tokens transform dedicated learnable embeddings.Figure 2 presents the two architectural designs.Source: Figure 2 · p. 5

4 Experiments

The experiments compare variable-length patch-derived and register-token bottlenecks across reconstruction, generation, inference speed, and latent-space organization. Under comparable training dynamics, both show similar overall performance, with outcomes varying by token budget and representation alignment.

4.1 Inference Speed

  • Patch-derived encoder and decoder throughput is constant across token budgets, while register-token encoding is slower and decoding slows as the budget grows.Patch-derived components each process 256 tokens; register-token decoding ranges from 257 to 512 tokens.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 Reconstruction Performance

  • At low reconstruction budgets, patch-derived tokens perform better; register tokens seem superior at medium budgets, and both are comparable at high budgets.At very low token counts, both approaches fail; REPA most notably deteriorates register-token rFID and LPIPS at high budgets.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Generative Performance

  • REPA-trained patch-derived tokens outperform register tokens at small generation budgets and achieve comparable performance at larger budgets.REPA can benefit generative performance when the generative model is sufficiently large; the only clear failure case is LlamaGen-B with register tokens and REPA, whose performance degrades as the budget increases.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 Latent Space Analysis

  • Register-token influence is generally more spatially diffuse, and REPA increases this trend; patch-derived tokens generally become more spatially concentrated at later token IDs.REPA partially mitigates the patch-derived trend. A maximum-entropy peak among register tokens is attributed to inactive tokens; many later register tokens also have low corruption sensitivity, consistent with diminishing rFID improvements at high budgets.Source: §4.4 · p. 9, Figure 5 · p. 10
  • Both token types often have localized influence; patch-derived tokens retain patch-based locality, while many register tokens become more diffuse with REPA.Register tokens can develop localized influence despite lacking a predefined token-to-patch correspondence, and qualitative examples show local effects remain visible after REPA.Source: §4.4 · p. 8, Figure 3 · p. 9, Figure 4 · p. 10

5 Conclusion

The comparison combines patch-derived and register-token bottlenecks with nested dropout for variable token budgets and downstream autoregressive generation. The approaches show different trade-offs in generative performance and spatial organization, while both learn effective compact representations.

5 Conclusion

  • Nested dropout enables patch-derived and register-token representations to support variable token budgets for reconstruction and downstream autoregressive generation.Source: §5 · p. 12
  • REPA particularly improves patch-derived-token generation; register tokens are comparable with sufficiently large generative models, but REPA is less robust at limited model capacity.Patch-derived tokens perform strongly even at small token budgets.Source: §5 · p. 12
  • Generative performance improves as token budgets rise to approximately 128–192 tokens, after which gains largely saturate.REPA improves generative quality when the downstream model has sufficient capacity.Source: §5 · p. 12
  • Patch-derived tokens retain localized influence, while register tokens develop spatial specialization without an imposed two-dimensional layout; REPA can broaden token influence, especially for register tokens.Both structured and unconstrained schemes learn effective compact representations, with different trade-offs in spatial organization and downstream generative performance.Source: §5 · p. 12
  • The tokenizer comparison is not compute-matched, and register tokens require more computation because more tokens are processed per image.A compute-matched comparison could clarify whether additional compute contributes to the performance gap observed without REPA.Source: §5 · p. 12
  • The downstream generative evaluation covers autoregressive models only; whether non-autoregressive models can exploit the learned ordering remains open.Nested dropout encourages an ordered representation.Source: §5 · p. 12

Related papers

Loading 2610.33540v1…