PaperlayerView exact revision on arXiv

Source-linked AI summary

Studying Image Tokenizers as Visual Languages in Unified Multimodal Models

Siting Li, Zhengyang Wang, Simon Shaolei Du, Xi Chen, Yang Liu

arXiv:2609.09143v1cs.CVcs.CL

TL;DR

Image-tokenizer evaluations often miss how visual tokens behave when jointly modeled with text. The paper builds a controlled pure-autoregressive testbed using task-specific losses, finding that losses must be interpreted by task and that I2T loss is a more consistent cross-tokenizer signal than T2I loss.

  • Problem

    Existing reconstruction metrics and generation- or understanding-only evaluations do not fully capture image-tokenizer behavior in unified joint text–image modeling.

  • Method

    The paper uses a controlled pure-autoregressive multimodal continual-pretraining testbed and tracks task-specific validation losses for text, image, T2I, and I2T prediction.

  • Results

    Task-specific losses scale differently and rank tokenizers differently; across tokenizers, I2T loss is more consistent than T2I loss, and it correlates with post-finetuning generation and general VQA performance.

  • Takeaways & Limitations

    Tokenizer evaluation should consider reconstruction fidelity together with joint modeling difficulty across tasks, because reconstruction quality can diverge from multimodal learnability and tokenizer choice can affect text modeling.

  • Takeaways & Limitations

    Image-token validation loss measures modeling quality over a tokenizer’s discrete latent codes and cannot be converted into a description length for the original pixels.

Abstract

from arXiv · show

Image tokenizers define the ``visual language'' of unified multimodal models, yet are commonly studied through isolated metrics or generation-/understanding-only evaluations. These evaluations do not fully capture how visual tokens behave when modeled jointly with text. We build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across text, image, text-to-image (T2I), and image-to-text (I2T) prediction. We examine how these losses scale and relate to downstream performance, then use them to study multimodal learnability---how well image and text tokens are jointly modeled---and tokenizer design. We find that (1) losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. (2) The loss--performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers, the T2I loss--performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal. I2T loss also correlates with both generation and visual understanding performance after supervised finetuning. Using losses as a lens, we show that (3) better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance, and that (4) image tokenizer choice can affect text modeling under joint optimization. As case studies, we revisit three tokenizer design axes---the discriminator, semantic supervision, and vocabulary size---to examine their effects on joint modeling and downstream performance. Together, our testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training.

1 Introduction

The paper studies image tokenizers as visual languages in unified autoregressive multimodal models, where image and text tokens are jointly modeled. It introduces task-specific losses to analyze tokenizer effects on joint modeling and downstream performance.

  • Image tokenizers define the discrete visual language that unified autoregressive models must learn, align with text, and use for generation and understanding.
  • Existing reconstruction metrics and single-axis generation or understanding evaluations do not fully reveal tokenizer effects on joint text–image modeling.
  • The study builds a controlled pure-autoregressive continual-pretraining testbed and tracks validation losses for text, image, T2I, and I2T prediction.
  • Task-specific losses scale differently and rank tokenizers differently, so averaged loss is too coarse for unified multimodal analysis.
  • Across tokenizers, I2T loss provides a more consistent signal than T2I loss because T2I loss shifts with the visual token space.
  • Reconstruction fidelity can diverge from multimodal learnability, while tokenizer choice can affect text modeling through joint image-token prediction.

2 Related Work

Related work studies unified multimodal architectures, loss-based evaluation, and image-tokenizer properties. These lines of work motivate analyzing tokenizers within joint multimodal modeling rather than through isolated reconstruction or downstream metrics.

  • Unified multimodal understanding and generation: Unified multimodal models combine visual understanding and generation using pure-AR, serial AR–diffusion, and hybrid AR–diffusion architectures.
  • Loss-based analysis of unified multimodal models: Loss-based analyses use metrics such as perplexity and bits-per-byte to study scaling, efficiency, tokenizer comparability, and downstream error.
  • Studying image tokenizer properties: Image-tokenizer studies evaluate reconstruction with pixel-wise and feature-based metrics, alongside benchmarks of preserved text, identity, and detail.

3 Preliminaries

The paper reviews discrete image-tokenizer architectures, compression choices, and training objectives before comparing single-codebook tokenizers in a controlled framework.

  • VQGAN-based tokenizers encode images into continuous vectors, quantize them using a codebook, model the resulting indices, and decode quantized vectors into reconstructed images.
  • Training objectives: The standard VQGAN objective combines vector-quantization and autoencoding losses, including reconstruction, perceptual, and adversarial components.
  • Architecture: The study focuses on single-codebook tokenizers to avoid extra sequence-organization and prediction choices introduced by multiple sub-codebooks.
  • The study compares IBQ, GigaTok, and UniTok, recording vocabulary size, semantic loss, discriminator type, codebook usage, and reconstruction FID.
  • Bitwise compression ratio: The framework fixes 16 × 16 image tokens for 256 × 256 inputs and varies vocabulary size as its main compression-related axis.
  • Training objectives: Semantic supervision is implemented with a CLIP contrastive loss in UniTok variants to encourage tokens to capture high-level semantics.

4 Framework Construction

The framework extends pretrained language models into unified autoregressive multimodal models and evaluates tokenizer effects through continual pretraining, supervised finetuning, and task-specific validation losses.

  • The framework extends pretrained Qwen3 language models into unified autoregressive models by adding discrete image-token embeddings and corresponding language-model output capacity.
  • Continual pretraining: Continual pretraining uses large-scale pure-text and image-text data, with conditional T2I and I2T sequence formats and masked cross-entropy on supervised tokens.
  • Supervised finetuning and validation: The recipe includes supervised finetuning on multimodal, text, and text-to-image instruction data, while its validation experiment is intended to establish a tokenizer study testbed rather than superior benchmark performance.
  • Validation loss: The study evaluates text, unconditional image, T2I, and I2T validation losses on held-out samples using the same loss-masking rules as training.
  • Validation loss: Validation loss measures mean negative log-likelihood over supervised discrete tokens, so image-token losses assess latent-modeling quality for a fixed visual language rather than raw-pixel description length.

5 Interpreting Loss in Unified Multimodal Training

Task-specific validation losses provide a more informative lens than a single aggregate loss for unified multimodal training. Their scaling and tokenizer rankings differ by task, while their relationship with generation and post-SFT performance depends on the predicted token space.

  • 5.1 Task-Specific Loss Scaling: All four task-specific losses decrease smoothly with data and model scaling, roughly following power-law trends.The losses cover text, image, T2I, and I2T prediction.
  • 5.1 Task-Specific Loss Scaling: Text, image, T2I, and I2T losses exhibit distinct scaling behavior, so averaging them obscures task-dependent differences.Text loss is more influenced by language-model initialization, whereas image-related losses benefit more from multimodal continual pretraining.
  • 5.1 Task-Specific Loss Scaling: No single tokenizer ranking holds across tasks: UniTok has the highest text loss but lowest T2I loss, while GigaTok shows the opposite pattern.These rankings reveal trade-offs between text modeling and image-token prediction under the unified autoregressive objective.
  • 5.2 Loss–Performance Relation: Within a fixed tokenizer, both T2I and I2T losses align with text-to-image generation quality, although T2I loss is the stronger alignment.This relationship holds across data scales and hyperparameter settings for the fixed tokenizer.
  • 5.2 Loss–Performance Relation: Across tokenizers, I2T loss provides a more consistent generation signal than T2I loss because it is computed over a shared text vocabulary.The T2I loss–performance relationship shifts with the image-token space; vocabulary normalization reduces, but does not eliminate, this shift.
  • 5.2 Loss–Performance Relation: I2T loss remains correlated with post-SFT generation and shows moderate correlation with post-SFT visual understanding across tokenizers.The signal is informative for VQAv2 and GQA as well as generation benchmarks.

6 What Unified Training Reveals About Image Tokenizers

Loss-based analysis reveals tokenizer effects that isolated reconstruction or single-axis evaluations miss. Reconstruction fidelity, multimodal learnability, text modeling, semantic associations, and downstream performance can favor different tokenizer designs.

  • 6.1 Does better reconstruction imply better unified multimodal learning?: Reconstruction fidelity can diverge from multimodal learnability and downstream performance under unified autoregressive training.Lower rFID does not necessarily yield better validation losses, generation, or understanding.
  • 6.1 Does better reconstruction imply better unified multimodal learning?: Improving GigaTok rFID from 0.81 to 0.51 with a DINO-based discriminator left GenAI unchanged at 0.720 and reduced VQAv2 from 52.25 to 51.31.None of the validation losses improved significantly, despite the reconstruction gain.
  • 6.3 Semantic supervision: How does semantic alignment change multimodal learnability?: Semantic supervision worsens reconstruction fidelity yet improves all three reported validation losses and both reported downstream metrics.It also strengthens image-token associations with object-level words, supporting a better visual words interpretation.
  • 6.2 Image token space and text modeling: Image token space affects text modeling difficulty under joint training, even when the text tokenizer and training recipe are fixed.UniTok-sem beats UniTok on text loss, and IBQ-1024 beats IBQ-8192; removing image-token prediction largely removes these gaps.
  • 6.3 Semantic supervision: How does semantic alignment change multimodal learnability?: Semantic supervision improves object-level image-token–word associations without consistently reducing empirical image-token n-gram entropy.UniTok-sem has higher PMI for COCO object words, while its 3- and 7-gram entropy exceeds UniTok’s.
  • 6.4 Vocabulary size: Is a larger vocabulary better for unified multimodal learning?: Vocabulary size affects multimodal learnability non-monotonically: IBQ-8192 has the lowest normalized T2I and image losses, while IBQ-16384 has the best I2T loss.IBQ-16384 nevertheless achieves the best post-SFT generation and understanding performance, possibly through higher reconstruction fidelity.

7 Conclusion

The paper presents a controlled, loss-based testbed for studying image tokenizers as visual languages in unified autoregressive multimodal models. Its findings show that tokenizer evaluation should consider reconstruction fidelity together with task-specific joint modeling across multiple design axes.

  • The testbed evaluates image tokenizers through task-specific losses and downstream behavior in unified autoregressive multimodal models.
  • Reconstruction fidelity can diverge from multimodal learnability under the shared autoregressive objective.
  • Image-token space can affect text modeling during joint training.
  • The study revisits discriminator choice, semantic supervision, and vocabulary size as tokenizer design axes.
  • Unified tokenizer design should account for reconstruction fidelity and joint modeling difficulty across tasks rather than optimizing a single proxy metric.

A Training Details

The training setup combines public image data, Qwen3, several image-tokenizer families, and supervised finetuning to create a reusable unified multimodal testbed. It also defines tokenizer variants and preprocessing choices for controlled comparisons.

  • Training setup: The continual-pretraining mixture uses a 1:8 text-to-image-text ratio to keep total image tokens comparable across training.The ratio was selected after observing that comparable performance requires matching the total number of image tokens seen during training.
  • Training setup: The framework uses public images with Qwen3 and evaluates generation and understanding after supervised finetuning.The resulting 8B model achieves similar performance to Liquid-7B on most benchmarks, supporting the testbed’s reuse for tokenizer studies.
  • Tokenizer variants: IBQ variants enable scalable large-codebook training, and the study uses vocabulary sizes 1024, 8192, and 16384 without semantic loss.The 262144 variant is excluded because it is much larger than the base model’s original vocabulary and might cause training difficulty.
  • Tokenizer variants: UniTok is evaluated with a single codebook of vocabulary size 16384, with and without CLIP-based semantic loss.This ablation tests whether semantic supervision changes downstream scaling under the unified multimodal framework.
  • Preprocessing and evaluation: UniTok uses zooming-then-cropping preprocessing, unlike the other tokenizer families, but the comparison remains within the same family.The authors state that this difference does not affect their conclusions because they do not compare tokenizers from different families.

B Supplementary Results on Training Dynamics

Supplementary analyses show that multimodal training dynamics differ across tasks and evolve over training. Image-related gradients dominate early optimization, while text gradients decay more slowly, consistent with distinct loss-scaling curves.

  • Loss scaling: Most data-scaling curves are convex for LAION-I2T, LAION-T2I, and LAION-Image, but slightly concave for text loss.The fitted curve shapes suggest different training dynamics across tasks, with image-related modeling dominating early and saturating later.
  • Gradient dynamics: The supplementary figures track gradient norms by modality across training steps for GigaTok and IBQ-16384.Both runs show the T2I gradient slowly saturating at the end of training.
  • Gradient dynamics: T2I gradients start larger than text gradients and gradually decay throughout continual pretraining.The text gradient decays more slowly and reaches a level similar to the T2I gradient near the end of training.
  • Data sources: T2I and image-loss trends remain similar across three data sources, while I2T trends differ slightly for LAION-I2T.The authors associate the LAION-I2T difference with longer, more detailed recaptioned captions that rely more on language ability.

C.2 Loss Scaling Results for Different Hyperparameters

Loss behavior depends strongly on task, hyperparameter setting, annealing, benchmark, and normalization. These results show that loss comparisons require task-aware interpretation and, for image-token losses, correction for vocabulary scale.

  • Hyperparameters: No hyperparameter setting dominates all tasks: aggressive settings favor image-related losses, while milder settings favor text loss.High learning rate and small batch size perform better on LAION-I2T, LAION-T2I, and LAION-Image, whereas mild settings improve text fitting.
  • Annealing: Annealing mainly reduces text loss and leaves image-related losses largely unchanged.An exception occurs for LAION-I2T, where lr=1e-4, bs=512 overtakes lr=3e-5, bs=512 after annealing.
  • Loss and VQA: I2T loss has an inconsistent relationship with TextVQA: it is negatively correlated at 0.6B but shows the opposite trend at 4B.Optimal hyperparameters also differ across VQA benchmarks, and TextVQA can favor mild settings despite higher I2T loss.
  • Loss normalization: Normalizing image-token loss by empirical entropy H1 produces nearly the same loss–performance relation as normalizing by log2 B.For all seven tokenizers, H1 lies within 1.3% of log2 B.
  • Loss normalization: Vocabulary-size normalization corrects loss scale but does not calibrate losses across tokenizer architectures at fixed vocabulary size.The analysis also excludes tokenizers with severe or complete codebook collapse, where nominal vocabulary size may not represent the effective prediction space.

C.6 Ablation on I2T Objective

Ablating the I2T objective does not remove the tokenizer gap in text or T2I losses, while I2T training complements text modeling during joint training.

  • Ablating I2T does not reduce the gap between UniTok and UniTok-sem on text and T2I losses.The ablation reformats I2T samples into T2I order and trains with Text+T2I data.
  • Text loss increases for both UniTok and UniTok-sem when the I2T objective is removed.This supports a complementary role for I2T in text modeling during joint training.
  • The I2T objective complements rather than competes with text modeling, while removing it slightly decreases T2I loss.

D Supplementary Benchmark Information and Results

The supplementary evaluations define the generation and understanding benchmarks, report qualitative generation comparisons, and show that seed sensitivity is much larger for POPE and MME-P than for the analyzed benchmarks.

  • Text-to-image generation: Generation evaluation uses VQAScore on GenAI-Bench and gFID on MJHQ-30K, with unfiltered softmax sampling and CFG scale=7.0.VQAScore ranges from 0 to 1 and measures text-image alignment through CLIP-FlanT5-XXL.
  • Text-to-image generation: GigaTok and GigaTok-DINO produce the best qualitative image quality and alignment, followed by UniTok and UniTok-sem.
  • Visual understanding: Visual understanding is evaluated with VQAv2 and GQA, while TextVQA is reported in Appendix C.4.POPE and MME are excluded from the analysis because runs showed large variance across shuffling seeds.
  • Benchmark sensitivity to random seeds: 1.9% relative difference separates seed runs for VQAv2, GQA, GenAI-all, and MJHQ-30K, compared with 7.07% for POPE and 4.98% for MME-P.The comparison uses one GigaTok configuration and two data-shuffling seeds.

E Limitations

The study’s conclusions are bounded by its controlled tokenizer set, validation-loss lens, incomplete scaling analysis, diagnostic measurements, fixed backbone family, and limited downstream coverage.

  • The study covers seven discrete image tokenizers with fixed token length and single-codebook designs, leaving sub-codebooks, larger resolutions, and different sequence lengths for future work.Varying sequence length would confound the numbers of image tokens and images seen.
  • Pretraining validation loss is a lens for joint modeling, not a replacement for downstream evaluation or tokenizer selection.I2T loss is less predictive for specialized tasks such as TextVQA.
  • The study does not derive compute-efficient frontiers from a full from-scratch scaling-law analysis because of high computational cost.Future work could compare tokenizer effects on the multimodal AR compute-efficient frontier.
  • Validation-set PMI and n-gram entropy analyses are diagnostic rather than mechanistic, and high-order empirical entropy can suffer finite-sample bias.
  • Findings are established for Qwen3-based pure-AR unified models and are not claimed to remain unchanged under another backbone family.Cross-family validation would require matched retraining and evaluation across tokenizers.
  • Downstream evaluation covers a limited set of generation and VQA benchmarks, with no dedicated region-level grounding or fine-grained spatial-reasoning benchmark.The study therefore does not claim extension to these specialized visual capabilities.
Loading 2609.09143v1…