PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Semantic Equivalence of Tokenization in Multimodal LLM

Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, Shuicheng Yan

arXiv:2406.05127v4cs.CV

TL;DR

Existing visual tokenization can fragment image content and leave vision and language tokens insufficiently aligned. SeTok dynamically groups visual features into semantic units, and SETOKIM uses those tokens in a multimodal model. The model outperforms patch-level tokenization and learnable queries across tasks, with a reported 3.6% improvement on GQA.

  • Problem

    Existing visual tokenizers can fragment objects or misalign tokens with semantic units, limiting fine-grained vision-language alignment.

    Source: §1 · pp. 1–2

  • Method

    SeTok dynamically clusters visual features into a variable number of semantic-complete tokens, which SETOKIM uses in a unified multimodal model.

    Source: §1 · pp. 2–3

  • Results

    SETOKIM outperforms standard patch-level tokenization and learnable queries across various tasks, with performance improvement on GQA by 3.6%.

    Source: §1 · p. 3, §4.1 · p. 7

  • Takeaways & Limitations

    The paper's analyses and visualizations show SeTok tokens are more interpretable and semantic-complete, achieving fine-grained vision-language alignment.

    Source: §1 · p. 3

    SETOKIM supports images up to 384×384, while higher-resolution images remain challenging, particularly for detailed visual reasoning.

    Source: B LIMITATION · p. 18

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in processing vision-language tasks. One of the crux of MLLMs lies in vision tokenization, which involves efficiently transforming input visual signals into feature representations that are most beneficial for LLMs. However, existing vision tokenizers, essential for semantic alignment between vision and language, remain problematic. Existing methods aggressively fragment visual input, corrupting the visual semantic integrity. To address this, this paper proposes a novel dynamic Semantic-Equivalent Vision Tokenizer (SeTok), which groups visual features into semantic units via a dynamic clustering algorithm, flexibly determining the number of tokens based on image complexity. The resulting vision tokens effectively preserve semantic integrity and capture both low-frequency and high-frequency visual features. The proposed MLLM (Setokim) equipped with SeTok significantly demonstrates superior performance across various tasks, as evidenced by our experimental results. The project page is at https://chocowu.github.io/SeTok-web/.

1 INTRODUCTION

Vision tokenization is central to MLLMs, but existing approaches can misalign visual tokens with semantic units in language. SeTok dynamically groups visual features into semantic tokens, and SETOKIM uses them in a unified multimodal model that performs better across varied vision-language tasks.

1 INTRODUCTION

  • MLLMs combine language-based LLMs with multimodal encoders and decoders, and existing vision tokenizers include patch-level continuous tokens, patch-level discrete tokens, and learnable queries.Figure 1 contrasts these token types with SeTok's semantic-equivalent continuous tokens.Source: §1 · p. 1, Figure 1 · p. 2
  • Existing patch-level tokens fragment objects, while fixed query tokens can misalign with visual semantic units and offer limited interpretability.Patch fragmentation disrupts visual semantic integrity and loses high-frequency details such as edges and contours.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • SeTok dynamically clusters visual features into semantic units, determines token counts from image content, and merges cluster features to retain high- and low-frequency information.Image reconstruction and concept-level image-text contrastive loss train the tokens for visual detail and semantic alignment.Source: §1 · p. 2, Figure 2 · p. 3
  • SETOKIM uses SeTok-generated visual tokens in a unified multimodal sequence and outperforms standard patch-level tokenization and learnable queries across various tasks.The paper reports gains across visual question-answering, image generation and editing, and referring segmentation.Source: §1 · p. 3

2 METHODOLOGY

The methodology builds SeTok to group image features into semantically complete visual tokens, then integrates those tokens with text in SETOKIM for multimodal understanding and generation.

2.1 SEMANTIC-EQUIVALENT VISION TOKENIZER

  • SeTok assigns image patches to a variable number of semantic masks, selecting cluster seeds by local density and distance before assigning remaining features until a stopping condition is met.Remaining embeddings receive an additional mask when the stopping condition is reached.Source: §2.1 · pp. 3–4
  • Within each cluster, SeTok combines 2D position embeddings with visual features, transforms and averages them, then uses inter-cluster layers to model relationships among tokens.Source: §2.1 · p. 4
  • SeTok training combines concept-level image-text contrastive loss with image reconstruction loss to target semantic completeness and pixel-level detail retention.The losses are combined as a weighted sum.Source: §2.1 · p. 4

2.2 SETOKIM

  • SETOKIM concatenates SeTok visual tokens with text tokens for LLM processing, then routes generated visual tokens to image detokenization and mask decoding.The mask decoder uses generated vision tokens to produce referring masks.Source: §2.2 · pp. 4–5
  • SETOKIM uses next-token prediction for autoregressive modeling, with text cross-entropy, image-generation diffusion loss, and mask-decoder binary cross-entropy and dice losses.Source: §2.2 · p. 5
  • Training proceeds from multimodal pretraining on image-text and language data to instruction tuning across visual QA, generation, and editing datasets.The mask decoder is separately trained on segmentation datasets during the first stage.Source: §2.2 · p. 5

3 SETTINGS

The experiments use a LLaMA-2-7B backbone with a pretrained SigLIP vision encoder and evaluate SETOKIM across visual understanding, generation, editing, and referring segmentation.

3 SETTINGS

  • SETOKIM uses LLaMA-2-7B, pretrained SigLIP-SO400M-patch14-384, 12 inner-cluster and 8 inter-cluster Transformer layers, and 512-dimensional semantic-equivalent tokens.Its detokenizer uses 12 blocks for mask tokens to query visual information.Source: §3 · pp. 5–6
  • Visual understanding is evaluated on Flicker30K, VQAv2, GQA, OK-VQA, POPE, MME, and MM-Vet, while generation fidelity is evaluated on MSCOCO.Source: §3 · p. 6
  • Image editing is tested on Magicbrush, EVR, and MA5K, and referring segmentation on refCOCOg, refCOCO+, and Reaseg.Source: §3 · p. 6

4 EXPERIMENTAL RESULTS

SETOKIM performs across visual understanding, generation, editing, and referring-expression segmentation; the analyses examine SeTok’s token quality, clustering choices, and qualitative behavior.

4.1 MAIN RESULTS

  • TiTok scores 72.6 vs 75.4 Top-1 accuracy; SeTok also achieves reconstruction quality comparable to well-trained VQ models.The measures are reconstruction rFID and ImageNet classification Top-1 accuracy.Source: §4.1 · p. 6, Table 3 · p. 6
  • SETOKIM is competitive on visual understanding, comparable on complex-prompt generation, stronger on editing metrics, and outperforms current SoTA on referring-expression segmentation.For segmentation, the reported comparison covers RefCOCO+/g and ReaSeg.Source: §4.1 · p. 7, Table 4 · p. 7

4.2 IN-DEPTH ANALYSIS AND QUALITATIVE EVALUATION

  • Removing contrastive loss, replacing the token merger with cluster averages, or removing positional encoding and transformers degrades downstream performance.Without contrastive loss, reconstruction remains commendable while visual-understanding performance decreases markedly.Source: §4.2 · p. 7, Table 6 · p. 7
  • Dynamic clustering strategies outperform fixed setups overall, accelerate training, and reduce inference costs; hard clustering performs better than soft clustering.Different datasets favor different fixed cluster counts.Source: Table 5 · p. 7, §4.2 · p. 8
  • Qualitative examples show image understanding and generation, including deciphering obscured text and generating images relevant to textual prompts.The analysis also reports object-focused descriptions that mitigate hallucinatory responses commonly observed in MLLMs.Source: §4.2 · p. 8, Figure 4 · p. 8
  • Editing examples show instruction adherence and preservation of unchanged details, including adding pizza toppings and removing an unusual object.The comparison includes cases where Emu-2-Gen and MGIE alter unintended elements or fail to perform the requested edit.Source: §4.2 · pp. 8–9, Figure 5 · p. 9
  • Token visualizations show a dynamic token count and variable granularity: a person may form one token or separate head, body, and leg tokens.Tokens also correspond to scene elements such as giraffes, grass, trees, and backgrounds.Source: §4.2 · p. 9, Figure 6 · p. 9

5 RELATED WORK

Related work traces the development of visual MLLMs and tokenization, then situates SeTok alongside scene-decomposition methods. It identifies semantic fragmentation and fixed token counts as relevant limitations of prior approaches.

5 RELATED WORK

  • Visual MLLMs expanded from multimodal-input understanding to visual generation, segmentation, and editing as encoders and decoders were integrated with LLMs.The section describes visual MLLMs as a major focus of research and application.Source: §5 · p. 10
  • Visual tokenization encodes image content as patch features or discrete codebook tokens for LLM input, but patchifying can compromise semantic-unit integrity and alignment.The paper presents Semantic Equivalent Tokenization as its proposed response to this bottleneck.Source: §5 · p. 10
  • Scene-decomposition methods often use fixed query counts and cross-attention; the paper contrasts these with dynamic semantic-object counts and simpler optimization structures.The section says fixed-query approaches may mismatch visual content and increase computational demands when combined with LLMs.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents SeTok and SETOKIM as a semantic-tokenization approach evaluated across several multimodal task types. The ethics statement notes output-quality and safety concerns, alongside data-privacy and bias-mitigation practices.

6 CONCLUSION

  • SeTok converts patch-level features into a variable number of semantically complete tokens, and SETOKIM uses a unified autoregressive objective and two-stage training.The conclusion reports performance across comprehension, generation, segmentation, and editing tasks.Source: §6 · p. 10

A ETHIC STATEMENT

  • SETOKIM may generate low-quality content because of limited fine-tuning data or base-model quality, and multimodal hallucinations may be harmful.The authors advise users to interpret results with caution and state that commercial use is not allowed.Source: A ETHIC STATEMENT · p. 18
  • The authors report using public or explicitly consented datasets and describe privacy protocols and dataset testing intended to identify and mitigate bias.The bias statement frames testing as intended to mitigate unfair outcomes in applications.Source: A ETHIC STATEMENT · p. 18

B LIMITATION

The authors identify model scale, image resolution, and hallucination as limitations of SETOKIM. These constraints bound the model configurations and visual settings discussed.

B LIMITATION

  • Evaluation is limited to 7B parameters; the authors leave potential benefits of 65B or 130B models for future work.They note that MLLM performance is limited by the scale of the core backbone LLM.Source: B LIMITATION · p. 18
  • SETOKIM supports images up to 384×384, while higher-resolution detailed reasoning and fine-grained details in low-resolution inputs remain challenges.The authors also identify generalization across diverse modalities as an area needing improvement.Source: B LIMITATION · p. 18
  • Hallucinations remain inevitable despite progress in mitigation through fine-grained vision-language alignment.The authors identify hallucination as an area for future exploration and enhancement.Source: B LIMITATION · p. 18

C DETAILED METHOD

The detailed method describes an iterative token-clustering procedure and a concept-level image-text contrastive objective. Together, these passages specify how visual features are assigned to masks and aligned with prompted object labels.

C.1 TOKEN CLUSTER

  • Each clustering iteration selects an unassigned seed, forms a kernel-based concept mask with a density penalty, updates the remaining scope, and stops by criterion before appending a residual mask.Seed selection uses seed scores and the current scope; mask assignment uses embedding distance from the selected seed.Source: §C.1 · pp. 18–19

C.2 CONCEPT-LEVEL IMAGE-TEXT CONTRASTIVE LOSS

  • The concept-level contrastive objective pairs images with prompted sentences for sampled object labels, giving each image K positive pairs and B(K −1) negative pairs in a batch.The loss is defined as the sum of two two-way contrastive losses.Source: §C.2 · p. 19
  • The concept representations for the prompted sentences come from a pre-trained CLIP-based text encoder that is frozen during training.Source: §C.2 · p. 19

D.1 IMPLEMENTATION DETAILS

The implementation details specify SeTok and SETOKIM components and identify the datasets used in training. The training-data passage also notes overlap between some Stage-I and Stage-II datasets.

D.1 IMPLEMENTATION DETAILS

  • For SeTok, the authors use pre-trained SigLIP-SO400M-patch14-384 (Zhai et al., 2023) as the vision encoder, setting the numbers of inner-cluster and inter-cluster transformer layers to 12 and 8, respectively.Source: §D.1 · p. 19
  • The detokenizer uses 12 cross-attention transformer layers to convert a dynamic token count into a fixed-size representation, alongside a CNN-based pixel decoder for image reconstruction.Source: §D.1 · p. 20
  • SETOKIM initializes its LLM backbone with LLaMA-2-7B and uses a mask decoder with prompt self-attention and bidirectional prompt-image cross-attention.The decoder upsamples image embeddings and maps output tokens to a dynamic linear classifier for foreground probabilities.Source: §D.1 · p. 20

D.2 TRAINING DATA

  • SeTok uses ImageNet-1K for reconstruction and OpenImages for reconstruction and alignment; some datasets overlap across Stage-I and Stage-II training.The passage names VQAv2, ShareGPT4V, and GQA as examples of datasets included in LLaVA-v1.5-mix-665.Source: §D.2 · p. 20, Table 7 · p. 20

D.3 TRAINING RECEIPT

The training-receipt passages specify the hardware used and point to tables covering training recipes and baseline configurations. The configuration comparison covers LLM version, vision encoder, and data size.

D.3 TRAINING RECEIPT

  • All training is conducted on 64× H100 (80G) GPUs, and Table 9 lists recipes for SeTok and both SETOKIM training stages.Source: §D.3 · p. 20, Table 9 · p. 21

D.4 BASELINES.

  • The baseline comparison covers LLM version, vision encoder, and data size; Table 8 presents the configurations for baselines and SETOKIM.Source: §D.4 · p. 21, Table 8 · p. 21

E EXTENDED EXPERIMENTAL ANALYSIS

Extended experiments compare SETOKIM with LLaVA and other tokenizers, then examine data proportions, tokenizer losses, and encoder freezing. Qualitative examples cover segmentation, reconstruction, generation, in-context learning, and multi-image reasoning.

E EXTENDED EXPERIMENTAL ANALYSIS

Loading 2406.05127v4…