PaperlayerView exact revision on arXiv

Source-linked AI summary

Planting a SEED of Vision in Large Language Model

Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, Ying Shan

arXiv:2307.08041v2cs.CV

TL;DR

SEED addresses the gap between LLMs’ emergent capabilities and multimodal models by exploring whether visual and textual representations can be processed interchangeably in a unified autoregressive Transformer. It introduces a VQ tokenizer with 1D causal, high-level semantic codes and reports competitive retrieval and image-generation performance against BLIP-2 and Stable Diffusion, respectively.

  • Problem

    Multimodal LLMs had yet to achieve the remarkable success of LLMs in emergent capabilities, motivating the premise that text and images can be represented and processed interchangeably in a unified autoregressive Transformer.

    Source: §1 · pp. 1–2

  • Method

    SEED is a VQ-based image tokenizer that produces discrete visual codes with 1D causal dependency and high-level semantics, aligned to an off-the-shelf LLM through LoRA tuning.

    Source: §1 · p. 2

  • Results

    SEED tokens show competitive performance in text-image retrieval compared to BLIP-2 and image generation compared to Stable Diffusion; the trained multimodal LLM handles image-to-text and text-to-image generation.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    SEED supports multimodal LLM training with the scalable original text next-word-prediction recipe and advances work toward emergent multimodal capabilities.

    Source: §5 · p. 9

    The work is an initial tokenizer-design exploration; scaling tokenizer training and using stronger LLMs for comprehensive multimodal pretraining and instruction tuning are left to future work.

    Source: §1 · p. 3

Abstract

from arXiv · show

We present SEED, an elaborate image tokenizer that empowers Large Language Models (LLMs) with the emergent ability to SEE and Draw at the same time. Research on image tokenizers has previously reached an impasse, as frameworks employing quantized visual tokens have lost prominence due to subpar performance and convergence in multimodal comprehension (compared to BLIP-2, etc.) or generation (compared to Stable Diffusion, etc.). Despite the limitations, we remain confident in its natural capacity to unify visual and textual representations, facilitating scalable multimodal training with LLM's original recipe. In this study, we identify two crucial principles for the architecture and training of SEED that effectively ease subsequent alignment with LLMs. (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. As a result, the off-the-shelf LLM is able to perform both image-to-text and text-to-image generation by incorporating our SEED through efficient LoRA tuning. Comprehensive multimodal pretraining and instruction tuning, which may yield improved results, are reserved for future investigation. This version of SEED was trained in 5.7 days using only 64 V100 GPUs and 5M publicly available image-text pairs. Our preliminary study emphasizes the great potential of discrete visual tokens in versatile multimodal LLMs and the importance of proper image tokenizers in broader research.

1 Introduction

SEED targets the gap between discrete visual tokens and LLM-compatible multimodal training by producing causal, high-level image codes. The paper introduces its tokenizer design, explains how it connects to an LLM, and reports preliminary comprehension and generation results.

1 Introduction

  • Earlier discrete-token systems often required super-scale training to converge, while VQ-VAE captured information too low-level for effective multimodal comprehension.The paper argues existing image tokenizers had not met the combined requirements of visual understanding, generation, and scalable multimodal training.Source: §1 · p. 2
  • SEED produces discrete visual codes with 1D causal dependency and high-level semantics, and the paper aligns a pretrained LLM to the tokenizer using LoRA.Visual codes are treated as new words by adding mapped codes to the LLM vocabulary.Source: §1 · p. 2
  • SEED converts 2D raster-ordered embeddings into 1D causally dependent semantic codes, aiming to match the high-level semantics of words and LLMs’ unidirectional attention.The paper presents tokenizer training as the first of three stages, followed by multimodal pretraining and instruction tuning.Source: §1 · p. 2
  • The tokenizer uses a ViT encoder and Causal Q-Former to form causal semantic embeddings, then a VQ Codebook and Reverse Q-Former to produce generation embeddings for the SD latent space.The generation embeddings are used with the off-the-shelf SD-UNet.Source: §1 · pp. 2–3
  • SEED tokens show competitive text-image retrieval against BLIP-2 and image generation against Stable Diffusion; after multimodal autoregressive training, SEED-OPT2.7B performs image-to-text and text-to-image tasks.The reported tasks include zero-shot image captioning and visual QA, alongside high-quality image generation.Source: §1 · p. 3
  • The authors reserve comprehensive multimodal pretraining and instruction tuning, as well as scaling training and using stronger LLMs, for future work.The introduction describes the presented effort as an initial exploration of tokenizer design.Source: §1 · p. 3

2 SEED Visual Tokenizer

SEED converts image features into discrete, causally ordered semantic codes and decodes them into embeddings for image generation. Its evaluations report competitive retrieval and image-generation performance, with semantic consistency in reconstructed images.

2.1 Pilot Experiments of Baseline Tokenizers

  • A high-level tokenizer achieves CIDEr 42.0 versus VQ-VAE's 34.0 in zero-shot COCO captioning, supporting semantic representations over low-level image-detail reconstruction.Both tokenizer representations were aligned with OPT2.7B on CC3M.Source: §2.1 · p. 3

2.2 Architecture

  • The tokenizer comprises a ViT encoder, Causal Q-Former, VQ Codebook, Reverse Q-Former, and UNet decoder, mapping 16×16 image features to 32 codes and 77 generation embeddings.The ViT encoder and UNet decoder derive from pretrained BLIP-2 and Stable Diffusion models, respectively.Source: §2.2 · p. 3

2.2.1 Training Stage I: Causal Q-Former

  • Causal Q-Former uses masked self-attention over 32 learnable queries and cross-attention to frozen ViT features, trained contrastively on 5M image-text pairs.The training data include CC3M, Unsplash, and COCO.Source: §2.2.1 · p. 3
  • Causal Q-Former embeddings outperform BLIP-2 on aggregated Recall@mean without losing retrieval performance against bidirectional attention.The zero-shot image-text retrieval evaluation uses COCO and Flickr30K and removes BLIP-2's ITM reranker for a fair comparison.Source: §2.2.1 · p. 4, Table 1 · p. 4

2.2.2 Training Stage II: Visual Quantization and De-tokenization

  • The VQ codebook discretizes 32 causal embeddings into 32 codes, while a Transformer reconstructs embeddings and Reverse Q-Former reconstructs Stable Diffusion text features.The displayed reconstruction examples show images with semantics consistent with the inputs; the Reverse Q-Former produces generation embeddings for the SD-UNet.Source: §2.2.2 · pp. 4–6, Figure 3 · p. 5
  • SEED slightly trails Stable Diffusion and outperforms GILL on image-generation CLIP similarity.The comparison evaluates COCO and Flickr30K; Table 2 reports CLIP similarity as the metric.Source: Table 2 · p. 5, §2.2.2 · p. 5

3 Multimodal Autoregression with SEED Visual Tokens

SEED-OPT2.7B uses LoRA-tuned autoregression with SEED visual tokens for image-to-text and text-to-image tasks. Its evaluations report promising zero-shot captioning and visual question answering, plus qualitative text-to-image results.

Image-to-Text Autoregression

  • LoRA tuning trains OPT2.7B for image-to-text autoregression using visual codes and an “A photo of” prefix to predict caption tokens.OPT2.7B is frozen while LoRA is fine-tuned.Source: §3 · p. 6

Text-to-Image Autoregression

  • Joint autoregression trains OPT2.7B to predict image codes from “Generate an image” and a caption, alongside image-to-text prediction.The model is frozen while LoRA is fine-tuned to predict the next vision token.Source: §3 · p. 6

Evaluation of Multimodal Generation

  • At inference, Reverse Q-Former embeddings are decoded by SD-UNet into realistic images that are semantically relevant to the description.Figure 6 presents qualitative text-to-image examples.Source: §3 · pp. 6–7

Evaluation of Multimodal Understanding

  • 5M image-text pairs, versus BLIP-2's 129M, accompany promising zero-shot image-captioning and visual question-answering results for SEED-OPT2.7B.The comparison covers COCO and NoCaps captioning and VQAv2 and GQA question answering.Source: §3 · p. 6, Table 3 · p. 7
  • The model answers open-ended visual questions despite having no VQA-dataset training, and produces captions with the “a photo of” prompt.Figure 5 shows qualitative captioning and visual-question-answering examples.Source: §3 · p. 7, Figure 5 · p. 7

4 Related Work

Related work explores LLM-based visual comprehension and generation, while prior visual tokenizers differ in their training targets and compatibility with autoregressive prediction.

Multimodal Large Language Models for Comprehension

  • Prior comprehension systems align pretrained image-encoder features with LLMs but train only next-text-token prediction, with no vision-data supervision.These systems can output text from multimodal inputs.Source: §4 · p. 7

Multimodal Large Language Models for Generation

  • CogView [14] reconstructs image pixels for tokenizer training and fine-tunes GPT models for image generation, while GILL [25] maps LLM embeddings to a frozen image generator.Both works target image generation without being explicitly designed for multimodal comprehension.Source: §4 · p. 8

Visual Tokenizer

  • VQ-VAE tokenizers [15] [13] [17] capture low-level image details through pixel reconstruction, whereas BEiT v2 [18] codes 2D vision features that conflict with LLM unidirectional attention.BEiT v2 [18] instead reconstructs high-level teacher features.Source: §4 · p. 8

5 Conclusion

The conclusion presents SEED as a discrete tokenizer whose high-level, causally ordered visual tokens are intended to fit LLM training and support multimodal tasks.

5 Conclusion

  • SEED is designed around high-level visual semantics and 1D causal token generation to make visual tokens compatible with LLMs.The paper says this lets multimodal LLMs follow the original next-word-prediction recipe and perform image-to-text and text-to-image generation.Source: §5 · p. 9
  • The authors suggest a more rational visual tokenizer could substantially reduce multimodal-training cost and complexity, supporting lower-carbon large-scale training.This is framed as a potential benefit, and the project is still in progress.Source: §5 · p. 9
  • The conclusion describes the work as a step toward emergent multimodal capabilities and calls for greater attention to visual tokenizers.Source: §5 · p. 9

Related papers

Loading 2307.08041v2…