PaperlayerView exact revision on arXiv

Source-linked AI summary

Making LLaMA SEE and Draw with SEED Tokenizer

Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, Ying Shan

arXiv:2310.01218v1cs.CV

TL;DR

Existing multimodal LLMs struggle to unify image comprehension and generation because visual representations are not sufficiently compatible with language-model autoregression. The paper introduces SEED, a causal semantic image tokenizer, and uses it to train SEED-LLaMA with unified next-word prediction; the resulting system achieves competitive multimodal performance and demonstrates compositional emergent abilities.

  • Problem

    Existing multimodal LLMs have not effectively unified visual comprehension and generation, while existing visual tokenizers can be costly, low-level, or incompatible with LLM autoregression.

  • Method

    SEED converts images into high-level discrete visual codes with 1D causal dependency, and SEED-LLaMA trains on interleaved visual-textual data using next-word prediction and instruction tuning.

  • Results

    SEED-LLaMA shows competitive results across multimodal comprehension and image-generation tasks and demonstrates multi-turn in-context multimodal understanding, reasoning, and generation.

  • Takeaways & Limitations

    A unified autoregressive framework can support multimodal comprehension and generation while exhibiting compositional emergent abilities.

Abstract

from arXiv · show

The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing to the gradual evolution of General Artificial Intelligence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world context. However, despite the considerable advancements made by recent multimodal LLMs, they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to overcoming the present impasse lies in enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce SEED, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw at the same time. We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a 1D causal dependency, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on a broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant.

1 Introduction

SEED addresses the difficulty of unifying multimodal comprehension and generation by representing images as semantic, causally dependent tokens compatible with LLM autoregression. SEED-LLaMA then uses these tokens for scalable multimodal pretraining and instruction tuning, showing competitive task performance and emergent multimodal abilities.

  • Existing multimodal LLMs improve visual tasks but do not yet match LLMs’ emergent capabilities or effectively unify comprehension and generation.
  • Existing visual tokenizers are costly to train, capture overly low-level information, and fail to support unified multimodal comprehension and generation.
  • SEED produces discrete visual codes with 1D causal dependency and high-level semantics for compatibility with LLM processing and both visual comprehension and generation.Causal dependency aligns image-token prediction with unidirectional attention, while high-level semantics match the abstraction of word tokens.
  • SEED combines a ViT encoder, Causal Q-Former, VQ Codebook, MLP, and UNet decoder to tokenize images and decode generation embeddings into realistic images.
  • SEED-LLaMA is pretrained and instruction-tuned on multimodal data with next-word prediction, demonstrating competitive performance and multi-turn in-context multimodal generation.

2 Related Work

Prior multimodal systems typically specialize in textual understanding or image generation, while discrete-token approaches have struggled to unify both capabilities. SEED instead targets semantic, autoregressive visual codes that support this unification.

  • Most MLLMs align image features with LLMs but train only next-text-token prediction, so they can output texts rather than images.
  • CogView and GILL generate images with LLMs, but neither is explicitly designed to unify multimodal comprehension and generation.
  • CM3Leon performs image-text autoregression but reports weaker visual comprehension because its tokenizer captures overly low-level information.On COCO image captioning, the cited comparison is CIDEr 61.6 for CM3Leon versus 126.9 for SEED.
  • Earlier VQ-VAE tokenizers reconstruct low-level image details, whereas Beit v2’s 2D visual codes are incompatible with dominant LLMs’ unidirectional attention for image generation.
  • SEED’s visual codes use causal dependency and high-level semantics, and its generation embeddings can be decoded into realistic images with frozen unCLIP-SD.

3 Method

SEED converts images into fixed-length causal semantic codes and decodes them through embeddings aligned with unCLIP-SD. SEED-LLaMA treats visual codes as vocabulary words in unified next-word prediction, followed by multimodal instruction tuning.

  • SEED Tokenizer: The Causal Q-Former converts 16×16 ViT features into 32 causal embeddings, which the VQ codebook discretizes into 32 visual codes.
  • SEED Tokenizer: Causal Q-Former queries attend only to previous queries while cross-attending to frozen image features, and contrastive learning trains the embeddings on image-text pairs.
  • SEED Tokenizer: The VQ codebook quantizes causal embeddings, while a Transformer reconstructs continuous embeddings and an MLP reconstructs a frozen unCLIP-SD image embedding.
  • Multimodal Pretraining: SEED-LLaMA discretizes visual inputs, interleaves visual codes with text tokens, and applies a unified next-word-prediction objective with visual codes treated as new words.
  • Multimodal Pretraining: The model adds 8192 visual codes to the vocabulary, expanding the embedding and decoder-head layers with randomly initialized parameters.
  • Multimodal Pretraining: Training first uses LoRA and then fine-tunes all parameters except the embedding layer, which is frozen to avoid unstable training loss.
  • Instruction Tuning: Multimodal instruction tuning uses supervised fine-tuning with an instruction-answer template, counting only answer content toward the loss.
  • SEED Tokenizer: Figure 5 traces reconstruction from the original image through SEED tokenization, causal codes, and de-tokenization to the reconstructed image.

4 Experiment

Experiments evaluate SEED’s semantic visual codes, reconstruction quality, and SEED-LLaMA’s multimodal comprehension and generation. Results support discrete codes for both understanding and image generation, including compositional and multi-turn capabilities.

  • SEED Tokenizer: Causal Q-Former outputs outperform BLIP-2 on aggregated Recall@mean for COCO and Flickr30K image-text retrieval.The evaluation uses Recall@K under a dual-stream inference setup without BLIP-2’s ITM re-ranking module.
  • SEED Tokenizer: SEED’s discrete codes achieve competitive image-text retrieval performance compared with BLIP-2, indicating high-level semantic representations.Retrieval uses reconstructed embeddings from the discrete codes.
  • SEED Tokenizer: SEED reconstruction only slightly reduces CLIP similarity compared with the unCLIP-SD upper bound on COCO and Flickr30K.Images are represented by 32 causal codes and one generation embedding before decoding with the unCLIP-SD-UNet.
  • Quantitative Evaluation: SEED-LLaMA achieves competitive image and video understanding performance against MLLMs using continuous visual representations.Larger pretrained LLMs improve SEED-Bench results, while instruction tuning further enhances performance.
  • Quantitative Evaluation: SEED-LLaMA generates text-to-image outputs with higher ground-truth similarity, using a frozen SD-UNet.Evaluation on MS-COCO and Flickr30K uses pair-wise CLIP similarity.
  • Emergent Abilities: SEED-LLaMA supports multi-turn multimodal responses, zero-shot compositional generation, and combinations of image and text inputs.Examples include sequential scenes, style transfer, image blending, multimodal composition, and instruction-followed image generation.
  • Training Strategy: Fully fine-tuning a LoRA-tuned model enhances both image comprehension and generation.The comparison evaluates image captioning with CIDEr and text-to-image generation with CLIP similarity on COCO.

5 Conclusion

The paper presents SEED as a discrete image tokenizer for high-level, causally generated visual tokens and SEED-LLaMA as a multimodal model trained with next-word prediction. The reported system performs across comprehension and image generation while exhibiting compositional emergent abilities.

  • Conclusion: SEED-LLaMA unifies multimodal comprehension and image generation through large-scale pretraining and instruction tuning under next-word prediction.The approach uses interleaved visual and textual data with the SEED tokenizer.
  • Conclusion: SEED-LLaMA demonstrates remarkable performance across multimodal comprehension and image generation tasks, together with compositional emergent abilities.The conclusion identifies multi-turn and compositional behavior as part of the model’s broader capabilities.

A SEED Tokenizer

SEED aligns its generation embedding with unCLIP-SD image embeddings so the unCLIP-SD-UNet can decode causal visual codes into realistic images.

  • Generation Embedding: SEED’s generation embedding is aligned with unCLIP-SD image embeddings and decoded into realistic images by the unCLIP-SD-UNet.This design is contrasted with SEEDtext, whose generation embeddings align with SD text embeddings.

B.1 Pretraining Data

SEED-LLaMA pretraining uses diverse image-text, video-text, and interleaved image-text datasets. The appendix describes filtering, sampling, prompts, hyperparameters, and evaluation benchmarks for this data pipeline.

  • Image-text Pairs: Pretraining data includes image-text pairs from CC3M, Unsplash, LAION-COCO, and MS-COCO.Samples are filtered by image resolution, aspect ratio, and visual-textual similarity, with image or text randomly placed first.
  • Video-text Pairs: Video-text pretraining uses WebVid-10M, removes extraneous metadata, and samples four frames per video.The dataset contains videos paired with captions.
  • Interleaved Image and Text: Interleaved image-text pretraining uses MMC4-core with 7.3 million samples and the complete OBELISC dataset with 141 million samples.MMC4 documents use sequences of length 1024 with shuffled images and corresponding texts selected by CLIP score.
  • Training Configuration: The appendix reports SEED-LLaMA’s pretraining hyperparameters and supervised instruction-tuning datasets and prompt templates.Instruction-tuning materials are summarized in Tables 7 and 8; MagicBrush contributes only single-turn multimodal prompt image generation examples.

D.1 Benchmarks

SEED-LLaMA is evaluated on 10 benchmarks spanning multimodal comprehension and image generation, using task-specific evaluation procedures and prompting templates.

  • 10 benchmarks assess SEED-LLaMA’s multimodal comprehension and image-generation abilities.Image generation uses CLIP-ViT-L/14 to calculate similarity between ground-truth and generated images.
  • SEED-Bench responses select the multiple-choice option with the highest log likelihood.
  • The 8B and 14B models use different prompting templates because they adopt Vicuna-7B and Llama2-chat-13B, respectively.
  • Table 10 summarizes the prompting templates used to evaluate SEED-LLaMA.

E Qualitative Cases

Qualitative cases show SEED-LLaMA generating images and text from multimodal or textual instructions while supporting diverse multimodal comprehension behaviors. The examples include multi-turn generation, compositional and text-to-image synthesis, image understanding, and story creation.

  • Multi-turn and compositional generation: SEED-LLaMA generates multimodal responses including synthesized, sequential, and instruction-followed images.The examples include multi-turn in-context generation and compositional image generation.
  • Text-to-image generation: Text-to-image examples show realistic images aligned with varied textual descriptions.
  • Multimodal comprehension: Multimodal comprehension examples cover in-context multi-image understanding, real-world knowledge grounding, complex reasoning, story creation, and video understanding.
  • Multimodal comprehension: The examples include prompts for identifying characters, explaining actions, and telling stories from multiple images.
  • Multimodal comprehension: A qualitative response describes Tom and Jerry in a comical chase scene involving a knife, a living room, and suspenseful action.
  • Story creation: A generated story begins with Max, a lonely dog in a far-off galaxy who dreams of becoming an astronaut.
Loading 2310.01218v1…