PaperlayerView exact revision on arXiv

Source-linked AI summary

SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation

Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, Ying Shan

arXiv:2404.14396v2cs.CV

TL;DR

Existing multimodal models remain limited in handling diverse visual inputs and user instructions for real-world use. SEED-X unifies arbitrary-resolution image comprehension with multi-granularity image generation through a ViT-bridged visual tokenizer, dynamic-resolution encoding, and instruction tuning. It supports multimodal assistants across domains and achieves 51% overall accuracy on GenEval.

  • Problem

    Multimodal models remain limited in responding to diverse user instructions and interacting with visual data of varying sizes and ratios for real-world applications.

  • Method

    SEED-X combines a ViT-based visual tokenizer and multi-granularity de-tokenizer with dynamic-resolution encoding and multimodal instruction tuning.

  • Results

    After instruction tuning, SEED-X functions as multimodal assistants across domains and achieves 51% overall accuracy on GenEval.

  • Takeaways & Limitations

    SEED-X demonstrates unified multi-granularity comprehension and generation for real-world multimodal applications.

Abstract

from arXiv · show

The rapid evolution of multimodal foundation model has demonstrated significant progresses in vision-language understanding and generation, e.g., our previous work SEED-LLaMA. However, there remains a gap between its capability and the real-world applicability, primarily due to the model's limited capacity to effectively respond to various user instructions and interact with diverse visual data. In this work, we focus on bridging this gap through integrating two enhanced features: (1) comprehending images of arbitrary sizes and ratios, and (2) enabling multi-granularity image generation. We present a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks. Besides the competitive results on public benchmarks, SEED-X demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning. We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications. The models, codes, and datasets are released in https://github.com/AILab-CVC/SEED-X.

1 Introduction

SEED-X addresses real-world limitations in multimodal models by supporting arbitrary image sizes and ratios alongside multi-granularity image generation. It unifies these capabilities into a foundation model that can be instruction-tuned for diverse multimodal assistants.

  • SEED-LLaMA handled multimodal comprehension and generation, but its generated content remained insufficiently accurate and diverse for real-world needs.
  • SEED-X integrates arbitrary-size image comprehension with multi-granularity generation spanning instructional image creation and low-level image manipulation.These capabilities target downstream tasks requiring different levels of visual semantics.
  • After instruction tuning, SEED-X supports multimodal assistants for interactive design, visual question answering, slide layouts, storytelling, and image-based suggestions.The reported assistants generate texts and images across different user needs and domains.
  • The model combines a visual tokenizer and multi-granularity de-tokenizer with dynamic-resolution image encoding for image understanding and generation.The visual de-tokenizer can incorporate a condition image to retain fine-grained input details during manipulation.
  • The models, codes, and datasets are publicly released to support research on unified multi-granularity comprehension and generation.

2 Related Work

Prior unified multimodal models support comprehension and generation, but their generated content remains limited for real-world needs. SEED-X targets the missing combination of multi-granularity visual semantics and application-oriented capabilities.

  • Existing unified MLLMs use discrete visual tokenizers or continuous visual embeddings to combine multimodal comprehension and generation.Continuous approaches decode regressed visual embeddings into images during inference.
  • Despite competitive VQA and text-to-image benchmark results, prior models fall short in the accuracy and diversity needed for real-world generated content.
  • Real-world applications require multi-granularity comprehension, including object detection and dynamic-resolution encoding, plus both instructional generation and image manipulation.
  • The comparison identifies no previous work supporting all of these application-oriented characteristics within one model.

3 Method

SEED-X uses a ViT-based visual tokenizer and staged de-tokenizer to unify image comprehension, generation, and detail-preserving manipulation. Dynamic-resolution encoding handles arbitrary image layouts, while multimodal objectives train text, visual features, and grounding outputs.

  • 3.1 Visual Tokenization and De-tokenization: The visual de-tokenizer first reconstructs semantically aligned images from 64 pooled ViT embeddings, then adds a condition image to preserve low-level details.The second stage encodes the condition image into latent space and concatenates it with noisy latent features for U-Net input.
  • 3.2 Dynamic Resolution Image Encoding: Dynamic-resolution encoding divides images into sub-images, adds extrapolatable 2D positional embeddings, and enables arbitrary sizes and aspect ratios, including unseen resolutions.A resized global image is also encoded to provide global visual context.
  • 3.3 Multimodal Pre-training: SEED-X trains on next-word prediction and image-feature regression, using learnable queries to reconstruct pretrained ViT features for later de-tokenization.Bounding-box tokens add detection and referencing abilities, while the ViT tokenizer decouples MLLM and visual de-tokenizer training.
  • 3.3.2 Training Stage II: Multimodal Instruction Tuning: Pre-training uses 158M multimodal samples, and instruction tuning covers editing, text-rich data, grounded and referencing QA, slide generation, and image generation.Specialized models include SEED-X-I, SEED-X-Edit, SEED-X-PPT, SEED-X-Story, and SEED-X-Try-on.
  • 3.3.2 Training Stage II: Multimodal Instruction Tuning: Instruction-tuned SEED-X achieves competitive multimodal understanding performance and excellent instruction-following for image generation.

4 Experiments

SEED-X achieves competitive multimodal comprehension and 51% GenEval accuracy, while qualitative studies show instruction-following image manipulation and real-world multimodal assistance.

  • Multimodal Comprehension: SEED-X-I achieves competitive performance across VQAv2, GQA, POPE, MME, SEED, MMB, MM-Vet, and MMMU.The comparison includes MLLMs specifically designed for multimodal comprehension.
  • Image Generation: 51% overall accuracy on GenEval demonstrates SEED-X's instruction-following capability for compositional image generation.GenEval evaluates object cooccurrence, position, count, and color.
  • Real-World Applications: Instruction-tuned SEED-X models support interactive design and personal-assistant scenarios involving image generation, modification visualization, arbitrary-size image comprehension, and suggestions.Examples include generating images without descriptive captions and providing relevant suggestions in multi-turn conversations.
  • Image Manipulation: SEED-X-Edit adheres to editing instructions while preserving low-level details of input images, including selectively adding sunglasses to one dog.Emu2-Gen and MGIE added sunglasses to both dogs in the cited example.
  • Image Manipulation: SEED-X-Edit removes a dog while preserving background details and the baby's features, whereas competing systems fail on removal or detail retention.Gemini retrieves Internet images rather than editing them in this comparison.
  • Ablation Study: Using 256 learnable queries to reconstruct 256 visual embeddings causes generated-image distortion compared with N = 64.The passage attributes this to the greater difficulty of regressing more visual features.

5 Conclusion

SEED-X is presented as a versatile foundation model for real-world multimodal assistants by unifying arbitrary-size image comprehension with multi-granularity image generation.

  • 5 Conclusion: SEED-X integrates arbitrary-size and ratio image comprehension with high-level instructional generation and low-level image manipulation.The conclusion frames this integration as supporting application in open-world contexts.
  • 5 Conclusion: After instruction tuning, SEED-X can serve as various multimodal AI assistants in real-world scenarios.The paper presents this capability as arising from unified multi-granularity comprehension and generation.
  • 5 Conclusion: The paper aims to inspire future research on versatile multimodal foundation models in real-world scenarios.

A Pre-training and Instruction Tuning Datasets

SEED-X uses broad multimodal pre-training and specialized instruction-tuning data to support comprehension, generation, editing, grounding, and referencing tasks.

  • A Pre-training and Instruction Tuning Datasets: Multimodal pre-training combines image-caption, grounded image-text, interleaved image-text, OCR, and pure-text data.
  • A Pre-training and Instruction Tuning Datasets: LAION-COCO and SAM images are re-captioned with more detailed descriptions to improve image comprehension and generation.
  • A Pre-training and Instruction Tuning Datasets: Instruction tuning uses public VQA data plus curated text-rich, grounded, and referencing QA datasets.These datasets target text-rich image comprehension and object detection requiring reasoning.
  • A Pre-training and Instruction Tuning Datasets: Conversational datasets support open-form text output, while image-caption pairs maintain image-generation ability during instruction tuning.
  • A Pre-training and Instruction Tuning Datasets: Image-manipulation training data are included alongside specialized instruction-tuning resources.The supplied passage introduces this component but ends before detailing it.

B Implementation Details

SEED-X combines ViT-based visual tokenization, latent-space visual de-tokenization, cross-attention interfaces, dynamic positional representations, and LoRA-based instruction tuning.

  • B Implementation Details: The visual tokenizer uses Qwen-VL's visual encoder and 1D average pooling to produce N = 64 visual embeddings.These embeddings are fed through cross-attention into an SDXL-initialized U-Net.
  • B Implementation Details: The second de-tokenization stage encodes a condition image into latent space and concatenates it with the noisy latent for image editing.The U-Net input channels expand from 4 to 8 and all U-Net parameters are optimized.
  • B Implementation Details: The multimodal model uses 64 visual embeddings as LLM inputs and 64 learnable queries to reconstruct visual embeddings through cross-attention.The LLM is initialized from Llama2-chat-13B and optimized using LoRA.
  • B Implementation Details: Instruction tuning updates LoRA, input and output cross-attention layers, and extrapolatable 2D positional embeddings.Specialized fine-tuning produces models tailored for tasks including image editing.

C Qualitative Examples

SEED-X-I demonstrates multimodal generation and comprehension across both high-level semantic alignment and fine-grained visual fidelity. Qualitative examples show stronger instruction adherence, detail preservation, and broad visual understanding.

  • Text-to-image Generation: SEED-X-I generates images that align closely with descriptive captions while exhibiting artistic qualities.Examples include accurately depicting a person in a small boat, a sword on its back, oriental landscapes, and vividly colored tigers.
  • Text-to-image Generation: The qualitative comparison includes SEED-X-I, SEED-LLaMA-I, Emu2-Gen, and Gemini for text-to-image generation.The comparison is visualized in Figure 9.
  • Image Manipulation: SEED-X-Edit follows image-editing instructions while preserving low-level details of the input image.It adds sunglasses to only the specified dog and removes a dog while retaining background details and the baby’s features.
  • Multimodal Comprehension: SEED-X-I supports fine-grained object detection, text-rich comprehension, mathematical computation, world knowledge, commonsense reasoning, and diagram understanding.These capabilities are presented as qualitative examples in Figures 10 and 11.
Loading 2404.14396v2…