PaperlayerView exact revision on arXiv

Source-linked AI summary

Emu: Generative Pretraining in Multimodality

Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, Xinlong Wang

arXiv:2307.05222v2cs.CV

TL;DR

Existing multimodal models often train mainly on text prediction and overlook scalable interleaved video-text data. Emu unifies visual and textual autoregressive prediction over diverse multimodal sequences, achieving strong performance across image and video tasks while supporting generalist multimodal interaction.

  • Problem

    Existing LMMs usually predict only text with frozen vision encoders and are trained mainly on image-text data, while scalable interleaved video data is underused.

  • Method

    Emu encodes visual signals into causal embeddings, interleaves them with text tokens, and trains end-to-end to classify the next text token or regress the next visual embedding.

  • Results

    Emu performs strongly across zero-shot and few-shot captioning, image and video question answering, and text-to-image generation, with Emu-I outperforming larger Flamingo-80B on several tasks.

  • Takeaways & Limitations

    The unified model acts as a generalist multimodal interface for image-to-text, text-to-image, in-context generation, image blending, and multimodal assistance.

  • Takeaways & Limitations

    Emu remains susceptible to hallucinations and slow autoregressive inference, and its predominantly English training makes performance in other languages delicate.

Abstract

from arXiv · show

We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu can serve as a generalist multimodal interface for both image-to-text and text-to-image tasks, and supports in-context image and text generation. Across a broad range of zero-shot/few-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to state-of-the-art large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.

1 INTRODUCTION

Emu addresses limitations in existing multimodal models by jointly modeling visual and textual elements from diverse interleaved data. It supports broad multimodal completion and is evaluated across understanding, generation, and assistant tasks.

  • Existing LMMs typically predict only text tokens with frozen vision encoders, limiting visual supervision and model capacity.
  • Emu unifies image, video, and text training by predicting the next visual or text element autoregressively.
  • Visual embeddings and text tokens form interleaved sequences that allow Emu to learn from videos, webpages, image-text pairs, and other web-scale sources.
  • Emu serves as a generalist interface for image-to-text and text-to-image completion, including in-context generation, image blending, video understanding, and knowledge grounding.
  • Evaluation covers zero-shot and few-shot image captioning, visual question answering, video question answering, and text-to-image generation, alongside an instruction-tuned multimodal assistant.

2 EMU: PREDICT THE NEXT IN MULTIMODALITY

Emu converts visual inputs into causal embeddings and models them with text in a unified autoregressive sequence. Separate heads predict text tokens or visual embeddings, which can be decoded into images.

  • 2.1 ARCHITECTURE: Emu comprises a Visual Encoder, Causal Transformer, Multimodal Modeling component, and Visual Decoder for multimodal completion.
  • 2.1 ARCHITECTURE: Images are encoded into dense visual features, transformed into N causal embeddings, and interleaved with text tokens; videos produce T × N embeddings for T frames.
  • 2.1 ARCHITECTURE: The Causal Transformer converts 2D visual signals into 1D latent causal sequences, avoiding direct raster-order autoregressive image modeling.
  • 2.2 TRAINING OBJECTIVE: The training corpus contains interleaved multimodal sequences, whose elements are either discrete text tokens or visual embeddings with causal dependencies.
  • 2.2 TRAINING OBJECTIVE: Text prediction uses cross-entropy classification, while visual prediction uses ℓ2 regression; generated visual embeddings are decoded into images by a latent diffusion model.
  • 2.2 TRAINING OBJECTIVE: The unified objective enables Emu to accept multimodal prompts and complete tasks across modalities, generating text tokens or N visual embeddings as required.

3 EMU TRAINING

Emu is pretrained on large-scale image-text, interleaved document, video-text, and interleaved video-text data represented as unified multimodal sequences. Its training combines pretrained encoders and language modeling components with end-to-end multimodal learning.

  • 3.1 DATA: Pretraining uses image-text pairs, interleaved image-text data, video-text pairs, and interleaved video-text data under a unified next-element objective.
  • 3.1 DATA: WebVid-10M supplies short videos with textual descriptions sourced from materials websites with diverse content.
  • 3.1 DATA: MMC4 contributes approximately 75 million image-text-interleaved documents containing 400 million images and 38 billion tokens.
  • 3.1 DATA: The interleaved video-text YT-Storyboard-1B dataset contains 18 million videos, about 1.8 billion storyboard images, and subtitles ordered by timestamps.
  • 3.2 PRETRAINING: Emu initializes its visual encoder with EVA-01-CLIP and its multimodal language model with 13B LLaMA, while the Causal Transformer is randomly initialized.
  • 3.2 PRETRAINING: The model is trained on 128 NVIDIA 80G-A100 GPUs for 10k steps and approximately 82M samples, totaling 150B tokens.
  • 3.2 PRETRAINING: After pretraining, only the U-Net of a Stable Diffusion v1.5 decoder is updated to transform Emu’s visual embeddings into images.

4 INSTRUCTION TUNING

Instruction tuning adapts Emu to follow multimodal user instructions through supervised conversational data while preserving the pretrained model parameters. The evaluation setup distinguishes the instruction-tuned Emu-I model and text-only prompting comparisons.

  • 4 INSTRUCTION TUNING: Multimodal instruction tuning uses publicly available conversational and language-instruction datasets to align Emu with human instructions.
  • 4 INSTRUCTION TUNING: All pretrained Emu parameters are frozen, and LoRA modules are added only to the Multimodal Modeling LLM’s self-attention layers.
  • 4 INSTRUCTION TUNING: Table 1 distinguishes Emu-I as the instruction-tuned model and marks text-only-example prompts with an asterisk.
  • 4 INSTRUCTION TUNING: The instruction-tuning template contains system, user, and assistant slots, with loss computed only on the assistant answer.

5 EVALUATION

Emu is evaluated across zero-shot, few-shot, in-the-wild, and qualitative multimodal settings, showing strong understanding, generation, in-context learning, and instruction-following capabilities.

  • Emu is evaluated on image captioning, image and video question answering, visual dialog, and text-to-image generation in zero-shot and few-shot settings.
  • Zero-shot evaluation: 112.4 CIDEr on COCO captioning surpasses other large multimodal models by a large margin.Emu also reaches 34.4% on VizWiz VQA, compared with 29.2% for Kosmos-1 and 28.8% for Flamingo-9B.
  • Zero-shot evaluation: Emu-I outperforms Flamingo-80B on VQAv2, VizWiz, and MSVDQA despite having 14B parameters.The reported comparisons are 62.0% vs. 56.3%, 38.3% vs. 31.6%, and 37.0% vs. 35.6%, respectively.
  • Few-shot evaluation: In four-shot evaluation, Emu reaches 58.4% VQAv2 accuracy, 41.3% VizWiz accuracy, and 21.8% MSRVTTQA accuracy, exceeding Flamingo on each cited comparison.Performance improves as the number of in-context examples increases from k = 0 to 8.
  • In-the-wild evaluation: Emu-I shows state-of-the-art in-the-wild capability on MM-Vet and exceeds LLaVA-65B in Rec, Know, Gen, and total score.
  • Qualitative evaluation: Qualitative evaluations demonstrate knowledge grounding, interleaved multi-image and video understanding, multimodal assistance, multi-turn dialogue, image blending, and in-context text-to-image generation.The paper attributes the new context-related image-generation ability to multimodal in-context capability.

6 RELATED WORK

Prior multimodal pretraining learns cross-modal interactions from large-scale data, while Flamingo established notable zero-shot and few-shot multimodal behavior by connecting pretrained vision and language models.

  • Multimodal pretraining learns cross-modal interactions from large-scale multimodal data.
  • Flamingo bridges pretrained vision and language models and demonstrates remarkable multimodal zero-shot and few-shot behaviors.

7 LIMITATIONS AND FUTURE TOPICS

Emu retains several limitations shared by large language and multimodal models, including hallucination risk, slow autoregressive inference, stale pretrained knowledge, non-factual outputs, and weak multilingual proficiency.

  • Emu remains susceptible to visual and language hallucinations and may generate non-factual content.
  • Autoregressive inference is slow, and the model stops receiving knowledge updates after pretraining.
  • Because Emu predominantly uses English-language data, its proficiency in other languages is currently delicate.The paper identifies hallucination reduction, faster inference, and expanded multilingual capability as future priorities.

8 CONCLUSION

Emu unifies visual and textual next-element prediction across diverse multimodal data, enabling broad multimodal task completion and instruction-tuned interaction.

  • Emu predicts the next visual or textual token with a unified autoregressive objective across image-text, video-text, and interleaved multimodal data.
  • The model supports image captioning, image and video question answering, text-to-image generation, in-context generation, and image blending.
  • An instruction-tuned Emu assistant exhibits human-aligned abilities such as multi-turn dialogue.
  • The work motivates further exploration of diverse web-scale multimodal data and generative pretraining beyond vision and language.

ETHICS STATEMENTS

The paper frames Emu as a preliminary research model with unresolved safety, privacy, bias, deployment, and data-curation concerns.

  • Emu is intended solely for preliminary research and should not be deployed without comprehensive risk analysis and mitigation strategies.
  • Its initialization and Internet-sourced pretraining data may propagate harmful, biased, inappropriate, or privacy-sensitive content.
  • Production deployment requires further investigation of model behavior and potential biases, while widespread dataset curation and model development exceed one paper's scope.
  • Suggested mitigation approaches include prompting, filtering, human oversight, evaluation, discriminator models, policy, product design, and improved ethics benchmarks.
  • Privacy-preserving face obfuscation or generation is identified as a possible direction but remains beyond the paper's scope.
  • The paper reiterates that deployment should be deferred until the identified issues are thoroughly considered and addressed.

A.1.3 ABLATIONS ON INTERLEAVED VIDEO-TEXT DATA

The ablation studies examine whether interleaved video-text pretraining data improves Emu's multimodal performance and in-context learning.

  • The ablation compares Emu models trained with and without the YT-Storyboard-1B dataset using reduced model and training settings.
  • The pretraining and visual-decoder experiments use separately summarized hyperparameter tables.
  • Table 6 reports quantitative comparisons between pretraining with and without interleaved video and text data, including zero-shot and in-context settings.
  • With YT-Storyboard-1B, Emu-7B achieves better zero-shot performance on MS-COCO, MSVDQA, and MSRVTTQA.
  • YT-Storyboard-1B also strengthens in-context learning under 4-shot evaluation.

B INSTRUCTION TUNING

Emu is instruction-tuned on language, image, and video instruction datasets to improve multimodal interaction and instruction following.

  • B.1 DATASET AND TRAINING DETAILS: Instruction tuning collects publicly available language, image, and video instruction datasets.
  • B.1 DATASET AND TRAINING DETAILS: Language instruction data includes about 70K ShareGPT dialogues and 52K Alpaca instruction-following examples.
  • B.1 DATASET AND TRAINING DETAILS: LLaVA supplies 158K image-text instruction samples spanning conversation, detailed description, and complex reasoning, with detailed descriptions removed during preliminary experiments.
  • B.1 DATASET AND TRAINING DETAILS: Video instruction data combines VideoChat-11K with an approximately 30K-sample subset of Video-ChatGPT-100k containing videos under one minute.
  • B.1 DATASET AND TRAINING DETAILS: Training uses 128 batch size, 10K steps, cosine learning-rate decay, and LoRA adapters on self-attention projections.
  • B.2 SYSTEM MESSAGES: Different system messages are used for language-, image-, and video-instruction datasets.
  • B.2 SYSTEM MESSAGES: The prompting templates are summarized in Table 8, while evaluation benchmarks are summarized in Table 9.

C EVALUATION

Emu is evaluated across diverse multimodal tasks using task-specific prompting, few-shot retrieval, and multimodal chain-of-thought strategies. It achieves strong benchmark performance, including state-of-the-art results on 5 of 6 evaluated benchmarks and qualitative multimodal capabilities.

  • Evaluation setup: Emu is evaluated on nine benchmarks spanning text-to-image generation, image and video question answering, and image-based visual dialogue.The benchmark suite covers multimodal image/video and language tasks.
  • Evaluation setup: Task-specific templates structure Emu and Emu-I outputs, while post-processing removes common redundant phrases and handles unanswerable VizWiz questions.For VizWiz, the model is prompted to determine whether an answer is known before producing an answer.
  • Evaluation setup: Multimodal Chain-of-Thought prompting first elicits an image or video caption, then supplies that caption with a task prompt to generate the final result.This test-time procedure is used to enhance the pretrained model’s capabilities.
  • Evaluation setup: Few-shot evaluation uses retrieved examples selected by RICES, which chooses visually similar training samples using EVA-CLIP features.For videos, the method compares mean frame features and limits support examples to two frames to respect context length.
  • Results: Emu achieves state-of-the-art results on 5 out of 6 evaluated benchmarks.This result is reported in Table 11’s zero-shot comparison with concurrent work.
  • Qualitative evaluation: Emu supports qualitative capabilities including interleaved multi-image understanding, real-world knowledge grounding, detailed video understanding, multimodal assistance, multi-turn dialogue, image blending, and text-to-image generation.The cited figures provide examples of these capabilities and generation settings.
Loading 2307.05222v2…