Source-linked AI summary
Generative Multimodal Models are In-Context Learners
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, Xinlong Wang
TL;DR
Current multimodal systems struggle to learn new tasks from demonstrations or simple instructions, motivating a more general in-context learner. The paper introduces 37-billion-parameter Emu2, trained with unified autoregressive prediction over multimodal sequences. Emu2 shows strong few-shot and instruction-tuned performance across understanding and grounded generation, while still failing on some complex scenes and trailing closed systems on question answering.
Problem
Current multimodal systems largely struggle to solve diverse new tasks from only a few demonstrations or simple instructions.
Method
Emu2 is a 37-billion-parameter generative multimodal model trained on large-scale multimodal sequences with a unified predict-the-next-element objective.
Results
Emu2 achieves strong few-shot multimodal understanding and state-of-the-art instruction-tuned results for visual question answering and grounded visual generation.
Takeaways & Limitations
Emu2 can serve as a base model and general-purpose interface for diverse multimodal understanding and generation tasks.
Takeaways & Limitations
Emu2 can fail on complex scenes or tasks, and its question-answering performance remains below closed multimodal systems.
Abstract
from arXiv · showhide
The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.
1. Introduction
Emu2 addresses the difficulty of generalizing across diverse multimodal tasks by scaling a generative model trained on multimodal sequences. It demonstrates in-context learning across few-shot understanding, visual prompting, and instruction-following generation, while motivating responsible deployment.
- Motivation and contribution: Previous multimodal systems commonly require task-specific architectures and sizable supervised datasets, whereas humans can solve new tasks from demonstrations or simple instructions.This repeated per-task scaling challenge motivates a general multimodal learner.
- Motivation and contribution: 37 billion parameters enable Emu2 to learn task-agnostic multimodal in-context abilities through a unified autoregressive objective.The model predicts either visual embeddings or textual tokens from large-scale multimodal sequences.
- In-context evaluation: Emu2 improves with more in-context examples and achieves state-of-the-art few-shot performance on multiple visual question-answering datasets.It is evaluated in both few-shot and instruction-tuning scenarios.
- In-context evaluation: In context, Emu2 can perform visual prompting and on-the-fly tasks such as recognition and counting, although it struggles at smaller scale or in zero-shot settings.Examples include following circles placed on images.
- Instruction-following adaptation: Instruction-tuned Emu2 achieves state-of-the-art visual question answering and generates images grounded in mixtures of text, locations, and images.The model can be adapted into a controllable visual generation system.
- Broader impact: The authors analyze societal implications and misuse concerns while emphasizing continued enhancement and responsible deployment.These concerns accompany the model’s broad capabilities.
2. Approach
Emu2 combines visual encoding, multimodal autoregressive modeling, and visual decoding in a unified architecture, then adapts the base model for instruction following and controllable generation. Its approach uses heterogeneous multimodal data and separate decoders for image and video outputs.
- Model architecture: Emu2 tokenizes images into visual embeddings, interleaves them with text tokens, predicts the next multimodal element, and decodes visual embeddings into images or videos.The architecture comprises a Visual Encoder, Multimodal Modeling, and Visual Decoder.
- Model architecture: The model uses a simpler connection between the Visual Encoder and Multimodal Modeling, mean-pooling images into 8 × 8 patches followed by linear projection.This replaces an additional C-Former used in Emu.
- Pretraining: Pretraining combines image-text pairs, video-text pairs, interleaved image-text and video-text data, grounded pairs, and language-only data.The data sources include LAION-2B, CapsFusion, WebVid-10M, MMC4, YT-Storyboard-1B, GRIT-20M, and Pile.
- Visual decoding: A separately trained visual decoder acts as a detokenizer, allowing visual embeddings from the encoder to be converted back into images without autoregressive language-model inference during decoder training.Together, the Visual Encoder and Visual Decoder form an image autoencoder.
- Visual decoding: The video decoder adapts a 2D denoising U-Net with temporal convolutions and spatial-temporal attention to generate clips conditioned on visual embeddings.It is initialized from Stable Diffusion 2.1 and fully trained for video generation.
- Instruction-following adaptation: Instruction tuning produces Emu2-Chat for multimodal dialogue and Emu2-Gen for generation grounded in text, locations, and images.Chat training supervises only answer tokens using academic-task and multimodal chat data.
3. Evaluation
Emu2 and Emu2-Chat are evaluated across few-shot multimodal understanding, instruction-following, grounding, and controllable generation tasks. Results show strong performance across these settings, including state-of-the-art or superior results on several benchmarks.
- Pretrained Base Model: Emu2 outperforms Flamingo-80B and IDEFICS-80B across all few-shot settings on VQAv2, VizWiz, and TextVQA despite its smaller 37B scale.Performance improves as more in-context samples are provided.
- Pretrained Base Model: Emu2 learns classification, counting, and visual prompting from a few examples in context.Examples include simultaneous classification and counting in a specified format and following red circles over images.
- Instruction-Following Chat: Emu2-Chat consistently outperforms other models on image question-answering benchmarks, including VQAv2 and GQA.It also achieves 49.0 on MSVD-QA and 31.4 on MSRVTT-QA without video question-answering training data.
- Instruction-Following Chat: Emu2-Chat achieves 48.5 on MM-Vet and 703.8 on TouchStone, outperforming on large multimodal model benchmarks.These benchmarks evaluate more comprehensive and complicated multimodal abilities.
- Instruction-Following Chat: Emu2-Chat achieves the best results among generalist models on RefCOCO, RefCOCO+, and RefCOCOg.Its strongest advantage is on RefCOCO+, which relies on appearance-based descriptions without position references.
- Controllable Visual Generation: Emu2-Gen accepts text, locations, and images to generate images in context across grounding, stylization, editing, and multi-entity composition tasks.The evaluation includes zero-shot controllable generation and subject fidelity comparisons using DINO and CLIP-I.
4. Related Work
Prior multimodal systems combine visual inputs with language models or use specialized training pipelines for multimodal understanding. In-context learning has emerged in large language models and has been extended to visual-linguistic tasks.
- Large Multimodal Models: CLIP pioneered large multimodal models through contrastive learning on massive image-text pair datasets.
- Large Multimodal Models: Flamingo and Kosmos demonstrate zero-shot and few-shot multimodal understanding by training on large-scale image-text interleaved data.
- Large Multimodal Models: Open-source language models have enabled multimodal systems that connect vision encoders and language models through small intermediate models.
- In-Context Learning: Scaling language-model size and data has been associated with stronger in-context learning and emergent abilities such as mathematical reasoning.
- In-Context Learning: Flamingo extends in-context learning to visual-linguistic tasks such as image captioning and OCR through language-based modeling.
5. Conclusion
Emu2 is presented as a versatile generative multimodal model for in-context understanding and generation. Its results support using scaled generative multimodal models as adaptable interfaces for diverse tasks.
- Conclusion: Emu2 is a 37-billion-parameter generative multimodal model with strong performance and versatility across in-context multimodal tasks.
- Conclusion: Emu2 achieves state-of-the-art results across a broad range of multimodal understanding and generation benchmarks.
- Conclusion: Emu2 provides controllable visual generation in multimodal context, including subject-grounded and text-grounded generation.
- Conclusion: Emu2 can serve as a base model and general-purpose interface for varied multimodal tasks.
A. Broader Impact and Limitations
Emu2’s in-context and generative capabilities may support applications including accessibility and creative work. The paper also identifies risks from hallucination, harmful or biased outputs, and failures on complex tasks.
- Broader Impact: Emu2’s in-context learning can help it adapt quickly to new tasks or environments with limited data.
- Broader Impact: Emu2’s generative capabilities may be valuable to creative industries.
- Limitations: Multimodal-model hallucinations may produce incorrect or unreasonable predictions in some cases.
- Limitations: Emu2 may generate harmful or biased content because generative-model training data may be biased or unsuitable.
- Limitations: Emu2’s in-context learning can fail in complex scenes or tasks, such as counting in a crowd.
- Limitations: Emu2 remains behind closed multimodal systems on question answering, with 48.5 on MM-Vet versus GPT-4V’s 67.7.
B.1. Dataset Details
Emu2 pretraining combines image-text, video-text, interleaved multimodal, grounded image-text, and language-only data across two stages. The datasets are organized into unified sequences with visual and textual elements plus boundary tokens.
- Pretraining Data: Stage 1 uses image-text pairs and video-text pairs, while stage 2 adds interleaved and language-only data.The two-stage setup broadens the multimodal sequence sources used during pretraining.
- Sequence Construction: Unified multimodal sequences interleave visual embeddings with text tokens and use <s> and </s> to mark sequence boundaries.Training hyperparameters for pretraining and visual decoding are summarized in Tables 6 and 7.
- Pretraining Data: Image-text pairs come from LAION-2B and CapsFusion-120M, while video-text pairs come from WebVid-10M.During stage 2, each image or video is randomly placed before or after its corresponding text.
- Interleaved Data: Interleaved image/video-text data comes from Multimodal-C4 and YT-Storyboard-1B to support multimodal in-context learning.For each Multimodal-C4 document, N = 8 images and corresponding sentences form a subsequence of L = 1024.
- Grounded Data: Grounded image-text data comprises CapsFusion-grounded-100M and GRIT, with phrases randomly positioned before or after corresponding coordinates.The positioning probability is 0.7.
- Language Data: Pile supplies language-only data to maintain text reasoning capabilities, with 3.6 billion tokens sampled for pretraining.Each sample is tokenized into 2048 tokens using the LLaMA tokenizer.
D.1. Dataset Details
Instruction fine-tuning uses academic task datasets, multimodal chat data, grounded generation and image-editing data, and high-quality image-text pairs. Separate hyperparameter summaries describe Emu2-Chat and Emu2-Gen training.
- Emu2-Gen Data: Grounded image-text pairs support grounded text-to-image generation, while InstructPix2Pix data supports image editing.The editing samples interleave an origin-image embedding, an editing prompt, and an edited-image embedding.
- Emu2-Gen Data: Text-to-image training uses filtered data from CapsFusion, LAION-Aesthetics, SA-1B, and LAION-High-Resolution.SAM provides segmentation results for corresponding grounding boxes.
- Emu2-Gen Data: High-quality fine-tuning uses approximately 500,000 image-text pairs sourced from Unsplash, Midjourney-V5, and DALL-E-3 outputs.The dataset is intended to refine the aesthetic quality of Emu2-Gen’s generated images.
- Training Settings: Training hyperparameters for Emu2-Chat and Emu2-Gen are summarized in Tables 8 and 9.
E. Evaluation Details
Evaluation covers few-shot base-model tasks, instruction-following chat, and controllable visual generation. The protocols specify example selection, prompting, decoding, video sampling, and DreamBench setup.
- Pretrained Base Model: Few-shot Emu2 evaluation selects examples with Retrieval In-Context Example Selection and places them before the test sample.The evaluation uses the prompt “[image] based on the picture, [question] short answer:”.
- Pretrained Base Model: Benchmark splits and metrics are detailed in Table 10, whose summary covers the evaluation benchmarks.
- Instruction-Following Chat: Emu2-Chat evaluation uses beam size 5 and uniformly samples 16 frames for video inputs.Short-answer benchmarks additionally use a concise-assistant system message and output-format controls.
- Controllable Visual Generation: DreamBench evaluation uses the same selected image for each object as Kosmos-G and modifies prompts with an object-placeholder instruction format.For example, “a red {}” becomes “{} Make it red”.
F. Qualitative Results
The qualitative-results section presents visual examples spanning Emu2-Gen and Emu2-Chat. The figures cover image autoencoding, text-to-image generation, controllable generation, video generation, and subject-driven generation.
- Emu2-Chat: Figures 12–14 provide qualitative cases of Emu2-Chat.The paper groups these figures separately from the Emu2-Gen cases.
- Emu2-Gen: Figure 10 illustrates zero-shot video generation using an interleaved vision-language prompt.
- Emu2-Gen: Figure 11 shows zero-shot subject-driven generation results on DreamBench.