PaperlayerView exact revision on arXiv

Source-linked AI summary

CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation

Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, Mohit Bansal

arXiv:2311.18775v1cs.CVcs.AIcs.CLcs.LGcs.SDeess.AS

TL;DR

Existing multimodal generative models face challenges with sophisticated zero-shot control, reasoning, multi-round interaction, and interleaved inputs. CoDi-2 uses an LLM with aligned multimodal inputs and diffusion-based generation, and reports capabilities across diverse zero-shot and few-shot tasks. The paper presents it as an exploration toward a GPT-like fundamental multimodal system.

  • Problem

    Existing multimodal generative models struggle with sophisticated zero-shot control and reasoning, consistent multi-round interaction, and inputs interleaving modalities.

    Source: §1 · p. 1

  • Method

    CoDi-2 aligns multimodal inputs with language for LLM processing and autoregressively predicts output features that are passed to diffusion models.

    Source: §1 · p. 2

  • Results

    CoDi-2 reports strong zero-shot and few-shot capabilities across diverse multimodal tasks, integrating in-context learning into interleaved, interactive any-to-any generation.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The authors present CoDi-2 as an exploration toward a GPT-like fundamental multimodal system for complex inputs and multimodal generation.

    Source: §6 · p. 9

Abstract

from arXiv · show

We present CoDi-2, a versatile and interactive Multimodal Large Language Model (MLLM) that can follow complex multimodal interleaved instructions, conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any input-output modality paradigm. By aligning modalities with language for both encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not only understand complex modality-interleaved instructions and in-context examples, but also autoregressively generate grounded and coherent multimodal outputs in the continuous feature space. To train CoDi-2, we build a large-scale generation dataset encompassing in-context multimodal instructions across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot capabilities for multimodal generation, such as in-context learning, reasoning, and compositionality of any-to-any modality generation through multi-round interactive conversation. CoDi-2 surpasses previous domain-specific models on tasks such as subject-driven image generation, vision transformation, and audio editing. CoDi-2 signifies a substantial breakthrough in developing a comprehensive multimodal foundation model adept at interpreting in-context language-vision-audio interleaved instructions and producing multimodal outputs.

1. Introduction

CoDi-2 targets multimodal generation that can follow interleaved instructions and in-context examples across multiple conversational turns. It combines an LLM-based approach with multimodal training data and reports capabilities across a broad range of generation tasks.

1. Introduction

  • Existing multimodal generative models struggle with sophisticated zero-shot control, reasoning, consistent multi-round interaction, and modality-interleaved inputs.Examples include transferring editing effects by analogy and subject-driven generation without fine-tuning.Source: §1 · p. 1
  • CoDi-2 uses an LLM to interpret interleaved multimodal instructions and in-context inputs, then autoregressively predicts output features for synchronized diffusion models.Diffusion-model generation loss also back-propagates to the LLM, which the paper says can enhance perceptual faithfulness to image or audio inputs.Source: §1 · p. 2
  • The authors convert instructional datasets across vision, audio, and language into in-context generation data and build additional multimodal learning data from text descriptions.Aligned encoders let textual descriptions, such as image captions, represent multimodal components in text-only datasets.Source: §1 · p. 2
  • Across diverse multimodal tasks, CoDi-2 shows strong zero-shot and few-shot prompting capabilities and integrates in-context learning into interleaved, interactive any-to-any generation.The reported tasks include audio fusion and editing, intricate image composition, in-context exemplars, reasoning, and video understanding and generation.Source: §1 · pp. 2–3

2. Related Work

Related work spans multimodal large language models, multimodal in-context understanding, and image-generation methods. The paper describes progress in these areas alongside limits in existing in-context generation and generalization.

2.1. Multimodal Large Language Models

  • Multimodal large language models extend language models to process multimodal inputs and generate multifaceted outputs, while related work also uses LLMs to ground image generation.The cited work includes multimodal understanding and generation across visual and other sensory data.Source: §2.1 · p. 3

2.2. Multimodal In-Context

  • Multimodal in-context understanding can involve interleaved images and text, but generating raw sensory outputs remains challenging and existing approaches are constrained by training regimes.The passage describes in-context image-generation techniques as nascent and lacking genuine in-context learning capabilities.Source: §2.2 · p. 3

2.3. Multimodal Generation

  • Image-generation research includes editing, exemplar learning, composition, and concept learning, but subject-driven methods often require task- or subject-specific tuning.Kosmos-G pursues zero-shot generation with interleaved image and text but is limited to image composition; the paper frames its work as aiming to transcend such limits.Source: §2.3 · p. 3

3. Model Architecture

CoDi-2 processes interleaved text, image, and audio inputs with a multimodal LLM, then uses diffusion models to generate multimodal outputs. Its training combines feature, diffusion, and token-prediction losses.

3. Model Architecture

  • CoDi-2 is designed to process text, image, and audio inputs with instructions and generate corresponding text, image, or audio outputs.The architecture overview describes audio and vision encoders and decoders alongside an LLM.Source: §3 · p. 3

3.1. Multimodal LLM as the Fundamental Engine

  • Aligned multimodal encoders convert input modalities into feature sequences bracketed by special tokens, allowing an LLM to process interleaved inputs such as text with image and audio features.The paper motivates the LLM as an engine for instruction following, chatting, and zero-shot learning.Source: §3.1 · p. 4

3.2. Multimodal Generation with MLLM

  • For multimodal generation, the MLLM predicts a conditional feature for a diffusion model, and training combines feature matching, diffusion loss, and text-token prediction.The stated loss includes mean squared error between the MLLM output feature and C_x(x), the diffusion loss, and text-token loss, with the MSE weighted by α.Source: §3.2 · p. 4

4. Building Diverse Multimodal In-Context Generation Data

The section assembles paired, instructional, and repurposed datasets for multimodal generation, then organizes tasks by zero-shot and example-based prompting.

4.1. Dataset Construction

  • MIMIC-IT adds 2.8 million text-only instruction-response pairs to support in-context understanding of multimodal inputs.Its examples include visual difference detection and predicting what happens next in a football video.Source: §4.1 · p. 4
  • Paired data spans image-text, audio, and video, supporting both caption generation from media and media generation from captions.The sources include LAION-400M, AudioSet, and WebVid.Source: §4.1 · p. 4
  • Audio editing data covers addition, removal, and replacement, with each version built on AudioSet and together totaling three times its size.Addition overlays two clips; removal reverses that construction, while replacement substitutes one sound segment for another.Source: §4.1 · p. 6
  • In-context editing prompts show a transformation between two images, then ask the model to apply it to a third; the template also adapts to audio.The target is a fourth image or, for audio, a corresponding edited output.Source: §4.1 · p. 6
  • The dataset also repurposes text-only instruction data by replacing selected words with text-encoder features to create interleaved inputs.The example converts phrases in an Alpaca sentence into text features while retaining the surrounding text.Source: §4.1 · p. 6

4.2. In-Context Instruction Task Types

  • Prompting tasks range from generating or transforming content without examples to learning from one or a few examples.Zero-shot examples include image style transfer and audio editing; one- or few-shot prompts provide examples before a similar task.Source: §4.2 · pp. 6–8
  • Exemplar learning provides an example of the desired output, while concept learning transfers shared attributes such as style or pattern to new content.Examples include applying a demonstrated seasonal transformation and reproducing floral patterns in a new image.Source: §4.2 · p. 8
  • Subject-driven learning generates new content from provided images while preserving the subject’s distinct features across changed poses, lighting, or context.The examples extend from generating new images of a cat to creating a video with sound featuring the same cat.Source: §4.2 · p. 8

5. Experiments

The experiments evaluate image and audio generation and describe a setup that feeds generated features to diffusion models, with original inputs added when greater fidelity is needed.

5.1. Model Setups

  • The implementation uses Llama-2-7b-chat-hf and ImageBind encoders, with diffusion models generating images, audio, or video from projected features.Image and audio features are projected into the LLM’s input dimension and generated features are projected back to ImageBind’s feature dimension.Source: §5.1 · p. 8
  • For higher fidelity to an original image or audio input, the setup also feeds that input to the diffusion model by concatenating diffusion noise.The passage says this approach is particularly effective at preserving perceptual features during edits such as adding content or changing style.Source: §5.1 · pp. 8–9

5.2. Image Generation Evaluation

  • CoDi-2 shows competitive zero-shot performance on subject-driven image generation and MSCOCO FID evaluation, indicating generalization to unseen tasks.Source: §5.2 · p. 9

5.3. Audio Generation Evaluation

  • CoDi-2 achieves lower LSD, KL divergence, and FD scores than previous methods across audio addition, dropping, and replacement tasks.The passage reports the lowest scores across all three metrics for all three editing tasks.Source: §5.3 · p. 9

6. Conclusion

CoDi-2 combines modality-interleaved instruction following, in-context generation, and multi-round interaction, producing latent features for diffusion-based multimodal generation.

6. Conclusion

  • CoDi-2 autoregressively produces latent features for diffusers after processing complex modality-interleaved inputs and instructions.The conclusion reports zero- and few-shot abilities in style adaptation, subject-driven generation, and cross-modal editing.Source: §6 · p. 9

A. Appendix Overview

The appendix expands on the paper’s training methodologies, model details, experimental setups, and dataset construction.

A. Appendix Overview

  • The supplementary section provides further detail on training, model and experiment setups, and dataset construction for multimodal in-context generation scenarios.Source: A. Appendix Overview · p. 11

B.1. Model Setups

CoDi-2 uses negative prompts as unconditioned input guidance for image generation and generated features, with prompts tailored to image and audio quality.

B.1. Model Setups

  • Image-generation negative prompts target defects including low quality, blur, watermarks, cropping, and visual distortions.The list also includes artifacts such as logos, signatures, and extra digits.Source: §B.1 · p. 11
  • Audio negative prompts target degraded sound, including noise, muffling, low volume, distortion, and interruptions.The list also covers off-key, repetitive, overlapping, and incoherent audio.Source: §B.1 · p. 11
  • These negative prompts function as unconditioned input guidance to enhance the quality of images and generated features.For image generation, MLLM-generated features also provide embedding guidance.Source: §B.1 · p. 11

B.2. Training Pipelines

Training alternates text, audio, and image generation phases, while prompt templates diversify multimodal and in-context generation tasks.

B.2. Training Pipelines

  • Training alternates text, audio, and image generation phases to avoid the cumbersome and inefficient model I/O of simultaneous generation.Fine-tuning uses rank-128 LoRA and updates LoRA weights, modality-to-LLM projection layers, and the decoder projection into diffusion input space.Source: §B.2 · p. 11

C. Extended Details of Multimodal In-Context

C.1. GPT-Generated Prompts

  • Each task type has 100 uniquely generated prompt prototypes, sampled uniformly during training.Task types include instructional editing, paired multimodal datasets, exemplar learning, and image composition.Source: §C.1 · p. 11
  • Prompt examples ask the model to learn an image transformation from exemplars and apply it to another image, or compose an image from two inputs.Other examples cover text-to-image generation and image or audio captioning.Source: §C.1 · p. 11

C.2. Addressing the Discrepancy Between Datasets and Practical Applications

The authors report that some applications remain underrepresented in their training datasets, even as the model excels at visual concept learning.

C.2. Addressing the Discrepancy Between Datasets and Practical Applications

  • Some applications remain insufficiently represented in the training datasets, including visual concept learning, where the model is reported to excel.The authors describe gathering or integrating datasets from many sources.Source: §C.2 · p. 11
  • The model architecture and task design aim to leverage the in-context capabilities of large language models.The authors pair this approach with diverse, interleaved in-context datasets from open-domain materials.Source: §C.2 · p. 11
  • The authors say these datasets complement the model design and enhance applicability across a broader range of real-world scenarios.Source: §C.2 · p. 11
Loading 2311.18775v1…