PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu

arXiv:2412.14164v1cs.CV

TL;DR

Existing multimodal LLMs primarily understand images but do not efficiently unify visual understanding and generation. The paper introduces VPiT, which trains a pretrained LLM to autoregressively predict text and continuous visual tokens from instruction-formatted multimodal sequences. Experiments show that the capabilities are mutually beneficial but asymmetric, and MetaMorph achieves competitive performance while using pretrained-LLM knowledge and reasoning for visual generation.

  • Problem

    Unified models for visual understanding and generation often require substantial architectural changes and multimodal pretraining or finetuning.

  • Method

    VPiT finetunes a pretrained LLM to predict discrete text tokens and continuous visual tokens from image-and-text instruction sequences, using separate text and vision heads.

  • Results

    MetaMorph achieves competitive performance on visual understanding and generation, while understanding-centric training improves both capabilities more effectively than generation-centric training.

  • Takeaways & Limitations

    Visual generation can emerge from improved visual understanding with modest additional generation data, suggesting pretrained LLMs contain visual capabilities that instruction tuning can activate.

Abstract

from arXiv · show

In this work, we propose Visual-Predictive Instruction Tuning (VPiT) - a simple and effective extension to visual instruction tuning that enables a pretrained LLM to quickly morph into an unified autoregressive model capable of generating both text and visual tokens. VPiT teaches an LLM to predict discrete text tokens and continuous visual tokens from any input sequence of image and text data curated in an instruction-following format. Our empirical investigation reveals several intriguing properties of VPiT: (1) visual generation ability emerges as a natural byproduct of improved visual understanding, and can be unlocked efficiently with a small amount of generation data; (2) while we find understanding and generation to be mutually beneficial, understanding data contributes to both capabilities more effectively than generation data. Building upon these findings, we train our MetaMorph model and achieve competitive performance on both visual understanding and generation. In visual generation, MetaMorph can leverage the world knowledge and reasoning abilities gained from LLM pretraining, and overcome common failure modes exhibited by other generation models. Our results suggest that LLMs may have strong "prior" vision capabilities that can be efficiently adapted to both visual understanding and generation with a relatively simple instruction tuning process.

1 Introduction

The paper proposes VPiT, a simple extension that trains pretrained LLMs to predict both text and continuous visual tokens, enabling unified multimodal understanding and generation. Experiments find that visual understanding and generation reinforce one another asymmetrically, with understanding-centric training improving both capabilities more effectively.

  • Motivation and approach: Unified multimodal models traditionally require major architectural changes and billions of image-text pairs for pretraining or finetuning.Prior approaches tokenize visual inputs, incorporate diffusion objectives, or separate understanding and generation modes.
  • Motivation and approach: VPiT extends visual instruction tuning by training an LLM to output discrete text tokens and continuous visual tokens from multimodal inputs.A separate diffusion model maps generated visual tokens back into pixel space for visualization.
  • Empirical findings: 200k additional visual generation examples can suffice when VPiT is trained jointly with sufficient visual understanding data.The paper reports that visual-token prediction emerges from understanding visual inputs and needs minimal additional training.
  • Empirical findings: Understanding-centric training substantially outperforms generation-centric training in improving both visual understanding and visual generation.Increasing understanding data raises VQA performance and lowers FID, while generation data improves both abilities to a lesser degree.
  • MetaMorph: MetaMorph achieves competitive performance on visual understanding and generation benchmarks while leveraging pretrained-LLM knowledge and reasoning during visual generation.The model can generate a butterfly after implicitly reasoning from a monarch caterpillar metamorphosis prompt.
  • Implications: The results suggest that LLMs possess strong pre-existing visual capabilities that instruction tuning can activate with relatively few samples.The paper presents unified instruction tuning as feasible and identifies prior visual capabilities in pretrained LLMs.

2 Visual-Predictive Instruction Tuning

VPiT adapts standard instruction tuning to arbitrary interleaved sequences of text and visual tokens, with separate heads and losses for language and vision. It supports diverse instruction-formatted data, while a diffusion model converts predicted visual tokens into images.

  • 2.1 From Unimodal to Multimodal Next-Token Prediction: VPiT extends instruction-tuning conversations so prompts and responses can contain both text and images.Text uses the LLM tokenizer, while images are encoded by a pretrained vision encoder and projected into the LLM dimension.
  • 2.1 From Unimodal to Multimodal Next-Token Prediction: The model processes arbitrary sequences of text and visual tokens with the original language head and a separate vision head.Both response modalities are predicted autoregressively using prompt tokens as context.
  • 2.1 From Unimodal to Multimodal Next-Token Prediction: Special image boundary tokens indicate visual-token spans and when the model should use the vision head.Unlike conventional visual instruction tuning, visual tokens are outputs as well as inputs.
  • 2.1 From Unimodal to Multimodal Next-Token Prediction: Language prediction uses cross-entropy loss, whereas visual prediction uses cosine similarity between predicted and vision-encoder visual tokens.Loss is incurred only on response tokens, consistent with instruction-tuning practice.
  • 2.2 Using Broad Types of Data: VPiT broadens training beyond question answering to visual understanding, visual generation, and other interleaved visual-data formats.The framework formats all data types as instruction-style prompt-response pairs, including image, video, visual-thinking, and image-to-image data.
  • 2.2 Using Broad Types of Data: Visual understanding data maps images or videos to text responses, while visual generation data maps image descriptions to predicted visual tokens.The generation set uses up to 5 million MetaCLIP pairs curated into question-answering formats.
  • 2.2 Using Broad Types of Data: Visual-thinking examples make the model produce visual tokens before textual answers, such as a zoomed-in image segment.Image-to-image examples instead generate transformed images conditioned on text and an input image.
  • 2.3 Mapping Tokens to Images through Diffusion: A separately finetuned diffusion model maps generated visual tokens into pixel-space images during inference.Generation begins after an image-start tag and ends at an image-end tag, after which the visual tokens condition the diffusion model.

3 Findings on Unlocking Visual Generation

The experiments show that visual generation can be unlocked efficiently through joint training, with visual understanding and generation improving each other but contributing asymmetrically. Understanding data is more effective overall, and vision-focused VQA tasks correlate more strongly with generation than knowledge-heavy tasks.

  • 3.1 Visual Generation Can Be Unlocked Efficiently by Joint Training with Visual Understanding: Visual understanding data, especially ImageQA and VideoQA, provides the largest improvement in visual generation when generation data is fixed at 200k.All additional data types improve generation, but visual understanding tasks contribute the most.
  • 3.1 Visual Generation Can Be Unlocked Efficiently by Joint Training with Visual Understanding: Joint training with visual understanding data unlocks visual generation with significantly less generation data than generation-only training.Generation-only training remains inferior even with over 3 million image-text pairs, whereas joint training can unlock generation with 5k generation samples and produce high-quality outputs with 200k.
  • 3.2 Visual Understanding and Generation are Mutually Beneficial: Increasing either understanding data or generation data improves both visual understanding and visual generation.With generation fixed at 200k, more VQA data improves VQA and generation; with VQA fixed at 1M, more generation data improves generation and VQA.
  • 3.3 Understanding Data Contributes More: Understanding data improves overall performance more than generation data, with increases in VQA data producing the strongest gains across Average VQA Score, FID Score, and CLIPScore.At 7M VQA samples, adding generation data contributes minimally.
  • 3.4 Certain Understanding Tasks Correlate More with Generation Performance: General, Vision-Centric, and Text&Chart VQA correlate strongly with generation performance, with Pearson correlation ρ above 0.85, while Knowledge VQA correlates weakly.High-Resolution VQA shows moderate correlation around ρ 0.7.

4 MetaMorph Model

MetaMorph is a unified model trained with VPiT to predict multimodal tokens using a pretrained LLM. It achieves competitive understanding and generation performance while leveraging LLM knowledge and reasoning during visual generation.

  • 4 MetaMorph Model: MetaMorph is trained on LLaMA-3.1 8B with VPiT to predict multimodal tokens.The evaluation covers quantitative performance, LLM knowledge in visual generation, and implicit reasoning in multimodal contexts.
  • 4 MetaMorph Model: MetaMorph achieves competitive performance on visual understanding and generation benchmarks and outperforms other unified models on most benchmarks.The comparison is not fully apples-to-apples because models use different datasets and base LLMs.
  • 4 MetaMorph Model: MetaMorph converts specialized LLM knowledge into accurate visual tokens for concepts such as Chhogori, Oncilla, and Chizarira.Stable Diffusion-3.5 8B struggles with these concepts despite producing high-quality images.
  • 4 MetaMorph Model: MetaMorph handles semantic distinctions involving negation, subjectivity, quantity, and degree more effectively than text-embedding-based text-to-image systems.Examples include “slightly” versus “very,” “few” versus “many,” and “without” versus “with.”
  • 4 MetaMorph Model: MetaMorph generates correct images for prompts requiring multi-step reasoning without explicit Chain-of-Thought hints.For example, it infers Einstein’s association with the violin and generates the corresponding visual tokens directly.

5 Related Work

Prior multimodal systems extend pretrained LLMs for understanding, while unified models generally require substantial architectural changes, multimodal objectives, or billion-scale data.

  • Instruction tuning and visual instruction tuning: Visual instruction tuning repurposes pretrained LLMs into multimodal understanding models using modest compute and millions of image-text question-answer pairs.These models commonly combine a pretrained vision encoder, language model, and modality connector.
  • From Multimodal LLMs to unified models: Unified-model approaches have used discrete visual tokens, diffusion objectives, or separate understanding and generation modes.Several representative systems require billions of image-text pairs for extensive pretraining and finetuning.

6 Discussion

VPiT extends visual instruction tuning to multimodal token prediction, and MetaMorph demonstrates competitive understanding and generation with LLM-derived knowledge and reasoning abilities.

  • 6 Discussion: VPiT enables LLMs to predict multimodal tokens and use diverse instruction-tuning data beyond visual question answering.The controlled experiments find that understanding data improves both capabilities more than generation data.
  • 6 Discussion: MetaMorph’s qualitative results show that pretrained LLM knowledge and reasoning abilities can support visual generation.The model generates specialized concepts and solves multimodal prompts requiring multiple reasoning steps.

A.1 MetaMorph Training

MetaMorph uses a two-stage recipe that adapts a pretrained LLM while preserving the vision backbone, then trains a projector to condition a pretrained diffusion model on visual embeddings.

  • Training recipe: Two-stage training first learns a two-layer GELU MLP adapter, then finetunes the full model except the vision backbone.The adapter is trained on Cambrian adapter data excluding LAION-sourced points; subsequent finetuning uses the instruction-tuning data.
  • Implementation: Experiments use DeepSpeed ZeRO-3 on H100 GPUs and run for one epoch.Hyperparameters are reported in Table 2.
  • Diffusion conditioning: A two-layer MLP projector maps SigLIP embeddings to the diffusion model’s cross-attention dimension.It expands inputs to 2048 dimensions with layer normalization and ReLU, then projects to the output dimension with a final layer normalization.
  • Implementation: The training schedule uses batch size 2112, a 2000-step logarithmic warm-up, and a peak learning rate of 1.1e-5.The learning rate then decreases linearly over 12000 steps; optimization uses AdamW with β parameters (0.9, 0.999) and weight decay 0.01.
  • Diffusion conditioning: The diffusion encoder components remain frozen while the projector and diffusion U-Net are trained.The conditioning is transformed from CLIP text to SigLIP image embeddings, using CFG level 0.7.

A.3 Evaluation Benchmarks

Evaluation covers visual understanding, video understanding, and image generation using a broad set of benchmarks and complementary generation metrics.

  • Benchmark suite: The evaluation includes nine ImageQA benchmarks, one VideoQA benchmark, and two generation benchmarks.The benchmark suite spans multimodal reasoning, visual detail, chart and text reading, science questions, and video understanding.
  • Understanding benchmarks: MMBench measures 20 multimodal ability dimensions, while Seed contains 19k human-annotated multiple-choice questions.These benchmarks target broad multimodal understanding and visual tasks, respectively.
  • Understanding benchmarks: V*STAR tests high-resolution image details, and MMVP evaluates “CLIP-Blind” pairs in vision-language models.Both benchmarks target specific visual understanding weaknesses.
  • Reasoning benchmarks: MMMU evaluates college-level multidisciplinary reasoning, while ChartQA evaluates visual and logical reasoning over charts.The suite also includes TextVQA and ScienceQA for text-in-image and science-related multimodal questions.
  • Video and real-world benchmarks: RealWorldQA targets real-world multimodal reasoning, and MV-Bench covers 20 video tasks that cannot be solved effectively from a single frame.Together they assess real-world and temporal understanding.
  • Generation metrics: FID compares generated-image feature distributions with real images, while CLIP Score measures alignment between generated images and text descriptions.These metrics provide complementary assessments of image quality and text-image alignment.

B Ablation Studies on Visual Prediction Objective

The ablations indicate that cosine similarity loss and diverse non-VQA data improve visual prediction and understanding, while the training mixture spans image, video, text, and visual-reasoning tasks.

  • Loss objective: Cosine similarity loss outperforms L1 loss on average and on most benchmarks.The comparison focuses on generation because differing output normalization prevents a direct fair generation comparison; VQA tasks are not directly affected by these vision losses.
  • Data mixture: Broader non-VQA data combined with cosine similarity loss improves performance across multiple benchmarks over Cambrian-7M alone.The authors connect this pattern to the benefit of using non-VQA data for visual generation and understanding.
  • Data composition: The MetaMorph dataset combines ImageQA and generation data with sources including Cambrian-7M and MetaCLIP.Figure 11 summarizes the distribution and categories of the data sources.
  • ImageQA: ImageQA uses Cambrian-7M examples in which the model answers questions or predicts transformed-image visual tokens.The rainy-day example illustrates visual-token prediction for image transformation.
  • VideoQA: VideoQA inputs frames at one frame per second and uses instruction-formatted question-answer examples.The cited example asks for a multiple-choice answer about the color of an object in an eight-second video.
  • Image generation tasks: Instruction-tuning examples include image transformation, image generation, and visual-token responses.Generation prompts require visual tokens in the response, unlike ImageQA and VideoQA.
  • Visual thinking: Visual-thinking data asks the model to predict an intermediate visual view before answering a question.The intended mode integrates visual skills such as zooming into reasoning steps.
  • Video prediction tasks: Pure-video tasks train forward prediction, partial completion, reverse prediction, and temporal reordering.These tasks are designed to develop temporal understanding and visual reasoning.

C.3 Potential Image Leakage in Testing Data

The evaluation analysis reports that joint training makes visual generation possible with relatively little data, while acknowledging possible image leakage in the multimodal data mixture.

  • Evaluation caveat: Some degree of data leakage may be inevitable because the training mixture contains numerous sources, despite selecting sources to avoid testing-set overlap.Image overlap does not necessarily mean the exact image-question pair appeared during training.
  • Generation data: Joint training with additional data significantly improves generation performance compared with training solely on generation data.At 5,000 samples, the model begins generating reasonably accurate visual tokens.
  • Joint training: Joint training with visual understanding data, especially ImageQA and VideoQA, provides the largest improvement in visual generation performance.This comparison evaluates 200k generation data combined with different data types.

D.2 Results of Joint training Different Understanding and Generation Data

Joint training experiments vary understanding and generation data across multimodal performance, showing that understanding data contribute more strongly to both task types. Stronger LLM backbones also improve visual understanding and generation.

  • Data scaling: Joint training evaluates 1M, 4M, and 7M understanding-data scales alongside 200k–4M generation-data scales.The results correspond to analyses of how data combinations affect visual understanding and generation performance.
  • Data scaling: Understanding data yield more substantial improvements in visual understanding and generation than increasing generation data.The comparison uses VQA performance for understanding and FID and CLIP scores for generation.
  • LLM backbones: Training uses 7M VQA examples and 1M generation examples across LLaMA-3 8B, LLaMA-3.1 8B, and LLaMA-3 70B backbones.These configurations are evaluated for both understanding and generation performance.
  • LLM backbones: Stronger LLM backbones improve both visual understanding and visual generation, supporting reciprocal development between the capabilities.The observation is reported for the evaluated LLaMA backbones.

E More Examples of MetaMorph

Examples show MetaMorph combining visual understanding, reasoning, transformation, and generation within and across conversations. The model also handles implicit puzzles and rare concepts using its multimodal capabilities.

  • Visual reasoning: For “What is the type of hat?”, MetaMorph generates hat-related visual tokens before answering “top hat” correctly.This example illustrates visual-token generation integrated with question answering.
  • Visual reasoning: MetaMorph performs image transformations and solves implicit letter-rearrangement puzzles before generating corresponding visual tokens.Shown transformations include cartoonizing an image and changing its appearance to daytime; “tca” is rearranged to cats.
  • Multimodal interaction: MetaMorph answers questions and transforms images within one conversation, while also generating images and rare concepts.The examples include image transformation, image generation, and leveraging LLM knowledge for rare concepts.
Loading 2412.14164v1…