PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamLLM: Synergistic Multimodal Comprehension and Creation

Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, Li Yi

arXiv:2309.11499v2cs.CVcs.CLcs.LG

TL;DR

Existing multimodal models have not fully realized synergy between comprehension and creation, partly because CLIP-aligned intermediate representations can lose modality-specific information. DREAMLLM directly models raw image and text distributions while generating interleaved documents, and it reports strong zero-shot multimodal performance together with free-form interleaved generation.

  • Problem

    Prior multimodal creation methods have not fully realized comprehension–creation synergy and remain deficient in multimodal comprehension.

  • Method

    DREAMLLM directly samples raw multimodal distributions and uses interleaved generative pre-training to model text, images, and their layouts.

  • Results

    DREAMLLM shows superior zero-shot multimodal-generalist performance, including 8.46 FID on MS-COCO and 49.1/35.9 on MMBench and MM-Vet.

  • Takeaways & Limitations

    The framework enables MLLMs to generate free-form interleaved content while supporting multimodal comprehension and creation.

  • Takeaways & Limitations

    Evaluations mainly use 7B base language models, leaving the benefits of larger models such as 65B or 130B for future exploration.

Abstract

from arXiv · show

This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io.

1 INTRODUCTION

DREAMLLM addresses limitations in multimodal models by generating raw language and image content autoregressively, including free-form interleaved documents. Its experiments report strong zero-shot multimodal performance and learning synergy between comprehension and creation.

  • Motivation: Existing MLLMs often align image outputs with CLIP representations, leaving modality-specific information and multimodal comprehension underdeveloped.The paper identifies marginal creativity gains and deficient comprehension as consequences of this limitation.
  • Approach: DREAMLLM learns image and text posteriors from raw multimodal inputs and outputs in an end-to-end framework.Dream queries preserve the MLLM output space while conditioning Stable Diffusion to decode raw images.
  • Approach: Interleaved Generative Pre-Training encodes and decodes internet-sourced interleaved image-text corpora using a special <dream> token for image placement.Its causal design generates content from multimodal history of arbitrary length.
  • Results: DREAMLLM-7B achieves an 8.46 FID on MS-COCO and scores 49.1/35.9 on MMBench and MM-Vet, respectively.The paper presents these results as evidence of zero-shot multimodal generalist performance.
  • Results: DREAMLLM generates free-form interleaved content after supervised fine-tuning and exhibits learning synergy between multimodal comprehension and creation.The framework is described as adaptable across modalities.

2 BACKGROUND & PROBLEM STATEMENT

The paper frames multimodal generation and comprehension as complementary but notes that CLIP-aligned diffusion strategies may discard modality-specific information. DREAMLLM instead uses direct pixel-space distribution modeling, learned queries, and interleaved autoregressive generation.

  • Background: Autoregressive language modeling factorizes a sequence probability into token-level conditional probabilities, and multimodal sequences can interleave images with words.Images are represented as visual embeddings before causal comprehension.
  • Background: Diffusion models transform data toward Gaussian noise through a forward process and generate data through a reversed denoising process.The denoising model estimates the noise needed to recover perturbed data under conditional embeddings.
  • Problem: Multimodal signals contain modality-specific yet complementary semantics, but multimodal creation’s potential to improve comprehension remains largely unexplored.The paper identifies this as the central learning-synergy gap.
  • Problem: CLIP-alignment strategies may discard modality-specific information because MLLM outputs are forced into CLIP’s intermediate space.The paper argues that this can create conflicts rather than synergies between comprehension and creation.
  • DREAMLLM: DREAMLLM models distributions through direct pixel-space sampling, using pretrained Stable Diffusion as a score metric for distribution distillation.The image posterior is learned through MLLM conditional parameterization.
  • DREAMLLM: Learned embeddings query MLLM-enriched semantics for diffusion conditioning instead of converting MLLM outputs to CLIP-aligned representations.Synthesis sampling implicitly models the resulting distribution.
  • DREAMLLM: Interleaved documents provide multimodal inputs and outputs, with generated images synthesized by Stable Diffusion and fed back for subsequent comprehension.The framework uses a <dream> token and sequential multimodal representations to coordinate generation.

3 DREAMLLM

DREAMLLM is a universal multimodal framework that learns text and image distributions through raw interleaved documents and unified causal modeling. Its three-stage training combines multimodal alignment, I-GPT pretraining, and supervised instruction fine-tuning.

  • Universal Multimodal Generative Modeling: DREAMLLM treats text-only, image-only, and mixed documents as interleaved corpora, enabling learning of conditional, marginal, and joint multimodal distributions.Its unified objective models causally conditioned posteriors for sequences containing either encoded images or words.
  • Interleaved Structure Learning: A special <dream> token marks image locations during training and lets DREAMLLM autonomously trigger image synthesis during inference.The token is predicted before images, after which conditional image synthesis is performed.
  • Conditional Synthesis through Score Distillation: Learnable dream queries condition image synthesis while preserving the MLLM output space, with a frozen Stable Diffusion decoder producing raw images.The image objective uses score distillation, and the Stable Diffusion parameters remain fixed.
  • Training Procedure: DREAMLLM uses a three-stage procedure: alignment training, unfrozen I-GPT pretraining, and supervised fine-tuning for multimodal comprehension and creative instruction following.The stages use approximately 30M image-text pairs, 2M MMC4-Core documents plus 2M LAION400M pairs, and approximately 80K visual instruction examples with document and image synthesis data.

4 EXPERIMENTS

Experiments evaluate DREAMLLM as a zero-shot multimodal generalist across comprehension, image generation, and interleaved document creation. The reported results include strong benchmark performance, free-form instruction-following generation, and automatic image placement with feedback into subsequent generation.

  • 4 EXPERIMENTS: DREAMLLM is evaluated as a zero-shot or in-context multimodal generalist across vision-language comprehension and synthesis tasks.The evaluation includes multimodal comprehension benchmarks, text-to-image generation, and instruction-following interleaved documents.
  • 4.1 MULTIMODAL COMPREHENSION: DREAMLLM outperforms other MLLMs across the reported comprehension benchmarks, including a +16.6 VQAv2 accuracy advantage over Emu-13B.It also achieves state-of-the-art performance against all 7B counterparts on MMBench and MM-Vet.
  • 4.2 TEXT-CONDITIONAL IMAGE SYNTHESIS: Table 2 reports zero-shot text-to-image FID on MS-COCO and LN-COCO across language-model, multimodal-generation, and free-form-interleaved-generation methods.The table annotates fine-tuned Stable Diffusion, retrieval augmentation, and results after stage I alignment training.
  • 4.3 MULTIMODAL JOINT CREATION & COMPREHENSION: DREAMLLM generates interleaved content from human instructions, placing images autonomously by predicting <dream> tokens without additional human intervention.Generated images are created at the selected locations and fed back as multimodal comprehension input for following content generation.
  • 4.3 MULTIMODAL JOINT CREATION & COMPREHENSION: The instruction-following document evaluation uses a held-out 15K-document subset spanning 30 topics and a human evaluation of 150 generated samples.Five volunteers compared generated and ground-truth MMC4 documents without identifying information; MMC4’s supportive rate was 77.24%.

5 DISCUSSIONS

DREAMLLM studies whether jointly learning multimodal creation and comprehension improves both capabilities. Its analyses examine interleaved training, in-context generation, and learned query attention.

  • Synergy between creation and comprehension: The study compares creation-only, comprehension-only, and joint-learning objectives using the same DREAMLLM architecture and training data.Joint learning models both image synthesis and language generation.
  • Synergy between creation and comprehension: Table 3 evaluates how interleaved data and I-GPT pretraining affect multimodal comprehension and image synthesis.The analysis also reports that adding CLIP alignment loss causes DREAMLLM to fail to converge.
  • Synergy between creation and comprehension: Figure 4 contrasts answers from comprehension-only models without interleaved training against answers from joint-learning models.
  • Multimodal in-context generation: DREAMLLM generates images from multimodal context for in-context editing, subject-driven generation, and compositional generation without downstream fine-tuning.The results suggest preservation of subject, identity, and semantic context.
  • Dream query attention: Figure 6 visualizes 64 dream-query attention maps arranged as an 8×8 grid and averaged across diffusion timestamps.The accompanying analysis describes the attention as structured, disentangled, and semantically oriented.

6 RELATED WORKS

Related work extends language models to multimodal comprehension through plugin-style systems and models trained to consume visual content. These approaches establish the surrounding design space for MLLMs.

  • Related works: One line of work integrates LLMs with specialized agents, using language as a general interface for multimodal interaction.
  • Related works: Another line of work trains LLMs to consume visual content directly rather than relying only on system integration.

7 CONCLUSIONS

DREAMLLM addresses the question of how multimodal understanding and creation can learn synergistically. It combines direct distribution learning with interleaved documents and reports broad empirical effectiveness.

  • Conclusions: DREAMLLM learns image and text posteriors through score distillation of conditional-image synthesis distributions, avoiding intermediate representation targets.
  • Conclusions: Interleaved documents enrich multimodal distributions by supporting multimodal encoding and decoding.
  • Conclusions: DREAMLLM performs effectively across diverse vision-language benchmarks and shows learning synergy between multimodal comprehension and creation.
  • Conclusions: After supervised fine-tuning, the framework takes an initial step toward free-form interleaved content creation.
  • Conclusions: DREAMLLM outperforms the Vicuna baseline on most evaluated language-only benchmarks after multimodal adaptation learning.The paper notes that stronger base language models could yield improved results.

A.2 ADDITIONAL MULTIMODAL COMPREHENSION RESULTS

Additional evaluations report strong zero-shot and few-shot multimodal comprehension, including fine-grained understanding, spatial reasoning, hallucination robustness, and in-context performance. Qualitative examples further illustrate format-following comprehension and subject-preserving generation.

  • Zero-shot multimodal comprehension: DREAMLLM-7B outperforms other 7B MLLMs overall on MMBench and MM-Vet, including some 13B models.The paper highlights fine-grained understanding and relational/spatial comprehension.
  • Visual hallucination: DREAMLLM-7B matches or surpasses 13B counterparts on POPE visual-hallucination evaluation and reaches the best or second-best performance in the hardest setting.POPE evaluates answers across random, popular, and adversarial nonexistent-object sampling strategies.
  • Few-shot evaluation: Few-shot evaluation with k=2, 4, and 8 demonstrates strong in-context learning compared with Emu and Flamingo.
  • Qualitative examples: Qualitative examples show that DREAMLLM follows required formats and logic during in-context multimodal comprehension.
  • Image consistency: Subject-representation fine-tuning is used to further investigate consistent image generation because MMC4 lacks image consistency.The experiment targets preservation of subject features during controllable generation.

A.5 ADDITIONAL ABLATION STUDY

The ablations examine query count, inference latency, and prompt-rewriting alternatives for DREAMLLM’s multimodal capabilities.

  • Query Number: 64 learnable <dream> queries achieve the best result, while 128 may reduce performance.The preferred query count depends on training-data size and diffusion-decoder strength.
  • Inference Latency: DREAMLLM adds 0.2s average inference latency relative to Stable Diffusion.The latency mainly comes from diffusion U-Net denoising rather than text-condition embedding.
  • Prompt Rewriting Strategy: Table 10 compares language processing and multimodal comprehension and creation capabilities between DREAMLLM and the rewrite-then-generate baseline.The table provides the evaluation framework for the additional comparison.
  • Prompt Rewriting Strategy: The rewrite-then-generate baseline matches DREAMLLM on language capability but falls short on COCO text-to-image generation.The comparison uses rewritten prompts and an off-the-shelf Stable Diffusion decoder without image decoders during training.

B ADDITIONAL QUALITATIVE EXAMPLES

Additional qualitative evaluations compare DREAMLLM with established VQA systems and show examples of multimodal dialogue and text-conditioned image synthesis.

  • Multimodal Dialogue: DREAMLLM provides more detailed and precise VQA responses than GPT-4 in the presented comparisons.The qualitative comparison includes LLaVA, BLIP-2, and OpenFlamingo.
  • Multimodal Dialogue: DREAMLLM avoids the imaginary visual elements frequently introduced by LLaVA.The observation is presented as consistent with earlier visual-hallucination evaluation findings.
  • Multimodal Dialogue: The dialogue examples illustrate comprehension and generation of long-context multimodal information across arbitrary input and output formats.Figures 10–12 provide qualitative multimodal-dialogue examples.
  • Text-condition Image Synthesis: Figures 13–14 present DREAMLLM text-conditioned image examples using prompts from DALL-E, DALL-E 2, GLIDE, Imagen, and Parti.Some prompts are extended with additional sub-prompts to construct further examples.
  • Evaluation Context: The appendix identifies the training recipes and evaluation-benchmark descriptions supporting these qualitative comparisons.Table 13 summarizes three training stages, while Table 14 describes evaluation benchmarks.

C.1 TRAINING DATA & HYPER-PARAMETERS

The appendix specifies DREAMLLM’s datasets, model components, training recipes, inference settings, and evaluation prompts.

  • Training Data: Training data combine LAION400M, LAION-COCO, MMC4, BLIP-LAION, and LLaVAPretrain-derived image-text resources.The listed sources support multimodal training across image-text and interleaved data.
  • Model Components: The default language model is Vicuna-7B1, based on LLaMA-1 trained on ShareGPT.Flash Attention and PyTorch FSDP are used to accelerate training.
  • Model Components: The visual encoder is frozen OpenAI CLIP-L/14, processing 224×224 images into 256-token sequences with boundary image tokens.This configuration follows CLIP pretraining resolution and prior vision-language practice.
  • Model Components: The frozen diffusion image decoder is Stable Diffusion v2.1 at 512×512 resolution.Images are resized to 512 resolution when constructing diffusion targets for the MSE loss.
  • Dream Queries: DREAMLLM uses 64 learnable dream queries and special <dream> and <dream/> tokens to gather semantics and mark image generation.The queries gather semantic context, while the tokens delimit the query sequence and signal image generation.
  • Inference Settings: Classifier-free guidance is applied only during inference, with default scale 7.5 and MS-COCO scale 2.0.Higher guidance generally improves authenticity while reducing diversity.
  • Evaluation: Systematic evaluation covers vision-language comprehension, content creation, and NLP using benchmark prompt templates.The appendix points to Figure 9 for templates and Table 13 for benchmark and dataset details.

E LIMITATIONS, FAILURE CASES & FUTURE WORKS

The paper identifies limitations in model scale, training data, prompt sensitivity, and compositional multi-image generation, while outlining extensions to broader modalities and tasks.

  • Limitations: Current evaluations mainly use 7B language models, leaving the benefits of 65B or 130B models for future exploration.Model scale is identified as the primary constraint.
  • Limitations: High-quality training data are difficult to procure and refine, and noise in MMC4 may affect output language and image style.The paper specifically cites commercial advertisements as one source of noise.
  • Limitations: Prompt sensitivity extends to MLLMs, requiring tailored prompts when concise VQA answers are desired.The authors note that MLLMs’ detailed responses can be unsuitable for short-answer settings.
  • Failure Cases: With two input images and an instruction such as “A and B,” DREAMLLM may merge their characteristics into one subject.The paper relates this failure to compositional complexity and scarce domain-specific data.
  • Future Works: Future work targets fine-grained visual comprehension, image-to-image translation, video, 3D and geometry-preserving tasks, perception, and additional modalities.Proposed directions include audio generation and unified zero-shot multimodal capabilities.
  • Additional Examples: The qualitative appendix includes multimodal dialogue, bird descriptions, and varied text-to-image examples, including material transformations of a duck sculpture.These examples cover arbitrary multimodal interactions and multiple image-generation prompts.
Loading 2309.11499v2…