PaperlayerView exact revision on arXiv

Source-linked AI summary

MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens

Kaizhi Zheng, Xuehai He, Xin Eric Wang

arXiv:2310.02239v4cs.CVcs.AI

TL;DR

Coherent interleaved image-and-text generation remains underdeveloped despite MLLMs’ multimodal understanding capabilities. MiniGPT-5 uses generative vokens and two-stage training for description-free generation, and reports better multimodal generation results on VIST and MMDialog, with a texture-maintenance limitation.

  • Problem

    Coherent simultaneous generation of images and text remains underdeveloped, despite MLLMs’ multimodal understanding capabilities.

    Source: §1 · p. 1

  • Method

    MiniGPT-5 connects an LLM to a text-to-image model with generative vokens and uses two-stage, description-free training with classifier-free guidance.

    Source: §1 · p. 1, §3 · p. 2, §3.2 · p. 4

  • Results

    MiniGPT-5 reports improvements over baselines on VIST and MMDialog and results comparable to the state of the art on CC3M; human evaluators preferred its coherent multimodal outputs in 56% of cases against the two-stage baseline.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported results support MiniGPT-5’s efficacy for interleaved multimodal generation on the evaluated datasets.

    Source: §1 · p. 2

    The authors report that object texture remains difficult to maintain in newly generated images.

    Source: §5 · p. 8

Abstract

from arXiv · show

The effectiveness of Multimodal Large Language Models (MLLMs) demonstrates a profound capability in multimodal understanding. However, the simultaneous generation of images with coherent texts is still underdeveloped. Addressing this, we introduce a novel interleaved vision-and-language generation method, centered around the concept of ``generative vokens". These vokens serve as pivotal elements contributing to coherent image-text outputs. Our method is marked by a unique two-stage training strategy for description-free multimodal generation, which does not necessitate extensive descriptions of images. We integrate classifier-free guidance to enhance the alignment of generated images and texts, ensuring more seamless and contextually relevant multimodal interactions. Our model, MiniGPT-5, exhibits substantial improvement over the baseline models on multimodal generation datasets, including MMDialog and VIST. The human evaluation shows MiniGPT-5 is better than the baseline model on more than 56\% cases for multimodal generation, highlighting its efficacy across diverse benchmarks.

1. Introduction

MiniGPT-5 addresses the difficulty of generating coherent interleaved vision-and-language outputs with generative vokens and a description-free, two-stage training strategy. It reports improvements on multimodal generation benchmarks and positive human evaluations against a two-stage baseline.

1. Introduction

  • Existing LLMs and vision-and-language models fall short at generating coherent multimodal outputs, especially when tasks require integrated vision and language.The introduction identifies this integration as important for next-generation LLMs.Source: §1 · p. 1
  • Generative vokens connect LLMs with Stable Diffusion, while description-free training addresses the scarcity of detailed image captions.The approach combines text-generation, latent-diffusion, and caption-alignment losses to improve generation quality.Source: §1 · pp. 1–2
  • Human evaluators preferred MiniGPT-5’s coherent multimodal outputs in 56% of cases compared with the two-stage baseline.The evaluation also favored its appropriate texts in 55% of cases and high-quality images in 53%.Source: §1 · p. 2
  • MiniGPT-5 reports superior performance on VIST and MMDialog and results comparable to the state of the art on CC3M.The reported human comparison is against a two-stage baseline that first generates intermediate image captions.Source: §1 · p. 2

2. Related Work

Related work spans multimodal comprehension models, text-to-image systems, and LLM-based multimodal generation. MiniGPT-5 is presented as a modular approach that reuses pretrained backbones and connects LLM outputs directly to diffusion conditioning.

2. Related Work

  • MiniGPT-4 aligns a pretrained vision component with an open-source LLM, and MiniGPT-5 builds on it to extend multimodal comprehension toward generation.The related-work discussion describes MiniGPT-4’s projection layer as aligning BLIP-2’s vision component with Vicuna.Source: §2 · p. 2
  • Text-to-image models bridge textual information and visual content; Stable Diffusion V2 generates conditional image features and reconstructs images from them.MiniGPT-5 uses this pretrained model as a basis for multimodal input and output.Source: §2 · p. 2
  • Prior LLM-based multimodal generation includes retrieval-augmented decoders, tokenized image streams, visual-token interfaces, and unified autoregressive models.The approaches cited include CM3Leon, Emu, NextGPT, GILL, SEED, Chameleon, Emu3, and Show-o.Source: §2 · p. 2
  • Unlike unified models that extensively retrain backbones, MiniGPT-5 reuses pretrained MLLM and diffusion backbones while tuning a lightweight mapper and LoRA.Its generative vokens project LLM states into diffusion conditioning and are trained with latent-diffusion loss.Source: §2 · p. 2

3. Method

MiniGPT-5 connects a pretrained multimodal LLM to a pretrained text-to-image model through generative vokens. Its method combines token positioning, voken-to-diffusion alignment, and two-stage training with classifier-free guidance.

3. Method

  • MiniGPT-5 integrates a pretrained multimodal LLM with a text-to-image model, using generative vokens to bridge textual and visual domains.The pipeline uses MiniGPT-4 for multimodal input understanding and Stable Diffusion for visual generation.Source: §3 · pp. 2–3, Figure 2 · p. 3

3.1. Multimodal Understanding Module

  • The understanding module adds image-positioning generative vokens to the LLM vocabulary, whose hidden states are used for image generation.The default is eight vokens; MiniGPT-4 pretrained weights are fixed, while extra voken input and output embeddings are trainable, alongside PEFT components.Source: §3.1 · pp. 3–4

3.2. Multimodal Generation Module

  • The generation module trains causal language modeling to position tokens and voken alignment to match diffusion conditioning features, enabling description-free learning from images.A feature mapper transforms voken states into conditioning features used in denoising; the LDM loss trains this alignment against ground-truth image latents.Source: §3.2 · p. 4

3.3. Training Strategy

  • Two-stage training first aligns vokens with image-generation features on single text-image pairs, then fine-tunes on interleaved data for text-only, image-only, and multimodal generation.Classifier-free guidance drops voken features with 10% probability during training; the resulting unconditional features serve as negative prompting at inference.Source: §3.3 · pp. 4–5

4. Experiments

Experiments evaluate MiniGPT-5 on CC3M, VIST, and MMDialog against unimodal and multimodal baselines using image, text, multimodal, and human measures. Main results favor MiniGPT-5 in VIST human judgments and MMDialog text accuracy and MM-Relevance, while ablations probe guidance, losses, context, architecture, and backbones.

4. Experiments

  • Experiments assess plausible image and text generation, comparisons on single- and multi-turn interleaved tasks, and the effects of module designs across CC3M, VIST, and MMDialog.The evaluation asks whether the model generates plausible images and reasonable text and how it compares with state-of-the-art models.Source: §4 · p. 5

4.1. Implementation Details

  • CC3M pretraining runs for two epochs; subsequent VIST and MMDialog fine-tuning runs for four epochs with nearly 6.6 million trainable parameters.Fine-tuning uses LoRA modules and can be completed on a server with four A6000 GPUs.Source: §4.1 · p. 5

4.2.1. Baselines

  • Comparisons include fine-tuned Stable Diffusion 2.1 and MiniGPT-4, a caption-first MiniGPT-4/Stable Diffusion pipeline, GILL, and Divter.The two-stage baseline uses InstructBLIP-13B to create synthetic captions for VIST; unlike GILL, MiniGPT-5 does not rely on caption data during multimodal-generation fine-tuning.Source: §4.2.1 · pp. 5–6

4.3. Main Results

  • On VIST, MiniGPT-5 produces more fitting narrations in around 55% of instances, superior image quality in around 53%, and more coherent multimodal outputs in around 56%; on MMDialog, it improves textual accuracy and MM-Relevance while image quality is similar.The VIST comparison is against the two-stage baseline; the MMDialog comparison is against Divter.Source: Table 3 · p. 6, Table 4 · p. 7, §4.3 · p. 7

4.2.2. Metrics

  • Image quality and diversity use IS and FID, text uses BLEU, Rouge-L, METEOR, and S-BERT, and multimodal evaluation includes CLIP-I and MM-Relevance.Human evaluation assesses language continuity, image quality, and multimodal coherence.Source: §4.2.2 · p. 6

4.3. Main Results

  • On VIST, MiniGPT-5 generates coherent, high-quality images from long-horizon multimodal prompts, and adding generative vokens does not hurt multimodal-comprehension performance.Tables 1 and 2 report image and narration generation results on the VIST validation set.Source: §4.3 · p. 6, Table 1 · p. 6, Table 2 · p. 6

4.4. Ablation Studies

  • Without CFG, all metrics worsen; caption loss aids image generation, and voken alignment further improves coherence and image quality.These ablations assess CFG and caption and conditional latent-diffusion losses on CC3M.Source: §4.4 · pp. 7–8, Table 5 · p. 7
  • More input information increases CLIP similarities; fixed queries trade higher IS and FID for lower CLIP similarities, while the encoder-decoder outperforms decoder-only across four metrics.Backbone comparisons find gains on VIST persist across MLLMs and improve with a stronger generator; MiniGPT-5 remains competitive on CC3M while supporting interleaved generation.Source: Table 6 · p. 8, Table 7 · p. 8, Table 8 · p. 8

5. Conclusion

MiniGPT-5 aligns an LLM with a pretrained text-to-image model for multimodal generation, but maintaining object texture remains difficult.

5. Conclusion

  • MiniGPT-5 aligns an LLM with a pretrained text-to-image model to augment multimodal generation.The authors aspire for it to set a benchmark for future multimodal generative models.Source: §5 · p. 8
  • The authors report substantial improvements from this approach.Source: §5 · p. 8
  • Maintaining object texture in newly generated images remains difficult.Source: §5 · p. 8

A.1. Datasets

The paper uses captioned image pairs for pretraining and multimodal storytelling and dialogue data for fine-tuning, with different history-context lengths across datasets.

A.1. Datasets

  • VIST contains five-image sequences paired with textual narratives; its training split has over 34 thousand sequences and its validation split has 4 thousand sequences with 8 thousand images.Source: §A.1 · p. 12
  • MMDialog consists of multimodal conversations averaging 2.59 images per dialogue.Source: §A.1 · p. 12

A.2. Data Format

  • Pretraining uses CC3M single-turn text-image pairs, appending image-placeholder vokens after captions for the LLM to generate.The resulting hidden features are used to compute conditional generation loss against the ground-truth image.Source: §A.2 · p. 12, §A.1 · p. 12
  • Fine-tuning uses VIST and MMDialog prompts with image placeholders and task instructions; outputs can be text, vokens, or both.Source: §A.2 · p. 12
  • VIST history includes all previous story steps, whereas MMDialog uses at most one previous turn because of computational-resource limits.Source: §A.2 · p. 12

B.1. Evaluation of Guidance Scale

The guidance-scale analysis finds improving image metrics as guidance increases, but above 10 coherence stabilizes while quality declines. The accompanying screenshot shows the human-evaluation interface comparing MiniGPT-5 with the two-stage baseline.

B.1. Evaluation of Guidance Scale

  • Above a guidance scale of 10, image semantic coherence stabilizes while image quality declines.The authors report that both Stable Diffusion and MiniGPT-5 generate better images as guidance increases, recommending a reasonable scale range.Source: §B.1 · p. 12
  • The analysis plots metric changes across varying guidance scales.Source: §B.1 · p. 12
  • The human-evaluation screenshot labels output 1 as MiniGPT-5 and output 2 as the two-stage baseline.Source: Figure 4 · p. 13

B.2. Evaluation of Voken Number

Increasing the number of vokens improves model performance, with the most noticeable gain occurring between one and four vokens.

B.2. Evaluation of Voken Number

  • Model performance consistently improves as the number of vokens increases from 1 to 8.Source: §B.2 · p. 13
  • The improvement is particularly noticeable when increasing the number of vokens from 1 to 4.Source: §B.2 · p. 13
  • The study tests models trained with varying counts of vokens.The tested range is 1 to 8 vokens.Source: §B.2 · p. 13

B.3. Texture Preservation via FID

The VIST final-step evaluation compares generated and ground-truth final images using FID under identical multimodal context. ViLGen attains the lowest FID in this setting.

B.3. Texture Preservation via FID

  • For each story, the model receives all preceding step images and narrations, then generates the final image.The evaluation assesses final-image generation with the preceding story context provided.Source: §B.3 · p. 13
  • ViLGen (Qwen2.5-VL + SD3) attains the lowest FID in the VIST final-step setting.The comparison is reported in Table 9.Source: §B.3 · p. 13
  • FID compares generated and ground-truth final-image sets, measuring realism and low-level appearance consistency under identical multimodal context.Source: §B.3 · p. 13

C. More Qualitative Examples

The qualitative examples span VIST, free multimodal generation, MMDialog, and CC3M text-to-image generation, with additional plots examining classifier-free guidance and voken count.

Classifier-free guidance

  • The CFG plots suggest appropriate guidance scales enhance image quality and coherence, with effectiveness comparable to SD2’s CFG strategy.Source: Figure 5 · p. 14

Voken count

  • As voken count increases, image quality and CLIP scores improve; the default is 8 vokens.Source: Figure 6 · p. 14

VIST image generation

Free multimodal generation

MMDialog dialog generation

CC3M text-to-image generation

Loading 2310.02239v4…