PaperlayerView exact revision on arXiv

Source-linked AI summary

Jointly Training Large Autoregressive Multimodal Models

Emanuele Aiello, Lili Yu, Yixin Nie, Armen Aghajanyan, Barlas Oguz

arXiv:2309.15564v2cs.LGcs.CLcs.CV

TL;DR

Existing multimodal models predominantly generate one modality, limiting integrated text-and-image output. JAM fuses autoregressive text and image models and adds specialized instruction tuning; the authors report coherent long-form outputs interleaving both modalities.

  • Problem

    Existing multimodal models predominantly generate one modality, while language models are limited to text output, leaving integrated text-and-image generation as a challenge.

    Source: §1 · p. 1

  • Method

    JAM merges pretrained autoregressive text-only and text-to-image models as an initialization for mixed-modal training, followed by instruction tuning on curated interleaved data.

    Source: §2.1.1 · p. 2, §2.2 · p. 4

  • Results

    The authors report a first-of-its-kind model that coherently generates long-form interleaved text and images; integration uses less than 1% of the parent models’ original pretraining data.

    Source: §1 · p. 1

  • Takeaways & Limitations

    The study reports the feasibility of combining two pretrained models into a cohesive structure with extended capabilities and fine-tuning it for interleaved image-text generation.

    Source: §5 · p. 10

    Generation is more effective when questions match the dataset style and images depict concrete objects; abstract concepts are often not represented faithfully.

    Source: A LIMITATIONS · p. 13

Abstract

from arXiv · show

In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains a significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned model demonstrates unparalleled performance in generating high-quality multimodal outputs and represents the first model explicitly designed for this purpose.

1 INTRODUCTION

The paper aims to address the limited multimodal generation of existing models by combining autoregressive text and image models. Its JAM framework pairs this fusion with specialized instruction tuning to generate long-form outputs that interleave text and images.

1 INTRODUCTION

  • Existing multimodal models still predominantly generate one modality, while language models are limited to text output.The paper frames integrated text-and-image generation as a way to address this limitation.Source: §1 · p. 1
  • JAM fuses specialized autoregressive text-to-image and text-only models, using less than 1% of both parent models’ original pretraining data.The authors cite architectural compatibility as enabling deep model fusion and joint training.Source: §1 · p. 1
  • The resulting model coherently generates long-form content with interleaved text and images, following specialized multimodal instruction tuning.The paper describes this as a first-of-its-kind large multimodal model.Source: §1 · p. 1

2 METHODS

JAM combines pretrained text-only and text-to-image autoregressive models, then trains the unified model on mixed-modal data. The methods explore weight averaging, width concatenation, cross-model fusion, and instruction tuning for interleaved image-text generation.

2 METHODS

  • The framework combines two decoder-only models and proceeds through model combination and alignment, followed by instruction tuning for interleaved image-text generation.The backbones are a 7B text-to-image model and a 7B text-only model.Source: §2 · pp. 1–2

2.1 CONTINUED PRETRAINING

  • The composite model is fine-tuned during continued pretraining on a hybrid dataset of text-only and image-text samples.Source: §2.1 · p. 2

2.1.1 MODEL MERGING

  • JAM-Uniform averages the pretrained LLM and text-to-image weights as an initialization for subsequent mixed-modal training, rather than as a training-free endpoint.Source: §2.1.1 · p. 2

2.1.3 CROSS MODEL FUSION

  • JAM-Cross interleaves bidirectional cross-attention between the original LLM and text-image blocks, exchanging corresponding-layer representations and projecting concatenated branch outputs.Source: Figure 2 · p. 3, §2.1.3 · pp. 3–4

2.1.2 WIDTH CONCATENATION

  • JAM-Width doubles hidden dimensions to 8192 from 4096 while retaining layer count, producing a 26B-parameter architecture; one variation initializes half its new parameters from averaged weights.Source: §2.1.2 · p. 3

2.2 MULTIMODAL CONVERSATIONAL INSTRUCT TUNING

  • Instruction tuning uses curated interleaved image-text data with standard supervised training, without reinforcement learning or human-preference optimization.For retrieval-augmented generation, the training mixture includes some Shutterstock image-text data; the authors report this helps preserve image quality when generating with retrieval augmentation.Source: §2.2 · p. 4

3 EXPERIMENTS

JAM experiments test whether pretrained text and image-text models can be combined while retaining their strengths and supporting interleaved generation. JAM-Cross leads the reported image-text comparisons, and its instruct-tuned outputs are reported as coherent and stronger than GILL qualitatively.

3.1 EXPERIMENTAL DETAILS

  • The experiments use CM3leon and a same-architecture LLM, with a shared text tokenizer, image VQ-VAE tokens, and the CM3 infilling objective.The image tokenizer represents each 256 × 256 image with 1024 tokens; masking does not cross modality-break tokens.Source: §3.1 · p. 4
  • Retrieval augmentation ranks multimodal documents using averaged text-and-image embeddings, while mixed-modal decoding alternates text and images at modality-break tokens.Retrieved candidates with scores at or below 0.9 are skipped; sampling also uses temperature, top-p, and classifier-free guidance.Source: §3.1 · pp. 4–6
  • Alignment training uses about 50B multimodal tokens; instruction tuning runs for 15 epochs, with the ninth-epoch checkpoint selected manually.The authors describe instruction tuning as data-efficient and report a run time of under two hours on 64 80GB A100s.Source: §3.1 · p. 5

3.1.1 DATASETS

  • The training data combine Shutterstock image-caption pairs, sampled public text corpora, LIMA prompts, and quality-filtered WikiHow interleaved articles.The WikiHow sample comprises 3,000 articles, balanced across categories and rated above 90/100; samples are limited to three images each.Source: §3.1.1 · p. 6

3.2.1 TEXT MODALITY

  • JAM-Cross yields the best reported text results, aligning with the primary LLM; JAM-Uniform slightly outperforms JAM-Width.The comparison evaluates zero-shot common-sense reasoning tasks and targets retention of parent-model performance after merging.Source: §3.2.1 · pp. 6–7, §3.2 · p. 6, Table 1 · p. 7

3.2.2 IMAGE-TEXT MODALITY

  • JAM-Cross achieves the best reported MS-COCO validation PPL among JAM strategies and surpasses CM3leon.The authors identify it as the best strategy for combining the two pretrained autoregressive models.Source: Table 2 · p. 7, §3.2.2 · p. 7

3.3 ABLATION STUDY

  • Ablations report that copying weights retains JAM-Width parent capabilities better than averaging, and mixed image-text pretraining data helps preserve MS-COCO PPL during instruction tuning.The JAM-Cross ablation varies cross-attention insertion frequency and includes a no-cross-attention condition; these runs use 25B tokens, versus 50B total training.Source: Table 3 · p. 7, Table 4 · p. 7, Table 5 · p. 7

3.2.3 INTERLEAVED GENERATION

  • In qualitative comparisons, instruct-tuned JAM-Cross produces coherent interleaved outputs and more complete text and relevant images than GILL.Figure 3 contrasts samples without retrieval augmentation (top) and with it (bottom); the authors report retrieval augmentation boosts image quality over standard image sampling.Source: Figure 3 · p. 5, §3.2.3 · p. 8, Figure 4 · p. 8

4 RELATED WORKS

The related work reviews autoregressive text-to-image generation, multimodal language models, and instruction tuning. It positions JAM's focus on interleaved image-text generation against prior work on visual understanding and image generation or retrieval.

Generative Text-to-Image Models

  • Autoregressive image models encode images as discrete tokens and use sequence modeling; retrieval augmentation has improved their training efficiency and generation performance.Yasunaga et al. (2022) report a training-efficiency gain, and Yu et al. (2023) report state-of-the-art image generation with 5x less training compute.Source: §4 · p. 9

Multimodal Language Models

  • Prior multimodal language models connect visual encoders to LLMs, while Flamingo injects visual features through cross-attention and GILL uses a mapping network with frozen models.JAM also uses cross-attention, but describes its mechanism as bidirectional between vision and language models.Source: §4 · p. 9

Instruction Tuning

  • Prior instruction tuning uses NLP datasets or generated instruction data; LIMA reports competitive results with 1,000 curated samples, while this work targets mixed-modal generation.The passage contrasts this focus with previous vision-language instruction tuning for visual-content understanding.Source: §4 · p. 9

5 CONCLUSIONS

The conclusion presents JAM as a way to combine pretrained autoregressive models and fine-tune them for interleaved image-text generation. It also identifies the study's model scale and architecture as a boundary and names directions for future work.

5 CONCLUSIONS

  • The study demonstrates the viability of combining two pretrained autoregressive models into a cohesive structure with extended capabilities.The integrated model is fine-tuned with a tailored procedure for interleaved image-text generation.Source: §5 · p. 10
  • The study is limited to 7B-parameter models with the same architecture.Future work could scale model size or apply cross-fusion asymmetrically to models of different sizes.Source: §5 · p. 10
  • The conclusion identifies longer context and multi-turn conversations as directions for further exploration.Source: §5 · p. 10

A LIMITATIONS

The authors describe limitations in inherited generation quality, instruction-tuning scope, and the model's performance on abstract concepts. They also identify data and context length as constraints on generating longer outputs.

A LIMITATIONS

  • JAM inherits potential hallucinations and biases from its LLM and autoregressive text-to-image backbones.The authors suggest that better autoregressive backbones may address these limitations.Source: A LIMITATIONS · p. 13
  • Instruction tuning focuses on wiki-style single-turn question answering, and generating longer outputs remains challenging with 4k-token context and 1,024 tokens per image.The authors say this is partially bottlenecked by training data and context length; the model usually generates one or at most two images.Source: A LIMITATIONS · p. 13
  • Generation is more effective when question style matches the dataset and images depict concrete objects; abstract subjects can fail to capture textual context.The authors cite “time travel,” “music,” and “research” as examples of abstract subjects.Source: A LIMITATIONS · p. 13

B GENERATED EXAMPLES

The section presents generated samples from instruction-tuned models using retrieval augmentation and standard image generation. Its figures include examples labeled as failure cases as well as examples from both generation settings.

B GENERATED EXAMPLES

Loading 2309.15564v2…