LMFusion: Adapting Pretrained Language Models for Multimodal Generation

1 Introduction

• Adapting pretrained LLMs raises the challenge of preserving text-only performance while adding visual understanding and generation; naive multimodal finetuning degrades language processing.The section notes that training unified multimodal models from scratch demands substantial compute.Source: §1 · p. 2
• LMFusion combines pretrained language modules with dedicated visual modules, freezing the text modules while training the image modules to preserve language capabilities and support visual learning.The design uses modality-specific QKV projections and FFNs with a joint self-attention layer for cross-modal interactions.Source: §1 · pp. 2–3
• Reusing pretrained LLMs avoids including text-only data in training, reducing computational demands while preserving pretrained text performance.The framework also reports adapting existing vision-language models such as LLaVA for multimodal generation.Source: §1 · p. 3
• LMFusion improves image understanding by 20% and image generation by 3.6% versus Transfusion, uses 50% of the FLOPs, and preserves Llama-3 text-only performance, outperforming Transfusion by 11.6%.The comparison uses a pretrained Llama-3 8B model and the same image data as Transfusion.Source: §1 · p. 3
• Ablations show that naive finetuning without modality separation causes catastrophic forgetting, while deep separation outperforms shallow separation and both outperform no separation.Shallow separation uses modality-specific FFNs only.Source: §1 · p. 3

2 Background: Transfusion

• Transfusion is a unified model for text generation, image understanding, and image generation, jointly modeling language tokens and diffusing image representations.Its architecture adds a U-Net structure to a standard Transformer to project image representations down and back up around diffusion.Source: §2 · p. 3

2.1 Language Modeling

• Language modeling predicts each text token from preceding tokens and learns by minimizing cross-entropy between the model and observed data distributions.When image data precedes language tokens, its representation can optionally condition the language-modeling objective.Source: §2.1 · p. 3

2.2 Image Diffusion

• Image diffusion encodes images as continuous latents, adds Gaussian noise over timesteps, then trains a model to predict that noise from noisy image data and context.The described context can include text prompts, and a U-Net downsamples image representations before the Transformer and upsamples them afterward.Source: §2.2 · pp. 3–4

2.3 Training Objective

• Transfusion combines the language-modeling loss on text inputs and diffusion loss on image inputs using a weighting hyperparameter.Source: §2.3 · p. 4

3 LMFusion

• LMFusion combines pretrained text-only Llama-3 with a dedicated image transformer, freezing text modules and finetuning visual modules to preserve language capabilities while supporting visual learning.The motivation is that Transfusion-style shared parameters pose a challenge for retaining Llama-3's text-only performance during image-capability training.Source: §3.1 · p. 4, §3 · p. 4

3.1 Model Architecture

• The decoder-only model uses modality-specific attention and FFNs, with text tokens and noisy image representations as inputs.The described configuration folds residual connections and layer normalization into self-attention and FFN modules.Source: §3.1 · pp. 4–5
• Separate QKV projections process text and image states, which are concatenated for cross-modal attention; the hybrid mask is causal for text and bidirectional for images.Attention outputs are projected back into each modality's hidden-state dimension using separate output weights.Source: §3.1 · p. 5
• After the transformer layers, text hidden states produce language logits while image hidden states produce predicted noise through a U-Net upsampler.Source: §3.1 · p. 5
• Text and image modules are initialized from the pretrained Llama model, and the main experiments freeze text parameters by setting their learning rate to zero.The image learning rate is decoupled from the text learning rate.Source: §3.1 · p. 5

4 Experiments

• The experiments compare LMFusion with Transfusion and report that LMFusion outperforms Transfusion in the FLOPs-match setting across text-only, image-understanding, and image-generation benchmarks.The reported comparison also includes training setup and evaluation results across the three task types.Source: §4 · p. 6

4.1 Training Setup

• LMFusion continues training from pretrained Llama-3 modules, freezes the text modules, and trains image modules with a 0.27B-parameter U-Net downsampler and upsampler.Images are compressed into 256 tokens, and the maximum context length is 4,096 tokens.Source: §4.1 · p. 6

4.2 Controlled Comparison with Transfusion

• The controlled comparison uses Transfusion 7B, trained on 0.25T text and 0.25T image-caption tokens; LMFusion also tests matching image data alone at about half the FLOPs.A second LMFusion configuration matches Transfusion’s total FLOPs, and language-only results are compared with Llama-3 8B.Source: §4.2 · p. 6

4.3 Evaluation Setup

• Evaluation covers language-only benchmarks, MS-COCO image captioning, and MS-COCO image generation measured with CIDEr, FID, and CLIP scores.Image generation uses 30K validation prompts, with results reported both without classifier-free guidance and with CFG coefficients of 1.55 or 1.6.Source: §4.3 · pp. 6–7

4.4 Results

• With half the training FLOPs, LMFusion preserves Llama-3’s text performance, improves image understanding by 20%, and outperforms Transfusion on image generation quality.The reported generation metrics are CLIP and FID; Figure 3 describes an advantage over Transfusion throughout training for captioning and generation.Source: Table 1 · p. 7, §4.4 · p. 7

5 Analysis

• The analysis also presents modality separation as central to retaining performance across modalities and describes extending the recipe to image-to-image generation.Image editing is used as an example requiring simultaneous understanding of input images and textual prompts.Source: §5 · p. 8, §5.2 · p. 10

5.1.1 Experimental Design

• The ablation compares dense, FFN-only shallow, and FFN-plus-attention deep separation, using small-scale models trained for 250K steps on 0.03T text and 0.03T image-caption tokens.The tested text-to-image learning-rate ratios are 0, 0.1, and 1.Source: §5.1.1 · pp. 8–9
• Deep modality separation outperforms both shallow separation and dense models on image benchmarks, with the gap over shallow separation widening notably for image generation.Deep separation uses modality-specific FFNs and attention; shallow separation uses modality-specific FFNs only.Source: Figure 5 · p. 9, §5.1.1 · p. 10
• Equal text and image learning rates leave HellaSwag 7% below Llama-3 after an initial 15% drop; a 0.1 ratio narrows the gap to 2% but reduces image capabilities in dense models.Learning-rate decoupling mitigates, but does not resolve, the trade-off in dense models without modality separation.Source: §5.1.1 · pp. 9–10
• Freezing text modules impairs image understanding and generation in dense models, but deep separation preserves language performance while performing strongly on both image tasks.The ablation compares learning-rate ratios including a ratio of zero, which freezes text-related components.Source: §5.1.1 · pp. 9–10, Figure 6 · p. 9

5.2 Image editing

• After finetuning on 8K image-editing examples, LMFusion produces qualitative edits on MagicBrush prompts that demonstrate image editing alongside its text, understanding, and generation capabilities.Each training example contains an input image, an edit prompt, and a resulting image.Source: §5.2 · p. 10

5.3 LLaVAFusion: extending LMFusion to vision-language models

• LLaVAFusion freezes LLaVA-NeXT’s transformer parameters and trains a parallel image-specific transformer; it reports strong image understanding and generation compared with unified multimodal models.Understanding is evaluated on MMMU, MME-Perception, ChartQA, and RealWorldQA.Source: §5.3 · pp. 10–11

Unified Models for Multimodal Generation

• Unified multimodal models represent images with discrete vector-quantized tokens or continuous embeddings; Transfusion combines autoregressive text generation with diffusion for images.The cited approaches include (Lu et al., 2022; Yu et al., 2023; Lu et al., 2024; Team, 2024b; Xie et al., 2024a; Wu et al., 2024b; Aiello et al., 2023) for discrete tokens and (Sun et al., 2024; Ge et al., 2024) for continuous embeddings.Source: §6 · p. 11

Model Sparsity

• Prior multimodal sparsity work uses modality-specific FFN experts, while LMFusion extends separation to modality-specific attention mechanisms.Concurrent work (Liu et al., 2024a) also reports the effectiveness of deeper separation in multimodal pretraining and image generation.Source: §6 · p. 11

Reuse of LLMs in Multimodal Training

• Prior models have continued training from pretrained language-only LLMs or vision LLMs, while this work focuses on reusing pretrained LLMs with Transfusion without compromising text-only capabilities.Examples of earlier reuse include (Ge et al., 2023; Sun et al., 2023; Dong et al., 2023; Xie et al., 2024a; Wu et al., 2024b; He et al., 2024b).Source: §6 · p. 11

• LMFusion uses Llama-3 for text generation and parallel transformer modules for image diffusion, reusing computation invested in pretrained LLMs.Source: §7 · p. 11
• Its modular design enables language and vision modules to develop independently, de-risking large-scale joint-modality pretraining.Source: §7 · p. 11
• Built on existing text-only LLMs, LMFusion could also benefit from visual-understanding LLMs while generating interleaved text and visual content.Source: §7 · p. 11