Source-linked AI summary
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, Armen Aghajanyan
TL;DR
Multimodal autoregressive models have faced a trade-off between strong image quality and high training cost, while their broader task generality remains limited. CM3Leon addresses this with retrieval-augmented pretraining, diverse multi-task supervised fine-tuning, and self-contained contrastive decoding. It achieves state-of-the-art text-to-image generation with 5x less training compute and supports controllable multimodal tasks after fine-tuning.
Problem
Autoregressive image generators can produce strong results and global coherence but are much more expensive to train and use than diffusion models, while broader multimodal task generality remains limited.
Method
CM3Leon combines a retrieval-augmented decoder-only multimodal model with large-scale pretraining, multi-task supervised fine-tuning, and self-contained contrastive decoding.
Results
CM3Leon achieves state-of-the-art text-to-image generation with 5x less training compute than comparable methods, reaching a zero-shot MS-COCO FID of 4.88, while SFT enables strong controllability across multimodal tasks.
Takeaways & Limitations
The results support autoregressive models as efficient, performant candidates for a broad range of text and image generation tasks.
Takeaways & Limitations
The study uses only licensed Shutterstock images and text, constraining its data sourcing to that setting.
Abstract
from arXiv · showhide
We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented, token-based, decoder-only multi-modal language model capable of generating and infilling both text and images. CM3Leon uses the CM3 multi-modal architecture but additionally shows the extreme benefits of scaling up and tuning on more diverse instruction-style data. It is the first multi-modal model trained with a recipe adapted from text-only language models, including a large-scale retrieval-augmented pre-training stage and a second multi-task supervised fine-tuning (SFT) stage. It is also a general-purpose model that can do both text-to-image and image-to-text generation, allowing us to introduce self-contained contrastive decoding methods that produce high-quality outputs. Extensive experiments demonstrate that this recipe is highly effective for multi-modal models. CM3Leon achieves state-of-the-art performance in text-to-image generation with 5x less training compute than comparable methods (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate unprecedented levels of controllability in tasks ranging from language-guided image editing to image-controlled generation and segmentation.
1 Introduction
CM3Leon adapts text-model training ideas to a retrieval-augmented, decoder-only multimodal model that generates and infills text and images. Scaling, diverse data, and instruction tuning enable efficient generation across varied multimodal tasks.
- CM3Leon is a retrieval-augmented, token-based, decoder-only multimodal model that generates and infills both text and images.
- Its training recipe combines large-scale retrieval-augmented pretraining with multi-task supervised fine-tuning on diverse instruction-style data.
- 5x less training compute yields state-of-the-art text-to-image generation, with a zero-shot MS-COCO FID of 4.88.
- After supervised fine-tuning, CM3Leon supports controllable language-guided image editing, image-controlled generation, and segmentation.
- The model also performs non-trivial image-to-text generation despite training on only 3B Shutterstock text tokens.
2 Pretraining
CM3Leon adapts token-based, decoder-only multimodal modeling with retrieval augmentation, cross-modal infilling, and scaled training. Its design uses licensed image data and reports better scaling than several text-to-image baselines.
- Architecture and training: CM3Leon builds on RA-CM3 by simplifying its objective, modifying the dataset, and incorporating multimodal scaling-law insights.The model explores token-based decoder-only architectures for text-to-image generation.
- Retrieval augmentation: Retrieval augmentation selects relevant, multimodal, and diverse documents from a memory bank using dense CLIP-based retrieval.The retriever separately encodes document text and images with frozen CLIP encoders and averages their representations.
- Retrieval augmentation: During training, CM3Leon samples three retrieved documents per caption-image pair, effectively quadrupling the pretraining tokens available.Two documents are retrieved based on image and text, respectively, before three samples are randomly selected for each example.
- Training objective: The CM3 objective converts multimodal inputs into infilling instances and applies standard next-token prediction to support both infilling and autoregressive generation.The same objective covers image and text generation, including caption-to-image generation.
- Training objective: CM3Leon removes query-pair loss upweighting from RA-CM3 because it adversely affects zero-shot image generation without retrieval.The method also avoids masking across modality-boundary <break> tokens, which could enable generation from an arbitrary midpoint.
- Scaling and implementation: The 350M, 760M, and 7B models are trained on 1.4T, 1.9T, and 2.4T tokens, respectively, using stable hyperparameters adapted from multimodal scaling-law work.The models use a decoder-only transformer architecture with a sequence length of 4096.
3 Text-To-Image Results
CM3Leon’s text-to-image results show that decoding choices and retrieval augmentation materially improve autoregressive generation. The model reaches a 4.88 zero-shot MS-COCO FID while using less training compute than comparable methods.
- Importance of Decoding Strategies: Classifier-free guidance blends conditional and unconditional logits to steer generation toward the input text.The unconditional stream replaces the text with a mask token, enabling guidance without fine-tuning.
- Importance of Decoding Strategies: CD-K modifies contrastive decoding by using text-conditioned and unconditioned probabilities rather than strong and weak models.The modification relaxes the original candidate constraint because it otherwise consistently becomes greedy decoding.
- Importance of Decoding Strategies: CD-K remains competitive with CFG and complements it, allowing FID to keep decreasing as the number of generated samples increases.CFG and CD-K independently stagnate as sample counts grow, whereas their combination continues improving FID.
- Quantitative Evaluations: 4.88 FID establishes CM3Leon-7B as state of the art on zero-shot MS-COCO text-to-image generation.The evaluation uses the zero-shot MS-COCO (30K) task and Fréchet Inception Distance.
- Quantitative Evaluations: Retrieval-augmented CM3Leon-7B achieves superior FID with one or two retrieved examples during inference.The result is reported as evidence that retrieval expands the model’s world knowledge for image generation.
- Quantitative Evaluations: The reported MS-COCO comparison generates eight samples per query and uses CLIP to select the best generation.The table summarizes text-to-image models using FID on zero-shot MS-COCO.
4 Supervised Fine-Tuning
CM3Leon is fine-tuned on diverse interleaved image-text, image-generation, image-editing, and vision-language tasks using the same objective as pretraining. The resulting SFT model shows controllable image generation and strong zero-shot vision-language performance, including outperforming Flamingo on VizWiz.
- Fine-tuning mixture: SFT combines interleaved image-text, text-to-image, image-to-text, editing, grounding, and spatial-generation tasks in one fine-tuning mixture.Tasks include text-guided editing, image-to-image grounded generation, spatially grounded generation, and how-to-write generation.
- Fine-tuning procedure: The fine-tuning process concatenates each task instruction with its output and trains with the same CM3 objective used during pretraining.Examples are organized as series of interleaved text and image inputs and outputs.
- Image generation and editing: Text-guided editing modifies images according to instructions, while grounded generation uses features such as edges, segmentation maps, key points, and human poses.The editing data contains around 600,000 examples, while ControlNet-based grounded-generation data contains 7 million examples.
- Image generation and editing: Qualitative results show edited images that preserve the original image while following text instructions, and distinct generations that preserve a shared input pose across prompts.The examples use separate image and text CFG values for text-guided editing and a single CFG value for structure-guided editing.
- Conditional text generation: SFT-CM3Leon achieves strong zero-shot performance on several vision-language tasks despite seeing approximately 3B text tokens, compared with 100B for Flamingo and 40B for OpenFlamingo.It surpasses Flamingo on VizWiz and can generate captions or answer varied image-conditioned questions.
5 Related Work
Related work spans diffusion, autoregressive token, and non-autoregressive token models for image generation, including retrieval-augmented approaches. CM3Leon builds on retrieval-augmented decoder-only models while focusing primarily on scaling this strategy.
- Diffusion models: Diffusion models generate images by adding noise sequentially and learning to reverse it from text inputs or other features.They have achieved significant progress in text-to-image generation and commonly use pretrained text or language representations.
- Retrieval augmentation: Retrieval augmentation addresses the difficulty of representing rare concepts and entities by incorporating retrieved examples during inference or training.Prior work reports efficiency gains when decoder-only text-to-image models are prefixed with statically retrieved instances during training.
- Token-based models: Autoregressive token models convert images into discrete latent variables, turning text-to-image generation into a sequence-to-sequence problem.Their related work is presented alongside newer non-autoregressive alternatives that aim to reduce decoding cost.
- CM3Leon’s position: CM3Leon builds on RA-CM3 by streamlining its objective, modifying the dataset, and incorporating multi-modal scaling-law insights.The paper states that its primary focus is scaling this strategy.
6 Conclusion
CM3Leon combines retrieval-augmented pretraining, diverse Shutterstock data, multi-task SFT, and contrastive decoding to generate and infill text and images. The conclusion highlights its efficiency, flexibility, and applicability across text and image tasks.
- CM3Leon efficiently and flexibly generates and infills both text and images.
- The training approach integrates retrieval-augmented pretraining, a diverse large-scale Shutterstock dataset, and multi-task supervised fine-tuning.
- Contrastive decoding further improves text and image generation quality.
B.1 Data Visualizations
The appendix visualizes tokenization, model and inference configurations, and qualitative image-generation prompts. These materials cover both system operation and example outputs across varied scenes and subjects.
- Figure 9 visualizes tokenization for a caption-image pair.
- Figure 10 visualizes tokenization for a full training sample containing retrieved and query caption-image pairs.
E.1 Hyper-Parameters
The appendix documents training and fine-tuning configurations for CM3Leon models. It describes model-training settings and provides a separate fine-tuning-parameter table.
- Training uses 4096-token sequences, balanced-data up/down sampling, and 64 or 128 80GB A100 GPUs for the 760M and 7B models.
- Preliminary validation experiments select learning rates and per-GPU batch sizes from specified candidate ranges.
- Table 4 lists fine-tuning parameters for CM3Leon models.
E.3 More Qualitative Samples
The appendix provides qualitative samples from finetuned CM3Leon models. It includes general generations as well as image-captioning and visual-question-answering examples.
- Figure 15 presents qualitative examples from the finetuned CM3Leon-7B model.Human faces are blurred to remove personally identifiable information.
- Table 5 documents datasets and prompt templates used for supervised fine-tuning.
- Figure 16 shows SFT-CM3Leon-7B generations for image captioning and visual question answering.Human faces are blurred to remove personally identifiable information.