Source-linked AI summary
Chameleon: Mixed-Modal Early-Fusion Foundation Models
Chameleon Team
TL;DR
Existing multimodal models often separate modalities, limiting arbitrary interleaving of images and text. Chameleon uses an early-fusion, fully token-based architecture with stability and alignment techniques, achieving strong vision-language and text-only performance while enabling mixed-modal generation. Its human evaluation shows preference over Gemini-Pro and GPT-4V for long-form mixed-modal responses, while comparisons rely partly on augmented non-native baselines.
Problem
Existing multimodal foundation models often use modality-specific encoders or decoders, limiting integration across modalities and generation of documents with arbitrary image-text sequences.
Method
Chameleon represents image, text, and code modalities as tokens in a shared early-fusion transformer, using architectural, optimization, and alignment techniques for stable scalable training.
Results
Chameleon shows state-of-the-art vision-language performance, competitive text-only performance, and new mixed-modal reasoning and generation capabilities in one model.
Takeaways & Limitations
Chameleon demonstrates that a single unified token-based model can support broad vision-language capabilities alongside interleaved image-text reasoning and generation.
Takeaways & Limitations
Human-evaluation prompts came from crowdsourcing with potentially limited coverage, and comparisons used baselines augmented with separately generated images rather than native mixed-modal models.
Abstract
from arXiv · showhide
We present Chameleon, a family of early-fusion token-based mixed-modal models capable of understanding and generating images and text in any arbitrary sequence. We outline a stable training approach from inception, an alignment recipe, and an architectural parameterization tailored for the early-fusion, token-based, mixed-modal setting. The models are evaluated on a comprehensive range of tasks, including visual question answering, image captioning, text generation, image generation, and long-form mixed modal generation. Chameleon demonstrates broad and general capabilities, including state-of-the-art performance in image captioning tasks, outperforms Llama-2 in text-only tasks while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and performs non-trivial image generation, all in a single model. It also matches or exceeds the performance of much larger models, including Gemini Pro and GPT-4V, according to human judgments on a new long-form mixed-modal generation evaluation, where either the prompt or outputs contain mixed sequences of both images and text. Chameleon marks a significant step forward in a unified modeling of full multimodal documents.
1 Introduction
Chameleon is designed as a unified early-fusion model that represents images, text, and code as tokens, enabling reasoning over and generation of arbitrarily interleaved multimodal documents. Its architectural and training innovations support broad performance across vision-language, text-only, and mixed-modal evaluations.
- Motivation and approach: Chameleon addresses the limitation of separately modeled modalities by using a shared architecture for arbitrarily interleaved image-text documents.The approach targets full multimodal document modeling rather than isolated image understanding, image generation, or text-only tasks.
- Motivation and approach: Images are quantized into discrete tokens so the same transformer can process image and text sequences without separate modality-specific encoders or decoders.All modalities are projected into a shared representational space from the start, though this creates optimization and scaling challenges.
- Motivation and approach: Query-key normalization, revised layer-norm placement, and adapted supervised finetuning enable stable training and alignment for mixed-modal models.Using these techniques, Chameleon-34B was trained on 5x as many tokens as Llama-2 while matching or outperforming existing LLMs on unimodal benchmarks.
- Evaluation: Chameleon-34B achieves state-of-the-art results on visual question answering and image captioning while remaining competitive on text-only benchmarks.It outperforms Flamingo, IDEFICS, and Llava-1.5 on the cited vision-language tasks and matches Mixtral 8x7B and Gemini-Pro on selected text-only tasks.
- Evaluation: A human evaluation of open-ended mixed-modal responses reports preference rates of 60.4% against Gemini-Pro and 51.6% against GPT-4V.The evaluation measures long-form responses containing interleaved images and text, complementing static public benchmarks.
- Significance: The work positions Chameleon as a step toward unified foundation models that flexibly reason over and generate multimodal content.The contribution includes a new large-scale human evaluation for open-ended mixed-modal reasoning and generation.
2 Pre-Training
Chameleon uses discrete image and text tokens in one early-fusion autoregressive model, trained on text-only, paired, and interleaved data. Stable scaling requires modifications addressing norm growth and softmax-related divergences.
- Data and tokenization: Chameleon represents images as discrete tokens and trains on arbitrary orderings ranging from text-only sequences to fully interleaved text-image documents.Images are encoded as 1024 discrete tokens from an 8192-entry codebook.
- Data and tokenization: The pre-training mixture combines 2.9 trillion text-only tokens, 1.5 trillion text-image tokens, and 400 billion interleaved text-image tokens.The second training stage lowers the first-stage data weight by 50% and adds higher-quality datasets while preserving a similar image-text token proportion.
- Stability: The standard LLaMA parameterization diverged during multimodal training because shared weights caused modality-dependent norm growth around softmax operations.The instability became especially problematic when scaling beyond 8B parameters and 1T tokens.
- Stability: Query-key normalization controls attention softmax input norms, and Chameleon-7B without it diverges after approximately 20% of a training epoch.The normalization is applied to query and key vectors within attention.
- Inference: The inference strategy is augmented for interleaved generation to improve throughput and reduce latency during alignment and evaluation.
3 Alignment
Chameleon aligns its mixed-modal capabilities through supervised fine-tuning on curated datasets spanning text, code, visual chat, image generation, interleaved generation, and safety. Balancing modality pairings is important because severe imbalance can bias the model toward generating one modality.
- Dataset composition: The supervised fine-tuning dataset covers Text, Code, Visual Chat, Image Generation, Interleaved Text/Image Generation, and Safety.Examples from these categories are presented in Figure 7.
- Dataset composition: Image-generation data selects the top 64K licensed images rated at least six by an aesthetic classifier and close to 512 × 512 in size and aspect ratio.
- Dataset composition: Visual Chat and interleaved generation data use high-quality third-party collection, and the authors exclude Meta user data.
- Dataset composition: Safety tuning pairs sensitive prompts with refusal responses across topics including violence, controlled substances, privacy, and sexual content.
- Fine-tuning strategy: Balancing modality pairings during supervised fine-tuning is important for high-quality alignment.Severe imbalance can teach an unconditional prior that mutes or exaggerates generation of a single modality.
- Fine-tuning strategy: The fine-tuning objective autoregressively masks prompt-token loss and optimizes primarily on answer tokens.Images in prompts are padded, while answer images are center-cropped for generation quality.
4 Human Evaluations and Safety Testing
Chameleon is evaluated on diverse human-authored prompts requiring mixed-modal responses, where it generally outperforms augmented and original Gemini and GPT-4V baselines. The evaluation also reveals category gaps, annotation ambiguity, safety results, and important coverage limitations.
- Evaluation setup: 1,048 prompts comprise 441 mixed-modal and 607 text-only cases after majority-vote filtering for clarity and expected image use.Prompts span 12 manually defined task categories and were collected from human annotators for real-life scenarios.
- Absolute evaluation: 55.2% of Chameleon responses completely fulfilled tasks, versus 37.6% for Gemini+ and 44.7% for GPT-4V+.Original text-only Gemini and GPT-4V responses fully fulfilled 17.6% and 23.1% of tasks, respectively.
- Absolute evaluation: Chameleon performs well on Brainstorming, Comparison, and Hypothetical categories but needs improvement on Identification and Reasoning.Performance differs little by input modality, although Chameleon is slightly better on text-only prompts while augmented baselines are slightly better on mixed-modal prompts.
- Relative evaluation: Chameleon’s overall win rates are 60.4% against Gemini+ and 51.6% against GPT-4V+ in pairwise human comparisons.Against original Gemini and GPT-4V responses, its win rates are 69.1% and 61.7%, respectively.
- Inter-annotator agreement: More than 10% of relative-evaluation pairs have no annotator agreement, while 55%–60% have one annotator differing from the other two.The authors interpret this pattern as indicating similar performance in many cases and challenging relative evaluation.
- Discussion: The human evaluation excludes or limits some visual-understanding coverage, including OCR and infographics, and uses crowdsourced rather than real-user prompts.Existing baseline APIs also provide text-only responses, so their image outputs are separately generated rather than native.
5 Benchmark Evaluations
Benchmark evaluations show that Chameleon remains competitive on text-only tasks and image-to-text capabilities while achieving strong results across several multimodal benchmarks. Its strongest results are in image captioning, with competitive but more mixed performance on visual question answering.
- Text-only benchmarks: Chameleon-34B outperforms Llama-2 70B on 5 of 8 commonsense-reasoning and reading-comprehension tasks and performs on par with Mixtral 8x7B.The evaluation covers zero-shot performance across eight benchmarks in these task families.
- Text-only benchmarks: 61.4 versus 56.8 maj@1 and 77.0 versus 75.1 maj@32 show Chameleon-34B outperforming Llama-2 70B and Mixtral 8x7B on GSM8K.Chameleon-7B also outperforms corresponding Llama-2 models on GSM8K and matches Mistral 7B at maj@8.
- Text-only benchmarks: 65.8 versus 70.6/71.8 on MMLU places Chameleon-34B below Mixtral 8x7B and Gemini-Pro while above its Llama-2 counterpart.The paper reports that both Chameleon models outperform their corresponding Llama-2 models.
- Image captioning: Chameleon-34B outperforms larger Flamingo and IDEFICS models on COCO image captioning using 2 shots versus their 32 shots.Fine-tuned and multitask Chameleon-34B variants outperform all other models on COCO, while the SFT variant leads on Flickr30k.
- Visual question answering: On VQA-v2, pretrained Chameleon-34B with 2 shots matches larger Flamingo and IDEFICS models at 32 shots, while fine-tuned performance approaches IDEFICS-80B-Instruct and Gemini Pro.It trails larger models including Flamingo-80B-FT, GPT-4V, and Gemini Ultra.
- Multimodal benchmarks: Overall, Chameleon is competitive on image captioning and VQA with fewer in-context examples and smaller model sizes than several comparison systems.The comparison includes pretrained and fine-tuned evaluations.
6 Related Work
Chameleon extends token-based multimodal learning into a fully token-based early-fusion model for arbitrary interleaved image-text sequences. It differs from related systems through unified modality processing and an end-to-end dense design.
- Token-based multimodal learning: Prior work explored discrete image tokens and mixed-modal documents with interleaved image and text tokens.Chameleon builds on this token-based lineage while extending model scale and architecture design.
- Early fusion: Unlike late-fusion models such as Flamingo, Chameleon encodes image and text in a shared token space rather than combining separately encoded modalities later.LLaVA, IDEFICS, and VisualGPT likewise maintain separate image and text encoders.
- Implications: Chameleon’s unified architecture supports reasoning over and generation of interleaved image-text sequences without modality-specific components.The paper identifies representation learning and alignment as challenges associated with this early-fusion approach.
- Comparison with Gemini: Chameleon and Gemini both use early-fusion token-based approaches, but Gemini uses separate image decoders whereas Chameleon is an end-to-end dense model without routing components.The paper presents Chameleon as more general-purpose across multimodal understanding and generation tasks.
7 Conclusion
The paper concludes that Chameleon combines unified image-text token modeling with stable early-fusion training to support broad multimodal capabilities in one model. Its strongest reported advantages span image captioning, visual question answering, text-only benchmarks, and mixed-modal open-ended interaction.
- Conclusion: Chameleon learns a unified representation over interleaved image and text tokens while achieving strong vision-language benchmark performance in a single model.The model also enables mixed-modal reasoning and generation capabilities.
- Conclusion: Quantizing images into discrete tokens and training on mixed-modal data from scratch supports joint image-text reasoning within a fully token-based architecture.The paper also introduces techniques for stable and scalable training of early-fusion models.
- Conclusion: Chameleon-34B outperforms Flamingo and IDEFICS on image captioning and visual question answering while remaining competitive on text-only benchmarks.The paper presents strong performance on a new benchmark for mixed-modal open-ended question answering.
A Samples
The samples show Chameleon responding to prompts that combine textual answers with generated images across animal and plant-related tasks. Responses provide factual descriptions, troubleshooting steps, and visual examples.
- Animal information and images: Mountain Cur responses combine breed information with generated images depicting the dogs in different settings.The responses describe appearance, temperament, exercise needs, and uses before providing example images.
- Animal information and images: The polar-bear response answers a factual color question and supplies a photograph of the animal in the wild.It also explains camouflage and the relationship between white fur and the Arctic environment.
- Plant troubleshooting: Plant-related prompts receive diagnosis and care guidance based on either a textual description or an input image.The pothos guidance covers watering, soil, light, fertilization, and pruning as possible troubleshooting steps.
B Additional Information of Human Evaluations
The human evaluation organizes prompts into twelve task categories and reports fulfillment and win rates by category and modality. Results are presented in separate tables for task fulfillment, modality fulfillment, and comparisons against Gemini and GPT-4V.
- Evaluation design: Twelve prompt task categories are described for the human evaluation.Table 8 provides the short description of each category.
- Evaluation design: Task fulfillment rates are broken down by task category and modality.The breakdown is reported in Tables 9 and 10.
- Comparative results: Chameleon’s win rates are broken down by task category and modality across four comparison tables.Tables 11–14 compare Chameleon with Gemini+, GPT-4V+, Gemini, and GPT-4V.