Source-linked AI summary
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, Jiaqi Wang
TL;DR
InternLM-XComposer2 addresses the need for vision-language models that support both free-form text-image composition and comprehension. It combines a vision encoder and InternLM2 with Partial LoRA applied only to image tokens, and reports strong composition and multimodal benchmark performance, including results matching or surpassing advanced closed-source models on some assessments.
Problem
Existing multimodal language models are limited in combining practical free-form text-image composition with strong vision-language comprehension.
Method
InternLM-XComposer2 combines InternLM2, a CLIP-based vision encoder, and Partial LoRA that adds adaptation exclusively to image tokens.
Results
InternLM-XComposer2 significantly outperforms existing open-source MLLMs and matches or surpasses GPT-4V and Gemini Pro in various benchmarks.
Takeaways & Limitations
The model provides integrated, customizable text-image generation alongside detailed perception, logical reasoning, and multimodal knowledge integration.
Abstract
from arXiv · showhide
We introduce InternLM-XComposer2, a cutting-edge vision-language model excelling in free-form text-image composition and comprehension. This model goes beyond conventional vision-language understanding, adeptly crafting interleaved text-image content from diverse inputs like outlines, detailed textual specifications, and reference images, enabling highly customizable content creation. InternLM-XComposer2 proposes a Partial LoRA (PLoRA) approach that applies additional LoRA parameters exclusively to image tokens to preserve the integrity of pre-trained language knowledge, striking a balance between precise vision understanding and text composition with literary talent. Experimental results demonstrate the superiority of InternLM-XComposer2 based on InternLM2-7B in producing high-quality long-text multi-modal content and its exceptional vision-language understanding performance across various benchmarks, where it not only significantly outperforms existing multimodal models but also matches or even surpasses GPT-4V and Gemini Pro in certain assessments. This highlights its remarkable proficiency in the realm of multimodal understanding. The InternLM-XComposer2 model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.
1. Introduction
InternLM-XComposer2 targets free-form text-image composition and comprehension, generating integrated multimodal content from flexible inputs while strengthening vision-language understanding. Its Partial LoRA design and diverse training data support this combined capability.
- Model capabilities: InternLM-XComposer2 generates high-quality integrated text-image articles from specifications, outlines, and reference images.It is designed for varied application contexts and supports detailed perception, logical reasoning, and knowledge integration.
- Key design elements: Its Partial LoRA design applies additional adaptation to image tokens while preserving the original language-token architecture.This design is presented as harmonizing composition and comprehension.
- Key design elements: The composition data emphasize complex instruction following, customization, stylistic diversity, and text editing.The comprehension data include broad high-quality pretraining and supervised fine-tuning multimodal data.
- Evaluation: InternLM-XComposer2 significantly exceeds existing open-source MLLMs and matches or surpasses GPT-4V and Gemini Pro in various benchmarks.The evaluation covers creation and multimodal comprehension across a broad benchmark set.
2. Related Works
The related work positions InternLM-XComposer2 within progress from language and vision-language models toward multimodal systems that combine perception, understanding, reasoning, and customizable text-image composition.
- Large Language Models: Modern LLMs use decoder-only autoregressive architectures, instruction tuning, and reinforcement learning from human feedback.Open-source systems have also enabled broader customization of language models.
- Multimodal Large Language Models: Vision-language models align image and text features in a shared embedding space through contrastive learning on image-text pairs.These models support strong zero-shot and few-shot generalization across downstream tasks.
- Multimodal Large Language Models: Recent MLLMs build on language models and visual encoders to perform visual perception, understanding, and reasoning across diverse tasks.Research has explored instruction tuning, efficient fine-tuning, high-resolution inputs, hallucination mitigation, and image generation.
- Free-form composition: InternLM-XComposer2 combines InternLM2 with CLIP ViT-Large and partial LoRA to generate customized interleaved text-image content from multimodal inputs.Inputs may include specifications, outlines, and reference images.
3. Method
InternLM-XComposer2 aligns visual and language modalities through a vision encoder, InternLM2, and Partial LoRA applied to visual tokens. Its staged training combines alignment, multimodal tasks, and free-form composition data to preserve language ability while enabling image-grounded generation.
- 3.1. Model Architecture: The model concatenates visual tokens from a CLIP-based vision encoder with tokenized text before processing them with InternLM2.The implementation uses OpenAI ViT-Large as the vision encoder and InternLM2-7B-Chat-SFT as the language model.
- 3.1. Model Architecture: Partial LoRA applies low-rank adaptation exclusively to the new-modality portion of the input, specifically all visual tokens.This addresses the trade-off between treating modalities identically and imposing a costly separate alignment mechanism.
- 3.1. Model Architecture: For visual tokens, the adapted layer adds W_BW_Ax_v to the original linear transformation, while language tokens retain the unadapted transformation.The input is partitioned into visual tokens x_v and language tokens x_t.
- 3.2. Pre-Training: During pre-training, the language model remains fixed while the vision encoder and Partial LoRA align visual tokens with language.The data target semantic alignment, world-knowledge alignment, and vision-capability enhancement.
- 3.2. Pre-Training: Partial LoRA enables adaptation to visual tokens while maintaining original language processing capabilities.The fixed language model also permits direct use of in-context learning performance to assess pre-training quality.
- 3.3. Supervised Fine-tuning: Supervised fine-tuning proceeds through multi-task training and free-form text-image composition while jointly tuning the vision encoder, language model, and Partial LoRA.InternLM2 supervised fine-tuning data account for a fixed 10% of the SFT data, and composition data combine pure-text and vision-language conversations.
- 3.4. Free-form Text-Image Composition: Free-form composition generates interleaved text and images customized to user-provided titles, outlines, writing materials, and optional image resources.The dataset covers varied writing styles, flexible editing, complex instruction adherence, and customization with text and images.
- 3.4. Free-form Text-Image Composition: User-provided image materials are inserted instead of retrieved images, and the composition data distribute approximately equally across four dimensions.The reported ratio across the dimensions is approximately 1:1:1:1.
4. Experiments
InternLM-XComposer2 is evaluated across multimodal understanding, hallucination, and creative-writing benchmarks, showing strong performance against open- and closed-source systems.
- The evaluation covers MathVista, MMMU, AI2D, MME, MMBench, SEED-Bench, LLaVA-Bench, QBench, MM-Vet, HallusionBench, ChartQA, and POPE.
- InternLM-XComposer2 outperforms competitors in 10 out of 11 open-source benchmark comparisons.
- 57.6% on MathVista and 78.9 on AI2D outperform the compared closed-source APIs, while MMMU reaches 43.0%.
- 87.7 average F1-score across the three POPE tracks establishes a new state-of-the-art result, while HallusionBench surpasses all open-source models.
- CreationBench evaluates average performance alongside Creativity, Richness, User Demand Fulfillment, and Logical Coherence, with and without GPT-4 reference answers.
- 6.24 overall without a GPT-4 referenced answer demonstrates strong CreationBench performance for creativity and logical structure.
5. Conclusion
InternLM-XComposer2 combines free-form text-image composition with vision-language understanding through Partial LoRA, which adds parameters only to image tokens. Across benchmarks, it outperforms existing multimodal models and matches or surpasses GPT-4V and Gemini Pro in some assessments.
- InternLM-XComposer2 demonstrates capabilities in vision-language understanding and free-form text-image composition.
- Partial LoRA applies additional LoRA parameters exclusively to image tokens, preserving pre-trained language knowledge while balancing visual understanding and text composition.
- The model supports highly customizable content creation and high-quality long-text multimodal generation.
- Across various benchmarks, InternLM-XComposer2 significantly outperforms existing multimodal models and matches or surpasses GPT-4V and Gemini Pro in certain assessments.