Source-linked AI summary
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, Jiaqi Wang
TL;DR
Vision-language models have largely focused on pure text outputs, leaving richer image-text composition and broad multimodal knowledge insufficiently addressed. InternLM-XComposer combines interleaved composition with multilingual vision-language training and achieves strong performance across English and Chinese benchmarks, while its composition quality is competitive with public solutions.
Problem
Existing vision-language large models focus on pure text outputs, while vision-language datasets remain limited in volume and diversity.
Method
InternLM-XComposer combines instructed article generation, context-aware image placement and selection, and multilingual vision-language training using over 11 million semantic concepts.
Results
InternLM-XComposer achieves state-of-the-art performance across multiple English and Chinese vision-language benchmarks and competitive interleaved composition scores against GPT3.5 and GPT4-V.
Takeaways & Limitations
InternLM-XComposer provides a vision-language model that integrates advanced text-image comprehension with coherent interleaved image-text composition.
Takeaways & Limitations
The authors will release concept names and source websites rather than the underlying concept data because of copyright concerns.
Abstract
from arXiv · showhide
We propose InternLM-XComposer, a vision-language large model that enables advanced image-text comprehension and composition. The innovative nature of our model is highlighted by three appealing properties: 1) Interleaved Text-Image Composition: InternLM-XComposer can effortlessly generate coherent and contextual articles that seamlessly integrate images, providing a more engaging and immersive reading experience. Simply provide a writing instruction, and our system will generate the corresponding manuscript. It can intelligently identify the areas in the text where images would enhance the content and automatically insert the most appropriate visual candidates. 2) Comprehension with Rich Multilingual Knowledge: The text-image comprehension is empowered by training on an extensive multi-modal multilingual database with carefully crafted strategies, resulting in a deep understanding of visual content. 3) State-of-the-art Performance: Our model consistently achieves state-of-the-art results across various mainstream benchmarks for vision-language foundational models, including MME Benchmark, MMBench, MMBench-CN, Seed-Bench, CCBench (Chinese Cultural Benchmark), QBench and Tiny LVLM. Owing to the absence of established metrics for quantitatively assessing text-image composition, we have devised a robust evaluation procedure that comprises both human and GPT4-Vision (GPT4-V) to ensure reliability. Notably, our InternLM-XComposer achieves competitive text-image composition scores compared to public solutions, including GPT4-V and GPT3.5. Collectively, InternLM-XComposer seamlessly blends advanced text-image comprehension and composition, revolutionizing vision-language interaction and offering new insights and opportunities. The InternLM-XComposer model series are publicly available at https://github.com/InternLM/InternLM-XComposer.
1. Introduction
InternLM-XComposer extends vision-language models beyond pure text outputs by combining advanced text-image comprehension with interleaved composition. It generates instructed articles, selects contextually suitable images, incorporates multilingual knowledge, and reports strong benchmark performance.
- InternLM-XComposer addresses the limitation of vision-language models that produce pure text by enabling advanced text-image comprehension and composition.
- Given a writing instruction, the model generates long-form articles, identifies image placement locations, and supplies suitable image descriptions.
- CLIP retrieves candidate images first, after which InternLM-XComposer uses comprehension to select the image that best complements the content.
- A multilingual vision-language dataset containing over 11 million semantic concepts supplements existing datasets to broaden visual-language knowledge coverage.
- InternLM-XComposer achieves state-of-the-art results across English and Chinese vision-language benchmarks and competitive composition scores against GPT3.5 and GPT4-V.
2. Related Works
Related work spans large language models, vision-language large models, and image-text retrieval models. InternLM-XComposer is distinguished by combining multimodal architecture and training with composition beyond image-text matching.
- Large Language Models: Large language models evolved from Transformer-based encoder-decoder and decoder-only architectures toward larger-scale autoregressive systems with instruction-following capabilities.
- Vision Large Language Models: Vision-language models align visual and textual features, but the related work passage notes limitations on higher-level understanding tasks.
- InternLM-XComposer: InternLM-XComposer uses a visual encoder, perceive sampler, and large language model, with pre-training for visual-language alignment and SFT for different capabilities.
- Image-text Retrieval Models: Image-text retrieval models primarily match images with aligned descriptions, whereas image-text article composition requires selecting images for complex interleaved context.
3. Method
InternLM-XComposer combines a visual encoder, perceive sampler, and large language model with staged multimodal training. Its composition pipeline generates articles, identifies image locations and captions, narrows candidates with retrieval, and selects images using article context.
- 3.1. Model Architecture: The architecture contains a visual encoder, perceive sampler, and large language model.The visual encoder uses EVA-CLIP and processes images resized to 224 × 224.
- 3.2. Training: Pre-training aligns visual and language knowledge using image-text pairs and interleaved image-text data, followed by multitask and instruction-tuning stages.The pre-training corpus includes 1.1 billion images and 77.7 billion text tokens, while fine-tuning uses multiple instruction datasets.
- 3.2. Training: The pre-training data includes over 11 million vision-language concepts collected from public websites alongside public datasets and retained InternLM text data.The retained textual data is intended to preserve the initial language model’s capabilities.
- 3.2. Training: The fine-tuning setup freezes the pre-existing language-model weights and adds Low-Rank Adaptation while jointly training the perceive sampler.The visual encoder remains frozen during multimodal pre-training, concentrating optimization on the perceive sampler and language model.
- 3.3. Interleaved Image-Text Composition: For interleaved composition, the model first generates an article, then predicts image locations and captions before selecting images from retrieved candidates.Final selection considers preceding text and prior images to support thematic and visual coherence.
4. Experiments
InternLM-XComposer is evaluated across English and Chinese vision-language benchmarks, interleaved image-text composition, image selection, and component ablations. The reported results show strong benchmark performance, multilingual robustness, and competitive composition quality.
- English-Based Benchmark results: 137.11% on MME Benchmark establishes state-of-the-art performance, exceeding Qwen-VL-Chat by more than 5.0%.The model reaches Top-3 performance on 10 of MME's 14 subtasks.
- English-Based Benchmark results: 74.4% accuracy on the MMBench test set outperforms previous methods by a large margin.MMBench evaluates vision-related reasoning and perception with multi-choice questions.
- English-Based Benchmark results: InternLM-XComposer achieves the best overall Seed-Bench image-set performance and the highest performance on 6 of 9 subtasks.Because subtasks are imbalanced, the paper also reports task-level average accuracy, which reaches state-of-the-art performance.
- English-Based Benchmark results: The model reaches state-of-the-art performance on all three Q-Bench tasks and surpasses previous methods by a large margin.Q-Bench covers low-level vision through perception, description, and assessment.
- Chinese-Based Benchmark results: The English-to-Chinese performance gap is 2.0% for InternLM-XComposer, versus 4.9% for Qwen and 7.9% for VisualGLM on the corresponding benchmarks.On the Chinese cultural benchmark, InternLM-based models reach 47.6%, compared with 39.3% for Qwen-VL-Chat.
- Interleaved Image-Text Composition: GPT-4V performs best on text-related composition dimensions, while InternLM-XComposer performs best on image-related dimensions and ranks second on text-related dimensions and average score.The evaluation combines human and GPT-4V scoring across eight dimensions; image selection is additionally studied against human-selected images.
- Ablation studies: The Perceive Sampler substantially influences most benchmarks, while the Attention and FFN components of LoRA are also important for strong performance.The ablation evaluates learnable components during multi-task supervised fine-tuning.
5. Conclusion
InternLM-XComposer combines multimodal understanding and composition through multilingual, multimodal pretraining, generating coherent interleaved image-text content and achieving state-of-the-art performance across vision-language benchmarks.
- InternLM-XComposer is a vision-language large model with multimodal understanding and composition capabilities.
- Its carefully designed pretraining provides rich multilingual and multimodal knowledge.
- The model generates coherent compositions that interleave images and text.
- InternLM-XComposer achieves state-of-the-art performance across various vision-language large-model benchmarks.
A.1. Pre-training
Pretraining combines large-scale image-text data, interleaved text-image data, and additional InternLM text to align multimodal and language knowledge.
- Pretraining uses 1.1 billion images and 77.7 billion text tokens from public datasets and in-house concept data.
- The paired and interleaved data includes 50.6 billion English text tokens and 17.1 billion Chinese text tokens.
- The training procedure uses AdamW with cosine learning-rate scheduling and linear warm-up over the initial 200 steps.
- The maximum learning rates are 2e-4 for the perceive sampler and 4e-5 for the large language model.
A.2. Supervised Fine-tuning
Supervised fine-tuning uses LoRA across multi-task training and instruction tuning, with separate optimization of the perceive sampler.
- Supervised fine-tuning consists of Multi-task Training followed by Instruction Tuning.
- LoRA is applied to both attention layers and the feed-forward network.
- The LoRA rank and alpha are both set to 256.
- Instruction tuning uses batch size 256 for 1000 iterations with learning rate 1e−5.
B. Concept Data
The concept-data pipeline builds multilingual vision-language concepts from Wikipedia and Baidu Baike, cleans interleaved articles, and trains tasks for article generation, image spotting, captioning, and retrieval selection.
- Concept Data: The concept dataset contains 2 million English and 9 million Chinese vision-language concepts curated from Wikipedia and Baidu Baike.
- Concept Data: Each concept pairs an image with corresponding descriptions.
- Text Generation: GPT-4 identifies advertisement, reference, and recommendation noise in sentences, which are removed during text-generation data cleaning.
- Text Generation: Article generation takes an instruction and produces a sequence of paragraphs from the first through the last paragraph.
- Image Spotting and Captioning: Image spotting selects paragraph locations for insertion and generates captions for those locations.
- Image Retrieval and Selection: Image retrieval first obtains candidate images with CLIP, then selects the appropriate image from those candidates.
- Image Retrieval and Selection: The selected image is represented by an index chosen from the retrieved candidate images.
C.2. Image-Text Composition Evaluation Details
The evaluation combines human experts and GPT4-V to assess interleaved image-text article quality across text, image, and overall-preference dimensions. Scores use three levels and normalize text- and image-related dimensions separately.
- Evaluation dimensions: The grading criteria cover eight dimensions: four text-related, three image-related, and one subjective preference.The text dimensions are instruction following, writing quality, logic, and factualness; image dimensions are image-text consistency, informativeness, and consistency.
- Scoring scheme: Each dimension assigns Excellent, Medium, or Poor ratings worth 5, 3, or 1 points, respectively.The criteria define dimension-specific interpretations for instruction following, writing quality, logic, factualness, and image quality.
- Evaluation procedure: Human experts and GPT4-V grade articles double-blind, with text scores normalized by 20 points and image scores by 15 points.GPT4-V receives the instruction, article content, and explanations or scoring levels for all eight dimensions through a structured prompt.
- Image criteria: Image-text consistency evaluates whether illustrations are relevant to the article topic, while image informativeness evaluates relevance and added supplementary information.A top informativeness score requires the image to convey supplementary information beyond the article.
- Image criteria: Image consistency evaluates whether subjects remain consistent across accompanying pictures, with the highest score requiring complete consistency.The examples distinguish closely similar subjects from genuinely consistent subjects.
D. More Benchmark results.
Additional results show InternLM-XComposer-VL performs strongly across five ability categories in Tiny LVLM, achieving the best overall result and Top-3 performance in most abilities.
- Tiny LVLM: InternLM-XComposer-VL achieves the best overall result on Tiny LVLM and Top-3 performance in most of its five abilities.The abilities are visual reasoning, visual perception, visual knowledge acquisition, visual commonsense, and object hallucination.
- Tiny LVLM: Tiny LVLM evaluates multimodal large models across visual reasoning, visual perception, visual knowledge acquisition, visual commonsense, and object hallucination.The benchmark reports results for all listed subtasks in its test set.
E. More Qualitative Results.
Qualitative examples present InternLM-XComposer’s interleaved image-text compositions and conversational interactions across multilingual and varied topical content. The examples include generated articles, travel and historical material, and Chinese-language content.
- Interleaved composition: Figures A8–A13 show interleaved image-text compositions generated by InternLM-XComposer.The examples are presented as qualitative demonstrations of the model’s composition ability.
- Conversation: Figures A14 and A15 show conversation cases with InternLM-XComposer.The paper describes the model as exhibiting multilingual interleaved composition and conversation ability in these demonstrations.
- Multilingual and topical content: Other demonstrations cover London travel guidance, a 318 self-driving itinerary, Shanghai-related material, and Chinese descriptions of aviation and scenic locations.The Chinese examples include the C919 aircraft, Hailuogou, and enthusiastic travel-style prose.