Source-linked AI summary
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang
TL;DR
Open-source LVLMs face limitations in versatile vision-language comprehension and composition, especially with long-context input and output. IXC-2.5 addresses these gaps through extended interleaved image-text context, upgraded comprehension capabilities, and LoRA-based composition modules. It achieves strong cross-task results, including state-of-the-art performance on 16 of 28 benchmarks with a 7B backend.
Problem
Open-source LVLMs lack diverse vision-language comprehension and composition capabilities and struggle with long-context input and output.
Method
IXC-2.5 combines 24K interleaved image-text training, 96K positional-encoding extrapolation, upgraded comprehension abilities, and extra LoRA parameters for webpage and article composition.
Results
16 out of 28 benchmarks achieve state-of-the-art results among previous open-source LVLMs, using an InternLM2-7B backend.
Takeaways & Limitations
IXC-2.5 provides versatile comprehension and composition capabilities with long-contextual input and output using a relatively modest 7B LLM backend.
Abstract
from arXiv · showhide
We present InternLM-XComposer-2.5 (IXC-2.5), a versatile large-vision language model that supports long-contextual input and output. IXC-2.5 excels in various text-image comprehension and composition applications, achieving GPT-4V level capabilities with merely 7B LLM backend. Trained with 24K interleaved image-text contexts, it can seamlessly extend to 96K long contexts via RoPE extrapolation. This long-context capability allows IXC-2.5 to excel in tasks requiring extensive input and output contexts. Compared to its previous 2.0 version, InternLM-XComposer-2.5 features three major upgrades in vision-language comprehension: (1) Ultra-High Resolution Understanding, (2) Fine-Grained Video Understanding, and (3) Multi-Turn Multi-Image Dialogue. In addition to comprehension, IXC-2.5 extends to two compelling applications using extra LoRA parameters for text-image composition: (1) Crafting Webpages and (2) Composing High-Quality Text-Image Articles. IXC-2.5 has been evaluated on 28 benchmarks, outperforming existing open-source state-of-the-art models on 16 benchmarks. It also surpasses or competes closely with GPT-4V and Gemini Pro on 16 key tasks. The InternLM-XComposer-2.5 is publicly available at https://github.com/InternLM/InternLM-XComposer.
1. Introduction
IXC-2.5 addresses open-source LVLM limitations in versatility and long-context management by supporting diverse comprehension and composition tasks with extended image-text contexts. It upgrades high-resolution, video, and multi-image dialogue capabilities and achieves strong benchmark results with a 7B backend.
- Open-source LVLMs still lag in versatility because of limited training diversity and difficulties managing long-context input and output.
- IXC-2.5 supports text-image conversation, OCR, video understanding, illustrated article composition, and webpage crafting.
- 24K interleaved image-text training contexts extend to 96K through positional encoding extrapolation.
- Compared with IXC-2.0, IXC-2.5 adds ultra-high-resolution understanding, fine-grained video understanding, and multi-turn multi-image dialogue.
- Extra LoRA parameters support webpage generation and high-quality text-image article composition, with CoT and DPO improving written-content quality.
- 16 out of 28 benchmarks achieve state-of-the-art results among previous open-source LVLMs, while matching or surpassing proprietary APIs on 16 benchmarks.
2. Related Works
The related-work passages place IXC-2.5 within the development of vision-language models and illustrate its focus on high-resolution, video, and webpage-generation capabilities.
- LVLMs integrate LLMs with vision encoders to extend language models toward understanding visual content and text-image conversation.
- IXC-2.5 capabilities include chat on 4K-resolution images, multi-turn multi-image dialogue, video, resume-to-homepage generation, and instruction-aware webpage generation.
Article Composing
The supplied passages combine article-composing context with visual-design and webpage-generation material, but do not provide a coherent account of the article-composing section itself.
- Video Understanding: Video analysis can represent sampled frames as a high-resolution composite picture, supporting fine-grained visual understanding.
3. Method
IXC-2.5 combines a 7B language backend with unified multimodal processing, long-context training, and specialized LoRA-based composition capabilities. Its method targets high-resolution images, fine-grained videos, multi-image inputs, webpages, and text-image articles.
- Model Architecture: IXC-2.5 uses a lightweight vision encoder, InternLM2-7B, and Partial LoRA for efficient vision-language alignment.The architecture mainly follows InternLM-XComposer2 and IXC2-4KHD.
- Multi-modal Input: The unified dynamic image partition strategy handles text, single or multiple images, and videos with varying resolutions and aspect ratios.Images are partitioned into 560 × 560 sub-images, while videos are represented by concatenated sampled frames.
- Training: Training freezes the LLM during pre-training while fine-tuning the vision encoder and Partial LoRA, then jointly fine-tunes components with a 24,000-token context window.Fine-tuning supports extremely large images and videos concatenated from at most 64 frames.
- Webpage Generation: Webpage generation uses LoRA to produce HTML, CSS, and JavaScript from screenshots, free-form instructions, or resumes.Instruction-aware data include 18,000 generated samples, while the resume-to-homepage dataset contains 2,000 samples.
- Article Composing: Article composition addresses scarce high-quality data with supervised fine-tuning, Chain-of-Thought prompt augmentation, response sampling, reward modeling, and DPO alignment.The pipeline starts from 5,000 instruction samples and constructs 30,000 preference pairs for DPO.
4. Experiments
IXC-2.5 is evaluated across video, structural high-resolution, webpage-generation, and broader multimodal benchmarks. It shows competitive or state-of-the-art performance against open-source models and closed-source APIs.
- Evaluation Setup: 28 benchmarks cover video understanding, structural high-resolution understanding, general visual question answering, multi-image dialogue, and webpage crafting.Evaluation uses OpenCompass VLMEvalKit for unified reproduction of results.
- Video Understanding: IXC-2.5 outperforms open-source models on 4 of 5 video benchmarks and remains on par with closed-source APIs.On MVBench, it reaches 69.1, 8.7% above VideoChat2-7B and 25.6% above GPT-4V.
- Structural High-resolution Understanding: 90.9% on DocVQA matches InternVL-1.5 despite IXC-2.5 having only 7B parameters and nearly four times fewer parameters than that model.It also exceeds DocOwl 1.5-8B on WikiTableQuestion, DeepForm, and TableFace.
- Webpage Generation: IXC-2.5 demonstrates potential for bridging visual design and code implementation in webpage-generation evaluation.Table 5 compares screenshot-to-code systems on the Design2Code benchmark.
5. Conclusion
IXC-2.5 combines long-contextual input and output with advanced vision-language comprehension and composition features. The authors position it as a promising direction for more contextual multimodal environments and real-world assistance.
- IXC-2.5 supports long-contextual input and output for ultra-high-resolution image understanding, fine-grained video understanding, multi-turn multi-image dialogue, webpage generation, and article composing.
- The model achieves competitive performance with a relatively modest 7B large language model backend.
- The authors identify long-context video understanding and extended interaction histories as future applications of the model’s contextual multimodal capabilities.