Source-linked AI summary
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Zhe Chen, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Kai Chen, Conghui He, Xingcheng Zhang, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang
TL;DR
InternLM-XComposer2-4KHD addresses the difficulty LVLMs face with fine-grained visual content by extending resolution support from 336 pixels to 4K HD and beyond. It uses dynamic resolution with automatic patch configuration and shows consistent gains as training resolution increases, with competitive performance across OCR-related and general-purpose benchmarks.
Problem
Limited LVLM resolution hampers processing of fine-grained visual content in charts, tables, documents, and infographics, constraining practical applicability.
Method
InternLM-XComposer2-4KHD uses dynamic resolution with automatic patch configuration, preserving image aspect ratios while varying patch counts and layouts across resolutions from 336 pixels to 4K HD.
Results
The model’s performance consistently improves as training resolution increases for HD-OCR tasks, with no saturation observed at 4KHD, while achieving competitive results across general-purpose and high-resolution benchmarks.
Takeaways & Limitations
High-resolution training and inference support enable strong OCR-related performance while retaining broad applicability across diverse image resolutions.
Abstract
from arXiv · showhide
The Large Vision-Language Model (LVLM) field has seen significant advancements, yet its progression has been hindered by challenges in comprehending fine-grained visual content due to limited resolution. Recent efforts have aimed to enhance the high-resolution understanding capabilities of LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and constrained to a relatively narrow resolution range. This paper represents InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently, considering the ultra-high resolution may not be necessary in all scenarios, it supports a wide range of diverse resolutions from 336 pixels to 4K standard, significantly broadening its scope of applicability. Specifically, this research advances the patch division paradigm by introducing a novel extension: dynamic resolution with automatic patch configuration. It maintains the training image aspect ratios while automatically varying patch counts and configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x 336), leading to dynamic training resolution from 336 pixels to 4K standard. Our research demonstrates that scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements. InternLM-XComposer2-4KHD shows superb capability that matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The InternLM-XComposer2-4KHD model series with 7B parameters are publicly available at https://github.com/InternLM/InternLM-XComposer.
1. Introduction
InternLM-XComposer2-4KHD addresses limited fine-detail understanding in LVLMs by supporting diverse image resolutions up to 4K HD through dynamic resolution and automatic patch configuration. The model reports consistent gains as training resolution increases and strong performance across diverse benchmarks.
- Limited resolution makes LVLMs struggle with fine details in charts, tables, documents, and infographics, constraining practical applicability.
- InternLM-XComposer2-4KHD supports images with any aspect ratio from 336 pixels up to 4K HD and beyond.
- Dynamic resolution with automatic patch configuration maintains image aspect ratios while varying patch counts and layouts for high-resolution inputs.
- Training at resolutions up to 4K standard produces consistent performance improvements, with further gains possible beyond 4K.
- The model achieves SOTA results in 6 of 16 benchmarks and surpasses GPT4V and Gemini Pro across ten benchmarks.
2. Related Works
Related work develops LVLMs for visual and document understanding, but high-resolution processing remains a central challenge. Existing approaches use high-resolution encoders or patch-based processing, while document tasks demand detailed handling of complex layouts and aspect ratios.
- LVLMs combine language models with vision encoders to leverage complementary textual and visual representations.
- High-resolution methods either adapt vision encoders directly or divide images into multiple low-resolution patches.
- InternLM-XComposer2-4KHD is presented as a simplified approach for varying resolutions and aspect-ratio inputs.
- Document understanding requires models to process high-resolution inputs, complex layouts, varying aspect ratios, and diverse document formats.
3. Method
InternLM-XComposer2-4KHD combines a lightweight vision encoder and language model with dynamic image partitioning, global-local views, and resolution-aware training. Its training uses diverse alignment and OCR-oriented data to support high-resolution and structural image understanding.
- 3.1. Model Architecture: The architecture uses OpenAI ViT-Large/14, InternLM2-7B, and Partial LoRA for efficient visual-language alignment.
- 3.2. High-Resolution Input: Dynamic image partitioning preserves the original aspect ratio while dividing high-resolution images into 336×336 patches.
- 3.2. High-Resolution Input: The global-local format combines a fixed 336 × 336 global view for macro understanding with reassembled local patch features.
- 3.3. Pre-Training: The model uses diverse pre-training objectives and additional OCR data to improve high-resolution and structural image understanding.
- 3.4. 4KHD Supervised Fine-tuning: Mixed-resolution training uses HD-55 for high-resolution tasks, allowing 4K inputs without additional image compression, while other tasks use dynamic resolutions.
- 3.4. 4KHD Supervised Fine-tuning: Using HD30 inference with HD25 training yields better results on most OCR-related tasks.
4. Experiments
Experiments compare InternLM-XComposer2-4KHD with open- and closed-source LVLMs across diverse benchmarks, then analyze resolution, global-view, newline, and token-merging choices. Results show strong high-resolution and OCR performance, with gains depending on task and resolution setting.
- Benchmark comparisons: 90.0% on DocVQA and 81.0% on ChartQA surpass GPT-4V and Gemini-Pro, while InfographicVQA approaches closed-source API performance.The model exceeds previous open-source results on InfographicVQA by nearly 20%.
- High-resolution evaluation: 68.6% on InfographicVQA surpasses DocOwl 1.5 by +17.9%, while 67.5% on OCRBench surpasses CogAgent by +8.5%.These comparisons target models specifically designed for high-resolution understanding.
- Resolution effects: +10.2% on InfographicVQA results from switching HD-9 to HD-16 during training, while higher inference resolution can add +8.1% without additional training.Resolution gains are relatively minor on other OCR-related and perception benchmarks.
- Component ablations: Removing the global view decreases performance across all benchmarks, including a −4.4% drop on MMBench EN-Test.The global view provides macro-level image understanding in the Global-Local Format.
- Component ablations: Dynamic newline indicators help interpret structurally complex images, while concatenation and C-Abstractor token merging achieve similar results on most benchmarks.The token-merging comparison uses the 4KHD setting and a compression rate of 0.25.
5. Conclusion
InternLM-XComposer2-4KHD combines dynamic resolution, automatic patch configuration, global-view patches, and a learnable newline token to support high-resolution LVLM understanding. It achieves strong OCR and general benchmark performance, with HD-OCR results continuing to improve through 4KHD despite computational limits on exploring higher resolutions.
- Contributions: InternLM-XComposer2-4KHD exceeds previous open-source models on OCR-related tasks and remains competitive on general-purpose LVLM benchmarks.The conclusion attributes support for up to 4KHD training to dynamic resolution and automatic patch configuration.
- Findings: Performance on HD-OCR tasks continues improving as training resolution increases, with no saturation observed at 4KHD.The upper improvement bound remains unexplored because higher-resolution inputs increase computational burden.
- Design choices: Global-view patches support macro understanding, while a learnable newline token handles varying input image resolutions.