Source-linked AI summary
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, Dahua Lin
TL;DR
Existing LMMs face sub-optimal modality alignment because high-quality image-text pairs are scarce and captions often omit fine-grained visual information. ShareGPT4V provides 1.2M descriptive captions, including 100K GPT4-Vision captions expanded by a caption model, and ShareGPT4V-7B achieves leading performance across most evaluated benchmarks.
Problem
Existing LMMs have sub-optimal modality alignment because available image-text datasets often contain brief captions with limited information content.
Method
ShareGPT4V combines 100K GPT4-Vision-generated captions with 1.2M captions produced by a caption model trained on the high-quality subset, using the data for pre-training and SFT.
Results
ShareGPT4V-7B attained the most superior performance in 9 of 11 benchmarks and achieved 1943.8 on MME, 95.6 points above Qwen-VL-Chat-7B.
Takeaways & Limitations
The results support highly descriptive image-text captions as an effective resource for improving LMM modality alignment and benchmark performance.
Takeaways & Limitations
The Qwen-VL-Chat-7B comparison used its final checkpoint fine-tuned on LLaVA-1.5’s 665K SFT dataset because the original pre-SFT checkpoint and SFT data were unavailable.
Abstract
from arXiv · showhide
In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data. To address this bottleneck, we introduce the ShareGPT4V dataset, a pioneering large-scale resource featuring 1.2 million highly descriptive captions, which surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. Specifically, ShareGPT4V originates from a curated 100K high-quality captions collected from advanced GPT4-Vision and has been expanded to 1.2M with a superb caption model trained on this subset. ShareGPT4V first demonstrates its effectiveness for the Supervised Fine-Tuning (SFT) phase, by substituting an equivalent quantity of detailed captions in existing SFT datasets with a subset of our high-quality captions, significantly enhancing the LMMs like LLaVA-7B, LLaVA-1.5-13B, and Qwen-VL-Chat-7B on the MME and MMBench benchmarks, with respective gains of 222.8/22.0/22.3 and 2.7/1.3/1.5. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM based on a simple architecture that has remarkable performance across a majority of the multi-modal benchmarks. This project is available at https://ShareGPT4V.github.io to serve as a pivotal resource for advancing the LMMs community.
1. Introduction
Existing LMMs align vision and language sub-optimally because image-text datasets often compress rich visual information into simplistic captions. ShareGPT4V addresses this limitation with high-quality captions and demonstrates strong benchmark performance for models trained with them.
- Low-quality captions reduce the information content available for modality alignment, despite the fine-grained semantics inherent in vision.
- Replacing a small portion of SFT captions with GPT4-Vision captions produced consistent performance gains across several LMMs and benchmarks.The replacement was only 3.5% of LLaVA-1.5 SFT data in one experiment.
- ShareGPT4V combines 100K GPT4-Vision captions with 1.2M captions generated by a caption model trained on the high-quality subset.
- The Qwen-VL-Chat comparison used its final checkpoint fine-tuned on LLaVA-1.5’s 665K SFT dataset because the pre-SFT checkpoint and corresponding data were unavailable.
- ShareGPT4V-7B outperformed other advanced 7B-scale LMMs across all 11 reported benchmarks.On MME, it scored 1943.8, exceeding Qwen-VL-Chat-7B by 95.6 points.
2. Related Work
Related work spans large language and multi-modal models, modality alignment, and image-text caption enhancement. ShareGPT4V is positioned against approaches limited by short, low-quality, or synthetic captions and by reliance on human annotation.
- Large multi-modal models commonly incorporate visual knowledge into language models through modality-alignment methods.
- CLIP exemplifies visual-textual alignment through contrastive learning on extensive image-text pairs.
- Image-text Data Enhancement: Caption-enhancement methods include LLM rewriting and caption fusion, but low-quality inputs or synthetic captions can limit their effectiveness.
- Image-text Data Enhancement: LLaVA uniquely uses human-annotated short captions and bounding boxes as inputs to GPT4 for detailed caption generation, while relying heavily on extensive human annotation.
3. ShareGPT4V Dataset
ShareGPT4V constructs descriptive captions from diverse image sources using GPT4-Vision, then scales them with a general captioner for pre-training. Its captions and resulting model show strong qualitative, human-evaluated, and benchmark performance.
- Data Collection: The dataset construction begins with GPT4-Vision captions for approximately 100K images gathered from diverse data sources.
- PT Data Generation: The Share-Captioner is trained on the 100K captions and generates 1.2M captions without requiring source-specific prompts.
- Data Collection: Data-specific prompts instruct GPT4-Vision to describe world knowledge, object attributes, spatial relationships, and aesthetic information.
- Evaluation: Replacing SFT captions with ShareGPT4V captions improved performance across varied LMMs, while Share-Captioner outputs were qualitatively comparable to GPT4-Vision.
- PT Data Generation: Share-Captioner captions were judged on par with GPT4-Vision in a human evaluation using 100 validation samples and 10 volunteers.
- Quantitative Evaluation: ShareGPT4V-7B outperformed competitors in 9 of 11 benchmarks and ranked second on the remaining benchmarks with 7B parameters.
4. ShareGPT4V-7B Model
ShareGPT4V-7B is a streamlined LMM based on LLaVA-1.5 that uses high-quality captions during pre-training and SFT. Its training jointly adapts the vision encoder, projector, and language model to align visual embeddings with caption details.
- Model Architecture: ShareGPT4V-7B follows LLaVA-1.5 with a CLIP-Large vision encoder, two-layer MLP projector, and Vicuna-v1.5 language model.The vision encoder uses 336×336 resolution and converts images into 576 tokens.
- Training Strategy: The model uses ShareGPT4V-PT during pre-training and replaces part of LLaVA-1.5’s 665k supervised data with ShareGPT4V captions.The paper emphasizes caption effectiveness rather than introducing a novel architecture.
- Training Strategy: Jointly fine-tuning the vision encoder, projector, and language model helps align visual embeddings with the intricate details expressed in high-quality captions.The authors state that fine-tuning only the MLP does not fully exploit these captions.
- Multimodal Dialogue: ShareGPT4V-7B generates detailed multimodal dialogue responses that combine image understanding with aesthetic assessment.The example describes both visual content and why the depicted combination is visually appealing.
5. Experiments
Experiments evaluate ShareGPT4V-7B across 11 multimodal benchmarks and ablate caption use in pre-training and SFT. The model performs strongly across benchmarks, while high-quality captions improve both training stages.
- Quantitative Comparison: ShareGPT4V-7B achieves the strongest performance among previous LMMs on 9 of 11 benchmarks.The evaluation spans academic VQA tasks and benchmarks designed for large multimodal models.
- Quantitative Comparison: 1.9 points separate ShareGPT4V-7B from LLaVA-1.5-13B on the LLaVA (in the wild) benchmark.The paper reports superior capabilities in detailed description and complex reasoning.
- Quantitative Comparison: 36.1 points separate the models’ MME perception scores, while ShareGPT4V-7B exceeds Qwen-VL-Chat by 15.7 points in MME cognition.ShareGPT4V-7B also reaches 68.8% on MMBench and 69.7% on SEED (image).
- Ablations: 31.4 points in MME perception, 2.5% on MMBench, and 0.5% on SEED result from using ShareGPT4V only during SFT.Using ShareGPT4V-PT only during pre-training yields gains of 46.5 points, 3.1%, and 2.3% on the same benchmarks.
- Ablations: More pre-training data produces consistent gains on MMBench and SEED Bench.The supplied figure passage states the scaling trend without reporting specific values.
- Ablations: 18.2, 1.9, and 2.0 separate ShareGPT4V-PT-558K from BLIP-558K on MME Perception, MMBench, and SEED Bench, respectively.The comparison uses the same images and end-to-end training strategy with different captions.
6. Conclusion
The paper introduces ShareGPT4V as a large-scale source of detailed captions and demonstrates its value for training multimodal models. ShareGPT4V-7B combines this data with both pre-training and SFT to achieve strong benchmark performance.
- Conclusion: ShareGPT4V contains 1.2 million detailed captions covering world knowledge, object attributes, spatial relationships, and aesthetic assessments.The dataset includes 100K GPT4-Vision captions expanded to 1.2M captions through a general caption model.
- Conclusion: ShareGPT4V-7B incorporates the dataset during both pre-training and SFT and achieves superior performance across recent multimodal benchmarks.The paper presents the model as a simple-architecture LMM supported by high-quality data.
- Conclusion: The authors intend ShareGPT4V to serve as a publicly accessible resource for advancing the LMM community.The conclusion states a commitment to making the dataset fully accessible.
A. Data Sources
ShareGPT4V’s 100K-image caption collection combines images from established datasets and web sources. Its pre-training subset uses 1.2M images selected from public datasets and captioned with Share-Captioner.
- Data Sources: The 100K-image collection includes 50K COCO, 30K LCS, 20K SAM, 500 TextCaps, 500 WikiArt, and 1K webcrawled images.The webcrawled images are split evenly between landmarks and celebrities.
- Data Sources: The pre-training subset contains 118K COCO, 570K SAM, and 558K LLaVA-1.5 pre-training images.These images are selected from existing public datasets and captioned using the pre-trained Share-Captioner.
B. Caption Analysis
Share-Captioner produces captions with diversity, linguistic expression, and lexical information comparable to GPT4-Vision. The comparison is presented through noun-verb visualizations and lexical analysis.
- Share-Captioner captions have diversity and linguistic expression comparable to GPT4-Vision captions.Figure 7 visualizes root noun-verb pairs occurring over 1% in both caption sets.
- Share-Captioner captions contain a comparable amount of information to GPT4-Vision captions.This conclusion is based on the lexical composition analysis reported in Table 7.
- Figure 7 compares the root noun-verb pairs produced by GPT4-Vision and Share-Captioner.
C. Prompts
The prompting strategy combines a general base prompt with a data-source-specific prompt to generate descriptions suited to diverse image sources.
- The prompting design uses a base prompt for generating general descriptions.
- A specialized prompt is added for each image data source.
- The prompt structure is intended to support content-related descriptions across diverse image sources.
D. Examples
The examples illustrate ShareGPT4V’s detailed image descriptions, including object attributes, scene context, artwork knowledge, and visual dialogue capabilities.
- Share-Captioner describes object identity, ordering, materials, and surrounding scene elements in detail.The suitcase example specifies relative sizes, brown leather, brass handles, a brick wall, a tree, and a blue tarp.
- ShareGPT4V captions combine artwork identification, artist knowledge, scene content, color, and visible brushstroke details.The example identifies The Sower as a Van Gogh oil painting and describes its subject, setting, colors, and texture.
- Figure 10 presents examples of detailed captions generated by ShareGPT4V.
- Figure 11 presents an example of a multi-round dialogue generated by ShareGPT4V-7B.