PaperlayerView exact revision on arXiv

Source-linked AI summary

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, Dahua Lin

arXiv:2311.12793v2cs.CV

TL;DR

LMM modality alignment remains suboptimal amid limited high-quality image-text pairs, while common captions often omit fine-grained visual semantics. ShareGPT4V builds a 1.2M-caption dataset from 100K GPT4-Vision captions and uses it in SFT and pretraining. ShareGPT4V-7B performs best on 9 of 11 benchmarks.

  • Problem

    LMM modality alignment remains suboptimal amid scarce high-quality image-text pairs, while mainstream datasets often reduce fine-grained visual semantics to brief salient-object captions.

    Source: §1 · p. 2

  • Method

    ShareGPT4V compiles 100K GPT4-Vision captions and expands them to 1.2M with a caption model, then uses the data in SFT and pretraining.

    Source: §6 · p. 9

  • Results

    ShareGPT4V-7B performed best in 9 of 11 benchmarks; its MME total score of 1943.8 surpassed Qwen-VL-Chat-7B by 95.6 points.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    The findings support detailed captions as a useful resource for LMM training.

    Source: §6 · p. 9

    For Qwen-VL-Chat-7B, the pre-SFT checkpoint and original SFT data were unavailable, so comparison used its final checkpoint fine-tuned on LLaVA-1.5’s 665K SFT dataset.

    Source: §1 · p. 2

Abstract

from arXiv · show

In the realm of large multi-modal models (LMMs), efficient modality alignment is crucial yet often constrained by the scarcity of high-quality image-text data. To address this bottleneck, we introduce the ShareGPT4V dataset, a pioneering large-scale resource featuring 1.2 million highly descriptive captions, which surpasses existing datasets in diversity and information content, covering world knowledge, object properties, spatial relationships, and aesthetic evaluations. Specifically, ShareGPT4V originates from a curated 100K high-quality captions collected from advanced GPT4-Vision and has been expanded to 1.2M with a superb caption model trained on this subset. ShareGPT4V first demonstrates its effectiveness for the Supervised Fine-Tuning (SFT) phase, by substituting an equivalent quantity of detailed captions in existing SFT datasets with a subset of our high-quality captions, significantly enhancing the LMMs like LLaVA-7B, LLaVA-1.5-13B, and Qwen-VL-Chat-7B on the MME and MMBench benchmarks, with respective gains of 222.8/22.0/22.3 and 2.7/1.3/1.5. We further incorporate ShareGPT4V data into both the pre-training and SFT phases, obtaining ShareGPT4V-7B, a superior LMM based on a simple architecture that has remarkable performance across a majority of the multi-modal benchmarks. This project is available at https://ShareGPT4V.github.io to serve as a pivotal resource for advancing the LMMs community.

1. Introduction

ShareGPT4V addresses suboptimal vision-language alignment by replacing brief captions with a large collection of detailed image descriptions. Its ShareGPT4V-7B model demonstrates strong results across multimodal benchmarks.

Introduction

  • Brief, salient-object-focused captions discard fine-grained visual information, and replacing SFT image-text pairs with equivalent GPT4-Vision captions produced consistent gains across LMMs and benchmarks.The replacement was only 3.5% of LLaVA-1.5’s SFT data in that case.Source: §1 · p. 2
  • The initial 100K captions came from varied image sources and data-specific prompts, averaged 942 characters, and trained a caption model that generates descriptions without specialized prompts.The descriptions capture comprehensive image information, including world knowledge, object properties, spatial relationships, and aesthetic evaluation.Source: §1 · p. 2
  • ShareGPT4V combines 100K GPT4-Vision captions with 1.2M captions generated by a model trained on that curated set.The captions cover world knowledge, object properties, spatial relationships, and aesthetic evaluations.Source: §1 · p. 2
  • 1943.8 on MME puts ShareGPT4V-7B 95.6 points ahead of Qwen-VL-Chat-7B, and it outperforms other 7B-scale LMMs on all 11 benchmarks.The paper notes that Qwen-VL-Chat-7B’s pre-SFT checkpoint and SFT data were unavailable, so it fine-tuned the final checkpoint using LLaVA-1.5’s 665K SFT dataset for comparison.Source: §1 · p. 2

2. Related Work

Prior LMMs advanced visual-language alignment, instruction following, and grounding, but the related work identifies image-caption quality as an overlooked limitation; the paper contrasts earlier caption-enhancement methods with direct image captioning by GPT4-Vision.

Large Language Models

  • With surging data and computational power, large language models boomed; early BERT [11] and T5 [46] encoderdecoder models and GPT decodercentric models leveraged Transformers [54] to excel in various NLP tasks.GPT3 popularized decoder-only architectures using auto-regressive decoding for predictions. PaLM [9] extended model parameters and dataset scale, while InstructGPT [42] and ChatGPT introduced fine-tuning and reinforcement learning for improved conversational interaction.Source: §2 · p. 2

Large Multi-modal Models

  • LMM research advanced visual-language alignment, complex-prompt understanding, and grounding, yet often overlooked image-caption quality.CLIP [45] aligned modalities through contrastive learning; later systems used refined data [26] [27], dialogue-focused pretraining and instruction fine-tuning [5], and grounding data and training techniques [6] [43] [57].Source: §2 · p. 3

Image-text Data Enhancement

  • Earlier image-text caption enhancement used rewriting, filtering, and fusion, while LLaVA [31] relied on human-annotated captions and boxes rather than direct image perception.LaCLIP [13] could hallucinate given limited visual information and low-quality original captions; filtering and blending methods [16] [38] and VeCLIP [23] were constrained by synthetic-caption quality. LLaVA’s annotation-based approach could produce hallucinations, whereas GPT4-Vision generates detailed captions from image inputs and deliberated prompts.Source: §2 · p. 3

3. ShareGPT4V Dataset

ShareGPT4V uses 100K GPT4-Vision captions to train a captioner that expands the dataset to 1.2M captions, and reports benefits in SFT and caption quality. The paper also reports strong benchmark results for ShareGPT4V-7B.

3.1. Overview

  • The pipeline collects 100K GPT4-Vision captions from varied image sources, then trains a captioner on these seeds to produce 1.2M captions for pretraining.The overview presents the dataset as a resource with longer captions and a broader range of sources than existing widely used caption datasets.Source: §3.1 · p. 3, Figure 3 · p. 4

3.2. ShareGPT4V Data Collection

  • GPT4-Vision produced 100K SFT captions from diverse image sources using prompts for attributes, spatial relations, relevant knowledge, and sometimes aesthetics.The sources included detection and segmentation images, text-containing images, and web images such as artworks, landmarks, and celebrities.Source: §3.2 · p. 4
  • Replacing corresponding detailed captions in existing LMM SFT datasets with the 100K GPT4-Vision captions significantly improved performance across the tested models.The comparison kept image data sources as consistent as possible.Source: §3.2 · p. 5

3.3. ShareGPT4V-PT Data Generation

  • Share-Captioner is fine-tuned on the 100K seed captions and generates captions for 1.2M public-dataset images without specialized prompts, using about 44 A100 GPU days.The authors motivate this scaling process by the large-scale data needs of pretraining for modality alignment.Source: §3.3 · p. 5
  • Share-Captioner captions were qualitatively close to GPT4-Vision on held-out examples and judged on par in a 100-caption comparison by 10 volunteers.The figure compares COCO, BLIP, LLaVA-1.5-7B, Share-Captioner, and GPT4-Vision captions; the human evaluation used 100 validation samples.Source: §3.3 · p. 5, Figure 4 · p. 5, Table 2 · p. 5
  • ShareGPT4V-7B outperforms competitors on 9 of 11 benchmarks and ranks second on the other two, despite competitors using larger datasets or more parameters.The paper characterizes ShareGPT4V as having more diverse image sources, a more advanced caption producer, more samples, and longer captions than widely used caption datasets.Source: §3.1 · p. 3, Table 3 · p. 6

4. ShareGPT4V-7B Model

ShareGPT4V-7B is a streamlined LMM baseline built to test high-quality ShareGPT4V data in both pretraining and supervised fine-tuning.

4.1. Model Architecture

  • ShareGPT4V-7B follows LLaVA-1.5 [30], using a 336×336 CLIP-Large encoder [45], a two-layer MLP projector, and Vicuna-v1.5 derived from LLaMA2 [53].The encoder converts each image into 576 tokens; the authors report that the lightweight 7B model can outperform many LMMs trained on larger datasets or at larger scales.Source: §4.1 · p. 6

4.2. Pre-Training

  • Pretraining on ShareGPT4V-PT jointly fine-tunes the vision encoder, projector, and language model so their visual embeddings align with caption details.Fine-tuning only the latter half of the vision encoder’s layers achieves optimal results with satisfactory training efficiency; training uses a 2e−5 learning rate, batch size 256, and roughly 4700 steps.Source: §4.2 · p. 6

4.3. Supervised Fine-Tuning.

  • SFT replaces 23K detailed-description examples in LLaVA-1.5’s 665K supervised examples with randomly sampled high-quality captions from ShareGPT4V’s 100K-caption set.The vision encoder is frozen while the projector and language model are fine-tuned, using a 2e−5 learning rate, batch size 128, and about 5200 steps.Source: §4.3 · p. 6

5. Experiments

ShareGPT4V-7B performs strongly across 11 multimodal benchmarks, while ablations show benefits from high-quality captions in both pretraining and SFT and from unlocking vision-encoder blocks. The experiments also examine caption-data scale and multimodal dialogue capabilities.

5.1. Benchmarks

  • The evaluation spans 11 benchmarks, combining academic VQA tasks with LMM-focused tests of conversation, reasoning, perception, cognition, and image understanding.It includes English and Chinese MMBench [33], MME [15], SEED [24], MM-Vet [58], Q-Bench [55], VQA-v2 [17], VisWiz [18], and LLaVA (in the wild) [31].Source: §5.1 · p. 7

5.2. Quantitative Comparison

  • ShareGPT4V-7B led 9 of 11 benchmarks, beating LLaVA-1.5-13B by 1.9 points on LLaVA (in the wild) and scoring 68.8% on MMBench and 69.7% on SEED (image).It also led MME perception by 36.1 points over LLaVA-1.5-13B and MME cognition by 15.7 points over Qwen-VL-Chat; its Q-Bench score was 63.4%.Source: §5.2 · p. 7

5.3. Multi-modal Dialogue

  • Figure 5’s dialogue examples show ShareGPT4V-7B understanding image details and performing aesthetic assessments.The paper presents these examples as further evidence of the significance of the collected high-quality captions.Source: §5.3 · p. 7

5.4. Ablations

  • Using ShareGPT4V in SFT raised MME perception by 31.4 points, while using ShareGPT4V-PT in pretraining raised it by 46.5 points over the LLaVA-1.5-7B baseline.The SFT subset also improved MMBench and SEED accuracy by 2.5% and 0.5%; pretraining improved them by 3.1% and 2.3%, respectively, and using ShareGPT4V in both stages further enhanced overall performance.Source: §5.4 · p. 8, Table 4 · p. 8
  • Replacing BLIP-558K captions with ShareGPT4V-PT-558K yielded gains of 18.2, 1.9 and 2.0 on the three benchmarks, supporting the importance of caption quality for pretraining.The comparison used the same images and end-to-end training strategy; BLIP-558K joint training showed only minor gains of 4.7 in MME perception and 0.1 on SEED.Source: §5.4 · p. 8, Table 5 · p. 8
  • Unlocking the latter half of the vision encoder’s transformer blocks during pretraining, rather than freezing it, improved MME perception by 52.2 points, MMBench by 2.2%, and SEED by 1.6%.The study identifies this as a more effective modality-alignment approach for high-quality captions.Source: §5.4 · p. 8, Table 6 · p. 8
  • With 100K high-quality pretraining captions, performance improved on MMBench and SEED; gains tended to saturate after more than 1000K captions.The study randomly sampled ShareGPT4V-PT subsets ranging from 100K to 1200K and describes high-quality captions as enabling efficient alignment at a relatively lightweight data scale.Source: §5.4 · p. 8, Figure 6 · p. 8

6. Conclusion

ShareGPT4V is a 1.2-million-caption image-text dataset built from 100K GPT4-Vision captions and expanded for pre-training. The study reports its effectiveness in LMM fine-tuning and in ShareGPT4V-7B, which uses the dataset in both pre-training and SFT.

6. Conclusion

  • ShareGPT4V contains 1.2 million detailed captions spanning world knowledge, object attributes, spatial relationships, and aesthetic assessments.The authors describe the captions as richer and more diverse than existing datasets.Source: §6 · p. 9
  • The dataset combines 100K high-quality GPT4-Vision captions for SFT with an expansion to 1.2 million captions for pre-training using a general caption model.Source: §6 · p. 9
  • The study validates ShareGPT4V through SFT results on recent LMMs and the reported superior performance of ShareGPT4V-7B, which uses the dataset in both pre-training and SFT.Source: §6 · p. 9
  • The authors commit to making ShareGPT4V fully accessible to the public as a resource for advancing LMM research.Source: §6 · p. 9

A. Data Sources

ShareGPT4V’s caption collection draws on a diverse 100K-image set, while ShareGPT4V-PT uses 1.2M images selected from public datasets.

A. Data Sources

  • 100K images from diverse sources formed the ShareGPT4V caption collection.The set includes 50K COCO [29], 30K LCS images from LAION [48], CC-3M, and SBU, 20K SAM [21], 500 TextCaps [51], 500 WikiArt [47], and 1K webcrawled landmark and celebrity images split evenly.Source: A. Data Sources · p. 9
  • 1.2M images selected from public datasets formed the ShareGPT4V-PT pretraining set.The pre-trained Share-Captioner generated the dataset, which includes 118K COCO [29], 570K SAM [21], and 558K LLaVA-1.5 pre-training images [30].Source: A. Data Sources · p. 9

B. Caption Analysis

The caption analysis finds that Share-Captioner’s captions are comparable to GPT4-Vision captions in root noun-verb diversity, linguistic expression, and lexical information content.

B. Caption Analysis

  • Figure 7 shows that Share-Captioner captions have root noun-verb diversity and linguistic expression comparable to GPT4-Vision captions.The visualization covers root noun-verb pairs occurring over 1% of captions.Source: Figure 7 · p. 9, B. Caption Analysis · p. 9
  • Table 7’s lexical composition analysis finds that Share-Captioner captions contain an amount of information comparable to GPT4-Vision captions.Source: Table 7 · p. 9, B. Caption Analysis · p. 9

C. Prompts

The authors designed prompts to produce content-related descriptions across diverse image sources, using a general-description base prompt and a specialized prompt for each source.

C. Prompts

  • A general-description base prompt was paired with a specialized prompt for each image source to address source diversity and elicit content-related descriptions.The prompt design is shown in Figure 8.Source: C. Prompts · p. 9

D. Examples

The section presents examples of captions generated by Share-Captioner and ShareGPT4V, with detailed and accurate parts emphasized.

D. Examples

Related papers

Loading 2311.12793v2…