CapsFusion: Rethinking Image-Text Data at Scale

1. Introduction

• Models trained on raw and CAPSFUSION captions show richer real-world knowledge than synthetic-caption models, whose outputs reduce to generic concepts.Figure 3 contrasts the models’ generated outputs.Source: Figure 3 · p. 2, §1 · p. 2
• Synthetic-caption training rapidly saturates and can degrade, especially on SEED-Bench, while its world knowledge outputs omit details about people, locations, and events.These shortcomings were less apparent on traditional COCO caption evaluation.Source: §1 · pp. 1–2
• Synthetic captions’ simplified language and missing real-world specifics are linked to captioning models trained on datasets that replace names and locations with generic placeholders.The paper says this may contribute to scalability deficiency and world knowledge loss.Source: §1 · p. 2
• CAPSFUSION uses ChatGPT to integrate noisy but knowledge-rich web captions with structured synthetic captions, then fine-tunes LLaMA on ChatGPT outputs for scalable fusion.The resulting model is reported to perform on par with ChatGPT on caption fusion.Source: §1 · p. 2
• 18.8 CIDEr points on COCO and 18.3 on NoCaps: CAPSFUSION captions outperform synthetic captions, with reported advantages in sample efficiency, world knowledge depth, and scalability.They require 11–16 times less computation to reach high performance similar to synthetic captions.Source: §1 · p. 3

Image-text Data Enhancement

• LaCLIP [16] rewrites raw captions but can hallucinate, while FuseCap [44] uses visual experts and CAPSFUSION uses large-scale web data to enhance captions.Other approaches filter and mix raw and synthetic captions [19] [39].Source: §2 · p. 3
• VeCLIP [31] also combines raw and synthetic captions, but uses an existing LLM directly and lacks CAPSFUSION’s explicit fusion instructions.CAPSFUSION fine-tunes an open-source LLM using ChatGPT-generated training data and instructs it to retain raw-caption knowledge and synthetic-caption structure.Source: §2 · p. 3

Large Multimodal Models

• Unlike recent work focused on CLIP, CAPSFUSION evaluates LMMs and reports that mixing raw and synthetic captions yields no improvement over using them separately [16] [31] [39].LMMs include models trained to predict text tokens and models that predict both image and text tokens [15] [21] [51] [63].Source: §2 · pp. 3–4

CAPSFUSION

• CAPSFUSION frames LMM training as predicting text tokens conditioned on an image, with noisy captions risking extraneous words and simplistic captions risking reduced language complexity.Its goal is to combine raw-caption knowledge with synthetic-caption structure in a refined dataset.Source: §3 · p. 4
• The method trains models to predict each text token from the image and preceding text tokens under an autoregressive objective.The training set consists of image-text pairs, with the text represented as a token sequence.Source: §3 · p. 4
• ChatGPT fuses raw and synthetic captions using instructions that distinguish their properties and request structured sentences retaining detailed real-world information.The prompt also tells ChatGPT not to concatenate the two sentences directly.Source: §3 · pp. 4–5
• Because ChatGPT’s cost is prohibitive at scale, CAPSFUSION fine-tunes 13B LLaMA-2 on raw-caption and synthetic-caption inputs paired with ChatGPT fusion outputs.On 100 human-evaluated validation cases, 80 outputs performed equally or better than ChatGPT; LaCLIP is described as prone to hallucination and failure on noisy raw captions.Source: §3 · p. 5, Figure 6 · p. 5
• CAPSFUS-LLaMA fuses 127,897,754 LAION-COCO image-text pairs in about 12 days on 128 A100-40G GPUs, yielding 120,724,312 filtered pairs in CAPSFUS-120M.The resulting captions have the highest unique-trigram count and longest average sentence length among the compared datasets.Source: §3 · p. 6

4.1. Setup

• The comparison holds images constant across caption datasets and evaluates models at 10M, 50M, and 100M scales, tuning only the bridging module.The setup uses LLaMA-2-7B and EVA-01-CLIP-g, freezes the LLM and vision encoder, and applies the same one-epoch schedule across captions.Source: §4.1 · pp. 6–7

4.2. Model Performance

• 18.8 and 18.3 CIDEr improvements on COCO and NoCaps, respectively, accompany CAPSFUSION’s broad advantage over baseline captions across the evaluated datasets.The experiments also describe CAPSFUSION as superior across effectiveness, efficiency, world knowledge depth, and scalability.Source: §4 · p. 6, §4.2 · p. 7
• Mixing captions does not consistently improve performance, and synthetic captions score 102.5 COCO CIDEr at 10M but 96.93 at 100M.On 10M COCO, raw-plus-language-rewrite mixing yields 72.62 CIDEr and 15.01 SPICE, between raw captions (75.13 CIDEr, 15.48 SPICE) and language rewrites (71.61 CIDEr, 14.6 SPICE).Source: §4.2 · p. 7

4.3. Sample Efficiency

• With 10M pairs, CAPSFUSION captions outperform other captions trained at 50M and 100M, and match the best baseline using 6M–9M samples across four benchmarks.The reported matches require 6M samples for NoCaps, 9M for TextCaps, 6M for Flickr30K, and 8M for COCO, corresponding to an 11–16 times speedup.Source: §4.3 · pp. 7–8, Figure 7 · p. 7

4.4. Scalability Analysis

• Synthetic captions saturate around 30M pairs and then gradually deteriorate, while raw captions rise with scale but remain noise-limited; CAPSFUSION outperforms both throughout.The passage describes the CAPSFUSION trend across all datasets and points to Figure 7 for the scalability comparison.Source: Figure 7 · p. 7, §4.4 · p. 8

4.5. Further Evaluation on SEED-Bench

• CAPSFUSION outperforms raw and synthetic captions on 7 of 9 SEED-Bench tasks, including instance counting, interaction, and scene understanding.Table 4 is identified as the zero-shot evaluation of models trained with different caption datasets on SEED-Bench.Source: §4.5 · p. 8, Table 4 · p. 8

4.6. Qualitative Evaluation on World Knowledge

• Models trained on raw and CAPSFUSION captions identify celebrities, artworks, authors, and event locations, while synthetic-caption models lose these capabilities.The paper presents these comparisons as qualitative evaluations of model outputs.Source: §4.6 · p. 8

4.7. Effects when Firing LLM

• At 10M scale, performance declines significantly when the LLM is tuned on synthetic captions, which the authors relate to their simplified language.The passage reports experiments on COCO, SEED, and MMLU and points to Table 5.Source: §4.7 · p. 8, Table 5 · p. 8

5. Conclusion

• The conclusion identifies scalability deficiency and world-knowledge loss in LMMs trained with synthetic captions.These are presented as issues identified by the work.Source: §5 · p. 8
• Web-based image-text pairs contain rich world knowledge but are too noisy to achieve decent performance.Source: §5 · p. 8
• CAPSFUS-120M exhibits all-around superiority over existing image-text datasets, and the authors present CAPSFUSION as a framework for scalable LMM training.Source: §5 · p. 8

6. More CAPSFUSION Examples

• Figure 8 provides further examples of raw web captions, BLIP-generated synthetic captions, and CAPSFUSION captions.Source: §6 · p. 12
• The examples describe CAPSFUSION captions as organically organizing information from raw and synthetic captions.Source: §6 · p. 12
• The examples identify CAPSFUS-LLaMA as the generator of the CAPSFUSION captions shown.Source: §6 · p. 12

7. Prompting Templates for Data Refining

• The template asks the model to merge and refine two supplied sentences: a raw caption and a synthetic caption.Source: §7 · p. 12
• It characterizes the raw sentence as detailed but structurally flawed, and the synthetic sentence as well-structured but lacking depth and potentially false.Source: §7 · p. 12
• The output should be a well-structured sentence that retains the raw caption’s detailed information without simply concatenating the inputs.Source: §7 · p. 12

• CAPSFUS-LLaMA and LMM training hyperparameters are presented in Tables 6 and 7, respectively.Source: §8 · p. 12

• SEED-Bench has 12 tasks, including spatial and temporal comprehension; the evaluation selects nine image-text tasks, excluding the leftmost three video-text tasks.Source: §9 · p. 12
• Five tasks assess scene understanding, instance identity, instance location, instance attributes, and instance counting.They cover holistic image information, object recognition and localization, visual appearance, and counting referred objects.Source: §9 · p. 12
• The remaining tasks assess spatial relations, instance interactions, visual reasoning using image information and commonsense, and textual elements in images.Source: §9 · pp. 12–14