PaperlayerView exact revision on arXiv

Source-linked AI summary

A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions

Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, Adriana Romero-Soriano

arXiv:2312.08578v2cs.CV

TL;DR

The paper identifies a lack of high-quality image descriptions tightly coupled to images for deep-alignment training and evaluation, alongside CLIP’s 77-token limit. It introduces DCI and sDCI for evaluating VLMs and reports that no evaluated model performs well on both tasks, while fine-tuning with high-quality pairs improves VL-C-Object.

  • Problem

    Available image descriptions lack the quality and image alignment needed to train or evaluate deep alignment, while CLIP’s 77-token limit complicates using dense captions.

    Source: §1 · p. 1, §3.3 · p. 6

  • Method

    The authors introduce DCI, a dataset of 7,805 images with dense mask-aligned descriptions, and under-77-token sDCI for negatives-based and subcrop-caption matching evaluations.

    Source: §1 · p. 2, §4.1 · p. 7

  • Results

    No evaluated model performs well on both evaluations, while negatives-based training trades improved negatives-based performance for worse SCM; high-quality-pair tuning raises VL-C-Object from 81.17% to 88.37%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    DCI has clear use as an evaluation benchmark, and its high-quality captions show initial potential for improving model performance through fine-tuning.

    Source: §1 · p. 2, §6 · p. 9

    Summarization may remove information needed to distinguish captions, so the test ideally awaits VLMs that can handle the unsummarized descriptions.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Curation methods for massive vision-language datasets trade off between dataset size and quality. However, even the highest quality of available curated captions are far too short to capture the rich visual detail in an image. To show the value of dense and highly-aligned image-text pairs, we collect the Densely Captioned Images (DCI) dataset, containing 7805 natural images human-annotated with mask-aligned descriptions averaging above 1000 words each. With precise and reliable captions associated with specific parts of an image, we can evaluate vision-language models' (VLMs) understanding of image content with a novel task that matches each caption with its corresponding subcrop. As current models are often limited to 77 text tokens, we also introduce a summarized version (sDCI) in which each caption length is limited. We show that modern techniques that make progress on standard benchmarks do not correspond with significant improvement on our sDCI based benchmark. Lastly, we finetune CLIP using sDCI and show significant improvements over the baseline despite a small training set. By releasing the first human annotated dense image captioning dataset, we hope to enable the development of new benchmarks or fine-tuning recipes for the next generation of VLMs to come.

1 Introduction

Existing image-text datasets often lack descriptions that are both rich in visual detail and tightly aligned to image regions. The paper introduces DCI and sDCI and evaluates fine-grained matching, finding that models struggle to perform well on both its evaluation tasks.

1 Introduction

  • No existing dataset provides descriptions tightly coupled enough to images to train for or evaluate deep alignment between the two domains.Web-derived pairs use alt-text or nearby webpage text, while curation trades quantity for quality.Source: §1 · p. 1
  • Existing benchmark progress can be difficult to interpret because language-prior methods can approach 100% accuracy on hard-negative tests without using images.Some benchmark training uses negatives constructed similarly to the tests, and reported improvements may not transfer to synthetic benchmarks with equally likely positive and negative captions.Source: §1 · pp. 1–2
  • DCI contains 7,805 images with dense, mask-aligned descriptions averaging over 1,000 words, and sDCI summarizes captions to fit within 77 tokens.The paper uses the summarized captions in a negatives-based test and a subcrop-caption matching task.Source: §1 · p. 2
  • No evaluated model performs well on both sDCI evaluation tasks, and gains from negatives-based training come at the cost of subcrop-caption matching performance.Subcrop-caption matching asks models to select captions for different regions of the same image.Source: §1 · p. 2
  • High-quality image-caption pairs match fine-tuning on at least 10× as much automatically annotated data on ARO and VL-Checklist, and improve VL-C-Object from 81.17% to 88.37% without explicit negatives.Source: §1 · p. 2

2 Related Works

Related work spans human-annotated datasets, web-scale image-text collections, compositionality benchmarks, and methods for vision-language pretraining. DCI is positioned among efforts to improve caption quality and density for model evaluation and training.

Vision-Language Datasets

  • Human-annotated datasets such as COCO, Visual Genome, and Flickr30k offer high quality but typically short captions and limited scale.Web-crawled collections such as YFCC100M, Conceptual Captions, and LAION-5B increase scale, with filtering trading data quality for scale.Source: §2 · p. 2
  • Localized Narratives provides dense visual grounding at about 36.5 words per caption, while DCI targets still denser descriptions.The passage reports approximately 850k text-image pairs for Localized Narratives.Source: §2 · p. 2

Vision-Language Evaluation Benchmarks

  • ARO, VL-CheckList, Winoground, CREPE, and PUG (Bordes et al., 2023) test object, relation, order, or compositional reasoning.Several of these benchmarks rely on COCO directly or through Visual Genome, and the passage notes difficulty finding sufficiently large evaluation data without COCO.Source: §2 · p. 4

Vision-Language models

  • CLIP (Radford et al., 2021) established contrastive image-text pretraining, while later approaches include negative-caption training, multimodal objectives, augmentations, and region-text matching.Examples include NegCLIP, FLAVA (Singh et al., 2022), Clip-rocket (Fini et al., 2023), and X-VLM (Zeng et al., 2021).Source: §2 · p. 4
  • DAC uses an automated LLM-based pipeline to increase caption quality and density; this paper explores human annotation for further increases.Source: §2 · p. 4

3 Dataset Construction

DCI pairs 7,805 SA-1B images with detailed descriptions aligned to submasks; a worker workflow creates annotations, and Llama2 summaries produce captions that fit CLIP’s 77-token limit.

3 Dataset Construction

  • DCI contains 7,805 SA-1B images with complete descriptions, much of them aligned to submasks that have their own labels and descriptions.Descriptions also cover relationships among visible entities and regions outside the submasks.Source: §3 · p. 4

3.1 Preparation

  • SAM extracts candidate masks using grid points and additional points near Canny-detected edges, then overlap thresholds organize masks into a hierarchy.Very small masks are removed, and the resulting masks do not cover every recognizable image detail.Source: §3.1 · p. 5

3.2 Collection Process

  • Qualified workers describe the full image, annotate submasks from leaf masks upward, and then write an overall description connecting the annotated regions.Workers can mark a region uninteresting or bad, helping focus annotation time on useful regions.Source: §3.2 · p. 5

3.3 Fitting DCI into 77 CLIP tokens

  • At 1,111 words and 1,279 CLIP tokens per image on average, original DCI descriptions exceed CLIP’s 77-token text limit.The median description length is 941 words.Source: §3.3 · p. 6
  • Llama2 recursively summarizes full-image and submask descriptions into multiple captions under 77 tokens and generates hard negatives for sDCI.Summaries can introduce noise, omit details, or mix references; fitting captions to the limit gives a lower bound on DCI’s evaluative resolution.Source: §3.3 · pp. 6–7

3.4 Statistics

  • DCI is 26x denser than the previous longest annotated dataset, Localized Narratives, and sDCI remains denser than LNCOCO and standard COCO captions.The comparison uses average CLIP tokens per image or caption.Source: Table 1 · p. 7, §3.4 · p. 7

4 Evaluating VLMs with summarized DCI

The section evaluates summarized DCI with subcrop-caption matching and negative-based tests. Existing models show uneven performance, with no model performing well across all sDCI tests.

4.1 Methodology

  • SCM batches images with their subcrops and summarized captions, testing whether models can match each caption to the corresponding image part.The evaluation uses a batch size of 8.Source: §4.1 · p. 7
  • The other tests vary negative selection, caption multiplicity, and image scope: All Neg, Pick5-SCM, Pick5-Neg, Base Neg, and All Hard-Negs.Pick5-SCM requires the worst-scoring positive to beat other images’ best-scoring captions; Pick5-Neg fails if any of five positives scores below its negative.Source: §4.1 · p. 7

4.2 Results

  • Because summaries may remove information needed to distinguish captions, the authors say testing unsummarized captions is preferable once models support 1000+ tokens.Source: §4.2 · p. 7
  • Table 2 compares baseline results; CLIP scored 40.12% on All SCM and 60.63% on All Neg, and only NegCLIP improved over CLIP on SCM.No model performed well across the full sDCI test set.Source: Table 2 · p. 8, §4.2 · p. 8
  • Flava’s Base Neg performance fell from 47.99% to 11.6%, indicating a longer-text comparison weakness not demonstrated in other models.Source: §4.2 · p. 8
  • Models without a CLIP component showed a smaller performance drop from All Neg to All Hard-Negs, supporting that these negatives capture a difficulty CLIP models and descendants share.The authors argue this is not simply a higher proportion of impossible negatives.Source: §4.2 · p. 8

5 Using summarized DCI as finetuning dataset

The paper fine-tunes CLIP with summarized DCI captions and compares the results with DAC caption training. The sDCI model improves over CLIP on most metrics and outperforms DAC ablations using comparable sample counts.

5 Using summarized DCI as finetuning dataset

  • The authors fine-tune ViT/32B CLIP on sDCI using LoRA, with and without a text-negative loss, and randomly select one caption per image and submask each epoch.The sDCI split contains 7,800 training, 100 validation, and 112 test samples; training runs for up to 10 epochs.Source: §5 · p. 8
  • The DAC comparison follows DAC’s ARO and VL-Checklist setup and tests DAC subsets of 10,000 images and 100,000 images against sDCI’s 8,012 images and 99,445 masks.The table reports means across five seeds and standard deviations when above 1% accuracy.Source: §5 · p. 8, Table 3 · p. 9

5.1 Results

  • sDCIP 1 improves over CLIP on most metrics and outperforms both sample-limited DAC ablations, though it trails DAC models trained on 3M images.The comparison suggests that a small number of highly aligned dense pairs can be more effective than larger quantities of loosely aligned or sparse data.Source: §5.1 · p. 9, Table 3 · p. 9
  • The sDCIP 1NL0 model does not improve across most benchmarks and somewhat degrades against CLIP, while showing a significant bump in VL-Object and some VL-Attribute improvement.The degradation is likely related to distribution shift and small sample size; LLM summaries without mask information can also lose relational information.Source: §5.1 · p. 9
  • LLM-generated negatives may not closely cover the test negatives, which could explain decreased VL-Object performance.The authors identify this as a possible explanation, not a confirmed cause.Source: §5.1 · p. 9

6 Conclusion and Future Work

The conclusion presents DCI as a dense, aligned resource for evaluation and initial fine-tuning experiments. The appendix describes how the authors generated multiple short summaries and negatives, along with examples and prompt-handling issues.

6 Conclusion and Future Work

  • DCI is presented as an evaluation resource with initial potential for fine-tuning, but current context limits require 77-token text and prevented full-caption fine-tuning.The authors also did not use pixel-level masks in their experiments, and say the dataset is too small to pretrain models for the full text.Source: §6 · p. 9
  • The authors propose that full human captions may become useful as models support longer context, and suggest DCI could help bootstrap highly aligned datasets using approaches like bitext mining.They compare the collection challenge to low-resource machine translation.Source: §6 · p. 9

A.1 Summarization

  • The summarization prompts target about 65 words for full-image and submask captions, then generate multiple alternatives emphasizing different details.The authors sought at least six captions per image or submask, including the initial summary.Source: §A.1 · p. 13

A.2 Negatives

  • Three negative prompts target basic edits, structural changes, or rearrangements of the same words; initial ablations found swaps produced the best model performance.Negatives were generated from the first summarized caption.Source: §A.2 · p. 13

A.3 Common Issues

  • LLaMA2 often missed the requested word boundary, so the authors tried shortening prompts and reverted to the original prompt if repeated attempts failed.They also removed prefatory affirmative text by deleting content before the last line break.Source: §A.3 · p. 13

A.4 Output Text Examples

  • The appendix gives a base-image description and several summaries, including variants emphasizing building details, plus a submask example focused on a street light.It also shows basic, layout, and swaps negatives for both examples.Source: §A.4 · p. 14, §A.4.1 · p. 14, §A.4.2 · p. 14

B Crowdsourcing Methodology

The crowdsourcing methodology uses model-generated masks and a staged worker-screening process. Workers are trained on the task interface, then monitored and audited for description and mask quality.

B Crowdsourcing Methodology

  • The authors replaced an initial workflow that asked workers to select regions and describe them together, because it was time-consuming and often lower quality than model-generated masks filtered by workers.They provide the full task instructions and reproduction code in the project repository.Source: B Crowdsourcing Methodology · p. 15

B.1 Quality Assurance

  • Worker eligibility was screened in stages, beginning with image-description and detail-noticing tasks, followed by interface-comprehension checks and regular audits of work quality.Audits informed feedback, limits on individual contributions, worker removal, and bonuses for exceptional quality.Source: §B.1 · p. 15

B.2 Instruction Details

  • Before the main task, workers were shown an example and guidance for cases such as duplicate masks, nested masks, and unusually simple images.The complete instructions were released with the code.Source: §B.2 · p. 15

B.3 Worker Metrics

The worker process narrowed a large qualification pool to a smaller group for the main task, then applied contribution controls and quality checks.

B.3 Worker Metrics

  • Over a month, the first qualification round drew over 800 submissions; roughly 250 workers advanced to the second task, and after reviewing around 600 second-stage tasks, 120 moved to the full task.Around 80 were regular contributors over the course of the collection.Source: §B.3 · p. 15
  • No worker could provide more than 10% of collected data at a time, and word-count, writing-rate, and unique-word metrics flagged examples for review.Examples outside expected metric bounds were hand-reviewed.Source: §B.3 · p. 15

C Extended Ablations

The extended ablations test caption selection, batching, negative-loss weighting, and negative construction, alongside dataset and training-set comparisons.

C.1.1 PickN Caption Training

  • PickN training randomly selects one caption per image each epoch for N = 1; for N > 1, it uses multiple captions and compares the worst positive with the best negative.The setup expects any caption for one image to score above captions from another image.Source: §C.1.1 · p. 16

C.1.2 Image-based Batching

  • ImgGroup uses other submasks from the same image as difficult negatives during CLIP-loss calculation.The ablation tests their impact on CLIP training.Source: §C.1.2 · p. 16

C.1.3 Negatives Loss

  • A separate run uses nine times the standard negatives loss to test whether benchmark negative-construction techniques can be gamed.Source: §C.1.3 · p. 16

C.1.4 Negatives Selection

  • For Localized Narratives and COCO, captions receive spaCy swaps of noun phrases, verbs, and adjectives; DCI runs also compare these swaps with LLM-generated negatives.Source: §C.1.4 · p. 16

C.2 Datasets and Training methods

  • The ablations compare sDCIsub, Localized Narratives, COCO, and matched 7,805-image subsets of the latter two datasets.Training uses the Section 5 parameters, and models are selected by validation performance on the dataset used for training.Source: §C.2 · p. 16
  • The sDCI split assigns 7,599 images to training, 98 to validation, and 108 to testing; all reported metrics use the 108-image test set.Source: §C.2 · p. 16

C.3.1 Aggregate DCI Ablations

The ablations show that performance depends on the benchmark and training choices: dataset-wide batching and Pick1 often help, while gains on negative tests can trade off against subcrop-caption matching.

C.3.1 Aggregate DCI Ablations

  • On ARO and VL-C, constructing batches from the entire dataset generally performs better than batching masks from the same image.Source: §C.3.1 · p. 16, Table 5 · p. 17
  • Pick1 is the most effective way to use the LLM caption summaries, outperforming Pick5 even when evaluated on Pick5.The authors attribute this to seeing more related text without potentially penalizing overlapping captions in a batch.Source: §C.3.1 · pp. 16, 18
  • Higher negatives loss improves most reported metrics, but not VL-Object or VL-Attribute, where constructed negatives may not match the test negatives.Source: §C.3.1 · p. 18
  • No negatives-loss setting leads on every metric: zero negatives loss favors Subcrop-Caption Matching, while negatives-trained models do better on negative tests.Source: §C.3.1 · p. 18

C.3.2 DCI fine-tuning performance

  • At 7,800 images without masks, DCI matches COCO on Subcrop-Caption Matching, while both outperform Localized Narratives by a noticeable margin.The authors suggest Localized Narratives may be less sample-efficient than baseline COCO captions.Source: §C.3.2 · p. 18
  • LLM-based captions improve performance on all negatives tests at the expense of Subcrop-Caption Matching.The authors suggest the generated captions may overfit to the test task.Source: §C.3.2 · p. 18
  • Training directly on negatives decreases Subcrop-Caption Matching performance across datasets and sDCI ablations.Source: §C.3.2 · p. 18

C.3.3 Linear Transferability

  • Linear-probe performance degrades slightly across Elevater datasets at all shot counts; sDCI trained without negatives loses less zero-shot ImageNet performance than its negatives-trained counterpart and is comparable to DAC.Source: §C.3.3 · p. 18

D Additional Selected Examples

The selected DCI examples illustrate dense descriptions of small details, complex scenes, and relationships between image regions.

D Additional Selected Examples

E Evaluating additional baselines on sDCI

Across additional pretrained baselines, larger models generally perform better, but no model excels on every task; the strongest models differ between subcrop matching and negative detection.

E Evaluating additional baselines on sDCI

F DCI Dataset Datasheet

This datasheet records DCI’s purpose and design: an initial resource of highly aligned image-text pairs for evaluating how existing models use them.

F.1 Motivation

  • DCI was created as an initial dataset of highly aligned image-text pairs for evaluating how well existing models use them.The dataset was created by Meta’s FAIR research team and funded by Meta.Source: §F.1 · p. 26

F.2 Composition

  • DCI contains 7,805 images randomly sampled from one SA-1B subset, with top-level captions and submask–subcaption pairs but no categorical labels.Annotations cover significant, not all, image content; annotator errors, omitted masks, and LLM-based caption-length augmentation may introduce noise.Source: §F.2 · p. 26

F.3 Collection Process

  • Images were collected from SA-1B through automated and manual annotation using Mephisto and custom interfaces, with crowdworkers paid above minimum wage.Collection took place from spring through fall 2023 and underwent internal review.Source: §F.3 · p. 26

F.4 Preprocessing/cleaning/labeling

  • The Segment Anything Model identified image regions for annotation, while raw images remained unmodified by extraction.Software used to construct the dataset was to be made available alongside the dataset release on the project GitHub.Source: §F.4 · p. 26

F.5 Uses

  • DCI is intended primarily as a test set, although train, validation, and test splits are provided for fine-tuning experiments; other uses are unplanned.Source: §F.5 · p. 26, §F.2 · p. 26

F.6 Distribution

  • The dataset is slated for broad download under CC-By-NC, with collection and preparation code on the project GitHub.Distribution was planned upon release of the associated publication.Source: §F.6 · pp. 26–27

F.7 Maintenance

  • Meta’s FAIR team will host DCI, but the authors specify no update schedule; known people in the images are face-blurred as in SA-1B.Others may extend or augment the dataset through the project GitHub under its terms.Source: §F.7 · p. 27

Related papers

Loading 2312.08578v2…