PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept-Aware Batch Sampling Improves Language-Image Pretraining

Adhiraj Ghosh, Vishaal Udandarao, Thao Nguyen, Matteo Farina, Mehdi Cherti, Jenia Jitsev, Sewoong Oh, Elisa Ricci, Ludwig Schmidt, Matthias Bethge

arXiv:2511.20643v1cs.CVcs.LG

TL;DR

Vision-language data curation often uses offline, concept-agnostic filtering, while concept-level distributions can vary with downstream evaluation. This paper pairs an annotated 128M-pair dataset with online concept-aware batch sampling; evaluations report benefits across classification and retrieval, with some gains over strong baselines.

  • Problem

    Many curation methods focus on individual-sample quality and operate offline without controlling concept-level data distributions.

    Source: §1 · p. 2

  • Method

    DataConcept annotates 128M image-text pairs with grounded concept information, and CABS uses those annotations to sample task-adaptive batches online.

    Source: §1 · p. 2

  • Results

    Across 28 tasks, CABS benefits CLIP and SigLIP pretraining, with gains up to 7% on ImageNet zero-shot classification and up to 9.1% on image-text retrieval over strong baselines.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    CABS provides flexible control over concept distributions and shows benefits across classification and retrieval compared with IID and other curation strategies.

    Source: §8 · p. 11

    Only 79% of the medium-scale pool had been downloaded as of September 2024.

    Source: §2 · p. 3

Abstract

from arXiv · show

What data should a vision-language model be trained on? To answer this question, many data curation efforts center on the quality of a dataset. However, most of these existing methods are (i) offline, i.e. they produce a static dataset from a set of predetermined filtering criteria, and (ii) concept-agnostic, i.e. they use model-based filters which induce additional data biases. In this work, we go beyond such offline, concept-agnostic methods and advocate for more flexible, task-adaptive online concept-based curation. Our first contribution is DataConcept, a collection of 128M web-crawled image-text pairs annotated with fine-grained details about their concept composition. Building on DataConcept, we introduce Concept-Aware Batch Sampling (CABS), a simple yet effective batch sampling framework that flexibly constructs batches on-the-fly based on specific target distributions. We propose two variants: (i) Diversity Maximization (CABS-DM) to curate batches with a broad coverage of available concepts, and (ii) Frequency Maximization (CABS-FM) to curate batches with high object multiplicity. Through extensive evaluations across 28 benchmarks, we demonstrate that our CABS method significantly benefits CLIP/SigLIP model classes and yields highly performant models. Overall, CABS represents a strong open-source alternative to proprietary online data curation algorithms, enabling practitioners to define custom concept distributions that optimize for specific downstream tasks.

1 Introduction

The paper argues for moving beyond offline, concept-agnostic filtering toward online, task-adaptive control of concept distributions. It introduces an annotated 128M-pair dataset and a framework for constructing batches around target concepts.

Motivation

  • Existing curation often filters individual samples offline, overlooking concept-level distributions, fixing the retained data subset, and risking bias from black-box models.The paper notes that discarded data is difficult to repurpose for other curation strategies.Source: §1 · p. 2

Contributions

  • DataConcept provides 128M image-text pairs with semantic concepts, bounding boxes, per-concept confidence scores, and concept-driven synthetic captions.The dataset is introduced to support modulation of visual concepts during pretraining.Source: §1 · p. 2
  • CABS constructs batches online for flexible, task-adaptive control rather than imposing a fixed, predetermined data distribution.Its variants maximize concept diversity for classification or concept multiplicity for retrieval.Source: §1 · p. 2
  • CABS-DM selects samples for concept diversity, while CABS-FM favors samples containing more objects.The paper describes these as task-adaptive online batch-sampling variants.Source: §1 · p. 2

Results

  • CABS improves vision-language pretraining across 28 tasks, with gains up to 7% on ImageNet zero-shot classification and up to 9.1% on image-text retrieval over strong baselines.The evaluations cover four visual backbones and both CLIP and SigLIP training objectives.Source: §1 · p. 2

2 Concept-Aware Dataset Augmentation

DataConcept is built from a 128M-pair image-text pool and augmented with grounded concept annotations and concept-aware synthetic captions.

Initial pool

  • The initial pool contains 128M image-text pairs, sampled from DataComp’s 12.8B-item XLarge pool because the standard download protocol suffers from link-rot.The authors report successfully downloading only 79% of the medium-scale pool as of September 2024.Source: §2 · p. 3

Concept bank

  • The concept bank combines sources including RAM++, V3Det, and OpenImages, yielding 19,261 concepts after deduplication and safety removal.The authors report a final vocabulary of 12,253 concepts across the annotated samples.Source: §2 · p. 3

Tagging and grounding

  • RAM++ supplies sample-level concept tags, and GroundingDINO adds concept-specific boxes to address confidence miscalibration and missing localization.Confidence seeding uses tags with at least 0.75 confidence, while resolution ensembling combines predictions across four image resolutions.Source: §2 · p. 3

Concept-aware recaptioning

  • The pipeline adds synthetic captions by prompting Qwen2-VL-7B with each sample’s detected concepts and original alt-text.The generated concept-aware caption is denoted R_i.Source: §2 · p. 4

Dataset output

  • DataConcept samples combine concept tags with confidence scores, localized bounding boxes, and concept-aware synthetic captions.Each sample is denoted (I_i, T_i, R_i, C_i).Source: §2 · p. 4

3 Concept-Aware Batch Sampling

CABS uses concept annotations to construct training batches tailored to task-relevant concept distributions, with variants designed for classification and retrieval.

3.1 Formulation

  • CABS computes concept-aware scores for samples in an IID-drawn superbatch, then selects a target-size sub-batch to induce a chosen concept distribution online.The target batch size is b = (1 − f)B; setting all scores equal recovers IID sampling.Source: §3.1 · p. 4

3.2 Task-Adaptivity of CABS

  • Classification motivates balanced concept coverage, whereas retrieval motivates batches rich in multi-concept samples; CABS-DM and CABS-FM target these respective patterns.The table identifies each variant’s heuristic, parameters, and whether scoring depends on the current sub-batch state.Source: §3.2 · p. 5, Table 1 · p. 5
  • In 4,096 sampled images, ImageNet images tended to contain single objects while MSCOCO retrieval images tended to contain multiple objects.The two CABS variants approximate these observed task characteristics in selected samples.Source: §3.2 · p. 5

3.3 Experimental Setup

  • Experiments use CLIP and SigLIP models, 25 classification and two retrieval benchmarks plus a long-tailed test set, and a 128M-sample training budget.Unless specified otherwise, the filter ratio is 0.8 and superbatches contain 20,480 samples; concepts guide curation but not the contrastive objective.Source: §3.3 · p. 5

4 CABS with Diversity Maximization

CABS-DM builds batches to balance concept coverage, and improves zero-shot classification against IID, offline curation, and online sampling baselines. The section also reports gains from concept-aware re-captions.

4.1 Formulation

  • CABS-DM iteratively selects samples containing under-represented concepts until each concept reaches a tunable upper count, producing a mostly flat distribution.An average curated sub-batch contains 1.5× more concepts than an IID-sampled batch.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Improvements on Zero-shot Classification

  • +5.0% for CLIP ViT-B-32 and +6.9% for SigLIP B-16-256 on ImageNet, with 1.0–2.4% gains on Let-It-Wag!, over IID sampling.CABS-DM also boosts average accuracy across the broader benchmark suite and model variants.Source: §4.2 · p. 7, Table 2 · p. 7
  • Concept-aware re-captions improve zero-shot classification over alt-texts, including +4.3% on ImageNet and +4.8% for CLIP ViT-B/32.For SigLIP ViT-B/16, the reported accuracy gains reach +11.6% and +12.2%.Source: §4.2 · p. 7

4.3 Improvements over State-of-the-art Methods

  • CABS-DM beats MetaCLIP by +3.8% on ImageNet and +2.9% on the average classification set, with further gains on long-tailed evaluations.The comparison is between online concept-balanced batch sampling and offline curation with a similar objective.Source: Table 3 · p. 7, §4.3 · p. 8
  • Against online samplers, CABS-DM improves up to +6.8% on ImageNet and +3.7% on average with SigLIP-ViT-B/16.GRIT-VLP and MAFA lag behind CABS-DM; with CLIP, they also struggle to outperform IID.Source: §4.3 · p. 8, Table 4 · p. 8

5 CABS with Frequency Maximization

CABS-FM favors image-text samples containing more annotated concepts, aiming to supply concept-dense batches for retrieval. It outperforms IID and other online samplers on retrieval benchmarks.

5.1 Formulation

  • CABS-FM scores each sample by its number of annotated classes and selects the top-scoring samples to form a concept-dense sub-batch.The method is motivated by retrieval scenes with multiple objects and uses frequency maximization to favor higher object multiplicity.Source: §5.1 · p. 8

5.2 Experiment Results

  • CABS-FM beats IID across configurations, with average retrieval gains of +3.5% for ViT-B-32-CLIP and +3.1% for ViT-B-16-SigLIP-256 on alt-text.With re-captions, those gains widen to +9.0% and +4.6%, respectively.Source: §5.2 · p. 9, Table 5 · p. 9
  • CABS-FM improves over GRIT-VLP by +3.9% for ViT-B-32-CLIP and +2.5% for ViT-B-16-SigLIP-256.GRIT-VLP and MAFA fail to surpass IID for ViT-B-32-CLIP and offer only modest improvements for ViT-B-16-SigLIP-256.Source: §5.2 · p. 9, Table 5 · p. 9

6 Data- & Compute-Constrained Experiments

The experiments test CABS under compute- and data-constrained training, including filtered datasets and extended training. CABS remains competitive across these settings, even with substantially more repeats.

6 Data- & Compute-Constrained Experiments

  • With CLIP-score-filtered data, CABS variants remain effective, and CABS-DM beats IID despite using a 2× higher data-repeat rate.The paper reports that CABS-FM also has unanimous gains across tested benchmarks and models under the same repeat protocol as CABS-DM.Source: Table 6 · p. 9, §6 · p. 10, Table 7 · p. 10
  • Under compute-constrained training, IID needs 3.2× more steps to reach CABS-DM’s ImageNet performance and 2× more to reach CABS-FM’s average retrieval performance.The comparison uses a ViT-B/32 CLIP backbone trained for 1.28B samples seen; gains diminish only slightly in the data-constrained regime.Source: §6 · p. 10
  • At the 50-repeat worst case, CABS remains competitive with IID, which uses 10 repeats, and the authors report compatibility with smaller curated datasets and multi-epoch web-corpus pretraining.For CABS, the paper hypothesizes that this extreme repeat setting over a comparatively small sample pool explains the slight performance decline.Source: §6 · p. 10

7 Related Work

Prior work examines data quality and concept distributions, and develops sampling methods that move beyond IID selection. This paper situates concept-aware batch construction within that line of work.

7 Related Work

8 Conclusion

The paper presents DataConcept and CABS as a flexible approach to concept-aware pretraining, with benefits across classification and retrieval evaluations. It also identifies annotation cost, filtering runtime, and limited training-scale coverage as constraints.

8 Conclusion

  • CABS outperforms IID and other curation strategies, including existing batch-sampling algorithms, across classification and retrieval tasks.The paper describes CABS as versatile and publicly releases DataConcept and CABS to encourage concept-awareness in data pipelines.Source: §8 · p. 11
  • DataConcept provides 128M concept-annotated image-text pairs, while CABS performs flexible online concept-aware batch sampling during pretraining.The paper frames concept-level information as an underexplored part of data-centric work.Source: §8 · p. 11
  • CABS annotation costs are amortizable through reuse, but runtime rises with the superbatch filtering ratio, and complex architectures and large-scale training runs remain untested.The authors report performance benefits even at low filtering ratios, where runtime overhead is more manageable.Source: §8 · p. 11

Supplementary Material

The supplementary material details how DataConcept’s concept vocabulary and object annotations are constructed, then describes multi-resolution box fusion and its evaluation. It also identifies dataset-wide and sample-level concept-count distributions as analyses relevant to CABS variants.

A.1 Vocabulary Construction

  • The vocabulary combines broader long-tailed sources and undergoes morphological, syntactic, semantic, and safety filtering, yielding 19,261 concepts.Semantic filtering removes near-identical concepts while preserving related but distinct terms.Source: §A.1 · p. 19

A.2 Object Tagging

  • Expanded-vocabulary tagging is miscalibrated; the authors select RAM++ confidence threshold 0.75, since 0.8 leaves substantially more images without tags.The supplementary figure also identifies 0.75 as the most suitable pool of concepts among the three thresholds tested.Source: §A.2 · p. 20, Figure 5 · p. 21

A.3 Object Detection

  • Because tags lack localization and captions may not match image content, RAM++ tags prompt GroundingDINO, which returns localized detections and bounding boxes.Using the full 19,261-concept pool as input risks hallucinated detections and slower processing; the approach applies confidence and similarity thresholds to retained detections.Source: §A.2 · p. 20, §A.3 · p. 22
  • DataConcept-M contains 12,253 unique concepts, which form CABS’s concept pool, although strict annotation protocols leave extremely rare concepts underrepresented..Source: §A.3 · p. 22

A.4 Weighted Box Fusion for Ensembling Bounding Boxes

  • Weighted Box Fusion combines same-class predictions across resolutions by confidence-weighted coordinate averaging, producing recalibrated annotations for DataConcept.The method uses predictions from resolutions 384, 512, 800, and 1000; unlike suppression methods, it aggregates overlapping boxes.Source: §A.3 · p. 22, §A.4 · pp. 24–25

A.5 Ensembling: Quantitative Results

  • Ensembling across all four resolutions provides the best bounding boxes in the ODinW evaluation, including the reported overall detection-accuracy comparison.The evaluation tests single resolutions and combinations using average precision for 10 selected classes.Source: §A.5 · p. 26, Table 8 · p. 26, Table 9 · p. 27

A.6 Concept Distribution

  • The concept-distribution analysis examines dataset-wide and sample-level concept counts, which inform different CABS variants..Source: §A.6 · p. 28

A.6.1 Dataset-wide Concept Count

DataConcept’s concept annotations reveal a long-tailed distribution across the dataset and generally low per-sample concept counts. The section also describes how Qwen2-VL-7B was selected for recaptioning and how its captions compare with raw alt-text.

A.6.1 Dataset-wide Concept Count

  • DataConcept has 486,303,998 annotations, ranging from 1 to 20,974,722 for man, a median concept count of 489, and an extreme long-tailed concept distribution.86 concepts have more than 1 million annotations.Source: §A.6.1 · p. 28, Figure 7 · p. 28

A.6.2 Sample-level Concept Count

  • DataConcept samples have a median of 3 detected concepts, counting repeated detections rather than only unique concepts.The authors interpret the low counts as evidence of a bias toward lower-complexity samples in IID batches.Source: §A.6.2 · p. 30, Figure 8 · p. 30

B.1 Selecting the Recaptioning VLM

  • Qwen2-VL-7B was selected for recaptioning because it combines prompt adherence and caption quality with relatively low hallucinations and moderate processing speed.Moondream2 was faster but lower fidelity; Molmo-7B-D-0924 was high quality but slower, more verbose, and more prone to hallucinations.Source: §B.1 · p. 31, Figure 9 · p. 31

B.2 Caption Quality

  • Qwen2-VL-7B recaptions have a median of 33.56 words versus 6 for raw alt-text, and synthetic recaptions improve measured concept adherence.The reported standard deviations are 16.45 for recaptions and 9.51 for raw captions.Source: §B.2 · p. 32

B.3 Qualitative Evaluation: Visualization Results

The qualitative examples illustrate concept-aware synthetic captions, while the accompanying code and training details describe CABS-DM and CABS-FM batch selection and implementation.

B.3 Qualitative Evaluation: Visualization Results

  • A Chinese book-cover alt-text is recaptioned in English with a translated title and visual details about its globe image and green-yellow color scheme.The authors describe the examples as showing concept adherence and multilingual understanding.Source: Figure 11 · p. 33, §B.3 · p. 34

C.1 CABS-DM

  • CABS-DM greedily selects samples with marginal gains toward concept-balancing targets, while respecting a maximum per-concept frequency.Its heuristic weights unmet targets by global concept frequency.Source: §C.1 · p. 37

C.2 CABS-FM

  • CABS-FM scores each sample by its number of detected concepts and favors samples with higher multiplicity.The code labels this score as frequency maximization.Source: §C.2 · p. 38

C.3 Hyperparameters

  • CABS is integrated into the open_clip codebase, with DataComp hyperparameters retained to support reproducibility of IID comparisons.The paper also provides general pretraining and CABS-specific hyperparameters.Source: §C.3 · p. 39, Table 11 · p. 39

D.1 Evaluation Suite: Further Details

The evaluation uses 28 benchmarks, and the reported results show CABS gains across additional model architectures and in continual pretraining. The continual-pretraining experiments extend training from an IID initialization.

D.1 Evaluation Suite: Further Details

  • The final evaluation pool contains 28 benchmarks: 26 zero-shot classification and 2 image-text retrieval tasks.The authors omit some proposed benchmarks after citing concerns about noise, errors, and variability.Source: §D.1 · p. 40, Table 12 · p. 40

D.2 Full Model Suite

  • CLIP ViT-S-16 and SigLIP ViT-SO400M show strong CABS gains, consistent with the reported results for other model variants.The authors conclude that CABS performs across varied model architectures and sizes.Source: §D.2 · p. 41, Table 13 · p. 41, Table 14 · p. 42

E Continual Pretraining

  • In continual pretraining, CABS-DM and CABS-FM outperform IID sampling across benchmarks for both alt-text and synthetic recaptions.The experiment continues a CLIP ViT-B/32 IID-trained initialization for another 128M samples, reaching 256M samples seen; the tables separate classification and retrieval results.Source: E Continual Pretraining · p. 43, Table 15 · p. 43, Table 16 · p. 43

F Ablation on Filter Ratios

The section examines the filter ratio and benchmark-level outcomes for CABS-DM, then extends comparisons to MetaCLIP. It reports gains on most classification benchmarks and a retrieval advantage for CABS-FM over MetaCLIP in the stated settings.

F Ablation on Filter Ratios

  • At the 128M-sample scale, filter ratio f = 0.8 is optimal among the tested values 0.5, 0.75, 0.8, and 0.9.The ablation is for CLIP ViT-B/32 on ImageNet; the figure caption says the ratio was also retained for CABS-FM.Source: F Ablation on Filter Ratios · p. 44, Figure 15 · p. 44

G Fine-grained Benchmark Performance

  • The authors argue that probing benchmarks within a complete evaluation suite can deepen understanding of model capabilities beyond commonly reported aggregate performance, a principle studied for language and autoregressive vision-language models that may also apply to CLIP.Source: G Fine-grained Benchmark Performance · p. 45

G.1 Expanded Analysis

  • CABS-DM outperforms IID sampling on 23 of 26 benchmarks in the cited CLIP ViT-B/32 comparison.The expanded analysis reports gains on datasets testing both long-tailed and common concepts.Source: §G.1 · p. 45, Figure 18 · p. 47

G.2 MetaCLIP: Further Details

  • CABS-FM outperforms MetaCLIP for CLIP ViT-B/32 and SigLIP ViT-B-16/256 at f = 0.8, using 25.6M samples.The comparison also includes CABS-DM against MetaCLIP; the filtered-pool IID baseline reached 15.1% ImageNet accuracy and underperformed unfiltered IID training.Source: §G.2 · p. 50, Table 17 · p. 50
Loading 2511.20643v1…