PaperlayerView exact revision on arXiv

Source-linked AI summary

DataComp-VLM: Improved Open Datasets for Vision-Language Models

Matteo Farina, Vishaal Udandarao, Thao Nguyen, Selim Kuzucu, Maximilian Böther, Andreas Hochlehnert, Adhiraj Ghosh, Marianna Nezhurina, Karsten Roth, Joschka Struber, Yuhui Zhang, Sebastian Dziadzio, Elaine Sui, Soumya Jahagirdar, Dhruba Ghosh, Hasan Hammoud, Thomas De Min, Simone Caldarella, Jehanzeb Mirza, Sedrick Keh, Mehdi Cherti, Hilde Kuehne, Bernt Schiele, Serena Yeung-Levy, Muhammad Ferjad Naeem, Federico Tombari, Ana Klimovic, Elisa Ricci, Matthias Bethge, Sewoong Oh, Ameya Prabhu, Alessio Tonioni, Jenia Jitsev, Massimiliano Mancini, Ludwig Schmidt, Nikhil Parthasarathy

arXiv:2606.28551v3cs.CVcs.CLcs.LG

TL;DR

Modern VLM data curation lacks systematic benchmarks despite relying on heterogeneous, already-curated datasets. DCVLM standardizes the pool, scaling, training, and evaluation setup to compare curation strategies, finding that instruction-heavy mixing outperforms additional filtering and yields a strong open baseline.

  • Problem

    Modern VLMs aggregate heterogeneous datasets with varying upstream curation, leaving the effects of filtering and mixture ratios poorly understood and largely irreproducible.

  • Method

    DCVLM fixes model architecture and training procedure while providing 160 datasets, four scales, and broad downstream evaluation for controlled curation experiments.

  • Results

    Instruction-heavy mixtures scale better than caption-heavy ones, while additional filtering on already-curated data provides diminishing or sometimes negative returns; DCVLM-BASELINE reaches 63.6% on the 33-task core set and exceeds FINEVISION by +5.4 pp.

  • Takeaways & Limitations

    Data mixture optimization, especially instruction-heavy mixing, is the most effective curation lever identified by DCVLM across model-size and compute scales.

  • Takeaways & Limitations

    The study is compute-limited to 200B pretraining tokens and does not empirically test model sizes of 20B or larger.

Abstract

from arXiv · show

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.

1 Introduction

DCVLM addresses the poorly understood and largely irreproducible role of data curation in modern autoregressive VLMs by enabling controlled experiments that vary data while fixing model and training procedure. Its experiments identify mixture optimization, especially instruction-heavy mixtures, as more effective than additional filtering, and produce a strong open baseline.

  • Motivation: Modern VLM data curation remains poorly understood because models aggregate heterogeneous, variably curated datasets while studies often focus on architectures, recipes, or evaluation.The open question is whether filtering, mixing ratios, or another choice drives quality under this aggregation-based regime.
  • Contribution: DCVLM introduces a benchmark that fixes model architecture and training procedure while varying data curation strategies for controlled comparisons.The benchmark is designed for the modern autoregressive VLM practitioner’s paradigm.
  • Contribution: 160 datasets span image-caption pairs, multimodal documents, text-only data, and multimodal instruction-tuning data within a 6T-token pool.The benchmark also provides a 1B–8B scaling ladder and evaluation across up to 52 benchmarks in 9 domains.
  • Findings: Additional filtering offers diminishing or sometimes negative returns on already-curated VLM data, whereas optimizing mixture ratios yields significant scale-dependent gains.Instruction-heavy mixtures outperform caption-heavy ones, with the gap widening as model size and token budget increase.
  • Findings: Pretraining rankings transfer to supervised fine-tuning with Pearson r = 0.99 across 54 SFT runs and remain similar across Qwen2.5-Base and Qwen2.5-Instruct initializations.This supports using pretraining-only metrics for data curation research with DCVLM.
  • Baseline: 63.6% accuracy on the 33-task core set lets DCVLM-BASELINE outperform FINEVISION by +5.4 pp at the 8B-model, 200B-token scale.The authors release the data pool, evaluation suite, checkpoints, and experimental infrastructure.

2 Related Work

Related work establishes data-centric benchmarking by fixing models and training procedures while varying data, but prior successes and practices leave VLM mixture and filtering choices insufficiently characterized.

  • Vision-Language Pretraining: Modern VLMs use modular vision-language architectures and increasingly diverse data, but precise mixture ratios, filtering criteria, and formatting choices remain largely unresolved.Earlier VLM pretraining relied predominantly on image-caption pairs, whereas recent systems train all parameters and incorporate diverse data types.
  • Benchmarking Data Curation: DataComp and DataPerf established controlled data-centric evaluation, while DCLM, FineWeb, and FineWeb-Edu demonstrated gains from filtering in language-model data.Related work generally fixes architecture and training procedure while varying only data.

3 The DCVLM Benchmark

DCVLM provides a controlled, scalable framework for constructing and evaluating VLM training datasets from a heterogeneous 160-dataset pool. Researchers can curate data across four compute scales, train with a fixed recipe, and assess models on broad capability benchmarks.

  • Benchmark Framework: DCVLM fixes the model and training recipe while participants filter and mix data from a shared pool to construct training sets.The resulting autoregressive VLMs are evaluated comprehensively across capabilities.
  • Data Pool Construction: The pool aggregates 160 publicly available datasets organized into image-caption, multimodal-document, text-only, and multimodal instruction-tuning data.These data types vary substantially in curation level and downstream role.
  • Data Pool Construction: The pool contains 6T multimodal tokens and deliberate heterogeneity across source quality, capabilities, visual and textual domains, and more than 20 languages.This setup supports realistic studies of data curation recipes.
  • Model Architecture and Training Recipe: The training recipe uses an InternViT-300M vision encoder, a 2-layer MLP projector, and Qwen2.5-Base, with AnyRes tiling and AdamW optimization.The architecture mimics InternVL3 models.
  • Competition Scales and Design Principles: Four scales—small, medium, large, and x-large—support curation studies across compute budgets, with model sizes and token budgets scaling together.The design uses a constant 30× pool-to-training-token ratio to permit aggressive filtering with constant data repetitions.
  • Evaluation: Participants can evaluate models on up to 52 benchmarks across 9 domains, filtered for stability and reliability.The evaluation is organized into validation, core, and extended tiers.

4 Towards a Strong Baseline on DCVLM

Controlled DCVLM experiments compare filtering and data mixing across scales. Filtering adds little to already-curated data, whereas instruction-heavy mixtures increasingly outperform alternatives as compute grows.

  • 4.1 Data Filtering: More than 60 filter configurations were tested at small and medium scales, using quality filters spanning image-text, text, and multimodal data.The experiments include CLIP-score, text-quality, and multimodal filtering approaches.
  • 4.1 Data Filtering: Local filtering preserves the global mixture, whereas global filtering changes source proportions and produces notably different performance outcomes.These fluctuations suggest that implicit mixture changes, rather than filtering alone, influence performance.
  • 4.1 Data Filtering: No tested quality filter robustly and significantly improves over the no-filter baseline, whether the global mixture is held fixed or allowed to change.The strongest outcome was a marginal +0.8pp improvement from globally filtering image-text pairs with SigLIP-2.
  • 4.1 Data Filtering: +2.4pp gain occurs with 25% pre-filtering, declining to +1.3pp at 65% and +0.6pp at 100%, showing diminishing returns from additional filtering.The result concerns downstream filtering applied after increasing amounts of upstream curation.
  • 4.2 Data Mixing: Instruction-heavy mixtures scale better: they begin worst at 1B×6.25B but become best at 2B×25B and remain best at 4B×25B.Mixture rankings at small scale therefore do not reliably transfer to larger scales.
  • 4.2 Data Mixing: The baseline study evaluates mixtures across 3 model sizes and 3 token budgets, then examines repetition robustness and transfer after supervised fine-tuning and across language-model backbones.The control experiments include 27 checkpoints and compare Qwen2.5-Base with Qwen2.5-Instruct.

5 Scaling Up Our Findings

DCVLM-BASELINE’s instruction-heavy mixture improves over FINEVISION across scales, with gains widening at larger scales and extending to broader evaluations. It also achieves higher performance with a smaller model and token budget.

  • Scaling Up Our Findings: The instruction-heavy DCVLM-BASELINE mix uses 10% Image-Caption, 5% Multimodal Documents, 15% Text-Only, and 70% Instruction-Tuning data.This mixture was found optimal at medium and large scales; the caption-heavy mixture was optimal at small scale.
  • Scaling Up Our Findings: +0.3pp, +1.1pp, +4.7pp, and +5.4pp are DCVLM-BASELINE’s progressive gains over FINEVISION on the 33-task Core evaluations from small through x-large scales.The gains increase with scale across all four benchmark scales.
  • Scaling Up Our Findings: A 4B model trained for 100B tokens on DCVLM-BASELINE outperforms an 8B model trained for 200B tokens on FINEVISION.This comparison corresponds to a fourfold reduction in compute.
  • Scaling Up Our Findings: 60.5% versus 56.6% is DCVLM-BASELINE’s x-large result versus FINEVISION’s x-large result on the 52-task Extended suite, an improvement of +3.9pp.The large-scale DCVLM-BASELINE model scores 56.0, nearly matching the x-large FINEVISION model.

6 Conclusion

The paper concludes that filtering already-curated VLM data provides little benefit, whereas scale-aware mixture optimization—especially instruction-heavy mixtures—improves performance. DCVLM offers a reproducible benchmark and infrastructure for studying these choices.

  • 6 Conclusion: DCVLM finds that individual quality filters provide negligible benefits when VLM source pools are already pre-filtered.The conclusion identifies this as one of the paper’s two central findings.
  • 6 Conclusion: Instruction-heavy data mixtures are the most effective curation lever, with gains that scale reliably with model size and compute.The conclusion contrasts mixture optimization with filtering.
  • 6 Conclusion: DCVLM releases 160 datasets, 52 benchmarks, model checkpoints at four scales, and experimental infrastructure as a reproducible testbed for future data research.The released artifacts cover the full data pool, evaluation suite, checkpoints, and infrastructure.
  • 6 Conclusion: DCVLM fixes model architectures and training pipelines while varying the data distribution to enable controlled comparisons of curation strategies.This benchmark paradigm follows earlier controlled data-curation benchmarks and is extended here to multimodal VLM data.
  • 6 Conclusion: The study uses a modular VLM with a pretrained vision encoder, language-model backbone, and connector, while recent systems increasingly train all parameters and use heterogeneous data.The paper situates its experiments within modern autoregressive VLM training regimes.

D Training and hyperparameter details

The appendix documents fixed training settings, learning-rate selection, and the composition and licensing of the 160-dataset DCVLM pool. It also details how datasets differ in samples, token totals, and tokens per sample.

  • D Training and hyperparameter details: The learning rate is selected from five values at each model scale using 10B training tokens with the base mixture, while other hyperparameters remain fixed.The sweep uses global batch size 1024, cosine scheduling, and 3% warmup.
  • D Training and hyperparameter details: The LR sweep used Qwen-Instruct backbones and 12 randomly selected benchmarks before the validation and Core sets were finalized.This setup differed from the later experimental configuration.
  • D Training and hyperparameter details: lr = 2 × 10−5 is consistently optimal across scales and backbones in the learning-rate sweep.Table 9 reports this as the best or second-best setting at every model scale and backbone group.
  • D Training and hyperparameter details: The DCVLM pool contains 160 datasets, 3.9B samples, and 6.0T multimodal tokens across image-caption, multimodal-document, text-only, and instruction-tuning data.The pool averages 1.5K tokens per sample.
  • D Training and hyperparameter details: Image-caption pairs account for 74% of samples and 83% of tokens, while text-only data accounts for 19% of samples and 5% of tokens.Visual tokens inflate image-caption token counts, whereas short text-only samples contribute no visual tokens.
  • D Training and hyperparameter details: Image-caption datasets cluster around 1–2K tokens per sample, while multimodal documents occupy a higher 3–6K regime.Text-only datasets span the widest range, from 100 to 15K tokens per sample.
  • D Training and hyperparameter details: The pool draws image-caption pairs from web-crawled and synthetic or human-curated sources, multimodal documents from web and PDF corpora, and instruction data from academic and synthetic sources.The listed sources include DataComp-1B, ReLAION-2B, PixMo-Cap, MINT-1T, and curated instructional datasets.
  • D Training and hyperparameter details: A Streamlit visualization tool supports browsing random samples from the 160 source datasets by data type and, for instruction-tuning data, capability.The tool is provided for quick visualization of source-dataset samples.

E.5 Multilingual Nature

The DCVLM pool is overwhelmingly English, with a smaller Chinese component and a thin multilingual tail. Language ablations suggest English carries nearly all measured signal, while multilingual performance remains essentially unchanged across language partitions.

  • Pool-wide distribution: English accounts for 91.1% of samples under Lingua and 92.8% under NLLB, while Chinese contributes 3.9% and 3.5%, respectively.The remaining languages are spread thinly across dozens of languages.
  • Sources of non-English data: The non-English tail comes partly from language-specific or synthetic datasets, including Chinese instruction and document corpora, translated captions, and multilingual generators.French, German, and Russian content is dominated by translated parallel corpora, while Japanese and Korean rely heavily on synthdog.
  • Implications: The pool is overwhelmingly English by design, and languages outside the top five are too sparse to sustain meaningful pretraining signal independently.Addressing this limitation would require deliberate ingestion of non-English vision-language datasets.
  • Language ablations: The English-only model trails the full pool by only 0.6pp despite discarding approximately 8% of the pool.This language ablation uses a fixed data mixture at the medium scale.
  • Language ablations: Multilingual performance is essentially flat across the all-language, English-only, and non-English-only pools: 39.8 / 39.9 / 39.1.The authors caution that repetition and noisy language annotations confound interpretation, especially for the non-English-only run.

F.1 Issues with Grounding Benchmarks

DCVLM excludes grounding benchmarks because COCO-derived datasets create severe contamination and split ambiguities that undermine held-out evaluation. The benchmark instead uses strict decontamination procedures to reduce train-test overlap in the retained suite.

  • Exclusion rationale: Grounding benchmarks were excluded because their contamination and split ambiguities make reliable data-centric evaluation difficult.The excluded examples include RefCOCO, RefCOCO+, and RefCOCOg.
  • Cross-benchmark overlap: Many grounding, VQA, and captioning benchmarks reuse overlapping MS COCO images, allowing one image to appear in training for one task and evaluation for another.Different annotation formats and custom splits make image-level decontamination difficult.
  • Split ambiguity: RefCOCO variants are not uniformly independent held-out datasets because their splits differ across image-level, object-level, and annotation-level schemes.They also differ in linguistic supervision, including expression length and whether spatial phrases are allowed.
  • Contamination risk: Open VLM recipes may group RefCOCO and RefCOCOg annotations under the same image, so nominally separate evaluation can measure familiarity with recurring COCO image-object pairs.This threatens interpretation of grounding generalization.
  • Exclusion rationale: Including grounding benchmarks would confound attribution because apparent gains could arise from uncontrolled COCO overlap rather than data curation choices.For this reason, grounding benchmarks are excluded from the main DCVLM evaluation suite.
  • Decontamination protocol: DCVLM performs strict two-way decontamination against the Extended evaluation suite, using separate procedures for multimodal images and text-only samples.The image pipeline uses SSCD similarity, while the text pipeline uses MinHash and exact substring matching.

G.2 Text-Based Decontamination

DCVLM decontaminates text-only training data by combining MinHash similarity screening with exact substring checks. Human annotation motivates the threshold choices, while the full protocol removes samples unevenly across sources.

  • Pipeline: Text-only decontamination matches concatenated human turns against prompts from 16 text-only evaluation benchmarks.The procedure follows the Tulu-3 protocol and operates at the prompt/question level.
  • MinHash matching: Training samples are represented by word-level 5-grams and 128-permutation MinHash signatures for approximate Jaccard matching.Exact brute-force signature comparison is feasible because the evaluation index contains approximately 86k samples.
  • Two-stage filtering: Samples with top-1 Jaccard similarity above 0.55 are discarded, while ambiguous matches from 0.3 through below 0.55 receive a bidirectional exact-substring check.The second stage recovers true overlaps whose Jaccard score is diluted below the first-stage threshold.
  • Hyperparameters: Five-grams detect meaningful overlap rates, whereas 8- and 13-grams detect almost none because many evaluation prompts are shorter.Increasing signatures from 128 to 1024 leaves match rates essentially unchanged.
  • Threshold selection: Human annotation found that true-positive rates rise steeply above approximately 0.55 and motivate the 0.55 first-stage threshold.Seven annotators labeled candidate matches across similarity bins.
  • Limitation: The tokenizer under-matches Chinese text because it is tuned for whitespace-delimited languages and does not segment Chinese well.This is an explicit limitation of the text decontamination procedure.
  • Removal rates: The full protocol removes samples unevenly, with concentration in sources sharing underlying training and evaluation distributions.InfoVQA loses 100% of samples, while pool-level removal remains much smaller.

H.1 Filtering rarely helps

Across filtering experiments, additional filtering rarely improves already curated VLM data, while gains diminish as upstream curation increases. The experiments also show that filtering can be implemented online without meaningful throughput or storage penalties.

  • Filtering results: Local filtering uniformly fails to improve over the no-filter baseline at the small scale.The only positive outcomes are global CLIPscore filters: SigLIP-2 +1.1pp and OpenAI-CLIP +0.6pp.
  • Filtering results: No tested configuration provides significant gains on the 13-task Validation suite.The experiments include CLIP-score, length heuristics, and image-quality assessment filters.
  • Diminishing returns: +2.4pp at 25% pre-filtering declines to +1.3pp at 65% and +0.6pp at 100%.The gain from downstream filtering shrinks as upstream curation increases.
  • Sampling control: Length-proportional sampling is near-optimal, while square-root sampling drops to 42.4% and near-uniform sampling to 40.0%.The broader cross-type mixing ratios are a more impactful lever than within-type temperature sampling.
  • Synthetic recaptioning: Synthetic captioning does not meaningfully improve average performance: original alt-text reaches 44.1%, versus 43.2% and 44.0% for two synthetic variants.The authors attribute the limited marginal value to high-quality instruction-tuning data already supplying densely described image-text pairs.
  • Filtering infrastructure: Online filtering matches unfiltered throughput and avoids materializing filtered subsets on disk.Runtime stays within 3.5% of baseline at 50M tokens and 0.3% at 500M tokens; three concurrent filters incur only 1.10× overhead.

I Fine-grained sweep of mixture optimization experiments

A finer-grained mixture sweep finds that the preferred image-caption versus instruction-tuning balance changes with scale. The section also describes streaming best-fit packing to satisfy token and image-tile constraints efficiently.

  • Mixture sweep: At small scale, the optimal mixture is 65% caption and 15% instruction data, with performance degrading as instruction share increases.The small-scale setting uses a 1B model and 6.25B tokens.
  • Mixture sweep: At medium scale, the best mixture reverses to 10% caption and 70% instruction data.This result demonstrates that curation strategies optimal at one scale need not transfer to larger scales.
  • Sequence packing: Each pack must respect both an L = 8192 token budget and an M = 24 image-tile budget.A pack completes when either budget is reached exactly.
  • Sequence packing: Incoming samples are assigned greedily to the fullest open buffer that satisfies both budgets.The buffers are sorted from fullest to emptiest, making the first feasible buffer a best-fit choice.
  • Sequence packing: A buffer is emitted when either budget is exactly full or the pool exceeds B = 20 open buffers.Pool overflow is the source of imperfectly filled examples and bounds memory and latency costs.
  • Sequence packing: Variable-length attention enforces independence among samples within a pack without re-padding.Cumulative segment boundaries are passed to a patched attention kernel to produce exact block-diagonal attention.

L.1 SFT Compute Budget and Learning-Rate Selection

SFT budgets are scaled in proportion to pretraining compute, and a learning-rate sweep selects a conservative SFT rate. Pretraining rankings remain predictive after fine-tuning across SFT datasets.

  • SFT compute budget: SFT budgets use an estimated 0.29 SFT-to-pretraining token ratio, yielding 1.8B, 3.6B, and 7.2B SFT tokens.The budgets correspond to 6.25B, 12.5B, and 25B pretraining-token checkpoints.
  • Learning-rate selection: The lowest tested SFT learning rate, 4 × 10^-6, is best on Core Avg at both checkpoints.The highest rate, 4 × 10^-5, most visibly reduces Text performance by 4.4pp and 4.5pp.
  • Transferability: Using Mammoth-VL-12M preserves Pearson r = 0.99 and Spearman ρ = 0.99 across all 27 checkpoints.The pretraining ranking remains preserved across the alternative SFT dataset.

M Extended Suite Evaluations

On the Extended suite, DCVLM-BASELINE retains an overall advantage over FINEVISION, especially on Vision-Centric and General Understanding tasks. However, it has substantial weaknesses on Safety, OCR, and Reasoning.

  • Overall comparison: +3.0pp and +3.9pp are DCVLM-BASELINE’s gains over FINEVISION at large and x-large scales.The Extended suite contains 52 tasks and adds Hallucination, Safety, and Reasoning categories.
  • Strengths: +23.1pp / +21.3pp in Vision-Centric and +8.4pp / +8.5pp in General Understanding are the largest gains.The paired values correspond to large and x-large scales.
  • Strengths: +3.4pp / +3.0pp on Hallucination favors DCVLM-BASELINE at both scales.The authors interpret this as better-calibrated visual grounding.
  • Weaknesses: Safety performance is 17.7pp / 19.2pp lower for DCVLM-BASELINE than FINEVISION.The authors identify safety alignment as a gap for future work.
  • Weaknesses: FINEVISION leads DCVLM-BASELINE on OCR by 6.5pp / 5.2pp and Reasoning by 4.3pp / 4.9pp.The authors associate these advantages with FINEVISION’s document-rich sources.

N.2 Image-level Safety Analysis

The study assesses image-level safety in a 20,000-image subset using ShieldGemma2, identifying images associated with dangerous, sexually explicit, and violent content.

  • 20,000 image–text pairs were analyzed for image safety using ShieldGemma2, a 4B-parameter image content moderation model.The model provides safety risk predictions across sexually explicit, violence and gore, and dangerous-content categories.
  • 117 images were classified as sexually explicit, 19 as dangerous, and 4 as containing violence and gore.
  • ShieldGemma2 was built on Gemma 3 and supplies confidence-scored predictions across multiple image-safety categories.

O Per-Dataset Core Results

The per-dataset results decompose the Core evaluation into category-specific benchmark tables, whose row means reproduce the aggregate category scores and overall Core average.

  • Each table corresponds to one of six Core categories and lists every constituent benchmark for the same models and scales as Table 3.
  • Averaging across all 33 datasets in the six tables recovers the Core average.
  • General Understanding: The General Understanding table averages seven datasets to reproduce its category score in Table 3.
  • Knowledge-Centric: The Knowledge-Centric table averages four datasets to reproduce its category score in Table 3.
  • OCR & Charts; Vision-Centric: The OCR & Charts table averages six datasets, while Vision-Centric averages five, to reproduce their respective Table 3 category scores.
  • Multilingual; Text-Only Understanding: The Multilingual table averages three datasets, and the Text-Only Understanding table averages eight, to reproduce their respective category scores.

P Limitations and Future Directions

The paper identifies scale, model size, pretraining paradigm, document transformation, mixture optimization, and data-quality scoring as open directions and scope boundaries for its findings.

  • The paper’s limitations and future directions are discussed in a dedicated Limitations and Future Directions section.
  • Scaling to longer token budgets: The experiments scale to 8B models and 200B tokens, but the authors remain compute-limited relative to open-weight models trained on 1.4T–2.2T tokens.
  • Scaling to larger model sizes: Whether the data-pool and mixture conclusions hold for model sizes ≥20B remains empirically untested.
  • Testing DCVLM data on native multimodal pretraining: The study uses pretrained vision encoders and language models, so instruction-heavy mixtures may not dominate under native multimodal pretraining from scratch.
  • Towards instructional multimodal documents: The results motivate transforming raw multimodal documents into denser instructional signals while preserving their breadth and diversity.
  • Quality scores might be used to upsample samples rather than discard data, because filtering can be exclusionary and may induce harmful biases.
Loading 2606.28551v3…