PaperlayerView exact revision on arXiv

Source-linked AI summary

Train a Unified Multimodal Data Quality Classifier with Synthetic Data

Weizhi Wang, Rongmei Lin, Shiyang Li, Colin Lockard, Ritesh Sarkhel, Sanket Lokegaonkar, Jingbo Shang, Xifeng Yan, Nasser Zalmout, Xian Li

arXiv:2510.15162v1cs.CVcs.CL

TL;DR

High-quality filtering for interleaved multimodal documents is under-explored, although MLLMs are pretrained on both caption and interleaved data. UniFilter trains an efficient MLLM classifier on semi-synthetic examples to filter both types. Its filtered data improves benchmark performance over baseline filters, including after visual fine-tuning.

  • Problem

    High-quality filtering for interleaved image-text documents is under-explored, while CLIPScore-based filtering is limited to single image-caption pairs.

    Source: §1 · p. 2

  • Method

    UniFilter trains an efficient MLLM data-quality classifier on sample-score pairs made by combining available images with generated text across four quality levels.

    Source: §1 · p. 2

  • Results

    UniFilter beat DFN and MLMFilter on all 5 zero-shot VQA datasets; after visual SFT, it surpassed the best baseline by +3.1 average VQA accuracy and +1.5 MMMU accuracy.

    Source: §1 · p. 3, §4.3 · p. 10

  • Takeaways & Limitations

    High-quality caption and interleaved data curated by UniFilter is associated with improved MLLM capabilities, with benefits persisting after supervised fine-tuning.

    Source: §1 · p. 3, §6 · p. 11

Abstract

from arXiv · show

The Multimodal Large Language Models (MLLMs) are continually pre-trained on a mixture of image-text caption data and interleaved document data, while the high-quality data filtering towards image-text interleaved document data is under-explored. We propose to train an efficient MLLM as a Unified Mulitmodal Data Quality Classifier to Filter both high-quality image-text caption and interleaved data (UniFilter). To address the challenge of collecting diverse labeled multimodal data, we introduce a semi-synthetic approach that leverages readily available raw images and generates corresponding text across four quality levels. This method enables efficient creation of sample-score pairs for both caption and interleaved document data to train UniFilter. We apply UniFilter to curate high-quality caption data from DataComp caption dataset and interleaved data from the OBELICS image-text interleaved dataset. MLLMs pre-trained on the filtered data demonstrate significantly enhanced capabilities compared to those trained on baseline-filtered data, achieving stronger zero-shot reasoning and in-context learning capabilities. After visual supervised fine-tuning, these UniFilter-induced MLLMs achieve stronger performance on various benchmarks, highlighting the downstream benefits of high-quality multimodal pre-training. We release the synthetic training data used for training UniFilter, the UniFilter model checkpoints, and the high-quality interleaved document subset OBELICS-HQ, curated by UniFilter, to the community for reproduction and further development.

1. Introduction

The introduction frames data quality as a bottleneck for multimodal pretraining and highlights the lack of filters for interleaved documents. UniFilter combines a unified MLLM classifier with semi-synthetic training data, and its filtered data improves pretraining and post-SFT benchmark performance over baselines.

1. Introduction

  • Because large-scale MLLMs train on less than one epoch of mined data, data quality is a major bottleneck, while CLIPScore handles only single image-caption pairs, not interleaved documents.Interleaved documents combine multiple images and long text paragraphs.Source: §1 · p. 2
  • UniFilter uses an efficient MLLM to score both caption and interleaved samples, trained on sample-score pairs generated by proprietary MLLMs across four quality levels.The synthetic labels correspond to scores 0, 1, 2, and 3.Source: §1 · p. 2
  • UniFilter beat DFN and MLMFilter on all 5 zero-shot VQA datasets; interleaved-data pretraining gained +0.7 and +2.8 average scores on 4-shot and 8-shot VQA.The introduction also reports +3.1 average improvement on VQA tasks and +1.5 on MMMU after instruction tuning.Source: §1 · p. 3
  • After visual supervised fine-tuning, UniFilter-pretrained MLLMs achieved +3.1 average improvement on VQA tasks and +1.5 improvement on the MMMU benchmark.The paper reports these gains over baseline-filtered models.Source: §1 · p. 3

2. Synthetic Data Construction

The paper constructs classifier training data by pairing real images with generated text at controlled quality levels. This semi-synthetic approach supplies diverse labeled examples for both captions and interleaved documents.

2. Synthetic Data Construction

  • Controlled synthetic data can be generated at scale, providing sufficient classifier training examples and clear boundaries among four quality levels.The authors state that this approach enhances classifier generalization across quality levels.Source: §2 · p. 3

2.1. Define Data Quality Requirements

  • The taxonomy replaces binary labels with easy negative, medium negative, hard negative, and positive categories spanning irrelevant samples, significant errors, subtle mismatches, and well-aligned data.Prompts guide Claude-3-Sonnet to generate examples for the defined quality levels.Source: §2.1 · pp. 3–4

2.2. Semi-Synthetic Data Generation

  • The semi-synthetic pipeline samples original web-crawled images and uses Claude-3-Sonnet to generate corresponding text at varied quality levels, combining visual realism with controlled text generation.The authors describe the process as scalable and efficient for creating diverse classifier-training data.Source: §2.2 · pp. 4–5
  • The dataset contains 40k synthetic captions and 40k synthetic interleaved documents across four score levels, plus 4k positive MSCOCO and Flickr captions; the combined pairs train one classifier.Five percent of the 80k synthetic examples are held out for validation, and generated text is scanned for safety concerns.Source: §2.2 · p. 5

3. UniFilter Architecture

UniFilter uses an MLLM architecture to process caption and interleaved inputs, with ablations balancing classification quality and inference efficiency. The selected configuration uses SigLIP, AvgPool, and Qwen-2.5-0.5B.

3. UniFilter Architecture

  • The architecture separately encodes interleaved images and text before restoring their original order; caption image encodings and text embeddings are concatenated, and a classification head outputs a quality logit.A single MLLM-based classifier handles both sample types.Source: §3 · pp. 5–6
  • The ablations address the cost of billion-parameter MLLMs by varying the vision encoder, visual projector, and LLM backbone, with image-token compression used to support efficiency.The study compares non-compressive MLP projection with compressive adaptive average pooling.Source: §3 · p. 6
  • The SigLIP-SO400M, AvgPool, and Qwen-2.5-0.5B configuration offers the best trade-off between quality-classification performance and efficiency.Qwen-2.5-1.5B performs best on classification but adds computational overhead; the chosen model achieves inference speed comparable to DFN-CLIP-Large.Source: Figure 3 · p. 7, §3 · p. 7

4. Experiments

Across caption filtering, interleaved-data filtering, and visual supervised fine-tuning, the experiments report stronger MLLM performance with UniFilter-curated data than with baseline-filtered data. They evaluate zero-shot VQA, multimodal in-context learning, and instruction-tuned benchmarks.

4.1. MLLM Pre-Training on Image-Text Caption Data Only

  • On average across five multimodal benchmarks, UniFilter-curated caption data outperformed DFN and all MLM-Filter metrics.The comparison uses DataComp-medium-128M, a noisy web-crawled caption pool with basic rule-based filtering; MLM-CTQ remained stronger on TextVQA, an OCR-focused task.Source: §4.1 · pp. 7–8

4.2. MLLM Pre-Training on Mixed Image-Text Caption and Interleaved Data

  • UniFilter improved average VQA performance over DFN by +3.2 in zero-shot, +0.7 in 4-shot, and +2.8 in 8-shot in-context learning.It outperformed the DFN variant on GQA, VizWiz, OKVQA, and TextVQA, but slightly lagged on VQA-v2; 4/8-shot accuracy averages five random seeds.Source: Table 4 · p. 9, §4.2 · p. 9

4.3. Visual Supervised Fine-Tuning

  • UniFilter pre-training surpassed the best baseline by +3.1 average VQA accuracy, +1.5 MMMU accuracy, and +1.6 MMBench accuracy after visual supervised fine-tuning.All pre-trained models benefited from visual SFT on out-of-distribution VQA tasks, while the SFT-only baseline lagged behind all pre-trained and fine-tuned models.Source: Table 5 · p. 10, §4.3 · p. 10

5. Related Work

Related work covers data-quality filtering for LLM and MLLM pre-training, including classifiers trained on synthetic labels. Prior multimodal approaches include caption filters and synthetic-score classifiers.

Data Filtering for LLM and MLLM Pre-training

  • LLM filtering approaches include educational-value classifiers, synthetic positive-versus-web negative data for binary classification, and multi-way quality scores.The passage discusses Phi, FineWebEdu-Classifier, Llama-3, and DCLM approaches.Source: §5 · p. 10
  • For multimodal caption data, LAION uses CLIPScore-based filtering, while BLIP uses Cap-Filt to generate high-quality training data.Source: §5 · p. 10

Data Quality Classifier Trained with Synthetic Data

  • Synthetic multimodal score classifiers include MLM-Filter's 100-way scores across four metrics and AITQE's unified 0–10 quality metric.Source: §5 · p. 10

6. Conclusion

The conclusion presents UniFilter as a unified MLLM-based classifier for caption and interleaved data. It links pre-training on the selected data with enhanced downstream capabilities.

6. Conclusion

  • UniFilter is an MLLM-based classifier designed to filter both image-text caption and interleaved data.Source: §6 · p. 11
  • The conclusion reports that pre-training on UniFilter-curated high-quality data can enhance general-purpose MLLMs' downstream-task capabilities.Source: §6 · p. 11
  • The stated scope includes improving zero-shot and few-shot multimodal in-context learning in pre-trained and fine-tuned MLLMs through unified data filtering.Source: §6 · p. 11

A. Additional Results on DataComp-Medium

DataComp-Medium compares filters on a fixed image-text pool using induced CLIP performance across 38 classification and retrieval tasks. UniFilter’s strongest results come from combining its selected data with DFN’s, while dataset-level comparisons reveal specific weaknesses.

Evaluation setting

  • DataComp-Medium fixes the image-text pool and controls training code and compute across filters, evaluating induced CLIP models on 38 classification and retrieval tasks.The reproduced comparisons use the 99.6M downloadable samples because 22.3% of the original 128M URLs were no longer downloadable as of June 2024.Source: A. Additional Results on DataComp-Medium · pp. 13–14
  • The baseline set includes no filtering, text- and image-based filters, CLIPScore, and DFN, alongside their specified combinations.Source: A. Additional Results on DataComp-Medium · p. 13

Results

B. Training Settings of MLLM Pre-Training

MLLM pre-training updates the visual projector and LLM backbone in one stage with the vision encoder frozen. The described datasets are URL-based, and only about half of OBELICS documents were fully downloadable as of June 2024.

B. Training Settings of MLLM Pre-Training

C. Statistics of Multimodal Datasets

  • The datasets are released as image URLs rather than image files, and only half of OBELICS interleaved documents were fully downloadable as of June 2024.A whole OBELICS document is discarded if any image in it is invalid; filtered subsets inherit the original dataset licenses.Source: C. Statistics of Multimodal Datasets · p. 16, Table 8 · p. 16

D. Statistics of the Curated Interleaved Document Dataset

UniFilter’s selected interleaved documents contain more images and text tokens per document than the No-Filter and DFN-Filter selections. Unlike DFN-CLIP-Filter, it supports document-level filtering with a selectable retention fraction.

E. Visual SFT Data Composition

The visual SFT mixture combines multiple visual instruction sources with ShareGPT text instructions. The appendix also describes synthetic-generation prompt placeholders and reports that adding <|endofchunk|> to few-shot demonstrations is significant for multimodal in-context learning.

E. Visual SFT Data Composition

  • The joint SFT dataset contains 575k multimodal instructions, drawing visual instructions from eight named datasets and text instructions from ShareGPT.The five VQA evaluation datasets are excluded from the joint SFT data.Source: E. Visual SFT Data Composition · p. 16, Table 10 · p. 17

F. Full Synthetic Data Generation Prompts

G. Ablations on In-Context Learning Prompt Templates

  • The prompt-template ablation finds <|endofchunk|> significant for multimodal in-context learning; adding it to each demonstration makes the examples resemble interleaved documents.The authors suggest this may help trigger parametric memory toward knowledge pre-trained from interleaved documents.Source: G. Ablations on In-Context Learning… · p. 17

H. Ablation Study on Filtering Fraction for Caption Data

The ablations favor retaining 30% of caption data over 15%, find that system prompts promote GQA performance, and contrast positive synthetic documents with hard negatives on knowledge content and image-text alignment.

H. Ablation Study on Filtering Fraction for Caption Data

  • Retaining 30% rather than 15% of the original pool is the better choice for both filtering methods, balancing average quality with data diversity.The authors note that retraining on less data generally hurts dataset diversity and distribution.Source: H. Ablation Study on Filtering… · p. 17, Table 13 · p. 19

I. Effects of Introducing System Prompts in Multimodal In-Context Learning

  • Introducing a system prompt in the in-context learning template promotes the induced MLLM’s GQA performance.The authors interpret this as evidence that multimodal pre-training trains the base MLLM to follow instructions.Source: Table 14 · p. 19, §I · p. 19

J. Examples of Claude-3 Generated Contrastive Interleaved Documents

Related papers

Loading 2510.15162v1…