PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Vision Language Model Training via High Quality Data Curation

Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, Jiao Ran

arXiv:2501.05952v3cs.CVcs.CL

TL;DR

Compact VLMs require stronger visual understanding and instruction following, but the effective use of available data and training budgets remains unresolved. SAIL-VL addresses this with scalable high-quality caption construction, large-scale pretraining, and curriculum SFT, achieving state-of-the-art performance across 18 benchmarks. The authors also identify limits in the explored model scale and data-magnitude range, as well as possible hallucinated, biased, or harmful outputs.

  • Problem

    Compact VLMs have limited visual understanding and instruction-following performance, while optimal use of public resources and high-quality SFT data remains unresolved.

  • Method

    SAIL-VL combines the SAIL-Caption construction pipeline, pretraining up to 655B tokens, and curriculum SFT with progressively higher-complexity data.

  • Results

    SAIL-VL achieves top-ranked or state-of-the-art performance across 18 open-source VLM benchmarks, with performance scaling logarithmically with pretraining data size.

  • Takeaways & Limitations

    The results support scaling high-quality data quantity and complexity as effective strategies for improving compact VLM visual comprehension and instruction following.

  • Takeaways & Limitations

    The scaling-law study covers a specific data-magnitude range and has not tested larger models; further improvement under optimized settings remains uncertain.

Abstract

from arXiv · show

In this paper, we introduce SAIL-VL (ScAlable Vision Language Model TraIning via High QuaLity Data Curation), an open-source vision language model (VLM) series achieving state-of-the-art (SOTA) performance in 2B and 8B parameters. The following three key improvements contribute to SAIL-VL's leading performance: (1) Scalable high-quality visual understanding data construction: We implement a data construction pipeline to enable hundred-million-scale high-quality recaption data annotation. The resulted dataset SAIL-Caption is validated to be of the highest data quality compared with opensource datasets. (2) Scalable Pretraining with High-Quality Visual Understanding Data: We scale SAIL-VL's pretraining budget up to 655B tokens and show that even a 2B VLM benefits from scaled up training data sizes, exhibiting logarithmic data size scaling laws in benchmark performance. (3) Scalable SFT via data quantity and complexity scaling: We curate a high-quality SFT dataset collection with leading data quantity scaling effectiveness and demonstrate that training with progressively higher-complexity data surpasses baseline one-stage training by a large margin. SAIL-VL series models achieve the highest average score in 18 widely used VLM benchmarks in our evaluation, with the 2B model takes the top position over VLMs of comparable sizes on OpenCompass 2024 (https://rank.opencompass.org.cn/leaderboard-multimodal), demonstrating robust visual comprehension abilities. SAIL-VL series models are released at HuggingFace (https://huggingface.co/BytedanceDouyinContent).

1 Introduction

SAIL-VL addresses limited visual understanding and instruction-following performance in compact VLMs through scalable, high-quality data construction, large-scale pretraining, and curriculum SFT. The resulting models achieve top-ranked performance across 18 open-source VLM benchmarks.

  • Motivation: Compact VLMs face limited fundamental visual understanding and unsatisfactory instruction-following performance, while optimal use of public resources remains unresolved.Prior pretraining often uses limited, low-quality caption data, constraining visual understanding before instruction tuning.
  • Contributions: SAIL-VL introduces a scalable pipeline for constructing SAIL-Caption, a large, high-quality visual understanding dataset.The dataset is described as having large quantity and the highest quality compared with open-source datasets.
  • Contributions: 655B tokens of pretraining data expose logarithmic model-performance scaling laws with respect to training data size.The paper presents these as the first proposed and discussed data-size scaling laws for VLM pretraining.
  • Contributions: SAIL-VL uses high-quality SFT curation and curriculum training to improve instruction following through progressively higher-complexity data.The curriculum strategy is presented alongside methodologies for high-quality SFT data curation.
  • Results: SAIL-VL models achieve top-ranked performance in evaluation across 18 open-source VLM benchmarks.The models are introduced in 2B and 8B parameter sizes with state-of-the-art performance claims.

2 Model Training Pipeline

SAIL-VL combines staged pretraining with curriculum SFT, progressively increasing model capacity and data complexity to develop visual understanding and instruction following. Its pipeline uses large-scale caption and OCR data before staged instruction-tuning datasets.

  • Training overview: SAIL-VL uses five total training stages spanning visual-understanding pretraining and instruction-following adaptation.The pipeline starts from Intern-ViT and Qwen-2.5 model series.
  • Pretraining: Pretrain-Alignment trains a randomly initialized MLP projector with approximately 131B tokens of detail-caption and OCR data.The visual encoder is initially not described as unlocked during this warm-up stage.
  • Pretraining: Pretrain-Advance unlocks the visual encoder and trains with approximately 524B tokens, using an evenly distributed SAIL-Caption subset and repeated high-quality OCR data.For SAIL-VL-8B, the two pretraining budgets are 20B and 32B tokens for efficiency.
  • SFT: Curriculum SFT trains all parameters with progressively higher-complexity data across Knowledge, Instruction, and Preference stages.The Instruction stage uses a customized 12M-sample visual instruction-tuning dataset, while Preference uses a smaller complex-data collection.

3 Towards Scalable VLM Training

SAIL-VL combines scalable high-quality visual data construction with data-size scaling in pretraining and quantity- and complexity-scaled instruction tuning. Its experiments show steadily improving performance with larger pretraining data, strong SAIL-Instruct scaling, and curriculum SFT outperforming all-in-one training.

  • Scalable High-Quality Visual Understanding Data Construction: SAIL-Caption is constructed through a scalable pipeline that collects diverse images, curates GPT4-O reference captions, trains SAIL-Captioner, and performs asynchronous large-scale annotation.The pipeline supports multi-task, multi-node, and multiprocessing annotation with flexible resource allocation.
  • Scalable High-Quality Visual Understanding Data Construction: 300M image samples comprise SAIL-Caption, whose captions show greater visual richness, linguistic diversity, and human-rated quality than open-source datasets.Evaluation samples 10,000 cases from SAIL-Caption and other open-source caption datasets.
  • Pretraining Data-Size Scaling: SAIL-VL-2B is trained with 131B and 524B tokens across two pretraining stages, enabling evaluation of performance dynamics over increasing data sizes.The evaluation suite covers visual understanding tasks including detail caption generation and OCR detection.
  • Pretraining Data-Size Scaling: In pretrain-alignment, visual understanding performance improves steadily, while exponentially increasing training data produces an approximately linear performance trend.Figure 2 reports performance for understanding benchmarks, caption tasks, OCR tasks, and average benchmark scores.
  • Pretraining Data-Size Scaling: In pretrain-advance, understanding benchmark scores improve markedly and show a similar linear scaling curve as data size increases.The authors attribute this stage’s improvement to the larger-capacity vision encoder optimized for visual understanding.
  • SFT Data Quantity Scaling: SAIL-Instruct achieves the highest model performance at every tested data scale among the compared open-source SFT collections, with stable logarithmic scaling.Figure 4 averages performance across 18 benchmarks and compares SAIL-Instruct with LLaVA-OneVision and Infinity-MM datasets.
  • SFT Data Complexity Scaling: The three-stage curriculum increases task difficulty, data complexity, and image-text relevance monotonically, while later-stage data becomes smaller and harder to collect.Human evaluation supports progressively higher-quality but more challenging later-stage SFT data.
  • SFT Data Complexity Scaling: Curriculum SFT produces a near-linear performance curve across stages and outperforms the logarithmic curve of all-in-one training.The result supports using progressively higher-complexity data, including smaller high-complexity datasets, in later stages.

4 Experiments

SAIL-VL is evaluated against comparable open-source VLMs across broad visual comprehension benchmarks. The 2B model substantially outperforms prior comparable systems, while the 8B model remains leading but has a smaller advantage.

  • Evaluation Setup: SAIL-VL is compared with previous SOTA VLM baselines of comparable sizes, including Qwen2-VL, InternVL2.5-MPO, and DeepSeekVL-2.The evaluation uses a customized version of VLMEvalKit.
  • Evaluation Setup: The evaluation covers General VQA, OCR VQA, Math&Knowledge, and Hallucination benchmarks.These categories include natural image and video VQA, OCR documents, reasoning, and world knowledge tasks.
  • Benchmark Results: SAIL-VL-2B scores 1.4 (2.06% ↑) higher average performance than InternVL2.5-MPO-2B.It achieves new SOTA performance in 3 of 4 evaluated subfields, except General VQA.
  • Benchmark Results: SAIL-VL-8B achieves leading visual comprehension performance over Qwen2-VL, DeepSeekVL-2, and InternVL2.5-MPO-8B.InternVL2.5-MPO-8B uses an additional reinforcement learning stage in training.
  • Limitations: The 8B performance advantage over SOTA baselines is smaller, possibly because relatively small data sizes were used for training.The authors describe these results as an early attempt at larger VLM training.

5 Analysis

The analysis shows that data quality strongly affects visual understanding and SFT outcomes. It also supports curriculum SFT: moderately complex instruction data helps, whereas excessively complex preference data may hinder learning.

  • Pretraining Data Quality: The data-quality experiment uses lightweight 16B-token pretraining with fixed data-type distributions and varied data composition.The comparison modifies the quality of the training data while holding the distribution across data types fixed.
  • Pretraining Data Quality: Models trained with SAIL-Caption achieve significantly higher visual understanding performance than models trained with other open-source caption datasets.This agrees with the reported data-quality evaluation results.
  • Pretraining Data Quality: Repeated high-quality OCR data outperforms diverse but relatively low-quality data in the frozen-LLM pretraining setting.The authors attribute this result to reduced overfitting under frozen-LLM pretraining.
  • SFT Data Quality: SFT-Instruction data produces a significant performance advantage over the other evaluated SFT-stage collections.The result validates the effectiveness of the proposed data-curation methods and aligns with its higher task difficulty, data complexity, and image-text relevance.
  • Curriculum SFT: SFT-Preference data fails to further improve performance despite improved data quality.The authors attribute this to excessively high data complexity hindering effective model learning.

6 Related Works

Related work establishes visual understanding data and visual instruction tuning as central components of VLM development. Prior studies improve data quality, scale, and composition, but leave SFT collection distribution methodology insufficiently established.

  • Visual Understanding Data: Visual understanding data pairs visual content with language descriptions and supports applications across vision-language models.The passage situates such data in representation learning, generative modeling, and recent VLM research.
  • Visual Understanding Data: Prior VLMs explore expanding high-quality visual understanding data after earlier systems used relatively low-quality caption datasets.This line of work includes MiniCPM-V, InternVL, and QwenVL series models.
  • Visual Instruction Tuning: Existing studies adjust SFT data distributions across domains and formats, but widely acknowledged methods for determining SFT collection distributions remain lacking.The related work includes multi-stage data-efficiency exploration alongside these distribution adjustments.
  • Visual Instruction Tuning: LLaVA introduced visual instruction tuning and provided a baseline for VLM SFT data curation.Later work refined instruction-tuning datasets, while BLIP3 added image-text interleaved data.

7 Conclusions

SAIL-VL combines scalable high-quality caption construction, large-scale pretraining, and curated phased SFT for compact VLMs. Its models show strong performance across benchmarks, while the training analyses report logarithmic-to-near-linear scaling improvements.

  • Conclusions: SAIL-VL uses a scalable caption construction pipeline and SAIL-Caption, described as the highest-quality open-source alternative.The dataset supports pretraining at up to 655B tokens.
  • Conclusions: SAIL-VL visual comprehension performance improves logarithmically as training data size increases.The paper reports that even compact VLMs benefit from larger-scale training data.
  • Conclusions: The curated SFT-Instruction collection has better performance scaling curves than open-source alternatives.The phased SFT strategy further changes the scaling curves from logarithmic to near-linear.
  • Conclusions: SAIL-VL models outperform existing comparable-size VLMs consistently in overall performance or domain-specific abilities across 18 open-source benchmarks.The authors describe this as promising for real-world applications.

8 Limitations

The authors identify two limitations: limited evidence from larger models and uncertainty about whether performance can improve beyond the studied data quantity. They also note potential hallucinated, biased, or harmful outputs.

  • Larger-model experiments could provide additional insights, but SAIL-VL’s findings are currently based on comparable-size models.
  • The scaling-law study covers a specific data magnitude, leaving further improvement under optimized settings uncertain despite observed saturation.
  • SAIL-VL may generate hallucinated, biased, or harmful information under certain circumstances.

D.1 Caption Data Quality Assessment

The caption-quality assessment combines GPT and human evaluation across SAIL-Caption variants and open-source alternatives. SAIL-Caption achieves higher measured quality and wins fine-grained comparisons against three baselines.

  • GPT evaluation scores candidate captions using the precision and recall of visual elements against ground-truth captions.The evaluation uses Azure GPT4O for efficiency.
  • Human experts evaluate 100 samples per dataset by comparing two candidate captions against the original image.The protocol asks annotators to judge candidate caption quality based on the original image.
  • A 10% inspection of annotated samples finds 95%+ accuracy, supporting the reliability of the human evaluation results.
  • SAIL-Caption variants achieve higher GPT and human evaluation scores than previous baseline datasets.The evaluation covers SAIL-Caption-DataComp, SAIL-Caption-SA1B, and SAIL-Caption-KALE.
  • 87%, 91%, and 79% are SAIL-Captioner’s win+tie rates against SA1B-QwenVL-Caption, DataComp-LLaVA-Caption, and BLIP3-KALE, respectively.GSB evaluation captures finer-grained caption-quality differences than a single rating.
  • Qualitative showcases report more detailed visual elements in SAIL-Caption than SA1B-QwenVL-Caption, DataComp-LLaVA-Caption, and BLIP3-KALE.The comparisons include both English and Chinese captions.
  • The released SAIL-Caption subset is intended to promote open-source VLM research.

E Visual Understanding Benchmark

The visual-understanding evaluation measures detailed captioning and OCR performance using curated benchmarks, GPT-based caption scoring, and ANLS for OCR. Evaluation uses left-out data and 500 randomly selected cases per benchmark.

  • The benchmark suite evaluates detailed captioning and OCR in both English and Chinese, matching the pretraining optimization objectives.The selected benchmark information is summarized in Table 9.
  • The suite includes DetailCaps-4870 plus OCR datasets spanning document pages, charts, single-word images, and compositional natural-image scenes.The OCR datasets include IDL-WDS, DocStruct, SynthText, SynthDog-EN, and SynthDog-ZH.
  • Captioning is scored by GPT on visual-element precision and recall, while OCR uses ANLS scores from 0 to 1.Higher ANLS indicates better prediction quality.
  • 500 randomly selected cases from each benchmark are used to evaluate pretrained checkpoints efficiently.All benchmark data comes from left-out subsets to avoid training–evaluation leakage.

G Generalizing Visual Understanding Abilities to Instruction Following Tasks

The study examines whether stronger pretrained visual understanding transfers to instruction following after SFT. It reports strong correlations across two SFT data settings and documents the evaluation setup and illustrative examples.

  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: The examples compare SAIL-Caption with SA1B-QwenVL-Caption and DataComp-LLaVA-Caption using multilingual captions and varied visual content.The showcased materials include urban streets, vintage cars, and promotional graphics.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: One comparison centers on a close-up reading “PLANNING MEETING FALL FESTI-VAL,” while another describes a promotional graphic titled “Fall Festival.”The passages provide both detailed and concise descriptions of the visual content.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: A wallet example pairs a concise caption with a more detailed description containing texture, lettering, a chain, and background information.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: The comparison materials include a bicycle-clock decorative object described with detailed structural, textual, and environmental attributes.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: The appendix evaluates SFT data using expert ratings of challenging, complexity, and relevance dimensions.The evaluation instructions frame the task around a Q&A conversation and its corresponding image.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: The appendix connects SFT data selection to visual-question-answering content rather than restricting evaluation to isolated image captions.The instructions explicitly require rating the combined Q&A conversation and image.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: Pretrained visual-understanding performance correlates strongly with post-SFT instruction-following performance across training strategies.The study plots this relationship for checkpoints trained with exponentially larger data sizes.
  • Generalizing Visual Understanding Abilities to Instruction Following Tasks: ρ = 0.97 and R2 = 0.94 for SFT-Instruction data, while LLaVA-Next SFT data reaches ρ = 0.99 and R2 = 0.98.These results are reported as evidence of ability generalization across training stages and tasks.

H.1 Experiment settings

SAIL-VL training uses streaming data infrastructure and sequence packing to support large-scale distributed training, with substantial NPU resources allocated across model sizes. Experiments include comparable open-source VLM baselines and benchmark evaluations, alongside a reported correlation between pretrained understanding and post-SFT benchmark performance.

  • Training data is persistently stored on HDFS and fetched in a stream during training for large-scale distributed data access.
  • Training uses PyTorch 2.1.0, CUDA 12.1, DeepSpeed 0.14.5, and Flash-Attention implemented for Huawei 910B NPU hardware.
  • A stream accumulator packs micro-batch sequences into long sequences, accelerating SAIL-VL training by approximately 40%.
  • The SAIL-VL-2B model uses 101,045 NPU hours across pretraining and SFT, while SAIL-VL-8B training consumes 19,575 NPU hours.
  • Evaluation reports complete results for SAIL-VL-2B and SAIL-VL-8B against comparable open-source VLMs, including InternVL2, InternVL2.5, and Aquila series models.
  • Figure 11 examines the correlation between pretrained-checkpoint understanding performance and post-SFT average scores on open-source benchmarks.
Loading 2501.05952v3…