Scalable Vision Language Model Training via High Quality Data Curation

• Compact VLMs face limited visual understanding and instruction following, while optimal pretraining data and budgets and SFT dataset distributions and stage allocation remain unresolved.The introduction attributes weak visual understanding to insufficient high-quality data or limited pretraining, and identifies a lack of reliable conclusions about pretraining budgets and data quality.Source: §1 · pp. 1–2
• 2B and 8B SAIL-VL models achieve new state-of-the-art performance across 18 widely used VLM benchmarks.The paper reports top-ranked performance in its evaluation of 18 open-source VLM benchmarks.Source: §1 · p. 2
• A scalable data-construction pipeline produces SAIL-Caption, which the paper describes as large-scale and highest quality compared with open-source datasets.The paper also describes its customized SFT collection as markedly higher quality than open-source datasets and uses it in a three-stage curriculum SFT strategy.Source: §1 · p. 2
• SAIL-VL scales pretraining to 655B tokens and reports logarithmic model-performance scaling laws with training data size.The paper presents these as the first proposed and discussed data-size scaling laws for VLM pretraining.Source: §1 · p. 2

2 Model Training Pipeline

• The five-stage pipeline combines visual-understanding pretraining with instruction-following adaptation, starting from Intern-ViT (Chen et al., 2023b) and Qwen-2.5 (Team, 2024b).The training strategy is summarized in Fig. 1.Source: §2 · p. 2

2.1 Pretrain

• Pretraining uses approximately 131B tokens for projector alignment and 524B tokens after unlocking the visual encoder, with an evenly distributed SAIL-Caption subset and repeatedly sampled high-quality OCR data.For SAIL-VL-8B, the respective stage budgets are 20B and 32B tokens.Source: §2.1 · pp. 2–3

2.2 SFT

• SFT progressively increases data complexity, moving from basic instruction following and world knowledge to a 12M-sample visual-instruction dataset and then a small set of complex instruction data.The stages use Infinity-MM Stage2, a customized visual instruction-tuning dataset, and LLaVA SFT, Molmo Caption, and Infinity-MM Stage4 data, respectively.Source: §2.2 · p. 3

3.1 Scalable High-Quality Visual Understanding Data Construction

• The four-step pipeline collects diverse public images, builds balanced GPT-annotated reference data, trains a captioner, and scales annotation through asynchronous multi-task, multi-node processing.Alt-texts can supplement reference annotations and caption-generation prompts, enabling captioning and recaptioning.Source: §3.1 · p. 3

3.2.1 Improving VLM Visual Understanding Performance via Data Size Scaling

• Across the 131B-token pretrain-alignment and 524B-token pretrain-advance stages, performance grows roughly linearly as data size scales exponentially; advance-stage understanding scores improve markedly.The authors attribute the advance-stage gains to the vision encoder’s capacity for visual understanding.Source: Figure 2 · p. 4, Figure 3 · p. 4, §3.2.1 · p. 4

3.2.2 Generalizing Visual Understanding Abilities to Instruction Following Tasks

• SFT models show near-linear scaling on an exponential data-size axis across both SAIL-Instruction and LLaVA-Next data, despite differing data compositions and final benchmark scores.The reported curves are for open-source VLM benchmarks.Source: Figure 3 · p. 4, §3.2.2 · p. 5

3.1 Scalable High-Quality Visual Understanding Data Construction

• 300M SAIL-Caption image samples exhibit leading visual-element richness and linguistic diversity, and receive higher human quality scores than existing open-source caption datasets.The quality comparison sampled 10,000 cases from SAIL-Caption and other open-source caption datasets.Source: §3.1 · p. 4

3.3.1 High-Quality SFT Data Curation for Data Quantity Scaling

• SAIL-Instruction achieves the highest performance at every data scale, with stable logarithmic performance scaling across training.The consistent ranking across datasets supports the proposed quick quality evaluation strategy.Source: Figure 4 · p. 5, §3.3.1 · p. 6

3.3.2 Multi-stage Instruction Tuning for Data Complexity Scaling

• Curriculum SFT yields near-linear performance scaling and outperforms logarithmic all-in-one training; later-stage small, high-complexity data produces more promising scaling curves.Human evaluation found task difficulty, data complexity, and image-text relevance increase across the three stages.Source: Figure 5 · p. 5, §3.3.2 · p. 6

3.3.1 High-Quality SFT Data Curation for Data Quantity Scaling

• SAIL-Instruction curation evaluates 2M-sample subsets by model performance, adjusts proportions of categorized data components, and incorporates new datasets that improve or maintain performance.The categorized components include closed-form and open-ended VQA, document VQA, math and reasoning, and pure text QA.Source: §3.3.1 · pp. 5–6

4.2 Benchmark Results

• 1.4 (2.06% ↑) higher average performance than InternVL2.5-MPO-2B, SAIL-VL-2B achieves new SOTA on 3 of 4 subfields, except General VQA.The paper attributes the General VQA exception to instability on benchmarks requiring long text generation, such as MMVet.Source: Table 4 · p. 7, §4.2 · p. 7

4.1 Experiment Setup

• The comparison includes prior SOTA VLM baselines of comparable sizes, including Qwen2-VL, InternVL2.5-MPO, and DeepSeekVL-2.Additional baseline comparisons are reported in Appendix H.2.Source: §4.1 · p. 7
• Evaluations use widely used benchmarks spanning General VQA, OCR VQA, Math&Knowledge, and Hallucination, with a customized VLMEvalKit.The evaluation covers VQA on natural images and videos, OCR-related documents, and tasks involving complicated reasoning and world knowledge.Source: §4.1 · p. 7

4.2 Benchmark Results

• SAIL-VL-8B leads Qwen2-VL, DeepSeekVL-2, and InternVL2.5-MPO-8B in visual comprehension, although the latter uses an additional reinforcement-learning stage.The authors note that SAIL-VL-8B's performance advantage over SOTA baselines shrank, possibly because training used relatively small data sizes; they describe this as an early attempt and say more competitive large models will follow.Source: §4.2 · pp. 7–8, Table 4 · p. 7

5.1 Pretrain Data Quality Determines Pretrained Model Performance

• SAIL-Caption pretraining yields significantly higher performance than other open-source caption datasets, while repeated high-quality OCR data outperforms diverse, lower-quality data.The comparison used lightweight 16B-token pretraining with a fixed data distribution; the authors attribute the OCR result to frozen-LLM pretraining mitigating overfitting from repeated data.Source: §5.1 · p. 8

5.2 SFT Data Quality Analysis

• SFT-Instruction data produces a significant performance advantage, consistent with its higher task difficulty, data complexity, and image-text relevance.Table 6 reports performance for models trained with SFT data from each stage.Source: Table 6 · p. 8, §5.2 · p. 8
• Despite improved data quality, SFT-Preference does not further improve performance; its excessive complexity may hinder learning, supporting curriculum SFT.The authors present this observation as further validation of the proposed curriculum VLM SFT strategy.Source: Table 6 · p. 8, §5.2 · p. 8

6.1 Visual Understanding Data

• Visual-understanding datasets range from low-quality alt-text pairs to small-scale API captions and recaptioner-generated detail captions at scale; resulting datasets are widely used in recent VLM research.The cited examples include LAION, TaiSu, Coyo, and DataComp; ShareGPT4V and ALLaVA; and recaptioning efforts such as CapsFusion and BLIP3-KALE.Source: §6.1 · pp. 8–9

6.2 Vision Language Model Pretrain

• Pretraining research progressed from lower-quality captions to scaling high-quality visual data, while this work studies the less-explored effects of SFT data quality and size.BLIP2 and LLaVA used relatively low-quality captions; MiniCPM-V, InternVL, and QwenVL explored larger high-quality datasets (Yao et al., 2024; Chen et al., 2024d; Team, 2024a; Chen et al., 2024c; Bai et al., 2023; Wang et al., 2024b).Source: §6.2 · p. 9

6.3 Visual Instruction Tuning

• Visual instruction tuning evolved from LLaVA’s baseline through dataset refinement and interleaved image-text and video-QA data; this work offers design guidelines for instruction datasets.The cited developments include LLaVA series refinements (Liu et al., 2023, 2024a; Li et al., 2024a), BLIP3’s interleaved data (Xue et al., 2024), and video-QA use in CogVLM, InternVL, and QwenVL (Hong et al., 2024; Chen et al., 2024d; Team, 2024a; Chen et al., 2024c; Wang et al., 2024b).Source: §6.3 · p. 9

7 Conclusions

• SAIL-Caption, curated with a scalable pipeline, supports pretraining up to 655B tokens, with visual comprehension improving logarithmically as training data grows.The authors report that even compact VLMs benefit from scaling training data.Source: §7 · p. 9
• SAIL-VL consistently outperforms comparable-size VLMs overall or in domain-specific abilities across 18 open-source benchmarks.The authors describe these results as promising for real-world applications.Source: §7 · p. 9
• Guidelines for high-quality SFT curation produced SFT-Instruction, whose performance scaling curves improve over open-source alternatives; phased SFT shifts the curves from logarithmic to near-linear.The passage spells the dataset name “SFT-Instrcution.”Source: §7 · p. 9

8 Limitations

• SAIL-VL’s scaling-law results remain limited: larger models and data quantities beyond the explored magnitude could clarify robustness and whether optimized training can improve performance.Performance appears to saturate at the explored data quantity, but further improvement under optimized settings is uncertain.Source: §8 · p. 9
• SAIL-VL may generate hallucinated, biased, or harmful information in certain circumstances, which future work will further discuss and mitigate.Source: §8 · p. 9

B SAIL-VL Model Card

• The paper provides a simplified model card for SAIL-VL-2B and SAIL-VL-8B.Source: B SAIL-VL Model Card · p. 14, Table 7 · p. 14

C SAIL-VL Showcases

• SAIL-VL-8B demonstrates language-interaction capabilities in both English and Chinese on vision-based queries drawn from common real-world interactions.The showcased images were selected from the internet.Source: C SAIL-VL Showcases · p. 14
• SAIL-VL-8B shows visual comprehension across varied input elements and recognizes famous landmarks, buildings, and artworks.The paper describes this recognition as demonstrating a broad reservoir of world knowledge.Source: C SAIL-VL Showcases · p. 14
• In meme examples, SAIL-VL-8B identifies visual elements and explains the contrast that makes the meme humorous.Source: C SAIL-VL Showcases · p. 14

D.1 Caption Data Quality Assessment

• SAIL-Caption-DataComp, SAIL-Caption-SA1B, and SAIL-Caption-KALE score higher than previous baselines in both GPT and human evaluation.The authors attribute the results to SAIL-Captioner and SAIL-Caption’s data quality.Source: Table 8 · p. 14, §D.1 · p. 15
• GPT evaluation compares candidate captions with Azure GPT4O-generated ground truths, while human experts score 100 samples per dataset and provide GSB judgments.Inspection of 10% of annotated samples showed 95%+ accuracy for the human evaluation.Source: §D.1 · pp. 14–15
• SAIL-Captioner achieves GSB win+tie rates of 87%, 91%, and 79% against SA1B-QwenVL-Caption, DataComp-LLaVA-Caption, and BLIP3-KALE, respectively.GSB captures fine-grained caption quality differences beyond a single rating.Source: §D.1 · p. 16
• SAIL-Captioner distills visual-understanding abilities from reference data annotated by powerful VLM APIs, enabling large-scale high-quality data generation with limited resources.Source: §D.1 · p. 16

D.2 SAIL-Caption Showcases

• Showcases indicate that SAIL-Caption captures more visual detail than alternative datasets in English and Chinese; a subset of the dataset will be released to support open-source VLM research.Source: §D.2 · p. 17

E Visual Understanding Benchmark

• Visual-understanding benchmarks cover detailed captioning and OCR in English and Chinese, with DetailCaps-4870 evaluated by GPT and OCR predictions scored using ANLS.The selected OCR sets include document pages, tables and charts, single-word images, and natural-image backgrounds with foreground text.Source: E Visual Understanding Benchmark · p. 17
• Benchmark data comes from left-out subsets to avoid leakage, and 500 randomly selected cases from each benchmark are used to evaluate pretrained checkpoints.Source: E Visual Understanding Benchmark · p. 17

F SFT Data Quality Evaluation

• Human experts annotate the three-stage SFT data for challenge, complexity, and relevance.Source: F SFT Data Quality Evaluation · p. 17

G Generalizing Visual Understanding Abilities to Instruction Following Tasks

• The experiments train pretrain-advance SAIL-VL checkpoints at exponentially larger data sizes, then use either LLaVA-Next SFT data or SFT-Instruction data.Source: G Generalizing Visual Understanding… · pp. 17, 19
• The supplied caption examples describe an urban street and a vintage-car outdoor gathering, including scene details such as parking signs, a crossing, and a blue tent.Source: G Generalizing Visual Understanding… · p. 18
• Figure 9 showcases SAIL-Caption against SA1B-QwenVL-Caption, DataComp-LLaVA-Caption, and BLIP3-KALE using images curated from those datasets.Source: Figure 9 · p. 18
• Figure 10 provides detailed instructions for human experts to judge SFT data quality.Source: Figure 10 · p. 19
• Stronger pretrained visual understanding corresponds to better post-SFT visual instruction following across training strategies, even with different visual instruction tuning datasets.The reported correlations support the authors’ stated rationale for pretraining VLMs to improve robust visual understanding.Source: G Generalizing Visual Understanding… · p. 19
• ρ = 0.97 and R2 = 0.94 with SFT-Instruction data, and ρ = 0.99 and R2 = 0.98 with LLaVA-Next SFT data.The authors interpret these results as evidence that visual-understanding abilities generalize across training stages and tasks.Source: G Generalizing Visual Understanding… · p. 19

H.1 Experiment settings

• SAIL-VL’s pretraining and SFT data is stored on HDFS and fetched as a stream during training.The passage says this enables training with large-scale distributed data storage.Source: §H.1 · p. 20
• Training used PyTorch 2.1.0 with CUDA 12.1, DeepSpeed 0.14.5, and FlashAttention implemented for Huawei 910B NPU.The passage cites PyTorch (Paszke et al., 2019), CUDA, DeepSpeed, FlashAttention (Dao et al., 2022; Dao, 2023), and Huawei NPU.Source: §H.1 · p. 20
• A stream accumulator packs micro-batch sequences into a long sequence, speeding SAIL-VL training by approximately 40%.Source: §H.1 · p. 20
• SAIL-VL-2B used 101,045 NPU hours (90,053 pretraining; 10,992 SFT), while SAIL-VL-8B used 19,575 (6,672 pretraining; 12,903 SFT).The 8B model used 26M pretraining samples and the same SFT data collections as the 2B model.Source: §H.1 · p. 20

H.2 Experiment Results.

• Supplementary evaluation baselines include InternVL2, InterVL2.5, and Aquila series models.The passage attributes these to Team (2024a), Chen et al. (2024c), and Gu et al. (2024), respectively.Source: §H.2 · p. 20
• Figure 11 relates pretrained-checkpoint understanding performance to post-SFT performance on open-source benchmarks.“OS BMK Score” denotes the average score on open-source benchmarks in the paper’s evaluations.Source: Figure 11 · p. 21
• Tables 10 and 11 provide complete evaluations of SAIL-VL-2B and SAIL-VL-8B against open-source VLMs of comparable sizes.The supplied passages do not include table values, so they do not establish which models perform best.Source: Table 10 · p. 21, Table 11 · p. 22