PaperlayerView exact revision on arXiv

Source-linked AI summary

SAIL-VL2 Technical Report

Weijie Yin, Yongjie Ye, Fangxun Shu, Yue Liao, Zijian Kang, Hongyuan Dong, Haiyang Yu, Dingkang Yang, Jiacong Wang, Han Wang, Wenzhuo Liu, Xiao Liang, Shuicheng Yan, Chao Feng

arXiv:2509.14033v2cs.CV

TL;DR

Efficient vision-language models face the compute, training, and deployment costs associated with scaling model size and data. SAIL-VL2 combines data curation, progressive training, and architectural changes to address multimodal understanding and reasoning. It reports state-of-the-art results across 106 benchmarks at the 2B and 8B scales and leading results on reasoning benchmarks and OpenCompass.

  • Problem

    Scaling model parameters and training data can bring performance gains but also substantial computational, training, and deployment costs.

    Source: §1 · p. 2

  • Method

    SAIL-VL2 combines scored and filtered data, progressive vision-to-language training, thinking-fusion SFT–RL, and sparse MoE architectures.

    Source: §2.1 · p. 4, §3.1.1 · p. 6, §4.2.5 · p. 14

  • Results

    Across 106 benchmarks, SAIL-VL2 reports state-of-the-art results at the 2B and 8B scales, leading results on MMMU and MathVista, and first place on OpenCompass among officially released models under 3B.

    Source: §7 · p. 20

  • Takeaways & Limitations

    The authors present SAIL-VL2 as an efficient, scalable foundation for the open-source multimodal community.

    Source: §1 · p. 2, §7 · p. 20

Abstract

from arXiv · show

We introduce SAIL-VL2, an open-suite vision-language foundation model (LVM) for comprehensive multimodal understanding and reasoning. As the successor to SAIL-VL, SAIL-VL2 achieves state-of-the-art performance at the 2B and 8B parameter scales across diverse image and video benchmarks, demonstrating strong capabilities from fine-grained perception to complex reasoning. Its effectiveness is driven by three core innovations. First, a large-scale data curation pipeline with scoring and filtering strategies enhances both quality and distribution across captioning, OCR, QA, and video data, improving training efficiency. Second, a progressive training framework begins with a powerful pre-trained vision encoder (SAIL-ViT), advances through multimodal pre-training, and culminates in a thinking-fusion SFT-RL hybrid paradigm that systematically strengthens model capabilities. Third, architectural advances extend beyond dense LLMs to efficient sparse Mixture-of-Experts (MoE) designs. With these contributions, SAIL-VL2 demonstrates competitive performance across 106 datasets and achieves state-of-the-art results on challenging reasoning benchmarks such as MMMU and MathVista. Furthermore, on the OpenCompass leaderboard, SAIL-VL2-2B ranks first among officially released open-source models under the 4B parameter scale, while serving as an efficient and extensible foundation for the open-source multimodal community.

1 Introduction

SAIL-VL2 targets efficient large vision-language models amid the compute, training, and deployment costs of scaling model size and data. It combines data, training, and architecture upgrades with strong results across multimodal understanding, reasoning, and fine-grained perception.

1 Introduction

  • Scoring and filtering improve data quality and distribution, while progressive training, thinking-fusion SFT–RL, and sparse MoE designs form the core approach.The training progression begins with SAIL-ViT, continues through multimodal pre-training, and culminates in the SFT–RL hybrid paradigm.Source: §1 · p. 2
  • Across 106 datasets, SAIL-VL2 achieves state-of-the-art results under the 2B-parameter scale.The report also notes competitive base-model performance on MMMU and MathVista without specialized prompting.Source: §1 · p. 2
  • SAIL-VL2-Thinking uses Chain-of-Thought and reinforcement learning strategies to improve complex-reasoning performance, often matching or surpassing much larger models.Source: §1 · p. 2
  • SAIL-VL2 reports high-fidelity OCR, high-resolution document-layout analysis, and chart interpretation, with visual grounding beyond models of similar scale.Source: §1 · p. 2
  • The authors present SAIL-VL2 as an efficient, extensible foundation intended to support the open-source multimodal ecosystem.Source: §1 · p. 2

2 Model Architecture

SAIL-VL2 combines a progressively trained vision encoder, a lightweight adapter, and a language model for multimodal understanding. Its vision-encoder training proceeds through staged alignment, while the model family supports multiple resolutions and dense or sparse language-model backbones.

2 Model Architecture

  • SAIL-VL2 comprises a vision encoder, an adapter, and an LLM, initialized from SAIL-ViT and Qwen3-series models.Source: §2 · p. 3

2.1 SAIL-ViT

  • SAIL-ViT is progressively optimized with multi-granularity knowledge to align visual information with the linguistic space.Source: §2.1 · p. 4

2.1.1 Progressive Training Strategy

  • Three instruction-tuning stages progress from adapter-only warm-up, to vision-encoder and adapter alignment with a fixed LLM, to joint training of all parameters.The stages introduce increasingly broad data and alignment objectives.Source: §2.1.1 · p. 4

2.1.2 SAIL-ViT Family

  • SAIL-ViT includes fixed-resolution and AnyRes variants; pre-training both variants produced visual encoders with strong visual representations.The AnyRes variant interpolates positional embeddings to match input resolution.Source: §2.1.2 · p. 4
  • The training-data pipeline curates and filters open-source datasets, generates synthetic data, and organizes both for different training stages.Source: Figure 3 · p. 5

2.2 Multi-modal Mixture-of-Experts

  • SAIL-VL2 uses Qwen3-based sparse MoE structures, with load-balancing and data-aware calibration strategies to support stable expert activation and preserve specialization.Calibration on language data maximizes expert activation entropy and improves entropy on multimodal sets.Source: §2.2 · p. 5

3 Pre-Training

The section covers pre-training data curation and organization alongside the training recipe.

3 Pre-Training

  • The section details pre-training data curation and organization, as well as the training recipe.Source: §3 · p. 5

3.1 Pre-Traning Data

SAIL-VL2 combines expanded caption, QA, instruction-tuning, and text data, with scoring and filtering to improve data quality and distribution. SAIL-Caption2 adds caption-quality judging and chart data, while caption-to-QA conversion expands synthetic VQA supervision.

3.1 Pre-Traning Data

  • The broader pre-training data combines upgraded captions, Caption2QA-generated pairs, pure text, and multimodal instruction-tuning VQA datasets.The text data is included to preserve the LLM’s language modeling and comprehension ability during multimodal pre-training.Source: §3.1 · p. 5

3.1.1 SAIL-Caption2

  • SAIL-Caption2 uses a caption judge to filter low-quality samples and adds chart-related captions to broaden data diversity.Chart training data includes code-generated chart descriptions and conventional datasets.Source: §3.1.1 · p. 6
  • 15–20% of sampled captions scored below 3; both judge models exceeded 90% accuracy and recall, and retained captions were estimated to be over 99% high quality.The Score Judge rates captions from 1–5, while the Yes-or-No Judge makes a binary quality decision.Source: §3.1.1 · p. 6
  • 250M general captions and 1.69M chart captions comprise SAIL-Caption2 after caption filtering and chart-data collection.The caption judge filters low-quality samples; chart captions combine automatically generated and conventional data.Source: §3.1.1 · pp. 6–7

3.1.2 Synthetic VQA Data

  • SAIL-Caption2 captions are converted into multiple question–answer pairs, scaling synthetic VQA data to support pre-training.Approximately 80M captions are sampled, and performance improves smoothly with training budget up to 180M samples despite possible linguistic bias.Source: §3.1.2 · p. 7

3.2 Pre-Training Recipe

SAIL-VL2 pre-training progresses from basic multimodal alignment to multi-task understanding and instruction following. The recipe also uses two-step data resampling, AdaLRS for basic pre-training, and expanded data scaling.

3.2.1 Training Pipeline

  • Pre-training first builds cross-modal alignment and understanding, then uses multi-task data to strengthen multimodal understanding and instruction following.Basic pre-training uses a pre-trained SAIL-ViT, a language-pretrained LLM, and a randomly initialized MLP adapter; multi-task pre-training unfreezes all parameters.Source: §3.2.1 · pp. 7–8

3.2.3 Adaptive Learning Rate Search

  • AdaLRS raises the initial learning rate from 2 × 10−4 to 6.75 × 10−4 and finishes with a loss advantage of over 0.06 versus the fixed-rate baseline.The reported comparison is for the basic pre-training stage.Source: §3.2.1 · p. 7

3.2.2 Data Resampling

  • Two-step resampling balances datasets during basic pre-training and rebalances n-gram patterns in large caption and synthetic VQA datasets during multi-task pre-training.The stated aims include increasing diversity, mitigating distributional bias, and improving data efficiency.Source: §3.2.2 · p. 8

3.2.3 Adaptive Learning Rate Search

  • AdaLRS uses backtracking line search to adjust the learning rate according to estimated loss-slope changes during basic multimodal pre-training.The procedure retains an increase when loss reduction accelerates, but rolls back parameters and optimizer state and decreases the rate when convergence slows.Source: §3.2.3 · p. 8

3.2.4 Scaling-law

  • At 360B tokens, performance improves monotonically across overall benchmarks, natural-scene VQA, and OCR VQA tasks.About half of the VQA data is synthesized; the authors report that increased scale and diversity enhance generalization and reasoning.Source: §3.2.4 · p. 8, Figure 4 · p. 9

4 Post-Training

SAIL-VL2 post-training combines foundational instruction tuning with a two-phase thinking-fusion SFT–RL pipeline to improve instruction following and reasoning.

4 Post-Training

  • Post-training is designed to strengthen instruction following and incentivize reasoning through a comprehensive pipeline.The pipeline begins with instruction-tuning SFT, then uses two-phase thinking-fusion training that progressively combines SFT with reinforcement learning.Source: §4 · p. 9
  • The first post-training stage uses SFT to inject general knowledge and improve fundamental instruction adherence.Source: §4 · p. 9
  • A two-phase thinking-fusion SFT–RL pipeline follows the initial SFT stage to strengthen reasoning ability.Source: §4 · p. 9

4.1 Post-Training Data

Post-training data curation covers video QA, visual instruction following, and multimodal chain-of-thought reasoning. The section describes selection and validation processes and reports stronger SFT performance from SAIL-Instruction2 at matched data budgets.

4.1.1 SAIL-Video

  • Scoring and filtering reduce 6.23M curated video-QA samples to a refined SAIL-Video corpus of 5.1M samples.Selection evaluates frame–instruction alignment, video-content richness, and QA difficulty.Source: §4.1.1 · p. 9

4.1.2 SAIL-Instruction2

  • SAIL-Instruction2 adds open-source datasets and long-answer or reasoning examples, retaining incremental data only after quality and performance-gain validation.Latent-class filtering creates fine-grained semantic buckets for uniform sampling and re-annotation; the final dataset contains 20M samples.Source: §4.1.2 · pp. 9–10
  • At the same data budget, SAIL-Instruction2 consistently outperforms prior instruction datasets in SFT comparisons.The comparison uses SAIL-VL2-2B after multi-task pre-training.Source: Figure 5 · p. 10, §4.1.2 · p. 10

4.1.3 Multimodal CoT Data

  • The multimodal reasoning dataset combines diverse public sources into complementary LongCoT SFT and Think-Fusion SFT subsets, followed by reinforcement learning across math, puzzle, science, OCR, and counting tasks.After refinement, it comprises 400K LongCoT SFT samples, 1M Think-Fusion SFT samples, 50K RL samples with verifiable rewards, and 100K RL samples with a mixed reward system.Source: §4.1.3 · p. 10

4.2 Post-Training Recipe

Post-training progresses from foundational instruction tuning through long-chain reasoning, verifiable-reward RL, and mixed reasoning/general-capability tuning. The recipe combines curated data, supervised objectives, model merging, and reward signals to develop multimodal understanding and reasoning.

4.2.1 Basic Supervised Fine-Tuning

  • Basic SFT progressively introduces more complex image data, then uses a balanced 1:1 video-image mixture to add video training while preserving image proficiency.The phases move from foundational instruction following through a 20M-sample visual instruction dataset and challenging instruction examples.Source: §4.2.1 · p. 11
  • Homogeneous model soups yield stable, notable gains after SFT, while merging heterogeneous models often degrades performance.The strategy therefore merges models with similar optimization trajectories, including comparable hyperparameters and data compositions.Source: §4.2.1 · p. 11

4.2.2 LongCoT Supervised Fine-tuning

  • LongCoT SFT trains on curated reasoning chains to teach step-by-step problem solving before the final answer.The pipeline filters and balances generated chains, yielding 400K LongCoT samples; training uses next-token prediction over the output sequence.Source: §4.2.2 · pp. 11–12

4.2.3 Reinforcement Learning with Verifiable Rewards

  • Verifiable-reward RL refines reasoning using answer-correctness and output-format rewards on challenging yet tractable problems.Difficulty filtering yields 70K STEM samples; both answer and format rewards are binary signals.Source: §4.2.3 · p. 12

4.2.4 Think-Fusion Supervised Fine-tuning

  • Think-Fusion SFT mixes general direct-answer examples with selected reasoning chains to retain broad capabilities while strengthening complex reasoning.The dataset contains 900K direct QA samples and 100K CoT samples; the loss covers answer tokens for direct pairs and the full sequence for CoT examples.Source: §4.2.4 · p. 13

4.2.5 Reinforcement Learning with a Mixed Reward System

  • Mixed-reward RL scores answers, reasoning paths, and format, using an LVM judge when ground truth is not deterministically verifiable.Training uses 150K curated samples, including hard reasoning cases and general-purpose examples; the authors report that the model can answer accurately without explicitly writing every reasoning step.Source: §4.2.5 · pp. 13–14

5 Training Infrastructure

The infrastructure targets efficient, robust training of large multimodal models through stream packing and hardware-adapted MoE training. The packing methods balance text and visual workloads, while the MoE stack addresses compute and communication overhead.

5 Training Infrastructure

  • The training infrastructure is designed to improve efficiency and robustness for large-scale multimodal models.The stream-packing and MoE infrastructure sections describe complementary approaches to that aim.Source: §5 · p. 14, §5.1 · p. 14, §5.2 · p. 15

5.1 Stream Packing

  • Stream packing concatenates variable-length samples to reduce padding, while visual packing balances visual-token counts across devices.Online micro-batch construction fills sequences to supported maximum length and periodically includes long samples; visual balancing also targets vision-encoder workload.Source: §5.1 · p. 14
  • Compared with the baseline, data packing nearly doubles SM utilization and accelerates training by 50%; visual packing adds a 48% efficiency gain.Packing ablations also report an average +0.7% improvement over baseline, with strong gains on open-ended QA benchmarks.Source: §5.1 · p. 14

5.2 MoE Infra

  • MoE infrastructure combines fused expert kernels with device-specific distributed strategies to reduce compute, memory, and communication overhead.The report states expert-kernel fusion achieves up to 3× training speedup; NPU training uses pipeline and expert parallelism, while NVIDIA devices use ZeRO-2 with CPU offloading.Source: §5.2 · p. 15

6 Experiment

The experiments evaluate SAIL-VL2 across a broad benchmark suite and examine its visual encoder, general multimodal understanding, and reasoning variants. Results report leading performance across comparable parameter scales, with separate findings for classification, feature alignment, and multimodal tasks.

6.1 Evaluation Setting

  • The evaluation spans 106 datasets; model configurations include fixed-resolution and AnyRes variants, alongside thinking versions assessed on OpenCompass reasoning benchmarks.The basic-model comparisons use a customized VLMEvalKit, while thinking-model comparisons primarily use official OpenCompass results.Source: §6.1 · pp. 15–16

6.2.1 Experimental Analysis on SAIL-ViT

  • SAIL-ViT-Large gained +1.5%, SAIL-ViT-Huge +2.11%, and scaled SAIL-ViT +2.73% over AIMv2; SAIL-ViT features also lie closer to LLM text spaces.The classification gains are averages across four test sets; the feature-space comparison uses multiple distance metrics.Source: §6.2.1 · pp. 16–17, Figure 6 · p. 17

6.2 Main Results

  • SAIL-VL2-2B/8B and SAIL-VL2-Thinking-8B achieve state-of-the-art average OpenCompass performance, surpassing publicly available open-source models at comparable scales.The basic models are reported in Tables 8 and 9, and thinking models in Table 10.Source: §6.2 · p. 16, Table 8 · p. 18, Table 9 · p. 19

6.2.2 Multimodal Understanding Tasks

  • SAIL-VL2-2B scored 86.77 on MMBench-v1.1, 72.29 on RealWorldQA, and 64.07 on MMStar, surpassing prior sub-4B state-of-the-art models on these visual question-answering benchmarks.The models are also reported to outperform same-size open-source counterparts across diverse benchmarks, including document and OCR tasks.Source: §6.2.2 · p. 18
  • SAIL-VL2-2B scored 895 on OCRBench and 93.10 on DocVQA; SAIL-VL2-8B scored 913 on OCRBench and 95.28 on DocVQA.The report describes these as state-of-the-art results below 4B and within the 8B scale, respectively.Source: §6.2.2 · pp. 18–19
  • SAIL-VL2-AnyRes-2B achieved 57.82 RefCOCOavg, while SAIL-VL2-2B attained 57.10 on Video-MME and 42.61 on MMIU; the report also notes leading LongVideoBench performance.The AnyRes model is reported to improve over fixed-resolution SAIL-VL2-2B on RefCOCO and to achieve sub-4B state of the art.Source: §6.2.2 · p. 19

6.2.3 Multimodal Reasoning Tasks

  • SAIL-VL2-8B-Thinking led open-source OpenCompass reasoning models with 54.4, while SAIL-VL2-MoE-Thinking scored 53.6 using 3B activated parameters.The MoE score exceeds Gemini-2.0-Flash at 50.6 and is competitive with GPT-4o-latest at 54.8.Source: Table 10 · p. 20, §6.2.3 · p. 20

7 Conclusion

SAIL-VL2 combines data curation, progressive training, and architectural changes to target efficient multimodal understanding and reasoning. The report describes state-of-the-art results at 2B and 8B scales across 106 benchmarks and positions the model as a foundation for the open-source multimodal community.

7 Conclusion

  • SAIL-VL2 combines data curation, progressive training, and architectural innovations, with results spanning multimodal understanding and complex reasoning.The report describes SAIL-VL2 as an open-suite vision-language foundation model.Source: §7 · p. 20
  • Across 106 benchmarks, SAIL-VL2 reports state-of-the-art results at the 2B and 8B scales, with SAIL-VL2-2B leading on MMMU and MathVista.The conclusion also reports first place on OpenCompass among officially released open-source models under 3B.Source: §7 · p. 20
  • The report presents SAIL-VL2 as a scalable foundation for the open-source multimodal community and plans further work on architectures, pre-training, and reinforcement learning.Source: §7 · p. 20

8 Contributors and Acknowledgment

The contributor section lists team members, a project leader, and supervisors.

8 Contributors and Acknowledgment

  • Core contributors are listed in reverse alphabetical order by last-name initial, with equal contributions within each group.The text notes that contributors are Douyin SAIL Team members unless otherwise specified.Source: §8 · p. 21
  • The listed contributors are Haiyang Yu, Dingkang Yang, Jiacong Wang, Han Wang, and Wenzhuo Liu.Source: §8 · p. 21
  • Xiao Liang is listed as project leader; supervisors are Xiao Liang, Shuicheng Yan, and Chao Feng.Source: §8 · p. 21

Related papers

Loading 2509.14033v2…