PaperlayerView exact revision on arXiv

Source-linked AI summary

RL makes MLLMs see better than SFT

Junha Song, Sangdoo Yun, Dongyoon Han, Jaegul Choo, Byeongho Heo

arXiv:2510.16333v2cs.CVcs.LG

TL;DR

The paper addresses limited understanding of how SFT and RL shape MLLMs and their vision encoders, a gap associated with an LLM-centric view of MLLM capabilities. It compares the strategies and analyzes visual representations, then proposes PIVOT. It reports stronger, more localized visual features with DPO and PIVOT-enhanced encoders outperforming larger or newer counterparts in MLLMs.

  • Problem

    Research lacks a systematic comparison of SFT and RL in MLLMs, especially analysis of how they reshape vision encoders and visual representations.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    The paper compares SFT with DPO, analyzes vision-encoder representations, and recasts RL training as PIVOT, an auxiliary vision-encoder training process.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    DPO achieves superior MLLM performance, particularly on detailed visual-comprehension tasks, and induces stronger, more localized visual features; PIVOT-enhanced encoders can outperform larger and newer counterparts in MLLMs.

    Source: §1 · p. 2, §7 · p. 11

  • Takeaways & Limitations

    PIVOT offers a promising direction for evolving MLLM vision encoders, with the reported training using fewer than 1% of the GPUs of standard vision pre-training.

    Source: §1 · p. 2

    The SFT–DPO comparison could be strengthened by equalizing supervision signals, for example by extending SFT to include negative examples.

    Source: §7 · p. 11

Abstract

from arXiv · show

A dominant assumption in Multimodal Language Model (MLLM) research is that its performance is largely inherited from the LLM backbone, given its immense parameter scale and remarkable capabilities. This has created a void in the understanding of the vision encoder, which determines how MLLMs perceive images. The recent shift in MLLM training paradigms, from Supervised Finetuning (SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the significant lack of analysis on how such training reshapes the vision encoder as well as the MLLM. To address this, we first investigate the impact of training strategies on MLLMs, where RL shows a clear advantage over SFT in strongly vision-related VQA benchmarks. Motivated by this, we conduct a critical yet under-explored analysis of the vision encoder of MLLMs through diverse and in-depth experiments, ranging from ImageNet classification and segmentation to gradient visualization. Our results demonstrate that MLLM's post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual representations. Specifically, the key finding of our study is that RL produces stronger and precisely localized visual representations compared to SFT, boosting the ability of the vision encoder for MLLM. We then reframe our findings into a simple recipe for building strong vision encoders for MLLMs, Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs, a PIVOT-trained vision encoder outperforms even larger and more heavily-trained counterparts, despite requiring less than 1% of the computational cost of standard vision pretraining. This result opens an effective and efficient path for advancing the vision backbones of MLLMs. Project page available at https://june-page.github.io/pivot/

1 Introduction

The paper challenges the LLM-centric account of MLLM capability by examining how SFT and RL affect both MLLMs and their vision encoders. It reports stronger visual representations with RL and proposes PIVOT as a recipe for evolving vision encoders for MLLMs.

1 Introduction

  • The literature lacks a systematic comparison of SFT instruction-following and RL preference alignment, including their effects on vision encoders and model scaling.The paper connects this gap to an implicit LLM-centric assumption about the source of MLLM capabilities.Source: §1 · p. 2
  • RL yields significant gains on vision-centric tasks and drives stronger visual representations than SFT, with gradient visualizations tracing optimization signals to the vision encoder.The paper focuses its RL analysis on DPO for simplicity.Source: §1 · p. 2
  • PIVOT recasts RL training as an auxiliary vision-encoder training process and is evaluated with CLIP, DINO, and MAE encoders.The paper reports that PIVOT-enhanced encoders outperform their original counterparts when used within MLLMs.Source: §1 · p. 2
  • 18 hours on 8 H100 GPUs and fewer than 1% of standard vision pre-training GPUs produced PIVOT results that surpassed substantially larger and subsequent-generation encoders in MLLMs.The reported comparisons include SigLIP2-So/16 + PIVOT outperforming SigLIP2-g/16 and SigLIP1-So/142 + PIVOT outperforming SigLIP2-So/16.Source: §1 · p. 2

2 MLLMs on RL: Where do we stand?

Prior work adopted LLM training advances in MLLMs, first using SFT to align pretrained language and vision models, then exploring RL for downstream performance. DPO became the predominant RL choice in this literature, motivating its use as the paper’s primary representative for comparison with SFT.

2 MLLMs on RL: Where do we stand?

  • Early MLLMs combined pretrained LLMs and vision models, then used SFT on vision-language data to align the LLM with visual representations.The passage cites LLaVA-Next and Cambrian as examples (Li et al., 2024b; Tong et al., 2024a).Source: §2 · p. 3
  • Prior MLLM studies applied RL as an auxiliary process to boost downstream performance, including approaches using visual preference data or modified objectives to mitigate hallucinations.The cited work also reports RL advantages in map navigation and robot action planning (Chu et al., 2025; Li et al., 2025b).Source: §2 · p. 3
  • DPO had become the predominant RL choice among MLLM studies using PPO or DPO, so the paper adopts DPO for a controlled comparison with SFT.The passage cites PPO studies (Sun et al., 2024) and identifies DPO as the predominant choice.Source: §2 · p. 3

3 How do SFT and RL affect MLLMs?

The section compares SFT and DPO across MLLM benchmarks, varying model and post-training data scales. DPO’s advantage is clearest on vision-related tasks.

3.1 Experimental setup & prerequisite

  • The experiments vary Qwen2.5 from 0.5B to 7B and SigLIP2 from B/16 to g/16, with a two-layer MLP projector.Training has projector-only and end-to-end pretraining stages before full-parameter SFT or DPO post-training.Source: §3.1 · p. 3
  • The controlled Stage 2 comparison uses the same number of image-query-response pairs for SFT and DPO, with 20K sampled post-training instances and evaluation across 16 tasks.The benchmark suite covers General, Knowledge, OCR & Chart, and Vision-Centric VQA.Source: §3.1 · pp. 3–4

3.2 Analysis and findings

  • Scaling SigLIP2 from B/16 to g/16 under DPO improves Vision-Centric VQA by +4.5%p and OCR & Chart VQA by +10.6%p, but Knowledge VQA by only +1.9%p.The language model is fixed to Qwen2.5-3B.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • DPO beats SFT most on vision-centric benchmarks: with SigLIP2-L/16, gains are +4.2%p on OCR & Chart and +2.4%p on Vision-Centric VQA, and remain strong as the LLM scales.With SigLIP2-g/16, the DPO–SFT gaps are +3.1%p on OCR & Chart and +4.2%p on Vision-Centric VQA.Source: Figure 2 · p. 4, Figure 3 · p. 4, §3.2 · p. 5
  • DPO with 3K samples scores 60.4%p, exceeding the 59.5%p of SFT trained on 40K samples.The comparison fixes the model at Qwen2.5-1.5B with SigLIP2-So/16.Source: §3.2 · p. 5, Figure 4 · p. 5

4 How does MLLM training affect visual representations?

The analysis evaluates visual representations independently of the MLLM’s language component. Across classification, gradients, and segmentation, DPO is associated with stronger and more localized representations.

4 How does MLLM training affect visual representations?

  • The section investigates how SFT and DPO change the vision encoder, extending the preceding MLLM benchmark comparison into standalone vision analysis.The preceding comparison found DPO gains on vision-related MLLM tasks.Source: §4 · p. 5

4.1 Experimental setup

  • The study isolates visual representations by detaching the vision encoder and projector, then evaluating the visual components on ImageNet classification and semantic segmentation.Performance is measured using features from the vision encoder alone or from the encoder-projector combination.Source: §4.1 · p. 6

4.2 Evaluating vision encoders beyond MLLM Benchmarks

  • DPO improves ImageNet Top-1 accuracy by +1.83%p for SigLIP2-So/16 with a Qwen-3B head and +1.96%p for SigLIP2-L/16 with a Qwen-1.5B head.For SigLIP2-So/16, training with a 7B rather than a 0.5B LLM yields a +4.4%p ImageNet accuracy increase under DPO.Source: §4.2 · p. 6, Figure 6 · p. 6
  • Only DPO benefits from increased Stage 2 data scale in visual-representation quality, although both SFT and DPO improve on MLLM benchmarks as data increases.The comparison uses Qwen2.5-1.5B and SigLIP2-So/16.Source: §4.2 · p. 6, Figure 5 · p. 6
  • DPO’s gradients focus more precisely on question-relevant regions than SFT’s, whose signals tend to be scattered.The authors hypothesize that DPO’s contrastive objective produces fine-grained visual gradients when distinguishing chosen from rejected responses.Source: §4.2 · p. 7, Figure 7 · p. 7
  • DPO-trained encoders outperform SFT-trained encoders on segmentation, including a 1.08%p patch-level recall increase with CLIP-L/14 336px.Qualitative segmentation maps from DPO-trained encoders align more closely with ground truth.Source: §4.2 · p. 7, Figure 8 · p. 7, Figure 9 · p. 8
  • DPO-trained vision encoders show stronger alignment with reference LLMs, and alignment scores rise when paired with a larger LLM.The authors interpret the larger-LLM trend as support for more informative backward signals reaching the vision encoder.Source: §4.2 · pp. 6, 8, Figure 10 · p. 8

5 Does RL’s advantage hold beyond DPO?

The paper tests whether the reported DPO advantage extends to another RL method, GRPO. GRPO outperforms SFT across the evaluated MLLM and vision-only dimensions.

5 Does RL’s advantage hold beyond DPO?

  • The GRPO comparison trains QwenVL-2.5-3B for 1,500 post-training steps with GRPO or SFT, holding other configurations identical.Evaluation includes VL benchmarks and vision-only ImageNet classification and segmentation probing.Source: §5 · p. 8
  • GRPO exceeds SFT across evaluated dimensions, with +3.1%p average MLLM score, +1.93%p on ImageNet, and +1.83%p on segmentation.Its largest MLLM gains are +4.3%p on OCR & Chart and +3.4%p on Vision-Centric tasks.Source: §5 · p. 8, Table 1 · p. 9
  • The reported gains support the conclusion that RL’s advantage over SFT is not specific to DPO but also appears with GRPO.The authors also report extended PPO and MPO experiments in Section B.1.Source: §5 · p. 9

6 What’s next: Unlocking vision model potential via RL

PIVOT applies DPO or SFT with an LLM head to existing vision encoders, then evaluates the frozen encoder with a new LLM. Across comparisons, PIVOT improves MLLM performance, including over some larger or newer encoders.

6.1 Experimental setup

  • PIVOT attaches a commonly used vision encoder to an LLM and trains it through pre-training followed by DPO or SFT, then freezes and evaluates the detached encoder with Qwen2.5-1.5B.Training uses 3M instruction-following samples and 20K preference pairs; evaluation also applies projector-only pre-training and instruction finetuning.Source: §6.1 · p. 9

6.2 Results

  • 53.2%p average VQA score for PIVOT-enhanced SigLIP1-So/14 surpasses the 52.4%p of newer SigLIP2-So/16.The PIVOT-enhanced SigLIP2-So/16 also scores 55.6%p versus 53.9%p for SigLIP2-g/16, despite having 2.5 times fewer parameters.Source: §6.2 · p. 9
  • PIVOT evaluates frozen, enhanced visual representations coupled to a new LLM, following prior MLLM evaluation protocols.This design assesses the encoder's standalone usefulness within MLLMs.Source: Figure 11 · p. 10
  • DPO-based PIVOT scores 56.7%p versus 55.4%p for SFT-based PIVOT with SigLIP2-g/16, so the study adopts DPO by default.All five tested classic vision models improve MLLM performance with PIVOT; gains include self-supervised MAE and MOCO and a supervised ImageNet classifier.Source: §6.2 · p. 10
  • Combining SigLIP1-So/14 and ConvNeXt-XXL raises the average score from 50.9%p to 51.4%p, while SigLIP1-So/14 + PIVOT alone reaches 53.2%p.Adding ConvNeXt-XXL to SigLIP1 + PIVOT further raises the score to 53.6%p.Source: §6.2 · p. 10

6.3 Importance of PIVOT

  • PIVOT reframes LLM-head training with DPO as an under-explored regime, and reports stronger RL-driven visual improvements than SFT.The paper positions PIVOT as a practical training recipe rather than a new method, and identifies room to improve even state-of-the-art encoders within MLLMs.Source: §6.3 · pp. 10–11

7 Conclusion & Broader impacts

The paper reports that DPO outperforms SFT on MLLM performance and produces stronger, more localized visual features, then validates PIVOT across vision encoders. It identifies unequalized SFT and DPO supervision as a comparison limitation.

7 Conclusion & Broader impacts

  • DPO achieves better MLLM performance than SFT, especially on detailed visual-comprehension tasks, and induces stronger, more localized visual features.The authors consolidate these findings into PIVOT and report its efficacy across diverse vision encoders.Source: §7 · p. 11
  • The comparison could be strengthened by equalizing SFT and DPO supervision, for example by giving SFT negative examples.The authors also identify designing dataset formats that better leverage DPO for visual representation learning as future work.Source: §7 · p. 11

A.1 MLLMs.

  • MLLMs connect a pretrained vision encoder to an LLM through a multimodal projector; later improvements have used larger components, better conversational data, and visual-understanding techniques.The cited background describes SFT as the dominant training strategy and RL as an emerging alternative.Source: §A.1 · p. 20

A.2 LLMs with RL.

  • DPO directly optimizes pairwise preferences, while GRPO uses group-wise relative rankings; the paper investigates how such training contrasts with SFT for MLLMs.The discussion situates this comparison alongside prior RL-versus-SFT studies of LLM adaptation and reported MLLM RL advances.Source: §A.2 · p. 20, §A.3 · p. 20

A.4 Vision-centric pre-trainings

The appendix reports that RL variants improve MLLM performance and visual representations beyond SFT, while DPO's learning-rate needs differ from SFT's. It situates PIVOT among vision encoder training approaches and explains why DPO enables a more controlled comparison.

A.4 Vision-centric pre-trainings

  • PIVOT is presented as a CLIP-style alternative that applies language-aligned supervision to existing encoders, using fewer than 1% of SigLIP2's GPUs and data.The paper contrasts this with image-only self-supervision and image-language supervised pretraining, and focuses on RL rather than SFT-driven representation changes.Source: §A.4 · p. 21

B.1 Other RL algorithms vs. SFT

  • DPO avoids a reward model and uses data formatted similarly to SFT, enabling a controlled comparison with the same number of image-query-response pairs.The authors note that PPO's external reward model and GRPO's reliance on verifiable signals make matched SFT comparisons harder.Source: §B.1 · pp. 21–22
  • MPO combines DPO, SFT, and binary preference-classification objectives; its vision-only results match DPO, while it slightly improves vision-centric and knowledge tasks when representations are evaluated within MLLMs.The appendix describes evaluation of MLLM benchmarks, vision-only benchmarks, and encoders re-integrated within MLLMs.Source: §B.1 · p. 22, Table D · p. 23
  • Across experiments with GRPO, PPO, and MPO, RL-based training improves MLLM performance and visual representations beyond SFT.PPO outperforms its SFT counterpart across 16 MLLM benchmarks; MPO's average-score gain over DPO is modest, and its vision-only results match DPO.Source: §B.1 · p. 22, Table D · p. 23

B.2 MLLM Training Sensitivity

  • DPO requires substantially smaller learning rates than SFT in the tested Qwen2.5-3B and SigLIP2-So/16 setup.The authors attribute this partly to DPO accounting for chosen and rejected responses, effectively doubling the batch size per iteration relative to SFT.Source: §B.2 · p. 23

B.3 MLLM performance under new data distributions

Across shifted-distribution, SFT-friendly, and extended benchmark settings, DPO generally retains an advantage over SFT, though the size of that advantage varies. Text-only results are mixed: DPO leads on MMLU, while HellaSwag and GSM8K remain comparable.

B.3 MLLM performance under new data distributions

  • At 80% new-distribution samples, DPO leads SFT by 17.9%p on vision-centric VQA, while SFT’s overall performance falls from 62.2%p at 0% to 53.2%p.DPO remains robust as shifted-distribution samples increase; the evaluation covers 16 benchmarks.Source: §B.3 · p. 24

B.4 MLLM performance under more SFT-friendly data

  • DPO’s advantage narrows on SFT-friendly data, but its performance does not fall below SFT’s across the 16 benchmarks.The authors hypothesize that shorter responses reduce the information gap between chosen and rejected samples, diminishing DPO’s effectiveness.Source: §B.4 · p. 25

B.5 MLLM performance on hallucination, captioning, and robustness benchmarks

  • DPO consistently outperforms SFT across hallucination, captioning, and real-world robustness benchmarks.The authors attribute these gains to contrastive DPO signals strengthening visual understanding.Source: §B.5 · p. 25

B.6 MLLM performance on text-only benchmarks

  • DPO-trained models outperform SFT on MMLU, while performance remains comparable on HellaSwag and GSM8K.The authors speculate that overlap between MPO training data and MMLU topics may explain this difference; they note further study is needed.Source: §B.6 · pp. 25–26

B.7 PIVOT ablation study

The ablations test PIVOT’s transfer across evaluation data scales and training protocols, as well as how much of its projector to reuse. PIVOT-based MLLMs compare favorably with existing models, including some larger counterparts.

B.7 PIVOT ablation study

  • Reusing the first frozen PIVOT-projector layer with one added layer yields the best downstream MLLM performance; deeper reuse performs worse.The authors adopt this 1+1 configuration in subsequent Section 6 experiments.Source: §B.7 · p. 26
  • With LLaVA-OV-3M evaluation data, PIVOT’s advantage persists; Qwen2.5-1.5B with PIVOT-enhanced SigLIP2-So/16 gains +2.3%p over the base encoder.The table caption states that PIVOT consistently improves performance with both Cambrian-737K and LLaVA-OV-3M Stage 3 data.Source: §B.7 · p. 26, Table I · p. 27
  • PIVOT-enhanced encoders consistently outperform their baseline counterparts even when all model parameters are updated during evaluation.This setting updates the vision encoder, LLM, and projector.Source: §B.7 · p. 27

B.8 PIVOT performance comparison with existing MLLMs

  • PIVOT (1.5B) outperforms other MLLMs built on language models under 2B parameters and exceeds larger models including LLaVA-1.5B, BLIP-2, and SPHINX.The comparison covers 16 benchmarks.Source: §B.8 · p. 27

C.1 MLLM training data scaling

Across controlled comparisons, DPO-trained MLLMs outperform SFT-trained models across data scales, encoders, and language models, with a stronger advantage on strongly vision-related tasks. Additional analyses find differences in visual representations and describe how the training objectives and task domains may relate to those differences.

C.1 MLLM training data scaling

  • DPO outperforms SFT as training data grows; on Knowledge VQA, the gap narrows from +4.2 to +0.7 as data increases from 3K to 10K samples.With fixed hyperparameters, SFT’s positive trend from 3K to 40K does not continue at larger scales; the authors note learning-rate or weight-decay adjustments may mitigate degradation.Source: §C.1 · pp. 27–28, Figure C · p. 28

C.2 Performance of MLLMs on all 16 benchmarks

  • DPO-trained MLLMs consistently surpass SFT-trained models across data scales, vision encoders, and language models, with a larger advantage on strongly vision-related than weakly vision-related tasks.The full comparison covers 16 benchmarks.Source: §C.2 · p. 28

C.3 ImageNet classification with a vision encoder

  • Linear-probe results reinforce that DPO improves vision-encoder representations more than SFT, and encoder performance rises with the size of its training LLM.This probe uses vision-encoder features, unlike the main Figure 6 analysis, which uses outputs from both the encoder and projector.Source: §C.3 · p. 28, Figure D · p. 31

C.4 Gradient visualization of a vision encoder

  • DPO produces more concentrated gradients on question-relevant image regions than SFT’s diffuse gradients; for global queries, both distribute gradients across the image.These results support the authors’ hypothesis that post-training data characteristics affect how DPO enhances visual representations.Source: §C.4 · p. 29

C.5 Segmentation probing with a vision encoder

  • With a CLIP-L/14 336px encoder and Qwen2.5-3B, DPO-trained vision encoders produce segmentation maps more consistent with ground truth than SFT-trained encoders.Source: §C.5 · p. 29

D.1 Understanding SFT and DPO

  • SFT maximizes the likelihood of the chosen response, whereas DPO uses chosen and rejected responses to increase the former and decrease the latter relative to a reference policy.Source: §D.1 · p. 30

D.2 Low performance gap on Knowledge VQA

  • DPO’s advantage is smaller on Knowledge VQA; the authors hypothesize that chosen answers may already contain sufficient facts, making rejected responses less informative comparatively.They identify the interplay between preference-data characteristics and task domains as a direction for future research.Source: §D.2 · pp. 30–31

D.3 Positioning our work within RL-based MLLM research

  • Even when both comparison models receive SFT in Stage 1, the model receiving DPO rather than SFT in Stage 2 retains an advantage.This comparison was designed to isolate the effect of Stage 2 RL more fairly than comparing SFT-only against SFT followed by DPO.Source: §D.3 · p. 31

D.4 Limitations

The authors identify unequal per-update supervision and possible preference-data bias as limitations, while noting efforts to test robustness across data variations. The appendices detail the training and evaluation protocols and identify further scope boundaries and future research directions.

D.4 Limitations

  • DPO updates use chosen and rejected responses, while SFT uses only chosen responses, so supervision per iteration is not perfectly matched; MPO preference pairs may also favor DPO.The authors test fewer DPO samples and varied data types, but say further investigation and a more balanced comparison incorporating negative responses into SFT remain necessary.Source: §D.4 · pp. 31–32

D.5 Future work

  • The authors identify testing other MLLM architectures and LLM backbones, and designing data formats that better leverage DPO for visual representations, as future work.Source: §D.5 · p. 32

E.1 Pre-training & Post-training

  • For Stage 2, the controlled setup uses a 1 × 10−6 learning rate for DPO and 5 × 10−6 for SFT; DPO’s learning rate is substantially smaller because it uses both responses.The experiments use LLaVA-OneVision code and a 20K MMPR-1.2 subset for post-training.Source: §E.1 · pp. 32–33

E.2 Evaluation benchmarks

  • The evaluation uses 16 benchmarks across four domains and rescales MME by a factor of 20 when computing the overall average.Source: §E.2 · p. 34, §E.1 · p. 34

E.3 ImageNet classification

  • ImageNet probing freezes the feature extractor and trains an L2-regularized logistic regression classifier; validation uses a random 50K subset.A prototype-based classifier is also used for faster validation, with similar performance trends to logistic regression.Source: §E.3 · p. 34

E.4 Grad-CAM

  • Gradient visualization uses one sample and inspects gradients around step 20 to avoid near-zero initial learning rates producing uninformative patterns.Source: §E.4 · pp. 34–35

E.5 Semantic segmentation

  • Segmentation probing freezes the vision encoder, trains a two-layer MLP head on ADE20K for five epochs, and evaluates patch-level recall on validation data.Source: §E.5 · p. 35

E.6 Representation alignment

  • Representation alignment uses a dataset of only 1,024 examples, so the authors evaluate three random seeds and average the results to address variability.Source: §E.6 · p. 35

E.7 PIVOT-enhanced vision model evaluation

The evaluation tests PIVOT-enhanced vision encoders in MLLMs through downstream training and visual probing; DPO-trained maps align more closely with segmentation ground truth than SFT-trained maps.

E.7 PIVOT-enhanced vision model evaluation

  • The evaluation first contrastively pretrains the vision encoder with CLIP or SigLIP, then applies preference-instructed SFT and DPO using 3M+20K samples.Source: §E.7 · p. 35
  • For downstream MLLM evaluation, the tuned encoder is frozen, paired with a new Qwen-1.5B LLM, and finetuned on Cambrian 737K.The setup uses batch size 256 and learning rate 1 × 10−5.Source: §E.7 · p. 35
  • DPO-trained CLIP-L/14 336px produces segmentation maps consistent with ground truth, unlike the broader maps from SFT.The probing uses Qwen2.5-3B as the base LLM.Source: §E.7 · p. 37
Loading 2510.16333v2…