PaperlayerView exact revision on arXiv

Source-linked AI summary

ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models

Yuqi Liu, Liangyu Chen, Jiazhen Liu, Mingkang Zhu, Zhisheng Zhong, Bei Yu, Jiaya Jia

arXiv:2510.10606v4cs.CV

TL;DR

LVLM post-training faces complementary gaps: SFT can generalize sub-optimally, while RLVR is bounded by existing knowledge, and sequential SFT →RLVR adds overhead and forgetting risk. ViSurf combines ground-truth demonstrations with on-policy rollouts in one stage and adds reward controls. Across diverse benchmarks, it consistently outperforms SFT, RLVR, and the sequential pipeline.

  • Problem

    SFT can yield sub-optimal generalization, while RLVR is constrained by the model’s existing knowledge; sequential SFT →RLVR adds computational overhead and remains vulnerable to forgetting.

    Source: §1 · p. 1

  • Method

    ViSurf interleaves ground-truth demonstrations and on-policy rollouts in a single-stage SFT–RLVR framework, with three reward-control strategies for stable optimization.

    Source: §1 · p. 2

  • Results

    ViSurf consistently outperforms standalone SFT, RLVR, and sequential SFT →RLVR across diverse benchmarks.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Stable VQA performance is reported as evidence that ViSurf mitigates catastrophic forgetting, while the analysis corroborates its framework and design principles.

    Source: §1 · p. 2, §5 · p. 8

    The current VeRL-based reinforcement-learning codebase mainly supports the Qwen-VL series; support for additional architectures is left for future work.

    Source: A. Limitations · p. 12

Abstract

from arXiv · show

Post-training Large Vision-and-Language Models (LVLMs) typically involves Supervised Fine-Tuning (SFT) for knowledge injection or Reinforcement Learning with Verifiable Rewards (RLVR) for performance enhancement. However, SFT often leads to sub-optimal performance, while RLVR remains constrained by the model's internal knowledge base. While a sequential SFT $\rightarrow$ RLVR pipeline can be used, it introduces significant computational overhead and suffers from catastrophic forgetting. To address these limitations, we propose ViSurf (\textbf{Vi}sual \textbf{Su}pervised-and-\textbf{R}einforcement \textbf{F}ine-Tuning), a unified, single-stage paradigm that integrates the strengths of both SFT and RLVR. By analyzing their training objectives, we establish a unified framework that injects ground-truth labels directly into RLVR rollouts, facilitating simultaneous external supervision and internal reinforcement. Furthermore, we introduce three novel reward control strategies to ensure training stability and optimization. Extensive experiments demonstrate that ViSurf consistently outperforms standalone SFT, RLVR, and the traditional two-stage pipeline across diverse benchmarks. In-depth analysis corroborates these findings, validating the derivation and design principles of ViSurf.

1. Introduction

SFT and RLVR have complementary strengths, but each has limitations, and sequentially combining them adds overhead while retaining forgetting risk. ViSurf unifies their strengths in one training stage and consistently outperforms the compared paradigms across diverse domains.

1. Introduction

  • SFT excels at acquiring out-of-distribution knowledge, whereas RLVR performs better on tasks aligned with existing capabilities and can fall below baseline beyond those capabilities.The contrast is summarized in Figure 1.Source: §1 · p. 1, Figure 1 · p. 1
  • ViSurf consistently outperforms SFT, RLVR, and the SFT →RLVR pipeline, with stable VQA performance cited as evidence of mitigated catastrophic forgetting.Figure 2 summarizes performance across paradigms and shows catastrophic forgetting for SFT and the two-stage pipeline.Source: Figure 2 · p. 2, §1 · p. 2
  • ViSurf is a theoretically grounded, single-stage framework that interleaves ground-truth demonstrations with on-policy rollouts to combine external supervision and internal reinforcement.The paper derives the unified objective from shared gradient patterns between SFT and RLVR.Source: §1 · p. 2
  • Three reward-control strategies address label–rollout preference alignment, exclusion of thinking rewards for static labels, and reward smoothing to prevent optimization spikes.The strategies are proposed to support training stability in the unified framework.Source: §1 · p. 2

2. Related Works

Prior LVLM post-training work includes supervised fine-tuning for expert-guided adaptation and reinforcement learning methods that use preference data, reward models, or verifiable outcomes.

2.1. Supervised Fine-tuning for LVLMs

  • SFT is a predominant LVLM post-training paradigm, using expert-annotated data and adapting models to specialized downstream tasks.Examples include the LLaVA, QwenVL, MGM, Eagle, and InternVL series, as well as image-quality assessment and autonomous-driving tasks.Source: §2.1 · p. 2

2.2. Reinforcement Learning for LVLMs

  • RLVR methods use verification functions rather than extensive human preference datasets or a reward model, with GRPO and DAPO among the cited algorithms.Recent LVLM advances include SegZero and VisualRFT.Source: §2.2 · p. 2

3. ViSurf

The section motivates ViSurf with a non-object segmentation contrast, then develops a single-stage objective that inserts ground-truth labels into RLVR rollouts. It presents reward controls and adaptive gradient weighting to regulate external supervision and internal reinforcement.

3.1. Preliminary

  • SFT minimizes negative log-likelihood against ground-truth labels paired with visual and textual inputs.Source: §3.1 · p. 3

Reinforcement Learning with Verifiable Rewards

  • RLVR samples G outputs from the old policy, scores them with a verifiable reward, computes rollout advantages, and optimizes a clipped policy objective.The implementation removes the KL divergence term.Source: Reinforcement Learning with Verifiable… · p. 3

3.2. Case Study: Non-Object Scenarios

  • SFT performs suboptimally on gIoU but tends to identify absent objects, while RLVR achieves higher overall gIoU yet often outputs masks when no object exists.This contrast motivates combining SFT’s external grounding with RLVR’s self-exploration.Source: §3.2 · p. 3, Figure 3 · p. 3

3.3. Combining SFT and RLVR

  • ViSurf includes the ground-truth label as a high-reward sample in an augmented RLVR rollout set, combining external and internal guidance in one stage.The SFT and RLVR gradients have similar forms but differ in guidance signal and coefficient.Source: §3.3 · p. 4

3.4. Reward Control for Ground-Truth Label

  • Three reward controls align label formatting with rollout preferences, zero the reasoning-format score for labels without reasoning annotations, and smooth label rewards when rollouts already score highly.When the maximum rollout reward reaches the label reward, smoothing sets the label reward to the rollout mean, making its advantage zero.Source: §3.4 · p. 5

3.5. Optimization Analysis During Training

  • When rollouts are poor, ground-truth advantage keeps external guidance strong; when rollouts are desirable, reward control makes that advantage approximately zero and RLVR dominates.The stated upper bound is RLVR when the old policy already produces desirable rollouts, and better than either SFT or RLVR alone when it does not.Source: §3.5 · pp. 5–6

4. Experiments

Across diverse visual benchmarks, ViSurf outperforms standalone SFT, RLVR, and the sequential SFT →RLVR pipeline. The experiments also examine reward-control ablations, training behavior, prompt dependence, cost, and state-of-the-art comparisons.

4.1. Experimental Settings

  • Benchmarks span non-object and reasoning segmentation, GUI grounding, industrial anomaly detection, medical imaging, and mathematical reasoning.Experiments use Qwen2.5VL-7B, with SAM2 where needed; methods use the same training steps, and rewards vary by task.Source: §4.1 · p. 6

4.2. Comparison of Different Training Paradigms

  • 38.6% average relative improvement over the baseline; ViSurf consistently surpasses existing methods across evaluated domains.The reported gain is most pronounced where the baseline initially struggles; SFT degrades performance on OmniACT, while pure RLVR underperforms the original model on RealIAD and gRefCOCO.Source: §4.2 · p. 6
  • On Qwen2VL-7B, ViSurf consistently outperforms its counterparts in the Table 3 comparison.The experiments also assess VQA performance on ChartQA and DocVQA after fine-tuning without VQA data.Source: §4.2 · pp. 6–7, Table 3 · p. 7

4.3. Ablation of Reward Control

  • Removing alignment of ground-truth labels with rollout preferences consistently degrades performance across multiple datasets.The authors relate this ablation result to the theoretical requirement πθ ≈πθold.Source: §4.3 · p. 7
  • Reasoning traces help on complex ReasonSeg and MathVista tasks, while omitting them improves results on simple-query gRefCOCO.Removing reward smoothing also leads to performance declines across datasets; the authors note the SFT term can be superfluous when self-rollouts are already higher quality.Source: §4.3 · p. 7

4.4. In-depth Analysis

  • ViSurf’s entropy drops initially and then converges more slowly, while its training remains more stable than RLVR and SFT →RLVR as training continues.The authors report that ViSurf’s performance gain depends on baseline strength: below 50%, improvement is substantial; above 50%, its upper bound aligns with RLVR alone.Source: §4.4 · pp. 7–8
  • ViSurf gains consistently with and without detailed formatting prompts, whereas RLVR fails without explicit formatting instructions.RLVR and ViSurf are more memory-efficient but take longer per training step because rollout generation adds overhead.Source: §4.4 · p. 8, Table 5 · p. 8, Table 6 · p. 8

4.5. Comparison with State-of-The-Arts

  • ViSurf achieves state-of-the-art performance on gRefCOCO and ReasonSeg, and qualitative examples show successful performance across multiple visual perception tasks.Figure 7 presents visualizations on various tasks.Source: §4.5 · p. 8, §4.6 · p. 8, Figure 7 · p. 8

5. Conclusion

ViSurf unifies SFT and RLVR in a single-stage framework, interleaving ground-truth labels with model-generated rollouts and using reward controls for stability. Across diverse benchmarks, it outperforms standalone methods and the sequential SFT →RLVR pipeline.

5. Conclusion

  • ViSurf integrates SFT and RLVR into a single-stage post-training framework, grounded in analysis of their objectives and gradient dynamics.Its implementation interleaves ground-truth labels with model-generated rollouts.Source: §5 · p. 8
  • Three reward-control strategies are used to maintain training stability during joint optimization.Source: §5 · p. 8
  • Across diverse benchmarks, ViSurf outperforms standalone SFT, RLVR, and the traditional sequential SFT →RLVR pipeline.The paper’s in-depth analysis is reported to corroborate its theoretical foundations.Source: §5 · p. 8

A. Limitations

The current reinforcement-learning codebase relies on VeRL, which mainly supports the Qwen-VL series; extending support to other architectures is left for future work as framework compatibility improves.

A. Limitations

B. Discussion and Future Work

ViSurf combines RLVR’s internal reinforcement with SFT’s external guidance. The authors note that although their labels contain only final answers, the paradigm is compatible with explicit reasoning traces and knowledge distillation.

B. Discussion and Future Work

C. Detailed Experimental Settings

The experimental settings cover segmentation, GUI grounding, anomaly detection, medical imaging, and mathematical reasoning. The prompt-design comparison contrasts detailed output-format guidance with a simpler answer placeholder.

C. Detailed Experimental Settings

  • ReasonSeg evaluates reasoning-based segmentation on 200 validation and 779 test images using gIoU; training uses Multi-objects-7K.gRefCOCO evaluation uses gIoU and N-Acc for non-object queries.Source: C. Detailed Experimental Settings · p. 12
  • OmniACT evaluation uses 6,101 training samples and tests whether a predicted point falls inside the target region.The task covers Desktop and Web GUI grounding.Source: C. Detailed Experimental Settings · p. 12
  • RealIAD uses 3,292 training and 2,736 disjoint test samples, with performance measured by ROC AUC.Source: C. Detailed Experimental Settings · p. 12
  • ISIC2018 lesion segmentation uses bbox accuracy, while MathVista-testmini contains 1,000 tasks and draws on around 10k training examples from several datasets.The bbox-accuracy threshold is IoU above 0.5; listed math-data sources include WeMath, MathVision, Polymath, SceMQA, and Geometry3K.Source: C. Detailed Experimental Settings · p. 12

D. Illustration of Prompt Design

  • Prompt 2 supplies detailed output-format instructions, whereas Prompt 1 uses the placeholder “answer here” between answer tags.The detailed example requests structured bbox and point outputs in JSON.Source: D. Illustration of Prompt Design · p. 12

E. Additional Explanation on pure RLVR

Pure RLVR performance is sensitive to randomness, though rare early successes can yield competitive results. Those cases remain marginally inferior to ViSurf.

E. Additional Explanation on pure RLVR

F. Non Object Training Data

The non-object training data adapts referring-expression samples and adds negative examples that teach the model to return an empty list when no target exists.

F. Non Object Training Data

G. Data Mixture in Non-object Scenarios

For non-object scenarios, ViSurf performance improves as the amount of non-object training data increases, and absent objects can be detected with as few as 50 negative samples.

G. Data Mixture in Non-object Scenarios

Related papers

Loading 2510.10606v4…