Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL

Sudong Wang; Weiquan Huang; Xiaomin Yu; Zuhao Yang; Hehai Lin; Keming Wu; Chaojun Xiao; Chen Chen; Wenxuan Wang; Beier Zhu; Yunjian Zhang; Chengwei Qin

Source-linked AI summary

Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL

Sudong Wang, Weiquan Huang, Xiaomin Yu, Zuhao Yang, Hehai Lin, Keming Wu, Chaojun Xiao, Chen Chen, Wenxuan Wang, Beier Zhu, Yunjian Zhang, Chengwei Qin

arXiv:2604.28123v3Apr 30, 2026cs.CVcs.AIcs.CL

TL;DR

SFT can introduce distributional drift in multimodal models, motivating PRISM, which adds black-box adversarial on-policy alignment with perception and reasoning experts before RLVR. PRISM consistently improves RLVR performance across algorithms and benchmarks, reaching +4.4 and +6.0 average points over SFT-to-RLVR on 4B and 8B models.

Problem
SFT can introduce distributional drift in multimodal models, compromising alignment with demonstrations and retention of the model’s original distribution.
Method
PRISM inserts black-box adversarial on-policy distillation between SFT and RLVR, using an MoE discriminator with dedicated perception and reasoning experts.
Results
+4.4 and +6.0 average points over SFT→GRPO on 4B and 8B, respectively, with similar improvements for DAPO and GSPO across multimodal benchmarks.
Takeaways & Limitations
PRISM provides a consistently better RLVR initialization across downstream RL algorithms and diverse multimodal benchmarks.
Takeaways & Limitations
PRISM adds training cost and relies on structured visual-caption and reasoning-trace formats, limiting applicability to tasks without this decomposition.

Abstract

from arXiv · show

The standard post-training recipe for large multimodal models (LMMs) applies supervised fine-tuning (SFT) on curated demonstrations followed by reinforcement learning with verifiable rewards (RLVR). However, SFT introduces distributional drift that neither preserves the model's original capabilities nor faithfully matches the supervision distribution. This problem is further amplified in multimodal reasoning, where perception errors and reasoning failures follow distinct drift patterns that compound during subsequent RL. We introduce PRISM, a three-stage pipeline that mitigates this drift by inserting an explicit distribution-alignment stage between SFT and RLVR. Building on the principle of on-policy distillation (OPD), PRISM casts alignment as a black-box, response-level adversarial game between the policy and a Mixture-of-Experts (MoE) discriminator with dedicated perception and reasoning experts, providing disentangled corrective signals that steer the policy toward the supervision distribution without requiring access to teacher logits. While 1.26M public demonstrations suffice for broad SFT initialization, distribution alignment demands higher-fidelity supervision; we therefore curate 113K additional demonstrations from Gemini 3 Flash, featuring dense visual grounding and step-by-step reasoning on the hardest unsolved problems. Experiments on Qwen3-VL show that PRISM consistently improves downstream RLVR performance across multiple RL algorithms (GRPO, DAPO, GSPO) and diverse multimodal benchmarks, improving average accuracy by +4.4 and +6.0 points over the SFT-to-RLVR baseline on 4B and 8B, respectively. Our code, data, and model checkpoints are publicly available at https://github.com/XIAO4579/PRISM.

1 Introduction

PRISM inserts a standalone distribution-alignment stage between SFT and RLVR to correct SFT-induced drift before reinforcement learning. It uses black-box adversarial on-policy distillation with an MoE discriminator that separates perception and reasoning corrections without requiring teacher logits.

Motivation: SFT can induce distributional drift, leaving the model mismatched with demonstrations and unable to retain its original favorable distribution.This drift can compound multimodal perception and reasoning failures during subsequent RLVR.
Alignment mechanism: PRISM uses a logit-free, black-box formulation that drives post-SFT policy responses toward the supervision distribution without an external teacher.The framework provides decoupled corrective signals for perception and reasoning drift.
Alignment mechanism: The alignment stage formulates a minimax game between the policy and an MoE discriminator with dedicated vision and reasoning experts.The discriminator probes perceptual grounding and reasoning consistency while distinguishing policy rollouts from the supervision pool.
Evaluation: Experiments on Qwen3-VL across diverse multimodal benchmarks and multiple RL algorithms show consistent gains over the standard SFT→RLVR pipeline.Further analysis indicates that alignment substantially narrows the distributional gap left by SFT.
PRISM: PRISM positions on-policy distillation as an intermediate alignment stage between SFT and RLVR.The resulting distribution-aligned policy is intended to provide a stronger initialization for downstream RLVR.

2 Related Work

RLVR has become a dominant paradigm for improving reasoning in language and multimodal models, while OPD addresses the distribution mismatch of standard off-policy distillation. PRISM extends this line by using OPD as an intermediate alignment stage before RLVR rather than a terminal objective.

Reinforcement learning with verifiable rewards: RLVR has emerged as a dominant paradigm for improving reasoning in large language and large multimodal models.DeepSeek-R1 showed that pure RL with verifiable rewards can elicit emergent chain-of-thought reasoning without human-labeled traces.
On-policy distillation: Standard teacher-output SFT distillation suffers from training–inference distribution mismatch, whereas OPD trains the student on its own generations.The passage identifies GKD and subsequent extensions as developments broadening the OPD paradigm.
On-policy distillation: PRISM positions OPD as an intermediate alignment stage before RLVR, contrasting with existing methods that typically treat distillation as the terminal training objective.PRISM also employs a mixture-of-experts discriminator with dedicated vision and reasoning components instead of a single undifferentiated discriminator or divergence signal.

3 Method

PRISM inserts adversarial on-policy distribution alignment between SFT and RLVR to repair SFT-induced drift before outcome-based optimization. Its MoE discriminator separates perception and reasoning errors, jointly guiding policy updates toward the supervision distribution.

Pipeline: PRISM uses three stages: SFT initializes a multimodal reasoning policy, adversarial OPD with an MoE discriminator recalibrates its distribution, and RLVR performs final improvement.The alignment stage explicitly intervenes between SFT and outcome-based RLVR.
Pipeline: 113K curated multimodal reasoning samples target problems with zero pass rate under strong contemporary models and contain Gemini 3 Flash solutions with fine-grained visual grounding and step-by-step deduction.Multi-stage filtering includes format validation and LLM-based correctness verification; 107K samples are used for SFT.
Distribution alignment: PRISM removes KL regularization during alignment because anchoring the policy near its SFT initialization would oppose correcting SFT-induced distributional drift.The resulting aligned checkpoint initializes the final RLVR stage.
Distribution alignment: The MoE discriminator assigns dedicated experts to visual grounding and logical reasoning because a single discriminator may be too coarse for these heterogeneous error modes.The perception expert evaluates visual descriptions, while the reasoning expert evaluates reasoning-trace consistency and deduction validity.
Distribution alignment: The discriminator is trained with Bradley-Terry loss to score reference responses above policy rollouts, while both experts jointly adapt throughout alignment as on-policy discriminators.The policy samples current rollouts and is optimized using the MoE discriminator’s reward; policy and discriminator updates alternate.

4 Experiments

Experiments on Qwen3-VL-4B and Qwen3-VL-8B show that PRISM improves downstream RLVR across algorithms and benchmarks, while ablations and training dynamics support the necessity of distribution alignment and disentangled multimodal feedback.

Main results: PRISM+GRPO improves over the SFT→GRPO baseline by +4.4 average points on 4B and +6.0 on 8B, with similar gains for DAPO and GSPO.The largest gains occur on MathVision and WeMath, and PRISM+GRPO uses fewer tokens per response.
Main results: The alignment-stage checkpoint has accuracy comparable to SFT before RLVR, indicating that alignment primarily corrects the response distribution rather than directly optimizing answer correctness.Its downstream value appears after RLVR.
Main results: SFT degrades the Instruct checkpoint more severely on 8B than 4B, while SFT→RLVR with GRPO and GSPO barely recovers original 8B performance.The result indicates that RLVR alone cannot fully compensate for SFT-induced distributional drift in the stronger model.
Ablations: Replacing the MoE discriminator with an equal-compute dense model causes a −3.4 avg. degradation, including −6.0 on WeMath and −4.9 on MathVerse.The dense discriminator collapses perception and reasoning feedback into one scalar, preventing disentanglement when the two axes move differently.
Ablations: Removing the alignment stage produces a −4.4 avg. drop, while using only 107K curated SFT samples instead of the full 1.37M corpus produces a −3.7 avg. drop.The reduced-data variant still reaches 62.5 avg. and outperforms the full-data SFT→RLVR baseline without alignment.
Training dynamics: Perception alignment peaks and converges quickly, whereas reasoning alignment rises gradually and oscillates more before both experts reach a comparable stable equilibrium.Structural proxies likewise show that alignment substantially reduces mismatches along reasoning-step and caption-description dimensions, with improvement persisting through RLVR.

5 Conclusion

PRISM mitigates SFT-induced distributional drift by inserting a black-box adversarial on-policy distillation alignment stage into a three-stage post-training pipeline. Its MoE discriminator uses dedicated perception and reasoning experts to provide disentangled corrective signals during the policy’s own rollouts.

Conclusion: PRISM is a three-stage post-training pipeline that explicitly aligns the post-SFT policy through black-box adversarial on-policy distillation.The alignment stage targets distributional drift introduced by SFT.
Conclusion: Its Mixture-of-Experts discriminator has dedicated perception and reasoning experts that provide disentangled corrective signals.These signals steer the policy toward the supervision distribution under its own rollout dynamics.

A Implementation Details · A.1 Training Details

PRISM’s three training stages use LlamaFactory for SFT and veRL with vLLM for alignment and RLVR, with stage-specific hyperparameters summarized in Table 3. SFT freezes visual components, alignment jointly trains policy and discriminator, and RLVR starts from the alignment checkpoint with longer response limits.

A Implementation Details: The SFT stage uses LlamaFactory, while alignment and RLVR use veRL with vLLM as the inference engine.These procedures cover all three stages of PRISM.
A.1 Training Details: Table 3 provides the detailed training hyperparameters for each PRISM stage.
A.1 Training Details: SFT performs full-parameter language-model fine-tuning while freezing the vision tower and multimodal projector.Training uses an 8192-token cutoff, online stream packing, one epoch, cosine scheduling with peak learning rate 1e-5 and warmup ratio 0.1, and DeepSpeed ZeRO-2.
A.1 Training Details: Alignment jointly trains the policy and MoE discriminator with a shared learning rate of 1e-6 and N = 16 rollouts per prompt at temperature 1.0.The MoE reward weight is α = 0.5, assigning equal importance to perception and reasoning experts; KL regularization is disabled with coefficient 0.0.
A.1 Training Details: RLVR starts from the alignment checkpoint and uses outcome-based training with global batch size 32 and N = 16 rollouts per prompt at temperature 1.0.The evaluated RL algorithms share the same hyperparameters.
A.1 Training Details: RLVR increases the maximum response length to 8192 tokens and trains for up to 1500 steps, selecting the best checkpoint by validation performance.Checkpoints are saved periodically during training.

A.2 Evaluation Details · A.3 Token Efficiency

Evaluations use a standardized lmms-eval/vLLM setup with fixed generation parameters and two-stage answer validation. On Qwen3-VL-4B, PRISM+GRPO achieves higher accuracy with fewer tokens than SFT+GRPO across MathVision, MathVerse, and MMMU-Pro.

A.2 Evaluation Details: All evaluations use the lmms-eval framework.The framework is paired with vLLM as the serving engine.
A.2 Evaluation Details: vLLM serves the evaluations across all benchmarks.
A.2 Evaluation Details: Generation uses temperature=1.0, top_p=0.7, and top_k=-1 throughout.
A.2 Evaluation Details: Answer extraction first applies a rule-based validator to minimize evaluation cost.
A.2 Evaluation Details: Unverified answers fall back to an LLM-as-judge validator powered by Qwen3-30B-A3B-Instruct.
A.3 Token Efficiency: PRISM+GRPO achieves higher accuracy than SFT+GRPO while using fewer tokens across MathVision, MathVerse, and MMMU-Pro.Figure 5 reports this comparison for Qwen3-VL-4B and characterizes the result as more concise and effective reasoning.

B Data Curation Pipeline … C.2 Cold-Start Data Samples

PRISM constructs a 113K verified multimodal reasoning dataset through source preparation, Gemini 3 Flash distillation, iterative filtering, and staged splitting. The resulting data supports SFT, alignment, and difficulty-matched RLVR, while structured prompts and examples enforce visual grounding, step-by-step reasoning, and final answers.

B Data Curation Pipeline: 113K verified samples are produced through source preparation, iterative generation with multi-stage filtering, and downstream data splitting.The source problems span mathematical reasoning, scientific diagrams, chart interpretation, and spatial reasoning.
B Data Curation Pipeline: Gemini 3 Flash is prompted to generate fine-grained visual descriptions, step-by-step reasoning traces, and concise final answers.The distillation template specifies the required reasoning and answer characteristics.
B Data Curation Pipeline: Responses pass truncation/failure, format, and correctness filters requiring valid <caption>, <think>, and <answer> sections and correct answers.An LLM-as-judge compares generated answers against ground truth.
B Data Curation Pipeline: Failed samples are regenerated, with correctness failures receiving the ground-truth answer in the prompt before repeating all three filters.This preserves the requirement for detailed reasoning while guiding the model toward correct solutions.
B Data Curation Pipeline: 6K of the 113K verified samples are held out for alignment and RL, while 107K are assigned to the SFT pool.The RLVR set further retains problems with pass rates in [0.2, 0.8] from N = 16 rollouts, yielding approximately 2K samples.
C.1 Prompts and Data Examples: PRISM uses a shared structured output format across SFT, RL training, and evaluation: <caption> for visual grounding, <think> for reasoning, and <answer> for the final response.The prompts are presented in Figure 6, while distillation and LLM-as-judge prompts are presented in Figures 7 and 8.
C Examples: The distillation prompt provides fine-grained rules for each output section to elicit detailed reasoning demonstrations from Gemini 3 Flash.A separate prompt supports LLM-as-judge evaluation.
C.2 Cold-Start Data Samples: Cold-start SFT examples illustrate visual grounding via <caption>, chain-of-thought reasoning via <think>, and final responses via <answer>.Figures 9 and 10 present representative examples using the structured three-part format.

C.3 Qualitative Data Samples

Qualitative rollouts after PRISM’s reinforcement learning stage illustrate the structured caption-think-answer reasoning pipeline. A shared system prompt enforces this three-part format during SFT, RL training, and evaluation.

Qualitative rollout examples: Figures 11 and 12 present representative post-PRISM reinforcement-learning rollouts.The examples show reasoning trajectories generated by the policy model.
Qualitative rollout examples: The rollouts demonstrate the structured caption-think-answer pipeline in practice.This format organizes the policy’s output into caption, reasoning, and answer stages.
Output-format enforcement: A shared system prompt enforces the <caption>, <think>, <answer> output format across SFT, RL training, and benchmark evaluation.The prompt is used consistently across all three stages.

D Full Training Procedure

PRISM uses a three-stage training pipeline: SFT initializes the policy, adversarial on-policy distillation aligns it with supervision, and the final stage switches the reward away from the discriminator. The alignment stage alternates discriminator and policy updates to reduce distributional mismatch.

Three-Stage Pipeline: PRISM proceeds through three sequential stages: SFT initialization, discriminator-based distribution alignment, and a final reward switch.Stage 1 trains an initial policy π_sft on the combined corpus; Stage 3 changes the reward from the discriminator.
Distribution Alignment: Stage 2 alternates MoE discriminator updates using Bradley-Terry loss with policy updates using GRPO and discriminator rewards.The discriminator contrasts supervision data with policy rollouts, while GRPO updates the policy.
Distribution Alignment: The alternating Stage 2 updates drive the policy distribution toward the supervision distribution.

E Limitations and Future Directions

PRISM adds training overhead through its alignment stage and MoE discriminator, despite running alignment for only 500 steps. Its current MoE design also depends on structured responses with explicit visual captions and reasoning traces to separate perception and reasoning feedback.

Limitations: 500 steps: PRISM’s alignment stage still increases memory and compute overhead because policy and discriminator are maintained jointly.This overhead exceeds that of a standard SFT→RLVR pipeline.
Limitations: Structured response formats: The current MoE design relies on explicit visual captions and reasoning traces to decompose perception and reasoning feedback.