Skill-Decoupled On-Policy Expert Distillation for Unified Vision-Language Models

1 INTRODUCTION

• Integrating complementary visual understanding capabilities across heterogeneous multimodal tasks remains a fundamental challenge in VLM post-training.Prior RL approaches improve specific visual understanding capabilities using task-specific rewards.Source: §1 · p. 1
• Directly aligning expert predictions can leave reusable strategies undistinguished from capability-specific patterns, including overfitting and reward-hacking behaviors.Joint optimization over diverse multimodal data can also produce capability-specific gradient conflicts.Source: §1 · pp. 1–2
• SCOPED independently trains capability-specific experts, extracts reusable solution patterns from their trajectories, and uses those skills to guide on-policy distillation into a unified student.The skills guide expert supervision during reasoning, rather than being supplied to the student at inference.Source: §1 · p. 2
• Across eight multimodal benchmarks, SCOPED enables an 8B model to achieve the highest average score among evaluated models, surpassing larger open-source and leading closed-source baselines.The reported analyses also identify benefits from expert distillation and skill augmentation, alongside conflicts in mixed-capability optimization.Source: §1 · p. 2

2 RELATED WORK

• Capability-specific RL specializes VLMs, while joint training across heterogeneous capabilities can create competing gradient signals that restrict specialized expertise.On-policy distillation and multi-teacher variants offer an alternative for integrating independently optimized experts, but leave trajectory-level reasoning patterns implicit.Source: §2 · p. 2
• Trajectory-derived skills can make execution experience explicit and reusable, but their use to regularize supervision from multiple independent experts during unified VLM integration remains underexplored.Prior examples include skill libraries, trajectory consolidation, multimodal tools, and skill-conditioned self-distillation.Source: §2 · pp. 2–3

3 METHODOLOGY

• SCOPED uses explicit skills as privileged directional anchors during on-policy distillation, structuring expert supervision while the student internalizes the guidance.The skills are not required for inference because regularized expert knowledge is embedded in the student’s weights.Source: §3 · p. 3, §3.2 · p. 5

3.1 LEARNING MLLM EXPERTISE WITH CAPABILITY-SPECIFIC REINFORCEMENT

• Each capability expert is independently optimized with its capability-specific reward objective, avoiding direct competition from other capabilities during expert training.The method partitions training data by capability and collects multiple expert rollouts to preserve concrete behavioral traces.Source: §3.1 · p. 3

3.2 UNIFYING CAPABILITY EXPERTS VIA SKILL-AUGMENTED DISTILLATION

• Reward-guided reflection contrasts successful and failed expert trajectories to extract reusable capability-specific solution patterns as explicit skills.The resulting skill conditions its expert’s supervision, guiding reasoning paths and suppressing erroneous or over-specialized tokens during distillation.Source: §3.2 · pp. 4–5
• The unified student is trained over all capability subsets using token-level reverse-KL distillation on student-generated responses, requiring no explicit skill conditioning at inference.The skills condition the expert distribution, while the student distribution is conditioned on its input and generated prefix.Source: §3.2 · p. 5

4 EXPERIMENTAL METHODOLOGY

• The evaluation spans eight benchmarks across fine-grained perception, OCR and chart understanding, symbolic visual reasoning, and multi-image reasoning.Capability-specific RL training data are drawn from six public multimodal datasets.Source: §4 · p. 5
• Accuracy is used for perception, chart understanding, symbolic visual reasoning, and multi-image tasks, while OCR uses micro-F1; the overall score is their eight-benchmark arithmetic mean..Source: §4 · p. 5
• SCOPED is compared with representative closed-source and open-source VLMs, including Claude, GPT, GLM, Qwen, InternVL, and MiMo models.The shared backbone is Qwen3-VL-8B-Instruct or Qwen3-VL-4B-Instruct.Source: §4 · p. 6

5.1 OVERALL PERFORMANCE

• SCOPED-8B improves Qwen3-VL on all eight benchmarks by 5.83 points on average and achieves the best overall performance among evaluated models, surpassing Qwen3-VL-32B.Source: §5.1 · pp. 6–7

5.2 ABLATION STUDY

• SCOPED gains 3.82 points over Mix-RL; on-policy distillation beats parameter merging, and skills add gains across four capability groups.Source: §5.2 · pp. 7–8

5.3 GRADIENT INTERFERENCE IN MULTI-CAPABILITY TRAINING

• Mix-RL trails independent RL in reward, aligns weakly or negatively with expert directions, and shows recurrent negative pairwise gradient cosine similarities.Source: Figure 2 · p. 8, §5.3 · p. 8

5.4 EFFECTIVENESS OF SKILL-AUGMENTED DISTILLATION IN SCOPED

• Skill augmentation improves expert-behavior retention and alignment, shifts within-capability trajectory similarity upward, and raises attention hit rates across most upper decoder layers.Source: Figure 3 · p. 9, §5.4 · p. 9

• SCOPED independently optimizes capability-specific experts and abstracts reusable solution patterns from reward-evaluated trajectories into explicit skills.Source: §6 · p. 10
• During on-policy distillation, skills guide capability experts in providing structured supervision for a unified student.Source: §6 · p. 10
• Experiments report consistent performance improvements, while analyses reveal mixed-optimization interference and better preservation and transfer of specialized expertise with skill augmentation.Source: §6 · p. 10

• Eight evaluation benchmarks cover fine-grained perception, OCR and chart understanding, symbolic visual reasoning, and multi-image reasoning.The reported evaluation sample counts reflect instances retained after preprocessing.Source: §A.1 · p. 15, Table 3 · p. 15
• The capability-specific RL corpus uses six public multimodal data sources organized into four capability-specific subsets.Source: §A.1 · p. 15, Table 4 · p. 15
• The resulting RL training corpus contains 37,999 samples after preprocessing and filtering.Source: §A.1 · p. 15, Table 4 · p. 15

A.2 ADDITIONAL IMPLEMENTATION DETAILS

• All experts and the unified student start from Qwen3-VL-8B-Instruct; additional model-scale experiments use Qwen3-VL-4B-Instruct.Expert training uses EasyR1, and skill-decoupled distillation uses veRL.Source: §A.2 · p. 15
• The reported settings include three training epochs, rollout batch size 128, actor global batch size 64, and learning rate 1 × 10−6.The listed configuration also includes eight responses per prompt, maximum prompt and response lengths of 8,192, and KL coefficient 1 × 10−2.Source: §A.2 · p. 16
• Experiments were conducted on NVIDIA A800 80GB GPUs, with rollout temperature 1.0, top-p 1.0, and bfloat16 training precision.The configuration also specifies tensor parallel size 2 and image pixel limits of 262,144 to 524,288.Source: §A.2 · p. 16
• The four capability experts share optimization and rollout settings, differing only in their capability-specific data and reward functions.During distillation, each expert receives its matching skill, while the student sees only the original multimodal input.Source: §A.2 · p. 16

A.3 REWARD DESIGN

• Textual perception uses token-level F1, while visual grounding parses labeled boxes and greedily matches same-label predictions to references by IoU.The grounding reward combines parse validity, label-aware detection F1 at IoU thresholds 0.5 and 0.75, and mean IoU from one-to-one matching.Source: §A.3 · p. 16
• OCR and chart-understanding tasks use token-level F1 after prediction normalization, with empty or non-overlapping predictions receiving zero reward.Boolean outputs are mapped to a unified yes/no representation before token matching.Source: §A.3 · p. 16
• Symbolic reasoning uses format-gated accuracy: malformed responses receive zero, and valid answers are matched by exact numeric equality or normalized text.Text predictions are extracted from the <answer> field when available, lowercased, and stripped of punctuation and articles.Source: §A.3 · pp. 16–17
• Multi-image spatial reasoning uses answer-type-specific matching, including option letters, image indices, canonical yes/no labels, normalized spatial relations, or containment.Malformed responses receive zero overall reward.Source: §A.3 · p. 17

A.4 ADDITIONAL ABLATION ON MODEL SCALE

• On Qwen3-VL-4B, SFT substantially underperforms the zero-shot backbone, while Mix-RL gives only a modest overall improvement and trails capability experts on several target capabilities.Independently trained experts achieve stronger performance on their respective capabilities.Source: §A.4 · p. 17
• At the 4B scale, matching each skill to its expert improves performance on the corresponding target capability, while non-target effects remain limited or mixed.This supports explicit skill-expert matching during distillation.Source: §A.4 · p. 17
• On Qwen3-VL-4B, full SCOPED yields the best average performance among the compared integration variants, with skill augmentation improving on SCOPED without it.SCOPED without skill augmentation also outperforms parameter merging.Source: §A.4 · pp. 17–18

A.5 SKILL CONSTRUCTION DETAILS

• Skills combine reward-grouped successful and failed expert trajectories into one capability-level guide, and using both sources yields the strongest overall performance.Before reflection, rollouts are deduplicated and balanced across task subtypes; successful and failed cohorts contribute effective behaviors and failure-avoidance guidance.Source: §A.5 · p. 18

A.6 DETAILS OF CROSS-CAPABILITY GRADIENT ANALYSIS

• The gradient analyses estimate capability-specific directions at shared Mix-RL parameters and compare their pairwise cosine similarities, with visual parameters frozen.Negative pairwise similarity indicates locally opposing optimization directions.Source: §A.6 · p. 19

A.7 CROSS-CAPABILITY GRADIENT INTERFERENCE IN MIX-RL

• Across capabilities, independently trained experts maintain positively aligned trajectories on average, while Mix-RL updates diverge across capabilities and show recurring gradient conflicts.The shared update advances some capability directions while opposing others; independent experts can still show local fluctuations.Source: §A.7 · p. 20, Figure 4 · p. 20

A.8 DETAILS OF EXPERT-RETENTION ANALYSIS

• Expert-retention metrics macro-average functional agreement with independently trained experts across four capabilities; they measure fixed-trajectory behavior, not downstream accuracy.Comparisons use fixed-trajectory teacher forcing so unified models are evaluated under identical response states.Source: §A.8 · p. 21

A.9 CAPABILITY-WISE EXPERT RETENTION AND SKILL ALIGNMENT

• SCOPED achieves higher expert-alignment and amplitude-retention measures across all four capabilities, while matched skills align guidance more closely with expert directions than random skills.Most matched-versus-random skill samples lie above the comparison diagonal.Source: Figure 5 · p. 21, §A.9 · p. 22

A.10 QUALITATIVE ANALYSIS OF SKILL-GUIDED VISUAL ATTENTION

• With a matched skill, a capability expert shifts more attention toward task-relevant visual evidence than it does without skill conditioning.The attention-change map provides a qualitative illustration consistent with the layer-wise attention analysis in the main text.Source: §A.10 · p. 22, Figure 6 · p. 22

A.11 CASE STUDIES

• In a chart-reasoning case, skill-augmented SCOPED identifies the correct age group while emphasizing supporting visual evidence and suppressing the erroneous trajectory.Other cases illustrate systematic counting, structural map comparison, and landmark-based movement inference across visual capabilities.Source: §A.11 · pp. 22–23, Figure 7 · p. 23

A.12 PROMPT EXAMPLES

• Construction prompts reflect on successful and failed trajectories before synthesizing a skill; evaluation uses benchmark-specific response formats, while students receive no explicit skill at inference.Capability experts, unlike the unified student at inference, are conditioned on their corresponding skill in ablation and analysis experiments.Source: §A.12 · p. 27